再议论文同行评议的可靠性——来自元分析的新证据  

A Further Discussion on the Reliability of Manuscript Peer Review:New Evidence Based on Meta-Analysis

在线阅读下载全文

作  者:王勇臻 Wang Yongzhen(School of Public Administration and Policy,Dalian University of Technology,Dalian 116024,China)

机构地区:[1]大连理工大学公共管理学院,辽宁大连116024

出  处:《科技管理研究》2024年第14期224-233,共10页Science and Technology Management Research

摘  要:论文同行评议的可靠性是科学界长期探讨却又悬而未决的议题。对此,选取目前广泛使用的两种评价者间信度指标,即组内相关系数(ICC)和卡帕系数(κ)来测度论文同行评议的可靠性,并通过元分析方法针对49项发表自1974至2022年间有关论文对同行评议场景下评价者间信度的实证研究进行量化归纳,继而开展亚组分析以考察情境和程序两大类共10种因素对论文同行评议可靠性的影响及其测度指标差异,以期拓展对论文同行评议质量的认识与理解。研究结果显示:论文同行评议的可靠性总体上不甚理想(ICC=0.3611,κ=0.1947);此外,论文同行评议的可靠性受评价对象、学科门类和录用难度(3种情境因素)以及盲法形式(单一程序因素)影响显著,而且当使用不同的测度指标对论文同行评议场景下的评价者间信度加以计量时,同一种因素所表现出来的影响模式可能大相径庭。The reliability of manuscript peer review(MPR)has long been a concerning issue for the scientific community.In this paper,two widely-used inter-rater reliability indicators,namely the intra-class correlation coefficient(ICC)and Cohen's kappa coefficient(κ),were selected to measure the reliability of MPR.A meta-analysis was conducted to provide a quantitative summary of 49 empirical studies published between the years 1974 and 2022 focusing on inter-rater reliability in the context of MPR,in order to expand the knowledge and understanding of the quality of MPR.After that,a series of subgroup analyses were carried out to investigate the effects of ten factors belonging to two categories,namely situational and procedural factors,on the reliability of MPR as well as how these effects differ between both indicators.The analytical results show that on the whole,the reliability of MPR is far from satisfactory(ICC=0.361,κ=0.195).More importantly,the reliability of MPR is significantly affected by the evaluation subjects,academic disciplines,acceptance rates(i.e.,three situational factors),and blind policies(i.e.,one procedural factor).In addition,when different indicators are used to measure the reliability of MPR,the effect pattern exhibited by the same factor can be quite different.

关 键 词:同行评议 论文同行评议 可靠性 评价者间信度 元分析 

分 类 号:G230[文化科学] F224[经济管理—国民经济] G301

 

参考文献:

正在载入数据...

 

二级参考文献:

正在载入数据...

 

耦合文献:

正在载入数据...

 

引证文献:

正在载入数据...

 

二级引证文献:

正在载入数据...

 

同被引文献:

正在载入数据...

 

相关期刊文献:

正在载入数据...

相关的主题
相关的作者对象
相关的机构对象