检索规则说明:AND代表“并且”;OR代表“或者”;NOT代表“不包含”;(注意必须大写,运算符两边需空一格)
检 索 范 例 :范例一: (K=图书馆学 OR K=情报学) AND A=范并思 范例二:J=计算机应用与软件 AND (U=C++ OR U=Basic) NOT M=Visual
作 者:王勇臻 Wang Yongzhen(School of Public Administration and Policy,Dalian University of Technology,Dalian 116024,China)
机构地区:[1]大连理工大学公共管理学院,辽宁大连116024
出 处:《科技管理研究》2024年第14期224-233,共10页Science and Technology Management Research
摘 要:论文同行评议的可靠性是科学界长期探讨却又悬而未决的议题。对此,选取目前广泛使用的两种评价者间信度指标,即组内相关系数(ICC)和卡帕系数(κ)来测度论文同行评议的可靠性,并通过元分析方法针对49项发表自1974至2022年间有关论文对同行评议场景下评价者间信度的实证研究进行量化归纳,继而开展亚组分析以考察情境和程序两大类共10种因素对论文同行评议可靠性的影响及其测度指标差异,以期拓展对论文同行评议质量的认识与理解。研究结果显示:论文同行评议的可靠性总体上不甚理想(ICC=0.3611,κ=0.1947);此外,论文同行评议的可靠性受评价对象、学科门类和录用难度(3种情境因素)以及盲法形式(单一程序因素)影响显著,而且当使用不同的测度指标对论文同行评议场景下的评价者间信度加以计量时,同一种因素所表现出来的影响模式可能大相径庭。The reliability of manuscript peer review(MPR)has long been a concerning issue for the scientific community.In this paper,two widely-used inter-rater reliability indicators,namely the intra-class correlation coefficient(ICC)and Cohen's kappa coefficient(κ),were selected to measure the reliability of MPR.A meta-analysis was conducted to provide a quantitative summary of 49 empirical studies published between the years 1974 and 2022 focusing on inter-rater reliability in the context of MPR,in order to expand the knowledge and understanding of the quality of MPR.After that,a series of subgroup analyses were carried out to investigate the effects of ten factors belonging to two categories,namely situational and procedural factors,on the reliability of MPR as well as how these effects differ between both indicators.The analytical results show that on the whole,the reliability of MPR is far from satisfactory(ICC=0.361,κ=0.195).More importantly,the reliability of MPR is significantly affected by the evaluation subjects,academic disciplines,acceptance rates(i.e.,three situational factors),and blind policies(i.e.,one procedural factor).In addition,when different indicators are used to measure the reliability of MPR,the effect pattern exhibited by the same factor can be quite different.
正在载入数据...
正在载入数据...
正在载入数据...
正在载入数据...
正在载入数据...
正在载入数据...
正在载入数据...
正在链接到云南高校图书馆文献保障联盟下载...
云南高校图书馆联盟文献共享服务平台 版权所有©
您的IP:216.73.216.171