新HSK书写成绩可靠性影响因素的概化理论分析  

Improving Dependability of New HSK Writing Test Scores:A Generalizability Theory Based Approach

作　　者：朱宇[1] 冯瑞龙[2] 辛涛[3] 

机构地区：[1]厦门大学海外教育学院,厦门361005 [2]北京师范大学香港浸会大学联合国际学院,珠海519085 [3]北京师范大学教育统计与测量研究所,北京100875

出　　处：《心理科学》2013年第2期479-483,共5页Journal of Psychological Science

基　　金：全国教育科学规划2009年度教育考试研究专项课题(GFA097004);北京师范大学香港浸会大学联合国际学院研究经费(CRG10-11/H/03)的资助

摘　　要：本研究以概化理论为视角,搜集了新HSK五级模拟书写题的作答和评分数据,估算了试题、评卷员、评阅速度效应及其交互效应的方差分量,考察了五级书写成绩的可靠性。基于概化理论和规划求解的数据分析发现了题量的调整方案及题型、题量、评卷人数的可行组合方案。本研究对评阅速度的分析属于前沿性的理论探索,而其他数据分析结果,则可能有益于旨在改进该测试质量的决策实践。Writing test scores of the New HSK, similar to those of any other language proficiency tests, are most vulnerable to reliability criticism. The study collected writing samples （n = 89） of two mock tests of writing of the New HSKS. Data analysis was performed from the perspective of Generalizability Theory. Variance components were estimated for effects of items, raters, and rating speeds. Phi was also estimated for various settings of the test. Major findings are： （a） according to the current test setting, the descending order of the Phi coefficients for each item type is the ordering of the inner- sentence components, writing based on the keywords given, and writing based on the photo given; （b） to keep Phi at least . 8 for each item type, ordering items needs to increase to 20 while each of the other two needs to increase 2 and 3 items ; （c） with current allocation of item quantities for each item type, if calculation of the comprehensive score of writing uses weight propor- tional to the raw scores, then the Phi coefficient for the writing test could marginally reach the level of. 74. The study explored various approaches reaching a Phi coefficient at least . 85 with relatively lower costs （for details, please refer to section 3.3.2 of this paper）. To do this, the analysis applied solver functions of Microsoft Excel ; （d） the study did not find a significant effect of rating speed. How- ever, this conclusion was limited to the two different speeds investigated： each rater＇ s comfortable speed, and a speed under which each rater felt a little rush but still had confidence about his/her rating quality. Effect of rating speeds needs to be investigated with more rigorous designs. The authors also called for more attention to reliability issues of writing test.

关 键 词：新HSK 写作测评 信度 概化理论 规划求解 

分 类 号：B842[哲学宗教—基础心理学]