Scoring and Consequential Validity Evidence of Computer-and Paper-Based Writing Tests in Times of Change  

变革时代中计算机与纸质写作测试评分及结果效度证据的分析

作　　者：María GUAPACHA-CHAMORRO Orlando CHAVES-VARÓN 玛丽亚·瓜帕查-查莫罗;奥兰多·查韦斯-维伦(山谷大学,哥伦比亚)

机构地区：[1]Universidad del Valle,Colombia

出　　处：《Chinese Journal of Applied Linguistics》2024年第3期450-480,525,共32页中国应用语言学（英文）

摘　　要：Little is known about how the assessment modality,i.e.,computer-based(CB)and paper-based(PB)tests,affects language teachers’scorings,perceptions,and preferences and,therefore,the validity and fairness of classroom writing assessments.The present mixed-methods study used Shaw and Weir’s(2007)sociocognitive writing test validation framework to examine the scoring and consequential validity evidence of CB and PB writing tests in EFL classroom assessment in higher education.Original handwritten and word-processed texts of 38 EFL university students were transcribed to their opposite format and assessed by three language lecturers(N=456 texts,152 per teacher)to examine the scoring validity of CB and PB tests.The teachers’perceptions of text quality and preferences for assessment modality accounted for the consequential validity evidence of both tests.Findings revealed that the assessment modality impacted teachers’scorings,perceptions,and preferences.The teachers awarded higher scores to original and transcribed handwritten texts,particularly text organization and language use.The teachers’perceptions of text quality differed from their ratings,and physical,psychological,and experiential characteristics influenced their preferences for assessment modality.The results have implications for the validity and fairness of CB and PB writing tests and teachers’assessment practices.当前学界对于测试方式(如计算机测试[CB]和纸质测试[PB])如何影响语言教师的评分、认知和偏好,进而对课堂写作评估的有效性和公平性所产生的影响的研究尚少。本研究为混合方法研究,采用Shaw和Weir(2007)提出的社会认知测试验证框架,分析CB和PB写作测试在高等教育EFL(英语作为外语)课堂中的评分标准和结果效度。为了评估CB和PB测试的评分效度,我们将38名EFL大学生的原始手写和在文字处理器中输入的文本进行了誊写和对调,并由三位语言教师对总共456篇文本评分,每位教师负责152篇。经分析,我们发现教师对文本质量的认知和采用的评分方式,对结果效度具有重要影响。研究发现评价方式的选择对教师的评分标准、认知和个人偏好产生影响。教师对原始手写文本和誊写的手写文本都给予了较高评分,特别是在文本的组织架构和语言运用方面。研究还发现教师在评估文本质量时所持的观点与其实际评分存在偏差。这种偏差的产生受到教师个体生理、心理以及长期累积的教学经验等多重因素的影响。本研究发现在CB和PB写作测试的有效性、公平性,以及教师的评估实践领域均有重要意义。

关 键 词：scoring validity consequential validity CB and PB writing tests classroom assessment 

分 类 号：H319.3[语言文字—英语] G434[文化科学—教育学]