开放式情境判断测验的自动化评分  

Automated scoring of open-ended situational judgment tests

作　　者：徐静 骆方 马彦珍 胡路明 田雪涛 XU Jing;LUO Fang;MA Yanzhen;HU Luming;TIAN Xuetao(School of Psychology,Beijing Normal University,Beijing 100875,China;Collaborative Innovation Center of Assessment toward Basic Education Quality,Beijing Normal University,Beijing 100875,China;Department of Psychology,School of Arts and Sciences,Beijing Normal University at Zhuhai,Zhuhai 519085,China)

机构地区：[1]北京师范大学心理学部,北京100875 [2]中国基础教育质量监测协同创新中心,北京100875 [3]北京师范大学珠海校区文理学院心理系,广东珠海519085

出　　处：《心理学报》2024年第6期831-844,共14页Acta Psychologica Sinica

基　　金：国家自然科学基金青年科学基金(62207002);国家自然科学基金面上项目(62377003);中国博士后科学基金特别资助(站前)(2022TQ0040);中国博士后科学基金面上资助(2022M720486)。

摘　　要：受限于评分成本,开放式情境判断测验难以广泛使用。本研究以教师胜任力测评为例,探索了自动化评分的应用。针对教学中的典型问题场景开发了开放式情境判断测验,收集中小学教师作答文本,采用有监督学习策略分别从文档层面和句子层面应用深度神经网络识别作答类别,卷积神经网络(ConvolutionalNeuralNetwork,CNN)效果理想,各题评分准确率为70%~88%,与人类评分一致性高,人机评分的相关系数r为0.95,二次加权Kappa系数(Quadratic Weighted Kappa,QWK)为0.82。结果表明,机器评分可以获得稳定的效果,自动化评分研究能够助力于开放式情境判断测验的广泛应用。Situational Judgment Tests(SJTs)have gained popularity for their unique testing content and high face validity.However,traditional SJT formats,particularly those employing multiple-choice(MC)options,have encountered scrutiny due to their susceptibility to test-taking strategies.In contrast,open-ended and constructed response(CR)formats present a propitious means to address this issue.Nevertheless,their extensive adoption encounters hurdles primarily stemming from the financial implications associated with manual scoring.In response to this challenge,we propose an open-ended SJT employing a written-constructed response format for the assessment of teacher competency.This study established a scoring framework leveraging natural language processing(NLP)technology to automate the assessment of response texts,subsequently subjecting the system's validity to rigorous evaluation.The study constructed a comprehensive teacher competency model encompassing four distinct dimensions:student-oriented,problem-solving,emotional intelligence,and achievement motivation.Additionally,an open-ended situational judgment test was developed to gauge teachers'aptitude in addressing typical teaching dilemmas.A dataset comprising responses from 627 primary and secondary school teachers was collected,with manual scoring based on predefined criteria applied to 6,000 response texts from 300 participants.To expedite the scoring process,supervised learning strategies were employed,facilitating the categorization of responses at both the document and sentence levels.Various deep learning models,including the convolutional neural network(CNN),recurrent neural network(RNN),long short-term memory(LSTM),C-LSTM,RNN+attention,and LSTM+attention,were implemented and subsequently compared,thereby assessing the concordance between human and machine scoring.The validity of automatic scoring was also verified.This study reveals that the open-ended situational judgment test exhibited an impressive Cronbach's alpha coefficient of 0.91 and demonstrated a goo

关 键 词：情境判断测验 自动化评分 教师胜任力 开放式测验 机器学习 

分 类 号：B848[哲学宗教—心理学]