ChatGPT与AES系统对大学英语写作的反馈效度比较  

Comparison of Feedback Validity in College English Writing Between ChatGPT and AES Systems

作　　者：殷小娟 林庆英 YIN Xiaojuan;LIN Qingying(College of Foreign Languages,Minjiang University,Fuzhou,Fujian 350108,China)

机构地区：[1]闽江学院外国语学院,福建福州350108

出　　处：《闽江学院学报》2024年第3期78-92,共15页Journal of Minjiang University

基　　金：闽江学院校级公共基础课程建设项目(MJU2022KC102)。

摘　　要：以45篇非英语专业大学生的英语作文为样本,比较了ChatGPT和国内典型3个作文自动评分系统(AES)的评分效度,并通过案例分析比较ChatGPT和AES系统代表批改网在整体评价、纠错反馈、篇章结构和观点合理性评估能力上的差异。研究发现,ChatGPT和3个AES系统评分平均都低于教师评分,且ChatGPT的评分最低;ChatGPT与教师评分结果呈中等正相关,但它与教师评分相邻吻合一致性不如3个AES系统。ChatGPT在内容反馈上的优势较为明显,其整体评价更加全面、个性化,错误识别更加准确,修改建议更加直接,对篇章结构和观点内容的判断能力显著优于AES系统,但也存在评分不稳定、提示语理解偏差和“类人化”惰性表现等局限性。通过了解ChatGPT的优势和局限性,有利于其更科学地赋能英语写作教学。Based on 45 English essays written by non-English major university students as samples,this article compares the scoring validity of ChatGPT and three typical domestic Automated Essay Scoring(AES)systems.Simultaneously,through case analysis,it compares the differences between ChatGPT and a representative AES system(Pigai)in terms of overall evaluation,corrective feedback,discourse structure,and logical reasoning assessment abilities.The research reveals that both ChatGPT and the three AES systems have lower average scores compared to the teacher’s grading,with ChatGPT having the lowest score.ChatGPT shows a moderate positive correlation with the teacher’s grading,but its consistency with manual grading is not as high as the three AES systems.ChatGPT demonstrates relatively significant advantages in providing feedback on essay content:its overall evaluation is more comprehensive and personalized,error identification is more accurate,modification suggestions are more direct,and its ability to assess discourse structure and content perspective surpasses that of AES systems.However,ChatGPT has limitations of unstable scoring,challenges in understanding prompts,and occasional laziness like humans.Understanding these strengths and limitations of ChatGPT contributes to a more scientifically informed application of its capabilities in English writing instruction.

关 键 词：大学英语写作 ChatGPT 作文自动评分系统 评分效度 内容反馈 

分 类 号：H319.3[语言文字—英语]