大语言模型在中学历史学科中的应用测评分析  

Evaluation and Analysis of Large Language Models Application in of Historical Discipline Middle Schools

作　　者：申丽萍[1] 何朝帆 曹东旭 朱云彬 吴永和[4] SHEN Li-Ping;HE Chao-Fan;CAO Dong-Xu;ZHU Yun-Shan;WU Yong-He(Department of Computer Science and Engineering,Shanghai Jiao Tong University,Shanghai,China 200240;High School Affiliated to Shanghai Jiao Tong University,Shanghai,China 200439;No.2 Middle School Affiliated to Shanghai Jiao Tong University,Shanghai,China 200240;Department of Education Information Technology,East China Normal University,Shanghai,China 200062)

机构地区：[1]上海交通大学计算机科学与工程系,上海200240 [2]上海交大附属中学,上海200439 [3]上海交大第二附属中学,上海200240 [4]华东师范大学教育信息技术学系,上海200062

出　　处：《现代教育技术》2024年第2期62-71,共10页Modern Educational Technology

摘　　要：大语言模型一经发布便获得广泛关注,但其在实际应用特别是教育领域的应用还存在诸多局限与挑战,因此需要对大语言模型在中文语境下的能力与风险进行测评。基于此,文章首先收集整理了一个包括10万条客观选择题与10套中学主观题测试卷的中学历史数据集,并在以ChatGPT、GPT-4和讯飞星火为代表的大语言模型上测试了该数据集中题目的回答表现。然后,文章详细分析了测试结果,发现虽然当前大语言模型的突出能力在于能够产生完整且流畅的表达,但其在中学历史知识测试中仍远低于适龄学生的平均水平,大语言模型应用于教育领域仍存在可靠性较差、可信度较低、具有偏见与歧视、推理能力不足、无法自动更新知识等问题。最后,文章针对大语言模型在中文语境下教育领域的应用提出建议,以期助力大语言模型在教育领域发挥更大的作用,为学生、教师带来更好的学习和教学体验。Large language models(LLMs)have received wide attention since its release,while there are still many limitations and challenges in their practical application,especially in the field of education.Therefore,it is necessary to evaluate the capability and risk of LLMs in the Chinese context.Based on this,this paper firstly collected and sorted out a historical dataset for middle school students including more than 100,000 objective multiple choice questions and 10 sets of subjective questions,and tested the answer performances of the questions in the data set of the LLMs represented by ChatGPT,GPT-4 and IFLYTEK Spark.Then,the paper analyzed the test results in detail and found that although the outstanding ability of the current LLMs lay in its ability to produce complete and fluent expression,and its performance in the history knowledge test of middle school was still far below the average level of school-age students.The application of LLMs in education still had some problems:such as poor reliability,low credibility,prejudice and discrimination,insufficient reasoning ability and inability to update knowledge automatically.Finally,some suggestions were proposed for the application of LLMs in the field of education in the Chinese context,in order to help LLMs play a greater role in the educational field and bring better learning and teaching experience for students and teachers.

关 键 词：大语言模型 ChatGPT 讯飞星火 教育应用 测评 

分 类 号：G40-057[文化科学—教育学原理]