大语言模型中文问答正确性对比实验研究——以ChatGPT 3.5、Claude 1.0和文心一言2.1为例  

A Comparative Experimental Study on the Accuracy of Chinese Question-answering in Large Language Models: Case Study of ChatGPT 3. 5, Claude 1. 0, and Wenxinyiyan 2. 1

作　　者：唐明伟[1] 陈宙 丁晗萱 朱翼 顾明辉 陈羽 Tang Mingwei;Chen Zhou;Ding Hanxuan;Zhu Yi;Gu Minghui;Chen Yu(School of Computer,Nanjing Audit University,Nanjing Jiangsu 211815)

机构地区：[1]南京审计大学计算机学院,江苏南京211815

出　　处：《情报探索》2024年第7期71-78,共8页Information Research

基　　金：教育部人文社会科学研究规划基金项目“面向情景推演的突发事件应急预案知识组织研究”(项目编号:23YJA870009);江苏高校哲学社会科学研究重大项目“面向应急决策的突发事件应急预案知识推理研究”(项目编号:2021SJZDA153);江苏高校“青蓝工程”;国家自然科学基金青年项目“行为科学视角下人类个体差异性对人机合作信任的影响研究”(项目编号:62006121)成果之一。

摘　　要：[目的/意义]对大语言模型中文问答正确性进行实验测评研究,为中文用户使用大语言模型提供一定的指导作用。[方法/过程]针对科技、教育、医学、生活、旅游美食和哲学文化6个领域,分别设计常识性、专业性和开放性三类问题,每类20个问题,共计360个问题。分别向ChatGPT 3.5、Claude 1.0和文心一言2.1提问,再针对回答进行正确性的人工评价。最后汇总评价结果,进行正确性的多方面对比分析。[结果/结论]实验分析表明中文语料数据的规模与质量,以及大语言模型的参数规模是影响大语言模型中文问答正确性的重要因素。[Purpose/significance]The paper conducts an experimental evaluation study on the accuracy of Chinese question-answering in large language models,aims to provide guidance for the Chinese users of large language models.[Method/process]Aiming at the six fields of science and technology,education,medicine,life,tourism and food,philosophy and culture,this paper designs three types of questions:common sense,professionalism and openness,20 questions in each category,a total of 360 questions.It asks questions to ChatGPT 3.5,Claude 1.0 and Wenxinyiyan 2.1 respectively,and then manually evaluates the correctness of the answers.Finally,the evaluation results are summarized and the correctness is compared and analyzed in many aspects.[Result/conclusion]The experimental analysis indicates that the scale and quality of Chinese corpus data and the parameter scale of large language models,are important factors influencing the accuracy of Chinese question-answering in large language models.

关 键 词：大语言模型 中文问答 实验研究 

分 类 号：G353[文化科学—情报学]