大型预训练语言模型基础逻辑能力测评研究  

Research on the evaluation of basic logic abilityof large-scale pre-trained language models

作　　者：倪睿康 肖达 高鹏 NI Ruikang;XIAO Da;GAO Peng(School of Cyber Science and Engineering,Qufu Normal University,273165,Qufu,Shandong;School of Cyberspace Security,Beijing University of Posts and Telecommunications,100876,Beijing,PRC)

机构地区：[1]曲阜师范大学网络空间安全学院,山东省曲阜市273165 [2]北京邮电大学网络空间安全学院,北京市100876

出　　处：《曲阜师范大学学报（自然科学版）》2024年第3期89-95,共7页Journal of Qufu Normal University(Natural Science)

基　　金：中国博士后科学基金(2023M732022);山东省自然科学基金(ZR2021QF061);曲阜师范大学科研基金(167/602801).

摘　　要：针对数量问题、集合关系、量词问题和常识推理等4类基本逻辑推理问题,构建少样本学习模板,对数据集进行自动生成扩展,设计了11个逻辑推理子任务.采用语境学习和提示微调2种少样本学习方法,从模型、测试、任务3个维度测试了GPT-Neo-1.3B、GPT-J-6B、GPT-3-Curie、GPT-3-Davinci等模型的逻辑推理能力.结果表明,GPT-3模型在数量问题、量词问题和常识推理问题方面相对优秀,GPT-Neo与GPT-J模型在集合关系问题上更具优势.相较于语境学习,对预训练模型进行提示微调能显著提升预测能力.For four basic logical reasoning abilities of quantity problem,set relationship,quantifier problem and common sense reasoning,we construct few-shot learning sample templates for few-sort learning,which contain 11 logical reasoning subtasks.Two few-shot learning methods of in-context learning and prompt tuning are used to test the logical reasoning ability of GPT-Neo-1.3B and other models from the three dimensions of model,test method and task.The experimental results show that GPT-3 is relatively excellent in quantity problem,quantifier problem and common sense reasoning problem,GPT-Neo and GPT-J have more advantages in set-relation problem.Compared with in-context learning,the pre-trained models can significantly improve the prediction ability by prompt tuning.

关 键 词：自然语言处理 预训练语言模型 语境学习 提示微调 少样本学习 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]