基于ChatGPT的多视角学术论文实体识别:性能测评与可用性研究  

ChatGPT-Based Scientific Paper Entity Recognition:Performance Measurement and Availability Research

作　　者：张颖怡 章成志[2] 周毅[1] 陈必坤 Zhang Yingyi;Zhang Chengzhi;Zhou Yi;Chen Bikun(School of Sociology,Soochow University,Suzhou 215123,China;School of Economics&Management,Nanjing University of Science and Technology,Nanjing 210094,China)

机构地区：[1]苏州大学社会学院,苏州215123 [2]南京理工大学经济管理学院,南京210094

出　　处：《数据分析与知识发现》2023年第9期12-24,共13页Data Analysis and Knowledge Discovery

基　　金：国家自然科学基金项目(项目编号:72074113);苏州大学人文社会科学青年交叉研究团队项目的研究成果之一。

摘　　要：【目的】分析基于大规模语言模型的提示学习方法在学术论文实体识别任务上的可用性。【方法】以ChatGPT这一大规模语言模型为例,将ChatGPT视为实体识别工具、伪标签生成工具以及训练数据生成工具,从性能、价格和时间等维度出发分析以上三个视角下ChatGPT的可用性。【结果】三个视角下基于ChatGPT的方法的F1值高于少量样本训练得到的神经网络基线模型,比如实体识别工具视角的F1宏平均值超过10个学术论文人工标注摘要训练得到的模型21.4个百分点。基于ChatGPT的方法在不同学科领域的学术论文数据集上性能较稳定。【局限】仅在英文学术论文摘要数据集上展开实验,但中文与英文学术论文、学术论文摘要与全文存在逻辑结构和表述上的差异。【结论】当缺少人工标注数据时,将ChatGPT视为实体识别工具可从学术论文摘要中识别出部分实体,但识别结果需进一步过滤以应用到下游任务中。[Objective]This paper aims to use a large language model for entity recognition tasks of academic papers.[Methods]We utilized ChatGPT,a large language model,as an entity recognition tool,a pseudo-label generation tool,and a training set generation tool.Then,we analyzed ChatGPT's performance,price,and time for the tasks.[Results]The F1 of the ChatGPT-based method in all three perspectives is higher than that of the neural network baseline model trained with a small dataset.For example,the F1 from the perspective of entity recognition was 21.4%higher than the model trained by manually annotating 10 abstracts.The ChatGPT-based methods had stable performance on academic paper datasets in different disciplines.[Limitations]We only examined the new method with English academic paper abstract datasets.More research is needed to examine it with the Chinese datasets.[Conclusions]ChatGPT can identify entities from academic paper abstracts with little manually annotated data.The recognition results need to be further filtered to be applied to downstream tasks.

关 键 词：ChatGPT AIGC 学术文本信息抽取 学术实体抽取 

分 类 号：G350[文化科学—情报学] TP391[自动化与计算机技术—计算机应用技术]