ChatGPT中文信息抽取能力测评——以三种典型的抽取任务为例  

Extracting Chinese Information with ChatGPT:An Empirical Study by Three Typical Tasks

作　　者：鲍彤 章成志[1] Bao Tong;Zhang Chengzhi(School of Economics&Management,Nanjing University of Science and Technology,Nanjing 210094,China)

机构地区：[1]南京理工大学经济管理学院,南京210094

出　　处：《数据分析与知识发现》2023年第9期1-11,共11页Data Analysis and Knowledge Discovery

基　　金：国家自然科学基金项目(项目编号:72074113)的研究成果之一。

摘　　要：【目的】评估ChatGPT在中文命名实体识别、关系抽取以及事件抽取等典型中文信息抽取任务中的性能,分析不同任务和领域ChatGPT的表现差异,给出ChatGPT中文场景下的使用建议。【方法】采用Prompt提示的方式,分别依据精确匹配和宽松匹配两种方式,测评ChatGPT在三个典型信息抽取任务、共7个数据集上的性能:在MSRA、Weibo、Resume和CCKS2019数据集评估ChatGPT的命名实体识别效果,并与GlyceBERT和ERNIE3.0模型对比;在FinRE和SanWen数据集测试ChatGPT与ERNIE3.0 Titan的关系抽取效果;在CCKS2020数据集测试ChatGPT与ERNIE3.0的事件抽取效果。【结果】ChatGPT在命名实体识别任务中的表现不及GlyceBERT和ERNIE3.0模型。在关系抽取任务中,ERNIE3.0 Titan优于ChatGPT。在事件抽取任务中,ChatGPT在宽松匹配下的表现优于ERNIE3.0。【局限】以Prompt提示的方式评估ChatGPT的性能表现存在主观性,不同的Prompt会产生效果差异。【结论】ChatGPT在典型的中文信息抽取任务上的表现还有很大改进空间,用户在使用过程中需选择合适的Prompt和问题。[Objective]This paper evaluates the performance of typical Chinese information extraction tasks such as named entity recognition,relationship extraction,and event extraction with ChatGPT.It also analyzes the performance differences of ChatGPT in different tasks and domains,which provides recommendations for ChatGPT in Chinese contexts.[Methods]We used manual prompts to evaluate the test results with exact matching or loose matching on three typical information extraction tasks across seven datasets.We evaluated the named entity recognition of ChatGPT on MSRA,Weibo,Resume,and CCKS2019 datasets and compared it with GlyceBERT and ERNIE3.0 models.We extracted the relationships with ChatGPT and ERNIE3.0 Titan on FinRE and SanWen datasets.We ran the event extraction of ChatGPT and ERNIE3.0 on the CCKS2020 dataset.[Results]In the named entity recognition task,ChatGPT was outperformed by GlyceBERT and ERNIE3.0 models.ERNIE3.0 Titan was also superior to ChatGPT significantly in the relationship extraction task.In the event extraction task,ChatGPT’s performance was slightly better than ERNIE3.0 under loose matching.[Limitations]The evaluation of ChatGPT’s performance using prompts is subjective,and different prompts may lead to different results.[Conclusions]ChatGPT needs to improve its performance on typical Chinese information extraction tasks,and users should choose appropriate prompts for better results.

关 键 词：ChatGPT 信息抽取 中文信息处理 预训练语言模型 

分 类 号：TP391[自动化与计算机技术—计算机应用技术] G350[自动化与计算机技术—计算机科学与技术]