基于Transformer的自然语言处理预训练语言模型概述  

Overview of Transformer-based Pre-trained Language Models for Natural Language Processing

作　　者：史童月 王中卿[1] SHI Tongyue;WANG Zhongqing(School of Computer Science and Technology,Soochow University,Suzhou Jiangsu 215006,China)

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006

出　　处：《信息与电脑》2022年第10期52-56,共5页Information & Computer

基　　金：国家自然科学基金(项目编号:62076175,61976146);江苏省双创博士计划和苏州大学大学生课外学术科研基金(项目编号:KY20220735B)。

摘　　要：在自然语言处理领域,谷歌提出Transformer模型之后,以生成式预训练模型(Generative Pre-Training,GPT)和深度双向预训练语言模型(Bidirectional Encoder Representat ions from Transformers,BERT)等为代表的一些基于Transformer的预训练语言模型(Transformer-based Pre-trained Language Models,TPLM)相继被提出,在大部分自然语言处理任务上取得很好的效果。TPLM使用自监督学习方法学习来自大量文本数据的通用语言表示,并将这些知识转移到下游任务中,为其提供了背景知识,避免了重新开始训练新任务模型的情况。笔者主要研究了基于Transformer的预训练语言模型与基于TPLM的“预训练+微调”的自然语言处理预训练新技术模式。首先,介绍预训练模型的发展背景;其次,解释各种有关TPLM的核心概念,如预训练、预训练方法、预训练任务、自监督学习与迁移学习等;再次,简要介绍相关TPLM模型与目前的进展;最后,提出改进TPLM的方法并总结。In the field of natural language processing,after Google proposed the Transformer model,some Transformer-based Pretrained Language Models(TPLM) such as Generative Pre-Training(GPT) and Bidirectional Encoder Representations from Transformers(BERT) have been proposed one after another.TPLM use a self-supervised learning approach to learn common language representations from large amounts of text data,and transfer knowledge to downstream tasks,providing it with background knowledge and avoiding the situation of restarting training models for new tasks.In this overview,we mainly introduce TPLM and the TPLM-based "Pre-trained + Fine-tuning" new paradigm.Firstly,we introduce the development background of the pre-trained model.Secondly,we explain various core concepts of TPLM,such as pre-train,pre-trained methods,pre-trained tasks,self-supervised learning,and transfer learning.Thirdly,we briefly introduce some related TPLM and current progress.Finally,we point out several possible improvements,efforts,and conclusions.

关 键 词：TRANSFORMER 自然语言处理 预训练模型 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术] TP183[自动化与计算机技术—计算机科学与技术]