文本词向量与预训练语言模型研究  

A Survey of Research on Word Vectors and Pretrained Language Models

作　　者：徐菲菲 冯东升 XU Feifei;FENG Dongsheng(School of Computer Science and Technology,Shanghai University of Electric Power,Shanghai 200090,China)

机构地区：[1]上海电力大学计算机科学与技术学院,上海200090

出　　处：《上海电力大学学报》2020年第4期320-328,共9页Journal of Shanghai University of Electric Power

基　　金：上海市自然科学基金(19ZR1420800)。

摘　　要：介绍了文本词向量及预训练语言模型的发展体系,系统整理并分析了其中重点方法的思想特点。首先,阐述了传统的文本词向量表征方法及基于语言模型的文本表征方法;然后,详述了预训练语言模型方法的研究进展,包括动态词向量的表征方法和基于Transformer架构的预训练模型;最后,指出了未来探究多模态间更有效的融合方式和迁移学习将成为该领域的发展趋势。This paper mainly introduces the development system of text word vectors and pre-trained language models,systematically organizes and analyzes the ideological characteristics of key methods.Firstly,we describe the traditional text word vector representation method and the language model-based text representation method,then we elaborate the research progress of the pre-trained language model method,including the dynamic word vector representation method and the Transformer architecture-based pre-training model.Finally,it is pointed out that in the future,exploring more effective fusion methods and transfer learning between multi-modalities will become a development trend in this field.

关 键 词：文本信息处理 词向量 预训练语言模型 Transformer架构 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]