基于Transformer的预训练语言模型在自然语言处理中的应用研究  

Research on the Application of Transformer-based Pre-trained Language Models in Natural Language Processing

作　　者：易顺明[1] 许礼捷[1] 周洪斌[1] Yi Shunming;Xu Lijie;Zhou Hongbin(Shazhou Professional Institute of Technology,Zhangjiagang 215600,Jiangsu,China)

机构地区：[1]沙洲职业工学院电子信息工程系,江苏张家港215600

出　　处：《沙洲职业工学院学报》2022年第3期1-6,共6页Journal of Shazhou Professional Institute of Technology

基　　金：2022年江苏高校“青蓝工程”优秀教学团队培养项目(苏教师函[2022]29号)。

摘　　要：自然语言处理(NLP)研究从早期基于规则的方法转向基于特征分析的机器学习,再转向无需事先进行特征抽取的深度学习,发展很快,但因其需要的文本数据量越来越大,模型训练所需的标注工作量巨大,对算力的要求也越来越高,而难以被广泛应用。基于Transformer的预训练语言模型(T-PTLM)提供了一个新的研究和应用路径:通过大规模无标注文本数据广泛学习语言现象,使模型具有很强的通用性,然后将模型进行迁移和微调,在NLP的许多具体任务应用中均取得了很好的效果。The research on natural language processing(NLP) has developed rapidly from the early rule-based methods to the machine learning based on the feature analysis, and then to the deep learning without prior feature extraction. However, due to the increasing amount of text data required, the labeling work required for training the models is huge, and the requirements for computing power are getting higher and higher, difficult to be widely used. The transformer-based pre-trained language models(T-PTLM) provide a new research and application path: the extensive learning language phenomena through large-scale unlabeled text data makes the model highly versatile, then transferring and fine-tuning this model in many NLP applications has achieved great results.

关 键 词：深度学习 TRANSFORMER 预训练语言模型 自然语言处理 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]