ALICE:一种面向中文科技文本分析的预训练语言表征模型  

ALICE:A Pre-trained Language Representation Model for Chinese Technological Text Analysis

作　　者：王英杰 谢彬[1] 李宁波[1] WANG Yingjie;XIE Bin;LI Ningbo(The 32nd Research Institute of China Electronics Technology Group Corporation,Shanghai 201808,China)

机构地区：[1]中国电子科技集团公司第三十二研究所

出　　处：《计算机工程》2020年第2期48-52,58,共6页Computer Engineering

基　　金：国家部委基金

摘　　要：深度学习模型应用于自然语言处理任务时依赖大型、高质量的人工标注数据集。为降低深度学习模型对大型数据集的依赖,提出一种基于BERT的中文科技自然语言处理预训练模型ALICE。通过对遮罩语言模型进行改进并将其与命名实体级遮罩相结合,改善基础模型在下游任务中的表现,使其学习到的语言表征更贴合中文的语言特性。实验结果表明,与BERT模型相比,ALICE模型对于中文科技文本的分类准确率和命名实体识别的F1值分别提高1.2%和0.8%。The deep model of natural language processing rely on huge,high-quality and human-annotated dataset.In order to alleviate such dependency,this paper proposes a BERT-based natural language processing pre-trained model for Chinese technological text named ALICE.Improve Masked Language Model(MLM)and combine it with entity-level mask to boost the base model’s performance on downstream tasks,and let the learned representations fit Chinese trait much better.Experimental results show that,compared with the BERT model,ALICE model improves the classification accuracy of Chinese technological texts and the F1 value of named entity recognition by 1.2%and 0.8%,respectively.

关 键 词：预训练模型 迁移学习 BERT模型 文本分类 命名实体识别 自然语言推断 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]