加入自注意力机制的BERT命名实体识别模型  

BERT named entity recognition model with self-attention mechanism

作　　者：毛明毅[1] 吴晨 钟义信[2] 陈志成[2] MAO Mingyi;WU Chen;ZHONG Yixin;CHEN Zhicheng(School of Computer and Information Engineering,Beijing Technology and Business University,Beijing 100048,China;School of Computer,Beijing University of Posts and Telecommunications,Beijing 100876,China)

机构地区：[1]北京工商大学计算机与信息工程学院,北京100048 [2]北京邮电大学计算机学院,北京100876

出　　处：《智能系统学报》2020年第4期772-779,共8页CAAI Transactions on Intelligent Systems

基　　金：北京市自然科学基金项目(4202016).

摘　　要：命名实体识别属于自然语言处理领域词法分析中的一部分,是计算机正确理解自然语言的基础。为了加强模型对命名实体的识别效果,本文使用预训练模型BERT(bidirectional encoder representation from transformers)作为模型的嵌入层,并针对BERT微调训练对计算机性能要求较高的问题,采用了固定参数嵌入的方式对BERT进行应用,搭建了BERT-BiLSTM-CRF模型。并在该模型的基础上进行了两种改进实验。方法一,继续增加自注意力(self-attention)层,实验结果显示,自注意力层的加入对模型的识别效果提升不明显。方法二,减小BERT模型嵌入层数。实验结果显示,适度减少BERT嵌入层数能够提升模型的命名实体识别准确性,同时又节约了模型的整体训练时间。采用9层嵌入时,在MSRA中文数据集上F1值提升至94.79%,在Weibo中文数据集上F1值达到了68.82%。Named entity recognition is a part of lexical analysis in the field of natural language processing.It is the basis for a computer to correctly understand natural language.In order to strengthen the recognition effect of the model on named entities,in this study,the pre-trained model BERT(bidirectional encoder representation from transformers)was used as the embedding layer of the model,and fixed parameter embedding was adopted to solve the problem of high computer performance required for BERT fine-tuning training.A BERT-BiLSTM-CRF model was built,and on the basis of this model,two improved experiments were carried out.Method one is to continue to add a self-attention layer.Experimental results show that the addition of the self-attention layer does not significantly improve the recognition effect of the model.Method two is to reduce the number of embedding layers of the BERT model.Experimental results show that moderately reducing the number of BERT embedding layers can improve the model’s named entity recognition accuracy,while saving the overall training time of the model.When using 9-layer embedding,thevalue on the MSRA Chinese data set increased to 94.79%,and thevalue on the Weibo Chinese data set reached 68.82%.

关 键 词：命名实体识别 BERT 自注意力机制 深度学习 条件随机场 自然语言处理 双向长短期记忆网络 序列标注 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]