多注意力机制BERT分类模型及其在电力运维中的应用  

Multi-attention mechanism-based BERT classification model and its application in electric power operation and maintenance

作　　者：王文娟 何晓莲 胡峰[2] 赵伟[3] 钟淘淘 WANG Wenjuan;HE Xiaolian;HU Feng;ZHAO Wei;ZHONG Taotao(Control Center of Information and Communications Branch,State Grid Chongqing Electric Power Company,Chongqing 401121,P.R.China;Chongqing Key Laboratory of Computational Intelligence,Chongqing Lniversity of Posts and Telecommunications,Chongqing 400065,P.R.China;Office of International Cooperation and Exchanges,Chongqing Lniversity of Posts and Telecommunications,Chongqing 400065,P.R.China;Technical Development Department of Information and Telecommunication Branch,State Grid Chongqing Electric Power Company,Chongqing 401121,P.R.China)

机构地区：[1]国网重庆市电力公司信息通信分公司调控中心,重庆401121 [2]重庆邮电大学计算智能重庆市重点实验室,重庆400065 [3]重庆邮电大学国际合作与交流处,重庆400065 [4]国网重庆市电力公司信息通信分公司技术发展部,重庆401121

出　　处：《重庆邮电大学学报（自然科学版）》2023年第1期156-163,共8页Journal of Chongqing University of Posts and Telecommunications(Natural Science Edition)

基　　金：国家自然科学基金(61772096);国家重点研发计划资助项目(2018YFC0832100,2018YFC0832102)。

摘　　要：为了提高变换网路中双向编码表示(bidirectional encoder representations from transformers,BERT)在文本分类中的应用效果,针对基于BERT的文本分类模型在注意力机制设计中无法关注文本中重点信息的问题,提出了一种基于多注意力机制的BERT分类模型。对BERT模型框架中后四层的每一层输入向量,设计词向量注意力机制,捕捉向量表中每一个词的重要性;对得到的BERT模型框架中后四层,设计层向量注意力机制,将这四层上由自适应权重计算得到的每一层输出向量进行融合,获得最终的BERT模型输出向量,更好地提升模型的特征抽取能力。在公开数据集IMDB和THUCNews上的实验表明,提出的模型相较于其他基线模型性能有明显提升。在电力系统运维项目管理的实际应用中,该模型也取得了比基线模型更好的效果,较好地解决了电力运维规模预测混乱问题。Aiming to solve the problem that text classification models based on BERT cannot focus on the importance information of text,we propose a BERT classification model based on multi-attention mechanism to solve the problem that BERT based text classification model cannot pay attention to the key information in the text in the attention mechanism design.For each input vector of the last four layers in the BERT framework,a word vector attention mechanism is designed to capture the importance of each word in the vector table.For the last four layers of the obtained BERT model framework,the layer vector attention mechanism is designed,and the output vectors of each layer calculated by adaptive weight on these four layers are fused to obtain the final BERT model output vector,which better improves the feature extraction ability of the model.Experiments on the public datasets IMDB and THUCNews show that the performance of the proposed model is significantly improved compared with other baseline models.In the practical application of power system operation and maintenance project management,the model has also achieved better results than the baseline model,and better solved the problem of confusion in power operation and maintenance scale prediction.

关 键 词：分类 多注意力机制 电力运维系统 特征抽取 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]