结合双预训练语言模型的中文文本分类模型  

Chinese text classification model based on dual pre-trained language model

作　　者：原明君 江开忠 YUAN Mingjun;JIANG Kaizhong(School of Mathematics,Physics and Statistics,Shanghai University of Engineering Science,Shanghai 201620,China)

机构地区：[1]上海工程技术大学数理与统计学院,上海201620

出　　处：《智能计算机与应用》2023年第7期1-6,14,共7页Intelligent Computer and Applications

基　　金：全国统计科学研究项目(2020LY080)。

摘　　要：针对Word2Vec等模型所表示的词向量存在语义模糊从而导致的特征稀疏问题,提出一种结合自编码和广义自回归预训练语言模型的文本分类方法。首先,分别通过BERT、XLNet对文本进行特征表示,提取一词多义、词语位置及词间联系等语义特征;再分别通过双向长短期记忆网络(BiLSTM)充分提取上下文特征,最后分别使用自注意力机制(Self_Attention)和层归一化(Layer Normalization)实现语义增强,并将两通道文本向量进行特征融合,获取更接近原文的语义特征,提升文本分类效果。将提出的文本分类模型与多个深度学习模型在3个数据集上进行对比,实验结果表明,相较于基于传统的Word2Vec以及BERT、XLNet词向量表示的文本分类模型,改进模型获得更高的准确率和F1值,证明了改进模型的分类有效性。To solve the problem of sparse features caused by Word2Vec models,a text classification method based on autocoding and generalized autoregressive pretrained language model is proposed.Firstly,BERT and XLNet are used to represent features such as polysemy,word location and relationship of the text respectively.Then,context features are extracted through Bi-directional Long Short-Term Memory(BiLSTM).Finally,features through self-attention and layer normalization of the two channel are fused to obtain the features closer to the original text and improve the effect of text classification.The proposed text classification model is compared with several deep learning models on three datasets.Experimental results show that compared with traditional text classification models based on Word2Vec,BERT and XLNet word vector representation,the improved model achieves higher accuracy and F_(1),which proves the validity of the improved model.

关 键 词：预训练语言模型 双向长短期记忆网络 自注意力机制 层归一化 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]