基于BERT的长文本分类方法  

BERT-based approach for long document classification

作　　者：刘博[1] 蒲亦非[1] LIU Bo;PU Yi-Fei(College of Computer Science,Sichuan University,Chengdu 610065,China)

机构地区：[1]四川大学计算机学院,成都610065

出　　处：《四川大学学报（自然科学版）》2023年第2期75-82,共8页Journal of Sichuan University(Natural Science Edition)

基　　金：国家自然科学基金面上项目(62171303);中国兵器装备集团(成都)火控技术中心项目(非密)(HK20-03);国家重点研发项目(2018YFC0830300)。

摘　　要：由于预训练模型输入分词数量限制,基于BERT的长文本分类任务效果与长文本分割后的文本段集合的处理及特征融合密切相关,现有的长文本分类研究在融合文本段特征时更关注文本段之间原始的顺序关系,而本文提出了一种基于BERT和集合神经网络的长文本分类模型.该方法以BERT为基础,可处理从同一文本样本分割得到的任意数量文本段,经过BERT后得到文本段特征,再将所有文本段特征输入到具有置换不变性的集合神经网络层中,提取出集合级别特征来优化长文本的特征表达.通过在三个数据上的实验分析,论文在平均分词长度较长的数据集上取得了90.82%的准确率,高出目前最优方法4.37%.Concerning the input limitation of pre-training model,a long document needs to be spit into a set of text segments.The performance of long document classification is closely related to the further processing of the segment set and feature fusion.Existing document classification models keep more attention on the sequential of segments in the text segment set.However,the athors consider that the sequential order of segments have a mild influence on drawing the feature of a long document.The authors propose a BERT based long document classification model,which utilizes deep sets to obtain the collection-level feature from the segment set.In the model,the authors obtain a set of text segment features after BERT,and this proposed network which is immune to permutation learns the identical feature of the set to represent the long document feature.The accuracy of our model on the 20 newsgroups dataset achieved 90.82%,which outperforms the state-of-the-art method by 4.37%.

关 键 词：文本分类 BERT 集合神经网络 长文本 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]