多重检验加权融合的短文本相似度计算方法  

Method for Calculating Short Text Similarity Using Multi-Check Weighted Fusion

作　　者：石彩霞 李书琴[1] 刘斌[1] SHI Caixia;LI Shuqin;LIU Bin(College of Information Engineering,Northwest A&F University,Yangling,Shaanxi 712100,China)

机构地区：[1]西北农林科技大学信息工程学院,陕西杨凌712100

出　　处：《计算机工程》2021年第2期95-102,共8页Computer Engineering

基　　金：中国博士后科学基金(2017M613216);陕西省自然科学基金(2017JM6059);陕西省重点研发计划(2019ZDLNY07);陕西省博士后基金(2016BSHEDZZ121)。

摘　　要：传统相似度计算方法仅考虑文本结构特征或语义信息,从而导致准确率较低。结合短文本特征稀疏的特性,提出一种多重检验加权融合的短文本相似度计算方法 MCWFS。使用基于改进编辑距离、考虑词频、基于Word2vec与LSTM的3种方法分别计算相似度,对满足多重检验标准的文本进行加权因子线性融合,以避免因一种相似度值过大或过小导致加权相似度值异常的问题。在此基础上,通过加权融合计算短文本相似度,使得计算结果更加准确合理。实验结果表明,相比层层检验和无检验融合方法,MCWFS方法的平均准确率分别提高16.01%和7.39%,且其F1值可达70.21%。Most of the existing similarity calculation methods consider only the text structure features or the semantic information,and thus reduce the accuracy. To address the problem,this paper proposes a method,MCWFS,for calculating short text similarity using multi-check weighted fusion to deal with the sparse features of short texts. The method calculates the similarity by using three methods:similarity calculation based on improved edit distance,semantic similarity calculation considering word frequency,and similarity calculation based on the Word2 vec and LSTM.Then weighted linear fusion is performed for texts that satisfy multi-check standards to avoid abnormal weighted similarity value caused by a single too large or too small similarity value.On this basis,weighted fusion is used to calculate the short text similarity to make the result more accurate and reasonable.Experimental results show that compared with the layer-by-layer check and non-check fusion methods,the proposed MCWFS method improves the average accuracy by16.01% and7.39% respectively,and its F1 value reaches70.21%.

关 键 词：短文本相似度 多重检验加权融合 编辑距离 语义信息 词频 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]