结合语义和结构的短文本相似度计算  

Similarity Calculation of Short Text Combined with Semantic and Structure

作　　者：倪高伟 李涛[1] 刘峥[1] NI Gao-wei;LI Tao;LIU Zheng(School of Computer Science,Nanjing University of Posts and Telecommunications,Nanjing 210046,China)

机构地区：[1]南京邮电大学计算机学院,江苏南京210046

出　　处：《计算机技术与发展》2018年第8期104-108,113,共6页Computer Technology and Development

基　　金：2015年教育部-中国移动科研基金项目(5-10);江苏省自然科学基金面上项目(BK20171447);江苏省高校自然科学研究面上项目(17JKB520024)

摘　　要：短文本相似度不仅包括语义相似度,还包括语法相似度。目前在短文本相似度度量算法中,大多只分析短文本语义层次的相似性,往往忽略了短文本的语法结构对短文本相似度的重要影响,导致无法捕获大量的文本语义信息,同时在短文本分类任务中召回率不够理想。通过分析短文本的特征,将EMD(earth mover’s distance)求解线性规划中运输问题的最优解应用于度量两个短文本的相似度,用Word2Vec度量两个单词的语义相似性,提出了词序位置相似度的概念,即在计算短文本相似度的同时考虑语句词组顺序对相似度的贡献。实验结果表明,在捕获大量文本语义信息的基础上,将算法应用于k近邻(k-nearest neighbor,KNN)文本分类中,有较好的准确率和召回率。Short text similarity includes both semantic similarity and syntax similarity. At present,similarity calculation method of short text based on word2Vec mostly only analyzes the semantic similarity,but often ignores the important influence of syntactic structure on short text similarity. As a result,a large amount of text semantic information could not be captured,and the recall rate is not satisfactory in the classified task of the short text. With the analysis of the features of short text,the optimal solution of EMD （earth mover’s distance） in linear programming is used to measure the similarity of two short texts. Word2Vec is used to measure the semantic similarity of two words,and the concept of word order similarity is proposed which means calculating the short text similarity while considering the sentence phrase order contributing to the similarity. The experiment shows that the algorithm applied to k-nearest neighbor （KNN） text classification on the basis of capturing a large number of text semantic information achieves better accuracy and recall rate.

关 键 词：EMD Word2Vec 短文本相似度计算 语义相似度 词序结构 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]