基于词向量和EMD距离的短文本聚类  

Short text clustering based on word embeddings and EMD

作　　者：黄栋[1] 徐博[1] 许侃[1] 林鸿飞[1] 杨志豪[1] 

机构地区：[1]大连理工大学信息检索研究室,辽宁大连116023

出　　处：《山东大学学报（理学版）》2017年第7期66-72,共7页Journal of Shandong University(Natural Science)

基　　金：国家自然科学基金资助项目(61572102;61602078;61562080);国家高技术研究发展计划(863)资助项目(2006AA01Z151);辽宁省自然科学基金资助项目(201202031;2014020003);教育部留学回国人员科研启动基金;高等学校博士学科点专项科研基金资助课题(20090041110002);中央高校基本科研业务费专项资金资助

摘　　要：短文本聚类在数据挖掘中发挥着重要的作用,传统的短文本聚类模型存在维度高、数据稀疏和缺乏语义信息等问题,针对互联网短文本特征稀疏、语义存在奇异性和动态性而导致的短文本聚类性能较差的问题,提出了一种基于特征词向量的文本表示和基于特征词移动距离的短文本聚类算法。首先使用Skip-gram模型(Continuous Skip-gram Model)在大规模语料中训练得到表示特征词语义的词向量;然后利用欧式距离计算特征词相似度,引入EMD(Earth Mover's Distance)来计算短文本间的相似度;最后将其应用到Kmeans聚类算法中实现短文本聚类。在3个数据集上进行的评测结果表明,效果优于传统的聚类算法。Short text clustering plays an important role in data mining. The traditional short text clustering model has some problems, such as high dimensionality,sparse data and lack of semantic information. To overcome the shortcomings of short text clustering caused by sparse features ,semantic ambiguity ,dynamics and other reasons, this paper presents a feature based on the word embeddings representation of text and short text clustering algorithm based on the moving distance of the characteristic words. Initially, the word embeddings that represents semantics of the feature word was gained through training in large-scale corpus with the Continous Skip-gram Model. Furthermore, use the Euclidean distance calculation feature word similarity. Additionally, EMD （Earth Mover＇s Distance） was used to calculate the similarity between the short text. Finally, apply the similarity between the short text to Kmeans clustering algorithm implemented in the short text clustering. The evaluation results on three data sets show that the effect of this method is superi- or to traditional clustering algorithms.

关 键 词：短文本 EMD距离 词向量 相似度计算 聚类 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]