一种结合TF-IDF方法和词向量的短文本聚类算法  

Short text clustering based on TF⁃IDF and word embedding

作　　者：赵晓平 黄祖源 黄世锋 王永和 ZHAO Xiaoping;HUANG Zuyuan;HUANG Shifeng;WANG Yonghe(Information Center,Yunnan Power Grid Co.,Ltd.,Kunming 650011,China;Yunnan Yundian Tongfang Technology Co.,Ltd.,Kunming 650220,China)

机构地区：[1]云南电网有限责任公司信息中心,云南昆明650011 [2]云南云电同方科技有限公司,云南昆明650220

出　　处：《电子设计工程》2020年第21期5-9,共5页Electronic Design Engineering

基　　金：国家自然科学基金青年项目(61702442)。

摘　　要：随着在线社交网络平台(微信、微博等)和APP(网易、学习强国)的快速发展和应用,产生了海量短文本。针对这些海量短文本,传统的文本聚类方法存在聚类性能较差的问题。本文融合TF-IDF方法和词向量,提出了一种短文本聚类方法。首先,使用TF-IDF方法,提取短文本中TFIDF值靠前的TOP-N关键词作为短文本的特征词集合;其次,在Word2Vec工具的支持下,使用Skip-gram模型在海量语料中训练得到特征词的向量表示;最后,使用WMD距离计算短文本间的相似度。将所提方法应用于4个数据集,实验结果表示,该方法比传统的文本聚类算法具有更好的效果。With the rapid development and application of online social network platforms(wechat,Weibo,etc.)and APP(Netease,learning power),a large number of short texts have been generated.For the massive short text,the traditional text clustering method has the problem of poor clustering performance.In this paper,we propose a short text clustering algorithm based on TF⁃IDF and word embedding.Firstly,TF⁃IDF method is used to extract TOP⁃N keywords in short texts,which are the TOP⁃N of TF⁃IDF value,as the feature words for these texts.Secondly,with the support of Word2Vec tool,Skip gram model is used to gain the vector representations of feature words by training in large scale corpus.Finally,WMD distance is used to calculate the similarity between two short texts.The experiments show that our algorithm has better effect than the traditional text clustering algorithms.

关 键 词：文本聚类 短文本 TF-IDF 词向量 自然语言处理 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]