改进特征权重的短文本聚类算法  

Short Text Clustering Algorithm with Improved Feature Weight

作　　者：马存 郭锐锋[2] 高岑[2] 孙咏[2] MA Cun;GUO Rui-Feng;GAO Cen;SUN Yong(University of Chinese Academy of Sciences,Beijing 100049,China;Shenyang Institute of Computing Technology,Chinese Academy of Sciences,Shenyang 110168,China)

机构地区：[1]中国科学院大学,北京100049 [2]中国科学院沈阳计算技术研究所,沈阳110168

出　　处：《计算机系统应用》2018年第9期210-214,共5页Computer Systems & Applications

摘　　要：短文本的研究一直是自然语言处理领域的热门话题,由于短文本特征稀疏、用语口语化严重的特点,它的聚类模型存在维度高、主题聚焦性差、语义信息不明显的问题.针对对上述问题的研究,本文提出了一种改进特征权重的短文本聚类算法.首先,定义多因子权重规则,基于词性和符号情感分析构造综合评估函数,结合词项和文本内容相关度进行特征词选择;接着,使用Skip-gram模型(Continuous Skip-gram Model)在大规模语料中训练得到表示特征词语义的词向量;最后,利用RWMD算法计算短文本之间的相似度并将其应用K-Means算法中进行聚类.最后在3个测试集上的聚类效果表明,该算法有效提高了短文本聚类的准确率.Short text research has been a hot topic in the field of natural language processing. Due to the sparseness of short texts and serious colloquialisms, its clustering model has the problems of high dimensionality, poor focus of theme,and unclear semantic information. In view of the above problems, this study proposes a short text clustering algorithm with improving the feature weight. Firstly, the rules of multi-factor weight are defined, the comprehensive evaluation function is constructed based on part-of-speech and symbolic sentiment analysis, and the feature words are selected according to the relevancy between the term and the text content. Then, a word skip vector model（continuous skip-gram model） trained in large-scale corpus to obtain a word vector representing the semantic meaning of the feature words.Finally, the RWMD algorithm is used to calculate the similarity between short texts and the K-means algorithm is used to cluster them. The clustering results on the three test sets show that the algorithm effectively improves the accuracy of short text clustering.

关 键 词：特征权重 情感分析 词向量 RWMD距离 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]