新闻类短文本聚类新方法的研究  

Research on a New Method of News Short Text Clustering

作　　者：傅承涛[1] 谢佳璇 牛永洁 FU Cheng-tao;XIE Jia-xuan;NIU Yong-jie(School of Continuing Education,Yan′an University,Yan′an 716000,China)

机构地区：[1]延安大学继续教育学院,陕西延安716000

出　　处：《延安大学学报（自然科学版）》2020年第4期67-71,76,共6页Journal of Yan'an University：Natural Science Edition

基　　金：国家社会科学基金项目(18BTQ042);延安大学继续教育教学改革研究专项(YDJZ2016-03);陕西省创新创业训练计划项目(S202010719082)。

摘　　要：针对短文本矢量化数据稀疏和维度高,基于词频统计语义缺失,传统聚类方法对非球面类别检测困难以及计算量大等问题,提出一种聚类新方法。该方法主要包含三个步骤:一对文本使用Doc2vec方法进行矢量化;二使用基于密度的聚类算法CFDP对矢量化后的文本数据进行聚类;三采用t-分布领域嵌入算法(t-SNE)将数据压缩到二维空间并进行可视化。并对聚类算法中关键参数进行了交叉测试,采用S_Dbw评价聚类的效果,得到CFDP算法最优的参数组合,最后对聚类效果进行了可视化,聚类结果与文本原始类别进行比较,F值达到89.24,聚类效果良好。Aiming at the sparseness and high dimensionality of short text vectorization data,the lack of semantics based on word frequency statistics,the difficulty of detecting aspheric categories and the large amount of calculation in traditional clustering methods,a new clustering method is proposed.The method mainly includes three steps:a pair of texts are vectorized using the doc2vec method;The second is to use the density-based clustering algorithm CFDP to cluster the vectorized text data;The third is to use the t-SNE algorithm to compress the data into two-dimensional space and visualize it.The key parameters of the clustering algorithm are cross-tested.S_Dbw is used to evaluate the effect of clustering,and the optimal parameter combination of the CFDP algorithm is obtained.Finally,the clustering effect is visualized,and the clustering result is compared with the original text categories.The F value reaches 89.24,and the clustering effect is good.

关 键 词：Doc2vec 数据压缩 密度 S_Dbw 聚类 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]