基于MapReduce的文本层次聚类并行化  

Parallel text hierarchical clustering based on MapReduce

作　　者：余晓山 吴扬扬[1] 

机构地区：[1]华侨大学计算机科学与技术学院,福建厦门361021

出　　处：《计算机应用》2014年第6期1595-1599,1680,共6页journal of Computer Applications

基　　金：福建省科技计划重大项目(2011H6016);福建省科技计划重点项目(2011H0028)

摘　　要：针对传统的层次聚类算法在处理大规模文本时可扩展性不足的问题,提出基于MapReduce编程模型的并行化文本层次聚类算法。将基于文本向量分量组特征统计的垂直数据划分算法应用于MapReduce的数据分发,将MapReduce的排序特性应用于合并点的选择,使得算法更加高效,同时有利于提高聚类精度。实验结果表明了利用该算法进行大规模文本聚类的有效性及良好的可扩展性。Conceming the deficiency in scalability of the traditional hierarchical clustering algorithm when dealing with large-scale text, a parallel hierarchical clustering algorithm based on the MapReduce programming model was proposed. The vertical data partitioning algorithm based on the statistical characteristic of the components group of text vector was developed for data partitioning in MapReduee. Additionally, the sorting characteristics of the MapReduce were applied to select the merge points, making the algorithm be more efficient and conducive to improve clustering accuracy. The experimental results show that the proposed algorithm is effective and has good scalability.

关 键 词：文本聚类 层次聚类 数据划分 MAPREDUCE 并行计算 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]