基于MapReduce的并行聚类算法设计与实现  

DESIGN AND IMPLEMENTATION OF PARALLEL CLUSTERING ALGORITHM BASED ON MAPREDUCE

作　　者：刘向东[1] 刘奎[1] 胡飞翔[2] 王翠荣[1] 

机构地区：[1]东北大学秦皇岛分校计算机与通信工程学院,河北秦皇岛066004 [2]东北大学信息工程学院,辽宁沈阳110000

出　　处：《计算机应用与软件》2014年第11期251-256,共6页Computer Applications and Software

基　　金：国家自然科学基金项目(61070162;71071028)

摘　　要：针对目前聚类算法对大数据处理效率较低的问题,研究云平台上高效并行化的聚类算法十分必要。在HDFS分布式文件系统基础上,设计一种并行聚类算法P-ISODATA,利用MapReduce编程框架的执行机制将传统ISODATA聚类算法并行化,并在Map阶段之后加入Combine阶段以减少网络传输开销,进一步提高执行效率。实验从著名UCI机器学习库上选取若干数据集作为测试数据,分析了新并行算法P-ISODATA性能,结果表明基于P-ISODATA算法具有优良的加速比、数据伸缩率和扩展率,可以有效地应用于大规模数据的处理。In light of the problem that current clustering algorithms are inefficient in efficiency when processing big data,it is necessary to study efficient parallel clustering algorithms on cloud platform.We design a new parallel clustering algorithm P-ISODATA based on HDFS（ Hadoop distributed file system）,it parallelises traditional ISODATA clustering algorithm by utilising the execution mechanism of MapReduce programming framework,and introduces Combine phase following behind the Map phase to reduce network transmission overhead and to further improve the execution efficiency.In experiments we select different datasets from famous UCI machine learning repository as the testing data,and analyse the performance of P-ISODATA algorithm.Results show that P-ISODATA-based algorithm has good speed-up ratio,data scale-up rate and size-up ratio,therefore it can be effectively applied to large-scale data processing.

关 键 词：大数据 聚类算法 云平台 并行处理 MAPREDUCE 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]