一种基于Hadoop平台的新聚类算法  

New Clustering Algorithm Bases on Hadoop

作　　者：缪裕青[1,2] 张锦杏[1] 刘少兵[1] 文益民[1] 明媚[1] 

机构地区：[1]桂林电子科技大学计算机科学与工程学院,桂林541004 [2]桂林电子科技大学广西可信软件重点实验室,桂林541004

出　　处：《计算机科学》2014年第4期269-272,共4页Computer Science

基　　金：广西可信软件重点实验室研究课题(KX201116);广西教育厅科研项目(201204LX122)资助

摘　　要：针对现有很多聚类算法不能有效处理大规模数据的问题,基于微簇和等价连接关系,提出一种能在Hadoop平台实现高效并行化的聚类算法bigKClustering。算法将紧凑的数据抽象成一个向量,然后通过等价关系对这些向量进行连接,得到最终的聚类结果。实验结果表明,bigKClustering算法不仅具有良好的时间效率和聚类效果,而且具有良好的可伸缩性、加速比和时间稳定性。Hadoop is a popular platform to handle huge datasets.But many clustering algorithms can not run effectively over it,for it lacks built-in support for iterative programs,which arises naturally in many clustering applications.We proposed big Clustering which can be easily parallelized in Hadoop MapReduce and done in quite a few MapReduce rounds.Our algorithm is based on the ideas of micro-cluster and equivalence relation.It divides a dataset into many groups and constructs one micro-cluster,which will be treated as a single point,corresponding to each group.All microclusters that are closed enough will be connected and put into the same group by the equivalence relation.The center of each group will be calculated and will be the center of a real cluster in the dataset.Experiments show that bigKClustering not only runs fast and obtains high clustering quality but also scales well.

关 键 词：微簇 等价连接 HADOOP平台 聚类 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]