基于Hadoop的仿射传播大数据聚类分析方法  

Affinity propagation clustering for big data based on Hadoop

作　　者：唐东明[1] 

机构地区：[1]西南交通大学信息化研究院,成都610031

出　　处：《计算机工程与应用》2015年第4期29-34,共6页Computer Engineering and Applications

基　　金：国家自然科学基金(No.61100118;No.61003142;No.61373009);中央高校基本科研业务费专项资金资助(No.2682014CX100)

摘　　要：仿射传播聚类算法(AP)是一个新的聚类分析方法,已经被广泛应用于各种领域。APC算法不能用于大型数据的分析。为了克服这个限制,在Hadoop分布式框架的基础上提出一种改进的放射传播聚类分析方法(基于Hadoop的仿射传播大数据聚类分析方法,简称APCH)。通过在Hadoop环境下重新设计算法流程,APCH算法成为了一个并行化的大数据聚类分析方法。此外APCH算法能够高效操作大数据,并能够直接决定聚类的个数。为了验证方法的性能,在多个数据集上进行了实验。实验结果表明APCH对大数据处理有很好的适应性和延展性。APCH采用开源的方式提供可执行软件程序和源代码,用户可以下载后部署在自己的分布式集群中或者是部署在亚马逊EC2等云计算环境中。所有编译后的执行程序,源代码,用户手册,部分测试数据集均可以从https://github.com/Hello World CN/Map Reduce APC上下载。Affinity Propagation Clustering（APC）is a new clustering algorithm. APC has been applied in various fields recently. However, AP can＇t be applied for analyzing large-scale data sets. To overcome this limitation, an improved Affinity Propagation cluster analysis algorithm（Affinity Propagation Clustering for Big Data Based on Hadoop, APCH）is proposed in the Hadoop distributed computing framework. After redesigning algorithm flow based on Hadoop framework, APCH becomes parallelized cluster analysis method for large-scale data. Moreover, APCH can efficiently operate big dada, and directly determine the number of clusters. To verify the provided method, we experiment its performance on many data sets. The experimental results show that APCH provides good scalability and flexibility on big data analysis. In addition,APCH is open-source software and can be freely downloaded. APCH can be deployed on your Hadoop clusters, or Amazon Elastic Compute Cloud（Amazon EC2）, etc. All compiled execution binary package, user manual, including some test data sets can be downloaded from https：//github.com/Hello World CN/Map Reduce APC.

关 键 词：仿射传播聚类 MAP REDUCE HADOOP 键值存储 大数据 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]