基于hadoop平台的分布式数据挖掘系统研究  

Distributed database system based on Hadoop platform Research

作　　者：郭建伟[1] 李瑛[1] 杜丽萍[1] 赵桂芬[1] 蒋继娅[1] 

机构地区：[1]北京市科学技术情报研究所信息技术部,北京100044

出　　处：《中国科技信息》2013年第13期81-83,共3页China Science and Technology Information

摘　　要：以Hadoop为代表的分布式系统,正在逐渐成为大数据挖掘系统必要组成部分。因此,就是在Hadoop分布式系统上完成数据挖掘任务的一次实践。主要任务是使用Hadoop搭建分布式集群环境,并在该环境上部署数据挖掘任务。研究Hadoop系统架构,对其分布式文件系统HDFS和MapReduce并行编程模型的原理和实现进行深入理解。系统掌握数据挖掘过程,将传统的数据挖掘算法使用MapReduce编程实现,并针对Hadoop平台的数据挖掘算法的执行情况进行研究,主要分析其执行效率和可扩展性。选择数据挖掘中的数据聚类任务作为代表,并选择K-means聚类算法做深入研究,掌握其原理并编写其MapReduce版本,在Hadoop平台上测试并验证其效果。通过不同集群规模和不同数据规模的对比试验得出,使用Hadoop分布式系统进行数据挖掘任务具有良好地加速比和效率,计算能力的扩展性能分析也显示了其具有较大的潜力。Distributed systems,symbolized by Hadoop,are becoming a necessary part of a large-scale data mining system.Therefore,this issue is exactly a kind of practice of data mining tasks on the Hadoop Distributed Systems.In this paper,the main task is to build a distributed cluster computation environment using Hadoop and implement a data mining task in the environment.Research Hadoop system architecture,and we get in-depth understanding of Distributed File System（HDFS） and MapReduce parallel programming model.We learn the data mining principle,and implement the traditional data mining algorithms using MapReduce programming model and study the implementation of Hadoop platform data mining algorithm,analysis of the efficiency and scalability.We select data clustering task as a representative,and select the K-means clustering algorithm to do in-depth research to grasp the principle and compile its MapReduce version,test and verify its effect on the Hadoop platform.Different cluster size and scale data comparative test derived that Hadoop Distributed System has good speedup and efficiency in data mining tasks and analysis of computing power extended performance also shows its great potential.

关 键 词：数据挖掘 K-MEANS 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]