基于大数据的数据挖掘引擎研究  

Research on data mining engine based on big data

作　　者：王小燕[1] 张丽敏[2] 

机构地区：[1]陕西广播电视大学,陕西西安710119 [2]西安外事学院信息与网络学院,陕西西安710077

出　　处：《电子设计工程》2017年第15期31-34,38,共5页Electronic Design Engineering

基　　金：陕西省教育厅科研项目(16JK2176);陕西工商职业学院2015年度教学改革研究项目(GJ1510)

摘　　要：为了解决数据挖掘在大数据中存在的问题,文中对大数据下的数据挖掘引擎进行了研究,以Spark作为核心引擎,并在Spark的内存计算算子的基础上,实现了多个传统数据挖掘算法的并行计算,使得传统的数据挖掘算法能在集群环境中并行运行,从而在大数据中得到较好的应用。然后通过系统分层方法,将数据挖掘系统进行分层设计,实现了一个完整的大数据挖掘平台。实验表明,基于Spark实现的Apriori算法跟PageRank算法的并行计算能有效减少执行时间,在大数据挖掘上具有较好的应用。In order to solve the problem of data mining in large data, the data of the large-scale data mining engine is studied, using Spark as the core engine, and in the memory of Spark operator on the basis of the implementation of a number of traditional data mining algorithms of parallel computing, the traditional data mining algorithm can run in parallel in a cluster environment, so as to obtain the very good application in big data. Then through the system layering method, the data mining system is designed, and a complete big data mining platform is realized. Experimental results show that the Apriori algorithm based on Spark algorithm and PageRank algorithm can effectively reduce the execution time, and it has a good application in large data mining.

关 键 词：大数据 数据挖掘 SPARK 引擎 

分 类 号：TN99[电子电信—信号与信息处理]