改进的FP-Growth算法及其分布式并行实现  

Improved FP-Growth Algorithm and Its Distributed Parallel Implementation

作　　者：马月坤 刘鹏飞 张振友 孙燕 丁铁凡 

机构地区：[1]华北理工大学信息工程学院,河北唐山063000

出　　处：《哈尔滨理工大学学报》2016年第2期20-27,共8页Journal of Harbin University of Science and Technology

基　　金：河北省科技支撑计划项目(15210110D);唐山市科技支撑计划项目(14130233B)

摘　　要：针对传统FP-Growth算法在大规模数据环境下挖掘效率低下的问题,提出了一种改进的FP-Growth算法.该算法主要是通过基于频繁闭项集策略对完备模式树进行剪枝进而减小搜索空间规模,达到提高算法挖掘效率的目的 .并将改进后的FP-Growth算法的分治策略与分布式计算框架Hadoop的MapReduce编程模式有机结合,进一步提高了大数据环境下的挖掘效率.实验证明,基于Hadoop的改进FP-Growth算法的效率较传统FP-Growth算法有所提高.The main problem solved in the research is that the traditional FP-Growth algorithm had a low efficiency and memory overflow under the big data environment. Aimed at the problem,this paper put forward a improved FP-Growth algorithm,which reduces the search space through complete pattern tree pruning by using frequent closed itemset to improve the mining efficiency of the algorithm. Then the FP-Growth algorithm is realized by using MapReduce a programming pattern of Hadoop. Thereby the mining efficiency is improved further. Experiments show that the efficiency of the improved FP-Growth algorithm implement by Hadoop is superior to traditional FP-Growth algorithm.

关 键 词：分布式并行 改进FP-Growth算法 剪枝 MapReduce编程模式 

分 类 号：TP311.1[自动化与计算机技术—计算机软件与理论]