基于Spark的并行频繁模式挖掘算法  

Parallel frequence pattern mining algorithm based on Spark

作　　者：曹博[1] 倪建成[2] 李淋淋[1] 于苹苹[1] 姚彬修 CAO Bo;NI Jiancheng;LI Linlin;YU Pingping;YAO Binxiu(College of Information Science and Engineering, Qufu Normal University, Rizhao, Shandong 276800, China;College of Software, Qufu Normal University, Qufu, Shandong 273100, China)

机构地区：[1]曲阜师范大学信息科学与工程学院,山东日照276800 [2]曲阜师范大学软件学院,山东曲阜273100

出　　处：《计算机工程与应用》2016年第20期86-91,共6页Computer Engineering and Applications

基　　金：国家自然科学基金(No.61402258);山东省本科高校教学改革研究项目(No.2015M102);校级教学改革研究项目(No.jg05021*)

摘　　要：在大数据环境下Apriori频繁模式挖掘算法在数据处理过程具有预先设定最小阈值、时间复杂度高等缺陷,为此采用多阶段挖掘策略实现并行化频繁模式挖掘算法PTFP-Apriori。首先将预处理数据以模式树的形式存储,通过最为频繁的k个模式得到最优阈值。然后根据该值删除预期不能成长为频繁的模式以降低计算规模,并利用弹性分布式数据集RDD完成统计项集支持度计数、候选项集生成的工作。实验分析表明相比于传统的频繁模式挖掘算法,该算法具有更高的效率以及可扩展性。Under the environment of big data, the frequent pattern mining algorithm Apriori has some defects, includingpresetting minimum threshold and high time complexity when in data processing process. Therefore, the multistage miningstrategy is adopted to realize the parallel frequent pattern mining algorithm(PTFP-Apriori). Firstly, the preprocessed datais stored in a pattern tree, and the optimal threshold is got by the most frequent K model. Subsequently, according to thethreshold, the frequent pattern that can’t grow up to be frequent patterns could be removed to reduce the computing scale.The RDD is used to accomplish the task of itemsets support counting and candidate itemsets generating. The experimentalresults show that the algorithm has higher effectivity and scalability than the traditional algorithm.

关 键 词：大数据 频繁模式挖掘 TOP-K 模式树 并行计算 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]