基于Spark框架的FP-Growth大数据频繁项集挖掘算法  

Frequent item sets mining algorithm for big data based on FP-Growth and Spark framework

作　　者：邵梁[1] 何星舟[2] 尚俊娜[3] Shao Liang;He Xingzhou;Shang Junna(Educational Technologies Center,Zhejiang College of Construction,Hangzhou 311231,China;Student Office,Zhejiang University of Technology,Hangzhou 311231,China;School of Communication Engineering,Hangzhou Dianzi University,Hangzhou 310018,China)

机构地区：[1]浙江建设职业技术学院教育技术中心,杭州311231 [2]浙江工业大学学生处,杭州311231 [3]杭州电子科技大学通信工程学院,杭州310018

出　　处：《计算机应用研究》2018年第10期2932-2935,共4页Application Research of Computers

基　　金：国家自然科学基金资助项目(166223123);浙江省自然科学基金资助项目(jg20160405)

摘　　要：针对大数据中的频繁项集挖掘问题,提出一种基于Spark框架的FP-Growth频繁项集并行挖掘算法。首先,根据垂直布局思想将数据按照事务标志符垂直排列,以此解决扫描整个数据集的缺陷;然后,通过FPGrowth算法构建频繁模式树,并生成频繁1-项集;接着,通过扫描垂直数据集来计算项集的支持度,从而识别出非频繁项,并将其从数据集中删除以降低数据尺寸;最后,通过迭代过程来生成频繁k-项集。在标准数据集上的实验结果表明,该算法能够有效挖掘出频繁项集,在执行时间方面具有很大的优越性。For the issues of the frequent item sets mining in big data,this paper proposed a parallel frequent item sets mining algorithm based on FP-Growth and Spark framework.Firstly,it arranged the data vertically according to the transaction identifier,in order to solve the defects of scan the entire data set.Then,it adopted the FP-Growth algorithm to construct the frequent pattern tree and generated frequent 1-item sets.After that,it calculated the support of the item set by scanning the vertical data set,so as to identify the non-frequent items,and deleted them from the data set to reduce the data size.Finally,it used the iterative process to generate frequent item sets.The experimental results on the standard dataset show that the algorithm can effectively excavate frequent item sets and have great superiority in execution time.

关 键 词：大数据 频繁项集挖掘 Spark框架 FP-GROWTH算法 垂直布局 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]