基于Hadoop的Apriori改进算法研究  

Reseach on Improved Apriori Algorithm Based on Hadoop

作　　者：黄剑[1] 李明奇[1] 郭文强[2] 

机构地区：[1]电子科技大学数学科学学院,成都611731 [2]新疆财经大学计算机科学与工程学院,乌鲁木齐830012

出　　处：《计算机科学》2017年第7期262-266,269,共6页Computer Science

基　　金：国家自然科学基金(61163066)资助

摘　　要：对于规模庞大的事务数据库,传统的并行Apriori算法在挖掘中会在数据IO上有较大的时间开销。从压缩事务、减少扫描次数、简化候选集生成3个方面对Apriori算法进行改进。提出了以元素"0"和"1"表示事务的布尔矩阵模型,并引入权值维度,压缩了相同事务的矩阵规模。同时,动态地进行剪枝,矩阵的"与"运算用于候选集合的生成。将改进后的算法在Hadoop框架上进行并行化实现,实验表明该算法适合大规模数据挖掘且具有良好的伸缩性与有效性。Traditional mining based on parallel Apriori algorithms needs much more time in data IO with the increasing size of large transaction database.In this paper,we improved Apriori algorithm in three aspects：compression in the transaction,reducing the number of scanning areas,and simplifying the candidate set generation.We proposed ＂0＂ and ＂1＂ as the entries to describe the transaction Boolean matrix model,and introduced the weight dimensions to compress the matrix size of the transaction.Meanwhile,dynamic pruning matrix is adopted,and ＂and＂operation of matrix is applied to generate a candidate set.The experiments of the improved algorithm running parallel in Hadoop framework show that the algorithm is suitable for large-scale data mining,and the algorithm has good scalability and effectiveness.

关 键 词：APRIORI算法 事务数据库 布尔矩阵 HADOOP 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]