基于Spark字典表压缩存储的关联规则算法优化  

OPTIMIZATION OF ASSOCIATION RULES ALGORITHM BASED ON SPARK DICTIONARY TABLE COMPRESSED STORAGE

作　　者：刘丽娜 姜利群 Liu Li'na;Jiang Liqun(Department of Computer Science and Engineering,Guangzhou College of Technology and Business,Guangzhou 510850,Guangdong,China)

机构地区：[1]广州工商学院计算机科学与工程系,广东广州510850

出　　处：《计算机应用与软件》2021年第8期37-43,共7页Computer Applications and Software

基　　金：广东高校优秀青年创新人才培养计划资助项目(2018KQNCX309);教育部2020年第一批产学合作协同育人项目(202002191035)。

摘　　要：针对大数据新型处理框架Spark执行Apriori算法存在速率低、内存负荷高等不足,提出一种改进的Apriori优化算法。基于字典表压缩存储的机制,结合Spark框架中列式存储模式对多维多属性值的数据集进行压缩,通过Spark集群进行数据并行处理。实验表明,该算法比原算法执行速率提高23%以上,且在数据量越大的情况下其优势更明显。该算法具有降低内存负荷量、去候选频繁项集、提高执行速率等优势,且解决了多维多属性值数据集的分析难题,具备一定的应用价值。A new Apriori optimization algorithm is proposed for the problem of low speed and high memory load during the implementation of the Apriori algorithm for the big data processing framework Spark.The algorithm was based on the dictionary table compression storage mechanism,combined with the column storage mode in the Spark framework to compress the data set of multi-dimensional multi-attribute values,and processed data parallelly through the Spark cluster.The experiments show that the improved algorithm is more than 23%faster than the original algorithm,and the advantage of the algorithm is more obvious when the amount of data is larger.It has the advantages of reducing the memory load,generating no candidate frequent itemsets,improving the execution rate,and solving the analysis problem of the multi-dimensional multi-attribute value data set,and it has certain application value.

关 键 词：联规则 算法优化 压缩存储 字典表 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]