基于Spark的并行信任进化算法  

PARALLEL TRUST EVOLUTION ALGORITHM BASED ON SPARK

作　　者：黄冬平 周夏冰 刘冠峰[1] Huang Dongping;Zhou Xiabing;Liu Guanfeng(School of Computer Science and Technology,Soochow University,Suzhou 215006,Jiangsu,China;Shanghai Stock Exchange Technology Limited Liability Company,Shanghai 200120,China)

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006 [2]上交所技术有限责任公司,上海200120

出　　处：《计算机应用与软件》2021年第3期243-248,共6页Computer Applications and Software

基　　金：国家自然科学基金青年基金项目(61702518);江苏高校优势学科建设工程项目。

摘　　要：为了提高信任进化(MET)算法在大数据环境下的时效性,提出一种基于Spark的并行MET算法——SparkMET。采用主从式架构,由主节点进行变异、交叉、选择过程,从节点计算适应度值。针对数据倾斜问题,提出一种新的数据分区策略——LBP算法,在SparkMET计算适应度值之前,根据合适的分区标签将数据重分区。在Epinions数据集上进行的实验表明,SparkMET算法能有效提高MET算法的运算效率。In order to improve the timeliness of trust evolution(MET)algorithm in big data environment,a parallel MET algorithm based on Spark-SparkMET is proposed.It adopted a master-slave architecture,in which the master node performed mutation,intersection,and selection processes,and the slave nodes calculated fitness values.Aiming at the data skew problem,a new data partitioning strategy-LBP algorithm was proposed.It repartitioned the data according to the appropriate partition label before SparkMET calculates the fitness value.Epinions dataset was used to conduct experiments.The experimental results show that the SparkMET algorithm can effectively improve the computational efficiency of the MET algorithm.

关 键 词：MET SPARK 并行计算 数据倾斜 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]