基于粒子群优化的全比较计算数据分发策略  

All-to-All Comparison Computing Data Distribution Strategy Based on Particle Swarm Optimization

作　　者：李雷孝 邓丹 李杰 王永生 LI Leixiao;DENG Dan;LI Jie;WANG Yongsheng(College of Data Science and Application,Inner Mongolia University of Technology,Hohhot 010080,China;Inner Mongolia Autonomous Region Software Service Engineering Technology Research Center Based on Big Data,Hohhot 010080,China)

机构地区：[1]内蒙古工业大学数据科学与应用学院,呼和浩特010080 [2]内蒙古自治区基于大数据的软件服务工程技术研究中心,呼和浩特010080

出　　处：《计算机工程与应用》2021年第15期109-117,共9页Computer Engineering and Applications

基　　金：内蒙古自然科学基金(2019MS06027);内蒙古自治区科技重大专项(2019ZD015);内蒙古自治区关键技术攻关计划(2019GG273);内蒙古工业大学科学研究项目(ZY201908)。

摘　　要：全比较计算数据分发策略是提高分布式集群系统整体计算性能的关键。针对现有数据分发策略存在的计算负载不均衡、数据不能完全本地化、存储空间浪费和计算速度慢等弊端,在满足数据完全本地化的前提下以负载均衡、最优化存储作为优化目标,结合优化的粒子群算法提出了数据分发模型(Data Distribution Based on Particle Swarm Optimization,DDBPSO)。DDBPSO模型分别以任务扰动、交换任务的方式对粒子进化规则进行了优化,有效避免了算法陷入局部最优。通过计算负载、存储占用和数据本地化等实验,结果表明,与开源框架Hadoop的数据分发策略相比,提出的DDBPSO模型与算法具有计算负载均衡、完全的数据本地化、存储空间占用小、计算速度快等优势。Data distribution strategy is the key to improve the overall computing performance of distributed cluster sys-tem in all-to-all comparison problems.To address the disadvantages of existing data distribution strategies,such as unbal-anced computing load,incomplete localization of data,waste of storage space and slow computing speed,etc.,on the premise of satisfying the requirement of data localization,a Data Distribution model Based on Particle Swarm Optimization(DDBPSO)is proposed with load balancing and optimal storage as the optimization objectives.DDBPSO model optimizes the particle evolution rules by task disturbance and task exchange,which makes the algorithm robust against local optima.The results in the numerial experiment show that the DDBPSO model has the advantages of computing load balancing,complete data localization,small storage space occupation and fast computing speed compared with the data distribution strategy in Hadoop.

关 键 词：全比较计算 数据分发 DDBPSO模型 粒子群优化 HADOOP 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]