面向负载均衡的动态均衡分区策略  

DYNAMIC BALANCING PARTITION STRATEGY FOR LOAD BALANCING

作　　者：杨迪[1] 赵家伟 王鹏[1] 赵建平[1] Yang Di;Zhao Jiawei;Wang Peng;Zhao Jianping(Changchun University of Science and Technology,Changchun 130000,Jilin,China)

机构地区：[1]长春理工大学,吉林长春130000

出　　处：《计算机应用与软件》2024年第8期46-52,共7页Computer Applications and Software

基　　金：中央引导地方科技发展资金吉林省基础研究专项(202002038JC)。

摘　　要：针对MapReduce计算框架处理倾斜数据集时造成Reduce端出现负载不均衡现象,提出一种动态均衡分区策略。在mapper阶段提出基于分治法的数据切分原则处理任务传入的数据组;结合最佳适应算法思想设计动态分配原则逐步将切分后的数据块均衡分配到预分区链表中;根据分区索引分配到各Reduce节点上实现负载均衡。实验结果显示,动态均衡分区策略与两个基准模型相比任务执行时长平均降低了7.7%,表明动态均衡分区策略更好地解决了数据倾斜问题,降低了任务执行时间,验证了模型的有效性。In view of the unbalanced load on the reduce side caused by MapReduce computing framework processing inclined data sets,this paper proposes a dynamic balanced partition strategy.In the mapper stage,the data segmentation principle based on divide and conquer method was proposed to process the incoming data groups.The dynamic allocation principle was designed combined with the idea of the best adaptive algorithm to gradually allocate the segmented data blocks to the server.According to the partition index,it was allocated to each reduce node to achieve load balancing.The experimental results show that,compared with the two benchmark models,the average task execution time of the dynamic balanced partition strategy is reduced by 7.7%,which indicates that the dynamic balanced partition strategy can better solve the problem of data skew,reduce the task execution time,and verify the effectiveness of the model.

关 键 词：MAPREDUCE 负载均衡 数据倾斜 数据分区 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]