MapReduce平台上面向倾斜数据连接的负载平衡方法  

Load balancing method for skewed data connection on MapReduce platform

作　　者：汪鑫[1] 芦立华 WANG Xin;LU Lihua(Shanghai Dianji University,Shanghai 201306,China)

机构地区：[1]上海电机学院,上海201306

出　　处：《现代电子技术》2021年第2期65-68,共4页Modern Electronics Technique

基　　金：国家自然科学基金青年项目(61703267)。

摘　　要：由于传统倾斜数据的连接负载平衡方法设置的负载布局位置不佳,导致对负载偏移的控制效果差,因此针对MapReduce平台上的倾斜数据,提出连接负载的新式平衡方法。优化设置MapReduce系统前端分发器和末端服务器,根据泊松流设置任务分配控制条件;提取倾斜数据特征,利用评价函数设置平衡约束指标;计算信息熵控制负载偏移量,使用平衡布局概率得出布局最优位置。实验结果表明,所提出负载平衡方法的负载偏移量比传统方法的测试结果低,证明该方法对负载偏移量的控制效果更好,符合现阶段的平衡控制要求。As the load layout position set by the connection load balancing method of the conventional skewed data is poor,which results in the inefficient control effect on the load offset,a novel balancing method of connecting the load is proposed for the skewed data on the MapReduce platform.The front end distributor and the end server of the MapReduce system are optimized,and the task allocation control conditions are set according to the Poisson flow.The features of skewed data are extracted,and the equilibrium constraint indexes are set by means of the evaluation functions.The information entropy is calculated to control the load offset,and the balanced layout probability is used to obtain the optimal position of the layout.The experimental results show that the load offset of the proposed load balancing method is lower than the test results of the traditional method,which verifies that this method has better control effect on the load offset and can meet the balance control requirements at the present stage.

关 键 词：倾斜数据 负载平衡 MapReduce平台 任务分配 特征提取 负载偏移控制 评价函数 

分 类 号：TN919-34[电子电信—通信与信息系统] TP391.9[电子电信—信息与通信工程]