基于负载感知的数据流动态负载均衡策略  

Dynamic data stream load balancing strategy based on load awareness

作　　者：李梓杨 于炯[1,2] 卞琛[2] 王跃飞[2] 鲁亮[2] 

机构地区：[1]新疆大学软件学院,乌鲁木齐830008 [2]新疆大学信息科学与工程学院,乌鲁木齐830046

出　　处：《计算机应用》2017年第10期2760-2766,2772,共8页journal of Computer Applications

基　　金：国家自然科学基金资助项目(61262088;61462079;61562086;61363083);新疆维吾尔自治区高校科研计划项目(XJEDU2016S106)~~

摘　　要：针对大数据流式计算平台中存在节点间负载不均衡、节点性能评估不全面的问题,提出基于负载感知算法的动态负载均衡策略,并将算法应用于Flink数据流计算平台中。首先通过有向无环图的深度优先搜索算法获取节点的计算延迟时间作为评估节点性能的依据,并制定负载均衡策略;然后基于数据分块管理策略实现流式数据的节点间负载迁移技术,通过反馈实现全局和局部的负载调优;最后通过实验评估时空代价论证算法的可行性,并讨论重要参数对算法执行效果的影响。经实验验证算法通过优化流式计算任务的负载分配提高了任务的执行效率,与采用Flink平台现有的负载均衡策略相比,任务执行时间平均缩短6.51%。Concerning the problem of unbalanced load and incomplete comprehensive evaluation of nodes in big data stream processing platform, a dynamic load balancing strategy based on load awareness algorithm was proposed and applied to a data stream processing platform named Apache Flink. Firstly, the computational delay time of the nodes was obtained by using the depth-first search algorithm for the Directed Acyclic Graph （DAG） and regarded as the basis for evaluating the performance of the nodes, and the load balancing strategy was created. Secondly, the load migration technology for data stream was implemented based on the data block management strategy, and both the global and local load optimization was implemented through feedback. Finally, the feasibility of the algorithm was proved by evaluating its time-space complexity, meanwhile the influence of important parameters on the algorithm execution was discussed. The experimental results show that the proposed algorithm increases the efficiency of the task execution by optimizing the load sharing between nodes, and the task execution time is shortened by 6.51% averagely compared with the traditional load balancing strategy of Apache Flink.

关 键 词：数据流 负载均衡 深度优先搜索 负载感知 APACHE Flink 

分 类 号：TP393.02[自动化与计算机技术—计算机应用技术]