基于Spark的异构集群调度策略研究  

Adaptive Scheduling Strategy for Heterogeneous Spark Cluster

作　　者：徐佳俊[1] 刘功申[1] 苏波[1] 孟魁[1] 

机构地区：[1]上海交通大学,上海

出　　处：《计算机科学与应用》2016年第11期692-704,共13页Computer Science and Application

基　　金：国家自然科学基金项目(编号:61472248)。

摘　　要：Spark的原生调度策略建立在集群同质化的基本假设上。然而随着硬件的更迭以及高性能硬件的引入,集群异质化现象日趋显著。因此现有的调度策略在异构集群环境下并不高效,短板效应严重。针对这个问题,本文提出了一种新的调度策略以优化Spark在异构集群下的表现。新策略引入了分层调度的思想,调度时综合考量了任务复杂度、节点性能及节点资源使用情况等因素,实现了更加高效公平的任务调度算法。通过仿真和真机实验,证明了新策略的效果相对于原策略有明显提升。The scheduling strategy of Spark assumes that cluster is homogenized. However, as the change or update of hardware in cluster, it becomes more and more heterogeneous. Thus, the original scheduling strategy cannot meet the performance requirement anymore and short board effect gradually emerges. The paper proposes a new strategy to solve this problem. The new strategy refers the idea of hierarchical scheduling. It combines the task complexity, worker performance and worker CPU usage as its scheduling factors to improve the scheduling performance. And ex-periments show that the new strategy is absolutely effective.

关 键 词：SPARK 异构集群 调度策略 

分 类 号：TP39[自动化与计算机技术—计算机应用技术]