数据依赖约束下的任务调度资源选择算法  

Task scheduling and resource selection algorithm with data-dependent constraints

作　　者：廖彬[1] 于炯[2] 张陶[3] 杨兴耀[2] 

机构地区：[1]新疆财经大学统计与信息学院,乌鲁木齐830012 [2]新疆大学软件学院,乌鲁木齐830008 [3]新疆医科大学医学工程技术学院,乌鲁木齐830011

出　　处：《计算机应用》2014年第8期2260-2266,共7页journal of Computer Applications

基　　金：国家自然科学基金资助项目(61262088;60863003;61063042);新疆维吾尔自治区自然科学基金资助项目(2011211A011)

摘　　要：大数据环境下的计算任务往往具有一定数据依赖性关系(如MapReduce),现有的分布式存储系统任务资源选择策略选择离请求者最近的数据块响应服务,忽略了对数据块所在服务器CPU、磁盘I/O与网络等资源负载状态的考虑。在分析研究系统集群结构、文件分块、数据块存储机制的基础上,定义了集群节点矩阵、CPU负载矩阵、磁盘I/O负载矩阵、网络负载矩阵、文件分块矩阵、数据块存储矩阵与数据块存储节点状态矩阵,为任务与数据之间的依赖性构建了基础数据模型,提出了一种数据依赖约束下的最优资源选择算法(ORS2DC)。任务调度节点负责维护基础数据,MapReduce任务与数据块读取任务由于依赖资源不同而采取不同的选择策略。实验结果表明:所提算法能够为任务选择质量更高的资源,提高任务完成质量的同时减轻了NameNode负担,减小了单点故障发生的概率。Like MapReduce, tasks under big data environment are always with data-dependent constraints. The resource selection strategy in distributed storage system trends to choose the nearest data block to requestor, which ignored the server＇s resource load state, like CPU, disk I/O and network, etc. On the basis of the distributed storage system＇s cluster structure, data file division mechanism and data block storage mechanism, this paper defined the cluster-node matrix, CPU load matrix, disk I/O load matrix, network load matrix, file-division-block matrix, data block storage matrix and data block storage matrix of node status. These matrixes modeled the relationship between task and its data constraints. And the article proposed an optimal resource selection algorithm with data-dependent constraints （ORS2DC）, in which the task scheduling node is responsible for base data maintenance, MapRedcue tasks and data block read tasks take different selection strategies with different resource-constraints. The experimental results show that, the proposed algorithm can choose higher quality resources for the task, improve the task completion quality while reducing the NameNode＇s load burden, which can reduce the probability of the single point of failure.

关 键 词：大数据计算 分布式存储系统 任务调度 数据模型 资源选择 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论] TP274[自动化与计算机技术—计算机科学与技术]