Hadoop平台下改进的推测任务调度算法  

Improved speculative task scheduling algorithm on Hadoop platform

作　　者：陈明丽[1] 刘旭敏[1] 

机构地区：[1]首都师范大学信息工程学院,北京100048

出　　处：《传感器与微系统》2017年第2期134-137,共4页Transducer and Microsystem Technologies

基　　金：国家自然科学基金资助项目(61272029)

摘　　要：研究对比Hadoop平台下默认的推测任务调度算法和异构环境下LATE调度算法的优势和不足,提出了一种基于Hadoop集群的改进的推测任务调度算法。该算法以节点历史信息对Reduce任务各阶段比例进行动态调整和更新,并对任务实时处理速率进行局部平滑处理来提高预估任务剩余完成时间的准确性,最后采用MCP模型对备份任务有效性进行验证。通过实验结果分析可知:该算法能够有效提升备份任务成功率,减少作业完成时间。After compare and analysis the advantage and disadvantage of the default speculative task scheduling algorithm on the Hadoop platform and LATE scheduling algorithm in heterogeneous environment, an improved speculative task scheduling algorithm based on Hadoop computer cluster is proposed. The algorithm dynamically adjusts and updates the proportion of stages in the Reduce task based on node history information, smoothes the real-time task progress rate to improve the forecast accuracy of the remaining completion time, and applies MCP model to verify the effectiveness for backup task. The experiment analysis show that the algorithm can improve the success rate of backup task and reduce job completion time.

关 键 词：MAPREDUCE 异构环境 推测执行 LATE 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]