基于调度器的Hadoop性能优化方法研究  

Research of Hadoop performance optimization based on the scheduler

作　　者：刘娟[1,2] 豆育升[2] 何晨[2] 唐红[1] 

机构地区：[1]重庆邮电大学计算机学院,重庆400065 [2]重庆邮电大学高性能计算与应用研究所,重庆400065

出　　处：《计算机工程与设计》2013年第1期190-194,共5页Computer Engineering and Design

基　　金：国家自然科学基金项目(21073242)

摘　　要：为了提高Hadoop调度器的调度性能,缩短Hadoop集群的任务整体响应时间,提出了一种基于CPU占用率的动态调度改进算法。首先对Hadoop传统的性能优化方法进行了对比,指出其存在问题的关键是缺乏动态性和灵活性。在此基础上,深入分析Hadoop默认任务调度模型,提出了一种以CPU占用率作为负载指标,在循环分配任务时根据反馈的负载指标判断节点负载情况的算法,动态适应负载变化。实验结果表明,该算法在Hadoop集群中,能有效提高集群性能。To improve the performance of Hadoop scheduler, reduce the task response time of Hadoop cluster, one kind of dy- namic scheduling algorithm based on the CPU occupancy rate is proposed. Firstly, traditional performance optimization method is compared. It is pointed out that lacking of dynamics and flexibility became the key problem. Then, the default task Scheduling model is analyzed deeply and the CPU occupancy as a load index, feeding back the load index in circular calculation is employed. It adapted load changes dynamically. The experimental results show that the algorithm in Hadoop cluster can effectively improve the cluster performance.

关 键 词：HADOOP 负载 CPU占用率 云计算 动态调度 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]