改进的Hadoop公平调度算法  

Improved Fair Scheduling Algorithm for Hadoop

作　　者：吴岳 WU Yue(National Administration of Glassland Forestry Industry Development Planning Institute,Beijing 100010,China)

机构地区：[1]国家林业和草原局产业发展规划院,北京100010

出　　处：《电脑与信息技术》2022年第6期39-41,共3页Computer and Information Technology

摘　　要：Hadoop是用于存储和处理大规模数据的基础架构和软件。MapReduce是Hadoop的一个JAVA程序设计框架模型,遵循MapReduce框架开发的应用程序可以在集群中的节点上并行执行。作业调度器是Hadoop系统中的重要组件。文章中,作者阐述了Hadoop中常见的三种调度器,并重点讨论了公平调度算法。通过分析公平调度算法在调度小型作业时容易影响整个集群吞吐量的原因,提出了一种改进的公平调度算法,并通过实验证明改进后的公平调度算法对整个集群的平均吞吐率有明显的提高。Hadoop is a kind of framework and software for mass data storage and processing.The job scheduler is an important component in the Hadoop system.In this article,the author explains three common schedulers in Hadoop,and focus on the fair scheduling algorithm.The author analyzes the reason that will affect the throughput of entire cluster on scheduling small job with fair scheduling algorithm,and advances an improved fair scheduling algorithm.The experimental result shows that the whole cluster average throughput significantly improved by the improved fair scheduling algorithm.

关 键 词：公平调度 MAPREDUCE HADOOP 数据本地性 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]