异构环境中HDFS数据块调度算法的设计与实现  

Design and Implementation of HDFS Data Block Scheduling Algorithm in Heterogeneous Environment

作　　者：高原[1,2,3] 任升[1,2,3] 顾文杰[1,2,3] 

机构地区：[1]南瑞集团公司(国网电力科学研究院),南京211106 [2]国电南瑞科技股份有限公司,南京211106 [3]智能电网保护和运行控制国家重点实验室,南京211106

出　　处：《计算机工程》2017年第8期82-89,共8页Computer Engineering

基　　金：国家电网公司科技项目"超大规模电网调控系统集群化关键技术研究"

摘　　要：针对Hadoop分布式文件系统(HDFS)的写性能在执行效率上的不足,提出一种在节点性能异构环境中对HDFS数据块进行并发传输的调度算法。该算法实时监控HDFS集群中每个节点的资源状态和内存缓存队列,动态地将接收节点与转发节点进行配对传输,使全系统节点的网卡和磁盘并发工作,缩短了所有副本写入分布式文件系统的时间。将数据写入磁盘后请求下一个数据块,保证数据安全性,同时也使得各个节点获得与自身性能相匹配的副本数,使性能异构的系统能达到较高的写入速度。性能测试结果表明,使用该算法的分布式文件系统的写入性能较原始的HDFS提高了1倍。Aiming at the insufficient of write performance of Hadoop Distributed File System （HDFS） , a scheduling algorithm for concurrent transmission of HDFS data blocks in a heterogeneous environment is proposed. The algorithm monitors the resource status and memory queue of each node in the HDFS cluster in real time, matches receiving nodes with the forwarding nodes dynamically, makes the network cards and disk of the whole system work concurrently and reduces the time to write all copies to the distributed file system. The algorithm ensures that the data are written to disk before requesting the next data block for the data security. In the meantime, it makes the number of copies of each node match its own performance, so that the heterogeneous systems can achieve a high rate of writing. Performance tests show that the write performance of the distributed file system using the proposed algorithm is improved by 1 times compared with the original HDFS.

关 键 词：异构 HADOOP分布式文件系统 并发 数据块 调度 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]