基于HDFS的分布式存储策略分析  

Research of distributed storage system based on HDFS

作　　者：王来[1] 翟健宏[1] 

机构地区：[1]哈尔滨工业大学计算机科学与技术学院,哈尔滨150001

出　　处：《智能计算机与应用》2016年第1期5-8,共4页Intelligent Computer and Applications

基　　金：国家自然科学基金(61202457);国家重点基础研究发展计划(973)(2011CB302605)

摘　　要：本文重点围绕HDFS机架感知和副本存放策略方面对HDFS分布式存储进行剖析。副本存放策略和机架感知主要通过Datanode节点形成的树状网络拓扑图来让Namenode节点获取,从而确定副本存放的位置,这种方式保证了对于数据的极高的容错性的同时也兼顾了数据本地化,即提高了数据在集群网络中的传输效率。基于此,提出一个设想,希望通过对副本存放策略的深入挖掘,根据Datanode数据节点的实时状态信息,实现对于数据块副本的定向存储,再由数据驱动任务分配,来为每一个Datanode数据节点分配更适合的任务,从而达到负载均衡提高资源利用率的作用。This article focuses on the strategy of Replication Target Chooser and Rack-Awareness to analyze HDFS distributed storage. To realize the strategy of Replication Target Chooser and Rack-Awareness,the HDFS forms a network topology tree of Datanode primarily to let Namenode nodes determine the location of replication,in such a way to ensure that for extremely high data fault tolerance while taking into account the data locally,and to improve the efficiency of data transmission in the cluster network. Based on this,the paper proposes an idea,hope to learn more of the strategy of Replication Target Chooser,based on real-time status information Datanode nodes to achieve the orientation for the data block,data-driven task redistribution to Datanode data for each node assign tasks better suited to achieve the effect of load balancing and improve resource utilization.

关 键 词：HDFS 分布式存储 副本存放策略 数据驱动 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]