面向大数据分布式存储的动态负载均衡算法  

Dynamic Load Balance Algorithm for Big-data Distributed Storage

作　　者：张栗粽 崔园[1] 罗光春[1] 陈爱国[1] 卢国明[1] 王晓雪[1] 

机构地区：[1]电子科技大学计算机科学与工程学院,成都611731

出　　处：《计算机科学》2017年第5期178-183,共6页Computer Science

基　　金：四川省科技厅应用基础(2015JY0228);科技支撑计划(2015SZ0045;2014GZ0174);电子科大基础研究(ZYGX2015J063);海外留学回国人员科研启动费项目基金资助

摘　　要：随着大数据时代的到来,分布式存储技术应运而生。目前主流大数据技术Hadoop的HDFS分布式存储系统的元数据存储架构上一直存在可扩展性差和写延迟高等问题,其在官方2.0版本中针对可扩展性的解决方案(Federation)仍不完美,仅解决了原有HDFS扩展性的问题,在元数据分配的问题上没有考虑NameNode的异构性能差异,也未解决NameNode集群动态负载均衡的问题。针对该情况,提出了一种动态负载均衡的分布NameNode算法,通过元数据多副本异构节点的动态适应性备份,使元数据在考虑节点性能及负载的情况下实现了动态分布,保证了元数据服务器集群的性能;同时结合缓存策略及自动恢复机制,提高了元数据的读写性及可用性。该算法在试验验证中达到了较为理想的效果。Distributed storage is the major approach for handling the ＂Big Data＂. Currently,the major technology is ha- doop distributed file system （HDFS） ,which has been beset by the issues of scalability and write latency. In official 2. 0 version,a new feature＇ HDFS Federation~ addresses this limitation by adding support for multiple NameNodes/name spaces to HDFS. However, it does not take the isomerism of NameNode into account, and still lacks of dynamic load balance ability. Consequently, a dynamic load balance algorithm for HDFS NameNode was proposed, and it dynamically allocated the metadata into a NameNodes cluster with multiple copies, in order to improve the performance of metadata utilizations. In addition, the proposed algorithm increases the readability by the adoption of metadata caches, and im- proves the stability by a built-in failover mechanism. Finally, an experiment was carried out, to illustrate and evaluate the utilizations of the proposed algorithm.

关 键 词：大数据 分布式存储 元数据管理 HDFS 

分 类 号：TP274[自动化与计算机技术—检测技术与自动化装置]