基于二元再生码的大数据存储系统研究  

Study on Binary Regenerating Codes for Big Data Storage Systems

作　　者：黄显霞[1,2,3] 李挥[1,2,3] 张宇蒙 侯韩旭[1,2,3] 周泰 郭涵[1,2,3] 张华宇 

机构地区：[1]北京大学深圳研究生院,广东深圳518055 [2]深圳市融合网络播控技术工程实验室,广东深圳518055 [3]深圳市云计算关键技术和应用重点实验室,广东深圳518055

出　　处：《计算机研究与发展》2013年第S2期54-63,共10页Journal of Computer Research and Development

基　　金：国家"九七三"重点基础研究发展计划基金项目(2012CB315904);国家自然科学基金项目(NSFC61179028);广东自然科学基金项目(S2011010000923;S2013020012822);深圳基础研究基金项目(JCYJ20130331144502026;JC201104210120A;JC201005260234A)

摘　　要：随着信息数据量越来越大,大数据存储的可靠性变得尤为重要,因而需要在存储系统中引入冗余.基于复制的传统策略虽然简单,但其存储效率和系统可靠性均不高.纠删码提供了一种不同于复制策略的有效存储方式,然而,当系统中有文件丢失时,纠删码(如Reed-Solomon,RS)需要较高的通信负载来进行文件修复,而且编解码效率不高.针对RS码的不足,在分布式系统中引入一种新颖的再生码,称为二元再生码(binary regenerating codes,BRGC).在Hadoop分布式存储系统中部署和实施了二元再生码和RS码.实验结果表明,BRGC在编解码效率和修复带宽上均优于RS码.With the amount of information becomes larger and larger,the reliability for big data storage becomes vitally important.Thus redundancy is introduced to storage systems.The traditional method of replication is simple,but it provides low storage efficiency and low system reliability. Erasure codes provide an efficient storage method which is different from the traditional replication method in distributed storage systems.However,when some of the encoded fragments are lost, erasure codes such as Reed-Solomn(RS)codes need a higher communication overhead for data recovery.Moreover,the encoding?decoding efficiency of RS codes is still low.To overcome the shortage of RS codes,we propose a novel family of regenerating codes called binary regenerating codes(BRGC),and implement BRGC and RS codes in Hadoop distributed file system(HDFS).Our experiments show that BRGC surpass RS codes in terms of both encoding?decoding efficiency and recovery bandwidth.

关 键 词：大数据 存储系统 纠删码 RS码 二元再生码 HADOOP 

分 类 号：TP333[自动化与计算机技术—计算机系统结构]