带重复数据删除的大规模存储系统可靠性保证  

Reliability provision mechanism for large-scale de-duplication storage systems

作　　者：顾瑜[1] 刘川意[1] 孙林春[1] 阎斌[1] 汪东升[1,2] 鞠大鹏[2] 

机构地区：[1]清华大学计算机科学与技术系,北京100084 [2]清华大学信息技术研究院,北京100084

出　　处：《清华大学学报（自然科学版）》2010年第5期739-744,共6页Journal of Tsinghua University(Science and Technology)

基　　金：国家自然科学基金资助项目(60833004)

摘　　要：重复数据删除技术将重复数据块归并存储,提高空间利用率却降低数据可靠性。该文提出一种数据可靠性保证机制——RLDS。RLDS将变长数据块打包成定长块,再使用ECC编码从一组数据块生成带纠错的块,分布到动态挑选的存储节点冗余组中。在节点发生错误时,RLDS分布式地恢复出错数据。实验结果表明:RLDS能取得与用RAID纠错机制的系统相近的空间利用率,同时提供与用多倍冗余复制机制的系统相近的数据可靠性,数据块平均恢复时延也比RAID系统大为缩短。RLDS通过数据再生还解决了节点超载问题,提供系统动态负载均衡功能。Data de-duplication technology achieves storage utilization by storing duplicate data chunks just once,but at the cost of error resilience or reliability.This paper presents a high reliability provision mechanism for large-scale de-duplication storage system（RLDS in short）.RLDS packages variable data chunks into fixed sized objects,while exploits ECC code to encode the objects and distributes them among storage nodes in a redundancy group,which is dynamically generated according to actual failure domains.RLDS uses a distributed recovery process upon failures.Experimental results show that RLDS can provide the same storage utilization as RAID schemes,as well as comparable reliability to replication based schemes with more redundancies.The results also show that the average recovery time of RLDS based configurations is much shorter than that of RAID schemes and that RLDS can provide run-time dynamic load balance by recovering data of overloaded nodes to light load nodes.

关 键 词：大规模存储系统 重复数据删除 可靠性 ECC纠错码 

分 类 号：TP309.3[自动化与计算机技术—计算机系统结构]