一种支持海量数据备份的可扩展分布式重复数据删除系统  

A Scalable Distributed Data Deduplication System to Backup Massive Storage

作　　者：刘厚贵[1,2] 邢晶[1] 霍志刚[1] 安学军[1] 

机构地区：[1]中国科学院计算技术研究所,北京100190 [2]中国科学院大学,北京100049

出　　处：《计算机研究与发展》2013年第S2期64-70,共7页Journal of Computer Research and Development

基　　金：国家"九七三"重点基础研究发展计划基金项目(2012CB316502);国家"八六三"高技术研究发展计划基金项目(2013AA01A211)

摘　　要：现代信息化社会中,数据的爆炸式增长使得存储和管理数据的成本越来越高,数据备份和恢复的时间越来越长,数据中心的能耗越来越严重.重复数据删除技术在缓解数据的高速增长和节约成本方面有很好的应用前景.然而面对日益增长的海量数据,重复数据删除技术需要在数据路由和指纹查询方面提供高可扩展的支持.对于数据路由,提出了基于分布式布隆过滤器(Bloom Filter)的有状态数据路由技术:拆分Bloom Filter并将其分布于多节点并发管理,可以在提高数据路由的扩展性同时降低其通信开销.对于去重节点内的指纹查询,提出了基于取样机制的指纹查询优化方法:通过可扩展Hash结构组织海量取样指纹信息,并根据取样指纹的时间局部性原理对指纹缓存进行管理.为了对研究工作进行评价,设计并实现了分布式重复数据删除系统3D-deduper(distributed data deduplication system),并进行了数据路由和去重性能相关的测试.实验结果表明,基于分布式布隆过滤器的数据路由技术能够使得分布式重复数据删除系统维持与单节点重复数据删除系统相当的去重率,波动低于5%.在取样查询方面,数据去重率在取样率为1?32时仍能达到非取样重复数据删除系统数据去重率的95%以上.在去重性能方面,单节点去重带宽最高可达到200MBps.在节点扩展时,去重带宽仍能保持近似线性的扩展性.With the advent of the information society,the explosive data growth leads to rapid increase of data storage and management cost,high overhead to finish a backup or rebuild of the data and huge energy consumed by data center.Deduplication technology has good application prospects on relieving the burden of data growth and saving money. However,to support massive data deduplication at hundreds of PB,we have to improve the scalability of data routing and fingerprint lookup in data deduplication.To improve the scalability of data routing,we proposes a stateful data routing method based on distributed Bloom Filter technology.To improve the efficiency of fingerprint,we propose a scalable fingerprint searching method based on sampling strategy. Extendible hashing and time locality aware cache management are adopted in fingerprint searching method to improve data deduplication efficiency. We design and implement a distributed data deduplication system named 3D-deduper(distributed data deduplication system)to evaluate our research.The experiment shows while using stateful data routing method,the deviation of deduplication ratio between distributed Bloom Filter method and one single Bloom Filter method is within 5 percent.As for fingerprint searching,we can obtain 95 percent of the Non-sampling deduplication by using the sampling rate of 1?32.Finally,the system can provide peak throughput of 200MBps on a single deduplication node.As the number of deduplication nodes grows,the aggregated deduplication throughput can achieve nearly linear scalability.

关 键 词：重复数据删除 数据路由 可扩展哈希 分布式布隆过滤器 

分 类 号：TP309.3[自动化与计算机技术—计算机系统结构]