内容分块算法中预期分块长度对重复数据删除率的影响  

Influence of Expected Chunk Size on Deduplication Ratio in Content Defined Chunking Algorithm

作　　者：王龙翔[1] 董小社[1] 张兴军[1] 王寅峰[2] 公维峰 魏晓林[1] 

机构地区：[1]西安交通大学电子与信息工程学院,西安710049 [2]深圳信息职业技术学院软件学院,广东深圳518172 [3]浪潮集团高效能服务器和存储技术国家重点实验室,济南250101

出　　处：《西安交通大学学报》2016年第12期73-78,共6页Journal of Xi'an Jiaotong University

基　　金：国家自然科学基金资助项目(61572394);国家重点研究发展计划资助项目(2016YFB1000303);深圳市基础研究资助项目(JCYJ20120615101127404;JSGG20140519141854753)

摘　　要：针对基于内容分块重复数据删除方法缺少能够定量分析预期分块长度与重复数据删除率之间关系的数学模型,导致难以通过调整预期分块长度优化重复数据删除率的问题,提出了一种基于Logistic函数的数学模型。在大量真实数据测观察基础上,提出了通过Logistic函数描述非重复数据的"S"形变化趋势,解决了该数据难以从理论上推导、建模的问题,证明了基于内容分块过程服从二项分布,并从理论上推导出了元数据大小模型。基于上述两种数据模型,通过数学运算最终推导得到重复数据删除率模型,并利用收集到的3组真实数据集对模型进行了实验验证。实验结果表明:反映数学模型拟合优度的R2值在0.9以上,说明该模型能够准确地反映出预期分块长度与重复数据删除率之间的数学关系。该模型为进一步研究如何通过调整预期分块长度使重复数据删除率最优化提供了理论基础。A logistic function based mathematical model is proposed to solve the problem that, mathematical model is not used to quantitatively analyze the relationship between the expected chunk size and the deduplication ratio in the content defined chunking （CDC） based deduplication method, resulting in the difficulty in optimizing the deduplication ratio by adjusting the expected chunk size. The logistic function is used to describe the S-shaped variation trend of unique data on the basis of observation on a large number of real data sets, and the problem that the unique data is hard to be modeled by theoretical derivation is solved. It is assumed that the CDC process fits a binomial distribution, and based on the assumption two metadata models are deduced theoretically. The deduplication ratio model is finally deduced based on these two data models.Three realistic datasets are used to verify the deduplication ratio model. Experimental results show that the R2 value, which denotes the goodness of fit of the model, is greater than 0. 9 in most results inferring the correctness of the model. The mathematical model may facilitate the study on optimization of deduplication ratio by reasonably setting the expected chunk size.

关 键 词：基于内容分块 重复数据删除率 Logistic函数 

分 类 号：TP333[自动化与计算机技术—计算机系统结构]