基于聚类分析和缓存优化的数据去重策略  

Data deduplication strategy based on clustering analysis and cache optimization

作　　者：裴世豪 刘颖[1] 李佳阳 郝欣哲 PEI Shihao;LIU Ying;LI Jiayang;HAO Xinzhe(School of Information Science and Technology,North China University of Technology,Beijing 100144,China)

机构地区：[1]北方工业大学信息学院,北京100144

出　　处：《计算机应用文摘》2024年第18期114-118,共5页

基　　金：山西省省级重点研发计划:面向物联网的计算机系统安全性分析(202202010101006)。

摘　　要：针对传统去重算法无法兼顾去重率和吞吐量的问题,设计了一种相似聚类重复数据删除算法。该算法基于数据相似性理论定义数据项间的相似度,将相似的数据项进行分类和标记,然后在缓存中保留每个聚类中的部分特征数据。当新数据录入时,算法根据数据特征选择合适的聚类进行数据去重。此外,为高效利用有限的缓存,提出了一种基于随机森林算法的缓存优化方法,用于优化去重过程中使用的指纹缓存,以提高缓存指纹的命中率。该缓存模型基于传统的随机森林分类器,并使用改进的烟花算法(ELU函数优化)对随机森林的超参数进行优化,能够有效应对数据量过大、指纹过多以及缓存利用有限的问题。实验验证表明,与基于数据相似原理的RMD和Shingle方法相比,所提算法在去重率和吞吐量方面均提高了10%~15%。Aiming at the problem that traditional deduplication algorithms cannot balance deduplication rate and throughput,a similarity clustering duplicate data deletion algorithm has been designed.This algorithm is based on the theory of data similarity to define the similarity between data items,classify and label similar data items,and then retain some feature data from each cluster in the cache.When new data is entered,the algorithm selects appropriate clusters based on data features for data deduplication.In addition,to efficiently utilize limited cache,a cache optimization method based on random forest algorithm is proposed to optimize the fingerprint cache used in the deduplication process and improve the hit rate of cached fingerprints.This caching model is based on a traditional random forest classifier and uses an improved fireworks algorithm(ELU function optimization)to optimize the hyperparameters of the random forest,which can effectively address the problems of large data volume,excessive fingerprints,and limited cache utilization.Experimental verification shows that compared with the RMD and Shingle methods based on the principle of data similarity,the proposed algorithm improves deduplication rate and throughput by 10%to 15%.

关 键 词：数据去重 灾难备份 数据相似性 烟花算法 随机森林 

分 类 号：TP309[自动化与计算机技术—计算机系统结构]