基于Hadoop的重复数据删除技术研究  

Research on Duplicate Data Deletion Technology Based on Hadoop

作　　者：郭强 GUO Qiang(Department of Management,Liaoning Normal University Haihua College,Shenyang 110167,China)

机构地区：[1]辽宁师范大学海华学院管理系,辽宁沈阳110167

出　　处：《现代信息科技》2019年第3期142-144,共3页Modern Information Technology

摘　　要：重复数据删除技术和云计算技术正在快速发展,但是Hadoop系统也仍然存在一些影响系统归档处理效率的因素。针对这个问题,文章提出一种新的增量指纹算法和改进TTTD算法——TDOB,并将这类算法应用到Hadoop分布式系统中,分析重复数据删除技术在Hadoop中的应用。Duplicate data deletion technology and cloud computing technology are developing rapidly,but there are still some factors that affect the efficiency of archiving processing in Hadoop system. In order to solve this problem,a new incremental fingerprint algorithm and an improved TTTD algorithm--TDOB are proposed and applied to Hadoop distributed system. The application of repeated data deletion in Hadoop is analyzed.

关 键 词：HADOOP 重复数据删除技术 TDOB算法 

分 类 号：TP309.3[自动化与计算机技术—计算机系统结构]