MapReduce模型下增量重复数据检测方法  

Incremental duplicate data detection method with MapReduce

作　　者：董富森 杨波[1] 马坤[1] 王文华[1] 

机构地区：[1]济南大学山东省网络环境智能计算技术重点实验室,山东济南250022

出　　处：《济南大学学报（自然科学版）》2015年第4期241-245,共5页Journal of University of Jinan(Science and Technology)

基　　金：国家自然科学基金(61173078)

摘　　要：针对重复数据检测过程中增量数据重复值检测问题进行分析,在基本近邻排序算法基础上,提出增量近邻排序比较算法。该算法通过跳动窗口形式比较相邻数据,大大减少了数据比较次数;同时引入MapReduce模型对该算法加以改进以提高其海量数据处理的能力。实验表明,改进后的增量近邻排序比较算法在保证检则结果准确的前提下,能够有效提高增量数据重复检测的速度,并且算法具有较高的稳定性,更适应海量数据环境中重复数据检测任务。The incremental duplicate data detection problem in data cleaning was analyzed. A new algorithm to detect the incremental duplicate data based on sorted- neighborhood method was proposed. The proposed algorithm can greatly reduce comparing times through jumping windows; MapReduce was combined with the new method to improve its detection performance in large-scale data environment. Experiments show that The improved algorithm can improve the speed of incremental data detection under high detection accuracy. The method is very stable,which made it become more useful in massive data situations.

关 键 词：增量重复数据检测 跳动窗口 MAPREDUCE模型 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]