大数据量的高效重复记录检测方法  

Efficient duplicate records detection method for massive data

作　　者：庞雄文[1] 姚占林[2] 李拥军[2] 

机构地区：[1]华南师范大学计算机学院,广东广州510300 [2]华南理工大学计算机科学与工程学院,广东广州510640

出　　处：《华中科技大学学报（自然科学版）》2010年第2期8-11,共4页Journal of Huazhong University of Science and Technology(Natural Science Edition)

基　　金：广东省安全生产科研专项资金资助项目(x2jsB2080910);广东省自然科学基金资助项目(9451063101002213)

摘　　要：针对目前重复记录检测方法不能有效处理大数据量的问题,提出了一种高效的重复记录检测方法.根据概念依赖图计算表的关键属性,并根据关键属性值将数据划分为记录集,在划分后的记录集中进行重复记录检测,大大减少需要比较的记录数,提高算法的效率;在记录集内进行重复记录检测时,将已匹配的记录合并后和已有的记录重新比较,提高了算法的准确度和效率.实验数据显示在大数据量情况下,计算效率提高47%.At present,the duplicate records of massive data can not be detected effectively by current methods.Thus an efficient duplicate records detection method for massive data is presented.The method firstly partitions records according to the key attributes which are calculated using concept dependency graph,and then detects duplicate records in each class of key attribute values;it can reduce the number of record comparisons greatly and improve the efficiency of duplicate detection.To improve the performance of duplicate detection,the method firstly merges the matching records into a new record,and then re-compares this record with records in the record set.We experimentally evaluate the method with actual data and reveal that the execute efficiency is improved up to 47% for massive data.

关 键 词：数据处理 重复记录检测 检测方法 概念依赖图 数据清洗 

分 类 号：TP274[自动化与计算机技术—检测技术与自动化装置]