基于N-Gram算法的数据清洗技术  

Data cleaning technology based on N-Gram algorithm

作　　者：马平全[1,2] 宋凯[1,2] 纪建伟[1] 

机构地区：[1]沈阳农业大学信息与电气工程学院,沈阳110866 [2]沈阳理工大学自动化与电气工程学院,沈阳110159

出　　处：《沈阳工业大学学报》2017年第1期67-72,共6页Journal of Shenyang University of Technology

基　　金：辽宁省教育厅科学研究项目(LG201610)

摘　　要：针对数据库中存在的大量相似重复数据,对相似重复记录的属性结构以及产生原因进行了分析,采用N-Gram算法对数据记录进行计算,得到代表每条记录属性的键值,即N-Gram值.依据该键值将数据库中的数据记录进行排序处理,建立有序的数据库,并对其中的数据记录进行相似度计算.运用排列合并的清洗思想对识别出来的相似重复数据记录进行清洗,实验结果表明,N-Gram算法有效提高了相似重复数据记录的查全率和查准率.Aiming at the plentiful approximately duplicate data in the database,the attribute structure of approximately duplicate records and the causing reason were analyzed.The data records were calculated with the N-Gram algorithm to get the key values,namely N-Gram values,which represented the attribute of every record.According to the key values,the data records in the database were ordered so as to form a well-organized database.In addition,the similarity of data records in the database was calculated.The identified approximately duplicate records were cleaned by applying the arranged combination cleaning idea.The experimental results show that the N-Gram algorithm effectively increases the recall ratio and precision ratio of approximately duplicate data records.

关 键 词：相似度 相似重复记录 属性 排序 合并 数据清洗 查全率 查准率 

分 类 号：TP311.11[自动化与计算机技术—计算机软件与理论]