对基于SNM数据清洗算法的优化  

Optimization algorithm for cleaning data based on SNM

作　　者：张建中[1] 方正[2] 熊拥军[1] 袁小一[1] 

机构地区：[1]中南大学信息科学与工程学院,湖南长沙410083 [2]中南大学化学化工学院,湖南长沙410083

出　　处：《中南大学学报（自然科学版）》2010年第6期2240-2245,共6页Journal of Central South University:Science and Technology

基　　金：国家自然科学基金资助项目(50874119)

摘　　要：对基本邻近排序算法SNM(basic sorted-neighborhood method)进行分析,指出其不足;提出基于SNM算法的一种优化算法,通过采集中南大学冶金矿物工程机构知识库的2 000多条文献记录作为样本数据进行实验研究,对记录的"脏数据"按照DC标准和相关规范进行清洗与排重。研究结果表明:与SNM算法相比,在同样的运算环境下,优化算法在招回率、误识别率和执行时间上有明显优势。The basic sorted-neighborhood method（SNM） was introduced and the analysis was made on its deficiency.An improved algorithm of data cleaning based on SNM was put forward.And the experiments were made on more than 2 000 sample records data from the mineral metallurgy institutional database of Central South University.Key task was cleaning dirty data and removing approximately duplicate records according to dublin core（DC） standard and other criterion.The results show that the improved algorithm is better than SNM in the aspects of recall,precision and run time in the same computer condition.

关 键 词：数据挖掘 数据清洗 重复记录 SNM算法 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]