改进的SNM中文语义重复记录检测算法  

Improved SNM Chinese Semantic Duplicate Record Detection Algorithm

作　　者：袁满[1] 穆永豪 王贵友 于再富 YUAN Man;MU Yonghao;WANG Guiyou;YU Zaifu(School of Computer and Information Technology,Northeast Petroleum University,Daqing 163318,China;Information Center,Zhaodong Branch of the Tenth Oil Production Plant of Daqing,Heilongjiang Province,Daqing 163000,China)

机构地区：[1]东北石油大学计算机与信息技术学院,黑龙江大庆163318 [2]黑龙江省大庆市第十采油厂肇东分公司信息中心,黑龙江大庆163000

出　　处：《吉林大学学报（信息科学版）》2021年第3期348-356,共9页Journal of Jilin University（Information Science Edition）

基　　金：黑龙江省哲学社会科学研究规划基金资助项目(19EDE334)。

摘　　要：为解决中文数据的重复检测问题,在SNM(Sorted-Neighborhood Method)算法的基础上提出了一种融合《同义词词林扩展版》和中文分词于一体的重复记录检测算法,通过利用《同义词词林扩展版》和Jaccard算法计算词语相似度,同时采用Python中Jieba中文分词的方式对语句进行分词,从而达到优化余弦相似度算法计算语句相似度。改进后的算法能有效的对记录为中文的字段和语句进行重复记录检测。并通过对某辅导机构学生测验信息数据集进行实验,实验结果表明,该算法的查全率及查准率比传统SNM算法有了很大提高。In order to detect the duplicate of Chinese data,we propose a duplicate record detection algorithm based on SNM( Sorted-Neighborhood Method) algorithm,which integrates the extended version of synonym word forest and Chinese word segmentation. Using the extended version of synonym word forest and Jaccard algorithm to calculate the similarity of words,the Chinese word segmentation in Python is used to segment sentences,to optimize cosine similarity algorithm and to calculate the similarity of sentences. The improved algorithm can effectively detect duplicate records of fields and sentences recorded in Chinese. The experiment on the test data set of students in a counseling institution shows that the recall ratio of the new algorithm is much higher than that of the traditional SNM algorithm.

关 键 词：相似重复记录 SNM算法 中文分词 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]