基于K-means的大数据相似重复记录检测  

Detection of Similar Duplicate Records of Big Data Based on K-means

作　　者：张平 程新莲 ZHANG Ping;CHENG Xinlian(School of Information Engineering,Anhui Vocational and Technical College,Hefei 230011,China;Jiashan D-max Electronics Co.,Ltd.,Jiaxing 314100,China)

机构地区：[1]安徽职业技术学院信息工程学院,安徽合肥230011 [2]嘉善万顺达电子有限公司,浙江嘉兴314100

出　　处：《现代信息科技》2022年第8期89-91,共3页Modern Information Technology

基　　金：2021年校级质量工程项目(2021xjtz107)。

摘　　要：目前大型企业存储了大量的数据,但是数据质量令人担忧,集中表现在相似重复冗余的数据特别多,以及多个数据源的合并加重数据的冗余。大数据相似记录检测环节是数据清洗研究的重要方向。针对大数据中存在的相似重复数据的检测问题,文章提出了一种基于k-means分组聚类的检测算法,实验分析表明,该方法在确保精度不变的情况下提高了检测效率。At present, large enterprises store a large amount of data, but the data quality is worrying. It expresses especially that there are much similar, repeated and redundant data, and the combination of multiple data sources aggravates the redundancy of data. Big data similarity record detection part is an important direction of data cleaning research. Aiming at the detection problem of similar and repeated data existing in big data, this paper proposes a detection algorithm based on K-means grouping clustering. Experimental analysis shows that this method improves the detection efficiency while ensuring the accuracy is constant.

关 键 词：相似重复记录 K-MEANS SNM 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]