Random Forests Algorithm Based Duplicate Detection in On-Site Programming Big Data Environment  

作　　者：Qianqian Li Meng Li Lei Guo Zhen Zhang 

机构地区：[1]University of Science and Technology Beijing,Beijing,100083,China [2]Beijing University of Posts and Telecommunications,Beijing,100876,China [3]Systems Engineering Institute AMS,Beijing,100071,China [4]Audio Analytic,2 Quayside,Cambridge,CB58AB,UK

出　　处：《Journal of Information Hiding and Privacy Protection》2020年第4期199-205,共7页信息隐藏与隐私保护杂志（英文）

基　　金：supported by the National Key R&D Program of China(Nos.2018YFB1003905);the National Natural Science Foundation of China under Grant No.61971032,Fundamental Research Funds for the Central Universities(No.FRF-TP-18-008A3).

摘　　要：On-site programming big data refers to the massive data generated in the process of software development with the characteristics of real-time,complexity and high-difficulty for processing.Therefore,data cleaning is essential for on-site programming big data.Duplicate data detection is an important step in data cleaning,which can save storage resources and enhance data consistency.Due to the insufficiency in traditional Sorted Neighborhood Method(SNM)and the difficulty of high-dimensional data detection,an optimized algorithm based on random forests with the dynamic and adaptive window size is proposed.The efficiency of the algorithm can be elevated by improving the method of the key-selection,reducing dimension of data set and using an adaptive variable size sliding window.Experimental results show that the improved SNM algorithm exhibits better performance and achieve higher accuracy.

关 键 词：On-site programming big data duplicate record detection random forests adaptive sliding window 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]