基于深度学习的自动化数据质量检测方法研究  

Research on Automated Data Quality Detection Methods Based on Deep Learning

作　　者：黄志远 秦莎莎 HUANG Zhiyuan;QIN Shasha(Guangdong Tobacco Shaoguan Co.,Ltd.,Shaoguan Guangdong 512029,China)

机构地区：[1]广东烟草韶关市有限公司,广东韶关512029

出　　处：《信息与电脑》2024年第24期178-180,共3页Information & Computer

基　　金：基于大数据分析的现代终端数据治理研究(项目编号:韶烟科项申202404)。

摘　　要：为了提升数据质量检测的精度与效率,文章分析了基于深度学习的自动化数据质量检测方法。利用卷积自编码器(Convolutional Autoencoder,CAE)、循环神经网络(Recurrent Neural Network,RNN)和生成对抗网络(Generative Adversarial Networks,GAN)对财务数据集、传感器数据集和零售终端数据集进行实验。结果表明,深度学习模型在低质量数据集上的检测精度为财务数据集94.1%、传感器数据集92.8%、零售终端销售数据集95.6%,在高质量数据集上检测精度进一步提升,表现出较高的适应性和鲁棒性。该方法在检测准确性上优于规则匹配方法,具有显著优势。In order to improve the accuracy and efficiency of data quality detection,the article analyzes the automated data quality detection method based on deep learning.Experiments are conducted on financial dataset,sensor dataset and retail terminal dataset using Convolutional Autoencoder(CAE),Recurrent Neural Network(RNN)and Generative Adversarial Network(GAN),and the results show that the detection accuracy of the deep learning model on the low-quality dataset is 94.1%for the financial dataset,94.8%for the sensor dataset 92.8%,and retail terminal sales dataset 95.6%,and the detection accuracy is further improved on high-quality datasets,showing high adaptability and robustness.The method outperforms the rule matching method in detection accuracy and efficiency,and has significant advantages.

关 键 词：深度学习 数据质量检测 卷积神经网络 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]