基于可靠性数据消冗技术的隐私大数据智能去重仿真  

Intelligent deduplication simulation of privacy big data based on reliable data redundancy technology

作　　者：孟锐 MENG Rui(Xi’an Technology and Business College,Xi’an 710043,China)

机构地区：[1]西安工商学院,陕西西安710043

出　　处：《电子设计工程》2025年第9期12-16,共5页Electronic Design Engineering

基　　金：陕西省教育厅科技科研计划项目(19JK0421)。

摘　　要：为了提高隐私大数据去重效果,设计了基于可靠性数据消冗技术的隐私大数据智能去重仿真方法。利用纠删码技术,通过编码与译码过程增强隐私大数据的可靠性。利用Bloom filter数据消冗技术,对比可靠性增强处理后隐私大数据的hash表与hash值,利用CDC分块算法设置滑动窗口,利用Rabin指纹算法运算滑动窗口内隐私大数据的指纹,完成隐私大数据分块。利用隐私大数据分块结果构建Bloom filter数据结构,设置数据对应的hash值作为隐私大数据的特征值。利用各数据特征值的相似性,更新隐私大数据对应的hash表,完成隐私大数据的智能去重处理。仿真分析结果表明,该方法可以实现文档、图像、源代码等不同类型隐私大数据的智能去重,存储开销均小于20%。To enhance the deduplication effectiveness of privacy-sensitive big data,a simulation method for intelligent deduplication of privacy-sensitive big data based on reliable data redundancy elimination technology has been designed.By utilizing erasure code technology,the reliability of privacy big data is enhanced through the encoding and decoding process.Using Bloom filter data redundancy technology,compare the hash table and hash values of privacy big data after reliability enhancement processing,use CDC blocking algorithm to set a sliding window,and use Rabin fingerprint algorithm to calculate the fingerprints of privacy big data within the sliding window,completing privacy big data blocking.Construct a Bloom filter data structure using the results of privacy big data partitioning,and set the corresponding hash value of the data as the feature value of privacy big data.By utilizing the similarity of feature values in each data,update the hash table corresponding to privacy big data to achieve intelligent deduplication of privacy big data.The simulation analysis results show that this method can achieve intelligent deduplication of different types of privacy big data such as documents,images,and source code,with storage costs all less than 20%.

关 键 词：可靠性 数据消冗技术 隐私大数据 智能去重仿真 纠删码技术 Bloom filter 

分 类 号：TN92[电子电信—通信与信息系统]