协同过滤下混合大数据无损挖掘算法研究  

Study on Nondestructive Mining Algorithm for Hybrid Big Data Under Collaborative Filtering

作　　者：卢思安[1] 刘江平[1] LU Si-an;LIU Jiang-ping(College of Computer and Information Engineering,Inner Mongolia Agricultural University,Hohhot Inner Mongolia 010018,China)

机构地区：[1]内蒙古农业大学计算机与信息工程学院,内蒙古呼和浩特010018

出　　处：《计算机仿真》2024年第4期485-488,共4页Computer Simulation

基　　金：内蒙古农业大学青年教师科研能力提升专项(BR 220116);内蒙古科技厅关键技术攻关项目(2020GG0169);内蒙古自治区自然科学基金项目(2022MS06026)。

摘　　要：大数据具有大规模性、多样性以及价值性,由于海量数据间的较高相似度,导致数据挖掘过程易受冗余干扰,出现数据丢失、损坏等问题。为解决上述问题,提出基于协同过滤算法的混合大数据无损挖掘方法。对混合大数据集成预处理,去除冗余,将不同来源的相同数据无损融合。采用协同过滤算法的时间衰减函数,计算挖掘项目间相似性。在混合大数据特征关联度的约束下,实现混合大数据无损挖掘。实验结果表明,所提方法应用下,混合大数据量高达25000MB时,数据挖掘所需时间仅为45ms左右,且挖掘精度高达95%以上,数据挖掘结果与目标具有一致性。Due to the high similarity between massive data,the data mining process is vulnerable to redundant interference,leading to data loss and data damage.Therefore,a lossless method of mining mixed big data based on collaborative filtering algorithm was presented.Firstly,the mixed big data were integrated,and the redundancy was removed.And then,the same data from different sources were integrated without loss.Moreover,the time decay function based on the collaborative filtering algorithm was used to calculate the similarity between mining items.Under the constraint of feature association degree of mixed big data,lossless mining for mixed big data was realized.Experimental results prove that when the mixed big data reaches 25000 MB,the time required for data mining is only about 45 ms,and the mining accuracy is more than 95%.The data mining results are consistent with the expected value.

关 键 词：协同过滤算法 混合大数据 无损挖掘 数据清理 数据集成 

分 类 号：TP242[自动化与计算机技术—检测技术与自动化装置]