Hadoop下改进布隆过滤器算法的网页去重  

An improved Bloom Filter algorithm under the Hadoop for duplicated web page removal

作　　者：黄伟建[1] 杨海龙[1] 

机构地区：[1]河北工程大学信息与电气工程学院,河北邯郸056038

出　　处：《计算机工程与科学》2017年第2期285-290,共6页Computer Engineering & Science

基　　金：河北省自然科学基金(F2015402077);河北省重点基础研究项目(14964206D)

摘　　要：针对服务器中存储的大量重复和相似数据造成的空间浪费问题,改进的布隆过滤器(Bloom Filter)算法通过增加位数组并根据位数组的重复命中次数所计算的权重来动态优化重复数据的副本数,然后在Hadoop分布式集群下对改进的算法进行并行实现,以进一步提高作业处理效率。实验结果表明,与传统网页去重算法相比,改进的Bloom Filter算法的并行实现不仅提高了作业的处理效率,而且通过基于位数组下动态重复次数对副本数的优化,在一定程度上节省了服务器的存储空间。To solve the space waste problem existing in the server space where a lot of duplicated and similar data are stored, we propose an improved Bloom Filter algorithm, which adds an array of bit and dynamically optimizes the number of copies of duplicated data according to the weight calculated by the repeated hits of the bit array. Then, the improved algorithm is parallelized in the Hadoop distributed cluster to further improve the processing efficiency. Experimental results show that compared with traditional web duplicate removal algorithms, the improved Bloom filter algorithm can not only improve the processing efficiency of jobs, but also save the server storage space to a certain extent by dynamically optimizing the number of copies of duplicated data according to the repeated hits of the bit array.

关 键 词：HADOOP 布隆过滤器 副本数 MAP REDUCE 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]