基于Larbin的网络爬虫体系结构的研究与改进  

Study and Improvement on System Architectures of Larbin Web Crawler

作　　者：李跃健[1] 朱程荣[1] 

机构地区：[1]同济大学计算机科学与技术系,上海201804

出　　处：《计算机技术与发展》2012年第7期147-150,170,共5页Computer Technology and Development

基　　金：国家863高技术发展计划项目(2010AA122200);上海市科委国际合作项目(10510712500)

摘　　要：Larbin是一种开源的网络爬虫/网络蜘蛛,抓取效率极高。它的url去重方法的设计,效率极高,占用的内存非常小,理论上下载6400万网页,使用的内存只有8M,然而它的冲突将会对它的性能大打折扣,实际上当达到10%的url时就已经有很大的冲突概率,导致内存利用率的降低以及很多网页不能被抓取。通过研究布隆过滤器,将url的hash算法进行改进,把原本一对一的映射变成多对一的映射,减小了冲突概率,同时也将大大地提高Larbin在url内存方面的利用率。经过实验检验,使用布隆过滤器,同样8M内存,当达到10%的url占有率时,采用7个映射,可以使得冲突概率最小,达到0.82%,而没采用Bloom filter的冲突概率则达到了10%。Larbin is an open source web crawler, it scratches pages very efficiently. On url comparing algorithm,it has great efficiency and cost very little memory. In theory,downloading 64 million pages cost only 8M memory,but its url conflict will greatly affect its perform- ance. In fact, when 10% of the urls are in memory, the new url will have 10% possibility to conflict, resulting in lower memory usage and many pages can not be crawled. By studying the Bloom filter,with the hash algorithm of url distinguish improves the original into a many -to-one mapping,reducing the probability of conflict,and also greatly enhance the Larbin＇s memory utilization. From the experiment,in the 8M memory with 10% used by url,if make the map number to be 7, the conflict percentage reaches to only 0.82% while it remains 10% if no bloom filter is applied to the algorithm.

关 键 词：Larbin 爬虫 哈希算法 url去重 布隆过滤器 

分 类 号：TP309[自动化与计算机技术—计算机系统结构]