分布式Web Crawler的研究:结构、算法和策略  

On Distributed Web Crawler: Architecture, Algorithms and Strategy

作　　者：叶允明[1] 于水[1] 马范援[1] 宋晖[1] 张岭[1] 

机构地区：[1]上海交通大学计算机科学与工程系,上海200030

出　　处：《电子学报》2002年第12A期2008-2011,共4页Acta Electronica Sinica

基　　金：国家自然科学基金(No.60221120145);上海市科委基础研究重点项目(No.02DJ14045)

摘　　要：本文介绍了一个大型分布式Web Crawler系统——Igloo 1.2版。它采用分布式的系统结构,通过我们设计的二级哈希映射算法使系统可以进行高效的任务分割,并且系统的规模动态可扩展.爬行网页的质量是评价Crawler的一个重要指标,Igloo以PageRank值作为网页质量评价的标准,从而提高了爬行质量.加快爬行速度的关键是如何解除Crawler系统中的性能瓶颈,本文对此也作了详细的讨论,并提出了一种基于“滞后合并”策略的UBL数据库存取方法.实验表明,Igloo在保持高性能的同时能快速爬行到高质量的网页.We describe a large - scale distributed Web Crawler system, i.e. Igloo VI.2. Igloo' s distributed architecture is based on our two-tiered Hash mapping algorithm, so that it can do efficient task partition while at the same time providing dynamic scalability. As the quality of crawled Web pages is an important factor for evaluating crawlers, it employs PageRank value as the evaluation metric of pages to improve its crawling efficiency. This paper also provides a detailed discussion of the performance bottlenecks in crawler systems,and proposes a new URL repository access method based on 'delayed merging' strategy to enable high-speed crawling. The experiments show Igloo can quickly crawl high-quality Web pages as well as present high performance.

关 键 词：WEB爬虫 爬行策略 分布式系统 计算机网络 网页 

分 类 号：TP393.092[自动化与计算机技术—计算机应用技术]