网络爬虫的设计与实现  

Design and Implementation of Webcrawler

作　　者：唐波[1] TANG Bo(Department of Computer Science and Technology,Huaihua University,Huaihua 418100,China)

机构地区：[1]怀化学院计算机系

出　　处：《电脑知识与技术》2009年第4Z期2867-2868,共2页Computer Knowledge and Technology

摘　　要：Web爬虫有两种爬取策略,基于Webcrawler(web爬虫)设计的BFS(广度优先)策略,文章使用MD5算法,来进行O(1)时间复杂度的链接判重。为了避免频繁的查询DNS服务器,建立DNS缓存。另外,也因一般行为模式的考量,在中加入了IP范围控制技术,网页过滤方法,和多线程并发技术。最后,给出了此爬虫所需的时间分析,以供评估并后续发展。This paper introduce an optimized Webcrawler based on the BFS(Breadth First Search).The MD5 algorithm was used to hash search time complexity.DNS cache is established in order to avoid frequent DNS server search.In addition,the IP technology and web are added because of general behaviors patterns.In the end,the time analysis was provided for further improvement.

关 键 词：Webcrawler 广度优先爬取 多线程 DNS缓存 url判重 

分 类 号：TP393.092[自动化与计算机技术—计算机应用技术;自动化与计算机技术—计算机科学与技术]