多线程网络爬虫的设计与实现  

Design and Implementation of Multi-threads Web Crawler

作　　者：张超[1] 闫宏印[1] 

机构地区：[1]太原理工大学计算机科学与技术学院,太原030024

出　　处：《电脑开发与应用》2012年第6期65-67,70,共4页Computer Development & Applications

摘　　要：针对互联网信息急剧增多,为了改善网络爬虫的爬行性能和提高爬虫程序的通用性,分析了网络爬虫的原理与架构,设计实现了一种高速多线程网络爬虫程序。该爬虫程序采用多个线程并行处理网页,采用宽度优先和深度优先结合的方式来控制网页爬取深度。实验证明该爬虫程序减少了网页下载过程中的平均等待时间,具有较好的性能。For the increasingly information of the Internet, a new High-speed and Multi-threads Web Crawler （HMWC） was designed and implemented to improve the performance and enhance the versatility in thispaper. The web crawler dealt with the web page by using multi-threads and controlled the depth of crawling by mixed method of breadth-first and depth-first. Experimental result show that as a web crawler, HMWC canreduce the average waiting time of web page downloaded and has better performance.

关 键 词：搜索引擎 网络爬虫 多线程 URL队列 宽度优先 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]