网络爬虫效率瓶颈的分析与解决方案  

Efficiency bottlenecks analysis and solution of Web crawler

作　　者：尹江[1] 尹治本[1] 黄洪[1] 

机构地区：[1]西南交通大学信息科学与技术学院,成都610031

出　　处：《计算机应用》2008年第5期1114-1116,1119,共4页journal of Computer Applications

摘　　要：网络爬虫的效率,直接关系到搜索引擎系统为用户提的供服务质量。如何设计高效、快速的网络爬虫,成为目前网络爬虫研究的热点。要提高网络爬虫的爬行效率,除了需要改进网络爬虫的爬行策略之外,还需要优化网络爬自身的设计,改进网络爬虫自身的结构,消除效率瓶颈。通过对网络爬虫结构、应用环境以及用户要求的分析,提出一个通用网络爬虫的改进设计方案,并通过实验得到较好的测试结果。The efficiency of a web crawler determines the quality of services a web searching system offers to its users. How to design a more efficient and faster web crawler is becoming a hot issue in the research of web crawler. In order to raise the crawling efficiency of a web crawler, the crawling strategy needs to be reformed. Besides, the design of the web crawler system has to be optimized and its structure also needs to be improved to eliminate bottlenecks. In this paper, an improved scheme of designing a general web crawler was presented through analyzing crawler＇s structure, application environment and user requirement, and the preferable testing result has proven better efficiency it has.

关 键 词：爬行策略 套接字 多线程 网络爬虫 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]