面向主题的网络蜘蛛技术研究及系统实现  

The Research and Implementation of Topic Spider Technique

作　　者：张博[1] 蔡皖东[1] 

机构地区：[1]西北工业大学计算机学院,陕西西安710072

出　　处：《微电子学与计算机》2009年第5期52-55,共4页Microelectronics & Computer

摘　　要：首先研究了面向主题的网络蜘蛛的关键技术:抓取目标描述、网页分析算法和网页搜索策略等,在此基础上,设计并实现了一个面向主题的网络蜘蛛系统(简称主题蜘蛛),采用向量空间模型来计算网页的主题相关度,使用改进的Shark-Search网页搜索策略来决定待抓取链接的访问次序,从种子网页开始,只爬行具有较高预测相关度的链接,仅采集与主题相关的网页,多线程对网页进行下载和分析,提高了主题网页采集的精度.This paper first researched key techniques of topic-oriented web spider,such as the crawling target description,web pages analyzing algorithms and the search strategy of web pages,then a topic-oriented web spider system called topic spider was designed and implemented.The topic spider system used vector space model to calculate the relevance of web pages and it used modified shark-search web search strategy to determine the visit order of hyperlinks waiting to crawl.It started from seed page,only crawling higher forecast of related hyperlinks,only gathering related web pages,and downloaded web pages using multi-threaded.It achieves high topic-specific information retrieval precision.

关 键 词：信息收集 主题蜘蛛 相关度计算 搜索策略 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]