一种Deep Web爬虫爬行策略  

Crawling Strategy of Deep Web Crawler

作　　者：刘徽[1] 黄宽娜[1] 余建桥[2] 

机构地区：[1]乐山师范学院数学与信息科学学院,四川乐山614004 [2]西南大学计算机与信息科学学院,重庆400715

出　　处：《计算机工程》2012年第11期284-286,共3页Computer Engineering

基　　金：四川省教育厅科研基金资助项目(10ZB023)

摘　　要：Deep Web包含丰富的、高质量的信息资源,由于没有直接指向Deep Web页面的静态链接,目前大多搜索引擎不能发现这些页面,只能通过填写表单提交查询获取。为此,提出一种Deep Web爬虫爬行策略。用网页分类器的分层结果指导链接信息提取器提取有前途的链接,将爬行深度限定在3层,从最靠近查询表单中提取链接,且只提取属于这3个层次的链接,从而减少爬虫爬行时间,提高爬虫的准确度,并设计聚焦爬行算法的约束条件。实验结果表明,该策略可以有效地下载Deep Web页面,提高爬行效率。Deep Web has rich,high-quality information resources.Because it does not have a static page links which is directly point to the Deep Web,most current search engines can not find these pages,but they can submit a query by filling out the form to obtain.This paper proposes a crawling strategy of Deep Web crawler.A hierarchical classification with the results of Web links guides the information extractor to extract the link.The strategy limits the crawling depth in three,extracts from the nearest link to the query form,and only extracts the links belonging to these three levels.Thus it reduces the reptiles crawling time,and improves the accuracy of reptiles.At the same time,it designs the Deep Web data sources for focused crawling algorithm.Experimental results show that this strategy can effectively download Deep Web pages to improve crawling efficiency.

关 键 词：DEEP WEB页面 反馈机制 爬行策略 聚焦爬虫 网络数据库 分类器 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]