面向主题爬取的多粒度URLs优先级计算方法  

Focused Crawling Oriented Multi-Granular Priority Computation for URLs

作　　者：陈竹敏[1] 马军[1] 韩晓晖[1] 雷景生[2] 

机构地区：[1]山东大学计算机科学与技术学院,山东济南250101 [2]海南大学信息科学技术学院,海南海口570228

出　　处：《中文信息学报》2009年第3期31-38,共8页Journal of Chinese Information Processing

基　　金：高等学校博士学科点专项科研基金项目(20070422107);山东省科技攻关项目(2007GG10001002);海南省自然科学基金项目(80546)

摘　　要：垂直检索系统中主题爬虫的性能对整个系统至关重要。在设计主题爬虫时需要解决两个问题:一是计算当前页面与给定主题的相关度,二是计算待爬取URLs的访问优先级。对第一个问题,给出利用页面的主题文本块和相关链接块的相关度计算方法;对第二个问题,给出基于主题上下文和四种不同的粒度(即站点级、页面级、块级和链接级)的优先级计算方法。在此基础上,提出基于上述方法的主题爬取算法。实验证明,新算法在不增加时间复杂度的前提下,在查准率和信息量总和方面明显优于其他三种经典的爬取算法。The performance of the focused crawler is crucial to a vertical search engine. Two scientific computation issues to be addressed in the design of focused crawlers are： （1） how to compute the relevance of a current visited Web page to a given topic, （2） how to compute the priorities of unvisited URLs in the queue. For the first issue, this paper describes the calculation of the relevance of a page to the topic based on the page＇s topical text blocks and related link blocks. For the second one, a novel approach is proposed to prioritize these unvisited URLs by hierar- chical topic context of four different granularities, i.e. site level, page level, block level and link level. Finally, a new focused crawling algorithm is presented. Experiments show that the new algorithm is more effective than three traditional algorithms in terms of precision rate and information amount without increasing time complexity.

关 键 词：计算机应用 中文信息处理 主题爬取 优先级计算 网页分块 相关度计算 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]