基于改进遗传算法的聚焦爬虫设计  

Design of a Focused Crawler Based on the Improved Genetic Algorithm

作　　者：范会联[1] 李献礼[2] 曾广朴[1] 

机构地区：[1]长江师范学院数学与计算机学院,重庆408100 [2]长江师范学院网络信息中心,重庆408100

出　　处：《计算机工程与科学》2010年第5期126-129,共4页Computer Engineering & Science

基　　金：重庆市教委科学技术研究项目(KJ091309)

摘　　要：本文提出以爬行控制器和页面分析过滤器为核心的聚焦爬虫设计方法。从待检索主题出发,在以改进的遗传算法为基础并结合内容评价和链接结构搜索策略优点的爬行策略引导下,以待爬行URL作为遗传个体,基于主题词集的向量空间模型评估个体适应度,引入新的URL实现交叉、变异操作,将具有相同URL前缀的链接按小生境处理。实践证明,该爬虫具有较好的性能。The paper presents the design method for a focused crawler based on the crawling controller and the page a- nalysis filter. Starting from the theme to be retrieved, the method based on the improved genetic algorithm combines with the advantages of both content evaluation and link structure. The crawler regards the URL link as the genetic individual, and the topic-words-hased VSM is applied to assess individual fitness, and imports new URLs to achieve crossover and mutation operations, and the URLs that have the same prefix are regarded as niche. The experimental results show that the approach has better performance.

关 键 词：聚焦爬虫 爬行控制器 主题相关度 数据抽取 

分 类 号：TP391.3[自动化与计算机技术—计算机应用技术]