基于统计模型的主题爬虫的研究与实现  

Research and implementation for topic crawler using statistic model

作　　者：金明珠[1] 丁岳伟[1] 

机构地区：[1]上海理工大学光电信息计算机工程学院,上海200093

出　　处：《计算机工程与设计》2010年第16期3700-3704,共5页Computer Engineering and Design

摘　　要：在研究了现存的主题爬虫的基础上,提出了一种基于统计模型的主题爬虫,它对抓取过程中可获得的信息进行分析,并运用统计模型计算的结果过滤URL,有效地解决了偏好特定主题的用户检索和Web信息的索引等相关问题。实验结果表明,与基于链接和网页内容分析的主题爬虫相比,该主题爬虫能够在检索较少的网页时,抓取到较多的与主题相关的网页,提高了抓取精度。Based on the analysis of the existed topic crawlers,another topic crawler using statistic model is proposed,which analyzes the information available and filters the URLs by using the results calculated from the statistic model during crawling,and to effectively addresses the problem how to index the mass web resource and how to find specific topic pages fit user’s interest.The experimental results show that,compared with topic crawlers based on linkage and page content analysis,topic crawler using statistical model can fetch more topic relevant web pages by retrieving less web pages,and improve the crawling accuracy.

关 键 词：统计模型 主题爬虫 URL过滤 特征信息 字段 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]