基于HMM的主题爬虫研究  

Research on Focused Crawler Based on HMM

作　　者：谢治军[1] 杨武[2] 李稚楹[1] 宋静静[1] 

机构地区：[1]重庆理工大学计算机科学与工程学院,重庆400054 [2]重庆理工大学信息与教育技术中心,重庆400054

出　　处：《重庆工商大学学报（自然科学版）》2012年第2期66-72,共7页Journal of Chongqing Technology and Business University:Natural Science Edition

摘　　要：主题爬虫是垂直搜索引擎的核心组成部分,它为面向主题的用户查询准备数据资源;提出了一种基于HMM的主题爬虫方法,方法不仅分析网页内容,而且还考虑网页的上下文链接结构,首先将当前网页的聚类结果作为观察状态、将当前网页到目标网页的链接距离作为隐含状态,然后通过HMM模型学习用户的主题浏览模式并利用它采集更多的主题网页;实验结果表明:方法能采集大量与指定主题相关的高质量网页,主题爬行效率优于Best-First主题爬虫。Focused crawler is a core component of the vertical search engine, it collected data resources for the subject-oriented user＇s query. This paper proposes an approach for focused crawler based on HMM, it not only considers the web content, but also analyzes the context of web link structure. Firstly, the observation state represents the clustering of the current web page, the hidden state represents the link distance from current web page to target web page, then through the HMM model learning user browsing patterns, more topic webpages are downloaded by using the model. Experiments show that the focused crawler based on HMM can capture a large number of high quality web pages related to target topics, and its crawling oerforms better than Best-First crawler.

关 键 词：主题爬虫 隐马尔科夫模型 向量空间模型 主题相关度 垂直搜索引擎 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]