面向网站群的主题爬虫研究  

Bootstrapping Focused Crawler for Websites of The Same Type

作　　者：徐昊 沈江明 XU Hao;SHEN Jiang-ming(School of Information Science,Yunnan University of Finance and Economics,Kunming 650221,China;Yunnan Universities Research Center for Data Operation and Management Engineering,Kunming 650221,China;Yunnan Corporation,China Telecom Group,Kunming 650221,China)

机构地区：[1]云南财经大学信息学院,云南昆明650221 [2]云南省高校数据化运营管理工程研究中心,云南昆明650221 [3]中国电信股份有限公司云南分公司企业信息化部,云南昆明650221

出　　处：《软件导刊》2020年第8期109-112,共4页Software Guide

基　　金：云南省高校数据化运营管理工程研究中心建设项目(2017)。

摘　　要：聚焦爬虫(Focused Crawler)又称为主题爬虫,是从网络上获取特定主题数据的有效工具。为了避免传统聚焦爬虫预训练主题相关性分类器的繁复工作,提出一种自举聚焦爬虫(Bootstrapping Focused Crawler),用于从特定网站群中收集主题数据。自举聚焦爬虫省略了预先训练分类器的步骤,转而采用一些样本页面以相似度排序的方式替代分类器功能。在实验中,自举聚焦爬虫以牺牲一定准确率为代价,取得了0.62的召回率以及0.45的F1值,表现优于传统聚焦爬虫(召回率0.16、F1值0.25)。对于网站群主题数据采集任务,采用相似度排序替代主题分类器,不仅可以减轻分类器训练负担,还可以达到更好的效果。Focused crawler(also known as theme crawler)is an effective tool to get data in any specific domain from Web.However,conventional focused crawlers need a classifier to filter out the irrelevant webpages,and to get such a classifier is usually labor-inten⁃sive.In this paper,we propose a Bootstrapping Focused Crawler(BFC)for collecting information from a group of websites in the same category.Instead of pre-training a tailored classifier,BFC adopts a ranking module to do the classification.In the experiments,the re⁃call and F1-score of BFC is significantly better than conventional focused crawler,from which we could draw the conclusion that our approach is more effective for the crawling tasks within a group of similar websites.

关 键 词：爬虫技术 信息检索 自举聚焦爬虫 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]