结合有监督广度优先搜索策略的通用垂直爬虫方法  

General Vertical Crawler Method Combined with Supervised Breadth-first Search Strategy

作　　者：高峰[1] 刘震[1,2] 高辉 GAO Feng;LIU Zhen;GAO Hui(School of Computer Science and Engineering,University of Electronic Science and Technology of China,Chengdu 611731,China;Big Data Research Center, University of Electronic Science and Technology of China,Chengdu 611731,China)

机构地区：[1]电子科技大学计算机科学与工程学院,成都611731 [2]电子科技大学大数据研究中心,成都611731

出　　处：《计算机工程》2018年第11期289-299,共11页Computer Engineering

基　　金：国家自然科学基金(61300018)

摘　　要：垂直爬虫程序无法直接移植到其他网站并且程序设计需要大量人工干预。为此,提出一种高可移植性的通用型垂直爬虫设计方法。自动识别目标主题和目录页面URL,并利用URL聚类生成URL正则表达式过滤器,以解决垂直爬虫中需人工维护初始URL队列的问题。然后,利用正则表达式过滤器和解析路径模板以及有监督的广度优先与网页赋权搜索策略,实现相关页面的精确定位和数据的快速准确提取。实验结果表明,该方法能够对不同网站实现高效、快速、通用的数据爬取。Aiming at the problem that the vertical crawler program can not be directly transplanted to other websites and the programming needs a lot of manual intervention,a highly portable and general vertical crawler design method is proposed.The target topic and directory page URL are automatically identified,and the URL regular expression filter is generated by URL clustering to solve the problem of manual maintenance of initial URL queue in vertical crawler.Using the regular expression filter,the parsing path template and the supervised breadth-first and Web page weighting search strategy,the accurate location of the relevant pages,the fast and accurate extraction of the data can be realized.Experimental results show that this method can achieve efficient,fast and universal data crawling for different websites.

关 键 词：垂直爬虫 URL聚类 赋权网页 路径模板解析 有监督广度优先搜索策略 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]