基于Hadoop的广域网分布式主题爬虫系统框架  

A framework of WAN distributed topic crawling system based on Hadoop

作　　者：王淑芬[1] 高军礼[1] 邹普 宋海涛[2] 

机构地区：[1]广东工业大学自动化学院,广东广州510006 [2]华南理工大学工商管理学院,广东广州510641

出　　处：《计算机工程与科学》2015年第4期670-675,共6页Computer Engineering & Science

基　　金：国家自然科学基金重大项目(710990403);中央高校基金项目(2014ZM0038);广东省省部产学研结合项目重点引导项目(2011B090400522)

摘　　要：广域网分布式爬虫与局域网爬虫相比有诸多的优势,而现有基于Hadoop分布式爬虫的设计主要是面向局域网环境的。为解决Hadoop分布式计算平台不适合部署于广域网的问题,设计了一个基于Hadoop的广域网分布式爬虫系统框架。爬虫系统利用消息中间件实现分布式可靠通信,数据存储采用可伸缩的Hadoop分布式文件系统HDFS,网页解析利用MapReduce并行处理,并基于模板匹配实现框架可定制。系统的性能仿真显示该框架具有支撑大规模爬虫并发工作的能力。Comparing with LAN crawling systems, WAN distributed crawling systems have lots of advantages, however, the existing crawling systems based on Hadoop are mostly used in LAN. To a- chieve a high computing speed of Hadoop in WAN, we present a crawler framework based on Hadoop. To achieve an extensible storage, all data are stored on the Hadoop distributed file system and the web pages are analyzed through MapReduce in parallel. To obtain reliable communication, a message oriented middleware is used. To make the framework customizable, a template matching method is proposed. The performance simulation shows that the crawler framework can support large scale crawling work.

关 键 词：分布式爬虫 HADOOP 爬虫框架 模板匹配 主题爬虫 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]