基于Hadoop的网络爬虫技术研究  

Research on Web Crawler Technology Based on Hadoop

作　　者：王艳红[1] 周军[1] 

机构地区：[1]南通航运职业技术学院管理信息系,江苏南通226010

出　　处：《吉林工程技术师范学院学报》2014年第8期87-89,共3页Journal of Jilin Engineering Normal University

摘　　要：网络爬虫一般从一个起始网页开始,读取网页的内容和网页中的链接,依次循环下去,直到找到此网页所有的链接网页为止;当要爬取的数据量比较大时,传统的技术存在一定弊端,而Hadoop开源云计算框架在数据采集方面会有一定的优势。在介绍Hadoop云计算框架的基础上,本文阐述网络爬虫的原理,并实现基于Hadoop的网络爬虫。The Web crawler usually starts from a starting Webpage, reads the content of webpage and Webpage links, successively circles until it finds all the webpage links; when you want to climb from the large amount of data, the traditional technology has some disadvantages, and the Hadoop open source cloud computing framework will have a certain advantages in data acquisition. On the basis of the introduction of Hadoop cloud computing framework, this paper describes the principle of the web crawler and realization of the web crawler based on Hadoop.

关 键 词：HADOOP 网络爬虫 MAPREDUCE 搜索引擎 

分 类 号：TP393.071[自动化与计算机技术—计算机应用技术]