基于Python的分布式多主题网络爬虫的研究与设计  

Research and Design of Distributed Multi-topic Web Crawler Based on Python

作　　者：张胜敏[1] 王爱菊 ZHANG Shengmin;WANG Aiju(College of Information Engineering,Kaifeng University,Kaifeng 475004,Henan;College of Information and Engineering,Zhengzhou Institute of Technology,Zhengzhou 450044,Henan)

机构地区：[1]开封大学信息工程学院,河南开封475004 [2]郑州工程技术学院信息工程学院,河南郑州450044

出　　处：《开封大学学报》2021年第1期93-96,共4页Journal of Kaifeng University

基　　金：河南省科技攻关计划项目(212102210398)。

摘　　要：针对传统网络爬虫抓取速度较慢的问题,提出了基于Python的分布式多主题网络爬虫的研究与设计。首先,通过Python集群与Storm集群的组建,搭建了网络爬虫的分布式物理架构,通过指标计算部分和处理部分,搭建了网络爬虫的分布式逻辑架构,两种架构共同组成了分布式多主题网络爬虫的总体架构;其次,通过API接口抓取数据、GUI接口下载数据、编写爬虫程序下载数据的设计,完成了网络爬虫的抓取途径设计。实验证明,这样设计出的网络爬虫拥有较高的抓取速度。Aiming at the problem of low crawl speed of traditional web crawlers,a research and design of distributed multi-topic web crawlers based on Python is proposed.Firstly,the distributed physical architecture of the web crawler is constructed through the establishment of the Python cluster and the Storm cluster.The distributed logical architecture of the web crawler is constructed through the indicator calculation part and the processing part.The two architectures together form the overall architecture of the distributed multi-topic web crawler.Secondly,the design of grabbing data through API interface,downloading data through GUI interface,and writing crawler program to download data completed the design of crawling way for web crawlers.The experiments show that the designed web crawler has a high crawling speed.

关 键 词：PYTHON 分布式 网络爬虫 Storm集群 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]