基于Scrapy框架的分布式网络爬虫系统设计与实现  

Design and Implementation of Distributed Web Crawler System Based On Scrapy Framework

作　　者：周毅 李威 何金 程蕾 柳璐 ZHOU Yi;LI Wei;HE Jin;CHENG Lei;LIU Lu(Information and Communication Branch of State Grid Liaoning Electric Power Supply Co.,Ltd.,Shenyang 110055,China)

机构地区：[1]国网辽宁省电力有限公司信息通信分公司,辽宁沈阳110055

出　　处：《现代信息科技》2021年第19期43-46,共4页Modern Information Technology

基　　金：国家电网有限公司科技项目:提高技术创新能力及群众性创新课题研究(2021YF-86)。

摘　　要：针对传统单机网络爬虫抓取效率低、稳定性差、数据量少等问题,文章利用Scrapy框架结合Redis技术,对传统网络爬虫框架进行改进和优化,设计出了分布式非结构化的网络爬虫系统,使采集到的信息能以非结构化形式存储于MongoDB数据库内,实现对数据信息的实时、有效处理分析。经过实际应用测试,证明基于Scrapy框架的分布式非结构化网络爬虫系统相较于传统单机系统具有更高的效率。Aiming at the problems of low capture efficiency,poor stability and small amount of data on traditional single-machine web crawler,this paper uses Scrapy framework and Redis technology to improve and optimize the traditional web crawler framework,and designs a distributed and unstructured web crawler system,which enables the collected information to be stored in the MongoDB database in an unstructured form,so as to achieve real-time and effective processing and analysis of data information.After practical application test,it is proved that the distributed and unstructured web crawler system based on Scrapy framework has higher efficiency than the traditional single-machine system.

关 键 词：分布式 Scrapy框架 网络爬虫 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]