基于Scrapy框架的微博用户信息采集系统设计与实现  

Design and Implementation of Weibo User Information Collection System Based on Scrapy Framework

作　　者：朱烨行[1] 赵宝莹 张明杰 魏笑笑[1] 卫昆 ZHU Yehang;ZHAO Baoying;ZHANG Mingjie;WEI Xiaoxiao;WEI Kun(School of Modern Post,Xi'an University of Posts and Telecommunications,Xi'an 710121,China;Information Technology Center,Luoyang Institute of Science and Technology,Luoyang 471023,China;College of Economics and Management,Xi'an University of Posts and Telecommunications,Xi'an 710121,China)

机构地区：[1]西安邮电大学现代邮政学院,陕西西安710121 [2]洛阳理工学院信息化技术中心,河南洛阳471023 [3]西安邮电大学经济与管理学院,陕西西安710121

出　　处：《现代信息科技》2023年第24期41-44,48,共5页Modern Information Technology

基　　金：陕西省社会科学基金项目(2020R048);陕西省科技计划项目(2021GY-180);陕西省科技厅项目(2021JM-467);陕西省教育厅科研计划项目(20JZ086)。

摘　　要：为深入了解新浪微博用户的有关情况,从中找出最有影响力的微博用户,发现当前新浪微博中的意见领袖,需要知道每位微博用户迄今为止已发表的微博数、关注数和粉丝数等信息。为此使用Python语言设计实现了一个基于Scrapy框架的网络爬虫,该爬虫根据输入的微博用户标识号ID抓取该用户目前已发表的微博数、关注数和粉丝数等信息。由于要连续多次访问微博网站,为了不让微博网站拒绝访问,在设计的爬虫中使用了用户代理和IP代理,其中IP代理选用隧道代理这一动态类型。实验结果表明下载七千多位微博用户的信息用时6小时22分钟。To gain a deeper understanding of Sina Weibo users'information,from which to identify the most influential Weibo users,discover opinion leaders on current Sina Weibo,it is necessary to know the number of Weibo posts,followers and fans of every Weibo user so far.To this end,a Web crawler based on the Scrapy framework is designed and implemented in Python language.The crawler captures the number of Weibo posts,followers,fans published by the user according to the Weibo user ID entered.We have to visit Weibo website for many times in succession,to prevent Weibo website from denying access,the User Agent and IP Proxy are used in the designed crawler.Among them,the IP proxy selects the dynamic type of tunnel proxy.The experimental results show that it takes 6 hours and 22 minutes to download the information of more than 7000 Weibo users.

关 键 词：Scrapy 网络爬虫 微博 用户代理 IP代理 

分 类 号：TP311.1[自动化与计算机技术—计算机软件与理论]