基于网络数据自动提取的爬虫设计与实现  

Design and Implementation of Web Crawler Based on Internet Data Automatic Extraction

作　　者：张璐璐 吴丽杰 孙俊杰 王星月 ZHANG Lu-lu;WU Li-jie;SUN Jun-jie;WANG Xing-yue(Anhui Vocational College of Grain Engineering,Department of Information Technology,Hefei Anhui,230011,China)

机构地区：[1]安徽粮食工程职业学院信息技术系,安徽合肥230031

出　　处：《广州航海学院学报》2022年第4期74-78,共5页Journal of Guangzhou Maritime University

基　　金：安徽省职业与成人教育学会教研规划重点课题(azcg44);安徽高校自然科学重点研究项目(KJ2021A1562);安徽省教育厅高校质量工程项目(2021jxtd088)。

摘　　要：“互联网+”的兴起打破了数据信息的壁垒,网络爬虫被广泛应用于数据采集与数据分析中.以“豆瓣电影top250”为目标,采用基于Python语言的爬虫技术,对比BeautifulSoup、Re和Xpath 3种网页解析方式的不同,完成目标数据的爬取.实验结果表明:在数据解析速度上,Re最优;在网页解析逻辑上,BeautifulSoup最优;在综合使用角度上,Xpath较为适宜.The rise of the strategy of“Internet plus”breaks the barriers of data and information.Web crawler is widely used in data acquisition and data analysis in the massive Internet plus information.Taking“Douban movie top250”as the goal,using the crawler technology based on Python language,this paper explains the four steps of web crawler in detail,compares the differences of three web page parsing methods:BeautifulSoup,Regular Expression(Re)and XPath,and completes the crawling of target data.The experimental results show that Re is the best in data analysis speed;In terms of web page parsing logic,beautiful soup is the best;From the perspective of comprehensive use,XPath is more suitable.

关 键 词：网络爬虫 PYTHON BeautifulSoup RE XPATH 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]