基于云平台的网页抓取架构的研究与设计  

Research and design of cloud-based web scraping

作　　者：范顺利 周亦敏[1] Fan Shunli;Zhou Yimin(School of Optical-Electrical and Computer Engineering,University of Shanghai for Science and Technology,Shanghai 200093,China)

机构地区：[1]上海理工大学光电信息与计算机工程学院,上海200093

出　　处：《计算机时代》2018年第9期21-23,27,共4页Computer Era

摘　　要：随着网络数据的迅速增长,网页数据抓取在处理大量数据时遇到了一些挑战,例如大量数据存储,需要密集计算能力以及数据提取的可靠性。提出了一个基于云平台的网页数据抓取架构,该架构使用AWS(Amazon Web Services)作为云平台,按需配置计算资源和数据存储;Selenium作为网页自动化工具,调用Web Driver API能够模拟用户使用浏览器。通过实验,比较了该架构与其他基于云的网页抓取架构的不同,并分析了它的优势。With the rapid growth of data on the Internet,web scraping encounters a number of challenges in processing large amounts of data,such as large amounts of data storage,the need for intensive computing capabilities,and the reliability of data extraction.So,a web scraping architecture based on the cloud platform is proposed.The architecture uses AWS(Amazon Web Services)as the cloud platform and configures computing resources and data storage on demand.Selenium is a web page automation tool;the supported WebDriver API simulates the process of users using the browser.Through experiments,the differences between this architecture and other cloud-based web scraping architectures are compared,and its advantages are analyzed.

关 键 词：AWS 网页抓取 大数据 SELENIUM 

分 类 号：TP311.5[自动化与计算机技术—计算机软件与理论]