基于Python的新闻聚合系统网络爬虫研究  

Research on the Python-based Web Crawler for News Aggregation System

作　　者：左卫刚 ZUO Wei-gang(Shanxi Management Vocational College, Linfen Shanxi 041051, China)

机构地区：[1]山西管理职业学院,山西临汾041051

出　　处：《长春师范大学学报》2018年第12期29-33,共5页Journal of Changchun Normal University

摘　　要：本文开发了一套基于Python的网络爬虫,并预留API,从而构建一个新闻聚合系统。新闻聚合系统中的新闻数据需要爬虫来获取,然而不同的网站有不同的页面布局,本研究旨在创建一个能够从不同页面布局中提取数据的开源爬虫,其中包括网络爬虫、API、网络爬虫调度器以及Socket服务器的实现等。开发过程中使用Python语言开发网络爬虫,利用Beautiful Soup作为网络爬虫的web提取工具,以Laravel为web应用程序框架,以PHP作为主要后端语言,承载CMS和API。网络爬虫可以通过利用用户创建的配置文件来适应从不同的页面布局中提取数据,并将提取的数据导出到JSON文件或数据库系统中。This paper develops a Python-based web crawler and reserves the API to construct a news aggregation system.The news data in the news aggregation system needs to be acquired by the web crawler,but different websites have different page layouts.This study aims to create an open source crawler able to extract data from different page layouts,including the implementation of web crawler,API,web crawler scheduler and Socket server.In the development process,Python language is applied to develop the network crawler,BeautifulSoup is used as the web extraction tool of the web crawler,Laravel is adopted as the web application framework,PHP is used as the main back-end language to support CMS and API.Web crawler can adapt to different page layouts and extract data from them by using the configuration files created by users and export the extracted data to JSON file or database system.

关 键 词：网络爬虫 HTML提取 新闻管理系统 应用程序接口 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]