Design of a Web Crawler for Water Quality Monitoring Data and Data Visualization  

作　　者：Ziwen Yu Jianjun Zhang Wenwu Tan Ziyi Xiong Peilun Li Liangqing Meng Haijun Lin Guang Sun Peng Guo 

机构地区：[1]College of Engineering and Design,Hunan Normal University,Changsha,410081,China [2]LIHERO Technology(Hunan)Co.,Ltd.,Changsha,410205,China [3]Big Data Institute,Hunan University of Finance and Economics,Changsha,410205,China [4]University Malaysia Sabah,Sabah,88400,Malaysia

出　　处：《Journal on Big Data》2022年第2期135-143,共9页大数据杂志（英文）

基　　金：This research was funded by the National Natural Science Foundation of China(No.51775185);Scientific Research Fund of Hunan Province Education Department(18C0003);Research project on teaching reform in colleges and universities of Hunan Province Education Department(20190147);Innovation and Entrepreneurship Training Program for College Students in Hunan Province(2021-1980);Hunan Normal University University-Industry Cooperation.This work is implemented at the 2011 Collaborative Innovation Center for Development and Utilization of Finance and Economics Big Data Property,Universities of Hunan Province,Open project,Grant Number 20181901CRP04.

摘　　要：Many countries are paying more and more attention to the protection of water resources at present,and how to protect water resources has received extensive attention from society.Water quality monitoring is the key work to water resources protection.How to efficiently collect and analyze water quality monitoring data is an important aspect of water resources protection.In this paper,python programming tools and regular expressions were used to design a web crawler for the acquisition of water quality monitoring data from Global Freshwater Quality Database(GEMStat)sites,and the multi-thread parallelism was added to improve the efficiency in the process of downloading and parsing.In order to analyze and process the crawled water quality data,Pandas and Pyecharts are used to visualize the water quality data to show the intrinsic correlation and spatiotemporal relationship of the data.

关 键 词：Water quality monitoring data web crawler data visualization 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]