基于大数据技术的数据分析处理平台设计与实现  

Design and Implementation of Data Analysis and Processing Platform Based on Big Data Technology

作　　者：杨宇 刘昉[1] YANG Yu;LIU Fang(Guizhou Vocational Technology College of Electronics&Information,Kaili 556000,Guizhou)

机构地区：[1]贵州电子信息职业技术学院大数据与人工智能系,贵州凯里556000

出　　处：《电脑与电信》2023年第1期81-85,共5页Computer & Telecommunication

基　　金：第二批国家级职业教育教师教学创新团队课题研究项目,项目编号:ZI2021120101。

摘　　要：针对目前大数据分析处理组件多而乱、平台技术架构复杂等问题,综合应用MapReduce、Hive、Sqoop等大数据技术,结合Scrapy数据采集框架,设计一种数据分析处理平台,利用SpringBoot及Vue技术搭建Web系统,实现处理结果的可视化。应用该平台对中国蔬菜网上的某一批发市场数据进行分析处理,得出年度蔬菜最高最低差价、年度价格排名前十的蔬菜品种等指标统计结果,分析结果对该批发市场的蔬菜投放及价格调控具有一定的指导意义。In view of the problems such as multiple and chaotic components of big data analysis and processing and complex platform technical architecture,this paper comprehensively applies big data technologies such as MapReduce,Hive and Sqoop,combined with Scrapy data acquisition framework,to design a data analysis and processing platform,and uses SpringBoot and Vue technologies to build a Web system.It realizes the visualization of processing results.In this paper,the platform is used to analyze and process the data of a wholesale market in China Vegetable Net,and the highest and lowest price difference of annual vegetables and the top ten vegetable varieties in annual price are obtained.The results of the analysis have a certain guiding significance for the vegetable supply and price control in the wholesale market.

关 键 词：大数据 数据分析 Scrapy Hive HDFS Sqoop Springboot Vule 中国蔬菜网 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]