基于Hadoop的智慧工作岗位分析大数据平台的设计与实现  

Design and Implementation of Intelligent Jobs Analysis Big Data Platform Based on Hadoop

作　　者：张书贵[1] ZHANG Shugui(School of Information Engineering,Kaifeng University,Kaifeng Henan 475004,China)

机构地区：[1]开封大学信息工程学院,河南开封475004

出　　处：《信息与电脑》2024年第5期112-114,118,共4页Information & Computer

摘　　要：随着互联网的发展,网上招聘方式日益普及。但当求职者面对海量招聘信息,对不同维度的信息进行综合分析时会变得异常繁杂。鉴于此,文章设计了基于Hadoop的智慧工作岗位分析大数据平台,为用户提供岗位数据整合、归档、检索、浏览和分析挖掘服务。该平台以大数据分析技术为核心,利用Python编程爬取Web页面数据,利用Flume对数据进行聚合和日志采集,采用Kafka对数据进行缓冲,使用Spark对实时数据进行计算,利用MapReduce对清洗后的数据进行运算,最后利用数据大屏对数据进行可视化展示。With the development of the Internet,online recruitment is becoming more and more popular.However,when job seekers face a large number of recruitment information,the comprehensive analysis of different dimensions of information will become extremely complicated.In view of this,the paper designs a big data platform for intelligent job analysis based on Hadoop to provide users with job data integration,archiving,retrieval,browsing and analysis mining services.The platform takes the big data analysis technology as the core,uses Python programming to crawl Web page data,uses Flume to aggregate data and collect logs,uses Kafka to buffer date,uses Spark to calculate real-time data,and uses MapReduce to compute the cleaned data.Finally,the data is visualized by using large data screen.

关 键 词：智慧 工作岗位 大数据 HADOOP 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]