面向电信网数据的ETL系统的设计与实现  

Design and Implementation of ETL System for Telecommunication Network Data

作　　者：安轲 马宏[1] 李英乐[1] 刘树新 AN Ke;MA Hong;LI Yingle;LIU Shuxin(Information Engineering University, Zhengzhou 450001, China)

机构地区：[1]信息工程大学,河南郑州450001

出　　处：《信息工程大学学报》2020年第4期442-447,共6页Journal of Information Engineering University

基　　金：国家自然科学基金青年科学基金资助项目(61803384)。

摘　　要：电信网数据常分布在若干个独立的信息系统中,具有数据量大、数据结构差异大等特点,数据如何从各个数据源汇聚到目的仓库中成为数据分析的基础,需要数据处理工具ETL(Extract-Transform-Load)。传统的ETL工具难以满足电信数据的分析要求,为此结合Hadoop生态圈设计一种基于Spark的分布式ETL系统,通过设计批处理层和流处理层可以对电信网中的历史数据和流数据分别进行数据转换。测试结果表明,该系统在处理电信网大数据时具有较好的性能,同时可以动态地监控管理ETL任务。Telecommunication network data is often distributed in several independent information systems,with the characteristics of large data volume and large differences in data structure.The aggregation of data from various data sources to the target warehouse becomes the basis of data analysis.This requires the data processing tool ETL(Extract-Transform-Load).Traditional ETL tools are difficult to meet the analysis requirements of telecommunication data.To this end,a distributed ETL system based on Spark is designed in combination with the Hadoop ecosystem.By designing batch layer and stream layer,historical data and stream data in telecommunication network can be converted respectively.The test results show that the system designed in this paper has good performance in processing telecom network big data and can monitor and manage ETL tasks dynamically.

关 键 词：电信网数据 ETL 分布式 SPARK 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]