基于Kafka、Disruptor技术对传统ETL的改进  

Improvement of Traditional ETL Based on Kafka and Disruptor Technology

作　　者：王梓 梁正和[1] 吴莹莹 WANG Zi;LIANG Zheng-he;WU Ying-ying(School of Computer and Information,Hohai University,Nanjing 211100,China)

机构地区：[1]河海大学计算机与信息学院,江苏南京211110

出　　处：《计算机技术与发展》2018年第11期26-29,共4页Computer Technology and Development

基　　金：国家自然科学基金(61272543)

摘　　要：ETL系统是构建和维护数据仓库的基本构件,对异构数据源中的业务数据进行抽取、清洗、转换可通过ETL工具将其装载到数据仓库中。但是,当数据量上升到一定程度时,传统的ETL在数据处理速度以及数据的准确性方面会大大降低,并且不能满足数据源多种多样的变化需求。针对如何同时具有高效的数据处理能力和通用的数据源访问能力的问题,提出一种对传统ETL进行改进的方案。利用Kafka和Disruptor并发框架相结合,从数据源中抽取数据放入Kafka集群,结合Disruptor高吞吐和低延迟的特点,实现了数据高效的传输,使数据可以在不同数据源之间进行清洗和转换,同时在数据传输准确性方面有了极大的改进,保证了数据传输的一致性。ETL system is the basic component of building and maintaining data warehouse,and business data in heterogeneous data sources can be extracted,cleaned,and transformed to be loaded into the data warehouse by ETL tools.However,when the data volume rises to a certain extent,the traditional ETL in terms of data processing speed and data accuracy will be greatly reduced,which can’t meet the diversified requirements of data source.Aiming at the problem of how to have both efficient data processing and universal data source accessing,we propose an improved scheme for traditional ETL.Combined Kafka with Disruptor concurrent framework,the data is drawn from the data source into Kafka cluster.According to high throughput and low delay for the Disruptor,the efficient data transmission is achieved,enabling data to be cleaned and transformed between different data sources.At the same time,it greatly improves the accuracy of data transmission and ensures the consistency of data transmission.

关 键 词：大数据 ETL Kafka 数据仓库 Disruptor 

分 类 号：TP311.133.1[自动化与计算机技术—计算机软件与理论]