面向ETL数据处理过程的优化研究  

Research on the optimization of ETL data processing

作　　者：张晓冉[1] 舒昝 

机构地区：[1]东北石油大学计算机与信息技术学院,黑龙江大庆163318

出　　处：《自动化与仪器仪表》2016年第7期109-110,共2页Automation & Instrumentation

摘　　要：针对ETL数据处理的数据转换阶段进行研究与优化,为了进一步提高数据处理的效率,对数据处理的规则进行优化合并,减少了相关进程的数目以及存储空间的消耗。另外引入了Map TrimReduce编程模型对ETL的大量数据进行分布式处理,规则上的优化处理也进一步缩短了MTR三个部分的执行次数,达到提高整体ETL数据处理效率的目的。In this paper,we study and optimize the data transformation phase of ETL data processing,in order to further improve the efficiency of data processing,optimize the rules of data processing,reduce the number of related processes and storage space. In addition,the Map TrimReduce programming model is introduced to deal with a large amount of data of ETL,and the optimization process of the rules further reduces the execution times of the MTR three parts,and achieves the purpose of improving the efficiency of the whole ETL data processing.

关 键 词：ETL 数据处理规则 Map-Trim-Reduce 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]