面向Flink的多表连接计算性能优化算法  

COMPUTATIONAL PERFORMANCE OPTIMIZATION ALGORITHM FOR MULTI-TABLE JOINS BASED ON FLINK

作　　者：李旺 双锴 Li Wang;Shuang Kai(Institute of Network Technology,Beijing University of Posts and Telecommunications,Beijing 100876,China)

机构地区：[1]北京邮电大学网络技术研究院,北京100876

出　　处：《计算机应用与软件》2021年第6期31-38,共8页Computer Applications and Software

基　　金：国家自然科学基金项目(U1534201)。

摘　　要：分布式计算引擎Flink已经被广泛应用到大规模数据分析处理领域,多表连接是Flink常见作业之一,因此提升Flink多表连接的性能可以加快数据处理和分析的速度。然而,直接将现有的多表连接优化算法应用到Flink上会带来两个问题:现有算法不能充分发挥Flink基于线程的轻量级计算模型的性能优势;连接算法需要shuffle的数据量过大。提出优化连接并行度的Multi Bushy Tree算法,尽可能提高多表连接计算的并行度;提出优化星型连接的Semi Join算法,可以大大减少需要shuffle的数据量。在TPC-H数据集上的实验结果表明,提出的算法可以有效提高多表连接计算的并行度,并缩短作业运行时间,减小星型连接中的网络IO代价。Flink has been widely used in large-scale data analysis processing.Multi-table join is one of the most common operations for data analysis,so improving the performance of Flink multi-table join can speed up data processing and analysis.However,applying the existing multi-table join algorithm directly to Flink brings two problems:the existing algorithm can not fully exploit the performance advantages of Flink s thread-based lightweight distributed computing;the connection algorithm needs too much shuffle data.The multi bushy tree algorithm for optimizing the parallelism of connections was proposed to improve the parallelism of multi-table join calculation as much as possible.The semi join algorithm for optimizing star connections was proposed,which could greatly reduce the amount of data that need to be shuffled.The experimental results based on the dataset of TPC-H show that the proposed algorithm can effectively improve the parallelism of multi-table join calculation,reduce job running time and reduce network IO cost in star connections.

关 键 词：Flink 多表连接 连接并行度 数据shuffle 星型连接 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]