基于稳定匹配的实时ETL弹性调度机制  

AN ELASTIC SCHEDULING MECHANISM FOR REAL TIME ETL BASED ON STABLE MATCHING

作　　者：刘旋律 顾进广[1,2,3,4] Liu Xuanlü;Gu Jinguang(School of Computer and Technology,Wuhan University of Science and Technology,Wuhan 430065,Hubei,China;Key Laboratory of Intelligent Information Processing and Real-time Industrial System in Hubei Province(Wuhan University of Science and Technology),Wuhan 430065,Hubei,China;Institute of Big Data Science and Engineering,Wuhan University of Science and Technology,Wuhan 430065,Hubei,China;Key Laboratory of Rich-media Knowledge Organization and Service of Digital Publishing Content,National Press and Publication Administration,Beijing 100038,China)

机构地区：[1]武汉科技大学计算机科学与技术学院,湖北武汉430065 [2]智能信息处理与实时工业系统湖北省重点实验室(武汉科技大学),湖北武汉430065 [3]武汉科技大学大数据科学与工程研究院,湖北武汉430065 [4]国家新闻出版署富媒体数字出版内容组织与知识服务重点实验室,北京100038

出　　处：《计算机应用与软件》2022年第2期266-273,共8页Computer Applications and Software

基　　金：国家自然科学基金项目(61673304)。

摘　　要：在数据生产速度波动较大的场景,为了实时ETL资源利用更合理,提出基于稳定匹配的ETL弹性调度机制。预测数据源的数据生产速度,并计算满足预测值的消费数据速度;使用贪婪负载均衡算法,调整ETL服务个数使节点负载均衡;确定ETL操作匹配关系,使消费数据速度最大且代价最小。该调度机制将匹配问题转化为最小费用最大流问题,并提出基于Dicnic算法的改进算法。实验结果表明,该调度机制在资源使用方面具有优势。In the case of large fluctuation of data production speed,in order to make real time ETL process resource utilization more reasonable,this paper proposes an ETL elastic scheduling mechanism based on stable matching.The data production speed of ETL data source was predicted,and the consumption data speed which needs to meet the predicted speed was calculated;greedy load balancing algorithm was adopted to adjust the number of ETL services to balance the load of nodes;we matched ETL operation relationship to make the consumption data speed the fastest and the cost the least.The ETL operation matching problem was transformed into the minimum cost maximum flow problem,and the improved algorithm based on Dicnic algorithm was proposed.The experimental results show that the scheduling mechanism has advantages in resource utilization.

关 键 词：实时ETL 弹性调度 稳定匹配 最小费用最大流 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]