一种面向异构大数据计算框架的监控及调度服务  

Monitoring and Dispatching Service for Heterogeneous Big Data Computing Frameworks

作　　者：胡雅鹏 丁维龙[1,2] 王桂玲 HU Ya-peng;DING Wei-long;WANG Gui-ling(Data Engineering Institute,North China University of Technolog;Beijing Key Laboratory on Integration and Analysis of Large-scale Stream Dat)

机构地区：[1]北方工业大学数据工程研究院,北京100144 [2]大规模流数据集成与分析技术北京市重点实验室,北京100144

出　　处：《计算机科学》2018年第6期67-71,95,共6页Computer Science

基　　金：国家自然科学基金(61702014);北京市自然科学基金(4172018)资助

摘　　要：各种类型的大数据计算框架存在各自专用的管理方法。传统的监控和调度服务在异构环境下的操作由于无法获取集群整体的运行状态而受到限制,且无法综合多粒度的运行时资源状态来调度不同的计算作业。这不仅浪费了集群的可用资源,而且增加了计算作业的等待时间。针对上述两个问题,提出了一种面向异构大数据计算框架的一体化监控及动态调度管理服务。该服务可以自动适应并监控多种类型的大数据计算框架及计算作业,并对多类型作业提供一体化调度。针对Hadoop和Storm两种计算框架,实现了原型系统并进行了实验。实验结果表明,所提服务在异构环境下的大数据计算框架中能降低人工操作的复杂度,并且能提高作业的调度效率。Various types of large data computing frameworks have their own management methods.The operation of traditional monitoring and scheduling service in heterogeneous environment is limited by the global status of cluster.It not only wastes resource of cluster,but also suffers long executive latencies of job.To solve these problems above,this paper presented an integrated monitoring and dynamic scheduling management service for heterogeneous big data computing framework.The service can monitor multiple types of computing framework automatically and provide integrated dispatching for diverse computing jobs.The work was implemented on Hadoop and Storm.The experimental results show that the service can reduce the complexity of manual operation in heterogeneous environment and improve job scheduling efficiency.

关 键 词：作业调度 集群监控 管理服务 作业提交 

分 类 号：TP315[自动化与计算机技术—计算机软件与理论]