Spark Streaming动态资源分配策略  

Dynamic resource allocation strategy in Spark Streaming

作　　者：刘备[1] 谭新明[1] 曹文彬[1] 

机构地区：[1]武汉理工大学计算机科学与技术学院,武汉430063

出　　处：《计算机应用》2017年第6期1574-1579,共6页journal of Computer Applications

基　　金：湖北省自然科学基金重点项目(2014CFA050)~~

摘　　要：针对Spark Streaming作为混合大数据计算平台流处理组件时资源调整周期长和不能满足多应用多用户个性化需求的问题,提出了一种多应用下动态资源分配策略(DRAM)。该策略增加了应用全局变量来控制动态资源分配过程。首先,获取历史执行数据反馈和应用全局变量;然后,进行资源增减计算;最后,进行资源增减执行。实验结果表明,所提策略能够有效调整应用资源配额,且在稳定数据流和不稳定数据流两种情况下,其处理延时相比原Spark平台的Streaming策略和Core策略都有所降低;同时该策略也能够提高集群资源利用率。The existing resource allocation strategy has long resource adjustment cycle and cannot sufficiently meet the individual needs of different applications and users when Spark Streaming is selected as stream processing component in hybrid large-scale computing platform. In order to solve the problems, a Dynamic Resource Allocation strategy for Multi-application （DRAM） was proposed. The global variables were added to control the dynamic resource allocation process in DRAM. Firstly, the historical data feedback and the global variables were obtained. Then, whether increasing or decreasing the number of resources in each application was determined. Finally, the increase or decrease of resources was implemented. The experimental results show that, the proposed strategy can effectively adjust the resource quota, and reduce the processing delay compared with the original Spark platform strategies such as Streaming and Core under both cases of the stable data stream and the unstable data stream. The proposed strategy can also improve the utilization rate of the cluster resources.

关 键 词：SPARK 实时数据流 多应用 动态资源分配 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]