流数据处理中负载突发感知的弹性资源分配  

Burstiness-Aware Elastic Resource Allocation in Stream Data Processing

作　　者：李丽娜[1] 魏晓辉[1,2] 李翔[3] 王兴旺 LI Li-Na;WEI Xiao-Hui;LI Xiang;WANG Xing-Wang(College of Computer Science&Technology,Jilin University,Changchun 130012;Symbol Computation and Knowledge Engineer of Ministry of Education,Jilin University,Changchun 130012;High Performance Computing Center,Jilin University,Changchun 130012)

机构地区：[1]吉林大学计算机科学与技术学院,长春130012 [2]吉林大学符号计算与知识工程教育部重点实验室,长春130012 [3]吉林大学高性能计算中心,长春130012

出　　处：《计算机学报》2018年第10期2193-2208,共16页Chinese Journal of Computers

基　　金：国家自然科学基金(61170004);吉林省科技厅项目(20140204013GX)资助~~

摘　　要：在分布式并行数据流处理中,面向实时变化且具有突发性的流数据负载,固定的资源分配将造成资源浪费或服务质量降低,因此,可伸缩的弹性资源分配是一个亟待解决的关键问题.然而,由于资源分配延迟和负载预测模型存在误差,已有的弹性资源分配策略无法准确地提供与突发负载相匹配的资源,且存在不必要的资源调整反复,增加了系统开销.该文主要解决弹性资源分配的调整延迟和调整颠簸问题.针对上述问题,主要的挑战在于突发负载的准确预测和节点间的协作.为此,该文提出了一个上、下游节点协同的弹性资源分配策略,最优化数据质量和资源使用率,兼顾考虑调整代价.在该策略中,基于数据负载关联模型和双向的控制机制,下游节点能够实时感知和预测上游节点产生的突发负载和负载的变化趋势,预先调整资源并避免调整颠簸;同时,上游节点能够基于反馈机制,动态调节数据处理速率以抑制下游节点的负载波动,降低其资源调整的可能性.实验结果表明,当负载变化较大时,该策略平均减少数据丢失达85%,并显著降低了系统资源调整开销,同时,提高了资源使用率.In distributed parallel data stream processing,facing the real-time-changing and bursting stream data,fixed resource allocation will cause waste of resources or reduce the quality of service.To achieve desirable performances without resource wastes,the scalable and elastic resource allocation is a critical problem to be solved,which allows applications to change dynamically their resource configuration in response to data load fluctuations at run-time.However,most elastic resource allocation policies only adjust their resources when the performance of nodes does not match the data load.The adjustment delay caused by the immediate resource allocation,and the unpredictable data load reduce the performance of the elastic allocation policy.In some work,the data load prediction is introduced into the elastic resource allocation,but in the data prediction model,the future data is predicted based on the historical data,which is not applicable to the bursty stream data.Moreover,the unnecessary resource adjustment bump increases the system overhead.This paper focuses on the adjustment delay and the adjustment jump in the elastic resource allocation.For the above problems,the main challenge lies in the prediction of burst load and the cooperation among nodes.Therefore,this paper firstly constructs a data load correlation model to predict the data load of nodes accurately.This model considers the correlation between nodes of the stream application,that is the node will experience the sudden load change,once its upstream nodes have carried out resource adjustments,then uses the queuing theory to predict the data load of the node in the next window,according to the states of its neighborhood upstream nodes in the current window,which includes the data load,the buffer occupy,the processing ability and the consumption ratio.Furthermore,a bi-directional control mechanism is designed to support the cooperative resource allocation between the upstream and downstream nodes,in which the feed forward control transmits the information

关 键 词：流数据 流数据处理 突发感知 资源分配 弹性调整 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]