分布式数据流计算系统的数据缓存技术综述  

Survey on data caching technology of distributed dataflow system

作　　者：袁旭初 付国 毕继泽 张岩峰 聂铁铮 谷峪 鲍玉斌 于戈 YUAN Xuchu;FU Guo;BI Jize;ZHANG Yanfeng;NIE Tiezheng;GU Yu;BAO Yubin;YU Ge(College of Computer Science and Engineeing,Northeastern University,Shenyang 110169,China)

机构地区：[1]东北大学计算机科学与工程学院,辽宁沈阳110169

出　　处：《大数据》2020年第3期101-116,共16页Big Data Research

基　　金：国家重点研发计划基金资助项目(No.2018YFB1003404);国家自然科学基金资助项目(No.61672141);中央高校基本科研业务费专项资金资助项目(No.N181605017,No.N181604016)。

摘　　要：数据流编程模型以其高度并行计算、支持流水线处理、支持函数式编程等优点被许多主流的计算系统采用。在分布式数据流系统和异构数据流系统中,算子之间数据生产和数据消化的速度不一致可能会导致数据堆积或者算子闲置等问题。为支持高效的数据流系统,需要设计缓存系统,以保证数据流的高效缓存和移动。选取了几个典型的分布式数据流系统与分布式消息队列系统进行系统分析,并总结了目前消息队列系统对数据流缓存系统的支持程度。最后对数据缓存技术进行了阐述,并分析了未来的数据流缓存系统的需求和研究方向。Dataflow model is adopted by several dataflow systems for its advantages of high parallel computing,pipeline processing and functional programming.In distributed dataflow systems and heterogeneous dataflow systems,due to the speed mismatch between the data production of data source operators and the data consumption of data sink operators,data could be delayed and operators could be idle.In order to support an efficient dataflow system,a dataflow cache system was desired to ensure efficient caching and movement of dataflow.Several distributed dataflow systems and distributed message queuing systems were analyzed,and the support degree of current message queuing system to data flow caching system was summarized.Finally,the cache technique was introduced,and the demands and research directions of future dataflow caching systems were analyzed.

关 键 词：数据流 缓存 分布式系统 消息队列 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]