QJoin:质量驱动的乱序数据流连接处理技术  

QJoin:Quality-driven Join Processing Technique over Out-of-Order Data Streams

作　　者：魏星贝 李陶深[1,2] 许嘉 吕品[1,2] 杨宁 WEI Xingbei;LI Taoshen;XU Jia;LV Pin;YANG Ning(School of Computer,Electronics and Information,Guangxi University,Nanning,Guangxi,530004,China;Guangxi Colleges and Universities Key Laboratory of Parallel and Distributed Computing Technology,Nanning,Guangxi,530004,China)

机构地区：[1]广西大学计算机与电子信息学院,广西南宁530004 [2]广西高校并行与分布式计算技术重点实验室,广西南宁530004

出　　处：《广西科学》2020年第3期266-275,共10页Guangxi Sciences

基　　金：国家自然科学基金项目(61402494);广西自然科学基金面上项目(2019JJA170045)资助。

摘　　要：数据流乱序现象会导致数据流处理结果的丢失,给数据流的分析处理带来了巨大困难。本研究探讨了质量驱动下的乱序数据流连接处理问题,提出一种质量驱动的乱序数据流连接处理技术(QJoin)。QJoin采用缓冲存储技术和对称连接策略,实现并确保对流元组进行即时分析处理,从而降低了流元组处理的平均等待时间。同时,基于质量驱动的理念,根据临近阶段连接处理过程中收集统计的数据,自适应地调整和优化内存缓存区的大小,从而在满足用户结果质量要求的前提下,降低系统内部历史数据的内存缓存量,尽可能保证迟到元组的连接处理完整性。真实数据集上的实验结果表明,与传统的数据流乱序处理技术MP-K-slack相比,QJoin在满足用户结果质量要求的前提下,确保能够即时地分析处理数据流的流元组,显著降低系统的内存开销。The out-of-order phenomenon of data streams will cause the missing of data stream processing results,which brings great challenges to the analysis and processing of the data stream.This study explores the problem of quality-driven join processing over out-of-order data streams and proposes a technique named QJoin.QJoin adopts cache storage technique and symmetric join processing strategy to ensure the real-time analysis and processing of each arriving stream tuple,thereby reducing the average waiting time of stream tuple processing.Meanwhile,based on the concept of quality-driven,QJoin collects statistic data during the join processing in the approaching stage and adaptively adjusts the size of the memory cache based on the statistic data,which reduces the amount of memory cache of the system’s internal history data and ensures the connection processing integrity of the late tuple as much as possible.The experimental results on the real data set show that compared with the traditional out-of-order data stream processing technique K-slack,on the premise of meeting the user’s quality requirements for results,QJoin ensures that stream tuples can analyze and process data streams in real time,significantly reducing the memory overhead of the system.

关 键 词：质量驱动 连接处理 乱序数据流 存储开销 流元组 缓存 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]