针对具有稀疏性的流式大数据卸载方法  

Load Shedding Methods for Big Data Stream with Sparsity

作　　者：王顺 李振星 连增申 曾国荪[1,3] 丁春玲[4] WANG Shun;LI Zhenxing;LIAN Zengshen;ZENG Guosun;DING Chunling(College of Electronics and Information Engineering,Tongji University,Shanghai 200092,China;Beijing AgileCentury Information Technology Co.,Ltd.,100085,China;Tongji Branch National Engineering&Technology Center of High Performance Computer,Tongji University,Shanghai 201804,China;College of Chemical Science and Engineering,Tongji University,Shanghai 200092,China)

机构地区：[1]同济大学电子与信息工程学院,上海200092 [2]北京捷软世纪信息技术有限公司,北京100085 [3]同济大学国家高性能计算机工程技术中心同济分中心,上海201804 [4]同济大学化学科学与工程学院,上海200092

出　　处：《同济大学学报（自然科学版）》2020年第2期276-286,共11页Journal of Tongji University:Natural Science

基　　金：国家社科基金（17BQT086）;国家海底科学观测系统子项目（2970000001/001/016）;CCF信息系统开放课题（CCFIS2018-01-03）

摘　　要：在保证实时性的前提下提高流式大数据卸载的准确性是一个重要问题。针对具有稀疏性的流式大数据开展2种典型场景下的卸载研究。对普通均匀业务的流式大数据进行空间建模,使用弹性距离对数据间的距离进行放缩,提出基于离心率的卸载方法。对异常检测业务流式大数据应用场景进行特征分析,使用预处理自动机对数据的动态处理过程进行描述,在综合考虑数据和处理行为相似度基础上,提出基于等价类划分的卸载方法。重复试验表明,所提出的卸载方法与传统卸载方法相比能明显提高卸载的有效性。How to improve the accuracy of load shedding under the premise of ensuring real-time performance is an important problem.Sparsity is a widespread feature of the big data stream.Therefore,we propose two loadshedding methods of the big data stream with sparsity in two scenarios.In the normal business scenario,we model the big data stream with the high dimensional space.Then we propose a load shedding method based on centrifugation,which uses the elastic distance to measure the distance of data.In the anomaly-monitoring scenario,we analyze the feature of the big data stream and propose a load shedding method based on equivalence class,which uses the combined similarity to divide the data set into equivalence classes.The combined similarity was composed of processing behavior similarity and data similarity to measure the difference between data.Repeated test results show that the two load shedding methods in this paper can significantly improve the accuracy compared with the conventional load shedding methods.

关 键 词：流式大数据 数据卸载 稀疏性 弹性距离 行为相似 

分 类 号：TP338[自动化与计算机技术—计算机系统结构]