MSOLA:基于多维分层采样的大数据在线聚集技术  

MSOLA:big data online aggregation based on multi-dimension stratified sampling

作　　者：史英杰[1] 杜方 尤亚东 

机构地区：[1]北京服装学院信息工程学院,北京100029 [2]宁夏大学信息工程学院,银川750021

出　　处：《计算机应用研究》2018年第2期375-380,共6页Application Research of Computers

基　　金：国家自然科学基金资助项目(61502279;61363018);山东省自然科学基金资助项目(ZR2015FM013);北京市教委科技计划项目(KM201710012008)

摘　　要：在线聚集通过统计计算估计查询结果,能够在查询完成前给用户反馈,在大数据分析领域具有重要意义。现有研究工作采用统一随机采样策略,当查询出现小分组或低选择率时,导致估计结果不准确及收敛速度缓慢。针对这一问题,提出了结合负载特征和数据分布进行多维分层采样的有偏采样策略,并结合大数据处理平台Storm设计了结果估计和置信区间计算方法。实验证明所提出的方案有效提高了在线聚集估计结果的准确度,并且具有良好的扩展性。Online aggregation estimates the query results through statistical computing, and it can provides feedback to users before the query finishes, which is of paramount importance in the field of big data analysis. The existing studies always adopt uniform sampling,which results in estimate inaccuracy and slow convergence. This paper proposed the multi-dimension strati- fied sampling technique based on workload characteristics and data distribution, and designed the result estimate and confi- dence interval compute algorithms based on Storm. The experiments demonstrate that the proposed online aggregation technique improves the accuracy of estimated results in online aggregation with efficient scalability.

关 键 词：在线聚集 大数据 多维分层采样 负载分析 

分 类 号：TP392[自动化与计算机技术—计算机应用技术]