Spark环境下的分布式OLAP查询  

Distributed OLAP Query in Spark

作　　者：崔红波[1] 游进国[1] 简兴明 张正凡 丁家满[1] 

机构地区：[1]昆明理工大学信息工程与自动化学院,昆明650500

出　　处：《小型微型计算机系统》2017年第11期2517-2522,共6页Journal of Chinese Computer Systems

基　　金：国家自然科学基金项目(61462050;61562054;51467007)资助;云南省自然科(KKSY201303095)资助

摘　　要：数据立方体是数据仓库的核心数据模型,其元组可以划分为封闭元组和非封闭元组,通过去掉非封闭单元进行压缩并分层形成了分层封闭立方体.以Hadoop为主的云计算环境通常以离线批量计算为主,查询分析有一定的延迟,不能达到OLAP的在线性和交互性.Spark是一个基于内存的快速通用的大数据并行计算框架,对此本文基于分层封闭立方体,利用Spark,设计和实现了两种有效的分布式OLAP查询算法:SLCCQuery及其优化算法SLCC_Layered Query.不同参数的数据集上的实验验证了本文提出的Spark环境下的分布式OLAP查询算法的有效性及其优化算法的相对高效性.Data cube is the core data model of data warehouse,Tuples of a data cube can be divided into closed-tuples and non-closed tuples. By removing all non-closed tuples and layering,a data cube can be translated to a layered closed cube. Cloud computing environment based on Hadoop is mainly dominated by offline batch computing. It can＇t satisfy online and interactive analysis. The Spark is a quick general big data parallel computing framework based on memory. In this paper,we design and implement two efficient distributed OLAP query algorithms based on Spark and layered closed cubes. They are SLCCQuery and its optimization algorithm SLCC_Layered Query. Experiments on data sets with different parameters demonstrate the efficiency of the distributed OLAP query algorithms in Spark proposed in this paper and high efficiency of its optimization algorithm.

关 键 词：分层封闭立方体 封闭单元 SPARK OLAP查询 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]