一种基于Spark的分布式时态索引方法  

A SPARK-BASED DISTRIBUTED TEMPORAL INDEXING METHOD

作　　者：郑晓东 王梅[1] 陈德华[1] 张碧莹 Zheng Xiaodong;Wang Mei;Chen Dehua;Zhang Biying(College of Computer Science and Technology, Donghua University, Shanghai 201620, China)

机构地区：[1]东华大学计算机科学与技术学院,上海201620

出　　处：《计算机应用与软件》2018年第5期102-108,163,共8页Computer Applications and Software

基　　金：上海市科技创新行动计划(16JC1400802)

摘　　要：基于Spark分布式计算平台提出一种分布式时态索引方法。该方法提出时态数据集的分段索引构造策略,对每一分段设计基于Spark的时态索引构建方法及基于Spark RDD的并行查询策略;根据时态查询所涉及的Spark RDD分区模式的不同,将其分为分区独立查询,跨区查询以及跨段查询,并分别针对不同模式的时态查询提出优化的辅助索引结构,提高查询效率;在基准数据上进行实验,验证了所提索引策略的实用性和高效性,同时表明所提方法对数据规模的有效自扩展性以及降低了集群硬件配置需求。Based on the Spark distributed computing platform,a distributed temporal indexing method is proposed.Firstly,this paper proposes a segment-indexing strategy of temporal data set,and designs Spark-based tense index construction method and Spark RDD-based parallel query strategy for each segment. According to the Spark RDD partition pattern involved in the temporal query,it is divided into domain independent query,cross-domain query,and cross-segment query,and proposes the optimized auxiliary index efficiency. Finally,the experiments on benchmark data verifies the practicability and efficiency of the proposed indexing strategy,indicating the method is effective on the scalability of the data scale and reduces the cluster configuration requirements.

关 键 词：时态数据 时态索引 SPARK 分布式 分段存储 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]