基于Spark的时态查询扩展与时态索引优化研究  

Research on Temporal Query Expansion and Temporal Index Optimization Based on Spark

作　　者：周亮[1] 李格非[1] 邰伟鹏[2] 郑啸[2] 

机构地区：[1]上海交通大学计算机科学与工程系,上海200240 [2]安徽工业大学计算机科学与技术学院,安徽马鞍山243032

出　　处：《计算机工程》2017年第7期22-28,37,共8页Computer Engineering

基　　金：安徽省高校自然科学研究重点项目"基于关键字的大规模地理数据查询方法研究"(KJ2015A310)

摘　　要：时空数据库和基于集群计算的时间分析工具大多基于外存,将其应用在大数据处理场景下系统性能将迅速降低。为此,基于Spark构建一个易用且高可扩展的时态大数据查询分析系统。通过扩展Spark SQL解析器,使其能够支持类SQL形式的时态操作,运用SIMBA开源项目的方法,引入全局过滤和局部时态索引2种优化策略,使得系统能以高吞吐量及低延迟执行时态查询操作。基于时态查询效率的评估实验结果表明,在不同影响参数下,该系统的时态查询性能优于原生的Spark SQL查询处理方案。There exists some temporal databases and temporal analysis tools based on cluster-based computing systems.However,most of them are disk-oriented and performance degenerate rapidly when processing big data. This paper proposes a system which is based on Spark,and provides accessible and scalable temporal query scheme with large temporal data for users. Specifically,it extends Spark SQL parser to support SQL-like temporal operations. Besides,it uses the index manager based on Spark SQL which is proposed by SIMBA,and embeds optimization strategies in two aspects：global filtering and local temporal index. Depending on these optimization rules,the system achieves high throughput and lowlatency in temporal operations. Evaluation experiment results on temporal query efficiency and effectiveness showthis system has improved temporal query performance over original Spark SQL in different factors.

关 键 词：时态大数据 Spark系统 SPARK SQL组件 时态查询 时态索引 高吞吐量 低延迟 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]