Spark环境下SQL优化的方法  

On the Method of SQL Optimization in Spark Environment

作　　者：杨彦彬 干祯辉 YANG Yanbin;GAN Zhenhui(Zhongtong Service Software Technology Co.,Ltd.,Shanghai 200050,China;Hangzhou Dongfang Communication Software Technology Co.,Ltd.,Hangzhou 310013,China)

机构地区：[1]中通服软件科技有限公司,上海200000 [2]杭州东方通信软件技术有限公司,浙江杭州310013

出　　处：《数字通信世界》2021年第7期13-14,共2页Digital Communication World

摘　　要：将传统SQL稍加改动后迁移到大数据平台是实现业务平稳过渡最佳办法,但是传统SQL所处理的数据量、运行环境与大数据平台完全不同,因而迁移后如何优化就很重要了。以当前主流大数据处理组件Spark SQL为例,论述了迁移产生问题背景、解决方案,以及优化前后性能比对结果。It is the best way to realize the smooth transition of business to move the traditional SQL to the big data platform after a little change,but the amount of data processed by the traditional SQL and the running environment are completely different from the big data platform.So how to optimize after migration is very important.In this paper,taking the current mainstream big data processing component spark SQL as an example,the background of migration problems,solutions,and performance comparison results before and after optimization are discussed.

关 键 词：SPARK 大数据 Spark SQL 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]