基于RDD非序列化本地存储的Spark存储性能优化  

Optimization of Spark RDD Based on Non-serialization Native Storage

作　　者：赵俊先 喻剑[2] ZHAO Jun-xian;YU Jian(Department of Computer Science and Technology,Tongji University,Shanghai 201804,China;Key Laboratory of Embedded System and Service Computing,Ministry of Education (Tongji University),Shanghai 201804,China)

机构地区：[1]同济大学计算机科学与技术系,上海201804 [2]嵌入式系统与服务计算教育部重点实验室(同济大学),上海201804

出　　处：《计算机科学》2019年第5期143-149,共7页Computer Science

摘　　要：Spark框架被越来越多的企业用作大数据的计算框架,但随着现有服务器的可用内存资源增加,Spark并不能与新环境相匹配。Spark运行在Java虚拟机上,随着堆空间内存被大量使用,Java虚拟机通过回收内存来为新对象提供空间(垃圾回收机制,GC)的时间开销占Spark作业总耗时的比例显著增加,但Spark作业的效率并未随着可用内存的增加而保持一定比例的提升。在使用非堆(本地)内存存储模式后,GC开销问题得以缓解,但缓存数据的序列化开销成为新的矛盾点。文中利用本地存储方式解决GC问题,同时通过减少序列化开销以加快作业速度,提出并修改了Spark的存储结构,改进了RDD的淘汰机制和缓存方式,将去序列化的数据引入到本地内存中,在保持较低的垃圾回收开销的同时,降低了序列化的开销。实验结果表明,与原Spark的堆上存储方式相比,非序列化的本地存储方法在单结点、大内存的服务器上的GC时间缩短到5%～30%,同时,序列化开销显著降低,吞吐量得到提升,作业耗时缩短8%以上。Spark framework is taken as the computing framework of big data by more and more enterprises.However,with the increasing of available memory resource of current severs,Spark can’t match with new environment well.Spark runs on Java Virtual Machine(JVM).As heap space memory is used heavily,the ratio of time cost produced by Java virtual machine to provide space for new objects by reclaiming memory(GC)to total time cost of Spark jobs increases significantly,but the efficiency of Spark jobs doesn’t improve with a certain ratio when the available memory increases.After using OffHeap(native)memory storage mode,the cost of serialization/deserialization becomes the new conflict point instead of GC.This paper used the way of native storage to deal with GC problem,and speeded up the job by reducing the overhead of GC.This paper also proposed and modified the storage structure of Spark,and improved the elimination mechanism and the caching way of RDD.The data without serialization are moved into native memory,realizing low garbage collection overhead and avoiding the time spending on serialization.Experimental results demonstrate that the GC cost of modification method on server with single node and large memory is 5%to 30%compared with the storage on heap of Spark.Meanwhile,the overhead of serialization decreases,the throughput increases and the running time of job can be reduced by more than 8%.

关 键 词：SPARK 垃圾回收 序列化 存储系统 本地内存 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]