Spark中一种高效RDD自主缓存替换策略研究  

Research on efficient RDD self-cache replacement strategy in Spark

作　　者：魏赟[1] 丁宇琛 Wei Yun;Ding Yuchen(School of Optical-Electrical&Computer Engineering,University of Shanghai for Science&Technology,Shanghai 200093,China)

机构地区：[1]上海理工大学光电信息与计算机工程学院,上海200093

出　　处：《计算机应用研究》2020年第10期3043-3047,共5页Application Research of Computers

基　　金：国家自然科学基金资助项目(61170277,61472256);上海市科委科研计划资助项目(16111107502)。

摘　　要：由于并行计算框架Spark缓存替换算法的粗糙性,LRU(least recently used)算法并未考虑RDD的重复使用导致易把高重用数据块替换出内存且作业执行效率较低等问题。通过优化权重模型和改进替换策略,提出了一种高效RDD自主缓存替换策略(efficient RDD automatic cache,ERAC),包括高重用自主缓存算法和缓存替换分级算法,可实现高效RDD的自主缓存和缓存目标的分级替换。最后利用SNAP(Stanford Network Analysis Project)提供的标准数据集将ERAC和LRU、RA(register allocation)等算法进行了对比实验,结果显示ERAC算法能够有效提高Spark的内存利用率和任务执行效率。Due to the roughness of cache replacement mechanism in Spark parallel computing framework,LRU algorithm does not consider the RDD data block reuse problem that easily results in high-reuse block to swap out of the cache and execution inefficiency.By optimizing the weight model and improving the replacement strategy,this paper put forward ERAC cache stra-tegy,which could achieve efficient RDD autonomous cache and hierarchical replacement,included high-reuse self-caching algorithm and cache replacement hierarchical algorithm.Applied ERAC,RA and LRU in standard dataset of SNAP,experimental results show that ERAC can effective improve the memory utilization and task execution efficiency.

关 键 词：SPARK 弹性分布式数据集 缓存替换 内存资源利用 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]