面向深度学习训练的内存交换机制综述  

Survey on Memory Swapping Mechanism for Deep Learning Training

作　　者：高赫然 吴恒[2,5,6] 许源佳 李修和 王焘 张文博[2,3,5,6] GAO He-Ran;WU Heng;XU Yuan-Jia;LI Xiu-He;WANG Tao;ZHANG Wen-Bo(University of Chinese Academy of Sciences,Beijing 100049,China;Technology Center of Software Engineering,Institute of Software,Chinese Academy of Sciences,Beijing 100190,China;State Key Laboratory of Computer Science(Institute of Software,Chinese Academy of Sciences),Beijing 100190,China;Institute of Electronic Countermeasures,National University of Defense Technology,Hefei 230037,China;University of Chinese Academy of Sciences,Nanjing,Nanjing 211135,China;Nanjing Institute of Software Technology,Nanjing 210000,China)

机构地区：[1]中国科学院大学,北京100049 [2]中国科学院软件研究所软件工程技术研究开发中心,北京100190 [3]计算机科学国家重点实验室(中国科学院软件研究所),北京100190 [4]国防科技大学电子对抗学院,安徽合肥230037 [5]中国科学院大学南京学院,江苏南京211135 [6]中科南京软件技术研究院,江苏南京210000

出　　处：《软件学报》2023年第12期5862-5886,共25页Journal of Software

基　　金：国家重点研发计划(2018YFB1402803);国家自然科学基金(61872344,61972386);山东省重大研发计划(2021CXGC010101)。

摘　　要：随着深度学习技术的快速发展和深入应用,深度学习训练规模持续增大,内存不足已成为影响深度学习可用性的主要瓶颈之一.内存交换机制是应对深度学习训练内存问题的关键技术,该机制利用深度学习训练内存需求的“时变”特征,在专用计算加速设备内存与外部存储之间按需移动数据,通过瞬时内存需求替代累积内存需求,保障深度学习训练任务的运行.对面向深度学习训练的内存交换机制进行综述,以深度学习训练内存需求的时变特征为研究视角,分别针对基于算子运行特征的内存换出机制、基于数据依赖关系的内存换入机制以及效能驱动的联合换出与换入决策等重要研究工作进行了总结分析,并针对该技术领域的发展方向进行了展望.With the rapid growth and further application of deep learning(DL),the scale of DL training continues to expand,and memory insufficiency has become one of the major bottlenecks threatening DL availability.Memory swapping mechanism is the key mechanism to alleviate the memory problem of DL training.This mechanism leverages the“time-varying”memory requirement of DL training and moves the data between specific computing accelerating device memory and external storage according to demands.The operation of DL training tasks can be ensured by replacing an accumulated memory requirement with an instant one.This study surveys the memory swapping mechanism for DL training from the aspect of time-varying memory requirements.Key studies of an operator feature-based memory swapping-out mechanism,a data dependency based swapping-in mechanism,and efficiency-driven joint swapping-in and swapping-out decisions are summarized.Finally,the development prospect of this technology is pointed out.

关 键 词：深度学习训练 内存交换 内存需求特征 

分 类 号：TP303[自动化与计算机技术—计算机系统结构]