面向国产超算系统的大模型训练优化方法  

Optimization Method for Large Language Models on Domestic Supercomputer System

作　　者：屈志勇[1] 王晓光 周纯葆[2] 史源香[1] 乔嘉伟 QU Zhiyong;WANG Xiaoguang;ZHOU Chunbao;SHI Yuanxiang;QIAO Jiawei(Shanxi Meteorological Information Center,Taiyuan,Shanxi 030006,China;Computer Network Information Center,Chinese Academy of Science,Beijing 100083,China)

机构地区：[1]山西省气象信息中心,山西太原030006 [2]中国科学院计算机网络信息中心,北京100083

出　　处：《数据与计算发展前沿(中英文)》2025年第2期120-129,共10页Frontiers of Data & Computing

基　　金：山西省气象局揭榜挂帅项目(SXKJBGS202409);山西省档案科技项目共同资助(2024-SX-002);国家气象信息中心重点创新团队(NMIC-2024-ZD08)。

摘　　要：【目的】为了降低国产超算系统上的大模型训练开销,研发一套大模型训练优化方法。【方法】本文基于MPI与UCC形成一套通信后端,将进程组快速构建与低延迟集合通信相结合,在此基础上引入基于压缩的集合通信优化方法。【结果】通过在国产超算系统上多种配置下的大模型训练实验,本文提出的优化方法可以有效减少训练开销。【结论】实验结果证明了本文提出的大模型训练优化方法在减少训练开销方面的有效性。[Objective]In order to reduce the training cost of large language models on domestic supercomputer systems,we propose an optimization method.[Methods]In this article,we build a communication backend based on MPI and UCC,combining the rapid construction of process groups with low-latency collective communication,and introduces a compression-based collective communication optimization method.[Results]Through training experiments for large language models with various configurations on domestic supercomputer systems,our proposed optimization method effectively reduces training costs.[Conclusions]Experimental results demonstrate the effectiveness of the proposed large model training optimization method in reducing training costs.

关 键 词：大语言模型 分布式训练 集合通信 数据压缩 

分 类 号：TP391[自动化与计算机技术—计算机应用技术] TP391.4[自动化与计算机技术—计算机科学与技术]