Llama2-70b模型的微调技术及其在材料领域的应用研究  

A Study of the Fine-Tuning Technique of the Llama2-70b Model and Its Application in the Field of Materials

作　　者：唐雷 陈子逸 梁锶翰 李凯[1] 万萌 张博尧[1] 刘淼 孟胜[3] 王彦棡[1,2] 周纯葆 王宗国[1,2] TANG Lei;CHEN Ziyi;LIANG Sihan;LI Kai;WAN Meng;ZHANG Boyao;LIU Miao;MENG Sheng;WANG Yangang;ZHOU Chunbao;WANG Zongguo(Computer Network Information Center,Chinese Academy of Sciences,Beijing 100190,China;University of Chinese Academy of Sciences,Beijing 100049,China;Institute of Physics,Chinese Academy of Sciences,Beijing 100190,China)

机构地区：[1]中国科学院计算机网络信息中心,北京100083 [2]中国科学院大学,北京100049 [3]中国科学院物理研究所,北京100190

出　　处：《数据与计算发展前沿(中英文)》2025年第1期163-174,共12页Frontiers of Data & Computing

基　　金：中国科学院网信专项“能源材料端到端设计的信息化智能平台”(CAS-WX2023SF-0101);中国科学院前沿科学重点研究计划“Ⅲ-Ⅴ族半导体材料的‘基因图谱’研究”(ZDBS-LY-7025);中国科学院青年创新促进会(2021167)。

摘　　要：【目的】为降低大语言模型的使用门槛,促进大语言模型在学科领域的应用。本文系统介绍了Llama2-70b模型的微调过程及其在材料领域应用的流程。【方法】本研究利用DeepSpeed框架和无机材料合成路径的指令式数据集,采用LoRA微调技术对开源大模型Llama2-70b进行微调,并对模型的超参数进行了调优,从模型训练中的损失值和模型稳定性两个方面对调优效果进行了评估,最终确定了一组适合模型的超参数组合。【结果】通过对模型的训练和优化,最终获得了一个在稳定性和性能方面表现优异的材料合成大语言模型。【结论】该研究为大语言模型在学科领域的应用提供了宝贵的经验和方法,所训练的材料大语言模型为材料合成设计提供了有意义的参考和支持。[Objective]To lower the barriers of using large language models and promote their applications in different fields,this paper systematically introduces the fine-tuning process of the Llama2-70b model and its application procedure in the field of materials science.[Methods]This study utilized the DeepSpeed framework and an instruction data set of inorganic material synthesis pathways,and employed the LoRA fine-tuning technique to fine-tune the open-source Llama2-70b model.The model’s hyperparameters were optimized,and the tuning effects were evaluated based on the loss value during model training and the model’s stability.A suitable combination of hyperparameters was finally determined.[Results]Through the training and optimization of the model,a large language model for material synthesis that performs excellently in terms of stability and performance was obtained.[Conclusions]This research provides valuable experience and methods for the application of large language models in academic fields.The trained material language model offers meaningful reference and support for material synthesis design.

关 键 词：Llama2-70b模型 LoRA 大模型微调 材料合成 

分 类 号：TB30[一般工业技术—材料科学与工程] TP18[自动化与计算机技术—控制理论与控制工程]