Optimizing Fine-Tuning in Quantized Language Models:An In-Depth Analysis of Key Variables  

作　　者：Ao Shen Zhiquan Lai Dongsheng Li Xiaoyu Hu 

机构地区：[1]National Key Laboratory of Parallel and Distributed Computing,National University of Defense Technology,Changsha,410073,China [2]Strategic Assessments and Consultation Institute,Academy of Military Science,Beijing,100091,China

出　　处：《Computers, Materials & Continua》2025年第1期307-325,共19页计算机、材料和连续体（英文）

基　　金：supported by the National Key R&D Program of China(No.2021YFB0301200);National Natural Science Foundation of China(No.62025208).

摘　　要：Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language Processing(NLP),driven by the pre-training and fine-tuning paradigm.While this approach allows models to specialize in specific tasks with reduced training costs,the substantial memory requirements during fine-tuning present a barrier to broader deployment.Parameter-Efficient Fine-Tuning(PEFT)techniques,such as Low-Rank Adaptation(LoRA),and parameter quantization methods have emerged as solutions to address these challenges by optimizing memory usage and computational efficiency.Among these,QLoRA,which combines PEFT and quantization,has demonstrated notable success in reducing memory footprints during fine-tuning,prompting the development of various QLoRA variants.Despite these advancements,the quantitative impact of key variables on the fine-tuning performance of quantized LLMs remains underexplored.This study presents a comprehensive analysis of these key variables,focusing on their influence across different layer types and depths within LLM architectures.Our investigation uncovers several critical findings:(1)Larger layers,such as MLP layers,can maintain performance despite reductions in adapter rank,while smaller layers,like self-attention layers,aremore sensitive to such changes;(2)The effectiveness of balancing factors depends more on specific values rather than layer type or depth;(3)In quantization-aware fine-tuning,larger layers can effectively utilize smaller adapters,whereas smaller layers struggle to do so.These insights suggest that layer type is a more significant determinant of fine-tuning success than layer depth when optimizing quantized LLMs.Moreover,for the same discount of trainable parameters,reducing the trainable parameters in a larger layer is more effective in preserving fine-tuning accuracy than in a smaller one.This study provides valuable guidance for more efficient fine-tuning strategies and opens avenues for further research into optimizing LLM fine-tuning in resource-constrained environment

关 键 词：Large-scale Language Model Parameter-Efficient Fine-Tuning parameter quantization key variable trainable parameters experimental analysis 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]