面向联邦学习的学习率裁剪梯度优化隐私保护方案  

Learning Rate Clipping Gradient Optimization Privacy Protection Scheme for Federated Learning

作　　者：孟向前 刘腾飞 谢绒娜[1] MENG Xiangqian;LIU Tengfei;XIE Rongna(Beijing Electronic Science and Technology Institute,Beijing 100070,P.R.China)

机构地区：[1]北京电子科技学院,北京市100070

出　　处：《北京电子科技学院学报》2024年第4期46-54,共9页Journal of Beijing Electronic Science And Technology Institute

基　　金：国家重点研发计划项目(2017YFB0801803)。

摘　　要：联邦学习中,攻击者通过模型梯度攻击来恢复训练数据集,使训练数据集的隐私性受到威胁,存在隐私泄露。为保护数据隐私性,差分隐私技术被引入到联邦学习中,但在神经网络训练过程中存在学习率过大导致梯度爆炸不收敛或学习率过小导致梯度收敛过慢的问题,降低学习的准确率。针对上述问题,本文提出一种具有自适应学习率的梯度优化算法(CAdabelief算法),该算法在神经网络中引入学习率裁剪动态界限的概念,动态调整学习率达到理想的值,并趋于稳定。将CAdabelief算法引入联邦学习差分隐私框架,提出了面向联邦学习的学习率裁剪梯度优化隐私保护方案。并采用MNIST数据集进行测试实证。在相同的隐私预算下,CAdabelief算法训练结果的准确率高于常用的SGD、Adam、Adabelief算法。In federated learning,attackers recover the training data set through model gradient attack,which threatens the privacy of the training data set and leads to privacy leakage.In order to protect data privacy,differential privacy technology is introduced into federated learning.However,in the process of neural network training,there is a problem that the learning rate is so large that causes the gradient to explode and not to converge or the learning rate is so small that causes the gradient convergence to be too slow,which reduces the accuracy of learning.In view of the above problems,this paper proposes a gradient optimization algorithm with adaptive learning rate(CAdabelief algorithm).This algorithm in⁃troduces the concept of learning rate clipping dynamic boundary in neural network,dynamically adjusts the learning rate to reach the ideal value and tends to be stable.The CAdabelief algorithm is introduced into the federated learning differential privacy framework,and a learning rate clipping gradient optimi⁃zation privacy protection scheme for federated learning is proposed.The MNIST data set is used for test verification.Under the same privacy budget,the accuracy of the training results of the CAdabelief algo⁃rithm is higher than that of the commonly used SGD,Adam,and Adabelief algorithms.

关 键 词：联邦学习 差分隐私 自适应 学习率裁剪 梯度优化 

分 类 号：TN919[电子电信—通信与信息系统]