面向联邦学习的学习率裁剪梯度优化隐私保护方案  

Learning Rate Clipping Gradient Optimization Privacy Protection Scheme for Federated Learning

作　　者：孟向前 刘腾飞 谢绒娜[1] MENG Xiangqiann;LIU Tengfei;XIE Rongna(Beijing Electronic Science and Technology Institute,Bejing 100070,P.R.China)

机构地区：[1]北京电子科技学院,北京市100070

出　　处：《北京电子科技学院学报》2023年第1期45-53,共9页Journal of Beijing Electronic Science And Technology Institute

基　　金：国家重点研发计划项目(项目编号:2017YFB0801803)。

摘　　要：联邦学习中,攻击者通过模型梯度攻击来恢复训练数据集,使训练数据集的隐私性受到威胁。为保护数据隐私性,差分隐私技术被引入到联邦学习中,但在神经网络训练过程中存在学习率过大导致梯度爆炸不收敛或学习率过小导致梯度收敛过慢的问题,降低学习的准确率。针对上述问题,本文提出一种具有自适应学习率的梯度优化算法(CAdabelief算法),该算法在神经网络中引入学习率裁剪动态界限的概念,动态调整学习率达到理想的值,并趋于稳定;继而将CAdabelief算法引入联邦学习差分隐私框架,提出了面向联邦学习的学习率裁剪梯度优化隐私保护方案,并采用MNIST数据集进行测试实证。实验表明,在相同的隐私预算下,CAdabelief算法训练结果的准确率高于常用的SGD、Adam、Adabelief算法。In federated learning,attackers recover the training data set using the model gradient attack,which threatens the privacy of the training data set and leads to privacy leakage.To protect the data pri-vacy,differential privacy technology is introduced to the federated learning.However,a problem that the learning rate is too large to converge the exploded gradient or too small to accelerate the gradient convergence speed exists in the process of neural network training,reducing the learning accuracy.To address the problem,a gradient optimization algorithm with adaptive learning rate(i.e.the CAdabelief algorithm)is proposed in this paper,where the concept of learning rate clipping dynamic boundary is introduced to the neural network to dynamically adjust the learning rate to reach the desired value as well as to be stable.Then the CAdabelief algorithm is introduced to the federated learning differential privacy framework and a learning rate clipping gradient optimization privacy protection scheme for feder-ated learning is proposed,whose performance is tested using the MNIST data set.Test result indicates that the CAdabelief algorithm achieves higher training accuracy than the popular SCD,Adam,and Ad-abelief algorithms under the condition of same privacy budget.

关 键 词：联邦学习 差分隐私 自适应 学习率裁剪 梯度优化 

分 类 号：TN01[电子电信—物理电子学]