Gradient sparsification for efficient wireless federated learning with differential privacy  

作　　者：Kang WEI Jun LI Chuan MA Ming DING Feng SHU Haitao ZHAO Wen CHEN Hongbo ZHU 

机构地区：[1]School of Electronic and Optical Engineering,Nanjing University of Science and Technology,Nanjing 210096,China [2]Zhejiang Lab,Hangzhou 311121,China [3]School of Information and Communication Engineering,Hainan University,Haikou 570228,China [4]Data61,Commonwealth Scientific and Industrial Research Organisation,Sydney 2015,Australia [5]School of Communications and Information Engineering,Nanjing University of Posts and Telecommunications,Nanjing 210003,China [6]School of Electronic Information and Electrical Engineering,Shanghai Jiao Tong University,Shanghai 200240,China [7]Key Laboratory of Computer Network and Information Integration Southeast University,Ministry of Education,Nanjing 211189,China

出　　处：《Science China(Information Sciences)》2024年第4期268-284,共17页中国科学（信息科学）（英文版）

基　　金：supported in part by National Natural Science Foundation of China(Grant Nos.62071296,62002170,62071234,U22A2002);National Key Research and Development Program of China(Grant No.2020YFB1807700);Fundamental Research Funds for the Central Universities(Grant No.30921013104);Key Technologies R&D Program of Jiangsu(Prospective and Key Technologies for Industry)(Grant Nos.BE2023022,BE2023022-2);Future Network Grant of Provincial Education Board in Jiangsu;Major Science and Technology Plan of Hainan Province(Grant No.ZDKJ2021022);Scientific Research Fund Project of Hainan University(Grant No.KYQD(ZR)-21008);Youth Foundation Project of Zhejiang Lab(Grant No.K2023PD0AA01);Collaborative Innovation Center of Information Technology,Hainan University(Grant No.XTCX2022XXC07);Sciences and Technology Commission of Shanghai Municipality(Grant Nos.22JC1404000,20JC1416502,PKX2021-D02)。

摘　　要：Federated learning(FL)enables distributed clients to collaboratively train a machine learning model without sharing raw data with each other.However,it suffers from the leakage of private information from uploading models.In addition,as the model size grows,the training latency increases due to the limited transmission bandwidth and model performance degradation while using differential privacy(DP)protection.In this paper,we propose a gradient sparsification empowered FL framework with DP over wireless channels,to improve training efficiency without sacrificing convergence performance.Specifically,we first design a random sparsification algorithm to retain a fraction of the gradient elements in each client's local model,thereby mitigating the performance degradation induced by DP and reducing the number of transmission parameters over wireless channels.Then,we analyze the convergence bound of the proposed algorithm,by modeling a non-convex FL problem.Next,we formulate a time-sequential stochastic optimization problem for minimizing the developed convergence bound,under the constraints of transmit power,the average transmitting delay,as well as the client's DP requirement.Utilizing the Lyapunov drift-plus-penalty framework,we develop an analytical solution to the optimization problem.Extensive experiments have been implemented on three real-life datasets to demonstrate the effectiveness of our proposed algorithm.We show that our proposed algorithms can fully exploit the interworking between communication and computation to outperform the baselines,i.e.,random scheduling,round robin,and delay-minimization algorithms.

关 键 词：federated learning differential privacy gradient sparsification Lyapunov drift convergence analysis 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程] TP309[自动化与计算机技术—控制科学与工程]