FedBone:Towards Large-Scale Federated Multi-Task Learning  

作　　者：Yi-Qiang Chen Teng Zhang Xin-Long Jiang Qian Chen Chen-Long Gao Wu-Liang Huang 陈益强;张腾;蒋鑫龙;陈前;高晨龙;黄武亮(Beijing Key Laboratory of Mobile Computing and Pervasive Devices,Institute of Computing Technology Chinese Academy of Sciences,Beijing 100190,China;University of Chinese Academy of Sciences,Beijing 100190,China)

机构地区：[1]Beijing Key Laboratory of Mobile Computing and Pervasive Devices,Institute of Computing Technology Chinese Academy of Sciences,Beijing 100190,China [2]University of Chinese Academy of Sciences,Beijing 100190,China

出　　处：《Journal of Computer Science & Technology》2024年第5期1040-1057,共18页计算机科学技术学报（英文版）

基　　金：supported by the Beijing Municipal Science and Technology Commission under Grant No.Z221100002722009;the National Natural Science Foundation of China under Grant No.62202455;the Youth Innovation Promotion Association of Chinese Academy of Sciences(CAS),the Hunan Provincial Natural Science Foundation of China under Grant No.2023JJ70034;the Science Research Foundation of the CAS-Aier Joint Laboratory on Digital Ophthalmology under Grant No.SZYK202201.

摘　　要：Federated multi-task learning(FMTL)has emerged as a promising framework for learning multiple tasks simultaneously with client-aware personalized models.While the majority of studies have focused on dealing with the non-independent and identically distributed(Non-IID)characteristics of client datasets,the issue of task heterogeneity has largely been overlooked.Dealing with task heterogeneity often requires complex models,making it impractical for federated learning in resource-constrained environments.In addition,the varying nature of these heterogeneous tasks introduces inductive biases,leading to interference during aggregation and potentially resulting in biased global models.To address these issues,we propose a hierarchical FMTL framework,referred to as FedBone,to facilitate the construction of large-scale models with improved generalization.FedBone leverages server-client split learning and gradient projection to split the entire model into two components:1)a large-scale general model(referred to as the general model)on the cloud server,and 2)multiple task-specific models(referred to as client models)on edge clients,accommodating devices with limited compute power.To enhance the robustness of the large-scale general model,we incorporate the conflicting gradient projection technique into FedBone to rectify the skewed gradient direction caused by aggregating gradients from heterogeneous tasks.The proposed FedBone framework is evaluated on three benchmark datasets and one real ophthalmic dataset.The comprehensive experiments demonstrate that FedBone efficiently adapts to the heterogeneous local tasks of each client and outperforms existing federated learning algorithms in various dense prediction and classification tasks while utilizing off-the-shelf computational resources on the client side.

关 键 词：federated learning multi-task learning split learning heterogeneous task 

分 类 号：TP39[自动化与计算机技术—计算机应用技术]