acceptodds
Desk rejected at ICLR 2027

FedMG: Model Growth for Federated Learning with System Heterogeneity

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated learning often encounters the challenge of system heterogeneity, manifested in diverse local hardware resource configurations and varying client-specific model capacities. To address this, several methods have been proposed: Knowledge distillation (KD)-based methods leverage a public dataset to transfer knowledge. Low-rank training methods utilize low-rank parameters. Partial training methods extract sub-models from the global model. Unlike these methods, we introduce a novel model-growth-based federated learning method FedMG to address the challenges of system heterogeneity. We first train an atomic model that can be trained on all participating clients. Subsequently, we introduce a new model growth strategy that focuses on newly introduced parameters, then progressively build larger models. During the training process, FedMG facilitates bidirectional knowledge transfer: Small models provide their parameters to the construction of larger models, and these larger models provide feedback to the small models. FedMG successfully assigns each client a model not exceeding its local capacity, and extensive experiments demonstrate that FedMG consistently outperforms existing methods across various scenarios.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Federated Learning, System Heterogeneity

Related papers

Discussion (0)

Sign in to comment.