FedMG: Model Growth for Federated Learning with System Heterogeneity

Federated learning often encounters the challenge of system heterogeneity, manifested in diverse local hardware resource configurations and varying client-specific model capacities. To address this, several methods have been proposed: Knowledge distillation (KD)-based methods leverage a public dataset to transfer knowledge. Low-rank training methods utilize low-rank parameters. Partial training methods extract sub-models from the global model. Unlike these methods, we introduce a novel model-growth-based federated learning method FedMG to address the challenges of system heterogeneity. We first train an atomic model that can be trained on all participating clients. Subsequently, we introduce a new model growth strategy that focuses on newly introduced parameters, then progressively build larger models. During the training process, FedMG facilitates bidirectional knowledge transfer: Small models provide their parameters to the construction of larger models, and these larger models provide feedback to the small models. FedMG successfully assigns each client a model not exceeding its local capacity, and extensive experiments demonstrate that FedMG consistently outperforms existing methods across various scenarios.