FedAura: Federated Visual Prompting for Cross-Model Adaptation

Federated learning (FL) faces the issue of model heterogeneity in the era of large models. Clients with diverse computational capacities often run models of different sizes. This renders traditional model aggregation methods ineffective. Existing approaches try to address this issue through knowledge distillation or partial training. The former relies on public data and often yields suboptimal performance, while the latter limits model architecture and often leads to uneven optimization. This paper presents FedAura, a federated learning scheme that accommodates model heterogeneity from the input space based on visual prompting. It enables cross-model adaptation by learning and aggregating lightweight visual prompts, which significantly reduces computation and communication overhead. FedAura introduces a model-contrastive loss to alleviate feature corruption in visual prompting caused by model heterogeneity. It aligns cross-model representation based on local consistency between pre-trained and prompted representations. In addition, it leverages a prompt-contrastive loss to alleviate data heterogeneity by driving semantic alignment and improving generalization across diverse data distributions. Extensive experiments across eight heterogeneous architectures (ResNet and ViT variants) on four image recognition datasets show the effectiveness of FedAura in accommodating model heterogeneity. Compared to state-of-the-art knowledge-distillation and partial-training baselines, FedAura improves model accuracy by up to $18.40\%$, reduces communication cost by up to $912\times$, and reduces training FLOPs by up to $7.8\times$. These results demonstrate that input-level prompt collaboration is a promising and resource-efficient direction for model-heterogeneous federated learning.