acceptodds
Under review as a conference paper at ICLR 2027

FedAura: Federated Visual Prompting for Cross-Model Adaptation

Abstract

Federated learning (FL) faces the issue of model heterogeneity in the era of large models. Clients with diverse computational capacities often run models of different sizes. This renders traditional model aggregation methods ineffective. Existing approaches try to address this issue through knowledge distillation or partial training. The former relies on public data and often yields suboptimal performance, while the latter limits model architecture and often leads to uneven optimization. This paper presents FedAura, a federated learning scheme that accommodates model heterogeneity from the input space based on visual prompting. It enables cross-model adaptation by learning and aggregating lightweight visual prompts, which significantly reduces computation and communication overhead. FedAura introduces a model-contrastive loss to alleviate feature corruption in visual prompting caused by model heterogeneity. It aligns cross-model representation based on local consistency between pre-trained and prompted representations. In addition, it leverages a prompt-contrastive loss to alleviate data heterogeneity by driving semantic alignment and improving generalization across diverse data distributions. Extensive experiments across eight heterogeneous architectures (ResNet and ViT variants) on four image recognition datasets show the effectiveness of FedAura in accommodating model heterogeneity. Compared to state-of-the-art knowledge-distillation and partial-training baselines, FedAura improves model accuracy by up to , reduces communication cost by up to , and reduces training FLOPs by up to . These results demonstrate that input-level prompt collaboration is a promising and resource-efficient direction for model-heterogeneous federated learning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
federated learning, visual prompting, model heterogeneity, contrastive learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_H9kXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FedAura, a federated learning scheme for the model-heterogeneity setting in which clients run pre-trained backbones of different architectures and sizes (CNN and ViT families). Instead of aggregating model weights or logits, each client freezes its backbone and learns a pixel-level visual prompt in the input space; because the prompt's dimension depends only on image resolution and is decoupled from the backbone, the server can aggregate prompts across heterogeneous architectures with plain FedAvg. To stabilize aggregation, FedAura adds two InfoNCE-style regularizers during local training: a model-contrastive loss aligning the prompted representation to the unprompted backbone representation pulling the local prompt's representation toward that of the global prompt.

Critical strengths

-Framing cross-model adaptation as aggregating an architecture-agnostic, input-level artifact is a practical and under-explored direction. The design is simple, parameter-light, and conceptually extends federated prompting beyond the homogeneous/Transformer-only settings that bound prior work.

-Beyond the main table, this work also includes more ablation studies to better understand the method design and the contributions of each component.

Critical weaknesses

-The authors' own motivation study (Fig. 3) shows prompts are strongly model/data-dependent: applying a prompt trained on one client to another architecture drops accuracy from 0.788→0.625 (ViT-Small) and 0.594→0.486 (ResNet-18) on CIFAR-10, with further drops under Model+Data-Het. Given that the aggregated global prompt is re-applied to different backbones each round, the entire premise rests on the global prompt transferring well — yet the paper never compares FedAura against each client training its own prompt in isolation (with the same two losses, no aggregation). All ablations (Table 3) keep the federated aggregation loop, so the "cross-model knowledge sharing" contribution is not isolated. Without this baseline, the reported gains could in principle come from the regularizers alone rather than from federation.

-The paper's central architectural premise, combining a server-aggregated global prompt with client-local prompt optimization, is not novel: this global/local prompt cooperation paradigm is already established in PromptFL, pFedPrompt/PromptFed, and PLAN (all cited by FedAura itself), and most directly in the CVPR 2026 work Controllable Federated Prompt Learning at Test Time (Zhu et al., COTE), which dynamically balances global and local prompts alongside a frozen CLIP prompt. FedAura's prompt-contrastive loss is functionally the same drift-alignment/personalization regularization these methods use, so the only genuinely new element is the pixel-level, input-space prompt interface that decouples from backbone architecture.

-Table 1 reports HeteroFL at 53.31% (CIFAR-10, IID) — far below the accuracy HeteroFL reports in its own paper on the same data, suggesting under-tuning; FedEcover similarly collapses on Tiny-ImageNet. Meanwhile several strong heterogeneous-FL methods cited in the related work (FedDSE, FlexiFed, FedDecorr) are never compared. The "up to 18.40%" abstract claim is also not traceable to any single Table-1 entry (closest is +18.01% on CIFAR-10 β=1 vs. FedMD; e.g., Flower-102 β=1 vs. FedMD is +19.09%). With the strongest competitors absent and remaining baselines apparently under-tuned, the reported margins are not conclusive.

-Figure 1 is a schematic restatement of the model-heterogeneity definition with no added insight and could be dropped, while Figure 2, the only method figure, is a generic block diagram that never shows where the pixel prompt is injected, how the two contrastive losses act on the four representations, or the aggregation vs. local-head data flow.

Questions for authors

-Can the authors report a no-federation control: each client trains its own prompt locally with the same two losses but no aggregation? This is the single most important experiment for validating that the global prompt transfers knowledge across heterogeneous models (given your own Fig. 3 shows prompts are highly model-dependent).

-Why do HeteroFL and FedEcover perform far below their published results on the same datasets? Please report results with the authors' recommended hyperparameters, and add comparisons with the heterogeneous-FL baselines you cite but do not evaluate (e.g., FedDSE, FlexiFed).

-In Eq. 8, LmL_m repels the prompted representation from the previous round's prompted representation of the same image. What is the motivation for using a temporally adjacent state as the negative, and did you ablate alternative negatives (e.g., other samples or the global-prompt representation as negative)?

Reviewer_nproOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FedAura is a federated learning method that enables collaboration across heterogeneous vision models by sharing pixel-level visual prompts. Clients freeze their pretrained backbones, train prompts and local classifier heads, and aggregate only the prompts. The method introduces two contrastive losses: one preserves the original backbone representations, while the other encourages consistency with the global prompt to reduce drift across clients. Experiments on four image-classification datasets report improved accuracy and reduced communication costs.

Critical strengths
  1. Compatibility across architectures. Pixel-level prompts provide a common aggregation interface for CNN and Transformer backbones, addressing a practical challenge in heterogeneous federated learning.

  2. Lightweight communication. Clients exchange only prompts while retaining their backbones and classifier heads locally, reducing communication compared with transmitting model weights.

  3. Supportive ablation results. Table 3 shows consistent improvements from combining both contrastive losses across four datasets, supporting their empirical usefulness.

Critical weaknesses
  1. Nonstandard paper formatting. The PDF uses approximately 6.5-inch text width and a 1-inch left margin, exceeding the layout allowed by the official template, which specifies 5.5 inches and 1.5 inches.

  2. Unjustified historical negatives. Both contrastive losses push representations away from previous local prompts, which may contain useful information. Is it possible to extract any knowledge from previous prompts instead of pushing them away directly?

  3. Cross-model collaboration is not clear. Figure 6 increases the proportion of larger backbones, so improved overall accuracy does not establish that weaker clients benefit from cross-model transfer.

  4. Potential mismatch between aggregated prompts and local classifiers. Prompts are jointly trained with local classifier heads but aggregated independently of them. The updated global prompt may produce features that are less compatible with local classifiers.

Questions for authors
  1. Please refer to the weakness part.

  2. Limited heterogeneity evaluation. Experiments consider label skew with β = 1 and 10, without stronger skew or cross-domain shifts. The effectiveness of a shared prompt under substantially different data distributions remains unclear.

Reviewer_zi9SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes to address the issues of model heterogeneity and data heterogeneity among clients in federated learning. Previous works utilizing knowledge distillation or partial training have achieved suboptimal results. FedAura introduces two losses to mitigate shift issues during training from the model and prompt perspectives, respectively. Specifically, the model-contrastive loss aligns the prompted representations with the frozen backbone's features to maintain semantic consistency across heterogeneous architectures, while the prompt-contrastive loss aligns local prompts with the global prompt to prevent semantic drift caused by non-IID data distributions. Compared to knowledge distillation and partial training methods, FedAura achieves improvements in model accuracy, communication cost, and training cost.

Critical strengths

1.By training only lightweight visual prompts on frozen backbones, FedAura reduces communication costs by up to 912x and training FLOPs by 7.8x compared to SOTA baselines.

2.Pixel-level visual prompts operate directly in the input space, decoupling from internal network dimensions. This guarantees universal compatibility across any CNN or Transformer architecture.

3.A model-contrastive loss aligns features to overcome model heterogeneity, while a prompt-contrastive loss aligns local-to-global prompts to prevent semantic drift under non-IID data.

4.FedAura eliminates the reliance on shared public datasets required by traditional knowledge distillation methods, achieving cross-model collaboration purely through safe prompt exchange.

Critical weaknesses

1.By solely aggregating visual prompts at the input layer, FedAura lacks deep fine-tuning capabilities. This input-level restriction likely imposes a theoretical limit on its ability to directly manipulate the model's high-level abstract semantics.

2.The proposed contrastive losses require retaining multiple feature representations (clean, current, previous, and global) in memory simultaneously. This design necessitates at least four forward passes through the backbone per batch, introducing significant computational overhead that is omitted from the paper's FLOPs analysis and making it unfriendly to resource-constrained devices.

3.FedAura fails under extreme conditions: extreme model heterogeneity breaks its contrastive alignment, severe data heterogeneity causes unpreventable prompt drift, and large domain gaps render the frozen backbone ineffective.

Questions for authors

1.What is the exact ratio of the different heterogeneous architectures used in the main experiments?

2.Could the authors explain why the previous-round prompt representation zkprez_k^{pre} is used as the negative sample in Eqs. (8) and (9), and what motivates repelling the current prompted representation from it?

3.The paper sets λm=λp=1{\lambda}_m={\lambda}_p=1, τ=0.5\tau=0.5, and local epochs =10=10 by default (and evaluates M∈{1,3,5}M\in\{1,3,5\} in Fig. 5), but provides no sensitivity analysis for these hyperparameters. Could the authors report how performance varies with λm{\lambda}_m, λp{\lambda}_p, τ\tau, MM, and the number of local epochs, and justify the default choices?

4.Even though the prompt is input-level, backpropagating ∇PkL\nabla_{P_k}\mathcal{L} still traverses the entire frozen backbone. How is this backward cost omitted from the FLOPs analysis in Section 4.5/Appendix B?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.