acceptodds
Under review as a conference paper at ICLR 2027

A Model-Heterogeneous Federated Learning Method for Mitigating Domain Shift via Virtual Feature Generation

Abstract

Federated learning (FL) becomes particularly challenging when model heterogeneity and domain shift occur simultaneously: heterogeneous architectures prevent direct parameter aggregation, while domain-dependent feature distributions hinder effective knowledge transfer across clients. Prototype-based approaches alleviate model heterogeneity by exchanging class-level feature summaries, yet a single prototype captures only the distribution center and overlooks domain-dependent variations in feature spread. To address these challenges, we propose FedVFG, a model-heterogeneous Federated Learning method for mitigating domain shift through Virtual Feature Generation. Each client summarizes its local feature distributions using class prototypes and diagonal variances, which are scale-calibrated and aggregated by domain on the server. During local training, clients interpolate domain-specific class prototypes and standard deviations to construct virtual domain statistics, which guide the transformation of local features into class-conditioned virtual features. The original and virtual features are then jointly used to optimize the local models, enabling them to learn from cross-domain feature variations across heterogeneous client models. Experiments on Digit-5 and PACS under domain shift in the model heterogeneous setting show that FedVFG achieves state-of-the-art performance, outperforming the strongest baselines by 6.54% and 1.66% in average accuracy, respectively.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Federated Learning, Domain Shift, Model Heterogeneity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JttvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FedVFG addresses federated classification with both domain shift and heterogeneous client architectures. Clients upload class means and diagonal variances. The server normalizes these statistics by a client RMS feature scale and aggregates them by domain and class. Clients interpolate statistics from two supported domains, transform local features through a bounded location-scale map, and train with cross-entropy on both original and virtual features. The paper reports heterogeneous and homogeneous experiments on Digit-5 and PACS, component ablations, communication/computation accounting, and a Digit-5 leave-one-domain-out evaluation.

Critical strengths
  1. The method combines an explicit feature transformation with an architecture-independent communication format and does not require a public dataset or an auxiliary generator.

  2. The reported Digit-5 improvement is substantial, and the paper goes beyond the main accuracy tables with three-seed results, homogeneous-model experiments, unseen-domain evaluation, and cost accounting.

Critical weaknesses
  1. Eqs. (5)-(7) apply one scalar RMS normalization and then average class means and diagonal variances coordinate by coordinate. Eqs. (9)-(13) subsequently use these coordinates to recenter and rescale another client's features. Equal dimensionality and equal RMS magnitude do not establish common semantic coordinates: a feature-channel permutation with a matching inverse permutation in the local classifier leaves that client's predictions unchanged, but changes the communicated statistics and hence the virtual features received by other clients. The stated training objective contains local and virtual-feature cross-entropy, but no explicit explanation or measurement showing how a consistent cross-model coordinate system emerges. This does not establish that the method cannot work empirically. It does leave unclear whether the transmitted variation represents domain structure, architecture-specific basis mismatch, or a mixture of both. 
  2. Removing L_VFG from Eq. (16) leaves only local cross-entropy, with no parameter aggregation. Under matched initialization, data, and optimization, this should recover the standalone learning rule. Yet Table 3 reports 34.21%/30.57% for 'w/o VFG loss', while Tables 1-2 report 27.89%/26.85% for Standalone: gaps of 6.32 and 3.72 points. Moreover, the full method improves over mean-shift-only by 1.15 points on Digit-5 and 0.43 on PACS, and over global statistics by 1.24 and 0.53. These are much smaller than the headline baseline gains. The contribution of diagonal spread and domain-specific statistics therefore needs stronger isolation, particularly against a matched class-conditioned feature-statistic augmentation control using the same information budget.
Questions for authors
  1. What enforces or empirically demonstrates shared feature coordinates across the independently trained architectures? 
  2. Why does 'w/o VFG loss' differ so substantially from Standalone? Please compare them with identical initial states, warmup, sample order, optimizer steps, and normalization-state handling, and describe any other remaining distinction. 
  3. Can you provide matched controls for RMS calibration, class-conditioned moment augmentation, and the added diagonal-variance information, together with accuracy versus cumulative communication on both datasets? How were the adapted heterogeneous baselines tuned?
Reviewer_EtGAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses 2 problems of FL: model heterogeneity and domain shift. Specifically, they exchange the domain-specific class means and variances between server and clients instead of raw data or the learning models. Among various components of the proposed scenario, the largest contributor is the generation of virtual features that improve classification.

Critical strengths

The presentation of the paper is easy to understand, and the discussion flow is easy to follow, which is very good in the current AI research area. The paper is clearly written by hand and does not rely on AI too much.

Critical weaknesses

However, the paper have limited contributions: such as the interpolating class means and standard deviations technique have been proposed for prototypes such as in [R1], sharing domain-specific prototype has also been studied. The new thing here is the model-heterogeneous federated setting but they all have the same output dimension for the prototype, which I consider a drawback. In addition, the paper lacks of the comparision with some latest generation approaches for the FL scenario such as [R2], even FedUSD paper did not consider the domain generation but I believe it can provide an outstanding results within the simulated scenario (they generate the synthetic at local and transmit those data, which can be suitable for the model-heterogeneous FL setting, frankly, FedUSD is not my paper, I just recently came across the paper ). The framework proposed various additional mechanisms that are associated with various coefficients such as interpolation, interpolation bounds, and transformation strengths, making the system quite complex and difficult to tune by hand. The simulations are also limited by the dataset; it lacks the DomainNet dataset.

[R1] Prototype Reminiscence and Augmented Asymmetric Knowledge Aggregation for Non-Exemplar Class-Incremental Learning [R2] FedUSD: Unbiased Synthetic Data for Federated Learning

Questions for authors

Please conduct more simulations with the latest feature/data generation approach for domain generalization. Conduct more simulations with more challenging datasets such as DomainNet and provide an explanation and answers to critical weakness

Reviewer_8BqEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a virtual feature generation method to address the model-heterogeneous federated learning problem, where individual clients employ different model architectures. Unlike conventional federated learning, model heterogeneity prevents the direct aggregation of gradients or model parameters across clients. Therefore, the authors adopt a prototype-based federated learning framework and generate virtual features by leveraging cross-client prototype feature statistics to mitigate domain shifts. These generated virtual features are then utilized for local training. Experimental results demonstrate that the proposed method achieves superior performance compared to existing approaches.

Critical strengths
  • The proposed method is quite simple and achieves substantial performance improvements despite its simplicity.
  • The paper is clearly written and easy to follow.
Critical weaknesses
  • The proposed method is evaluated on only two relatively simple datasets. More extensive evaluations on additional datasets, such as Office-10, are needed to further validate the effectiveness and generalizability.
  • Although the paper provides basic ablation studies, it lacks sufficient experimental analysis and discussion of potential challenges arising from different degrees of model heterogeneity. Additional experiments are needed to investigate how varying levels of model heterogeneity affect the performance and robustness of the proposed approach (The experimental results on the homogeneous model setting in the appendix would be valuable to include in the main text).
  • Usage of \citep{} and \citet{} is incorrect.
Questions for authors
  • Virtual feature-based domain alignment may become less effective when there is a substantial difference in representation capacity across heterogeneous models (e.g., a network with a few M params vs. one with only a few K params). In particular, when clients with significantly different model capacities are paired, the limited representation capacity of the smaller model may reduce the effectiveness of training with augmented virtual features. Therefore, additional analyses are needed to investigate the extent of model heterogeneity that the proposed method can effectively handle.
  • When some clients have only a limited number of samples available for estimating prototype statistics, or when the number of samples fluctuates significantly across training episodes, the resulting statistics may exhibit high variance, potentially leading to instability in virtual feature generation. Therefore, further analysis is needed to evaluate the robustness and stability of the proposed method under such conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.