acceptodds
Under review as a conference paper at ICLR 2027

What Does the Global Model Leave Behind? Client-Conditioned Residual Experts for Federated Learning

Abstract

Federated learning typically treats a trained global model as the endpoint of collaborative optimization, yet even strong shared models can leave structured, client-dependent residual errors. We ask what predictive structure remains after shared learning and whether it can be corrected without relearning the deployed model. We introduce FedCoREx (Federated Client-Conditioned Residual Experts), which freezes a deployment-selected shared model and federatively learns a bank of shared residual experts together with a client-conditioned router. The router combines frozen sample representations, base prediction probabilities, and train-only client contexts summarizing the base model's class-conditional prediction behavior to produce additive logit corrections, while preserving the original predictor as an always-on substrate. We further characterize the residual correction problem theoretically: the unrestricted log-loss headroom of a frozen predictor decomposes into sample-only model misspecification and additional client-conditioned information, while the gain realized by FedCoREx also depends on the expressiveness of its residual family and the effectiveness of learning. This perspective explains why statistical heterogeneity alone does not imply correctable residual structure. Across controlled and natural federated settings, FedCoREx yields substantial improvements when strong client-conditioned residual structure is exposed, while producing modest or near-neutral corrections in settings where little usable residual headroom is captured by the current context and correction family. Mechanism controls and evaluations across multiple shared-model substrates further support the roles of conditional context and residual composition. These results position residual correction as a complementary layer to global federated optimization rather than a replacement for it.

Primary area:
optimization
Keywords:
Federated Learning, Non-IID Data, Mixture of Experts, Residual Learning, Client Conditioning, Conditional Routing, Model Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pkKQOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FedCoREx, a post-hoc federated correction framework that learns a shared residual correction on top of a frozen base predictor. Each client constructs a confusion-based context from its local training data, which is combined with sample representations and base predictions to route inputs through shared residual experts. The experts and router are jointly trained with federated averaging, while validation selects the correction checkpoint, including a zero-correction option. The method is evaluated across multiple federated learning settings to assess its effectiveness in correcting client-dependent prediction errors.

Critical strengths
  1. The base predictor is frozen throughout correction training, and zero-output initialization makes the initial correction exactly equivalent to the original predictor. Moreover, validation explicitly includes the zero-correction candidate, providing a principled way to avoid deploying a harmful correction.

  2. Rather than learning independent client-specific correction models, FedCoREx uses globally shared residual experts while introducing client dependence through a locally constructed confusion context. This provides a compact mechanism for adapting the same correction functions to different client-level error patterns.

Critical weaknesses
  1. All experts are globally shared and jointly optimized with the router using the same corrected-prediction cross-entropy objective, without an explicit diversity or specialization constraint. Since all experts are evaluated for every sample and their outputs are combined through a dense mixture, it is unclear whether the learned experts develop complementary correction functions or become partially redundant. This matters directly to the necessity of the proposed expert-bank architecture.

  2. FedCoREx introduces a bank of shared residual networks, a context-conditioned router, and an additional client-side context-construction procedure on top of the frozen base. Although the paper provides parameter, computation, and communication accounting, the methodological question remains whether this additional complexity is necessary relative to simpler correction mechanisms. In particular, the paper would benefit from a clearer assessment of how the performance gain changes with the number of experts and how this gain should be weighed against the additional training and inference cost of evaluating all experts.

Questions for authors
  1. All experts share the same architecture and are jointly optimized with the same corrected-prediction cross-entropy objective, without an explicit specialization mechanism. What is the intended mechanism that makes different experts learn complementary correction functions rather than redundant ones?

  2. The method evaluates all KK residual experts for every input and then performs dense mixture through the router. Thus, both training and inference costs increase with the number of experts on top of the frozen base model. What motivates this dense expert formulation over a simpler single residual function, and how should the additional computation be viewed relative to the intended correction benefit?

  3. The context cic_i is constructed once from the frozen base's predictions on local labeled training data and then kept fixed throughout correction training and inference. What assumptions justify treating this historical confusion profile as a stable conditioning signal for subsequent residual correction?

  4. The framework assumes that heterogeneous client errors can be represented by a common bank of residual functions whose mixtures vary across clients and samples. What properties of the client heterogeneity make this decomposition appropriate?

4 comments
AuthorsOpenReview ↗

The response of W1&Q1&W2&Q2

W1&Q1

Independent hidden projections and input-dependent routing give the experts different learning signals.

(a) Let pp be the corrected prediction, eye_y the one-hot label, and g=p−eyg=p-e_y. With residual r=∑kqkΔkr=\sum_kq_k\Delta_k, expert parameters θk\theta_k, and pre-softmax routing scores sks_k, the gradients are

=q_k\left(\frac{\partial\Delta_k}{\partial\theta_k}\right)^\top g, \qquad \frac{\partial\ell}{\partial s_k} =q_k\,g^\top(\Delta_k-r).$$ Each expert receives an update modulated by its routing weight and its own Jacobian. The router adjusts scores according to the loss contribution of each expert relative to the mixture. At initialization, the output projections, residual, and task gradient to the router are zero. Independently initialized hidden projections produce different activations, so the expert output projections receive different initial updates; the router and experts subsequently learn jointly. This explains how different functions can emerge under a shared objective. (b) The observed routing is selective: the original Full has mean per-example effective expert counts of approximately 1.404/1.244 on Concept/Feature+Concept, compared with 1.809/1.435 for the MLP4 router. These statistics describe how inputs use the expert bank; the prediction effects are evaluated through the structural controls. The revision explains the gradients and initialization and uses these concrete observations to describe expert behavior. **W2&Q2** We use direct architectural controls and expert-budget comparisons to explain the motivation for expert composition and its performance–cost tradeoffs. (a) The design represents reusable corrections through shared experts, with sample state and client context determining their composition. The base executes once, the lightweight experts operate on its frozen representation, and dense soft mixing supports joint training. We also evaluated Single-Conditional as a simpler alternative with the same information sources and a closely matched parameter count. On the original CIFAR-10 backbone, Full reaches 67.916±3.336 / 54.942±1.632% on Concept/Feature+Concept, versus 73.693±0.328 / 58.364±1.424% for Single. Full/Single results are 72.885±1.798 / 72.995±1.237% on FEMNIST and 66.205±1.160 / 66.219±1.159% on Sentiment140. Both implement shared conditional residual learning; Single achieves higher accuracy in the tested CIFAR-10 settings, while their mean accuracy is close on the natural tasks. The revision includes this direct comparison to clarify the motivation and observed performance of expert composition. (b) Within the expert architecture, we compared K=2/4/8 at approximately 6.85k total correction parameters, with expert widths 34/16/7. For 50 clients, Concept accuracy is 65.373±0.192 / 67.916±3.336 / 71.098±2.134%, and Feature+Concept accuracy is 52.556±0.812 / 54.942±1.632 / 56.244±1.587%. Allocating the near-matched budget to more, narrower experts improves mean accuracy in these two settings. Increasing K at a fixed expert width increases computation; the budget-matched study instead reduces expert width as K increases. The revision distinguishes these comparisons and presents them alongside the parameter, training, communication, and inference accounting. All experts are evaluated, and routing concentration describes their mixing weights.
AuthorsOpenReview ↗

The response of Q3&Q4

Q3

Context summarizes the frozen base's class-conditional prediction behavior on each client's training distribution. With the base fixed, an approximately stable client distribution, and training samples representative of deployment data, this summary provides a consistent conditioning signal.

We additionally tested distribution changes by multiplying image brightness by 0.7 or 1.3. Keeping the learned parameters and original context fixed, correction improves accuracy over the base under the same transformation by approximately 8.0–11.1 pp on Concept/Feature+Concept, with corresponding improvements in bottom-client accuracy and NLL. Thus, the historical context retains correction value under the tested brightness shifts. Refreshing it from labeled training data with the same transformation changes mean accuracy by −0.071 to 0.000 pp; the original context already retains useful information in these conditions. The revision clarifies the fixed-context assumptions and includes this deployment diagnostic.

Q4

The decomposition is appropriate when the frozen representation retains useful predictive information, client errors contain reusable correction patterns, and context together with sample state can distinguish the required corrections. Repeated label-mapping differences over shared visual representations provide one such example. Our analysis separates conditional-information headroom under log-loss from the approximation and learning gaps of a finite residual family. The permutation-style model explains context identifiability through a shared prediction channel, positive class support, and distinguishable channel rows.

The cross-setting results are consistent with this motivation. With 50 clients, K=2/4/8 gives approximately 74.92/74.88/74.86% accuracy under IID, 65.37/67.92/71.10% under Concept, and 52.56/54.94/56.24% under Feature+Concept. With 100 clients, K=4 improves over its own base by 12.413±3.373 and 9.022±2.069 pp in the two concept-shift settings. FEMNIST also shows positive mean accuracy effects across three bases. The revision connects these results to the reusable-error-pattern interpretation and states the corresponding theoretical conditions explicitly.

Thank you to the authors for addressing my questions and providing additional experiments. I also apologize for my late reply.

After reviewing the rebuttal and the additional experiments from the other reviewers, I still have some reservations about the MoE design. The comparison with Single-Conditional shows that a simpler conditional network can achieve comparable or even better performance than the original Full model under a similar parameter budget. While Context-Experts narrows this gap, it does not fully establish the advantage of using multiple experts. In addition, although the authors provide explanations of expert optimization and routing statistics, stronger evidence is still needed to demonstrate that the experts learn distinct and complementary correction functions.

I appreciate the authors' efforts to address these concerns and have adjusted my score to 2 accordingly. However, I believe the motivation for the MoE design and the role of expert specialization could be further clarified.

Thank you for reviewing our responses and additional experiments, and for reassessing our work. Your comments have helped us identify the shortcomings of the current design more clearly: the original MoE architecture makes limited use of conditional information, and the evidence for distinct, complementary correction functions across experts remains insufficient. We accept this criticism and will use it to guide further development and evaluation.

Our starting question was whether a strong federated global model can still leave client-distribution-dependent prediction errors that can be corrected through parameters shared across clients. Our experiments provide empirical support for this idea. We also see potential value in practical settings such as newly arriving clients: local support data could provide the context needed to adapt a shared correction module to a client's distribution, without maintaining a separate trainable model for every client.

We chose MoE with the intuition that different experts could learn different error patterns across heterogeneous clients and develop complementary correction capabilities. This intuition motivated the architecture, but your feedback has made clear the need to test more directly whether such specialization actually emerges and provides additional benefits. We initially prioritized a unified shared-correction mechanism and sought to avoid designing separate architectures and contexts for different types of heterogeneity. Consequently, we did not explore alternative parameterizations and context representations thoroughly enough, leaving some weaknesses of the implementation unidentified.

Our recent exploratory results suggest that, in some of the conditions examined, changes to the correction location and context representation can improve upon the original MoE and Single-Conditional configurations. These observations encourage us to investigate which information reveals the correctable errors left by a strong global model, and how shared learning can use that information effectively. Our longer-term goal is to develop a well-grounded correction method that responds to client heterogeneity, rather than to identify a universally optimal residual architecture.

Thank you again for your careful and concrete feedback. It has helped us understand the limitations of the current work and identify a more focused direction for improving it.

Reviewer_Hn9COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Federated learning can leave systematic client-dependent errors in a trained global model. This paper proposes FedCoREx, which freezes such a global model and trains residual experts together with a client-conditioned router to correct its predictions without modifying the base model. The paper employs an MoE-like strategy to construct experts and a router to correct the residual of the base model.

Critical strengths
  • The method seems interesting, and novel enough to make a paper.
  • The paper contains a useful theory to motivate the implementation of the proposed method.
  • The method seems to perform well in most cases.
  • One thing I was most concerned about was the context leaking outside the clients and being sent to the server. The method says that they are kept local, which is a nice thing.
Critical weaknesses
  • For some cases, FedCoREx does not show an impressive performance gap from the baseline FedGMT. The paper explicitly says that FedCoREx uses the frozen FedGMT base so I asked myself: how can one know if the impressive performance comes from the frozen base or the method itself? I can tell from Table 3 and Table 5 that FedCoREx is effective for various base models. But those tables are only for two cases of CIFAR-10. The strongest evidence for FedCoREx is concentrated on synthetic permutation-style concept shifts, whereas the gains on natural heterogeneity are modest or negligible. It is therefore unclear how broadly the proposed residual structure occurs in realistic FL problems. Moreover, the experiments use relatively small architectures, so it remains unclear whether the observed gains extend to stronger and more realistic modern backbones. The authors should do experiments more with complex architectures.
  • How much is the proposed method affected by the performance of the router? The router is deliberately lightweight. While the Full-vs-NoContext ablations demonstrate that client-conditioned routing is important, it is unclear whether the chosen router has sufficient capacity as client heterogeneity and task complexity increase. How sensitive is FedCoREx to router capacity?
  • This is a question on the method: how much is the proposed method dependent on the number of experts, especially when the number of clients is numerous? I see that the paper already tests 2/4/8 experts and finds that K=8K = 8 is the better. But is it dependent on the number or the heterogeneity of the clients? If so, it could be a weakness of the method.
Questions for authors
  1. Can you do more experiments for IID and Label skew cases with various frozen base models, aside from FedGMT?
  2. Can you do more experiments with networks of a more complex architecture, like ResNet-18? If the two questions above are resolved, I would like to raise the score to acceptance.
  3. (Not necessary) If I am right, the training of the router and experts is being done by local SGD (FedAvg). What if the training is done by a more advanced method such as SCAFFOLD or FedDyn? Would it enhance the performance of the method?
6 comments
AuthorsOpenReview ↗

The response of W1&Q1&Q2

W1&Q1&Q2

(a) The gains measure correction of the same deployed base. Base and Full use exactly the same deployment-selected base checkpoint, whose weights remain frozen throughout correction training. On the original CIFAR-10 Concept and Feature+Concept settings, Full improves over FedGMT Base by 11.444±2.957 and 9.151±1.271 pp. This paired comparison directly measures the contribution of the correction layer.

(b) The cross-base evidence covers controlled and natural heterogeneity. The existing appendix includes FedAvg/FedProx/FedGMT results on Fashion-MNIST Concept/Feature+Concept, FEMNIST, and Sentiment140. Mean accuracy gains on FEMNIST are 1.701/3.075/1.440 pp for these three bases; the Sentiment140 change over FedGMT is +0.255±0.244 pp. We have now added CIFAR-10 IID/Label experiments with the other bases:

Setting / baseFull−Base accuracy (pp)Full−Base NLL
IID / FedAvg+0.160 ± 0.061−0.00147 ± 0.00047
IID / FedProx−0.022 ± 0.149−0.00149 ± 0.00100
IID / FedGMT+0.022 ± 0.073−0.00108 ± 0.00097
Label / FedAvg+0.467 ± 0.187−0.01685 ± 0.00811
Label / FedProx+0.720 ± 0.346−0.01613 ± 0.00787
Label / FedGMT−0.107 ± 0.096−0.02491 ± 0.00092

Mean NLL improves in all six settings, with accuracy gains of 0.467 and 0.720 pp for FedAvg and FedProx under label skew. Alongside the natural-data results, these experiments show how correction utility varies with the client-dependent errors left by the frozen predictor. The conditional-information, approximation, and learning terms in our analysis provide a way to interpret these differences.

(c) Correction also yields positive mean gains with ResNet-18. We added CIFAR-10 experiments using a FedAvg-trained ResNet-18, with three training repetitions per setting:

SettingBase accuracy (%)Full accuracy (%)Full−Base accuracy (pp)Full−Base NLL
Concept63.773 ± 0.48871.613 ± 3.234+7.840 ± 3.634−0.4362 ± 0.1707
Feature+Concept53.138 ± 0.94058.849 ± 0.860+5.711 ± 1.800−0.2928 ± 0.0689

Mean accuracy increases by approximately 7.8 pp on Concept and 5.7 pp on Feature+Concept, together with lower NLL. These results extend the evidence for correction to a stronger backbone. The revision includes the cross-base and ResNet-18 results and clearer main-text pointers to the natural-data evidence.

AuthorsOpenReview ↗

The response of W2&W3&Q3

W2

We examine both the routing function at a matched parameter budget and the effect of increasing router capacity.

We fixed the expert bank and the 200-round budget and compared the original linear router, a factorized linear router (Fact4), and MLP routers with hidden widths 4, 16, and 64. Fact4 and MLP4 have exactly the same parameter count, close to the original Full budget.

RouterTotal correction parametersConcept accuracy (%)Feature+Concept accuracy (%)
Original linear Full6,85667.916 ± 3.33654.942 ± 1.632
Fact46,87267.867 ± 3.49854.960 ± 1.646
MLP46,87270.364 ± 1.74553.920 ± 1.766
MLP169,24870.604 ± 2.32754.289 ± 0.934
MLP6418,75270.427 ± 2.88553.796 ± 4.440

At the same parameter count, MLP4 improves over Fact4 by 2.498±2.192 pp on Concept, while MLP16 reaches 70.604%. The original lightweight linear router remains competitive on Feature+Concept. Thus, a more expressive routing function can help particular tasks, while the benefit of additional capacity depends on the data condition. The linear router provides a low-cost default. The revision includes the full capacity comparison.

W3

The choice of expert count depends on heterogeneity, per-expert capacity, and local data support; we examine these factors across constructions and client populations at closely matched budgets.

We compared K=2/4/8 with closely matched total correction budgets of 6,848/6,856/6,872 parameters and expert widths of 34/16/7. The 50-client evaluation covers four heterogeneity settings; Concept and Feature+Concept are additionally evaluated on nested 100-client partitions.

Clients / settingK=2 accuracy (%)K=4 accuracy (%)K=8 accuracy (%)
50 / IID74.920 ± 0.40174.884 ± 0.34174.858 ± 0.357
50 / Label73.356 ± 0.79673.449 ± 0.79574.991 ± 0.883
50 / Concept65.373 ± 0.19267.916 ± 3.33671.098 ± 2.134
50 / Feature+Concept52.556 ± 0.81254.942 ± 1.63256.244 ± 1.587
100 / Concept62.524 ± 3.43267.418 ± 4.34668.889 ± 3.878
100 / Feature+Concept50.253 ± 0.24352.836 ± 2.22652.218 ± 1.130

The three allocations perform similarly under IID. Concept shows higher mean accuracy with increasing K at both client counts. For Feature+Concept, the highest mean occurs at K=8 with 50 clients and K=4 with 100 clients. With 100 clients, K=4 still improves over its own base by 12.413±3.373 and 9.022±2.069 pp.

The shared bank can therefore be much smaller than the client population. A suitable K depends jointly on heterogeneity, per-expert capacity, and local sample support. The nested partition preserves the total data and mode set while splitting each parent's train/validation/test data between two clients; local data size is halved, bases are retrained, and the participation fraction is unchanged. The revision includes these results and the partition specification.

Q3

The correction layer accommodates different federated optimizers, with SCAFFOLD further improving mean performance in both tested settings.

We trained the correction layer with FedAvg, SCAFFOLD, and FedDyn using the same frozen FedGMT base, context, K=4 architecture, and 200-round budget.

Correction optimizerConcept accuracy (%)Feature+Concept accuracy (%)
FedAvg67.916 ± 3.33654.942 ± 1.632
SCAFFOLD68.707 ± 3.16655.533 ± 2.538
FedDyn69.556 ± 6.94554.636 ± 2.074

SCAFFOLD improves accuracy over FedAvg by 0.791±2.205 and 0.591±0.908 pp, with improvements in mean NLL and bottom-client accuracy as well. FedDyn reaches 69.556% on Concept and 54.636% on Feature+Concept. These results show that the correction framework accommodates other federated optimizers, with SCAFFOLD further improving mean performance in both settings. The revision includes the comparison and the associated control-vector and historical-state costs.

Thank you for the detailed comments on my questions! It really helps a lot understanding the strengths and weaknesses of the paper.

As far as I know of, (pp) measures the difference in the percentage (%) of the accuracies, right? As I have concerned, the improvement in the accuracy is not meaningful for the cases of IID client composition and Label skew cases. Considering the effort to run the proposed method, I am not sure why we should use the proposed method for those cases.

Also, the number of experts seems to be dependent on the client heterogeneity, and the performance gap is quite large between the case with less experts and more experts. The budget might increase much more for extreme cases, which is a weakness of this method.

A strange thing is that the accuracy can actually decrease for more complex archictectures of the router. This is not necessarily a weakness, but is certainly a strange point for those who might actually deploy this method in practice.

I think that the proposed method is interesting, but the experimental results are not impressive enough to appear in a top venue such as ICLR. Therefore, I maintain my score as it is.

If I am wrong, do not hesitate to give me additional comments!

Reviewer_7jYtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to correct client-dependent errors after a federated global model has been selected and frozen. FedCoREx uses each client’s training data to construct a confusion-based context. A shared router combines this context with the frozen model’s sample representation and predictions to weight residual experts, whose outputs correct the base logits. The paper also analyzes the potential log-loss reduction available from sample-only correction and additional client information. Experiments show substantial gains under controlled visual concept shifts, but smaller gains on natural client datasets.

Critical strengths
  1. Learning a shared correction while preserving the deployed global model is well motivated and clearly defined.
  2. FedCoREx improves accuracy by approximately 9–17 percentage points over the same frozen FedGMT base in the Concept and Feature+Concept settings, with corresponding improvements in NLL and bottom-client accuracy.
  3. The paper uses paired comparisons with identical base checkpoints and includes controls for client context, residual composition, additional base-training rounds, and alternative upstream bases.
  4. The analysis distinguishes potential information-level improvement from the gain achieved by a learned correction.
Critical weaknesses
  1. Existing controls do not give a single residual model the same client context as FedCoREx. They establish the value of client conditioning, but do not clearly show the added value of expert routing.

  2. Gains are large under constructed concept shifts but modest on FEMNIST and near zero on Sentiment140. The correction stage adds substantial training cost, without a direct comparison to lightweight personalization methods on the same frozen base.

  3. The C^2^-dimensional confusion context may become costly and difficult to estimate with many classes or sparse local data. Its robustness to distribution shift after deployment is also unknown.

Questions for authors
  1. How does FedCoREx compare with a parameter-matched single conditional residual network receiving the same sample inputs and client context, under the same training and checkpoint-selection protocol?
  2. Under comparable total costs, how does it compare with a local linear head, lightweight adapter, or client-specific logit correction on the same frozen base?
  3. How do performance and context quality change with more classes, fewer local examples per class, or client distribution shift after deployment?
2 comments
AuthorsOpenReview ↗

The response of W1&Q1&W2&Q2

W1&Q1

To examine the role of residual parameterization, we compared single-network and expert architectures with the same information sources and closely matched parameter budgets.

(a) We added Single-Conditional, which receives the same sources of information as FedCoREx: the frozen representation, base probabilities, and client context. It uses a closely matched correction-parameter budget, the same 200-round training budget, and the same validation-based checkpoint selection. Full and Single contain 6,856 and 6,765 correction parameters, respectively. We also evaluated Context-Experts, which retains the original router and additionally supplies context directly to each expert. This variant has 7,016 correction parameters, 2.33% more than Full.

CIFAR-10 settingOriginal Full accuracy (%)Context-Experts accuracy (%)Single accuracy (%)
Concept67.916 ± 3.33673.431 ± 0.38773.693 ± 0.328
Feature+Concept54.942 ± 1.63257.551 ± 2.05058.364 ± 1.424

Context-Experts improves over Full by 5.516±2.955 and 2.609±1.145 percentage points (pp), narrowing the mean gap to Single to 0.262 and 0.813 pp. Its experts receive the frozen representation and normalized context; Single also receives base probabilities and uses its specified input normalization. Expert width is reduced from 16 to 8 to keep the budgets close. These results identify expert-side conditioning as a useful way to improve the expert architecture.

(b) On the natural datasets, Full and Single achieve similar mean accuracy: 72.885±1.798 versus 72.995±1.237% on FEMNIST, and 66.205±1.160 versus 66.219±1.159% on Sentiment140. Together, the results support shared client-conditioned correction and clarify how its parameterization affects performance. The expert bank offers a compositional implementation, and the location at which context enters the correction function matters. The revision includes the direct comparisons and input specifications to make these roles explicit.

W2&Q2

We address the performance–cost tradeoff through natural-data results and a comparison of shared, private, and combined correction for different deployment needs.

(a) The natural datasets also show positive mean accuracy effects. On FEMNIST, correction over FedGMT improves accuracy by 1.440±1.493 pp and reduces mean NLL by 0.06022. The accuracy improvements over FedAvg and FedProx are 1.701±1.005 and 3.075±1.385 pp. On Sentiment140, the accuracy change over FedGMT is +0.255±0.244 pp. The larger visual concept-shift gains are consistent with more pronounced client-dependent error patterns that the correction can exploit.

(b) Following your suggestion, we evaluated a zero-initialized private linear logit-residual head on the same frozen FedGMT base, with only 850 parameters per client, and a combination of shared and private correction. The following comparisons match 400 stages of client-training opportunities and sample access. Computation and communication depend on the respective architectures.

MethodConcept accuracy (%)Feature+Concept accuracy (%)
Shared correction: Full40068.662 ± 4.65455.044 ± 1.549
Private correction: Private40073.431 ± 0.28358.827 ± 1.372
Shared + private: Full200+Private20073.582 ± 0.41858.560 ± 1.052

The private head achieves higher accuracy in these two settings and requires no aggregation of its parameters during private adaptation. Both approaches are meaningful for different deployment needs. FedCoREx maintains one shared set of trainable correction parameters and uses local context for client-dependent predictions. Personalization instead trains and maintains client-specific weights. Shared correction suits a common-model deployment, while private adaptation suits deployments that allow dedicated client parameters and local training.

The approaches are also compatible: a private residual can be learned on top of the shared correction. The combined system reaches 73.582% and 58.560%, close in mean accuracy to private adaptation alone. We provide this comparison here to clarify the performance and cost tradeoffs between shared conditioning and client-specific adaptation.

AuthorsOpenReview ↗

The response of W3&Q3

W3&Q3

We assess the practical use of context through deployment shifts, per-class sample support, and class-dependent costs.

(a) Correction retains substantial gains under the tested post-deployment distribution shifts. We fixed the base and correction parameters and multiplied test-image brightness by 0.7 or 1.3, keeping client assignments and labels unchanged. With the original context, the paired accuracy gains over the base evaluated under the same shift are:

Shifted test viewConcept gain (pp)Feature+Concept gain (pp)
Darker, brightness ×0.7+10.573 ± 2.858+8.022 ± 1.134
Brighter, brightness ×1.3+11.111 ± 3.176+8.800 ± 1.181

Bottom-10% accuracy also improves by approximately 9.7–14.1 pp, and NLL decreases by approximately 0.23–0.28. These results show robustness of the relative correction benefit across several metrics under the two brightness shifts. Absolute accuracy decreases under the shifts, while the original context remains useful. Refreshing context from labeled training data with the same brightness transformation changes mean accuracy by −0.071 to 0.000 pp; the original context already preserves an effective conditioning signal in these settings.

(b) A small number of examples per class approaches full-context performance. Keeping the learned model fixed, we recomputed context using 1, 5, 20, or all training examples per client and class. With five examples, the accuracy differences from full context are only −0.333±0.058 and −0.391±0.359 pp for Concept and Feature+Concept; with 20, they are −0.196±0.056 and +0.036±0.095 pp. With one example, the differences are −2.809±0.712 and −2.596±1.404 pp. These settings retain support for every class and quantify the sensitivity of the smoothed context estimate to sample support.

(c) Class-dependent costs can be quantified explicitly. Context contains C2C^2 values per client, and the linear router has K(dh+C+C2)K(d_h+C+C^2) parameters. With an 84-dimensional representation and four width-16 experts, 10/50/100/200 classes require 6,856/19,176/52,576/179,376 correction parameters and 400/10,000/40,000/160,000 bytes of FP32 context storage. The actual 62-class FEMNIST experiment uses a 3,844-dimensional context and 25,368 correction parameters, approximately 26.33% of the base-model parameter count, and obtains the positive mean accuracy effects reported above. The revision presents the analytic costs, support-sensitivity results, and brightness-shift evaluation separately.

Comments on the paper

3 comments

Thank you for checking. I consider the MoE direction reasonable and potentially promising. However, I still do not think the current experimental evidence is sufficient to establish its advantage over the simpler alternatives. Addressing this concern would require more extensive experiments and analysis, which is not something that can be adequately resolved within the short rebuttal period. Therefore, I will maintain my original score.

My initial concern has not really been addressed, and I don't think it can be fully resolved without changing the proposed method. So I didn't push the discussion further.

Overall, my main concerns are as follows:

  1. Is the multi-expert architecture really necessary?

Full uses a Router + Experts architecture, but Single-Conditional achieves higher accuracy with fewer parameters. So it's unclear what the actual advantage of using multiple experts is.

  1. How do the experts learn complementary functions?

My original question was: "What is the intended mechanism that makes different experts learn complementary correction functions rather than redundant ones?" The paper doesn't really explain how experts learn different correction patterns or avoid redundancy. Also, Context-Experts still performs slightly worse than Single-Conditional, which makes this question even more relevant.

  1. Is the additional complexity worth it?

If the multi-expert architecture delivers weaker performance, the additional computational cost and inference overhead introduced by the Router and multiple Experts become important concerns.

Apologies, it seems that the authors could not see it due to the visibility setting. I have now enabled it. Please feel free to correct me if I have misunderstood anything.

AuthorsOpenReview ↗

Summary of Contributions and Discussion

Dear Area Chair and Reviewers,

Thank you for the thoughtful discussion. We would like to summarize the central contribution of FedCoREx: client-dependent prediction errors can be addressed through a shared conditional correction layer while preserving an already-trained global model.

Our contribution combines three elements:

  1. A theoretical characterization of correction opportunities. We decompose the available log-loss improvement into sample-level model misspecification and additional client-conditioned information, separating these from approximation and learning gaps. This explains why distributional heterogeneity alone does not determine the benefit of correction.

  2. A concrete mechanism for shared, client-conditioned correction. FedCoREx uses training-derived summaries of the frozen base’s prediction behavior to condition shared residual experts. All trainable neural parameters remain shared, while local context enables client-dependent predictions. This separates correction from upstream model training and allows the method to complement different federated optimizers.

  3. Evidence identifying where this approach is useful. Experiments across four datasets, multiple upstream methods, and two image backbones characterize its benefits and tradeoffs. On CIFAR-10, correction over FedGMT improves mean accuracy by 11.44 percentage points under concept shift and 9.15 points under combined feature–concept shift. The revised manuscript also includes direct architectural controls and deployment robustness analyses.

The structural comparisons clarify an important distinction: Single-Conditional performs better in the tested CIFAR-10 settings, while providing context directly to experts substantially narrows that gap. These findings strengthen the evidence for shared conditional correction and identify conditioning placement as an important architectural choice.

We hope the final assessment will consider the contribution of this combined theoretical and empirical account of what remains correctable after global federated training, and how shared models can exploit client-local evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.