- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes FedCoREx, a post-hoc federated correction framework that learns a shared residual correction on top of a frozen base predictor. Each client constructs a confusion-based context from its local training data, which is combined with sample representations and base predictions to route inputs through shared residual experts. The experts and router are jointly trained with federated averaging, while validation selects the correction checkpoint, including a zero-correction option. The method is evaluated across multiple federated learning settings to assess its effectiveness in correcting client-dependent prediction errors.
Critical strengths
-
The base predictor is frozen throughout correction training, and zero-output initialization makes the initial correction exactly equivalent to the original predictor. Moreover, validation explicitly includes the zero-correction candidate, providing a principled way to avoid deploying a harmful correction.
-
Rather than learning independent client-specific correction models, FedCoREx uses globally shared residual experts while introducing client dependence through a locally constructed confusion context. This provides a compact mechanism for adapting the same correction functions to different client-level error patterns.
Critical weaknesses
-
All experts are globally shared and jointly optimized with the router using the same corrected-prediction cross-entropy objective, without an explicit diversity or specialization constraint. Since all experts are evaluated for every sample and their outputs are combined through a dense mixture, it is unclear whether the learned experts develop complementary correction functions or become partially redundant. This matters directly to the necessity of the proposed expert-bank architecture.
-
FedCoREx introduces a bank of shared residual networks, a context-conditioned router, and an additional client-side context-construction procedure on top of the frozen base. Although the paper provides parameter, computation, and communication accounting, the methodological question remains whether this additional complexity is necessary relative to simpler correction mechanisms. In particular, the paper would benefit from a clearer assessment of how the performance gain changes with the number of experts and how this gain should be weighed against the additional training and inference cost of evaluating all experts.
Questions for authors
-
All experts share the same architecture and are jointly optimized with the same corrected-prediction cross-entropy objective, without an explicit specialization mechanism. What is the intended mechanism that makes different experts learn complementary correction functions rather than redundant ones?
-
The method evaluates all residual experts for every input and then performs dense mixture through the router. Thus, both training and inference costs increase with the number of experts on top of the frozen base model. What motivates this dense expert formulation over a simpler single residual function, and how should the additional computation be viewed relative to the intended correction benefit?
-
The context is constructed once from the frozen base's predictions on local labeled training data and then kept fixed throughout correction training and inference. What assumptions justify treating this historical confusion profile as a stable conditioning signal for subsequent residual correction?
-
The framework assumes that heterogeneous client errors can be represented by a common bank of residual functions whose mixtures vary across clients and samples. What properties of the client heterogeneity make this decomposition appropriate?
4 comments
The response of Q3&Q4
Q3
Context summarizes the frozen base's class-conditional prediction behavior on each client's training distribution. With the base fixed, an approximately stable client distribution, and training samples representative of deployment data, this summary provides a consistent conditioning signal.
We additionally tested distribution changes by multiplying image brightness by 0.7 or 1.3. Keeping the learned parameters and original context fixed, correction improves accuracy over the base under the same transformation by approximately 8.0–11.1 pp on Concept/Feature+Concept, with corresponding improvements in bottom-client accuracy and NLL. Thus, the historical context retains correction value under the tested brightness shifts. Refreshing it from labeled training data with the same transformation changes mean accuracy by −0.071 to 0.000 pp; the original context already retains useful information in these conditions. The revision clarifies the fixed-context assumptions and includes this deployment diagnostic.
Q4
The decomposition is appropriate when the frozen representation retains useful predictive information, client errors contain reusable correction patterns, and context together with sample state can distinguish the required corrections. Repeated label-mapping differences over shared visual representations provide one such example. Our analysis separates conditional-information headroom under log-loss from the approximation and learning gaps of a finite residual family. The permutation-style model explains context identifiability through a shared prediction channel, positive class support, and distinguishable channel rows.
The cross-setting results are consistent with this motivation. With 50 clients, K=2/4/8 gives approximately 74.92/74.88/74.86% accuracy under IID, 65.37/67.92/71.10% under Concept, and 52.56/54.94/56.24% under Feature+Concept. With 100 clients, K=4 improves over its own base by 12.413±3.373 and 9.022±2.069 pp in the two concept-shift settings. FEMNIST also shows positive mean accuracy effects across three bases. The revision connects these results to the reusable-error-pattern interpretation and states the corresponding theoretical conditions explicitly.
Thank you to the authors for addressing my questions and providing additional experiments. I also apologize for my late reply.
After reviewing the rebuttal and the additional experiments from the other reviewers, I still have some reservations about the MoE design. The comparison with Single-Conditional shows that a simpler conditional network can achieve comparable or even better performance than the original Full model under a similar parameter budget. While Context-Experts narrows this gap, it does not fully establish the advantage of using multiple experts. In addition, although the authors provide explanations of expert optimization and routing statistics, stronger evidence is still needed to demonstrate that the experts learn distinct and complementary correction functions.
I appreciate the authors' efforts to address these concerns and have adjusted my score to 2 accordingly. However, I believe the motivation for the MoE design and the role of expert specialization could be further clarified.
Thank you for reviewing our responses and additional experiments, and for reassessing our work. Your comments have helped us identify the shortcomings of the current design more clearly: the original MoE architecture makes limited use of conditional information, and the evidence for distinct, complementary correction functions across experts remains insufficient. We accept this criticism and will use it to guide further development and evaluation.
Our starting question was whether a strong federated global model can still leave client-distribution-dependent prediction errors that can be corrected through parameters shared across clients. Our experiments provide empirical support for this idea. We also see potential value in practical settings such as newly arriving clients: local support data could provide the context needed to adapt a shared correction module to a client's distribution, without maintaining a separate trainable model for every client.
We chose MoE with the intuition that different experts could learn different error patterns across heterogeneous clients and develop complementary correction capabilities. This intuition motivated the architecture, but your feedback has made clear the need to test more directly whether such specialization actually emerges and provides additional benefits. We initially prioritized a unified shared-correction mechanism and sought to avoid designing separate architectures and contexts for different types of heterogeneity. Consequently, we did not explore alternative parameterizations and context representations thoroughly enough, leaving some weaknesses of the implementation unidentified.
Our recent exploratory results suggest that, in some of the conditions examined, changes to the correction location and context representation can improve upon the original MoE and Single-Conditional configurations. These observations encourage us to investigate which information reveals the correctable errors left by a strong global model, and how shared learning can use that information effectively. Our longer-term goal is to develop a well-grounded correction method that responds to client heterogeneity, rather than to identify a universally optimal residual architecture.
Thank you again for your careful and concrete feedback. It has helped us understand the limitations of the current work and identify a more focused direction for improving it.
The response of W1&Q1&W2&Q2
W1&Q1
Independent hidden projections and input-dependent routing give the experts different learning signals.
(a) Let p be the corrected prediction, ey the one-hot label, and g=p−ey. With residual r=∑kqkΔk, expert parameters θk, and pre-softmax routing scores sk, the gradients are
=q_k\left(\frac{\partial\Delta_k}{\partial\theta_k}\right)^\top g, \qquad \frac{\partial\ell}{\partial s_k} =q_k\,g^\top(\Delta_k-r).$$ Each expert receives an update modulated by its routing weight and its own Jacobian. The router adjusts scores according to the loss contribution of each expert relative to the mixture. At initialization, the output projections, residual, and task gradient to the router are zero. Independently initialized hidden projections produce different activations, so the expert output projections receive different initial updates; the router and experts subsequently learn jointly. This explains how different functions can emerge under a shared objective. (b) The observed routing is selective: the original Full has mean per-example effective expert counts of approximately 1.404/1.244 on Concept/Feature+Concept, compared with 1.809/1.435 for the MLP4 router. These statistics describe how inputs use the expert bank; the prediction effects are evaluated through the structural controls. The revision explains the gradients and initialization and uses these concrete observations to describe expert behavior. **W2&Q2** We use direct architectural controls and expert-budget comparisons to explain the motivation for expert composition and its performance–cost tradeoffs. (a) The design represents reusable corrections through shared experts, with sample state and client context determining their composition. The base executes once, the lightweight experts operate on its frozen representation, and dense soft mixing supports joint training. We also evaluated Single-Conditional as a simpler alternative with the same information sources and a closely matched parameter count. On the original CIFAR-10 backbone, Full reaches 67.916±3.336 / 54.942±1.632% on Concept/Feature+Concept, versus 73.693±0.328 / 58.364±1.424% for Single. Full/Single results are 72.885±1.798 / 72.995±1.237% on FEMNIST and 66.205±1.160 / 66.219±1.159% on Sentiment140. Both implement shared conditional residual learning; Single achieves higher accuracy in the tested CIFAR-10 settings, while their mean accuracy is close on the natural tasks. The revision includes this direct comparison to clarify the motivation and observed performance of expert composition. (b) Within the expert architecture, we compared K=2/4/8 at approximately 6.85k total correction parameters, with expert widths 34/16/7. For 50 clients, Concept accuracy is 65.373±0.192 / 67.916±3.336 / 71.098±2.134%, and Feature+Concept accuracy is 52.556±0.812 / 54.942±1.632 / 56.244±1.587%. Allocating the near-matched budget to more, narrower experts improves mean accuracy in these two settings. Increasing K at a fixed expert width increases computation; the budget-matched study instead reduces expert width as K increases. The revision distinguishes these comparisons and presents them alongside the parameter, training, communication, and inference accounting. All experts are evaluated, and routing concentration describes their mixing weights.