acceptodds
Under review as a conference paper at ICLR 2027

Label-Efficient Residual Correction for Mixture-of-Experts via Their Routed Readouts

Abstract

Pretrained mixture-of-experts (MoE) predictors are increasingly reused on heterogeneous deployment data. In such settings, the model's error, whether caused by distribution shift or by limited training, can differ across the input regions served by different experts, yet retraining is often infeasible due to restricted access to data or model parameters. Existing post-hoc adaptation methods typically treat the model as a black-box predictor, ignoring the routing weights and expert outputs that form each prediction. We propose ATF-kNN (Affine Teacher Features with kNN residual imputation), a post-hoc adaptation method for a frozen MoE teacher, which leverages the routing probabilities and expert-specific outputs exposed at inference time. Our approach fits one offset and one gain per expert in one step, gated by the model's own routing probabilities, and imputes residuals for unlabeled deployment data via k-nearest neighbors in this readout space. We characterize the population gain over scalar teacher correction and prove a finite-sample risk certificate under regularity conditions. Experiments on real and synthetic data show consistent improvements over the frozen teacher under both limited training and distribution shift, robustness to a misspecified number of experts and over a range of imputation hyperparameters, and, given enough labels, gains over scalar correction when the teacher's residual depends on its routing.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Mixture-of-Experts, Post-hoc Adaptation, Semi-supervised Learning, Finite-sample Error Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uMUbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ATF-kNN, a post-hoc adaptation method for frozen mixture-of-experts (MoE) regression models. It uses exposed routing probabilities and expert outputs to construct a 2K−1-dimensional feature representation, imputes residuals for unlabeled deployment examples via k-nearest neighbors in this representation, and fits a weighted ridge correction with only 2K coefficients. The paper provides a population projection analysis and a finite-sample risk certificate under regularity and imputation assumptions. Experiments on synthetic benchmarks and several real datasets compare the method with scalar correction, semi-supervised baselines, and fine-tuning. Results are generally favorable, especially in scarce-label settings, although the incremental benefit of kNN imputation varies across datasets.

Critical strengths
  1. The method exploits MoE routing and expert-level readouts rather than treating the teacher as a scalar black box. The resulting correction is lightweight, interpretable, and requires no access to model weights.
  2. The paper includes a useful theoretical decomposition of the population gain over scalar correction, plus broad ablations and comparisons across label budgets, expert-count misspecification, and distribution shifts.
Critical weaknesses
  1. The empirical evidence is promising but still limited in external validity. Several results rely on synthetic or deliberately corrupted teachers, while the real-data evaluation covers a small number of regression settings. It is unclear how well the method transfers to large, sparse, modern neural MoEs or less structured deployment shifts.
  2. The practical benefit of the kNN-imputation component is inconsistent: ATF-no-kNN matches or outperforms the full method in some reported settings, and the gain is small on Diamonds. The paper would benefit from a clearer characterization of when unlabeled residual imputation helps, along with stronger evidence that the default unlabeled weight is robust across datasets.
Questions for authors
  1. Can the authors provide a more systematic analysis of when kNN residual imputation improves over ATF-no-kNN, particularly when the labeled sample is very small or the readout features are poorly conditioned?
  2. How sensitive are the results to the default unlabeled weight w_u=0.5 and to the geometry of the standardized readout space under realistic deployment shifts? Can the authors report evidence beyond the current synthetic sensitivity study?
Reviewer_NBtVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies post-hoc adaptation of pretrained MoE models. The authors argue that existing prediction-powered and semi-supervised adaptation methods largely treat the teacher model as a black-box predictor and therefore ignore information exposed by the MoE architecture. To address this, they propose ATF-KNN (Affine Teacher Features with kNN residual imputation), a lightweight adaptation method that keeps the teacher frozen, constructs features from routing probabilities and expert readouts, imputes residuals using kNN, and fits a weighted ridge correction.

Critical strengths
  • ATF-KNN consistently performs well across several synthetic and semi-synthetic benchmarks, including various settings such as corrupted experts, misspecified numbers of experts, deployment shift.
  • The central idea is simple and intuitive: if different experts exhibit different systematic errors, adaptation should exploit expert-level information rather than only the final scalar prediction.
  • Lightweight and does not require fine-tuning.
  • Experimental results are strong.
Critical weaknesses
  • The paper would be strengthened by more interpretation and discussion of the observed behaviors. For example, one of the central findings is that MoE-specific signals provide adaptation benefits beyond scalar predictions, but more discussion of the quantitative results would help readers understand when and why these gains arise.
  • It is unclear whether the synthetic gains reflect a real-world phenomenon or primarily the benchmark construction. Additional discussion of how the corruption settings relate to realistic expert-level errors would strengthen the paper.
  • The paper clearly demonstrates settings where ATF-KNN performs well, but it is less clear when the method should be expected to fail.
    • For example, what happens under severe routing failures, or highly nonlinear residuals?
    • More systematic discussion of such failure modes would help clarify the practical applicability of the method and better separate the contribution of the affine correction from benchmark-specific settings.
  • I would expect comparison against the work of Cowen-Bree et al. (ICLR 2026), given the similar PPI/SSL-style setup.
  • Instead of section 4, I would focus on interpreting/discussing the results and proposing insights about each component of proposed method. I do not find the theoretical analysis particularly compelling.

Minor Comments

  • While terms such as finite-sample risk certificate and routed residual directions are valid, they often obscure relatively simple underlying ideas and make the paper harder to follow.
  • Although kNN-MoE is currently only available on arXiv, it appears sufficiently related that it may be worth discussing in the positioning section.
Questions for authors

Most of my questions are reflected in the weaknesses above.

1 comment

Thank you for your response and clarification. While the answers for clarification are appreciated, I view the gains as incremental after reviewing the rebuttal/comments from the other reviewers. I still consider this paper to be borderline and will maintain my current score.

Reviewer_Q2Y7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to correct a frozen mixture-of-experts model using a small number of labels from the deployment distribution. ATF-kNN uses the model's routing probabilities and expert outputs instead of relying only on its final prediction. It estimates residuals for unlabeled examples from nearby labeled examples in this feature space, then fits a weighted ridge correction. The result is one offset and one gain per expert, with no changes to the teacher's parameters. The authors describe when these features can improve on scalar correction and give a finite-sample risk bound under stated assumptions. Experiments include real data, controlled shifts, and teacher-independent synthetic tasks. The method often improves on the frozen teacher and is competitive with fine-tuning, although the extra benefit of residual imputation varies across settings.

Critical strengths
  1. Using the routing weights and expert outputs is a sensible way to find errors that are hidden in the final prediction. The correction only fits one offset and one gain per expert, so it is useful when the original model cannot be retrained.

  2. The experiments include useful controls and repeated runs. The authors test a real temporal shift, different label budgets, and synthetic shifts defined before teacher training. The no-kNN and readout-permutation experiments help separate the value of the features from the rest of the method (Tables 1–2; Appendix C).

  3. The analysis makes the trade-off clear: richer features may capture more residual signal, but estimation and imputation errors can remove this advantage. The population and finite-sample results help explain why scalar correction may work better when labels are very limited (Section 4).

Critical weaknesses
  1. The methodological contribution appears incremental. The routed feature design is sensible, and the paper provides useful analysis of the resulting correction. However, residual imputation, kNN, and ridge regression are established tools, so the main advance lies in how they are combined for frozen MoE models. The analysis helps explain this design, but I find the overall methodological advance modest.

  2. The benefit of kNN imputation varies across settings, as the authors acknowledge. It helps at very small label budgets on Bike-Sharing, but the no-kNN version is statistically tied and slightly better on average on Electricity, and the difference is small on Diamonds. These results support the value of the routed features more consistently than the need for the additional imputation step.

  3. The experiments use MoE models with relatively few experts, including an expert-count sweep from two to five. The paper already explains how feature dimension and conditioning affect estimation, and it discusses scaling to larger MoEs. Still, the reported few-label gains have not been established for models with substantially more experts, where the correction requires more coefficients. This limits the empirical scope of the results.

Questions for authors
  1. The paper already compares MoE-adapted PP-SSL and SADA. Could you also clarify how ATF-kNN relates to the imputation-and-refitting approach of Chakrabortty and Cai (2018), beyond the choice of features? Would a comparison using the same routed features help separate the effect of the estimator from that of the representation?

  2. The paper already uses labeled-set cross-validation to tune hyperparameters. Could the same procedure also select between ATF-kNN and ATF-NO-kNN? How would this compare with always using either version, especially at the smallest label budgets?

  3. Could you extend the expert-count experiment to larger values of K while keeping the label budget fixed? Reporting feature conditioning alongside prediction error would help connect the observed performance to the dimension and conditioning terms in the existing analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.