- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ATF-kNN, a post-hoc adaptation method for frozen mixture-of-experts (MoE) regression models. It uses exposed routing probabilities and expert outputs to construct a 2K−1-dimensional feature representation, imputes residuals for unlabeled deployment examples via k-nearest neighbors in this representation, and fits a weighted ridge correction with only 2K coefficients. The paper provides a population projection analysis and a finite-sample risk certificate under regularity and imputation assumptions. Experiments on synthetic benchmarks and several real datasets compare the method with scalar correction, semi-supervised baselines, and fine-tuning. Results are generally favorable, especially in scarce-label settings, although the incremental benefit of kNN imputation varies across datasets.
Critical strengths
- The method exploits MoE routing and expert-level readouts rather than treating the teacher as a scalar black box. The resulting correction is lightweight, interpretable, and requires no access to model weights.
- The paper includes a useful theoretical decomposition of the population gain over scalar correction, plus broad ablations and comparisons across label budgets, expert-count misspecification, and distribution shifts.
Critical weaknesses
- The empirical evidence is promising but still limited in external validity. Several results rely on synthetic or deliberately corrupted teachers, while the real-data evaluation covers a small number of regression settings. It is unclear how well the method transfers to large, sparse, modern neural MoEs or less structured deployment shifts.
- The practical benefit of the kNN-imputation component is inconsistent: ATF-no-kNN matches or outperforms the full method in some reported settings, and the gain is small on Diamonds. The paper would benefit from a clearer characterization of when unlabeled residual imputation helps, along with stronger evidence that the default unlabeled weight is robust across datasets.
Questions for authors
- Can the authors provide a more systematic analysis of when kNN residual imputation improves over ATF-no-kNN, particularly when the labeled sample is very small or the readout features are poorly conditioned?
- How sensitive are the results to the default unlabeled weight w_u=0.5 and to the geometry of the standardized readout space under realistic deployment shifts? Can the authors report evidence beyond the current synthetic sensitivity study?
Thank you for your response and clarification. While the answers for clarification are appreciated, I view the gains as incremental after reviewing the rebuttal/comments from the other reviewers. I still consider this paper to be borderline and will maintain my current score.