acceptodds
Under review as a conference paper at ICLR 2027

Residual Routing over Shapley Priors: Student-Adaptive Multi-Teacher Distillation under Distribution Shift

Abstract

Multi-teacher knowledge distillation (MKD) promises to transfer the complementary expertise of specialised models to a compact student and, with it, better out-of-distribution (OOD) detection. We report an overlooked failure mode of this promise: measured against the label-only supervised student that MKD studies usually omit, static per-sample teacher weighting, whether derived from exact Shapley attribution or from confidence and entropy priors, does not improve the student's OOD detection and in most cases degrades it. The cause is not imperfect expert identification but the rigidity of a static allocation that cannot adapt to the student's learning state. We propose Shapley-Anchored Residual Routing (SRR), which keeps a fixed teacher prior and trains a lightweight router, jointly with the student, to predict a per-sample residual gate and a mixing coefficient, using no outlier data for training or checkpoint selection. Across OOD benchmarks SRR raises mean energy-score AUROC by ( points) over uniform distillation and by points over the supervised baseline, with routing gains of to points on label-conditioned and label-free priors alike and the feature-space readout left intact. The routing coefficient is also a deployable dividend: a lightweight probe that regresses it from frozen features, even of an untouched supervised student, matches feature-space -NN detection (– above the energy score) with about less storage and no per-query feature-bank retrieval.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multi-teacher knowledge distillation, Out-of-distribution detection, Shapley values, Residual routing, Adaptive teacher weighting.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Z4WCOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies knowledge distillation from multiple teacher models, while focusing on the metric of OOD detection. Under the papers setting, the teachers provide complementary expertise to the student. The paper's main motivation is the observation that current multi KD schemes usually omit comparison to standard one-hot training (i.e. not distilling at all), and that when this comparison is added, OOD detection performance of the methods often degrades compared to standard one-hot training. To address this issue, the authors propose the method Shapley-Anchored Residual Routing (SRR) which keeps a fixed teacher-weighting prior and trains a lightweight router, jointly with the student, which predicts a per-sample correction to that prior. They also propose reusing the router’s per-sample mixing coefficient as an OOD score, by fitting a small probe on frozen features. Their experimental setting is performed on CIFAR10 with two teacher-student architecture pairs (Res18 to Res34 and the same Res18 teachers to MobileNetV2). They then evaluate the resulting student's OOD detection capabilities compared to test sets from SVHN, CIFAR-100, TinyImageNet, and DTD. their method improves on the other distillation methods and roughly matches, but does not exceed, the one-hot supervised student (no distillation) trained for the same number of epochs. Their probe detects OOD as well as a standard nearest-neighbour detector, without storing the training-set features.

Critical strengths
  1. The authors compare students trained in multi-teacher KD settings against standard one-hot training, which is the simplest and most obvious baseline that other multi-teacher KD papers usually omit. They show that existing methods consistently degrade OOD detection relative to standard one-hot training. This finding is useful for the community regardless of the proposed method.
Critical weaknesses
  1. The main weakness of the paper is limited contribution. The main finding is that existing multi-teacher distillation methods make OOD detection worse than simply training the student on the labels. However, the proposed method only brings distillation back to the performance level one-hot training. For two students trained with an equal budget, one trained with SRR from multiple teachers, and one trained from one-hot labels, the resulting student performance in terms of energy AUROC is the same, and in terms of FPR@95 the SRR student is worse (the 0.511 vs 0.429 result). Moreover, the experimental evidence is limited. Results are shown solely on CIFAR10, and only 2 teacher-student pairs (with a student that is larger than its teachers), and in the MobileNetV2 student setting SRR doesn't beat standard training from one-hot labels. Also, while the authors state that the proposed probe matches existing detector performance but needs significantly less storage, it requires a full routed run to produce its targets. So it isn't really cheaper, it's actually a cheaper way to reach existing performance, under conditions that make it expensive to obtain. It is not a better OOD detector. Overall, it is unclear why a practitioner would choose to do multi-teacher KD instead of standard supervised training from the labels.

  2. The paper is very difficult to follow. This is mainly because it never explains what the method does and why it should work. Key concepts are used before they are defined, and the central idea, that a static prior fails because it ignores the student’s “learning state” and that routing fixes this “mismatch”, is stated repeatedly but never actually shown. It remains unclear what the method actually corrects or why that should improve OOD detection. In Section 5, the method is presented as a collection of loss terms, schedules and constants with little or no motivation for why each of them is needed. The results are spread across the paper and appendices under changing metrics, and the conclusions are often given as short slogans rather than supported by evidence. This makes the reader have to reconstruct the paper’s argument on their own, which made it hard for me to judge whether the conclusions are actually supported by the experiments.

Questions for authors

Overall, the main issues that need to be addressed for the paper to be considered for acceptance are:

  1. The paper needs to be substantially rewritten so that it explains what the method does and why it should work, defines concepts before using them, and reports results under consistent metrics.

  2. The proposed method needs show meaningful improvement over standard one-hot training (no distillation). Showing improved results compared to other multi-teacher KD methods is not enough if standard one-hot training is better to begin with.

  3. Additional evidence is needed, results should be shown on additional in-distribution datasets (e.g. CIFAR100, ImageNet, WILDS), as well as with additional teacher–student pairs, including teachers that are larger than the students.

Additional questions:

  1. Can you report a Res34 student run of SRR with the router disabled (fixed r) but everything else kept (the anchor, initialization, and additional epochs)? Currently in the paper the router and anchor are removed together, so it is unclear if the gain can be attributed to the routing.

  2. What is the “student’s learning state” and the “student–prior mismatch” that SRR fixes? can it be measured?

  3. Why are there missing values in Table 1? Can you report them?

Reviewer_Vs1LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates multi-teacher knowledge distillation for out-of-distribution (OOD) detection. The authors observe that static teacher weighting strategies often fail to outperform a label-only supervised student, despite improving over uniform distillation. To address this issue, they propose Shapley-Anchored Residual Routing (SRR), which combines a fixed teacher prior with a student-adaptive routing mechanism to dynamically adjust teacher weights. Additionally, the learned routing coefficient is used to construct a lightweight OOD detector without requiring a feature bank. Experimental results show that SRR improves OOD detection over conventional distillation methods, while the routing-head probe achieves performance comparable to feature-space k-NN with substantially lower storage requirements.

Critical strengths
  1. The paper investigates an interesting and meaningful problem in multi-teacher knowledge distillation. It highlights that existing teacher weighting strategies, despite outperforming uniform distillation, may still perform worse than a label-only supervised student in OOD detection. This observation provides a valuable perspective on the limitations of conventional multi-teacher distillation methods.

  2. The proposed residual routing framework is intuitive and well-motivated. By combining a fixed teacher prior with a student-adaptive correction, the method allows teacher weights to be adjusted according to the student's learning state while preserving the information provided by the original prior.

  3. The routing-head probe provides an interesting extension with practical value. By using the learned routing coefficient as an OOD detection signal, the proposed probe achieves performance comparable to feature-space k-NN while requiring substantially less storage and avoiding feature-bank retrieval.

Critical weaknesses
  1. The methodological novelty is somewhat limited. The proposed residual routing framework mainly combines a fixed teacher prior with a learnable student-dependent gate and a mixing coefficient. These components are relatively conventional, and their integration does not introduce substantial methodological innovations.

  2. The experimental evaluation is relatively limited. The main experiments are conducted on CIFAR-10 with a controlled pool of class-specialist teachers, which may not fully reflect more realistic multi-teacher settings. Moreover, the proposed method fails to outperform supervised training on MobileNetV2, raising concerns about its generalizability across student architectures and model capacities.

  3. The ablation study does not sufficiently isolate the contributions of the key components. Although Appendix J provides some ablation results, these experiments are conducted on MobileNetV2, where the routing coefficient saturates and the proposed adaptive mechanism becomes ineffective. Ablations on the main ResNet-34 setting are needed to clarify the individual contributions of the anchor loss, prior supervision, and routing regularization.

  4. Although the probe achieves competitive OOD detection performance with low storage overhead, obtaining its training targets still requires a separately trained SRR model. A more complete analysis of the overall training cost would help assess its practical efficiency.

Questions for authors

Please see the Weaknesses.

Reviewer_wxgnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies multi-teacher knowledge distillation (MKD) for out-of-distribution (OOD) detection with class-specialist teachers, and it measures every method against the label-only student of the same architecture.

Critical strengths
  • There are comprehensive controls in experiments: The checkpoint-selection protocol, gate geometry, the specialisation dial, three priors, probe controls (own-MSP target, noise target, frozen supervised student), teacher-ensemble scores (Table 8) and a latency benchmark with mean ± sd over repeats
  • On the supervised ResNet-34, a 0.26 MB head reaches the AUROC of k-NN with a 92 MB bank. If this generalises, it is useful beyond distillation.
Critical weaknesses
  • "Matches the supervised student" rests on one metric and no equivalence test. In Table 1 the 100-epoch supervised student has FPR@95 0.429 (bold); every distillation row has 0.511–0.542. SRR's 0.511 is 8.2 points worse. The abstract and introduction nonetheless say that SRR "matches" the supervised student and has "the lowest FPR@95 of all distillation variants"
  • The routing gain is not attributed to per-sample, student-adaptive routing. Every routed student is initialised from the converged static student, trained for 30 + 50 more epochs, and tied by L_anchor to a frozen copy of that student, which is a self-distillation term. The same 50 extra epochs give the supervised student +0.9, close to the +1.0 and +1.1 routing gains, and Table 2 has no intervals. The only router-free control is a Shapley-gate continuation on a two-set mean (0.867 vs 0.873). Its description does not say whether L_anchor was kept, and there is no such control for the LogP and entropy priors.
  • Eq. (2) follows from the definition of gap for any pair of scores; I re-derived it for every row of Table 1 (to within 0.01). It describes a mechanism but does not test one. That the head fails to "read" the features is an interpretation that no experiment isolates. Refitting the linear head of the static student on frozen features, or applying the cited ReAct or ASH to its energy score, would test it.
Questions for authors
  • What are the per-set FPR@95 values for the supervised student and SRR? Why is SRR 8.2 points worse at the high-TPR operating point while its AUROC is higher? Will "matches" be restricted to energy AUROC?
  • On ResNet-34/cov3 and all three priors, how do r ≡ 0 with L_anchor, r ≡ 0.2 with the learned candidate gate, and full SRR compare under the same schedule? Was L_anchor kept in the Shapley-gate continuation control? Is r(x) detached inside L_anchor, and how does r(x) correlate with KL(p_a ‖ p_s) per sample?
  • Can Table 7 include the supervised student? Does "no static gate beats the supervised student" hold under MSP, GEN and Mahalanobis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.