- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies knowledge distillation from multiple teacher models, while focusing on the metric of OOD detection. Under the papers setting, the teachers provide complementary expertise to the student. The paper's main motivation is the observation that current multi KD schemes usually omit comparison to standard one-hot training (i.e. not distilling at all), and that when this comparison is added, OOD detection performance of the methods often degrades compared to standard one-hot training. To address this issue, the authors propose the method Shapley-Anchored Residual Routing (SRR) which keeps a fixed teacher-weighting prior and trains a lightweight router, jointly with the student, which predicts a per-sample correction to that prior. They also propose reusing the router’s per-sample mixing coefficient as an OOD score, by fitting a small probe on frozen features. Their experimental setting is performed on CIFAR10 with two teacher-student architecture pairs (Res18 to Res34 and the same Res18 teachers to MobileNetV2). They then evaluate the resulting student's OOD detection capabilities compared to test sets from SVHN, CIFAR-100, TinyImageNet, and DTD. their method improves on the other distillation methods and roughly matches, but does not exceed, the one-hot supervised student (no distillation) trained for the same number of epochs. Their probe detects OOD as well as a standard nearest-neighbour detector, without storing the training-set features.
Critical strengths
- The authors compare students trained in multi-teacher KD settings against standard one-hot training, which is the simplest and most obvious baseline that other multi-teacher KD papers usually omit. They show that existing methods consistently degrade OOD detection relative to standard one-hot training. This finding is useful for the community regardless of the proposed method.
Critical weaknesses
-
The main weakness of the paper is limited contribution. The main finding is that existing multi-teacher distillation methods make OOD detection worse than simply training the student on the labels. However, the proposed method only brings distillation back to the performance level one-hot training. For two students trained with an equal budget, one trained with SRR from multiple teachers, and one trained from one-hot labels, the resulting student performance in terms of energy AUROC is the same, and in terms of FPR@95 the SRR student is worse (the 0.511 vs 0.429 result). Moreover, the experimental evidence is limited. Results are shown solely on CIFAR10, and only 2 teacher-student pairs (with a student that is larger than its teachers), and in the MobileNetV2 student setting SRR doesn't beat standard training from one-hot labels. Also, while the authors state that the proposed probe matches existing detector performance but needs significantly less storage, it requires a full routed run to produce its targets. So it isn't really cheaper, it's actually a cheaper way to reach existing performance, under conditions that make it expensive to obtain. It is not a better OOD detector. Overall, it is unclear why a practitioner would choose to do multi-teacher KD instead of standard supervised training from the labels.
-
The paper is very difficult to follow. This is mainly because it never explains what the method does and why it should work. Key concepts are used before they are defined, and the central idea, that a static prior fails because it ignores the student’s “learning state” and that routing fixes this “mismatch”, is stated repeatedly but never actually shown. It remains unclear what the method actually corrects or why that should improve OOD detection. In Section 5, the method is presented as a collection of loss terms, schedules and constants with little or no motivation for why each of them is needed. The results are spread across the paper and appendices under changing metrics, and the conclusions are often given as short slogans rather than supported by evidence. This makes the reader have to reconstruct the paper’s argument on their own, which made it hard for me to judge whether the conclusions are actually supported by the experiments.
Questions for authors
Overall, the main issues that need to be addressed for the paper to be considered for acceptance are:
-
The paper needs to be substantially rewritten so that it explains what the method does and why it should work, defines concepts before using them, and reports results under consistent metrics.
-
The proposed method needs show meaningful improvement over standard one-hot training (no distillation). Showing improved results compared to other multi-teacher KD methods is not enough if standard one-hot training is better to begin with.
-
Additional evidence is needed, results should be shown on additional in-distribution datasets (e.g. CIFAR100, ImageNet, WILDS), as well as with additional teacher–student pairs, including teachers that are larger than the students.
Additional questions:
-
Can you report a Res34 student run of SRR with the router disabled (fixed r) but everything else kept (the anchor, initialization, and additional epochs)? Currently in the paper the router and anchor are removed together, so it is unclear if the gain can be attributed to the routing.
-
What is the “student’s learning state” and the “student–prior mismatch” that SRR fixes? can it be measured?
-
Why are there missing values in Table 1? Can you report them?