- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies knowledge distillation (KD) from partially expert teachers (PETs), whose predictions are reliable on some inputs but potentially misleading on others. It develops an SGD convergence analysis that characterizes the sample-wise trade-off between one-hot and teacher supervision. The resulting bounds identify two competing factors: the intrinsic diversity of Bayes class probabilities (BCPs), which affects the cost of one-hot supervision, and teacher prediction error, which affects the cost of distillation. The paper proposes an uncertainty-guided selective KD method that uses aleatoric and epistemic uncertainty as proxies for these factors and employs a Gaussian mixture model to route each sample to either one-hot or teacher supervision. Experiments across vision, audio, and high-energy physics classification tasks demonstrate improvements over standard and adaptive KD baselines.
Critical strengths
S1. The paper addresses an important limitation of conventional KD: a teacher may be reliable only on a subset of the input space. The sample-wise decision of whether to use teacher or one-hot supervision is well motivated and practically relevant, particularly when the teacher and student training distributions differ.
S2. The SGD convergence analysis provides an interesting characterization of the competing costs of one-hot and teacher supervision. In particular, relating these costs to BCP diversity and teacher error offers a principled explanation for why different supervision sources may be preferable for different samples.
S3. The proposed method achieves consistent improvements across vision, audio, and high-energy physics classification tasks, outperforming a broad range of standard and adaptive KD baselines. The additional experiments showing compatibility with existing KD objectives further support the practical usefulness of the proposed routing mechanism.
Critical weaknesses
W1. The theoretical analysis derives a sample-wise routing criterion by comparing the costs of one-hot and teacher supervision. However, the practical method replaces these costs with the ratio of epistemic to total uncertainty, followed by a two-component Gaussian mixture model. The theoretical costs depend on several factors, including teacher bias, prediction variability, and the student's Jacobian, which are not explicitly captured by the proposed uncertainty ratio. Although the paper provides empirical evidence supporting the use of AU and EU as proxies, it remains unclear whether the resulting routing decisions reliably approximate those prescribed by the theoretical analysis.
W2. The convergence analysis relies on strong quasi-convexity and student realizability assumptions, which may not hold for practical neural networks, particularly when the student has substantially lower capacity than the teacher.
W3. The proposed method uses epistemic uncertainty to estimate teacher reliability. However, a teacher may make confidently incorrect predictions with low epistemic uncertainty, potentially leading to inappropriate routing decisions.
Questions for authors
Q1. Under what conditions does the uncertainty-based routing rule approximate the theoretically optimal routing criterion?
Q2. How does limited student capacity affect the theoretical routing criterion between teacher and one-hot supervision?
Q3. How does the proposed method handle confidently incorrect teacher predictions with low epistemic uncertainty?
2 comments
Response to Reviewer u6y8 (2/2)
(continued) As anticipated, the accuracy of all three methods degrades as the student shrinks. Our method nevertheless attains the highest accuracy in all 30 configurations, with a gain over the better of standard one-hot training and standard distillation of 4.2 to 7.1 percentage points on UrbanSound8K and 0.4 to 7.4 points on CIFAR-10. In particular, the smallest audio students, with 6.3k and 3.5k parameters, still gain 5.5 and 4.9 points. We thank the reviewer for this important question, which strengthened the results and insights in the paper.
Q3 & W3. We agree with the reviewer. As stated in our limitations section, confidently wrong predictions with low epistemic uncertainty in Bayesian NNs are a well-known challenge that may arise from the posterior approximation or from model misspecification [4,5]. Our work is not aimed at solving this challenge, which has several active research directions, including characterizing what information-theoretic EU estimators actually measure and when they fail [6] and alternative uncertainty representations such as credal sets [7].
To answer the reviewer's question, we highlight that the core contribution of the paper is a new understanding of the quantities that govern distillation for students trained by SGD, how to approximate them, and how to use them to improve the distillation process. The method is therefore based solely on AU and EU, and it does not leverage other signals on which many previous methods rely, including agreement with the one-hot labels, the student's current predictions, or a validation objective optimized in a bilevel or meta step. Although the design is intentionally kept free of these signals so that the gain can be attributed to the uncertainty signals, they carry complementary information, which motivated us to combine our approach with other methods, as shown in Figures 3 and 18. To answer the reviewer's question, such combinations can correct errors that arise, for example, from confidently incorrect teacher predictions with low EU. In addition to Figures 3 and 18, we provide an intuitive example below. Inspecting the routing decisions of our method, such predictions appeared mainly in the audio experiments. The simplified rule supported by [8] distills a sample when the teacher's top prediction agrees with the one-hot label, and does not distill it otherwise. Our method sends almost all confidently incorrect predictions with low EU to the teacher, whereas the label-agreement rule sends all of them to the one-hot label, but it makes other routing errors that our method handles correctly, as explained in the paper. We therefore tested a simple combination for these cases: when the two rules agree we use their routing, and when they disagree we set , which gave:
| Teacher → Student | Low-EU confidently wrong (% of samples) | Routed to teacher by ours (% of those) | Routed to one-hot by agreement (% of those) | Agreement (Acc [%]) | Ours (Acc [%]) | Hybrid (Acc [%]) |
|---|---|---|---|---|---|---|
| UrbanSound8K: CNN-8 → CNN-4 | 10.9 | 91.4 | 100 | 64.21 | 65.95 | 66.76 |
| UrbanSound8K: ResNet-14 → ResNet-8 | 10.3 | 99.4 | 100 | 53.72 | 57.31 | 57.72 |
| UrbanSound8K: ResNet-14 → CNN-4-N | 10.3 | 99.4 | 100 | 58.93 | 59.56 | 61.05 |
The students trained with the hybrid rule outperform those trained with either rule alone. This further demonstrates how complementary signals can correct each method's individual errors, and, as stated in the paper, we encourage practitioners to combine such mechanisms.
We hope our responses have adequately addressed the reviewer's concerns and are grateful the reviewer acknowledged the strengths of our work. We would be grateful if the reviewer would consider raising their score to a positive one.
[1] "Knowledge distillation performs partial variance reduction", NeurIPS 2023.
[2] “SGD-based knowledge distillation with Bayesian teachers: Theory and guidelines", ICLR 2026.
[3] “Handbook of convergence theorems for (stochastic) gradient methods”, ArXiv 2023.
[4] "Deep deterministic uncertainty: A new simple baseline", CVPR 2023.
[5] "On the expressiveness of approximate inference in Bayesian neural networks", NeurIPS 2020.
[6] "Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks", NeurIPS 2024.
[7] "Credal wrapper of model averaging for uncertainty estimation in classification", ICLR 2025.
[8] "Correctness-aware knowledge distillation for enhanced student learning", TMLR 2026.
Response to Reviewer u6y8 (1/2)
We thank the reviewer for the careful evaluation and the thoughtful feedback. We address the reviewer's concerns below. Table and figure numbers refer to the revised paper.
Q1 & W1. As noted by the reviewer and referenced in our limitations, while our method is derived based on the theoretical characterization (whose formulation involves quantities such as the true BCPs and loss-landscape constants μ, L, and L and requires AS1-AS3), the practical method (that has no access to these quantities) may not share these theoretical guarantees. The theoretical results are intended to provide insights on the inner-working mechanisms of KD, what affects it, and to provide support and guidelines for practical method design. On top of the strengths acknowledged by the reviewer which support the use of the method, we can show the approximations of the uncertainty-based routing rule and the theoretically optimal routing criterion are aligned in various scenarios. Following the reviewer’s suggestion, we added the new Subsection E.5 in the Appendix which explores this correspondence in the 9 settings of Appendix E. We include the main results here:
The practical and theoretical rules agree on 88% of the samples on average across the 9 test cases, the practical score r(x) predicts the theoretical routing with an average AUROC of 0.94, and students trained with the practical rule are within 1.0 point of those trained with the theoretical rule. Additional results, comparisons to solely EU-based and AU-based routers, and visualization of the correspondence are presented in the newly added Appendix E.5. The results further strengthen the connection between the theoretical result and the practical method. We thank the reviewer for this important suggestion.
Q2 & W2. As the reviewer noted, and as stated in our limitations, our theoretical analysis relies on assumptions which may not hold in practice, but are typically used in works that analyze convergence in KD [1,2,3]. Quasi-convexity and smoothness assumptions are used to maintain a tractable convergence characterization, and we refer the reviewer to the "Assumptions" paragraph in Section 3.1 which explains where the smoothness assumption may come from for typical architectures via bounded Jacobians. The realizability assumption, as stated in Section 3.1, is a deliberate modelling choice which abstracts away student capacity so that the analysis can characterize the sample-wise costs through the BCPs rather than through the student's in-class optimum. From a theoretical perspective, incorporating limited student capacity into the characterization in the paper requires re-deriving the proofs under different modelling choices, which would substantially change the paper. We therefore leave it for future work.
To answer the reviewer's question empirically, we ran additional experiments that examine how limited student capacity affects our method relative to standard one-hot training and standard KD. We fix the teacher and scale the width of the student for four teacher-student pairs across two modalities, for a total of 30 student configurations ranging from 3.5k to 25M parameters. The full sweeps are shown in Figure 19 at the end of Subsection G.1 of the revised paper, together with a discussion. The table below reports student test accuracy at the largest, nominal, and smallest student of each pair.