acceptodds
Under review as a conference paper at ICLR 2027

Selective Knowledge Distillation from Partially Expert Teachers: An SGD Analysis

Abstract

Knowledge Distillation (KD) typically relies on a teacher whose predictions are assumed to provide useful supervision throughout the student's training data. In practice, pretrained teachers can be reliable on some inputs while being misleading on others. We study Stochastic Gradient Descent (SGD)-based KD from such partially expert teachers and ask, on a sample-by-sample basis, when the student should learn from the observed one-hot label and when it should learn from the teacher. We derive convergence bounds whose asymptotic error decomposes into competing sample-wise costs associated with the two supervision sources. Our analysis shows that the cost of one-hot supervision is governed by the intrinsic diversity of the Bayes Class Probabilities (BCPs), whereas the cost of teacher supervision increases with the teacher error relative to them, yielding a principled sample-wise routing rule. Since the underlying BCPs are unavailable in practice, we relate these quantities to predictive uncertainty: aleatoric uncertainty provides a proxy for BCP diversity, while epistemic uncertainty indicates limited teacher expertise. Building on this connection, we propose an uncertainty-guided selective distillation method that routes each sample between label- and teacher-based supervision. Experiments across vision, audio, and high-energy physics classification tasks with partially expert teachers demonstrate consistent improvements over state-of-the-art KD approaches.

Primary area:
learning theory
Keywords:
knowledge distillation, stochastic gradient descent, convergence analysis, sample-wise supervision, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_u6y8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies knowledge distillation (KD) from partially expert teachers (PETs), whose predictions are reliable on some inputs but potentially misleading on others. It develops an SGD convergence analysis that characterizes the sample-wise trade-off between one-hot and teacher supervision. The resulting bounds identify two competing factors: the intrinsic diversity of Bayes class probabilities (BCPs), which affects the cost of one-hot supervision, and teacher prediction error, which affects the cost of distillation. The paper proposes an uncertainty-guided selective KD method that uses aleatoric and epistemic uncertainty as proxies for these factors and employs a Gaussian mixture model to route each sample to either one-hot or teacher supervision. Experiments across vision, audio, and high-energy physics classification tasks demonstrate improvements over standard and adaptive KD baselines.

Critical strengths

S1. The paper addresses an important limitation of conventional KD: a teacher may be reliable only on a subset of the input space. The sample-wise decision of whether to use teacher or one-hot supervision is well motivated and practically relevant, particularly when the teacher and student training distributions differ.

S2. The SGD convergence analysis provides an interesting characterization of the competing costs of one-hot and teacher supervision. In particular, relating these costs to BCP diversity and teacher error offers a principled explanation for why different supervision sources may be preferable for different samples.

S3. The proposed method achieves consistent improvements across vision, audio, and high-energy physics classification tasks, outperforming a broad range of standard and adaptive KD baselines. The additional experiments showing compatibility with existing KD objectives further support the practical usefulness of the proposed routing mechanism.

Critical weaknesses

W1. The theoretical analysis derives a sample-wise routing criterion by comparing the costs of one-hot and teacher supervision. However, the practical method replaces these costs with the ratio of epistemic to total uncertainty, followed by a two-component Gaussian mixture model. The theoretical costs depend on several factors, including teacher bias, prediction variability, and the student's Jacobian, which are not explicitly captured by the proposed uncertainty ratio. Although the paper provides empirical evidence supporting the use of AU and EU as proxies, it remains unclear whether the resulting routing decisions reliably approximate those prescribed by the theoretical analysis.

W2. The convergence analysis relies on strong quasi-convexity and student realizability assumptions, which may not hold for practical neural networks, particularly when the student has substantially lower capacity than the teacher.

W3. The proposed method uses epistemic uncertainty to estimate teacher reliability. However, a teacher may make confidently incorrect predictions with low epistemic uncertainty, potentially leading to inappropriate routing decisions.

Questions for authors

Q1. Under what conditions does the uncertainty-based routing rule approximate the theoretically optimal routing criterion?

Q2. How does limited student capacity affect the theoretical routing criterion between teacher and one-hot supervision?

Q3. How does the proposed method handle confidently incorrect teacher predictions with low epistemic uncertainty?

2 comments
AuthorsOpenReview ↗

Response to Reviewer u6y8 (1/2)

We thank the reviewer for the careful evaluation and the thoughtful feedback. We address the reviewer's concerns below. Table and figure numbers refer to the revised paper.

Q1 & W1. As noted by the reviewer and referenced in our limitations, while our method is derived based on the theoretical characterization (whose formulation involves quantities such as the true BCPs and loss-landscape constants μ\mu, LL, and L\mathcal{L} and requires AS1-AS3), the practical method (that has no access to these quantities) may not share these theoretical guarantees. The theoretical results are intended to provide insights on the inner-working mechanisms of KD, what affects it, and to provide support and guidelines for practical method design. On top of the strengths acknowledged by the reviewer which support the use of the method, we can show the approximations of the uncertainty-based routing rule and the theoretically optimal routing criterion are aligned in various scenarios. Following the reviewer’s suggestion, we added the new Subsection E.5 in the Appendix which explores this correspondence in the 9 settings of Appendix E. We include the main results here:

ExperimentBayesian teacherAUROC of r(x)r(x) against λ∗(x)\lambda^*(x)Practical rule and λ∗(x)\lambda^*(x) agreement (%)Student acc, practical rule (%)Student acc, theoretical rule λ∗(x)\lambda^*(x) (%)
ALaplace0.95788.378.6 (0.8)79.2 (0.9)
AVI0.91285.678.3 (0.7)79.3 (0.8)
AEnsemble0.94184.478.7 (0.2)79.7 (1.0)
BLaplace0.95089.078.3 (0.9)78.5 (0.3)
BVI0.85381.878.2 (0.5)78.9 (0.5)
BEnsemble0.91585.078.5 (0.4)78.8 (0.5)
CLaplace0.97994.474.6 (0.3)74.9 (0.6)
CVI0.97893.874.5 (0.4)74.5 (0.5)
CEnsemble0.96493.274.3 (0.5)74.6 (0.4)

The practical and theoretical rules agree on 88% of the samples on average across the 9 test cases, the practical score r(x)r(x) predicts the theoretical routing with an average AUROC of 0.94, and students trained with the practical rule are within 1.0 point of those trained with the theoretical rule. Additional results, comparisons to solely EU-based and AU-based routers, and visualization of the correspondence are presented in the newly added Appendix E.5. The results further strengthen the connection between the theoretical result and the practical method. We thank the reviewer for this important suggestion.

Q2 & W2. As the reviewer noted, and as stated in our limitations, our theoretical analysis relies on assumptions which may not hold in practice, but are typically used in works that analyze convergence in KD [1,2,3]. Quasi-convexity and smoothness assumptions are used to maintain a tractable convergence characterization, and we refer the reviewer to the "Assumptions" paragraph in Section 3.1 which explains where the smoothness assumption may come from for typical architectures via bounded Jacobians. The realizability assumption, as stated in Section 3.1, is a deliberate modelling choice which abstracts away student capacity so that the analysis can characterize the sample-wise costs through the BCPs rather than through the student's in-class optimum. From a theoretical perspective, incorporating limited student capacity into the characterization in the paper requires re-deriving the proofs under different modelling choices, which would substantially change the paper. We therefore leave it for future work.

To answer the reviewer's question empirically, we ran additional experiments that examine how limited student capacity affects our method relative to standard one-hot training and standard KD. We fix the teacher and scale the width of the student for four teacher-student pairs across two modalities, for a total of 30 student configurations ranging from 3.5k to 25M parameters. The full sweeps are shown in Figure 19 at the end of Subsection G.1 of the revised paper, together with a discussion. The table below reports student test accuracy at the largest, nominal, and smallest student of each pair.

Teacher → studentStudent parameters (largest / nominal / smallest)Largest: KDLargest: one-hotLargest: oursNominal: KDNominal: one-hotNominal: oursSmallest: KDSmallest: one-hotSmallest: ours
UrbanSound8K: CNN-8 → CNN-425M / 390k / 6.3k57.2061.3366.7756.6960.1065.9546.8849.2554.71
UrbanSound8K: ResNet-14 → ResNet-813M / 210k / 3.5k50.9751.3557.7249.4751.3057.3143.0146.5751.46
CIFAR-10: WRN-16-2 → WRN-16-k4.3M / 175k / 12k67.4484.0586.8265.2778.6683.5857.6670.3570.79
CIFAR-10: ConvNet-6 → ConvNet-31.5M / 95k / 6.4k61.6869.3376.7258.9269.2673.1850.1660.5461.83
AuthorsOpenReview ↗

Response to Reviewer u6y8 (2/2)

(continued) As anticipated, the accuracy of all three methods degrades as the student shrinks. Our method nevertheless attains the highest accuracy in all 30 configurations, with a gain over the better of standard one-hot training and standard distillation of 4.2 to 7.1 percentage points on UrbanSound8K and 0.4 to 7.4 points on CIFAR-10. In particular, the smallest audio students, with 6.3k and 3.5k parameters, still gain 5.5 and 4.9 points. We thank the reviewer for this important question, which strengthened the results and insights in the paper.

Q3 & W3. We agree with the reviewer. As stated in our limitations section, confidently wrong predictions with low epistemic uncertainty in Bayesian NNs are a well-known challenge that may arise from the posterior approximation or from model misspecification [4,5]. Our work is not aimed at solving this challenge, which has several active research directions, including characterizing what information-theoretic EU estimators actually measure and when they fail [6] and alternative uncertainty representations such as credal sets [7].

To answer the reviewer's question, we highlight that the core contribution of the paper is a new understanding of the quantities that govern distillation for students trained by SGD, how to approximate them, and how to use them to improve the distillation process. The method is therefore based solely on AU and EU, and it does not leverage other signals on which many previous methods rely, including agreement with the one-hot labels, the student's current predictions, or a validation objective optimized in a bilevel or meta step. Although the design is intentionally kept free of these signals so that the gain can be attributed to the uncertainty signals, they carry complementary information, which motivated us to combine our approach with other methods, as shown in Figures 3 and 18. To answer the reviewer's question, such combinations can correct errors that arise, for example, from confidently incorrect teacher predictions with low EU. In addition to Figures 3 and 18, we provide an intuitive example below. Inspecting the routing decisions of our method, such predictions appeared mainly in the audio experiments. The simplified rule supported by [8] distills a sample when the teacher's top prediction agrees with the one-hot label, and does not distill it otherwise. Our method sends almost all confidently incorrect predictions with low EU to the teacher, whereas the label-agreement rule sends all of them to the one-hot label, but it makes other routing errors that our method handles correctly, as explained in the paper. We therefore tested a simple combination for these cases: when the two rules agree we use their routing, and when they disagree we set λ(x)=0.5\lambda(x)=0.5, which gave:

Teacher → StudentLow-EU confidently wrong (% of samples)Routed to teacher by ours (% of those)Routed to one-hot by agreement (% of those)Agreement (Acc [%])Ours (Acc [%])Hybrid (Acc [%])
UrbanSound8K: CNN-8 → CNN-410.991.410064.2165.9566.76
UrbanSound8K: ResNet-14 → ResNet-810.399.410053.7257.3157.72
UrbanSound8K: ResNet-14 → CNN-4-N10.399.410058.9359.5661.05

The students trained with the hybrid rule outperform those trained with either rule alone. This further demonstrates how complementary signals can correct each method's individual errors, and, as stated in the paper, we encourage practitioners to combine such mechanisms.

We hope our responses have adequately addressed the reviewer's concerns and are grateful the reviewer acknowledged the strengths of our work. We would be grateful if the reviewer would consider raising their score to a positive one.

[1] "Knowledge distillation performs partial variance reduction", NeurIPS 2023.

[2] “SGD-based knowledge distillation with Bayesian teachers: Theory and guidelines", ICLR 2026.

[3] “Handbook of convergence theorems for (stochastic) gradient methods”, ArXiv 2023.

[4] "Deep deterministic uncertainty: A new simple baseline", CVPR 2023.

[5] "On the expressiveness of approximate inference in Bayesian neural networks", NeurIPS 2020.

[6] "Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks", NeurIPS 2024.

[7] "Credal wrapper of model averaging for uncertainty estimation in classification", ICLR 2025.

[8] "Correctness-aware knowledge distillation for enhanced student learning", TMLR 2026.

Reviewer_5poAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies knowledge distillation (KD) from partially expert teachers, focusing on when the student should rely on teacher predictions rather than observed one-hot labels under SGD-based KD. Through an SGD analysis, the authors derive a sample-wise supervision selection rule governed by the underlying Bayes Class Probabilities (BCPs). They then use AU and EU as proxies for BCP diversity and teacher error to construct a practical sample-wise routing rule between label-based and teacher-based supervision. Experiments demonstrate the effectiveness of the proposed approach.

Critical strengths
  1. This paper addresses a meaningful problem in sample-wise knowledge distillation: determining how much the student should trust teacher predictions for different training samples. In the setting of partially expert teachers, explicitly analyzing the relative benefits of the two supervision sources is worthwhile.
  2. The paper provides relatively comprehensive experimental validation, including ablation studies of the different components. Although the GMM serves as an adaptive thresholding mechanism, the authors evaluate several alternative strategies to demonstrate its effectiveness. These experiments provide empirical support for the proposed method.
Critical weaknesses
  1. The connection between the theoretically optimal routing rule for lambda(x) and the practical AU/EU-based rule is not sufficiently established. Through an SGD analysis, the authors derive a supervision selection rule based on C(x) B(x) A(x), and their associated coefficients. However, the practical algorithm does not directly compute or approximate these quantities. Instead, it determines the routing decisions using r(x)=EU/(EU +AU). Authors explain the respective roles of AU and EU, providing a qualitative motivation for this design. However, it remains insufficiently validated to what extent the practical routing rule follows the theoretically derived selection rule. Given the synthetic experiments already established in the paper, I suggest comparing the practical AU/EU-based routing with the routing derived from the theoretical quantities in these controlled settings.

  2. Could the potential performance upper bound of the sample-wise selection strategy be further investigated? The experiments demonstrate that the proposed sample-wise selection strategy outperforms fixed mixing weights and other compared methods. I wonder whether a small-scale controlled experiment could provide further insight into its potential performance upper bound. Specifically, for a small number of representative samples, one could enumerate different supervision weight assignments using an independent validation set, initially considering binary weights (0 or 1) and potentially including soft mixing if computationally feasible. Comparing the best performance obtained through this search with the proposed method could help assess the remaining room for improvement in the sample-wise selection strategy and provide an additional reference for its empirical effectiveness.

  3. The evaluation in the real-world PET settings could be more complete. The authors construct PET settings by training the teacher only on region A, while the student is trained on samples from both regions A and B. Although the paper reports the overall performance, I suggest additionally reporting the teacher's predictive performance separately on the two regions, as well as the proportion of samples assigned to teacher supervision by the proposed routing strategy in each region. This would help validate the reliability of the proposed routing strategy.

Questions for authors

My main question concerns Critical Weakness 1. Could the authors further clarify or empirically validate the relationship between the theoretically derived selection rule based on C(x) B(x) A(x), and the practical routing strategy based on AU/EU and GMM? Even a comparison in synthetic experiments would provide more direct evidence of how the theoretical analysis relates to the practical algorithm.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5poA (1/2)

We sincerely thank the reviewer for their positive evaluation and constructive feedback. We address the reviewer’s concerns below. Table and figure numbers refer to the revised paper.

W1 & Q1. We agree with the reviewer. The theoretical characterization relies on modelling choices and involves quantities such as the true BCPs and the loss-landscape constants μ\mu, LL, and L\mathcal{L}, to which the practical method has no access. The theoretical results are intended to provide insights on the inner-working mechanisms of KD and what affects it, and to provide support and guidelines for practical method design. We therefore understand the importance of comparing the practical routing with the routing derived from the theoretical quantities in the synthetic experiments of Appendix E, and we added the experiments requested by the reviewer in Appendix E.5 of the revised paper. We include the main results here:

ExperimentBayesian teacherAUROC of r(x)r(x) against λ∗(x)\lambda^*(x)Practical rule and λ∗(x)\lambda^*(x) agreement (%)Student accuracy, practical rule (%)Student accuracy, theoretical rule λ∗(x)\lambda^*(x) (%)
ALaplace0.95788.378.6 (0.8)79.2 (0.9)
AVI0.91285.678.3 (0.7)79.3 (0.8)
AEnsemble0.94184.478.7 (0.2)79.7 (1.0)
BLaplace0.95089.078.3 (0.9)78.5 (0.3)
BVI0.85381.878.2 (0.5)78.9 (0.5)
BEnsemble0.91585.078.5 (0.4)78.8 (0.5)
CLaplace0.97994.474.6 (0.3)74.9 (0.6)
CVI0.97893.874.5 (0.4)74.5 (0.5)
CEnsemble0.96493.274.3 (0.5)74.6 (0.4)

In these settings, the practical and theoretical rules agree on 88% of the samples on average across the 9 test cases, the practical score r(x)r(x) predicts the theoretical routing with an average AUROC of 0.94, and students trained with the practical rule are within 1.0 point of those trained with the theoretical rule. Additional results, comparisons to solely EU-based and AU-based routers, and visualization of the correspondence are presented in the newly added Appendix E.5. The results further strengthen the connection between the theoretical result and the practical method. We thank the reviewer for this important suggestion.

W2. As requested by the reviewer, we conducted the proposed controlled experiment, with binary weights and with soft mixing, in the synthetic setting of Appendix C. Since searching over a separate weight per sample is computationally infeasible, we split the training samples into 10 groups (samples where the teacher is an expert or unreliable, each divided into five levels of BCP diversity) and search over one weight per group. For binary weights we train all 210=10242^{10}=1024 assignments in each of 33 settings (three types of PETs, non-expert fractions ff from 0 to 1). For soft weights in {0,0.25,0.5,0.75,1}\lbrace 0, 0.25, 0.5, 0.75, 1 \rbrace, an exhaustive search would require 510≈9.85^{10}\approx 9.8M students per setting, so we conduct a greedy search guided by the validation set. The full results are presented and visualized (Figure 8) at the end of Appendix C.3 of the revised paper, and are summarized here:

PET typeBest binary − ours (pp, avg over ff)Greedy soft − ours (pp, avg over ff)
Half-space0.110.93
Radial0.061.09
Diversity0.030.65

Among binary assignments, the rule of Proposition 2 is the best of all 1,024 in 23 of the 33 settings and within 0.64 points otherwise, leaving essentially no room for improvement above it in these settings. The greedy soft search improves over the rule by 0.89 points on average, mostly at large ff (1.71 points for f≥0.7f\ge0.7, 0.19 for f≤0.3f\le0.3). The assignments it finds keep λ\lambda close to 1 where the teacher is an expert and place small weights on unreliable samples with high BCP diversity, which is consistent with the label-smoothing effect of distillation [1], also observed for the best constant λ\lambda at large ff in Figure 6. Such room above binary routings is expected in real settings, since KD benefits from several mechanisms, as supported by Figures 3 and 18, where combining our routing with existing KD methods yields additional gains. We thank the reviewer for this insightful suggestion.

AuthorsOpenReview ↗

Response to Reviewer 5poA (2/2)

W3. As requested by the reviewer, we additionally report teacher's predictive performance separately on the two regions, as well as the proportion of samples assigned to teacher supervision by the proposed routing strategy in each region. The additional metrics are presented in Table 9 of Appendix G.1 of the revised paper, followed by a discussion, and are also reported here:

JetClass: MLP-256CIFAR-10: WRN-16-2CIFAR-10: ResNet-32CIFAR-10: ResNet-20CIFAR-10: WRN-40-2CIFAR-10: ConvNet-6UrbanSound8K: CNN-8UrbanSound8K: ResNet-14UrbanSound8K: CNN-8-N
Teacher accuracy in region A (%)70.4992.4391.5490.5494.0488.9375.2471.4975.07
Teacher accuracy in region B (%)31.0911.2514.7513.5512.979.7026.3835.2230.73
Region A samples routed to the teacher by our method (%)98.1491.7371.4782.1671.3498.1697.6299.7198.95
Region B samples routed to the teacher by our method (%)0.000.602.031.932.670.2019.7813.7819.33
Routing-region agreement (%)98.7094.0379.4286.9379.1498.6592.4095.6793.47

The metrics show an average of 90.9% of the routing decisions agree with the regions across all teachers, further validating the reliability of the proposed method.

We thank the reviewer for recognizing the strengths of our work and for suggesting valuable and insightful extensions and further investigations of its inner workings. We hope our responses have adequately addressed the reviewer's concerns.

[1] "Revisiting knowledge distillation via label smoothing regularization", CVPR 2020.

Thank you for the detailed responses and additional experiments. The responses have adequately addressed my concerns.

AuthorsOpenReview ↗

Response to Reviewer 5poA

We sincerely thank the reviewer for the positive evaluation of our work and for taking the time to read the responses and revised paper. We are glad the additional experiments adequately addressed the concerns.

Reviewer_W4dzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies knowledge distillation when the teacher is reliable on part of the input space and unreliable elsewhere. Modeling the teacher as a Dirichlet perturbation of the Bayes class probabilities with input dependent bias and concentration, the authors derive SGD convergence bounds for a per sample mixing weight and show the asymptotic error splits into a one hot cost that grows with the Gini diversity of the true label distribution and a teacher cost that grows with teacher bias and variance. The bound is minimized by routing each sample to whichever source is cheaper. Since neither quantity is observable, they use a Bayesian teacher's aleatoric uncertainty as a proxy for label diversity and epistemic uncertainty as a proxy for teacher error, compute the EU fraction, and fit a two component GMM over the training set to split samples into teacher supervised and label supervised. Experiments on CIFAR-10 with corruption pairs, UrbanSound8K with a frequency mask, and JetClass with jet momentum bands show gains of roughly 1 to 3 points over the best of thirteen baselines and that the router composes with other KD losses.

Critical strengths
  1. The theory and the method are actually connected. Proposition 3 gives the Gini index interpretation of the one hot cost, and the AU and EU proxies map onto the two terms in a way that is validated on synthetic data where the Bayes probabilities are known. This is more than the usual decoration.
  2. The experiments are careful. Every number has a standard deviation over 3 or 10 seeds, baselines are grid searched including their published settings, a best constant λ row is included, and the comparison spans three modalities and seventeen pairs across Tables 1, 6 and 7. The ablations in Tables 9 and 10 report paired differences rather than raw numbers.
  3. The ablation on the routing signal in Table 9 is the most convincing part. EU alone, agreement, and max probability all lose, and the explanation for why each fails lines up with the theory.
Critical weaknesses
  1. The practical decision rule is not from the theory. Proposition 2 compares two costs with unknown constants, and the paper admits the threshold on r(x) cannot be derived. The GMM is a heuristic, and Table 10 shows a plain median gate matches or beats it on four of six pairs. So the contribution of the theory is the direction of the trade off, which is close to the intuition one would have started with.
  2. The PET construction makes the teacher weak in absolute terms. On CIFAR-10 the teacher gets 65 to 70 percent on the student test set while the one hot student gets 80 to 82 percent. A teacher that is worse than the student trained from scratch on 10k images is an unusual regime, and pure KD losing by up to 16 points is mostly a consequence of that. The more interesting case, a strong teacher that is wrong on a minority of inputs, is only covered by the f sweep in the appendix.
  3. There is no oracle routing baseline. The regions are known by construction, so routing by true region membership would give an upper bound and tell us how much of the achievable gain the uncertainty proxies capture. Without it the 1 to 3 point improvements have no reference point.
  4. The assumption that µ, L and L hold uniformly over all routings λ is doing a lot of work. The risk function changes with λ, so the constants should too, and the bound comparison across routings depends on this.
  5. The gain over the best baseline is small, 0.6 to 2.9 points, and WLS or LUD is within about a point on several pairs. Meanwhile the method needs a Bayesian teacher with 30 posterior samples per training example. UBKD has the same requirement, but the other baselines do not, and the comparison does not account for that cost.
  6. The Dirichlet PET model is acknowledged as an abstraction, but the analysis uses its mean and variance structure directly in D(x) and B(x). It is not shown that a real teacher's error on region B looks like this rather than, for example, confidently wrong with low variance, which is exactly the case where a last layer Laplace approximation would give low EU.
Questions for authors
  1. What does oracle routing by true region give on the Table 1 pairs? This is the single number that would most change my read.
  2. Table 10 shows a median gate on r is better than the GMM on four of six pairs. Why keep the GMM as the default?
  3. Can you show one setting where the teacher is stronger than the one hot student overall but unreliable on a minority region, and whether the gain persists there?
  4. How often does the last layer Laplace EU fail to flag region B samples, that is, what is the routing accuracy against true region membership?
  5. Can you confirm the Beta-KD and WTTM implementations reproduce their published numbers in a non PET setting?
5 comments
AuthorsOpenReview ↗

Response to Reviewer W4dz (1/3)

We thank the reviewer for the careful evaluation and the constructive feedback. We address the reviewer's concerns below. Table and figure numbers refer to the revised paper.

Q1 & W3. As requested, we implement the oracle baseline which routes based on true region membership. The oracle baseline performance compared to our method is presented below.

Teacher → StudentOurs (Acc [%])Oracle baseline (Acc [%])Oracle − Ours (pp)
JetClass: MLP-256 → MLP-6464.4364.480.05
CIFAR-10: WRN-16-2 → WRN-16-183.5883.910.33
CIFAR-10: ResNet-32 → ResNet-2083.3984.471.08
CIFAR-10: ResNet-20 → ResNet-2083.2684.150.89
CIFAR-10: WRN-40-2 → WRN-40-184.7985.390.60
CIFAR-10: ConvNet-6 → ConvNet-373.1873.210.03
UrbanSound8K: CNN-8 → CNN-465.9567.101.15
UrbanSound8K: ResNet-14 → ResNet-857.3158.311.00
UrbanSound8K: CNN-8-N → CNN-4-N60.3462.011.67

Our method is within 0.03 to 1.67 points of the oracle baseline on every pair. This empirically strengthens our practical method, which consistently recovers most of the oracle-based achievable gain, with an average gap to the oracle at least four times smaller than that of any other baseline. We added the oracle row to Table 1 in the main body of the revised paper and a short explanation about it in the discussion of the main results in Section 4.3. Thank you for this suggestion!

Q2 & W1. Note that the relation between uncertainty in predictions and its use in distillation is a long-lasting source of disagreement in previous works (see paragraph “Uncertainty in KD” in the related works section). Our convergence analysis characterizes how BCP diversity and teacher error affect the per-sample trade-off between one-hot and teacher supervision, which is then related to aleatoric and epistemic uncertainty in predictions, providing insights further beyond “the direction of the trade off”. While the practical decision rule indeed approximates the theoretical costs and constants which depend on the unknown BCPs and on the loss-landscape constants μ\mu, LL, and L\mathcal{L}, the correspondence between the approximated practical and theoretical rules can be shown in synthetic settings. We perform this through experiments in the controlled settings of Appendix E in the newly added Subsection E.5, which demonstrate that the practical rule agrees with the theoretical rule of Proposition 2 on 88% of the samples on average across 9 test cases, and has an average AUROC of 0.94 against the theoretical rule. The results further strengthen the connection between the theoretical result and the practical method. Regarding the median gate, note first that it is not a separate baseline, but rather a method that entirely relies on our theoretical framework and our proposed ratio r(x)r(x), changing only the threshold on r(x)r(x). We keep the GMM as our default because a median fixes the routed fraction at one half of the training set regardless of the true proportion of expert and non-expert samples, whereas the GMM estimates that proportion from the data with no hand-set threshold. While we agree with the reviewer that the median gate yields minor gains on some pairs in Table 13 (previously Table 10), it receives lower accuracy on average, and is less robust to an expert fraction sweep (see Appendix G.3).

Q3 & W2. As requested, we point to settings where the teacher is stronger than the one-hot student overall but unreliable on a minority region. In these settings plain KD does not exploit the teacher's advantage: in Table 1 it falls below one-hot training in all four, consistent with the student inheriting the teacher's bias on unreliable samples. Our method improves over both baselines in every setting. The relevant pairs from Table 1 are shown below.

Teacher → StudentTeacher − one-hot (pp)KD − one-hot (pp)Ours − one-hot (pp)Ours − KD (pp)
JetClass: MLP-256 → MLP-64+0.5-1.8+6.3+8.1
UrbanSound8K: CNN-8 → CNN-4+0.5-3.4+5.9+9.3
UrbanSound8K: ResNet-14 → ResNet-8+9.3-1.8+6.0+7.8
UrbanSound8K: CNN-8-N → CNN-4-N+4.6-3.3+3.2+6.5

In Table 8, the teacher outperforms the one-hot student in all four UrbanSound8K pairs (+0.5 to +10.4 pp), with the following results.

Teacher → StudentTeacher − one-hot (pp)KD − one-hot (pp)Ours − one-hot (pp)Ours − KD (pp)
UrbanSound8K: CNN-8 → ResNet-8+9.3+2.0+8.3+6.3
UrbanSound8K: ResNet-14 → CNN-4+0.5-8.0+1.4+9.4
UrbanSound8K: ResNet-14 → CNN-4-N+3.5-6.7+2.4+9.1
UrbanSound8K: CNN-8-N → ResNet-8+10.4+3.0+7.5+4.5

The gains also persist at lower non-expert fractions in Figure 20, for example f=0.2 for JetClass and f=0.1 for CIFAR-10 WRN-16-2 → WRN-16-1.

AuthorsOpenReview ↗

Response to Reviewer W4dz (2/3)

Q4. Following the reviewer’s request, we provide additional insights into our method’s behavior including statistics on regions and routing decisions, which are included in Appendix G.1 in the revised paper (new Table 9 followed by a discussion). The resulting routing accuracy against true region membership is also reported here:

JetClass: MLP-256CIFAR-10: WRN-16-2CIFAR-10: ResNet-32CIFAR-10: ResNet-20CIFAR-10: WRN-40-2CIFAR-10: ConvNet-6UrbanSound8K: CNN-8UrbanSound8K: ResNet-14UrbanSound8K: CNN-8-N
Region B samples routed to the teacher by our method (%)0.000.602.031.932.670.2019.7813.7819.33
Routing-region agreement (%)98.7094.0379.4286.9379.1498.6592.4095.6793.47

The routing agrees with the true region membership on 90.9% of training samples on average across all teachers.

Q5. Our implementations of Beta-KD and WTTM are based on the authors’ official code. As such, they reproduce the results of their corresponding papers when applied to the same settings. We did have to adapt them to our pipeline, and we updated Appendix F.1 to clarify this. While these methods are designed to perform well in standard non-PET settings, their accuracy in our tables is consistent with their design here: both distill every sample, so they perform close to plain KD in a PET setting where the teacher is unreliable.

W4. We agree with the reviewer that this is a modelling assumption and that it may not reflect all scenarios. As stated in Section 3.1 and in the limitations, it was a deliberate choice which provides the analytical tractability needed for a per-sample analysis in which λ\lambda varies across inputs, and it allows us to reach a meaningful characterization of how teacher error and BCP diversity govern the sample-wise bounds and routing. We also note that Theorems 1 and 2 hold for any single routing λ\lambda with constants specific to that routing and the true risk, uniform constants are only invoked in Proposition 2 as a common bounding envelope to compare asymptotic error terms across routings on equal footing.

W5. We would like to emphasize that our main contribution is a theoretical understanding of the quantities governing SGD-based distillation with PETs, and a method that leverages this understanding to achieve consistent improvements using uncertainty-based signals alone. Access to teacher uncertainty is indeed an additional requirement compared with some existing methods, but it is precisely this information that enables our theoretically motivated routing.

Regarding the Bayesian teacher requirement, for fair comparison, all methods in Table 1 use the same teacher predictions, obtained by averaging 30 posterior samples and precomputed before student training, and our method also exploits them to estimate AU and EU. Note that the Bayesian teacher can be obtained from a pretrained deterministic model through a post-hoc, last-layer Laplace approximation, and posterior sampling is performed only once before student training.

Regarding the gains, our method consistently outperforms the strongest baseline across all nine teacher-student pairs by 0.6–2.9 percentage points. These improvements are achieved without leveraging additional signals used by other methods, such as label agreement, student predictions, or meta-optimization. Moreover, Figures 3 and 18 demonstrate that our approach provides complementary information that can be combined with existing methods for further improvements. We thus believe that our results demonstrate the value of uncertainty-guided routing, both as a standalone approach and as a complementary component of existing distillation methods.

AuthorsOpenReview ↗

Response to Reviewer W4dz (3/3)

W6. Indeed, the teacher model with errors that follow a Dirichlet distribution with input-dependent concentration parameters and bias is used as a modelling choice in our analysis to show how they affect the final decisions. As explained in the “Analytical PET Model” paragraph in Section 3.1, it is not intended to imply that the predictions of a trained teacher necessarily follow a biased Dirichlet distribution. Nevertheless, as noted in our limitations, confidently wrong predictions with low EU may arise in practice due to approximate Bayesian inference or model misspecification [1,2], motivating active orthogonal research into specialized uncertainty estimators [3] and credal sets [4]. As mentioned in W5, while our design is intentionally kept free of other signals to isolate the effect of the uncertainty signals, combining our approach with methods that utilize complementary signals (as shown across architectures and modalities in Figures 3 and 18) helps make up for routing errors, which can include confidently wrong low-EU predictions.

We provide an additional intuitive example: examining the predictions made by our method, such confidently incorrect predictions with low EU were mainly observed in the audio experiments. The simplified method supported by [5] distills samples where top teacher predictions agree with the one-hot label, and doesn’t distill otherwise. Our method routes almost all confidently wrong samples with low EU to the teacher, while the label-agreement rule routes all of them to the one-hot label, but makes other routing errors that our method correctly handles, as explained in the paper. We therefore considered a simple combination for such cases: when both rules agree use their routing, and when they disagree use λ(x)=0.5\lambda(x)=0.5, which resulted in:

Teacher → StudentLow-EU confidently wrong (% of samples)Routed to teacher by ours (% of those)Routed to one-hot by agreement (% of those)Agreement (Acc [%])Ours (Acc [%])Hybrid (Acc [%])
UrbanSound8K: CNN-8 → CNN-410.991.410064.2165.9566.76
UrbanSound8K: ResNet-14 → ResNet-810.399.410053.7257.3157.72
UrbanSound8K: ResNet-14 → CNN-4-N10.399.410058.9359.5661.05

The students trained with the hybrid rule outperform those trained with either rule alone. These results further demonstrate how complementary signals can correct each method's individual errors, and, as stated in the paper, practitioners are encouraged to combine such mechanisms.

We hope our responses have adequately addressed the reviewer's concerns and appreciate the suggestions that strengthen our work. We would be grateful if the reviewer would consider raising their score to a positive one.

[1] "Deep deterministic uncertainty: A new simple baseline", CVPR 2023.

[2] "On the expressiveness of approximate inference in Bayesian neural networks", NeurIPS 2020.

[3] "Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks", NeurIPS 2024.

[4] "Credal wrapper of model averaging for uncertainty estimation in classification", ICLR 2025.

[5] "Correctness-Aware knowledge distillation for enhanced student learning", TMLR 2026.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and effort in reviewing this paper. Please take a moment to read the authors’ rebuttal and the other reviewers’ comments, and participate in the discussion. It would be helpful to indicate which concerns have been addressed and which remain unresolved, and to follow up with the authors where further clarification is needed. Please also update your review and score if warranted, or briefly explain why your assessment remains unchanged. Your engagement will be valuable in reaching a well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.