acceptodds
Under review as a conference paper at ICLR 2027

Distilling Bayesian Uncertainty into a Single Forward Pass

Abstract

Deep ensembles give reliable Bayesian uncertainty estimates, but they need forward passes for every input. We treat the uncertainty of a Bayesian teacher as a function of the input and train a single deterministic network, our student, to output it in one forward pass. Distilling such posterior quantities has been done before, but the targets are Monte Carlo estimates from members: they change when the ensemble is retrained, and what a student learns from these noisy targets has not been analysed. We model the student's uncertainty head as a regression on the noisy teacher uncertainty, allowing correlated noise and a biased teacher, and give an exact condition under which the student is closer to the ideal uncertainty than its teacher. When the teacher's bias and the part of its noise that does not average out over members are small, the best regularisation strength decreases as grows. A controlled simulation and ensembles trained from scratch follow the predicted trend in . To learn the uncertainty function, our student model first learns the teacher's predictions; a small head on the frozen backbone then learns the teacher's uncertainty on clean, corrupted, mixed and masked inputs, with a log-space regression loss and a ranking loss. Any teacher that provides this uncertainty can be used. On image classification, segmentation and text, our student stays close to the teacher's accuracy at the cost of one forward pass. Scored against ground-truth labels on image classification, it has the lowest or tied-lowest selective-prediction risk among single-pass methods that do not distil the teacher, is competitive with other distillation methods, and detects distribution shift without labels as well as the teacher. Its out-of-distribution detection is close to the teacher's.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Quantification, Bayesian Deep Learning, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_m7yUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to distill a deep ensemble into a single model, and keep the aleatoric-epistemic uncertainty estimators by additionally adding an uncertainty head that distills the mutual information of the teacher ensemble on augmented inputs.

Critical strengths
  1. The paper is well written, it is easy to understand and presented clearly.
  2. The appendix contains extensive supplementary results.
Critical weaknesses
  1. The method can be seen as distilling a deep ensemble into a Treenet where one head predicts . However, the idea that a single pass can somehow capture epistemic and aleatoric uncertainty is not supported theoretically. The student's uncertainty estimation works in the evaluation provided can perhaps be explained by the high empirical correlation between aleatoric and epistemic uncertainty, hence

  2. The proposed distillation method relies on multiple augmentation methods to train the uncertainty estimator, but those augmentations were not used for baselines, which suggests a potentially unfair comparison. Even then the reported results are not very strong on the utility table (with the proposed method losing to baselines on 7/8 Miscls/OOD AUROC in table 2).

Questions for authors
  1. How did you ensure fairness when comparing your distillation method to the baselines, did you match the amount of augmentation in training?
  2. I noticed that for three of the datasets you evaluated on, you start with a single pre-trained model, and use LoRA fine-tuned checkpoints as ensemble members to distill from. Generally, deep ensembles require members to be trained from different random initialization so they can provide the necessary predictive diversity useful for uncertainty quantification. Why did you make that experimental decision, and does it affect the interpretation of your reported results?
  3. Can you add experiment to evaluate the entanglement of aleatoric and epistemic uncertainty given by your distilled student model?
3 comments
AuthorsOpenReview ↗

Response to Reviewer m7yU (1/2)

We thank Reviewer m7yU for the clear and focused questions and the kind words on the writing and appendix. We tested whether our EU head only reflects aleatoric uncertainty, rebuilt all baselines under matched data and backbones, and clarified the role of the LoRA ensembles, reporting also where the evidence is weaker. G1 to G4 refer to our General Response on top (held-out pipeline, GPED and DLF, matched baselines, tests of Sec. 4).

W1, Q3. Can one forward pass estimate epistemic uncertainty, or does it only reflect aleatoric correlation? We agree that one forward pass does not sample a posterior at test time, and our claim is narrower. Once the KK-member teacher is trained, its MI uT(x)u_T(x) (Eq. 3) is a deterministic function of xx. Stage 2 learns this function by regression from frozen features, as in amortized inference, and Sec. 4 analyzes when this regression is accurate.

To test whether the EU head only reflects the correlation between aleatoric (AU) and epistemic (EU) uncertainty, we ran the checks below (ID test sets, Spearman unless noted).

CheckCIFAR-10MNISTSVHN (new)
Teacher AU vs. teacher EU0.6470.8370.531
Student AU head vs. student EU head0.4970.789n/a
Student EU head vs. teacher EU / AU0.801 / 0.5400.801 / 0.7570.773 / n/a
Partial on H[pS]H[p_S] (vs. teacher EU)0.7070.5680.706
Head on features only vs. softmax only0.802 vs. 0.627Pearson 0.791 vs. 0.6390.771 vs. 0.591
Mean OOD AUROC, EU head vs. H[pS]H[p_S]0.900 vs. 0.8780.991 vs. 0.3010.977 vs. 0.968
  • Distillation does not merge EU into AU. On CIFAR-10 and MNIST, the student's AU and EU heads correlate less than the teacher's AU and EU, and the EU head tracks teacher EU much more closely than teacher AU.
  • Features carry EU beyond softmax. Controlling for H[pS]H[p_S], the EU head still correlates 0.57 to 0.71 with teacher EU. A head on features alone matches the full head, whereas a head on softmax alone drops clearly.
  • MNIST OOD detection does not come from AU. On MNIST, H[pS]H[p_S] gives 0.301 mean OOD AUROC while our EU head on the same backbone reaches 0.991, and a head trained on teacher AU reaches only about 0.92.

Where AU and EU are harder to separate. On CIFAR-100, the teacher's AU and EU already correlate strongly (0.91 with 5 members, 0.95 with 30), and our student follows (0.94), leaving a smaller extra EU signal (partial Spearman 0.36). Even there, training on a 30-member target over the same frozen 5-member backbone raises partial Spearman with 30-member EU from 0.443 to 0.546, showing the features hold more EU signal than a 5-member target teaches. We will add this table and discussion to the paper.

AuthorsOpenReview ↗

Response to Reviewer m7yU (2/2)

W2, Q1. Fair baselines and Table 2. You are right that the submitted baselines did not get our auxiliary inputs and mostly used one teacher member's backbone. We fixed both in G3, rebuilding all 15 CIFAR-10 variants on our student's backbone or with Tier 2 corrupted inputs. Tier 3 inputs (mixup, masking) have no class label, so they apply only to teacher-student methods, and giving them to GPED, Hydra, and DLF improves all three (G2). After matching, our head leads all 15 baselines in fidelity (at least +0.064 Spearman) and stays at or near the best in AURC, error detection, and OOD detection. Matching also closes some gaps (ViM ties us on CIFAR-10 OOD), and the revision will report the matched versions.

In Table 2, "Best other" took the best of 12 baselines separately per cell, so 5 different methods fill the 8 cells. No single baseline beats our student across metrics. EDL has higher CIFAR-10 error-detection AUROC (0.946 vs. 0.942) but 88% higher AURC (0.0064 vs. 0.0034), and ViM has 0.905 OOD AUROC but 41% higher AURC and only 0.765 on CIFAR-100. We will replace "Best other" with one row per method.

Q2. Why LoRA ensembles, and does this change the interpretation? Large pretrained models are normally fine-tuned rather than trained from scratch, and training 16 copies from scratch is costly. We agree that LoRA members from one pretrained model are less diverse, limiting what those three benchmarks show. We use them only to show that the pipeline scales and keeps accuracy, while Sec. 4, G4, and all new baseline comparisons use scratch ensembles (MNIST, CIFAR-10, CIFAR-100, and 48-member SVHN). Diversity still matters. On Tiny ImageNet, varying LoRA rank, target modules, unfrozen blocks, and data subsets raises member disagreement by 62% over a seed-only teacher and raises OOD AUROC for teacher (0.883 to 0.897) and student (0.834 to 0.860, App. B.5, with confounds listed there). VOC and SST-2 vary initialization scale, label smoothing, dropout, and optimizer settings (App. C.1).

Thank you again. These questions improved the paper, and we would be glad to discuss any of them further.

I thank the authors for the detailed response, which helps clarify the paper.

The fair baselines, however, show that the proposed method's lead on utilities are weak. The one consistent lead is on fidelity, but that is unfair because the method makes that an explicit learning signal, and that alignment with teacher ensemble MI is not useful without high utility, since teacher MI is itself an estimate. Additionally, I find the empirical results undermined by the small and saturated datasets used for the from-scratch ensembles.

Reviewer_UynhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a single network to reproduce an ensemble’s predictions and uncertainty. It first trains the predictor, then freezes its features and fits an uncertainty head. The authors analyse when learning from noisy ensemble targets can improve uncertainty estimates. Experiments across several tasks show substantial inference savings while retaining much of the teacher’s predictive performance.

Critical strengths
  1. The method offers a useful reduction in inference cost. The student runs around 5–16 times faster than the teacher, with relatively small changes in predictive performance. The selective-prediction and monitoring experiments show that its uncertainty estimates can also be useful in practice.

  2. The paper examines when distillation helps and when it fails. The analysis accounts for teacher bias and correlated noise, and the experiments with independently trained ensembles provide useful evidence about the effect of ensemble size.

Critical weaknesses
  1. Some main results use test data to select checkpoints. On Tiny ImageNet, corrupted test images are also used to train the uncertainty head. The held-out reruns in Appendix B.8 are reassuring, but they do not replace all affected comparisons with an evaluation that keeps training, validation and test data separate throughout. This reduces confidence in the smaller reported improvements.

  2. The improvement over existing uncertainty-distillation methods is not fully established. GPED is identified as the closest prior work but is not included in the experiments. Results against other distillation methods are mixed, with EnD performing better on selective prediction and error detection on CIFAR-100. A comparison with GPED under the same training and evaluation conditions is needed to assess what the proposed method adds.

Questions for authors
  1. Could you report the affected main comparisons using separate validation data for all checkpoint selection, including teacher checkpoints, and without test-derived auxiliary inputs? The reruns in Appendix B.8 are helpful, but do not cover all affected comparisons.

  2. How does the method compare with GPED using the same teacher, auxiliary data and comparable student capacity? Since GPED is identified as the closest prior work, this comparison would help establish the contribution of the proposed method.

  3. Were ensemble members evaluated sequentially or in parallel in the timing experiments? If sequentially, how much of the reported speed advantage remains when ensemble members are run in parallel on the same hardware?

3 comments

We thank Reviewer Uynh for highlighting our inference speedups, deployment experiments, and finite-KK analysis. We retrained the full pipeline on a held-out split, compared against GPED [1] under matched conditions, analyzed the CIFAR-100 trade-off with EnD², and timed parallel ensembles. All main metrics hold under held-out retraining, our student beats GPED, and it stays 3.7× to 57.8× faster than a parallel ensemble. G1 to G4 refer to our General Response on top (held-out pipeline, GPED and DLF, matched baselines, tests of Sec. 4).

W1, Q1 (Strict train/validation/test separation, including teacher checkpoints and Tiny ImageNet). See G1. On CIFAR-10 and MNIST, we held out 5,000 training images as a dedicated validation split and retrained all 16 teacher members, Stage 1, and Stage 2 from scratch on the remaining training images, selecting every checkpoint (including each teacher member) on validation. Every metric in G1 stays within 0.010 of the submission (for example, CIFAR-10 mean OOD AUROC is 0.900 ± 0.004 over 3 seeds vs. 0.901). On Tiny ImageNet, building Tier 2 strictly from training images and selecting Stage 2 on held-out validation gives 0.858 ± 0.004 mean OOD AUROC over 5 seeds (App. B.8). We will replace the affected main-table numbers in the revision.

W2, Q2 (Matched GPED comparison and CIFAR-100 trade-off with EnD²). See G2 for GPED. Using the same teacher, auxiliary inputs, and ResNet-18 student capacity, our student outperforms GPED (w=1w = 1 and tuned w=11.2w = 11.2) by +0.013 mean OOD AUROC and +0.144 to +0.148 Spearman on CIFAR-10, and by +0.054 mean OOD AUROC and +0.109 Spearman on CIFAR-100.

On CIFAR-100 (Table 19), when each method's epistemic uncertainty (EU) score is used for every task, EnD² performs better on ID selective prediction (AURC 0.054 vs. 0.059) and ID error detection (AUROC 0.875 vs. 0.853), whereas our EU head performs better on OOD detection (mean AUROC 0.807 vs. 0.798). This reflects what each task requires. Ranking a 100-class classifier's in-distribution errors depends mainly on total predictive confidence rather than EU. Even for the teacher ensemble itself, predictive confidence 1−max⁡pˉ1 - \max \bar p achieves lower AURC than the teacher's own EU (0.045 vs. 0.054), while our EU head already matches the teacher's EU on error detection (0.853 vs. 0.846). Because our student outputs both Stage 1 probabilities pSp_S and the Stage 2 EU head in one forward pass, users can pair 1−max⁡pS1 - \max p_S for ID selective prediction (0.0530 AURC) with the EU head for OOD detection (0.807 AUROC) at no extra cost.

Q3 (Sequential vs. parallel ensemble timing). Table 1 timed the 16 teacher members sequentially at batch size 256. To test parallel execution on the same RTX 4090 GPU, we stacked all 16 members into one batched forward pass via torch.func.vmap. The table below reports our student's speedup over the 16-member teacher across batch sizes.

Benchmark (K=16K = 16)Sequential, batch 256Parallel (vmap), batch 256Parallel (vmap), batch 16Parallel (vmap), batch 1
CIFAR-10 (ResNet-18)16.0×30.7×17.5×5.2×
MNIST (4-layer CNN)15.5×57.8×6.2×3.7×

At batch size 256, one model already saturates GPU compute, so vmap parallelization needs 8.6 GB of peak memory on CIFAR-10 and runs slower than sequential evaluation, making our student 30.7× to 57.8× faster. Parallel execution helps the ensemble most at batch size 1, where our student still runs 3.7× to 5.2× faster. We will add this table to App. D.7.

We hope these results resolve your questions and would appreciate it if you would consider raising your score.

[1] Generalized Bayesian Posterior Expectation Distillation, Vadera et al., UAI 2020

I thank the authors for their rebuttal. However, I will be keeping my score of 2. This is because the idea seems to have largely been done before by other people. Moreover, the use of test data was never justified. To be fair, they reran the experiments without the test data and the results changed only slightly. That said, Tables 1, 2 and 5 in the revised paper still use the models selected on test data, and the full clean results are only in the appendix for now. Furthermore, I believe the new comparisons against parallelized ensembles and GPED do make the paper stronger, but I still believe a score of 2 is appropriate for ICLR.

We thank Reviewer Uynh for the follow-up and for noting that the GPED and parallel ensemble comparisons strengthen the paper.

Test data. We have uploaded a second revision in which CIFAR-10 and MNIST models in Tables 1, 2, and 5 now use validation-selected checkpoints as well, matching Table 3, and all conclusions hold.

Prior work. Your review identified GPED as the closest prior work and asked for a matched comparison to establish what our method adds (W2, Q2). Under the same teacher, auxiliary inputs and student capacity, the comparison shows clear gains over GPED and DLF (Table 3). Beyond the method, the paper shows when a student estimates uncertainty more accurately than its finite-KK teacher, confirmed on trained networks (Table 4), and gives a training-derived auxiliary set that also improves GPED, Hydra and DLF. If specific prior work covers these points, we would be grateful for the reference so that we can discuss it.

Since all three questions from your initial review (Q1 validation-selected main tables, Q2 GPED comparison, and Q3 parallel timing) are agreed to be resolved in the rebuttal, we sincerely ask if you would consider raising your score!

Reviewer_RsATOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for distilling the predictions and epistemic uncertainty of a deep ensemble into a single deterministic student network, allowing both to be estimated in a single forward pass. The student first learns to match the teacher's mean predictions. The backbone is then frozen, and a separate uncertainty head is trained to approximate the ensemble's epistemic uncertainty.

On the theoretical side, the authors model the uncertainty head using Kernel Ridge Regression (KRR) and analyze conditions under which smoothing the estimation noise in uncertainty targets obtained from a finite ensemble can reduce the student's expected squared error.

Critical strengths
  • The paper addresses a practical problem: reducing the inference cost of deep ensembles while retaining their ability to estimate uncertainty.

  • The authors provide a theoretical analysis of conditions under which smoothing by the student can yield a lower expected squared error than the teacher in a KRR setting.

  • Across six benchmarks, the proposed method generally preserves the predictive performance of deep ensembles while improving throughput. The authors also investigate different aspects of the method through ablations and supplementary experiments.

Critical weaknesses
  • GPED (Vadera et al., 2020) already proposed distilling posterior expectations and entropy-based uncertainty into a student model, and the authors acknowledge it as the closest prior work. However, the practical advantages of the proposed two-stage training procedure and uncertainty head over GPED remain unclear. A more explicit discussion of what these design choices add, supported by a direct experimental comparison, would help establish the methodological contribution.

  • DLF (Park et al., 2025), which approximates the output distribution of a teacher ensemble using matrix operations and latent sampling after a single backbone forward pass, is not discussed in the Related Work section. By modeling the mean and covariance of the output distribution, DLF can support the computation of multiple uncertainty measures. In contrast, the proposed method directly predicts a scalar epistemic uncertainty measure. If a different uncertainty measure is needed and cannot be computed from the existing output, new teacher targets must be generated and an additional uncertainty head trained. A comparison with DLF, together with a discussion of the benefits of direct regression despite this difference in representational flexibility, would help clarify the contribution of the proposed approach.

  • Rather than directly computing the teacher's epistemic uncertainty, the method approximates it from the student's learned features. Since the backbone is frozen after prediction distillation, it is unclear whether these features retain sufficient information to capture disagreement among ensemble members. This concern is particularly relevant for out-of-distribution (OOD) inputs, where the uncertainty head may need to generalize beyond the regions covered during training. Additional experiments examining how accurately the student reproduces teacher uncertainty on unseen OOD data would provide stronger support for the approach.

  • The main theoretical analysis considers a population KRR surrogate rather than the proposed neural network itself. Since the actual student is a nonlinear neural network, the theoretical conclusions do not directly carry over to the implemented method. The paper would benefit from a clearer explanation of how the KRR results relate to the training dynamics and uncertainty estimates of the neural network.

  • Theorem 2 assumes that the kernel is independent of the teacher realization. In the proposed method, however, the student backbone is trained using teacher predictions, meaning that the learned features, and hence the corresponding kernel, depend on the teacher. Thus, this independence assumption does not appear to hold for the actual method.

  • The theoretical results largely rely on standard properties of KRR and the usual bias-variance decomposition, which limits their novelty. Moreover, the conditions in Theorem 2 involve the unobservable ideal uncertainty and teacher bias. Although the authors provide empirical verification under certain special settings, it remains difficult to determine whether these conditions hold in general applications or to use them to guide training. This limits the practical implications of the theoretical analysis.

  • Some experiments use test or evaluation data for checkpoint selection or auxiliary data generation, raising concerns about the independence of the evaluation. Although the authors include additional held-out experiments, this separation is not consistently maintained across all settings. Differences in training protocols and backbone performance across baselines may also affect the uncertainty evaluation, making it difficult to isolate the contribution of the proposed uncertainty head. A more consistent evaluation protocol, along with comparisons under controlled training conditions, would strengthen the experimental evidence.

References

  • Vadera, M. P., Jalaian, B., & Marlin, B. M. (2020). Generalized Bayesian posterior expectation distillation for deep neural networks. Proceedings of Machine Learning Research, 124, 719–728.

  • Park, S., Lee, J., Shin, Y., Ohn, I., & Kim, Y. (2025). Knowledge distillation of uncertainty using deep latent factor model. Advances in Neural Information Processing Systems, 38.

Questions for authors
  • GPED and DLF are existing approaches to distilling Bayesian or ensemble uncertainty into a single model. Could the authors compare these methods under the same teacher, backbone, training data, and computational environment, reporting both uncertainty estimation performance and inference cost? Such a comparison would help establish the advantages of the proposed approach.

  • The proposed method and some of the baselines use different auxiliary datasets and data augmentation strategies. Does the performance advantage persist when all methods are evaluated using the same training data and augmentation settings? This would help determine how much of the reported improvement comes from the method itself.

  • It would be useful to know whether the approach remains effective at a larger scale. For example, experiments with models such as ViT-Large, Llama, or Qwen could provide further insight into its scalability in terms of both uncertainty estimation and inference efficiency.

  • Some of the mathematical notation and definitions would benefit from further clarification, as this would make the theoretical sections easier to follow.

2 comments

We thank Reviewer RsAT for the thorough review and for recognizing the practical problem and our analysis. We compared against GPED [1] and DLF [2] under matched conditions, rebuilt all 15 baselines, reran the full pipeline on a held-out split, and tested Theorem 2 on trained networks. Our student beats both methods, and the trained head follows the 1/K1/K predictions of Theorem 2. G1 to G4 refer to our General Response on top (held-out pipeline, GPED and DLF, matched baselines, tests of Sec. 4).

W1, Q1 (GPED). See G2. With the same teacher, backbone, and inputs, our student beats GPED (w=1w = 1 and tuned w=11.2w = 11.2) on CIFAR-10 in AURC (0.0036 vs. 0.0044 and 0.0047), mean OOD AUROC (+0.013), and Spearman (+0.144 to +0.148), and by +0.054 OOD AUROC and +0.109 Spearman on CIFAR-100. G2 also isolates what each design choice adds. Unlike GPED's online SGLD teacher, our method works with cached targets from any offline teacher, and Sec. 4 analyzes finite KK.

W2, Q1 (DLF). We added DLF to Sec. 2 and ran its official code on our teacher (G2). Our student leads DLF by +0.042 mean OOD AUROC and +0.196 / +0.110 Pearson / Spearman, and still leads by +0.021 OOD AUROC and +0.058 / +0.044 fidelity after DLF gets our auxiliary set, while running faster (1.00× vs. 0.82 to 0.94×). On flexibility, another uncertainty measure only needs a new head on the frozen backbone (1 to 2 minutes on cached features), and TU, AU, and EU heads reach Pearson 0.863, 0.845, and 0.811 on CIFAR-10.

W3 (Do frozen features retain ensemble disagreement?). Three checks.

  • Signal beyond softmax probabilities. Controlling for student softmax entropy H[pS]H[p_S], our head keeps a partial Spearman correlation with teacher MI of 0.71 (CIFAR-10, SVHN) and 0.57 (MNIST). A head on features h(x)h(x) alone matches the full head on CIFAR-10 (0.802 Spearman), whereas a head on softmax pS(x)p_S(x) alone drops to 0.627 (CIFAR-10) and 0.591 (SVHN).
  • Capacity beyond a 5-member target. On CIFAR-100, training Stage 2 on a 30-member target over the same frozen 5-member backbone raises partial Spearman from 0.443 to 0.546 and mean OOD AUROC to 0.825 (30-member teacher 0.828), so the features hold more than a 5-member target teaches.
  • Unseen OOD data. Across OOD sets never used in training (6 on CIFAR-10 and SVHN, 5 on MNIST), the held-out head matches the 16-member teacher in mean OOD AUROC (0.900 vs. 0.903, 0.977 vs. 0.974, and 0.992 vs. 0.994).

W4 (KRR surrogate vs. neural head). See G4. Across 36 CIFAR-10 subsets, the trained neural head follows the 1/K1/K law of Theorem 2 (R2=0.991R^2 = 0.991), cuts the teacher's noise slope by nearly half (5.70 vs. 10.79 ×10−3\times 10^{-3}), and its error crosses the teacher's at K=2.69K = 2.69 vs. a predicted K⋆=2.88K^\star = 2.88.

W5 (Teacher-dependent kernel). G4 tests Theorem 2 on an independent backbone, as the theorem assumes. Distilling the backbone from the same KK members, as in our method, gives an even lower MSE ratio (0.78 vs. 0.88 at K=2K = 2 on CIFAR-10, and 1.08 vs. 1.49 at K=5K = 5 on CIFAR-100). Thus feature dependence helps denoising.

W6 (Novelty and practical guidance). Prior KRR analyses of distillation study regularization across self-distillation rounds [5] or noisy one-hot labels [6]. We analyze finite-KK Monte Carlo variance alongside teacher bias in continuous uncertainty targets. While u⋆(x)u^\star(x) cannot be read from one ensemble, retrained ensembles give an independent reference for it (G4), and the fits give practical guidance. A 5-member teacher brings the student within 0.008 OOD AUROC of a 16-member student (saving about 2/3 of teacher cost), and distillation stabilizes uncertainty across retrained ensembles (Spearman 0.902 vs. 0.825).

W7, Q2 (Protocol and matched baselines). See G1 and G3. Our head leads all 15 rebuilt baselines in fidelity and is at or near the best on every metric.

Q3 (Scale). We agree this is an important test, but training ViT-Large or LLM ensembles and their students takes longer than the rebuttal period allows. The paper covers DeiT-Small, SegFormer-B2, and BERT-base (App. F), and since deployment costs one member's pass, the speedup is about KK or higher on all six benchmarks (Table 33).

Q4 (Notation). We will fix the four overloaded symbols (biasK\mathrm{bias}_K, Tk\mathcal{T}_k, ρj\rho_j, mm).

We hope these results address your concerns and would be grateful if you would reconsider your score.

[1] Generalized Bayesian Posterior Expectation Distillation, Vadera et al., UAI 2020
[2] Knowledge Distillation of Uncertainty Using Deep Latent Factor Model, Park et al., NeurIPS 2025
[5] Self-Distillation Amplifies Regularization in Hilbert Space, Mobahi et al., NeurIPS 2020
[6] Understanding Self-Distillation in the Presence of Label Noise, Das & Sanghavi, ICML 2023

Thank you for the detailed rebuttal and additional experiments. The comparisons with GPED and DLF, the experiments using an independent validation set, and the analysis of the neural student have addressed some of my previous concerns. I also understand the practical challenges of conducting additional experiments with large-scale models.

I would appreciate some further clarification on the following points.

W2. Additional Uncertainty Measures

  • The rebuttal mentions that a new uncertainty head can be trained in 1–2 minutes using cached features. Does this mean that all teacher outputs are stored in advance? Storing all possible teacher outputs does not seem practical. Otherwise, estimating a new uncertainty measure may require running all KK teacher models again on a large number of inputs to generate new targets. I would appreciate clarification on whether this additional computational cost is included.
  • Moreover, DLF can compute multiple uncertainty measures from its learned output distribution, whereas the proposed method may require a separate head for each measure. Could you please clarify the practical advantages of the proposed approach in this regard?

W3. Information Preservation in the Frozen Backbone

  • The additional experiments provide useful evidence that the frozen features retain information relevant to predicting the teacher's MI.
  • However, Spearman correlation measures ranking agreement, and the proposed method explicitly uses a ranking loss. Therefore, these results do not necessarily show how accurately the student reproduces the teacher's uncertainty values.
  • Would it be possible to additionally report the MSE or log-space MSE between the teacher's MI and the student's uncertainty estimates, particularly on unseen OOD data?

W5. Teacher-Dependent Kernel

  • The experiment using an independently trained backbone is helpful, as it more closely matches the assumptions of Theorem 2.
  • However, the proposed method trains the backbone and uncertainty head using the same teacher, so the kernel may still depend on the teacher. The improved results with a shared backbone also suggest that this dependence may influence the student's behavior.
  • Could you please clarify to what extent the additional experiments support the applicability of Theorem 2 to the proposed method?
Reviewer_ev26OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a two-stage distillation framework that amortizes Bayesian epistemic uncertainty from a multi-pass teacher (deep ensemble or other Bayesian approximations) into a single deterministic student that outputs both predictions and uncertainty in one forward pass. Theoretically, it models Stage-2 uncertainty regression as population kernel ridge regression with noisy, finite-K Monte Carlo targets, deriving an exact condition under which the student reduces teacher error and analyzing how the optimal regularization depends on K in the presence of teacher bias and correlated noise. Empirically, across image classification, segmentation, and text benchmarks, the student achieves speedups , maintains accuracy close to the teacher, improves or matches label-based uncertainty utility versus single-pass baselines, and exhibits competitive OOD detection and label-free shift monitoring performance.

Critical strengths
  1. The efficiency benefits are clear. The student achieves predictive performance close to the teacher while providing approximately 4.9–16.1× higher inference throughput. The monitoring and routing experiments demonstrate useful deployment applications.
  2. The evaluation is broad. The paper distinguishes fidelity to the teacher from utility measured against ground-truth labels. It includes ablations, independently trained teacher pools, several multi-seed experiments, and supplementary held-out validation results. The appendices also document settings where the theoretical assumptions fail or the method performs less favorably.
  3. Theorems and assumptions are explicitly stated, and the pipeline and roles of losses and auxiliary sets are well-motivated with accessible intuition.
Critical weaknesses
  1. The authors identify GPED as the closest predecessor but do not report a direct experimental comparison. GPED already distills posterior predictive distributions and expected entropy and evaluates uncertainty ranking and OOD detection. Consequently, single-model uncertainty distillation and general downstream usefulness cannot independently establish the novelty of this work.
  2. The comparisons with EnD² and Hydra are useful, but they do not fully establish the benefit of the proposed recipe over direct posterior-expectation distillation under matched teachers, student capacity, auxiliary data, and training budgets. Moreover, Table 5 reports higher mean OOD AUROC for plain MSE than for the complete loss, 0.909 versus 0.901, and limited benefits from Tier-3 inputs. These results suggest trade-offs between metrics rather than consistent benefits from every component.
  3. Section 5.1 and Appendices B.8 and C.1 disclose that some teacher and student checkpoints are selected using test accuracy, several Stage-2 checkpoints are selected using agreement with the teacher on evaluation inputs, and the main Tiny ImageNet experiment uses corrupted evaluation images to train the uncertainty head. These results therefore do not constitute a fully independent evaluation of inductive generalization.
  4. Theorem 2 provides an interpretable analysis of population regression with a fixed kernel. In the actual method, however, the backbone is distilled from the teacher and therefore depends on the teacher realization; the uncertainty head also involves finite width, finite samples, and a ranking objective. Appendix A.1 explicitly acknowledges these differences. The empirical NTK diagnostic in Appendix B.4 additionally finds amplification in 52.3% of components, rather than the coordinate-wise shrinkage of kernel ridge regression.
  5. Comparisons against distillation baselines are limited in breadth (primarily EnD² and Hydra) and settings; newer hybrid or self-distillation-based uncertainty estimators are not extensively explored.
Questions for authors
  1. Can you provide a stricter experiment variant where all model selection uses a held-out validation split (no evaluation data, no corrupted test images)?
  2. How sensitive is performance to the exact composition and size of auxiliary tiers across datasets, particularly Tiny ImageNet and CIFAR-100?
  3. Have you tried simple post-hoc recalibration of the student’s MI scale (e.g., monotone isotonic mapping on a small labeled calibration set) to mitigate OOD scale drift? Does this help maintain absolute scale without hurting rank-based utility?
  4. To what extent do the conclusions hold if the teacher is not an ensemble but a different Bayesian approximation (e.g., SWAG/Laplace) providing MI-like proxies?
  5. Could you provide a matched comparison with GPED-style posterior-expectation distillation on one representative benchmark? Alternatively, please precisely identify the algorithmic differences from that framework and explain how the existing experiments isolate their benefits.
  6. How are the regularization coefficients in Table 11 selected for each ensemble size? Is the independent reference ensemble used exclusively for final evaluation? For the neural uncertainty head, which observations validate specific predictions of the theory beyond qualitative consistency with a bias–variance trade-off?
1 comment

We thank Reviewer ev26 for the careful reading of the paper and appendices and for noting our broad evaluation and explicit assumptions. We compared against GPED [1] and DLF [2] with a step-by-step breakdown, reran the full pipeline on a held-out split, tested Theorem 2 with zero tuning on the reference, and ran the requested ablations. Each design choice adds a measurable gain, and the trained head follows the quantitative predictions of Theorem 2. G1 to G4 refer to our General Response on top (held-out pipeline, GPED and DLF, matched baselines, tests of Sec. 4).

W1, W5, Q5 (Matched GPED and DLF comparisons, algorithmic differences, and novelty). See G2. Under matched teachers, backbones, and inputs, our student outperforms GPED (+0.013 mean OOD AUROC and +0.144 to +0.148 Spearman on CIFAR-10, and +0.054 and +0.109 on CIFAR-100) and DLF with our auxiliary set (+0.021 OOD AUROC and +0.044 Spearman). The step table in G2 isolates our three algorithmic differences from GPED. The auxiliary set gives the largest OOD gain, the log-space and ranking loss the largest fidelity gain, and two stages add more on CIFAR-10 and CIFAR-100. Beyond the pipeline, the paper contributes the finite-KK denoising analysis verified in G4 and a training-derived auxiliary set that also improves GPED, Hydra, and DLF.

W2 (Loss trade-offs and Tier 3 role in Table 5). We agree that the two Stage 2 loss terms serve complementary goals. All four loss variants in Table 37 stay within 0.01 mean OOD AUROC (plain MSE 0.909 vs. our loss 0.901), while adding the ranking term raises Spearman fidelity from 0.650 to 0.799 (Table 5) and keeps student ranking closer to the reference ensemble than its teacher's at every KK (G4). We also agree that Tier 3 adds little on CIFAR-10. On CIFAR-100, Tier 1+2 matches all three tiers at equal total size (0.801 vs. 0.798), so we will present Tier 3 as optional.

W3, Q1 (Strict held-out evaluation). See G1. Full held-out retraining, including all teacher members, changes every metric by at most 0.010, and Tiny ImageNet with training-derived Tier 2 and held-out Stage 2 selection gives 0.858 ± 0.004 (App. B.8).

W4, Q6 (Table 11 λ\lambda selection, quantitative neural-head predictions, and NTK diagnostics).

  • Regularization selection in Table 11. Here λ\lambda was selected against the reference. Repeating with 20-split cross-fitting (selecting on half of the test inputs and evaluating on the other half) changes values by less than 0.02×10−30.02 \times 10^{-3}, and fixing λ=0\lambda = 0 (Table 13) still beats the teacher at K=2,3K = 2, 3. All new tests in G4 do zero tuning on the reference, using it strictly for evaluation.
  • Quantitative predictions for the neural head. G4 tests three predictions of Theorem 2, error linear in 1/K1/K, a smaller student slope, and the crossover K⋆K^\star. Both ridge and our neural head fit the 1/K1/K law (R2R^2 0.979 to 0.997) and cut the noise slope by 34% to 47%, and the predicted K⋆K^\star is close to the measured one (2.88 vs. 2.69 for our CIFAR-10 head).
  • Teacher-dependent backbone and NTK modes. Sharing the teacher between Stage 1 and Stage 2 lowers small-KK MSE further than an independent backbone (0.78 vs. 0.88 at K=2K = 2). In App. B.4, the largest amplification ratios occur where teacher target energy is near zero. Weighted by mode energy, the head's empirical NTK contracts the overall score to 0.745, matching Theorem 1(a) in aggregate.

Q2 (Auxiliary tier composition and size). Clean Tier 1 images alone lower mean OOD AUROC by 0.018 (CIFAR-10), 0.035 (CIFAR-100), and 0.537 (MNIST). On CIFAR-10 and MNIST, halving or quartering the auxiliary set changes mean OOD AUROC by at most 0.003, and replacing it with external Tiny ImageNet images adds only 0.001. On MNIST, removing any one of the 8 corruptions changes it by at most 0.004.

Q3 (Isotonic recalibration). Without recalibration, the head's mean output across the 6 CIFAR-10 OOD sets is already 0.90× to 1.05× teacher mean MI, showing little scale drift after one distillation round. Fitting an isotonic map on held-out validation lowers ID test MSE by 7.3% (0.00468 to 0.00434).

Q4 (Other Bayesian teachers). App. B.10 (Table 18) distills MC Dropout, SWAG, and last-layer Laplace teachers. In each case, the student's mean OOD AUROC exceeds its teacher's (for example, 0.915 vs. 0.863 for SWAG).

We hope these results resolve your concerns and would be grateful if you would consider raising your score.

[1] Generalized Bayesian Posterior Expectation Distillation, Vadera et al., UAI 2020
[2] Knowledge Distillation of Uncertainty Using Deep Latent Factor Model, Park et al., NeurIPS 2025

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response (1/3). Three Questions the Paper Studies and Planned Revisions

We thank all reviewers for their constructive feedback. We ran the requested experiments, including a from-scratch held-out pipeline, matched comparisons with GPED [1] and DLF [2], 15 baselines rebuilt on our backbone and data, and a pre-registered 48-member SVHN benchmark trained from scratch. Beyond compressing an ensemble, our paper studies three questions about distilling noisy uncertainty targets, clarified below with new evidence and shared updates G1 to G4. Full tables and code will be in the revision.

1. When can a student estimate uncertainty more accurately than its teacher? (Sec. 4) A KK-member teacher's mutual information (MI) uT(x)u_T(x) is a Monte Carlo estimate of the ideal uncertainty u⋆(x)u^\star(x), so it changes when the ensemble is retrained. Stage 2 fits a head on frozen features by regularized regression, smoothing out part of this finite-KK noise along with part of the signal. Under consequence (ii) of Theorem 2, teacher squared error to u⋆(x)u^\star(x) scales as aT+bT/Ka_T + b_T/K, while the student keeps a smaller noise slope bS<bTb_S < b_T. If aS>aTa_S > a_T, the student is closer to u⋆(x)u^\star(x) than its teacher for K<K⋆=(bT−bS)/(aS−aT)K < K^\star = (b_T - b_S)/(a_S - a_T). G4 verifies this on trained networks. Two practical benefits follow. A student distilled from 5 members is within 0.008 mean OOD AUROC of one distilled from 16 on CIFAR-10, and students from two independently trained 16-member teachers agree more closely (Spearman 0.902) than the two teachers do (0.825).

2. Why Stage 2 needs inputs beyond clean training images (Sec. 3.4) Ensemble members nearly always agree on clean training images (Tier 1). On CIFAR-10, mean teacher MI is 0.023 on clean images versus 0.27 on corrupted (Tier 2) and mixed or masked (Tier 3) inputs, so a head trained on clean images rarely sees high targets. Using clean images alone lowers mean OOD AUROC by 0.018 (CIFAR-10), 0.035 (CIFAR-100), and 0.537 (MNIST). Adding our auxiliary set to GPED, Hydra, and DLF also raises their mean OOD AUROC by 0.02 to 0.47. Prior work [3, 4] noted a related issue for distilling predictive distributions, using perturbation search and mixup. Our set uses only training-set transforms, targets scalar uncertainty, and transfers across distillation methods.

3. Why two-stage training instead of joint training (Sec. 3.2) We agree that the joint-training row in Table 5 was not a matched comparison because it used clean images only without the ranking loss. Under identical inputs, head loss, and validation selection, two-stage training still beats joint training on CIFAR-10 (+0.017 mean OOD AUROC, +0.031 Pearson fidelity) and across total, aleatoric, and epistemic uncertainty (TU, AU, EU) on CIFAR-100 (+0.014 to +0.036 OOD AUROC, +0.059 to +0.075 Pearson). Decoupling also needs no joint loss weight and keeps the classifier fixed. A new uncertainty measure only needs a new head on cached features (1 to 2 minutes), and TU, AU, and EU heads reach Pearson 0.863, 0.845, and 0.811 on CIFAR-10.

Revisions. We will (1) replace the joint row of Table 5 with the matched comparison above and update Sec. 3.2, (2) report G1 in the main tables, (3) add the GPED and DLF comparisons of G2, (4) expand "Best other" in Table 2 into one row per method, and (5) cite [2, 3, 4] in Sec. 2.

[1] Generalized Bayesian Posterior Expectation Distillation, Vadera et al., UAI 2020
[2] Knowledge Distillation of Uncertainty Using Deep Latent Factor Model, Park et al., NeurIPS 2025
[3] Diversity Matters When Learning From Ensembles, Nam et al., NeurIPS 2021
[4] Functional Ensemble Distillation, Penso et al., NeurIPS 2022

AuthorsOpenReview ↗

General Response (2/3). Fully Held-Out Pipeline and Matched GPED and DLF Comparison

G1. Fully held-out pipeline (RsAT W7, Uynh W1 and Q1, ev26 W3 and Q1) On CIFAR-10 and MNIST, we held out 5,000 training images for validation and retrained all 16 teacher members, Stage 1, and Stage 2 from scratch (3 seeds), selecting every checkpoint including each teacher member on this validation split. Fidelity is correlation with teacher MI.

SettingMiscls. AUROC ↑Mean OOD ↑Pearson / Spearman ↑CIFAR-10-C monitoring ↑
CIFAR-10, submitted0.9420.9010.815 / 0.7990.932
CIFAR-10, rebuttal held out0.9430.9000.812 / 0.8090.931
MNIST, submitted0.9730.9910.818 / 0.805n/a
MNIST, rebuttal held out0.9770.9920.828 / 0.806n/a

Every metric moves by at most 0.010. On CIFAR-10-C shift monitoring, the held-out head still has the highest power (0.931), above every training-free score on the same backbone in all 3 seeds (best is Energy, 0.921) and the 16-member teacher (0.923). On Tiny ImageNet, building Tier 2 from training images and selecting Stage 2 on held-out validation gives 0.858 ± 0.004 mean OOD AUROC over 5 seeds (App. B.8).

G2. GPED and DLF under matched conditions (RsAT W1, W2 and Q1, Uynh W2 and Q2, ev26 W1, W2, W5 and Q5) GPED [1] uses an online SGLD teacher, so on our 16-member CIFAR-10 teacher and ResNet-18 we implemented a GPED-style baseline that regresses pˉ\bar p, TU, AU, and MI end to end (weight w=1w = 1 or tuned w=11.2w = 11.2). For DLF [2] we ran the official code.

CIFAR-10 (K=16K = 16)Acc. (%) ↑AURC ↓OOD ↑Pearson / Spearman ↑Speed ↑
Teacher (16 passes)96.960.00230.9031 / 10.06×
GPED-style, w=1w = 1 (3 seeds)95.930.00440.8910.744 / 0.6601.00×
GPED-style, w=11.2w = 11.2 (3 seeds)95.700.00470.8910.723 / 0.6561.00×
DLF95.590.00430.8620.615 / 0.6940.82 to 0.94×
DLF + auxiliary set95.810.00390.8830.753 / 0.7600.82 to 0.94×
EnD² head + auxiliary set (Table 3)95.720.00380.8970.684 / 0.7731.00×
Hydra + auxiliary set96.210.00370.8960.753 / 0.6720.98×
Ours (3 seeds)95.780.00360.9040.811 / 0.8041.00×

Our student has the lowest AURC (18% below GPED), highest OOD AUROC, and highest fidelity at single-pass speed, and beats GPED on CIFAR-100 by +0.054 OOD AUROC and +0.109 Spearman. The table below adds our three design choices to clean-image GPED-style regression one at a time.

StepMetricCIFAR-10MNISTCIFAR-100
1. Add auxiliary setMean OOD AUROC+0.018+0.473+0.095
2. Log-space and ranking loss, validation selection (still end to end)Spearman+0.173+0.192+0.080
3. Two stages instead of joint trainingMean OOD / Pearson+0.017 / +0.031tie+0.036 / +0.059

[1] Generalized Bayesian Posterior Expectation Distillation, Vadera et al., UAI 2020
[2] Knowledge Distillation of Uncertainty Using Deep Latent Factor Model, Park et al., NeurIPS 2025

AuthorsOpenReview ↗

General Response (3/3). Matched Baselines and Tests of Section 4

G3. Baselines under matched conditions (RsAT W7 and Q2, m7yU W2 and Q1) In the submission, most baselines used one teacher member's backbone and clean images only. On CIFAR-10 we rebuilt all 15 variants. Post-hoc and density scores now use our student's backbone, and EDL, SNGP, and MC Dropout are trained with Tier 2 corrupted inputs. The last three columns are reported for training-free scores.

Method (matched)AURC ↓Miscls. ↑Spearman ↑CIFAR-10 OOD ↑CIFAR-100 OOD ↑MNIST OOD ↑CIFAR-10-C ↑
MC Dropout + Tier 20.00480.9420.6400.895n/an/an/a
EDL / SNGP + Tier 2 (best)0.00640.9250.6380.870n/an/an/a
DDU / LLLA / DEUP on student (best)0.00380.9400.7400.894n/an/an/a
Softmax entropy H[pS]H[p_S]0.00330.9450.5520.8780.8070.3010.911
Energy on student0.00380.9390.5030.8880.8130.2870.921
ViM on student0.00490.9180.6360.9060.7650.9950.888
Ours0.00360.9440.8040.9040.8070.9920.931

Our head leads all 15 in fidelity (at least +0.064 Spearman) and is at or near the best in every column. Every training-free score fails somewhere, Energy and H[pS]H[p_S] on MNIST OOD (0.287, 0.301) and ViM in CIFAR-10 AURC and CIFAR-100 OOD (0.765).

G4. Testing Section 4 on trained networks (RsAT W4 to W6, ev26 W4 and Q6) We measure log⁡(1+u)\log(1+u) squared error against an independent evaluation-only reference ensemble, with zero tuning on it. CIFAR-10 uses 36 random subsets (K∈{2,3,4,5,8,12}K \in \{2,3,4,5,8,12\}, 6 per KK) on a backbone distilled from a separate ensemble, as Theorem 2 assumes. Pre-registered SVHN uses 48 scratch members (6 pools of 8). A ratio below 1 means the student is closer to the reference than its teacher.

Modelaa (10−310^{-3})bb (10−310^{-3})R2R^2K⋆K^\star pred. / meas.Ratio at K=2K = 2 / 33
CIFAR-10 teacher1.0210.790.997n/an/a
CIFAR-10 ridge2.405.910.9793.53 / 4.230.87 / 0.92
CIFAR-10 our head2.795.700.9912.88 / 2.690.88 / 1.04
SVHN teacher0.313.880.998n/an/a
SVHN ridge0.752.260.9883.71 / 3.950.86 / 0.89
SVHN head, log loss only0.432.540.99711.1 / above 80.77 / 0.77

All errors are linear in 1/K1/K, the student cuts the noise slope by 34% to 47%, and predicted K⋆K^\star is close to the measured crossing. Distilling the backbone from the same KK members lowers the ratio to 0.78 at K=2K = 2 and 0.84 at K=3K = 3 on CIFAR-10 (5 runs), and to 1.08 vs. 1.49 for an independent backbone at K=5K = 5 on CIFAR-100 (5 pairs). With the ranking loss, the student also ranks closer to the reference than its teacher at every KK (Spearman +0.201 to +0.011 on CIFAR-10, +0.134 to +0.045 on SVHN).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.