- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to distill a deep ensemble into a single model, and keep the aleatoric-epistemic uncertainty estimators by additionally adding an uncertainty head that distills the mutual information of the teacher ensemble on augmented inputs.
Critical strengths
- The paper is well written, it is easy to understand and presented clearly.
- The appendix contains extensive supplementary results.
Critical weaknesses
-
The method can be seen as distilling a deep ensemble into a Treenet where one head predicts . However, the idea that a single pass can somehow capture epistemic and aleatoric uncertainty is not supported theoretically. The student's uncertainty estimation works in the evaluation provided can perhaps be explained by the high empirical correlation between aleatoric and epistemic uncertainty, hence
-
The proposed distillation method relies on multiple augmentation methods to train the uncertainty estimator, but those augmentations were not used for baselines, which suggests a potentially unfair comparison. Even then the reported results are not very strong on the utility table (with the proposed method losing to baselines on 7/8 Miscls/OOD AUROC in table 2).
Questions for authors
- How did you ensure fairness when comparing your distillation method to the baselines, did you match the amount of augmentation in training?
- I noticed that for three of the datasets you evaluated on, you start with a single pre-trained model, and use LoRA fine-tuned checkpoints as ensemble members to distill from. Generally, deep ensembles require members to be trained from different random initialization so they can provide the necessary predictive diversity useful for uncertainty quantification. Why did you make that experimental decision, and does it affect the interpretation of your reported results?
- Can you add experiment to evaluate the entanglement of aleatoric and epistemic uncertainty given by your distilled student model?
3 comments
Response to Reviewer m7yU (2/2)
W2, Q1. Fair baselines and Table 2. You are right that the submitted baselines did not get our auxiliary inputs and mostly used one teacher member's backbone. We fixed both in G3, rebuilding all 15 CIFAR-10 variants on our student's backbone or with Tier 2 corrupted inputs. Tier 3 inputs (mixup, masking) have no class label, so they apply only to teacher-student methods, and giving them to GPED, Hydra, and DLF improves all three (G2). After matching, our head leads all 15 baselines in fidelity (at least +0.064 Spearman) and stays at or near the best in AURC, error detection, and OOD detection. Matching also closes some gaps (ViM ties us on CIFAR-10 OOD), and the revision will report the matched versions.
In Table 2, "Best other" took the best of 12 baselines separately per cell, so 5 different methods fill the 8 cells. No single baseline beats our student across metrics. EDL has higher CIFAR-10 error-detection AUROC (0.946 vs. 0.942) but 88% higher AURC (0.0064 vs. 0.0034), and ViM has 0.905 OOD AUROC but 41% higher AURC and only 0.765 on CIFAR-100. We will replace "Best other" with one row per method.
Q2. Why LoRA ensembles, and does this change the interpretation? Large pretrained models are normally fine-tuned rather than trained from scratch, and training 16 copies from scratch is costly. We agree that LoRA members from one pretrained model are less diverse, limiting what those three benchmarks show. We use them only to show that the pipeline scales and keeps accuracy, while Sec. 4, G4, and all new baseline comparisons use scratch ensembles (MNIST, CIFAR-10, CIFAR-100, and 48-member SVHN). Diversity still matters. On Tiny ImageNet, varying LoRA rank, target modules, unfrozen blocks, and data subsets raises member disagreement by 62% over a seed-only teacher and raises OOD AUROC for teacher (0.883 to 0.897) and student (0.834 to 0.860, App. B.5, with confounds listed there). VOC and SST-2 vary initialization scale, label smoothing, dropout, and optimizer settings (App. C.1).
Thank you again. These questions improved the paper, and we would be glad to discuss any of them further.
I thank the authors for the detailed response, which helps clarify the paper.
The fair baselines, however, show that the proposed method's lead on utilities are weak. The one consistent lead is on fidelity, but that is unfair because the method makes that an explicit learning signal, and that alignment with teacher ensemble MI is not useful without high utility, since teacher MI is itself an estimate. Additionally, I find the empirical results undermined by the small and saturated datasets used for the from-scratch ensembles.
Response to Reviewer m7yU (1/2)
We thank Reviewer m7yU for the clear and focused questions and the kind words on the writing and appendix. We tested whether our EU head only reflects aleatoric uncertainty, rebuilt all baselines under matched data and backbones, and clarified the role of the LoRA ensembles, reporting also where the evidence is weaker. G1 to G4 refer to our General Response on top (held-out pipeline, GPED and DLF, matched baselines, tests of Sec. 4).
W1, Q3. Can one forward pass estimate epistemic uncertainty, or does it only reflect aleatoric correlation? We agree that one forward pass does not sample a posterior at test time, and our claim is narrower. Once the K-member teacher is trained, its MI uT(x) (Eq. 3) is a deterministic function of x. Stage 2 learns this function by regression from frozen features, as in amortized inference, and Sec. 4 analyzes when this regression is accurate.
To test whether the EU head only reflects the correlation between aleatoric (AU) and epistemic (EU) uncertainty, we ran the checks below (ID test sets, Spearman unless noted).
Where AU and EU are harder to separate. On CIFAR-100, the teacher's AU and EU already correlate strongly (0.91 with 5 members, 0.95 with 30), and our student follows (0.94), leaving a smaller extra EU signal (partial Spearman 0.36). Even there, training on a 30-member target over the same frozen 5-member backbone raises partial Spearman with 30-member EU from 0.443 to 0.546, showing the features hold more EU signal than a 5-member target teaches. We will add this table and discussion to the paper.