- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a post-hoc UQ method that leaves a classifier’s predictions unchanged. Its starting point is that class labels identify predictive probabilities, but do not directly identify the spread of a distribution over those probabilities. Standard evidential objectives nevertheless learn both together, which can affect the predictive mean. BEDI freezes a temperature-scaled classifier and fits the dispersion of a Dirichlet distribution around its probabilities. It uses feature-space effective sample size to define a dispersion prior and last-layer Laplace disagreement to construct pseudo-observations. A hierarchical model combines these signals and predicts dispersion for new inputs. The experiments evaluate OOD detection across several modalities and include a controlled study of missing training support versus label ambiguity.
Critical strengths
- The paper makes a useful distinction between estimating predictive probabilities and estimating uncertainty about those probabilities. Proposition 4.1 gives a concrete example of how controlling dispersion through the expected-MSE evidential objective can distort the predictive mean.
- Keeping the classifier fixed is a practical advantage. BEDI can add uncertainty estimates to a pretrained model while preserving its probabilities and decisions by construction.
- The method gives training support and predictive disagreement separate roles. This makes the uncertainty estimate easier to examine than an evidence output learned only from class labels. The ablation in Table 13 shows that combining the signals generally improves OOD detection over using either alone.
- The two-moons study is a useful part of the evaluation. It asks whether uncertainty responds to missing information rather than merely following class ambiguity. This tests the intended behaviour more directly than reporting OOD detection scores alone.
Critical weaknesses
-
Theorem 4.4 varies relative effective sample size while holding the predictive mean, representation, kernel residual, and fitted parameter posterior fixed. The resulting decrease in uncertainty follows from the imposed negative dependence on effective sample size in eq. 4. This is a desirable property, but it does not show what happens when more data are collected and the method is refitted. Since effective sample size is normalised by its training set average, increasing the dataset need not increase the relative quantity used by BEDI. The discussion in Section 4 should distinguish this monotonicity result from a statistical contraction guarantee.
-
The main experiments do not clearly establish what the hierarchical inference adds beyond combining the two signals. This matters because the proposed inference procedure accounts for a substantial part of the method’s complexity.
-
Some reported improvements should be attributed to the base classifier rather than dispersion inference. BEDI preserves the predictive mean, so its classification accuracy is inherited from the underlying predictor. Section 5.1 also defines misclassification detection using total predictive uncertainty, which eq. 11 identifies as the entropy of that fixed mean. Dispersion inference therefore does not change this detection score. These metrics are useful for showing that predictive behaviour is preserved, but the evidence for the new uncertainty model comes primarily from OOD detection and the controlled study.
-
The reliability weighting of the disagreement signal needs more justification. The paper uses the mismatch between the Laplace-derived covariance and its Dirichlet approximation to determine the pseudo-likelihood variance. However, fitting the Dirichlet covariance structure well does not establish that the underlying Laplace approximation accurately captures model uncertainty. An inaccurate disagreement estimate could still have a small projection mismatch and receive substantial weight. The paper should explain more clearly what this reliability measure captures and what errors it cannot detect.
-
Fixing the predictive mean also restricts the uncertainty that BEDI can express. Eq. 11 bounds epistemic uncertainty by the entropy of the frozen predictive probabilities. If the classifier is highly confident on an unfamiliar input, the epistemic score cannot become large even when the support estimate indicates little relevant training information. Preserving predictions is useful, but this limitation is directly relevant to confidently wrong predictions. The main evaluation does not make clear how the method behaves in those cases.
Questions for authors
- Is Theorem 4.4 intended only as a monotonicity result for the fitted model? What supports the broader interpretation of contraction as training information increases?
- Is there a comparison with a simpler dispersion regressor using the same support features and disagreement-derived targets? Which result most directly isolates the contribution of hierarchical inference?
- How should the covariance projection mismatch be interpreted as a reliability measure when the Laplace approximation itself may be inaccurate?
- Can the existing OOD results be broken down by the base classifier’s predictive confidence to assess the fixed-mean restriction?
7 comments
(2. Contribution of hierarchical dispersion inference beyond combining the two signals)
We thank the reviewer for suggesting this comparison. To assess what hierarchical inference adds beyond combining ESS and Bayesian disagreement, we added direct and reliability-weighted dispersion regressors using the same support features, disagreement-derived targets, and functional form for the dispersion mean. The reliability-weighted regressor provides the most direct comparison: it already combines both signals and uses the same inverse marginal-variance weights as BEDI’s collapsed pseudo-likelihood. BEDI achieves an average OOD/shift detection AUPR of 88.07%, compared with 85.71% for direct regression and 85.62% for reliability-weighted regression.
These comparisons complement Table 13, which examines the value of combining ESS and Bayesian disagreement. Here, we assess what hierarchical inference adds when both signals are already available.
Matched regression alternatives. All variants use the same frozen classifier checkpoints, data splits, calibration procedure, support-feature construction, disagreement-target construction, and evaluation protocol. They share the mean function
where is the normalized frozen classifier representation and is the relative ESS. With the disagreement target , the regression data-fitting objectives are
=\frac{1}{N}\sum_i\left[t_i-\mu_\theta(x_i)\right]^2, \qquad \mathcal L_{\mathrm{weighted}}(\theta) =\frac{1}{N}\sum_i\frac{\left[t_i-\mu_\theta(x_i)\right]^2}{V_i}, \quad V_i=\tau_i^2+\sigma_{\mathrm{LA},i}^2$$ Thus, the reliability-weighted regressor uses $w_i^{\mathrm{reg}}=1/V_i$, matching Eq. (9). The support-kernel residual and covariance projection mismatch enter through $\tau_i^2$ and $\sigma_{\mathrm{LA},i}^2$, respectively. *What hierarchical inference adds.* The regressors fit point estimates of the shared parameters. BEDI instead combines the collapsed observation model $t_i\mid\theta,x_i\sim\mathcal N(\mu_\theta(x_i),V_i)$ with a parameter prior, learns a variational posterior, and propagates it through test-time dispersion inference. **The comparison evaluates this hierarchical procedure as a whole, including prior regularization and posterior averaging.** **Table R2. Hierarchical inference versus matched dispersion regressors.** OOD/shift detection AUPR (%) averaged over each setting's OOD/shift datasets. Entries are mean ± standard deviation over five seeds for CIFAR and ImageNet and three seeds for OpenBookQA; Dry Bean aggregates three held-out-class settings × five seeds. BEDI values are from the evaluation runs used for these comparisons and differ slightly from those reported in the main paper. Bold indicates the highest mean in each column. Higher is better. | **Method** | **CIFAR-10 ↑** | **CIFAR-10-LT ↑** | **CIFAR-100 ↑** | **ImageNet ↑** | **Dry Bean ↑** |**OpenBookQA ↑** | **Avg. ↑** | |:---|---:|---:|---:|---:|---:|---:|---:| | Direct regression | 91.69 ± 1.0 | 78.55 ± 1.1 | **85.61 ± 0.5** | 86.41 ± 0.7 | **92.07 ± 4.2** | 79.95 ± 6.2 | 85.71 | | Reliability-weighted regression | **91.79 ± 1.0** | 79.18 ± 1.2 | 84.49 ± 0.5 | 86.09 ± 0.9 | 91.88 ± 4.2 | 80.26 ± 6.1 | 85.62 | | BEDI | 91.71 ± 1.0 | **79.43 ± 1.1** | 85.22 ± 0.6| **88.62 ± 0.2** | 91.73 ± 4.1 | **91.74 ± 4.1** | **88.07** | **Results and interpretation.** BEDI’s largest gains occur in pretrained-model settings, a central use case for its post-hoc uncertainty estimation. On ImageNet and OpenBookQA, BEDI improves AUPR over reliability-weighted regression by 2.53 and 11.48 points, and over direct regression by 2.21 and 11.79 points, respectively. Even with identical support features, disagreement targets, and reliability weights, the reliability-weighted regression falls short of BEDI’s average detection performance. **These gains demonstrate the practical importance of hierarchical dispersion inference as a key component of BEDI in pretrained-model settings, beyond combining and weighting the two signals.**Computational cost. Using cached features and pseudo-targets, we timed dispersion-head fitting for 200 optimization iterations on an NVIDIA RTX A6000.
Table R3. Dispersion-head fitting time. Median wall-clock time over five repeats on one checkpoint per setting. Feature extraction, support estimation, and disagreement-target construction are excluded. Lower is better.
| Method | ImageNet (s) ↓ | OpenBookQA(s) ↓ |
|---|---|---|
| Direct regression | 1.04 | 1.40 |
| Reliability-weighted regression | 1.12 | 1.34 |
| BEDI (default) | 1.51 | 1.76 |
Relative to reliability-weighted regression, BEDI adds 0.39 seconds on ImageNet and 0.42 seconds on OpenBookQA. This adds less than half a second to one-time dispersion-head fitting in both settings.
These comparisons show that BEDI’s hierarchical inference improves average detection performance beyond combining and weighting the two signals, with modest additional head-fitting cost in the two timed settings. We will include these comparisons in Appendix E.5 and summarize the benefit of hierarchical inference over direct and reliability-weighted regression in Section 5.2.
(3. Attribution of prediction-related performance and the role of these metrics)
We thank the reviewer for this comment. BEDI’s classification accuracy and entropy-based misclassification-detection performance are inherited from the calibrated base classifier, rather than improved by dispersion inference. BEDI preserves the calibrated predictive mean, and the total uncertainty used for misclassification detection is its entropy, , as specified in Eq. (11).
Why we report these metrics. A central motivation for BEDI is to avoid interference between predictive-mean learning and dispersion inference. Proposition 4.1 establishes a predictive-mean distortion mechanism for the standard EDL expected-MSE objective. BEDI’s fixed-mean construction prevents dispersion fitting altering the calibrated predictive mean.
We report classification accuracy, misclassification detection, and calibration to show the predictive behavior retained by this separation. The evidence for the learned dispersion comes from EU-based OOD/shift detection and the controlled uncertainty studies, rather than improvements in these inherited prediction-related metrics.
We will clarify this attribution in Sections 5.1 and 5.2 and align the corresponding descriptions in Appendix E.
Thank you for your careful review of our paper and for the insightful and constructive comments. Building on the two-moons study, which you highlighted as a strength, we have added a comparison with a near-exact Bayesian reference to further strengthen our direct validation of epistemic uncertainty (Tables R8 and R9 in our response to Reviewer mU8W’s Comment 4).
We will upload a revised manuscript incorporating the additional results, analyses, and clarifications described below before the end of the rebuttal period. Below, we provide detailed responses to the weaknesses and questions raised in your review.
(1. Scope of Theorem 4.4 and uncertainty reduction under model refitting)
We thank the reviewer for highlighting this important distinction. Theorem 4.4 establishes conditional monotonicity with respect to relative ESS for a fixed BEDI model, rather than statistical contraction as training data increase and the model is refitted. To examine how BEDI’s uncertainty changes as training data increase under full refitting, we conducted an additional experiment on progressively larger CIFAR-10 training sets. Mean epistemic uncertainty decreases consistently across the evaluated training-set sizes.
What Theorem 4.4 establishes. The theorem varies relative ESS while holding the predictive mean, representation, kernel residual, and fitted parameter posterior fixed. Although Eq. (4) imposes a decreasing relationship between relative ESS and the prior logit-dispersion mean, the theorem establishes that this ordering also holds for BEDI’s predictive epistemic uncertainty after integrating over dispersion and the fitted parameter posterior. Establishing statistical contraction under full refitting would require additional assumptions and control over how the learned representation, ESS normalization, and dispersion posterior change as data accumulate. We therefore distinguish the theorem’s conditional monotonicity from the empirical behavior under full refitting.
Relative ESS under increasing training data. As the reviewer notes, increasing training-set size need not increase relative ESS, because its training-set normalization also changes. However, relative ESS can increase when the ESS of held-out inputs grows proportionally faster than the training-set average. This may occur as additional data improve coverage of previously less-supported inputs. Whether this occurs under full refitting is an empirical question, which we examine below.
Evaluation under full refitting. Motivated by prior studies examining how epistemic uncertainty changes with training-data availability [1, 2], we used class-stratified CIFAR-10 subsets with N∈1,000,2,000,5,000,10,000,25,000,47,500. At each size, we retrained VGG-16 and refitted temperature scaling, support estimation, Laplace disagreement, and hierarchical dispersion inference. Each fitted model was evaluated on the same ID test set over five seeds.
Table R1. BEDI under increasing training-set size and full refitting. All quantities are averaged over the fixed CIFAR-10 test set. Raw ESS is reported as a seed mean; relative ESS, dispersion, and EU are reported as mean ± standard deviation over five seeds.
Results and interpretation. Mean EU, averaged over five seeds, decreases with each increase in training-set size, from 0.233 ± 0.036 to 0.020 ± 0.002, accompanied by decreasing mean dispersion from 0.088 to 0.011. Raw ESS increases from 9.9 to 497.4, and relative ESS increases from 0.49 to 0.95. Thus, in this experiment, held-out inputs become better supported relative to the training-set average despite the changing normalization.
These results provide empirical evidence of epistemic uncertainty reduction under full refitting, complementing the theorem’s conditional monotonicity for a fixed model.
We will rename Theorem 4.4 as Conditional monotonicity of epistemic uncertainty in relative effective sample size, update Section 4 and Appendix B.4, and revise the corresponding contraction wording in the abstract and conclusion. We will include the full-refitting experiment in Appendix E.7.
References
[1] Maohao Shen et al. Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? NeurIPS 2024.
[2] Taeseong Yoon and Heeyoung Kim. Uncertainty Estimation by Flexible Evidential Deep Learning. NeurIPS, 2025.