acceptodds
Under review as a conference paper at ICLR 2027

Separating Prediction from Epistemic Uncertainty in Evidential Deep Learning

Abstract

Evidential deep learning (EDL) predicts a Dirichlet distribution over class-probability vectors in a single forward pass, using its mean for prediction and its dispersion to represent epistemic uncertainty. However, class-label supervision directly informs the predictive mean but provides no direct target for the dispersion. Standard EDL nevertheless uses the same labels to learn both, creating a *supervision–estimand mismatch*: the objective intended to shape dispersion lacks a direct dispersion supervision and can also alter the predictive mean. We address this mismatch with *Bayesian Evidential Dispersion Inference* (BEDI), a probabilistic framework that separates predictive-mean estimation from dispersion inference. BEDI fixes the Dirichlet mean at the calibrated predictive mean of a trained classifier, preserving its class probabilities and predictions by construction. It then infers dispersion using two complementary signals: effective sample size, which quantifies the training information relevant to an input, and Bayesian predictive disagreement, which reflects variation in class probabilities under model uncertainty. Through hierarchical dispersion inference, BEDI uses effective sample size to define a prior over input-specific dispersion and predictive disagreement to update this prior, while learning a shared relationship between effective sample size and dispersion across inputs. This shared relationship enables dispersion inference at new inputs with a single neural-network forward pass. Our theoretical analysis formalizes the supervision–estimand mismatch and establishes contraction of epistemic uncertainty as effective sample size increases. Across standard and long-tailed classification benchmarks and pretrained-model settings, BEDI achieves strong performance on a range of downstream tasks requiring reliable uncertainty quantification.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
uncertainty quantification, evidential deep learning, epistemic uncertainty, single-pass uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PccjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a post-hoc UQ method that leaves a classifier’s predictions unchanged. Its starting point is that class labels identify predictive probabilities, but do not directly identify the spread of a distribution over those probabilities. Standard evidential objectives nevertheless learn both together, which can affect the predictive mean. BEDI freezes a temperature-scaled classifier and fits the dispersion of a Dirichlet distribution around its probabilities. It uses feature-space effective sample size to define a dispersion prior and last-layer Laplace disagreement to construct pseudo-observations. A hierarchical model combines these signals and predicts dispersion for new inputs. The experiments evaluate OOD detection across several modalities and include a controlled study of missing training support versus label ambiguity.

Critical strengths
  • The paper makes a useful distinction between estimating predictive probabilities and estimating uncertainty about those probabilities. Proposition 4.1 gives a concrete example of how controlling dispersion through the expected-MSE evidential objective can distort the predictive mean.
  • Keeping the classifier fixed is a practical advantage. BEDI can add uncertainty estimates to a pretrained model while preserving its probabilities and decisions by construction.
  • The method gives training support and predictive disagreement separate roles. This makes the uncertainty estimate easier to examine than an evidence output learned only from class labels. The ablation in Table 13 shows that combining the signals generally improves OOD detection over using either alone.
  • The two-moons study is a useful part of the evaluation. It asks whether uncertainty responds to missing information rather than merely following class ambiguity. This tests the intended behaviour more directly than reporting OOD detection scores alone.
Critical weaknesses
  • Theorem 4.4 varies relative effective sample size while holding the predictive mean, representation, kernel residual, and fitted parameter posterior fixed. The resulting decrease in uncertainty follows from the imposed negative dependence on effective sample size in eq. 4. This is a desirable property, but it does not show what happens when more data are collected and the method is refitted. Since effective sample size is normalised by its training set average, increasing the dataset need not increase the relative quantity used by BEDI. The discussion in Section 4 should distinguish this monotonicity result from a statistical contraction guarantee.

  • The main experiments do not clearly establish what the hierarchical inference adds beyond combining the two signals. This matters because the proposed inference procedure accounts for a substantial part of the method’s complexity.

  • Some reported improvements should be attributed to the base classifier rather than dispersion inference. BEDI preserves the predictive mean, so its classification accuracy is inherited from the underlying predictor. Section 5.1 also defines misclassification detection using total predictive uncertainty, which eq. 11 identifies as the entropy of that fixed mean. Dispersion inference therefore does not change this detection score. These metrics are useful for showing that predictive behaviour is preserved, but the evidence for the new uncertainty model comes primarily from OOD detection and the controlled study.

  • The reliability weighting of the disagreement signal needs more justification. The paper uses the mismatch between the Laplace-derived covariance and its Dirichlet approximation to determine the pseudo-likelihood variance. However, fitting the Dirichlet covariance structure well does not establish that the underlying Laplace approximation accurately captures model uncertainty. An inaccurate disagreement estimate could still have a small projection mismatch and receive substantial weight. The paper should explain more clearly what this reliability measure captures and what errors it cannot detect.

  • Fixing the predictive mean also restricts the uncertainty that BEDI can express. Eq. 11 bounds epistemic uncertainty by the entropy of the frozen predictive probabilities. If the classifier is highly confident on an unfamiliar input, the epistemic score cannot become large even when the support estimate indicates little relevant training information. Preserving predictions is useful, but this limitation is directly relevant to confidently wrong predictions. The main evaluation does not make clear how the method behaves in those cases.

Questions for authors
  1. Is Theorem 4.4 intended only as a monotonicity result for the fitted model? What supports the broader interpretation of contraction as training information increases?
  2. Is there a comparison with a simpler dispersion regressor using the same support features and disagreement-derived targets? Which result most directly isolates the contribution of hierarchical inference?
  3. How should the covariance projection mismatch be interpreted as a reliability measure when the Laplace approximation itself may be inaccurate?
  4. Can the existing OOD results be broken down by the base classifier’s predictive confidence to assess the fixed-mean restriction?
7 comments

Thank you for your careful review of our paper and for the insightful and constructive comments. Building on the two-moons study, which you highlighted as a strength, we have added a comparison with a near-exact Bayesian reference to further strengthen our direct validation of epistemic uncertainty (Tables R8 and R9 in our response to Reviewer mU8W’s Comment 4).

We will upload a revised manuscript incorporating the additional results, analyses, and clarifications described below before the end of the rebuttal period. Below, we provide detailed responses to the weaknesses and questions raised in your review.

(1. Scope of Theorem 4.4 and uncertainty reduction under model refitting)

We thank the reviewer for highlighting this important distinction. Theorem 4.4 establishes conditional monotonicity with respect to relative ESS for a fixed BEDI model, rather than statistical contraction as training data increase and the model is refitted. To examine how BEDI’s uncertainty changes as training data increase under full refitting, we conducted an additional experiment on progressively larger CIFAR-10 training sets. Mean epistemic uncertainty decreases consistently across the evaluated training-set sizes.

What Theorem 4.4 establishes. The theorem varies relative ESS while holding the predictive mean, representation, kernel residual, and fitted parameter posterior fixed. Although Eq. (4) imposes a decreasing relationship between relative ESS and the prior logit-dispersion mean, the theorem establishes that this ordering also holds for BEDI’s predictive epistemic uncertainty after integrating over dispersion and the fitted parameter posterior. Establishing statistical contraction under full refitting would require additional assumptions and control over how the learned representation, ESS normalization, and dispersion posterior change as data accumulate. We therefore distinguish the theorem’s conditional monotonicity from the empirical behavior under full refitting.

Relative ESS under increasing training data. As the reviewer notes, increasing training-set size need not increase relative ESS, because its training-set normalization also changes. However, relative ESS can increase when the ESS of held-out inputs grows proportionally faster than the training-set average. This may occur as additional data improve coverage of previously less-supported inputs. Whether this occurs under full refitting is an empirical question, which we examine below.

Evaluation under full refitting. Motivated by prior studies examining how epistemic uncertainty changes with training-data availability [1, 2], we used class-stratified CIFAR-10 subsets with N∈1,000,2,000,5,000,10,000,25,000,47,500N \in \\{1{,}000, 2{,}000, 5{,}000, 10{,}000, 25{,}000, 47{,}500\\}. At each size, we retrained VGG-16 and refitted temperature scaling, support estimation, Laplace disagreement, and hierarchical dispersion inference. Each fitted model was evaluated on the same ID test set over five seeds.

Table R1. BEDI under increasing training-set size and full refitting. All quantities are averaged over the fixed CIFAR-10 test set. Raw ESS is reported as a seed mean; relative ESS, dispersion, and EU are reported as mean ± standard deviation over five seeds.

Training size NMean raw ESSMean relative ESSMean dispersionMean EU
1,0009.90.49 ± 0.030.088 ± 0.0180.233 ± 0.036
2,00020.30.54 ± 0.040.064 ± 0.0090.159 ± 0.019
5,00056.00.68 ± 0.040.032 ± 0.0040.078 ± 0.009
10,000106.70.74 ± 0.030.026 ± 0.0020.059 ± 0.003
25,000258.90.87 ± 0.040.019 ± 0.0040.037 ± 0.006
47,500497.40.95 ± 0.010.011 ± 0.0010.020 ± 0.002

Results and interpretation. Mean EU, averaged over five seeds, decreases with each increase in training-set size, from 0.233 ± 0.036 to 0.020 ± 0.002, accompanied by decreasing mean dispersion from 0.088 to 0.011. Raw ESS increases from 9.9 to 497.4, and relative ESS increases from 0.49 to 0.95. Thus, in this experiment, held-out inputs become better supported relative to the training-set average despite the changing normalization.

These results provide empirical evidence of epistemic uncertainty reduction under full refitting, complementing the theorem’s conditional monotonicity for a fixed model.

We will rename Theorem 4.4 as Conditional monotonicity of epistemic uncertainty in relative effective sample size, update Section 4 and Appendix B.4, and revise the corresponding contraction wording in the abstract and conclusion. We will include the full-refitting experiment in Appendix E.7.

References

[1] Maohao Shen et al. Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? NeurIPS 2024.

[2] Taeseong Yoon and Heeyoung Kim. Uncertainty Estimation by Flexible Evidential Deep Learning. NeurIPS, 2025.

(2. Contribution of hierarchical dispersion inference beyond combining the two signals)

We thank the reviewer for suggesting this comparison. To assess what hierarchical inference adds beyond combining ESS and Bayesian disagreement, we added direct and reliability-weighted dispersion regressors using the same support features, disagreement-derived targets, and functional form for the dispersion mean. The reliability-weighted regressor provides the most direct comparison: it already combines both signals and uses the same inverse marginal-variance weights as BEDI’s collapsed pseudo-likelihood. BEDI achieves an average OOD/shift detection AUPR of 88.07%, compared with 85.71% for direct regression and 85.62% for reliability-weighted regression.

These comparisons complement Table 13, which examines the value of combining ESS and Bayesian disagreement. Here, we assess what hierarchical inference adds when both signals are already available.

Matched regression alternatives. All variants use the same frozen classifier checkpoints, data splits, calibration procedure, support-feature construction, disagreement-target construction, and evaluation protocol. They share the mean function μθ(x)=β0+ω⊤u(x)−γlog⁡(1+n~eff(x)),γ>0,\mu_\theta(x)=\beta_0+\omega^\top u(x)-\gamma\log\left(1+\tilde n_{\mathrm{eff}}(x)\right),\qquad \gamma>0,

where u(x)u(x) is the normalized frozen classifier representation and n~eff(x)\tilde n_{\mathrm{eff}}(x) is the relative ESS. With the disagreement target ti=logit⁡(d^LA,i)t_i=\operatorname{logit}(\hat d_{\mathrm{LA},i}), the regression data-fitting objectives are

=\frac{1}{N}\sum_i\left[t_i-\mu_\theta(x_i)\right]^2, \qquad \mathcal L_{\mathrm{weighted}}(\theta) =\frac{1}{N}\sum_i\frac{\left[t_i-\mu_\theta(x_i)\right]^2}{V_i}, \quad V_i=\tau_i^2+\sigma_{\mathrm{LA},i}^2$$ Thus, the reliability-weighted regressor uses $w_i^{\mathrm{reg}}=1/V_i$, matching Eq. (9). The support-kernel residual and covariance projection mismatch enter through $\tau_i^2$ and $\sigma_{\mathrm{LA},i}^2$, respectively. *What hierarchical inference adds.* The regressors fit point estimates of the shared parameters. BEDI instead combines the collapsed observation model $t_i\mid\theta,x_i\sim\mathcal N(\mu_\theta(x_i),V_i)$ with a parameter prior, learns a variational posterior, and propagates it through test-time dispersion inference. **The comparison evaluates this hierarchical procedure as a whole, including prior regularization and posterior averaging.** **Table R2. Hierarchical inference versus matched dispersion regressors.** OOD/shift detection AUPR (%) averaged over each setting's OOD/shift datasets. Entries are mean ± standard deviation over five seeds for CIFAR and ImageNet and three seeds for OpenBookQA; Dry Bean aggregates three held-out-class settings × five seeds. BEDI values are from the evaluation runs used for these comparisons and differ slightly from those reported in the main paper. Bold indicates the highest mean in each column. Higher is better. | **Method** | **CIFAR-10 ↑** | **CIFAR-10-LT ↑** | **CIFAR-100 ↑** | **ImageNet ↑** | **Dry Bean ↑** |**OpenBookQA ↑** | **Avg. ↑** | |:---|---:|---:|---:|---:|---:|---:|---:| | Direct regression | 91.69 ± 1.0 | 78.55 ± 1.1 | **85.61 ± 0.5** | 86.41 ± 0.7 | **92.07 ± 4.2** | 79.95 ± 6.2 | 85.71 | | Reliability-weighted regression | **91.79 ± 1.0** | 79.18 ± 1.2 | 84.49 ± 0.5 | 86.09 ± 0.9 | 91.88 ± 4.2 | 80.26 ± 6.1 | 85.62 | | BEDI | 91.71 ± 1.0 | **79.43 ± 1.1** | 85.22 ± 0.6| **88.62 ± 0.2** | 91.73 ± 4.1 | **91.74 ± 4.1** | **88.07** | **Results and interpretation.** BEDI’s largest gains occur in pretrained-model settings, a central use case for its post-hoc uncertainty estimation. On ImageNet and OpenBookQA, BEDI improves AUPR over reliability-weighted regression by 2.53 and 11.48 points, and over direct regression by 2.21 and 11.79 points, respectively. Even with identical support features, disagreement targets, and reliability weights, the reliability-weighted regression falls short of BEDI’s average detection performance. **These gains demonstrate the practical importance of hierarchical dispersion inference as a key component of BEDI in pretrained-model settings, beyond combining and weighting the two signals.**

Computational cost. Using cached features and pseudo-targets, we timed dispersion-head fitting for 200 optimization iterations on an NVIDIA RTX A6000.

Table R3. Dispersion-head fitting time. Median wall-clock time over five repeats on one checkpoint per setting. Feature extraction, support estimation, and disagreement-target construction are excluded. Lower is better.

MethodImageNet (s) ↓OpenBookQA(s) ↓
Direct regression1.041.40
Reliability-weighted regression1.121.34
BEDI (default)1.511.76

Relative to reliability-weighted regression, BEDI adds 0.39 seconds on ImageNet and 0.42 seconds on OpenBookQA. This adds less than half a second to one-time dispersion-head fitting in both settings.

These comparisons show that BEDI’s hierarchical inference improves average detection performance beyond combining and weighting the two signals, with modest additional head-fitting cost in the two timed settings. We will include these comparisons in Appendix E.5 and summarize the benefit of hierarchical inference over direct and reliability-weighted regression in Section 5.2.

(3. Attribution of prediction-related performance and the role of these metrics)

We thank the reviewer for this comment. BEDI’s classification accuracy and entropy-based misclassification-detection performance are inherited from the calibrated base classifier, rather than improved by dispersion inference. BEDI preserves the calibrated predictive mean, and the total uncertainty used for misclassification detection is its entropy, H(m~(x))H(\tilde m(x)), as specified in Eq. (11).

Why we report these metrics. A central motivation for BEDI is to avoid interference between predictive-mean learning and dispersion inference. Proposition 4.1 establishes a predictive-mean distortion mechanism for the standard EDL expected-MSE objective. BEDI’s fixed-mean construction prevents dispersion fitting altering the calibrated predictive mean.

We report classification accuracy, misclassification detection, and calibration to show the predictive behavior retained by this separation. The evidence for the learned dispersion comes from EU-based OOD/shift detection and the controlled uncertainty studies, rather than improvements in these inherited prediction-related metrics.

We will clarify this attribution in Sections 5.1 and 5.2 and align the corresponding descriptions in Appendix E.

Reviewer_if6nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Evidential deep learning (EDL) produces mean as prediction and dispersion as epistemic uncertainty. Supervision provides direct target for mean but not the dispersion. To mitigate the supervision-estimand mismatch in standard EDL, the authors propose Bayesian Evidential Dispersion Inference (BEDI), a probabilistic framework that separates mean estimation from dispersion inference. Experiments across multiple settings show strong performance in classification accuracy, misclassification detection, and OOD/shift detection.

Critical strengths
  • The paper identifies a genuine weakness of standard EDL, as class labels do not provide direct supervision for epistemic uncertainty, and separating prediction from dispersion is well-motivated.
  • The post-hoc design preserves the backbone classifier accuracy. Compared to many UQ methods that sacriface accuracy for uncertainaty, this is a more suitable design option.
  • The experiment covers a broad range of datasets, settings, metrics, and baselines. Sufficient implementation details are provided in the supplementary as well, including the runtime and memory comparison.
  • Compared to the baselines, the proposed method achieves relatively strong performance across multiple settings.
Critical weaknesses
  • The Bayesian disagreement signal is itself based on a highly approximate class-factorized last-layer Laplace model and a first-order softmax linearization. Thus the proposed approach may inherit the biases of the approximate posterior that it uses as a pseudo-target.
  • Sensitivity to the representation, kernel bandwidth, inducing-point selection, and auxiliary noise assumptions might be further explored.
  • In some cases, a few baselines are stronger. Some explanations are desirable to explain when the proposed approach is weaker (failure modes).
Questions for authors

Authors could refer to the aforementioned weaknesses.

7 comments

Thank you for your careful review of our paper and for the insightful and constructive comments. We will upload a revised manuscript incorporating the additional results, analyses, and clarifications described below before the end of the rebuttal period. Please find our detailed responses to your comments below.

(1. Sensitivity to approximations in Bayesian predictive disagreement)

We agree that BEDI may inherit errors from the approximate posterior used to construct its disagreement-derived pseudo-observations. To assess the practical impact of these approximation choices, we compared BEDI’s default construction with alternatives that incorporate cross-class posterior dependencies, Monte Carlo propagation, or both. Across six evaluated settings, the default class-factorized, first-order construction achieves the highest average OOD/shift detection AUPR. The tested alternatives do not provide consistent improvements.

Experimental comparison. We vary two aspects of the disagreement construction:

  • Laplace posterior covariance: class-factorized versus joint covariance, allowing cross-class dependencies.
  • Predictive-probability propagation: first-order softmax linearization versus Monte Carlo propagation using 256 posterior draws.

The joint construction uses a full joint covariance on CIFAR-10, CIFAR-10-LT, OpenBookQA, and Dry Bean, and a Kronecker-factored approximate curvature (KFAC) approximation [1] on CIFAR-100 and ImageNet. For each variant, we recompute the disagreement-derived pseudo-observations and pseudo-likelihood variances and refit the same hierarchical dispersion model. The remaining components and evaluation protocol are held fixed.

Table R1. Sensitivity of BEDI to Bayesian disagreement approximations. OOD/shift detection AUPR (%) averaged over each setting’s OOD/shift datasets, reported as mean ± standard deviation. Results are from refits for this comparison and may therefore differ slightly from those reported in the main paper. Avg. is the unweighted mean of the six setting-level means. For Dry Bean, the mean and standard deviation are computed across the 15 held-out-class–seed combinations. Bold indicates the highest mean in each column. Higher is better.

VariantLaplace posteriorSoftmax propagationCIFAR-10 ↑CIFAR-10-LT ↑CIFAR-100 ↑ImageNet ↑OpenBookQA ↑Dry Bean ↑Avg. ↑
BEDI (default)Class-factorizedFirst-order91.71 ± 1.079.43 ± 1.185.22 ± 0.688.62 ± 0.291.74 ± 4.191.64 ± 4.288.06
BEDI + MCClass-factorizedMonte Carlo89.70 ± 0.679.35 ± 1.285.40 ± 0.688.27 ± 0.291.67 ± 4.290.49 ± 3.687.48
BEDI + JointJointFirst-order91.66 ± 1.079.29 ± 1.084.43 ± 0.688.72 ± 0.291.58 ± 4.291.69 ± 4.487.90
BEDI + Joint + MCJointMonte Carlo89.92 ± 0.579.24 ± 1.184.65 ± 0.688.41 ± 0.291.53 ± 4.390.51 ± 3.487.38

Results and interpretation. BEDI’s default construction achieves an average AUPR of 88.06%, compared with 87.48% for MC propagation, 87.90% for the joint posterior construction, and 87.38% when both changes are applied. MC propagation yields a small mean improvement on CIFAR-100, while the joint construction yields small mean improvements on ImageNet and Dry Bean. The default achieves the highest mean on CIFAR-10, CIFAR-10-LT, and OpenBookQA.

Neither incorporating cross-class posterior dependencies nor replacing first-order propagation with Monte Carlo consistently improves downstream detection. These results support the practical effectiveness of BEDI’s default disagreement construction among the evaluated alternatives, providing empirical evidence of how different approximation choices affect downstream uncertainty estimation.

We will include these comparisons in Appendix E.5 and clarify the rationale for the default disagreement construction in Section 3.2.2.

[1] James Martens and Roger Grosse. Optimizing Neural Networks with Kronecker-factored Approximate Curvature. ICML, 2015.

(2. Sensitivity to representation and support-estimation choices)

We thank the reviewer for suggesting this analysis. To assess how support-estimation choices affect BEDI, we varied representation normalization, RBF bandwidth, inducing-point selection, and the auxiliary model’s variance scale. Detection performance remains relatively stable under moderate bandwidth changes, alternative inducing-point selection, and the tested variance scales. Removing L2 normalization has a larger effect on ImageNet. We also directly examine how moderate bandwidth changes affect the prior and inferred dispersion.

Experimental settings. We use the same frozen classifier checkpoints, data splits, calibration protocol, and disagreement-construction procedure, vary one support-estimation factor at a time, and refit the support estimator and dispersion model. Specifically, we compare:

  • Representation: raw versus L2-normalized frozen features used for support estimation.
  • RBF bandwidth: multipliers from 0.25× to 4× the dataset-specific default.
  • Inducing-point selection: class-wise k-means, global k-means, class-balanced random selection, and uniform random selection.
  • Auxiliary variance scale: writing the auxiliary model’s coefficient prior as a∼N(0,σa2I)a\sim\mathcal N(0,\sigma_a^2 I), we vary the noise-to-prior variance ratio r=σn2/σa2∈0.1,0.3,1,3,10r = \sigma^2_n/\sigma^2_a\in \\{0.1, 0.3, 1, 3, 10\\}, where σn2\sigma^2_n is the observation-noise variance. We fix σn2=1\sigma_n^2=1 and set σa2=1/r\sigma_a^2=1/r, with r=1r=1 corresponding to the default.

All settings use L2-normalized support features and class-wise k-means by default. The auxiliary variance-ratio comparison changes the relative variance scale of the support model; it does not vary the Laplace posterior or dispersion-prior hyperparameters.

Table R2. Sensitivity of BEDI to support-estimation choices. OOD/shift detection AUPR (%) averaged over each setting’s OOD/shift datasets. Entries are mean ± standard deviation over five seeds for CIFAR and ImageNet, and three held-out-class settings × five seeds for Dry Bean. Default† uses L2-normalized support features, RBF with each setting’s default bandwidth, class-wise k-means inducing-point selection, and r=1. Each alternative changes the indicated support-estimation factor. Higher is better.

FactorConfigurationCIFAR-10 ↑CIFAR-10-LT ↑CIFAR-100 ↑ImageNet ↑Dry Bean ↑
ReferenceDefault†91.67 ± 1.0179.40 ± 1.0585.24 ± 0.6088.62 ± 0.2191.71 ± 4.18
RepresentationRaw features91.57 ± 1.1978.99 ± 0.9985.27 ± 0.4084.84 ± 0.7291.76 ± 4.06
Bandwidth multiplier0.25×91.56 ± 0.9079.44 ± 0.9984.48 ± 0.7288.65 ± 0.3091.78 ± 4.11
0.5×91.57 ± 1.0279.39 ± 1.0284.75 ± 0.6188.87 ± 0.2291.73 ± 4.22
0.75×91.64 ± 1.0379.43 ± 1.0385.07 ± 0.5988.73 ± 0.2091.61 ± 4.21
1.5×91.70 ± 0.9779.29 ± 1.0385.36 ± 0.5988.48 ± 0.2091.79 ± 4.12
2×91.71 ± 0.9779.20 ± 1.0285.39 ± 0.5988.29 ± 0.2291.87 ± 4.10
4×91.70 ± 0.9878.96 ± 0.9885.36 ± 0.5987.33 ± 0.2791.99 ± 4.08
Inducing-point selectionGlobal k-means91.64 ± 1.0079.16 ± 0.9585.25 ± 0.5788.65 ± 0.1991.64 ± 4.18
Class-balanced random91.80 ± 0.8979.23 ± 1.1785.66 ± 0.6188.59 ± 0.1891.70 ± 4.15
Uniform random91.81 ± 0.9478.81 ± 1.0185.63 ± 0.5788.57 ± 0.2091.79 ± 4.15
Auxiliary variance ratior=0.191.67 ± 1.0379.52 ± 1.0585.23 ± 0.5989.53 ± 0.2291.64 ± 4.17
r=0.391.67 ± 1.0279.47 ± 1.0585.24 ± 0.5989.07 ± 0.2191.64 ± 4.17
r=391.67 ± 0.9979.33 ± 1.0485.22 ± 0.6088.31 ± 0.2291.64 ± 4.18
r=1091.68 ± 0.9879.24 ± 1.0385.20 ± 0.6088.06 ± 0.2391.63 ± 4.18

† The default configuration was rerun with new random seeds for this comparison, resulting in small numerical differences from the earlier ablation results reported in the paper.

Results and interpretation. Across Table R2, the largest absolute changes from the default mean AUPR are 0.14, 0.59, and 0.76 percentage points on CIFAR-10, CIFAR-10-LT, and CIFAR-100, respectively, and 0.28 points on Dry Bean.

On ImageNet, moderate bandwidth changes (0.5×–2×) yield AUPR between 88.29 and 88.87, and the four inducing-point selection strategies span only 88.57–88.65. Varying the auxiliary variance ratio yields AUPR between 88.06 and 89.53. In contrast, using raw rather than L2-normalized support features reduces ImageNet AUPR from 88.62 to 84.84. Thus, support-feature normalization is a consequential choice in this setting.

Direct sensitivity of intermediate estimates. We additionally compare relative ESS, expected local-prior location, and inferred expected dispersion on matched ID and OOD inputs across the four image settings. For bandwidth multipliers of 0.75× and 1.5×, the lowest seed-averaged Spearman correlations with the corresponding default are 0.978 for ESS, 0.983 for prior location, and 0.981 for dispersion. These results directly show that moderate bandwidth changes largely preserve the input-wise ordering of the intermediate estimates, complementing the downstream detection results. Detailed results and explanations are provided in our response to Reviewer mU8W, Comment 1 (Table R1).

Together, these results support practical stability under the tested bandwidth, inducing-point selection, and auxiliary variance-scale changes, while identifying representation normalization as a more consequential choice on ImageNet. We will include these analyses in Appendix E.5 and clarify the support-estimation choices in Section 3.2.1.

Reviewer_mU8WOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a supervision–estimand mismatch in Evidential Deep Learning (EDL), where class-label supervision does not directly identify Dirichlet dispersion and may interfere with predictive-mean estimation. To address this issue, the authors propose Bayesian Evidential Dispersion Inference (BEDI), which preserves the calibrated predictive mean of a pretrained classifier while inferring dispersion from effective sample size and Bayesian predictive disagreement through hierarchical Bayesian inference. The paper provides theoretical analyses of the proposed framework and demonstrates competitive uncertainty quantification performance across multiple classification benchmarks and pretrained models.

Critical strengths
  1. The paper addresses an important conceptual issue in EDL concerning the identifiability of Dirichlet dispersion under class-label supervision. Separating predictive-mean estimation from dispersion inference offers a meaningful perspective on the reliability of evidential uncertainty estimation.
  2. The proposed method combines training-data support and approximate Bayesian predictive disagreement within a hierarchical inference framework. The individual components have clear statistical motivations, and preserving the predictive mean enables post-hoc uncertainty estimation without modifying the original classifier.
  3. The experiments cover multiple model architectures, datasets, and modalities, including pretrained models. The reported results demonstrate promising performance in OOD and distribution-shift detection. The supplementary material also provides additional evaluations, ablation studies, and computational analyses.
Critical weaknesses
  1. The proposed ESS is estimated using an auxiliary Bayesian linear model with Nyström kernel features. Since this estimate depends on the kernel function, bandwidth, and inducing points, how sensitive are the resulting prior and dispersion estimates to these choices? Additional comparisons using different kernels and parameter settings would help establish the robustness of the proposed prior.
  2. BEDI estimates Bayesian predictive disagreement using a last-layer Laplace approximation while treating the feature extractor as deterministic. However, uncertainty in the feature extractor may substantially affect predictions. Could the authors investigate whether incorporating uncertainty from additional layers or the entire network would improve dispersion estimation?
  3. BEDI fixes the predictive mean of the pretrained classifier, even when its predictions are incorrect. In particular, epistemic uncertainty is upper-bounded by the entropy of this fixed mean, potentially limiting its ability to identify overconfident errors. The authors should investigate whether the inferred dispersion remains informative when the underlying classifier makes highly confident incorrect predictions.
  4. Although the paper reports competitive results on several uncertainty-related tasks, these results primarily demonstrate improvements in downstream detection performance rather than directly validating the reliability of the inferred epistemic uncertainty. More targeted experiments are needed to establish whether BEDI accurately captures model uncertainty and to substantiate its central methodological claims.
Questions for authors

Please refer to the Weakness Part.

9 comments

Thank you for your careful review of our paper and for the insightful and constructive comments. We will upload a revised manuscript incorporating the additional results, analyses, and clarifications described below before the end of the rebuttal period. Please find our detailed responses to your comments below.

(1. Sensitivity of the ESS-based prior to kernel and Nyström choices)

We thank the reviewer for suggesting these comparisons. To assess how kernel and Nyström choices affect the ESS-based prior and inferred dispersion, we conducted additional experiments varying the kernel family, bandwidth, and inducing-point selection method. The results support stable prior-location and dispersion rankings under moderate bandwidth changes and alternative inducing-point selection in the examined settings. Comparisons with alternative kernels show broadly similar detection performance for some choices, with larger differences for others.

Experimental comparison. We vary three support-estimation choices:

  • Kernel family: RBF, Laplacian, Matérn-3/2, and rational quadratic.
  • Bandwidth: multipliers from 0.25× to 4× the dataset-specific default.
  • Inducing-point selection: class-wise k-means, global k-means, class-balanced random selection, and uniform random selection, with the inducing-point budget held fixed.

Each configuration recomputes the support estimates and refits the dispersion model using the same classifier weights, data splits, and calibration protocol. RBF is BEDI’s original default kernel. Alternative kernels use the same bandwidth rule and scale, without kernel-specific tuning.

Direct sensitivity of the prior and dispersion. We compare relative ESS, expected prior location, conditional prior variance, and inferred expected dispersion against the corresponding default models on matched ID and OOD inputs. Spearman correlations measure whether the input-wise ordering is preserved, while median absolute log changes quantify changes in dispersion and prior variance. Expected dispersion is evaluated by deterministic quadrature to avoid Monte Carlo evaluation noise.

Table R1. Direct sensitivity of the ESS-based prior and inferred dispersion under moderate bandwidth changes. Each entry reports the range of seed-mean results across bandwidth multipliers 0.75× and 1.5×, and two evaluation summaries: ID results and the equally weighted average of OOD-domain results. Absolute log changes are computed per input and summarized by the within-domain median before aggregation. These ranges describe sensitivity across comparisons, not confidence intervals. Five seeds are used for each image setting, three for OpenBookQA, and three held-out classes × five seeds for Dry Bean.

SettingESS Spearman ρPrior-location Spearman ρDispersion Spearman ρMedian abs. Δ log dispersionMedian abs. Δ log prior variance
CIFAR-100.986–0.9990.983–0.9950.981–0.9950.120–0.1330.007–0.175
CIFAR-10-LT0.988–0.9980.999–0.9990.994–0.9990.081–0.1450.013–0.251
CIFAR-1000.978–0.9830.998–0.9990.999–0.9990.009–0.0150.019–0.039
ImageNet0.999–0.9990.999–1.0000.999–1.0000.011–0.014<0.001
OpenBookQA0.989–0.9970.995–0.9990.995–0.9990.015–0.059<0.001
Dry Bean0.904–0.9650.994–0.9980.994–0.9980.053–0.1640.026–0.499

Seed-averaged prior-location and dispersion correlations remain at least 0.983 and 0.981, respectively, across all six settings. ESS is more sensitive on Dry Bean, with correlations of 0.904–0.965, but the resulting prior-location and dispersion correlations remain at least 0.994. Median absolute log changes in dispersion range from 0.009 to 0.164, while prior-variance changes reach 0.499 on Dry Bean. Thus, moderate bandwidth changes preserve the ordering of the inferred quantities more consistently than their absolute values.

Direct comparisons on ImageNet and OpenBookQA also show prior-location and dispersion correlations of approximately 0.997–1.000 under alternative inducing-point selection methods. Kernel-family changes can be more consequential: on ImageNet, the Laplacian kernel yields an ID dispersion correlation of 0.572 ± 0.029, compared with approximately 0.991 for Matérn-3/2 and 1.000 for rational quadratic.

Downstream sensitivity. Table R2 evaluates how these design choices affect OOD/shift detection.

Table R2. Sensitivity of BEDI’s detection performance to kernel and Nyström choices. OOD/shift detection AUPR (%) averaged over each setting’s OOD/shift datasets, using ID examples as positives. Entries are mean ± standard deviation over five seeds for CIFAR and ImageNet, three seeds for OpenBookQA, and three held-out-class settings × five seeds for Dry Bean. Default† uses RBF, each setting’s default bandwidth, and class-wise k-means. Each alternative changes the indicated factor. Higher is better.

FactorConfigurationCIFAR-10 ↑CIFAR-10-LT ↑CIFAR-100 ↑ImageNet ↑Dry Bean ↑OpenBookQA ↑
ReferenceDefault†91.67 ± 1.0179.40 ± 1.0585.24 ± 0.6088.62 ± 0.2191.71 ± 4.1891.74 ± 4.12
Kernel familyLaplacian91.62 ± 1.0078.42 ± 0.9583.99 ± 0.7679.60 ± 1.0691.62 ± 4.2288.25 ± 4.57
Matérn-3/291.67 ± 1.0178.96 ± 1.0084.57 ± 0.6487.90 ± 0.2891.63 ± 4.2191.81 ± 3.92
Rational quadratic91.70 ± 1.0079.23 ± 1.0185.04 ± 0.5988.50 ± 0.2291.69 ± 4.1891.73 ± 4.12
Bandwidth multiplier0.25×91.56 ± 0.9079.44 ± 0.9984.48 ± 0.7288.65 ± 0.3091.78 ± 4.1191.66 ± 3.47
0.5×91.57 ± 1.0279.39 ± 1.0284.75 ± 0.6188.87 ± 0.2291.73 ± 4.2291.90 ± 3.81
0.75×91.64 ± 1.0379.43 ± 1.0385.07 ± 0.5988.73 ± 0.2091.61 ± 4.2191.87 ± 3.97
1.5×91.70 ± 0.9779.29 ± 1.0385.36 ± 0.5988.48 ± 0.2091.79 ± 4.1291.29 ± 4.24
2×91.71 ± 0.9779.20 ± 1.0285.39 ± 0.5988.29 ± 0.2291.87 ± 4.1090.68 ± 4.66
4×91.70 ± 0.9878.96 ± 0.9885.36 ± 0.5987.33 ± 0.2791.99 ± 4.0890.01 ± 4.62
Inducing-point selectionGlobal k-means91.64 ± 1.0079.16 ± 0.9585.25 ± 0.5788.65 ± 0.1991.64 ± 4.1891.70 ± 4.15
Class-balanced random91.80 ± 0.8979.23 ± 1.1785.66 ± 0.6188.59 ± 0.1891.70 ± 4.1591.59 ± 4.17
Uniform random91.81 ± 0.9478.81 ± 1.0185.63 ± 0.5788.57 ± 0.2091.79 ± 4.1591.60 ± 4.18

† The default configuration was rerun with new random seeds for this comparison, resulting in small numerical differences from the earlier ablation results reported in the paper.

Under the default RBF kernel, moderate bandwidth changes (0.5×–2×) produce AUPR ranges spanning at most 0.64 percentage points in each image setting, 0.26 points on Dry Bean, and 1.22 points on OpenBookQA. All four inducing-point selection strategies also yield similar performance, with the AUPR range spanning at most 0.59 points within any setting.

Matérn-3/2 and rational quadratic perform similarly to RBF in most settings. In contrast, Laplacian reduces AUPR by 9.02 points on ImageNet and 3.49 points on OpenBookQA, consistent with the greater kernel-family sensitivity observed in the intermediate estimates.

Together, these results show that moderate bandwidth changes largely preserve prior-location and dispersion ordering, while downstream detection remains stable under these changes and alternative inducing-point selection methods. The kernel-family comparisons further show that the similarity structure used to define training support matters for both the inferred estimates and detection performance. We will include these analyses in Appendix E.5 and clarify their implications for the ESS-based prior in Section 3.2.1.

(2. Effect of incorporating uncertainty beyond the classifier head)

We thank the reviewer for suggesting this comparison. To investigate whether incorporating feature-extractor uncertainty improves BEDI’s downstream detection, we evaluated disagreement signals derived from Laplace approximations covering the classifier head, the final feature block plus head, or the entire network. Across three CIFAR settings, BEDI’s head-only construction achieves the highest average OOD detection AUPR while requiring substantially less computation. Full-network Laplace yields only marginal mean gains on CIFAR-10 and CIFAR-100 and lower performance on CIFAR-10-LT.

Experimental comparison. Within each setting and seed, all variants use the same classifier architecture and trained checkpoint. We compare (i) the final linear classifier head only (BEDI default), (ii) the last convolutional block of VGG-16 or last residual stage of ResNet-18 together with the head, and (iii) the entire feature extractor and head. For each construction, we obtain disagreement-derived dispersion pseudo-observations and refit BEDI’s hierarchical dispersion model. The calibrated predictive mean and ESS-based support construction remain fixed.

For computational tractability, the broader variants use diagonal Laplace approximations, while the head-only default retains its class-factorized construction. All variants use first-order linearization to compute predictive-probability covariance.

Table R3. Comparison of Laplace constructions with different parameter scopes. OOD detection AUPR (%) is averaged over the two OOD datasets for each ID setting, using ID examples as positives, and reported as mean ± standard deviation over five seeds. The BEDI default values are from refitting and evaluation runs used for this comparison and differ slightly from the earlier ablation results in the paper. Time covers disagreement construction and dispersion-model fitting. Time and peak GPU memory are measured on an NVIDIA RTX A6000 using one seed per setting. Average rows report unweighted means across the three settings. Bold indicates the best value within each comparison group. Higher AUPR and lower cost are better.

ID settingLaplace parameter scopeParameters in posteriorOOD AUPR ↑Time (s) ↓Peak GPU memory (GB) ↓
CIFAR-10Classifier head only5.1 K91.71 ± 1.0340.55
Last conv block + head7.09 M90.36 ± 1.51721.31
Entire network14.73 M91.72 ± 0.43431.38
CIFAR-10-LTClassifier head only5.1 K79.43 ± 1.1220.36
Last conv block + head7.09 M78.41 ± 1.5661.31
Entire network14.73 M78.15 ± 2.11321.38
CIFAR-100Classifier head only51.3 K85.22 ± 0.6410.87
Last residual stage + head8.31 M85.12 ± 0.62151.29
Entire network11.05 M85.30 ± 0.84401.31
AverageClassifier head only—85.45320.59
Last block / stage + head—84.631511.30
Entire network—85.063051.36

CIFAR-10 and CIFAR-10-LT use VGG-16; CIFAR-100 uses ResNet-18. CIFAR-10-LT has an imbalance ratio of 0.01.

Results and interpretation. Full-network Laplace changes mean AUPR by +0.01, −1.28, and +0.08 points on CIFAR-10, CIFAR-10-LT, and CIFAR-100, respectively. Including the final feature block yields lower mean AUPR across all three settings. Consequently, head-only Laplace achieves the highest average AUPR of 85.45%, compared with 84.63% for the final block/stage plus head and 85.06% for the entire network.

The broader constructions also incur substantially higher computational cost. Averaged across settings, disagreement construction and dispersion fitting take approximately 151 seconds for the final block/stage plus head and 305 seconds for the entire network, compared with 32 seconds for head-only inference. Average peak GPU memory increases from 0.59 GB to 1.30 and 1.36 GB, respectively.

These findings are consistent with the practical motivation for post-hoc last-layer Laplace in Laplace Redux [1], which reports that it often performs better than applying Laplace to all weights while being substantially cheaper.

These results support head-only Laplace as an effective and efficient choice for BEDI: the tested broader constructions provide no consistent detection benefit despite substantially higher computational cost. Feature-extractor uncertainty may still be relevant, but the marginal gains in two settings do not translate into higher average detection performance.

We will include these comparisons in Appendix E.5 and clarify the computational motivation for last-layer inference in Section 3.2.2.

[1] Erik Daxberger et al. Laplace Redux—Effortless Bayesian Deep Learning. NeurIPS, 2021.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of Rebuttal Responses and Manuscript Revisions

We thank the reviewers for their careful reading and constructive feedback. Their comments have helped us clarify BEDI’s methodological contributions and strengthen the empirical evidence supporting its effectiveness and the reliability of its epistemic uncertainty estimates. We have uploaded a revised manuscript incorporating the additional experiments, analyses, and clarifications developed during the rebuttal.

Revisions are highlighted in red\color{#B40000}{\text{red}} for ease of reference. Below, we summarize the key findings and manuscript revisions, with references to the corresponding reviewer comments and revised manuscript sections.

Key findings

1. Direct validation of epistemic uncertainty
(mU8W: Comment 4; Pccj: Comment 1)

Building on the existing controlled two-moons study, which was highlighted as a strength, we added a comparison with a near-exact Bayesian reference in a low-dimensional setting. BEDI achieves the lowest mean error across all three evaluated metrics and the lowest mean Wasserstein-1 distance across all five training-support conditions. Separately, under full pipeline refitting on progressively larger CIFAR-10 training sets, mean epistemic uncertainty decreases at every evaluated size, from 0.233 ± 0.036 at 1,000 training samples to 0.020 ± 0.002 at 47,500 samples. These results strengthen the direct validation of BEDI’s epistemic uncertainty beyond downstream detection performance.

Revised manuscript: Section 5.3 and Appendix E.7.

2. The contribution of hierarchical dispersion inference
(Pccj: Comment 2)

The reliability-weighted baseline uses the same support features, disagreement targets, dispersion-mean parameterization, and inverse marginal-variance weights as BEDI. BEDI achieves an average OOD/shift detection AUPR of 88.07%, compared with 85.71% for direct regression and 85.62% for reliability-weighted regression. The largest gains occur on ImageNet and OpenBookQA, where BEDI improves over reliability-weighted regression by 2.53 and 11.48 AUPR points. These results demonstrate the practical importance of hierarchical dispersion inference beyond combining and weighting the two signals, particularly in pretrained-model settings, a central use case for BEDI. We also report the additional dispersion-head fitting cost.

Revised manuscript: Section 5.2 and Appendix E.5.5.

3. Evaluation of support estimation and disagreement approximations
(mU8W: Comments 1–2; if6n: Comments 1–2)

Sensitivity analyses show stability under moderate bandwidth changes and alternative inducing-point selections, while documenting the effects of kernel family, feature normalization, and auxiliary variance settings. The tested broader Laplace constructions incur substantially higher fitting costs without consistent detection gains. Alternative covariance constructions and Monte Carlo predictive propagation likewise provide no consistent improvement. These results support BEDI’s default construction as an efficient and effective choice for post-hoc uncertainty estimation.

Revised manuscript: Section 3.2 and Appendices E.5.2–E.5.4.

4. Characterizing the fixed-mean trade-off
(mU8W: Comment 3; Pccj: Comment 5)

Confidence- and entropy-controlled analyses show that BEDI retains useful OOD discrimination among similarly confident ID and OOD inputs, including when predictive entropies are approximately matched. We also examine highly confident incorrect predictions and document limited detection when extremely confident OOD inputs are compared against the full ID population. These analyses clarify both the information captured by dispersion and the constraint imposed by the fixed predictive mean.

Revised manuscript: Section 6 and Appendix E.8.

Further clarifications incorporated into the manuscript

  • Theoretical scope (Pccj: Comment 1). Theorem 4.4 now explicitly establishes conditional monotonicity for a fixed fitted model, distinguished from the empirical uncertainty reduction observed under full refitting. We updated the abstract, Section 4, Appendix B.4, and conclusion accordingly.

  • Interpretation of the reliability proxy (Pccj: Comment 4). We clarified that covariance mismatch measures compatibility with the fixed-mean Dirichlet covariance family and specified which upstream approximation errors it can leave undetected (Section 3.2.2 and Appendix A.2).

  • Predictive-metric attribution and failure modes (if6n: Comment 3; Pccj: Comment 3). We clarified that predictive accuracy and entropy-based misclassification detection are inherited from the calibrated base classifier. We expanded the discussion of weaker results and limitations associated with the frozen representation (Sections 5–6 and Appendix E.1).

We thank the reviewers again for their time and thoughtful suggestions. We hope that our responses and revisions have addressed the concerns raised, and we welcome any further questions or comments.

AuthorsOpenReview ↗

Response to Reviewer Pccj’s Follow-up Comments (1/2)

Thank you for carefully reading the revised manuscript and for your positive reassessment of our work. We particularly appreciate the depth and technical precision of your feedback. Your suggestions have helped us clarify the theoretical scope, strengthen the empirical evaluation, and present BEDI’s contributions and limitations more precisely. We are glad that the revisions have addressed your main concerns.

Theorem 4.4. Thank you for confirming that the revised statement and discussion clarify the intended theoretical scope. Following your suggestion, we also report the SVHN OOD results from the same CIFAR-10 full-refitting experiment. At each training-set size, we retrain the classifier and refit temperature calibration, support estimation, the Laplace approximation, and dispersion inference.

Table. ID and OOD epistemic uncertainty under increasing training-set size and full pipeline refitting. Mean EU is evaluated on fixed CIFAR-10 ID and SVHN OOD sets and reported as mean ± standard deviation over five seeds.

Training-set sizeCIFAR-10 ID mean EUSVHN OOD mean EU
1,0000.233 ± 0.0360.256 ± 0.084
2,0000.159 ± 0.0190.212 ± 0.018
5,0000.078 ± 0.0090.159 ± 0.026
10,0000.059 ± 0.0030.135 ± 0.015
25,0000.037 ± 0.0060.107 ± 0.005
47,5000.020 ± 0.0020.093 ± 0.011

Mean EU decreases on both ID and OOD inputs at every evaluated training-set size, while SVHN retains higher mean EU than CIFAR-10 throughout. These results extend the empirical full-refitting evaluation to OOD inputs, complementing the theorem’s conditional monotonicity for a fixed fitted model. We will include the SVHN results in Table 24 and update the discussion in Appendix E.7.2.

Attribution of accuracy and misclassification detection. We appreciate your confirmation that the revised explanation makes clear that these metrics are inherited from the calibrated base classifier.

Reliability proxy. Thank you for confirming that the distinction between projection compatibility and the accuracy of the supplied covariance resolves your concern. We retain this distinction explicitly in Section 3.2.2 and Appendix A.2.

Fixed predictive mean. We agree that describing only the magnitude limitation does not fully convey its implications for detection. We will revise Section 6 to state explicitly that highly confident OOD inputs can receive lower EU scores than ID inputs, impairing their ranking for OOD detection even when inferred dispersion remains informative. We will also make the interpretation of Table 26 explicit in Appendix E.8.2: the mean EU AUROC of 43.02% for OOD confidence ≥0.99 is below chance across the three eligible settings. This will connect the main-text limitation directly to the observed failure case and the explanation already provided in Appendix E.8.5.

Bayesian reference. We agree that this experiment should be interpreted as a controlled sanity check in a low-dimensional setting with a fixed representation. It enables direct comparison with a numerically integrated Bayesian reference, but does not establish comparable distributional accuracy in high-dimensional settings. We will make this scope explicit in Section 5.3 and Appendix E.7.1.

AuthorsOpenReview ↗

Response to Reviewer Pccj’s Follow-up Comments (2/2)

Hierarchical inference. We agree that the gains are not uniform across settings and that OpenBookQA contributes substantially to the average improvement. We will qualify Section 5.2 to state explicitly that the largest observed benefits occur on ImageNet and OpenBookQA, while simpler regressors remain competitive on CIFAR and Dry Bean.

To investigate the OpenBookQA results, we analyzed the original three fitted checkpoints without retraining or selecting additional seeds. BEDI improves over reliability-weighted regression in all three seeds, by 8.46, 12.50, and 13.48 AUPR percentage points, respectively. The average improvement therefore does not arise from a single exceptional seed, although three seeds remain a limited basis for assessing variability.

OpenBookQA presents a data-limited dispersion-fitting problem: 500 inputs from the development set provide pseudo-targets for a model using 4,096-dimensional representations. The direct and weighted regressors fit the pseudo-targets more closely. Evaluating the logit-dispersion mean function at the regressors’ point estimates and BEDI’s posterior parameter mean against the same logit-transformed development pseudo-targets gives mean squared errors of 0.14, 0.15, and 1.08 for direct regression, reliability-weighted regression, and BEDI, respectively, averaged over the three seeds. However, the regressors also learn substantially larger feature coefficients and exhibit greater variation in the feature-dependent term ω⊤u(x)\omega^\top u(x) across ID inputs.

Table. OpenBookQA dispersion diagnostics. Entries are mean ± standard deviation across the three original seeds. Both detection columns report ID-positive AUPR, using negative dispersion and negative EU as the respective scores, consistent with Table 19. Coefficient norms and feature-term variation are evaluated at the regressors’ point estimates and BEDI’s posterior parameter mean. “ID feature-term SD” denotes the within-ID standard deviation of ω⊤u(x)\omega^\top u(x) in the logit-dispersion predictor. Bold indicates the highest mean detection score.

MethodFeature coefficient norm ∥ω∥2\lVert\omega\rVert_2ID feature-term SDDispersion AUPR (%) ↑EU AUPR (%) ↑
Direct regression18.05 ± 2.381.01 ± 0.1679.81 ± 6.4779.95 ± 6.16
Reliability-weighted regression17.97 ± 2.441.01 ± 0.1680.14 ± 6.4480.26 ± 6.10
BEDI3.49 ± 0.310.19 ± 0.0492.02 ± 4.4891.74 ± 4.11

Using the same metric and positive-class convention, BEDI improves over reliability-weighted regression by 11.88 percentage points in dispersion AUPR and 11.48 points in EU AUPR. Thus, the improvement is already present in the inferred dispersion and is retained in the final EU score.

BEDI’s hierarchical inference balances fitting the disagreement-derived pseudo-targets against a Gaussian parameter prior through the KL term in Eq. (9). This regularization discourages excessive deviations from the prior center, which can be particularly useful when learning feature-dependent dispersion from limited development data. Consistent with this interpretation, BEDI fits the pseudo-targets less closely but learns smaller feature coefficients and achieves better ID–OOD dispersion separation. Reliability weighting alone leaves the larger feature-dependent variation largely unchanged.

These diagnostics suggest that prior regularization may help explain BEDI’s advantage on OpenBookQA. This is one plausible explanation consistent with the observed fitting behavior, but the diagnostics do not establish a causal link between prior regularization and the performance gap. We will include these diagnostics and their interpretation in Appendix E.9, with a cross-reference in Appendix E.5.5, alongside the qualified main-text claim.

Thank you again for helping us improve the clarity and precision of the paper.

AuthorsOpenReview ↗

Additional Results and Manuscript Updates

Following the discussion with Reviewer Pccj, we have uploaded an updated manuscript incorporating the additional results and clarifications described in our follow-up responses. These updates supplement our earlier “Summary of Rebuttal Responses and Manuscript Revisions.”

  • OOD evaluation under full refitting. We extended the training-size experiment to SVHN OOD inputs. Mean epistemic uncertainty decreases at every evaluated training-set size on both CIFAR-10 and SVHN, while remaining higher on SVHN throughout (Appendix E.7.2, Table 24).

  • OpenBookQA diagnostics and the scope of hierarchical-inference gains. We added diagnostics showing improvements over reliability-weighted regression in all three original seeds. The observed fitting behavior suggests prior regularization as one plausible explanation for BEDI’s advantage (Appendix E.9, cross-referenced in E.5.5). Section 5.2 now explicitly states that the largest gains occur on ImageNet and OpenBookQA, while simpler regressors remain competitive on CIFAR and Dry Bean.

  • Implications of the fixed predictive mean. We explicitly describe how highly confident OOD inputs can receive lower epistemic uncertainty scores than ID inputs, impairing OOD detection, and connect this limitation to the observed results (Section 6 and Appendix E.8.2).

  • Scope of the Bayesian-reference comparison. We clarify that this experiment provides a controlled sanity check in a low-dimensional setting with a fixed representation; it does not establish comparable distributional accuracy in high-dimensional settings (Section 5.3 and Appendix E.7.1).

These updates are highlighted in red. We thank the reviewers again for helping us strengthen the evaluation and clarify the scope of our findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.