acceptodds
Under review as a conference paper at ICLR 2027

A Ground-Truth Framework for Uncertainty Disentanglement with Posterior Risk

Abstract

Reliable uncertainty estimates are critical in safety-sensitive applications. For such estimates to be useful in practice, it is crucial to understand the sources underlying a model’s uncertainty, motivating the disentanglement of total uncertainty into epistemic and aleatoric uncertainty. Existing notions of uncertainty differ in the sources they capture and, consequently, in their definitions of aleatoric and epistemic uncertainty, with no universally accepted definition. We define uncertainty through sample-conditional pointwise posterior risk, which is the expected loss of a predictor under the distribution of plausible ground-truth functions given the observed sample. This definition unifies probabilistic and risk-based concepts of uncertainty. To assess state-of-the-art uncertainty disentanglement methods, we develop a framework that directly compares their estimates against ground-truth uncertainty defined primarily by posterior risk, alongside commonly used alternative uncertainty definitions. We find that Spectral-normalized Neural Gaussian Processes and Variational Latent Gaussian Processes most closely recover the ground-truth uncertainty, while most methods track posterior variance more closely than posterior risk, missing the predictors' bias. Beyond method rankings, we investigate how strongly estimated aleatoric and epistemic uncertainty are entangled and how sensitive uncertainty quality is to modeling choices, yielding practical guidance for uncertainty disentanglement. To support further method development and validation, we release semi-synthetic UCI/OpenML datasets with known posteriors, enabling the computation of ground-truth uncertainty. Code and data will be made publicly available upon acceptance.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Disentanglement, Epistemic Uncertainty, Aleatoric Uncertainty, Posterior Risk, Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n13oOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the distinction between traditional aleatoric uncertainty and risk-based uncertainty by defining uncertainty through the sample-conditional pointwise posterior risk, which captures the expected loss of a predictor under the posterior distribution of the true function conditioned on observed samples. Based on this definition, the authors establish a unified framework that integrates both perspectives. Furthermore, the paper develops an evaluation framework for existing uncertainty quantification (UQ) disentanglement methods based on the proposed definition. To this end, the authors construct 13 semi-synthetic datasets with known generative processes and conduct a comprehensive comparison of 17 UQ disentanglement techniques under various conditions. Empirical results demonstrate that existing methods tend to track posterior variance more closely than posterior risk. The paper also provides an in-depth analysis of the entanglement between estimated aleatoric and epistemic uncertainties, as well as the impact of architectural and modeling choices on uncertainty estimation.

Critical strengths

1. Rigorous theoretical formulation and carefully validated oracle construction

The paper clearly formalizes the concept of sample-conditional posterior risk and systematically explains the components of uncertainty through Bregman decomposition. In particular, the authors construct a correctly specified reference posterior using a semi-synthetic generative process, and rigorously validate the reliability of this oracle through posterior calibration and convergence diagnostics. This represents a major strength, as the framework goes beyond a simple heuristic benchmark to enable direct evaluation of mathematically defined quantities in a controlled environment.

2. Comprehensive and carefully designed empirical evaluation

The study systematically compares 17 uncertainty estimation methods across diverse datasets, generating priors, noise structures, and data coverage conditions. Notably, it evaluates alternative uncertainty targets—such as posterior variance and posterior risk—while conducting extensive analyses including AU/EU entanglement, OOD detection, sensitivity to modeling choices, and ablation studies. Rather than merely presenting a method ranking, the experiments are designed to uncover the underlying causes and characteristics of each method, with assumptions and limitations discussed in a transparent manner.

Critical weaknesses

1. Insufficient conceptual justification for ground-truth epistemic uncertainty

While the paper acknowledges the lack of a universally agreed-upon definition of epistemic uncertainty, it adopts the posterior excess risk as the primary benchmark for ground-truth epistemic uncertainty (EU). However, posterior risk incorporates not only the generalized variance of the reference posterior but also the discrepancy of the deployed predictor. From the decision-theoretic perspective of Bickford Smith et al. (2025), the latter represents a component of predictive performance, which is distinct from predictive uncertainty. Consequently, a clear distinction must be made between the fact that posterior excess risk can be computed precisely and the claim that it represents ground-truth EU. Furthermore, the aleatoric uncertainty/epistemic uncertainty (AU/EU) decomposition inherently depends on the choice of reference prior and generative assumptions. Given these considerations, the paper requires a clearer justification regarding how the proposed quantity relates to existing probabilistic and risk-based notions of EU, and why it should be interpreted as ground-truth EU.

2. Limited component-level validation and interpretability of the benchmark comparisons

Although the paper adopts the combined target of generalized variance and deployment gap as its primary evaluation metric, it does not sufficiently isolate or analyze the magnitude and estimation quality of each individual component.

Specifically, because the deployment gap depends on the deployed predictor, each method is evaluated against a different reference target. Furthermore, while the Spearman correlation demonstrates the overall ranking agreement of total EU, it does not reveal whether this agreement is driven by the generalized variance or the deployment gap.

As a result, the following points remain unclear:

  • Whether the inability of existing methods to track posterior risk stems from limitations of the estimators themselves, or because they were originally designed to target a different uncertainty quantity.
  • How much generalized variance and deployment gap contribute individually, and how they respond to interventions such as data collection or model architecture changes.

Without a detailed component-level analysis, it is difficult to fully assess the interpretive significance of the proposed decomposition and the generalizability of the method rankings.

Questions for authors

1. Conceptual Justification of Posterior Risk

Posterior risk corresponds to the expected loss of a deployed action under a reference distribution. How do the authors justify framing this choice specifically as "uncertainty"? Furthermore, could the authors discuss how this formulation aligns with or differs from the distinctions between predictive uncertainty, predictive performance, and data dispersion outlined by Bickford Smith et al. (ICML 2025)?

2. Sensitivity to the Reference Prior

For a given tuple (S,f∗)(S, f^*), how much does the evaluation target vary when conditioned on different reference priors? For instance, when evaluating data generated by a Gaussian Process (GP) conditioned under HBART or BNN reference posteriors, could the authors report the Spearman correlations between the resulting targets and the corresponding shifts in AU/EU decomposition ratios?

3. Component-Level Evaluation

What is the Spearman correlation for each method when evaluated solely against the deployment gap target? Additionally, across different experimental conditions, what is the relative magnitude of the deployment gap compared to the generalized variance?

5 comments

We thank the reviewer for the careful review, for highlighting our formulation and validated oracle, and especially for pointing us to Bickford Smith et al. (2025), which we had regrettably overlooked. We now cite it in the introduction.

W1/Q1: Posterior risk as a reference target for predictive reliability. We agree with Bickford Smith et al. (2025) that predictive uncertainty and predictive performance are distinct, and that the deployment gap is a performance component in their taxonomy. We include it in the target because a method's uncertainty estimate is used to judge how reliable its own prediction is, and posterior risk is exactly what such an estimate should predict given the sample. For an exact, correctly specified Bayesian learner that deploys its Bayes act, the gap vanishes, and posterior risk equals its predictive uncertainty. Their subjectivity argument (Secs. 3.5–3.6) is why we keep the reference posterior separate from each estimator.

With Pf=P(⋅∣x,f)P_ f=P(\cdot\mid x,f), model predictive distribution qSq_ S and its Bayes act a^S\hat a_ S, averaging their Eqs. 4–5 under our reference posterior gives

RS=EF∣S[EY∼PF[ℓ(a^S,Y)]],R_ S=\mathbb{E}_ {F\mid S}[\mathbb{E}_ {Y\sim P_ F}[\ell(\hat a_ S,Y)]],

AUS=EF∣S[h(PF)],EUS=EF∣S[d(qS,PF)].\mathrm{AU}_ S=\mathbb{E}_ {F\mid S}[h(P_ F)],\quad \mathrm{EU}_ S=\mathbb{E}_ {F\mid S}[d(q_ S,P_ F)].

These are posterior-expected performance, dispersion, and discrepancy.

For models averaging over stochastic parameters, their Props. 2–3 interpret total uncertainty, and AU estimates as subjective Bayes estimates of performance and dispersion under quadratic estimation loss and their difference estimates discrepancy. Our reference posterior equals the generator's actual conditional distribution of FF given SS. Together with the known noise distribution, it yields externally grounded AU and EU targets for fixed deployed predictions, making estimation and disentanglement directly benchmarkable. "Ground truth" is conditional on the declared generator and loss.

Under squared loss, AUS(x)=σε2(x)\mathrm{AU}_ S(x)=\sigma_ \varepsilon^2(x) exactly, since all plausible ff share the known noise variance, while EUS=VS+(a^S−mS)2\mathrm{EU}_ S=V_ S+(\hat a_ S-m_ S)^2, with mS,VSm_ S,V_ S the reference posterior mean and variance. A model's own Bayes act makes its within-model gap zero, but its reference gap can remain and increase posterior expected loss. For an exact, correctly specified Bayesian learner deploying its Bayes act, posterior risk equals predictive uncertainty. This EU differs from their method- and policy-dependent expected uncertainty reduction (Sec. 5.2): misspecification need not vanish with more data (W2).

W2/Q3: Component-level analysis: methods' EU tracks the generalized variance but not the deployment gap, which is a significant component of uncertainty. We added the requested analysis (new App. D.3,4,9).

  • Correlation with each component. Against the deployment gap alone, every method's EU reaches only ρ=0.10\rho=0.10–0.310.31, compared with 0.22–0.86 against the generalized variance. The small correlation with the gap comes from shared location: gap and variance both grow where data are sparse (ρ(V,G)=0.17\rho(V,G)=0.17–0.300.30, higher under covariate shift), not from methods detecting their own gap.
  • Relative magnitude. The deployment gap is not small. Its median share of posterior excess risk ranges from 0.42 to 0.86 across methods. We report the deployment gap’s share of posterior risk in (new App. D.3). This highlights the relevance of considering it whenever one seeks to judge the reliability of a model's predictions.
  • Are unstable posterior risk estimations an estimator limitation or a different target? Both, and our benchmark is able to disentangle them. Every method tracks posterior variance better than posterior excess risk (e.g. VLGP 0.86 vs. 0.70), because often its EU is designed as a variance estimate and has no access to its own gap: that is a target mismatch. Among methods that target the same variance, quality still ranges from 0.22 to 0.86: that is estimator quality. We want to encourage the development of methods that approximate posterior risk.
  • Posterior risk is a different target per method. Because the gap depends on the predictor, each method's posterior risk is method-specific. This is intended, since each method is accountable for its own predictions.

W2: More data reduces the generalized variance while a more accurate specification reduces the deployment gap. The two components respond to different interventions: the generalized variance depends only on the data, while the deployment gap also depends on the model. In a controlled experiment (new Fig. 10; kin8nm, exact GP oracle, nested training sets of 64-1024 points), more data shrinks the generalized variance from 0.64 to 0.28. For a well-specified GP, the deployment gap also vanishes (0.20 →\to 0.01). For a misspecified linear GP, it grows (0.26 →\to 0.49): its error stays nearly constant (0.87 →\to 0.75) but shifts from variance to deployment gap. The right kernel family decreases it 47x at n=1024n = 1024. This is why the target must include the gap: for misspecified models, more data moves the risk into exactly the part that variance-based estimates miss.

Q2: Sensitivity to the reference prior. We ran the requested experiment (new App. D.9): the identical sample, conditioned on the generating prior and on a different prior family. AU is unaffected, since the noise is known. Where the data leave the function undetermined depends little on the prior (posterior variance ρ\rho = 0.72–0.80 between priors). The posterior excess risk agrees less (ρ\rho = 0.51–0.60), because the deployment gap is measured from a different posterior mean. The level of EU, and with it the AU/EU split, shifts with the prior (e.g., an EU share of 0.18 under the correct BNN prior vs. 0.49 under a GP prior), and mismatched posteriors are miscalibrated (90% intervals cover the true function at 37–93% of points).

We hope these additions (all included in the uploaded revision) address the reviewer’s concerns about posterior risk as a target and the questions regarding the empirical findings for the deployment gap. If so, we would be grateful if the reviewer would consider updating their score. If not, we would be glad to know what remains unclear so that we can further address it.

Dear Reviewer n13o,

The discussion period ends in 48 hours.

We have added the analyses you requested to the rebuttal, which we believe further enrich our benchmark:

  1. Component-level analysis (W2, Q3): The methods' EU tracks the generalized variance (ρ\rho from 0.22 to 0.86) but not the deployment gap (ρ\rho from 0.10 to 0.31), although the deployment gap's median share of the posterior excess risk is 0.42 to 0.86 (App. D.3).

  2. Response to interventions (W2): More data shrinks the generalized variance, while for a misspecified model the deployment gap grows (from 0.26 to 0.49), moving risk into the part that variance-based estimates miss (App. D.4, Fig. 10).

  3. Sensitivity to the reference prior (Q2): Conditioning the same sample on a different prior family largely preserves the posterior variance (ρ\rho from 0.72 to 0.80), preserves the posterior excess risk less well (ρ\rho from 0.51 to 0.60), and shifts the AU/EU split (App. D.9).

  4. Relation to Bickford Smith et al. (2025) (W1, Q1): We position posterior risk in their taxonomy as posterior-expected predictive performance and now cite the paper in the introduction.

We would greatly appreciate it if you could let us know whether these additions address your concerns.

Best wishes, The Authors

Reviewer_whpYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

Goal: Evaluating how well different methods can disentangle epistemic uncertainty (EU) and aleatoric uncertainty (AU).

Method & experiments: Their evaluation method would, in principle, be applicable to regression and classification. Their empirical evaluation is restricted to regression datasets.

Their evaluation method relies on having access to ground-truth posteriors (i.e., defining reference priors for which the posterior can be computed precisely). I think (a) this computability of the posterior restricts the methodology quite a bit and (b) that the choice of these reference priors matters more than how the paper frames it, but the rest of the methodology makes sense.

The paper spends a lot of time arguing why comparing the posterior uncertainty to the Posterior Excess Risk is a good evaluation criterion, and I find their arguments quite convincing (if realistic reference priors are selected).

They define multiple reference priors (none of them is extremely realistic in my opinion), generate semi-synthetic datasets from these priors, and conclude that rankings of methods are quite stable across their selected reference priors.

Critical strengths

The idea of using the Posterior Excess Risk makes sense in my opinion (if one has access to good reference posteriors).

The breadth of the study and showing stability across multiple different reference priors is valuable.

Critical weaknesses

W1: Restriction to reference priors with computable posterior: My personal opinion is that there are hardly any realistic priors for which posteriors are easily computable. Realistic priors should be very hierarchical in my opinion, making posteriors computationally difficult. Realistic priors should not assume known noise structure and should allow for more complicated relation structures between noise structure and mean: priors over the joint distribution of (x,y) are more realistic than priors over functions. This makes everything computationally challenging.

W2: Unrealistic priors: Have you done any type of reality check how well your priors agree with real-world data? Your priors don’t really have the desired properties mentioned in W1, I think?

E.g., I think that realistic priors should be very hierarchical in the following sense for example, that the output mainly depends on a low-dimensional latent space, but the map from inputs to this latent space and the dimension of this latent space are not known. For example, the empirical success of RealMLP supports this hypothesis. Maybe you could define a reference prior as a BNN on a 100-dimensional space where the first hidden layer is only 2-dimensional, and afterwards you have multiple wider layers to also cover such settings to some extent. Approximating this posterior precisely is computationally quite expensive, though.

E.g., lines 428-431 “SN raises local distance preservation for every method (pooled +0.21 vs. +0.01 for strong WD; Appendix D.4). For the four methods whose EU is read from a single network’s features, SN improves the correlation with the posterior excess risk by +0.10 to +0.11 and with the posterior variance by up to +0.26.” strongly suggest that you are overfitting to your synthetic reference priors. Your synthetic reference priors seem to have the property that a lot of prior mass is given to functions that change approximately equally much in all directions. Especially in deep learning applications, it is believed that most functions mainly change in a few directions of the input space, while most directions of the input space have almost no influence on the output. I think there is a very large 3rd-order uncertainty regarding the mismatch between your collection of priors and realistic priors. These experiments suggest that the discrepancy between your collection of priors and realistic priors might be quite large.

(W3: Maybe you could emphasize more why people are using variance-based estimators for epistemic uncertainty in practice, as you don’t have access to a ground truth in practice). For example, in Lines 55-56, emphasizing the lack of such ground truth in practice might be helpful. Otherwise, one might ask, why do you need uncertainty if you fix one ground truth. If you have access to the ground truth, you are not uncertain anymore.)

Questions for authors

In the following, some remarks (R) and questions (Q):

R1: In Figure 1: labeling x-axes with “x” and y-axes with “y” and maybe mentioning the word “regression” would be helpful for the reader. a^S\hat{a}_S as linear curve in Fig 1a would fit better into the overall narrative, I think, as you mentioned that underparametrization is a typical source of the deployment gap. Overall, I like Figure 1.

R2: In lines 358-359, spelling out what the abbreviations of the best and worst models correspond to ( or even quickly explaining them would be helpful.)

Q3: Lines, 366-367: “prior like their own. The best and worst methods, however, are the same under every oracle (Figure 7 in Appendix D.1).” seems surprising to me. Methods with a perfectly matched prior should always perform best on their prior with ρ=1\rho=1? Do you agree?

R4: Lines: 401-402: “less severe than previously reported (Mucs ́anyi et al., 2024).”. You study regression and they study classification, which makes a difference here. This should maybe be noted. There are obvious reasons why EU and AU are more entangled for classification than regression.

Q4: I don’t understand the sentence “We then match red- and white wine evaluation points in distance to the training data. The red wines still differ in feature combinations and have higher GP-oracle uncertainty compared to white wines (Appendix D.3).” in lines 411-412. How are these "evaluation points" defined? How do you match them? Are "evaluation points" a quantity you define or a feature of the wine dataset?

Q5: I’m not sure if I understood “The EU of VLGP and SNGP-Het, the best methods in our benchmark, hardly changes, although they track the ground-truth EU within the white-wine data far more closely than the ensembles. This has two consequences.” correctly. What hardly changes? They assign almost the same EU to white wine and red wine?

9 comments

We thank the reviewer for the thorough and constructive review, and for finding the posterior-excess-risk evaluation convincing given realistic reference priors. This makes prior realism the decisive question, and we seek to address it in three steps: (1) our priors are more realistic than their parametric families suggest, because they inherit the real covariates and kernels fitted to the real targets (2) the benchmark’s conclusions transfer to the real targets of the same datasets (3) this includes the spectral-normalization finding.

W1, W2: Our tasks inherit the feature structure of real data. To make the tasks realistic, we built them from real data wherever possible. All tasks use the real covariates of each dataset, so inputs follow the correlated distribution of real data. For the GP prior, the kernel hyperparameters are fitted per feature using automatic relevance determination (ARD), to each dataset's real targets. We expanded this description in App. B.2 Gaussian Process (GP). As a reality check, we measured how unevenly the targets depend on individual features (new App. B.8). In all three prior families, the generated targets are anisotropic: the most informative feature (mutual information with the ground truth) is 1.21.2–3.6×3.6\times as informative as the average feature (median 1.81.8–2.12.1), comparable to the real targets (1.21.2–4.74.7, median 2.32.3). Datasets with more anisotropic real targets also yield more anisotropic tasks (Spearman ρ=0.72\rho=0.72–0.840.84 across datasets). Method rankings are stable across all three prior families (Kendall's W=0.81W=0.81 on ranks, 0.920.92 on ρ\rho; Table 7), so our conclusions do not rest on a single prior.

W1, W2: The benchmark's conclusions transfer to real targets. As an additional reality check, we benchmark six datasets with their real targets, using triage as the task: deferring a model's least reliable predictions to an expert, for which posterior risk is provably the optimal score (new App. A.8 and Sec. 5). Splits and tuning are the same as in the benchmark, only the targets differ. The paper's benchmark ranking (Figure 2) predicts real triage quality: Kendall's τ=0.78\tau=0.78 without shift and 0.700.70 under covariate shift (new paragraph in Sec 5 and App. D.8).

W1, W2: The spectral-normalization finding does not depend on isotropy and holds on real targets. We agree that the SN result could in principle be biased by our priors. To check this we tested if the improvement holds on the real UCI datasets with the actual targets. It does: SN improves real-world triage by 24% (Laplace) and 20% (FSP-Laplace) (App. D.8). The SN gain is also unrelated to how anisotropic a dataset is (Spearman −0.09 for NN-Lap and +0.19 for NN-FSP across the 13 datasets; App. D.6). We added to the Limitations that, in higher-dimensional problems where only a few directions matter, preserving distances in irrelevant directions could limit the benefit of SN.

W3: We revised lines 55–56 to stress that the true function is unknown in practice. Thank you for this suggestion. Because practitioners cannot condition on the true function, they must infer EU from the data, which is why variance-based estimators are widely used. Posterior risk keeps this learner's perspective: it averages the loss over all functions that remain plausible given the sample, rather than conditioning on one known function.

Remarks and questions. Thank you for these careful comments, we adopted all of them in the revision.

  • R1: We are revising Figure 1 as suggested. Thank you for this suggestion! The caption now states that the example is a one-dimensional regression. We are re-plotting the figure with the deployed predictor a^S\hat a_{\mathcal S} as an underfit linear model (in 1a and 1b to make them correspond to each other, which illustrates the deployment gap more directly, and with axes labeled xx and yy. We will upload the updated figure in a revision within the next days.
  • R2: We now spell out the methods: VLGP (variational latent GP), SNGP-Het (heteroscedastic spectral-normalized neural GP), NN-DEUP (direct epistemic uncertainty prediction) and NN-EDL (evidential deep learning).
  • Q3: We agree: with exactly the reference prior, likelihood and exact inference, a method would reach ρ=1\rho=1. Our methods share only the prior family. The GP baseline uses a Matérn-plus-linear kernel and homoscedastic noise on every dataset rather than the generating kernel and noise model (App. C.1), the tree methods are not BART, and the NN methods use a residual architecture rather than the one-hidden-layer prior network. This is deliberate, so that no method inherits knowledge of the oracle. We now clarify it in Sec. 5.
  • R4: Good point. We now state that our entanglement results concern regression, whereas Mucsányi et al. (2024) study classification, where AU and EU estimates are seemingly more strongly coupled.
  • Q4: Evaluation points are query points: 1,500 held-out white wines and 1,500 red wines. We compute each point's Mahalanobis distance to the white-wine training inputs, cut the pooled distances into 20 quantile bins, and keep equally many red and white points per bin (134 per class on average). Both classes then have the same distance distribution, so distance alone scores an AUROC of about 0.5. We rewrote this passage in Sec. 5.
  • Q5: Yes. VLGP and SNGP-Het assign almost the same EU to the distance-matched red and white wines datasets (median ratios 1.05 and 0.85), although the true EU is 1.36$\times$ higher on the red wines. Within the white-wine data, both track the EU far more closely than the ensembles (partial Spearman ρ\rho given distance 0.90 and 0.76 vs. 0.37). They reproduce the posterior's variation within the training distribution, but not its rise on the red wines. We rephrased the sentence accordingly.

We hope these additions (all included in the uploaded revision) address the reviewer's concerns about prior realism. If so, we would be grateful if the reviewer would consider updating their score. If not, we would be glad to know what remains unclear so that we can address it further.

Thanks again for your suggestions on Figure 1. We’ve incorporated them into the revised figure, which is now available in the latest revision of the manuscript.

Reviewer_BhNsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the lack of a universally accepted ground truth for uncertainty disentanglement (aleatoric uncertainty and epistemic uncertainty) by proposing sample-conditional posterior risk, which is the expected loss of a predictor under the distribution of plausible ground-truth functions given the observed sample. To validate their framework, the authors introduce 13 semi-synthetic UCI/OpenML datasets with known posteriors, evaluating 17 state-of-the-art uncertainty disentanglement methods.

Critical strengths
  1. I agree with the authors that proxies such as OOD detection are not faithfully equivalent to the assessment of uncertainty estimation or its disentanglement. The strongest contribution is the construction of an explicit reference posterior against which uncertainty estimates can be evaluated. This addresses a genuine limitation of benchmarks that rely primarily on OOD detection.

  2. The experimental coverage is reasonably thorough, including the use of three generative prior families, multiple noise configurations, and different data-coverage settings.

Critical weaknesses
  1. Insufficient justification for the proposed ground-truth framework. The paper motivates the limitations of proxy-based uncertainty evaluation, but does not convincingly establish why posterior expected risk should be considered the ground truth for epistemic uncertainty. The proposed quantity combines posterior uncertainty with predictor-dependent error, and its interpretation as ground-truth uncertainty is not clear. More importantly, the paper does not demonstrate why evaluating against this target is preferable or more useful than existing uncertainty evaluation criteria.

  2. Limited applicability and unclear practical value. The framework relies on access to a correctly specified reference posterior, which is available by construction in the semi-synthetic experiments but generally unknown for real-world problems. While the theoretical formulation is more general, its practical evaluation procedure is difficult to apply outside controlled settings. The paper also provides limited evidence that its benchmark rankings translate into better uncertainty estimation or decision-making in real applications. The disagreement with OOD detection rankings alone does not establish the superiority of the proposed framework.

Questions for authors
  1. What makes posterior expected risk a more appropriate ground-truth uncertainty target than existing definitions, beyond being computable in the proposed benchmark?

  2. How can the framework be applied to real-world datasets where the true prior, likelihood, and reference posterior are unknown?

  3. Is there evidence that selecting methods based on the proposed benchmark leads to better uncertainty estimation or downstream decisions in practice?

4 comments

We thank the reviewer for recognizing the explicit reference posterior as our key contribution and for agreeing that proxies such as OOD detection do not faithfully evaluate uncertainty. Below we show why evaluating against ground-truth uncertainty is more informative than proxy criteria (W1, Q2), why posterior risk is the right ground truth (W1, Q1), and that the benchmark ranking transfers to real-world decisions (W2, Q3).

Evaluating against ground-truth uncertainty instead of existing evaluation criteria separates the method's performance on a task from the method's capabilities to produce faithful uncertainty estimates (W1, Q2). A proxy task scores how well a method solves one task. That mixes two questions: is the method’s uncertainty estimate accurate, and is uncertainty the right signal for this task? A poor score cannot tell them apart (Li et al., 2025; Paplhám et al., 2026), and the verdict can change with how the task is set up. In our wine experiment (App. D.5 / Figure 3b), VLGP ranks 3rd of 19 when the red wines (OOD samples) are far from the training data and 11th once distances between ID (white wines) and OOD (redwines) are matched, while the deep ensemble moves from 8th to 1st. Which method a practitioner should choose depends on a detail of the benchmark, not entirely on the methods. We agree that this instability alone does not show our framework is better. Instead, it shows that the proxy’s verdict is fragile. Evaluating against ground-truth uncertainty separates the two questions. The benchmark measures the quality of the uncertainty estimate once, and that ranking applies to tasks for which uncertainty is the right signal. Ground truth evaluation also explains the ranking shuffle between the OOD experiments. The distance-matched red wines are genuinely more uncertain (true uncertainty 1.36× higher). VLGP and SNGP-Het track the true uncertainty most closely within the training distribution but barely raise their EU on the distance-matched red wines (1.05×, 0.85×), whereas the deep and bagging ensembles do (1.49×, 1.30×).

The properties of posterior risk make it a quantity real decisions can rely on (W1, Q1). We have established why we need to evaluate against ground truth uncertainty directly. Still, we need a notion of ground truth uncertainty that is useful in the real world. Expanding on the presented properties of posterior risk in Sec. 2 and Figure 1 we want to add a real world task example that illustrates the advantages of posterior risk. In our motivating example in the introduction (referral to human experts of high uncertainty prediction, namely triage), referring the points with the highest posterior risk is Bayes-optimal for every referral budget (new App. A.8). Posterior variance is only optimal if the prediction is the posterior mean, which no practical method guarantees, demonstrating that neglecting the deployment gap can lead to overconfident predictions and wrong real-world decisions. Our semi-synthetic benchmark confirms this (new Appendix Results. D.7): ordering by ground truth posterior risk leads to better triage decisions than relying on posterior variance being only beaten by the actual expected risk, which doesn't align with the learner's perspective.

The benchmark ranking transfers to real-world triage (W2, Q3). We repeated triage on six benchmark datasets with their real targets, using all 17 methods plus the spectrally normalized Laplace variants (new paragraph in Sec. 5 and App. D.8). Splits and tuning are the same as in the benchmark, only the targets differ. The benchmark ranking aligns with real triage quality: Kendall's τ=0.70−0.78\tau=0.70-0.78. Triage quality is measured by the normalized AURC, which divides out each method's overall error, so the transfer reflects the quality of the uncertainty estimates. Specific benchmark findings also transfer: spectral normalization, which the benchmark identified as improving Laplace approximations, improves real-world triage by 24% (Laplace) and 20% (FSP-Laplace). The benchmark's results can therefore directly inform method choice for real-world tasks.

The framework validates methods where the truth is known, before they are applied to real data (Q2). Ground-truth uncertainty cannot be computed on real data. Our tasks keep the real covariates and synthesize only the targets, and the real-target experiment above shows that rankings and fixes carry over. We release all 3,120 task instances with posterior draws so that new methods can be validated before deployment, and we now state this role more explicitly in Sec. 1 and 6.

We hope these additions (all included in the uploaded revision) address the reviewer’s concerns about posterior risk as a target and the practical value of the framework. If so, we would be grateful if the reviewer would consider updating their score. If not, we would be glad to know what remains unclear so that we can further address it.

Dear Reviewer BhNs,

The discussion period ends in 48 hours.

We have added new results addressing your questions to the rebuttal, which we believe further strengthen our benchmark:

  1. Real-world transfer (Q3): On six datasets with their real targets, the methods' triage quality follows the benchmark ranking (Kendall's τ=0.78\tau = 0.78 balanced, 0.700.70 under covariate shift; App. D.8). Spectral normalization, which the benchmark identified as improving Laplace methods, also improves real-world triage.

  2. Usefulness of the target (W1, Q1): Ranking by posterior risk is Bayes-optimal for triage at every referral budget (new App. A.8). On the synthetic tasks, ranking by posterior risk leads to better triage decisions than ranking by posterior variance (App. D.7).

  3. Fragility of proxy evaluation (W1): In the distance-matched wine OOD experiment, the proxy ranking changes with the task setup, while ground-truth uncertainty explains why (App. D.5).

We would greatly appreciate it if you could let us know whether these additions address your concerns.

Best wishes, The Authors

I thank the authors for replying to my comments. I agree with the paper's motivation that the community lacks a convincing evaluation framework for uncertainty disentanglement. However, I remain unconvinced that the proposed framework bridges this gap since computing the reference posterior requires knowing the data-generating prior while in real-world tasks they are unknown. Therefore I find no strong reasons to overturn my original rating.

Thank you for engaging with our rebuttal. We agree that the data-generating prior is unknown in real-world tasks. This is exactly why a ground-truth benchmark has to be (semi-)synthetic. True uncertainty is never observed in real data, so any evaluation against a ground truth must generate the targets from a known prior. This known prior is what makes the explicit reference posterior possible, which the reviewer named as our strongest contribution. What matters is whether conclusions drawn where the prior is known hold on real data where it is not. Our results show that they do:

  1. The conclusions do not depend on knowing the right prior. Across three different prior families (GP, HBART, BNN), the method ranking is stable (Kendall's W≥0.81W \geq 0.81). The best (GP, VLGP, SNGP-Het) and worst (NN-DEUP, NN-EDL) methods are the same under every prior (App. D.1).
  2. The conclusions transfer to real targets, where the prior is unknown. On six datasets with their real targets, triage quality follows the benchmark ranking (Kendall's τ=0.70\tau = 0.70-0.780.78; App. D.8). Spectral normalization, which the benchmark identified as improving Laplace methods, also improves real-world triage by 24% (Laplace) and 20% (FSP-Laplace).

As this was the remaining reason for the current rating, and we have addressed it with evidence on real data, we kindly ask the reviewer to reconsider their score.

Comments on the paper

1 comment

Dear AC and reviewers,

We sincerely thank all three reviewers for their thorough reviews and their engagement throughout the discussion. Their suggestions led to several new analyses and experiments, and we believe the paper is substantially stronger as a result. Below we summarize where the discussion stands.

Reviewer positions after the discussion

  • n13o (3, weak accept) keeps a positive evaluation: the component-level and prior-sensitivity analyses “have addressed my concerns”, and “the proposed paper has merit”. One conceptual point remains (see below).
  • whpY (2) wrote that the revision “did significantly improve the paper” and that their current score is a "2.6" (with the possibility to raise it to a 3 in the closed discussion). They also said they would raise Presentation from 2 to 3.
  • BhNs (2) keeps one concern, that the reference posterior requires a known prior (see below).

Strengths noted by reviewers: the explicit reference posterior as ground truth (BhNs, n13o), posterior excess risk as the target (whpY), breadth and stability across priors (whpY, n13o), and rigorous oracle validation (n13o).

Added during the discussion (all in the revised PDF):

  1. Transfer to real data. On six datasets with their real targets, triage quality follows the benchmark ranking (Kendall's τ=0.70\tau = 0.70–0.780.78, App. D.8). In an EU-dominated regime suggested by whpY, the EU-only ranking also agrees (Kendall's τ=0.60\tau = 0.60). Spectral normalization improves real-world triage by 24% (Laplace) and 20% (FSP-Laplace).
  2. Why posterior risk. Ranking by posterior risk is Bayes-optimal for triage at every referral budget (App. A.8) and beats ranking by posterior variance (App. D.7).
  3. Component-level analysis (suggested by n13o). Methods' EU tracks the generalized variance (ρ=0.22\rho = 0.22–0.860.86) but not the deployment gap (ρ=0.10\rho = 0.10–0.310.31), even though the gap makes up 42–86% of posterior excess risk (App. D.3).
  4. Realistic priors (suggested by whpY). ARD kernels fitted to real targets give anisotropy comparable to the real data (App. B.8). Rankings are stable across the three prior families (Kendall's W≥0.81W \geq 0.81, Table 7), and App. D.9 adds a sensitivity analysis on the reference prior requested by n13o.

Remaining concerns

  1. The reference posterior needs a known prior (BhNs, whpY). This holds for any ground-truth benchmark, because true uncertainty is never observed in real data. We address it by showing that conclusions are stable across prior families and carry over to real targets (items 1 and 4 above).
  2. Is the deployment gap part of epistemic uncertainty? (n13o) n13o views the deployment gap as predictive performance rather than epistemic uncertainty. We agree the two are conceptually distinct and report both targets (App. D.3). We include the gap because tasks such as triage need to know where the deployed predictor is wrong, and there, posterior risk is Bayes-optimal (App. A.8) and outperforms posterior variance (App. D.7).

Once more, we are very grateful to the reviewers and the AC for their time and constructive feedback.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.