- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses the distinction between traditional aleatoric uncertainty and risk-based uncertainty by defining uncertainty through the sample-conditional pointwise posterior risk, which captures the expected loss of a predictor under the posterior distribution of the true function conditioned on observed samples. Based on this definition, the authors establish a unified framework that integrates both perspectives. Furthermore, the paper develops an evaluation framework for existing uncertainty quantification (UQ) disentanglement methods based on the proposed definition. To this end, the authors construct 13 semi-synthetic datasets with known generative processes and conduct a comprehensive comparison of 17 UQ disentanglement techniques under various conditions. Empirical results demonstrate that existing methods tend to track posterior variance more closely than posterior risk. The paper also provides an in-depth analysis of the entanglement between estimated aleatoric and epistemic uncertainties, as well as the impact of architectural and modeling choices on uncertainty estimation.
Critical strengths
1. Rigorous theoretical formulation and carefully validated oracle construction
The paper clearly formalizes the concept of sample-conditional posterior risk and systematically explains the components of uncertainty through Bregman decomposition. In particular, the authors construct a correctly specified reference posterior using a semi-synthetic generative process, and rigorously validate the reliability of this oracle through posterior calibration and convergence diagnostics. This represents a major strength, as the framework goes beyond a simple heuristic benchmark to enable direct evaluation of mathematically defined quantities in a controlled environment.
2. Comprehensive and carefully designed empirical evaluation
The study systematically compares 17 uncertainty estimation methods across diverse datasets, generating priors, noise structures, and data coverage conditions. Notably, it evaluates alternative uncertainty targets—such as posterior variance and posterior risk—while conducting extensive analyses including AU/EU entanglement, OOD detection, sensitivity to modeling choices, and ablation studies. Rather than merely presenting a method ranking, the experiments are designed to uncover the underlying causes and characteristics of each method, with assumptions and limitations discussed in a transparent manner.
Critical weaknesses
1. Insufficient conceptual justification for ground-truth epistemic uncertainty
While the paper acknowledges the lack of a universally agreed-upon definition of epistemic uncertainty, it adopts the posterior excess risk as the primary benchmark for ground-truth epistemic uncertainty (EU). However, posterior risk incorporates not only the generalized variance of the reference posterior but also the discrepancy of the deployed predictor. From the decision-theoretic perspective of Bickford Smith et al. (2025), the latter represents a component of predictive performance, which is distinct from predictive uncertainty. Consequently, a clear distinction must be made between the fact that posterior excess risk can be computed precisely and the claim that it represents ground-truth EU. Furthermore, the aleatoric uncertainty/epistemic uncertainty (AU/EU) decomposition inherently depends on the choice of reference prior and generative assumptions. Given these considerations, the paper requires a clearer justification regarding how the proposed quantity relates to existing probabilistic and risk-based notions of EU, and why it should be interpreted as ground-truth EU.
2. Limited component-level validation and interpretability of the benchmark comparisons
Although the paper adopts the combined target of generalized variance and deployment gap as its primary evaluation metric, it does not sufficiently isolate or analyze the magnitude and estimation quality of each individual component.
Specifically, because the deployment gap depends on the deployed predictor, each method is evaluated against a different reference target. Furthermore, while the Spearman correlation demonstrates the overall ranking agreement of total EU, it does not reveal whether this agreement is driven by the generalized variance or the deployment gap.
As a result, the following points remain unclear:
- Whether the inability of existing methods to track posterior risk stems from limitations of the estimators themselves, or because they were originally designed to target a different uncertainty quantity.
- How much generalized variance and deployment gap contribute individually, and how they respond to interventions such as data collection or model architecture changes.
Without a detailed component-level analysis, it is difficult to fully assess the interpretive significance of the proposed decomposition and the generalizability of the method rankings.
Questions for authors
1. Conceptual Justification of Posterior Risk
Posterior risk corresponds to the expected loss of a deployed action under a reference distribution. How do the authors justify framing this choice specifically as "uncertainty"? Furthermore, could the authors discuss how this formulation aligns with or differs from the distinctions between predictive uncertainty, predictive performance, and data dispersion outlined by Bickford Smith et al. (ICML 2025)?
2. Sensitivity to the Reference Prior
For a given tuple , how much does the evaluation target vary when conditioned on different reference priors? For instance, when evaluating data generated by a Gaussian Process (GP) conditioned under HBART or BNN reference posteriors, could the authors report the Spearman correlations between the resulting targets and the corresponding shifts in AU/EU decomposition ratios?
3. Component-Level Evaluation
What is the Spearman correlation for each method when evaluated solely against the deployment gap target? Additionally, across different experimental conditions, what is the relative magnitude of the deployment gap compared to the generalized variance?
5 comments
W2/Q3: Component-level analysis: methods' EU tracks the generalized variance but not the deployment gap, which is a significant component of uncertainty. We added the requested analysis (new App. D.3,4,9).
- Correlation with each component. Against the deployment gap alone, every method's EU reaches only –, compared with 0.22–0.86 against the generalized variance. The small correlation with the gap comes from shared location: gap and variance both grow where data are sparse (–, higher under covariate shift), not from methods detecting their own gap.
- Relative magnitude. The deployment gap is not small. Its median share of posterior excess risk ranges from 0.42 to 0.86 across methods. We report the deployment gap’s share of posterior risk in (new App. D.3). This highlights the relevance of considering it whenever one seeks to judge the reliability of a model's predictions.
- Are unstable posterior risk estimations an estimator limitation or a different target? Both, and our benchmark is able to disentangle them. Every method tracks posterior variance better than posterior excess risk (e.g. VLGP 0.86 vs. 0.70), because often its EU is designed as a variance estimate and has no access to its own gap: that is a target mismatch. Among methods that target the same variance, quality still ranges from 0.22 to 0.86: that is estimator quality. We want to encourage the development of methods that approximate posterior risk.
- Posterior risk is a different target per method. Because the gap depends on the predictor, each method's posterior risk is method-specific. This is intended, since each method is accountable for its own predictions.
W2: More data reduces the generalized variance while a more accurate specification reduces the deployment gap. The two components respond to different interventions: the generalized variance depends only on the data, while the deployment gap also depends on the model. In a controlled experiment (new Fig. 10; kin8nm, exact GP oracle, nested training sets of 64-1024 points), more data shrinks the generalized variance from 0.64 to 0.28. For a well-specified GP, the deployment gap also vanishes (0.20 0.01). For a misspecified linear GP, it grows (0.26 0.49): its error stays nearly constant (0.87 0.75) but shifts from variance to deployment gap. The right kernel family decreases it 47x at . This is why the target must include the gap: for misspecified models, more data moves the risk into exactly the part that variance-based estimates miss.
Q2: Sensitivity to the reference prior. We ran the requested experiment (new App. D.9): the identical sample, conditioned on the generating prior and on a different prior family. AU is unaffected, since the noise is known. Where the data leave the function undetermined depends little on the prior (posterior variance = 0.72–0.80 between priors). The posterior excess risk agrees less ( = 0.51–0.60), because the deployment gap is measured from a different posterior mean. The level of EU, and with it the AU/EU split, shifts with the prior (e.g., an EU share of 0.18 under the correct BNN prior vs. 0.49 under a GP prior), and mismatched posteriors are miscalibrated (90% intervals cover the true function at 37–93% of points).
We hope these additions (all included in the uploaded revision) address the reviewer’s concerns about posterior risk as a target and the questions regarding the empirical findings for the deployment gap. If so, we would be grateful if the reviewer would consider updating their score. If not, we would be glad to know what remains unclear so that we can further address it.
Dear Reviewer n13o,
The discussion period ends in 48 hours.
We have added the analyses you requested to the rebuttal, which we believe further enrich our benchmark:
-
Component-level analysis (W2, Q3): The methods' EU tracks the generalized variance ( from 0.22 to 0.86) but not the deployment gap ( from 0.10 to 0.31), although the deployment gap's median share of the posterior excess risk is 0.42 to 0.86 (App. D.3).
-
Response to interventions (W2): More data shrinks the generalized variance, while for a misspecified model the deployment gap grows (from 0.26 to 0.49), moving risk into the part that variance-based estimates miss (App. D.4, Fig. 10).
-
Sensitivity to the reference prior (Q2): Conditioning the same sample on a different prior family largely preserves the posterior variance ( from 0.72 to 0.80), preserves the posterior excess risk less well ( from 0.51 to 0.60), and shifts the AU/EU split (App. D.9).
-
Relation to Bickford Smith et al. (2025) (W1, Q1): We position posterior risk in their taxonomy as posterior-expected predictive performance and now cite the paper in the introduction.
We would greatly appreciate it if you could let us know whether these additions address your concerns.
Best wishes, The Authors
We thank the reviewer for the careful review, for highlighting our formulation and validated oracle, and especially for pointing us to Bickford Smith et al. (2025), which we had regrettably overlooked. We now cite it in the introduction.
W1/Q1: Posterior risk as a reference target for predictive reliability. We agree with Bickford Smith et al. (2025) that predictive uncertainty and predictive performance are distinct, and that the deployment gap is a performance component in their taxonomy. We include it in the target because a method's uncertainty estimate is used to judge how reliable its own prediction is, and posterior risk is exactly what such an estimate should predict given the sample. For an exact, correctly specified Bayesian learner that deploys its Bayes act, the gap vanishes, and posterior risk equals its predictive uncertainty. Their subjectivity argument (Secs. 3.5–3.6) is why we keep the reference posterior separate from each estimator.
With Pf=P(⋅∣x,f), model predictive distribution qS and its Bayes act a^S, averaging their Eqs. 4–5 under our reference posterior gives
RS=EF∣S[EY∼PF[ℓ(a^S,Y)]],
AUS=EF∣S[h(PF)],EUS=EF∣S[d(qS,PF)].
These are posterior-expected performance, dispersion, and discrepancy.
For models averaging over stochastic parameters, their Props. 2–3 interpret total uncertainty, and AU estimates as subjective Bayes estimates of performance and dispersion under quadratic estimation loss and their difference estimates discrepancy. Our reference posterior equals the generator's actual conditional distribution of F given S. Together with the known noise distribution, it yields externally grounded AU and EU targets for fixed deployed predictions, making estimation and disentanglement directly benchmarkable. "Ground truth" is conditional on the declared generator and loss.
Under squared loss, AUS(x)=σε2(x) exactly, since all plausible f share the known noise variance, while EUS=VS+(a^S−mS)2, with mS,VS the reference posterior mean and variance. A model's own Bayes act makes its within-model gap zero, but its reference gap can remain and increase posterior expected loss. For an exact, correctly specified Bayesian learner deploying its Bayes act, posterior risk equals predictive uncertainty. This EU differs from their method- and policy-dependent expected uncertainty reduction (Sec. 5.2): misspecification need not vanish with more data (W2).