- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces AUB-Wiki: a deterministically generated (LLM-free) benchmark with interpretation count as the aleatoric axis and sitelink coverage as the epistemic axis, plus matched unambiguous controls. The authors validate both axes behaviorally and benchmark SE, P(True), an ambiguity probe, and the SD/ICE decompositions. Key findings: total uncertainty scores do not track ambiguity (SE anticorrelates), while the direct probe works; SD/ICE components align with their intended axes.
Critical strengths
Existing ambiguity datasets provide only an aleatoric reference; this is the first benchmark where both components can be checked against independent references on the same questions — a prerequisite for evaluating disentanglement. The motivation is well argued, borrowing from and going beyond Mucsányi et al. (2024) in vision.
Critical weaknesses
-
The knowledge proxy degrades for frontier models. Coverage AUROC falls with capability (Opus 5: 0.65, still answering 58% of low-coverage clarifications correctly) — sitelinks are a poor knowledge proxy for models trained on corpora far beyond Wikipedia, which is exactly where evaluation matters most.
-
Weak evidence for "disentanglement". The axes are engineered to be near-orthogonal (ρ≈−0.12) and disentanglement is shown only via marginal Spearman correlations. No calibration, additivity, or covarying-sources test is provided. The ρ=−0.19 between SD components (Fig. 1) may be a mechanical consequence of the entropy-subtraction identity rather than true disentanglement.
-
Thin method coverage. Only SD/ICE are tested — one clarification-ensembling family whose aleatoric term is structurally aligned with the benchmark's ambiguity axis. The results may be a property of the method family, not of decomposition methods generally.
Questions for authors
-
Rerun SD/ICE using ground-truth clarifications instead of Sol's: do the aleatoric AUROC and the axis correlations survive?
-
How robust is Table 4 to the choice of clarifier (e.g., an open-weight model)?
-
Why does SE anticorrelate with interpretation count even when the model answers all clarifications correctly?
-
How should the knowledge axis be fixed as coverage becomes uninformative for stronger models?
6 comments
Comment 2/5
W2: Evidence for disentanglement
Why the axes are near-orthogonal by design. As discussed by Mucsányi et al. [3], many datasets have an inevitable entanglement of aleatoric and epistemic uncertainty as the ground-truth signals themselves are correlated. In image classification, for instance, ambiguous images are undersampled in training sets, so they automatically have a high epistemic uncertainty as well. On such data, a correlation between the estimated components is confounded: it may reflect a failure of the decomposition or simply the correlation already present between the ground-truth quantities. We therefore constructed AUB-Wiki with near-orthogonal ground-truth axes to remove this confounder. If the estimated components are correlated on AUB-Wiki, the correlation can only be attributed to the estimator and is a limitation of the decomposition; if they are not, the decomposition reflects the decorrelation of the ground truth. Near-orthogonality is thus the property that makes the disentanglement test conclusive, not a weakness of it.
Is the low correlation in Figure 1 a consequence of the entropy-subtraction identity? It is not. Mucsányi et al. [3] use the same entropy-subtraction identity, with an ensemble over models instead of clarifications, and find strongly correlated components (rank correlation of at least 0.78 across methods). To test this directly on AUB-Wiki, we kept the SD formula, embedder, kernel and entropy unchanged and changed only the groups. Instead of one group per clarification, the groups are now six models (the five open models and Llama-3.1-8B), each with 10 answers sampled for the ambiguous question. As in the standard ensemble decomposition [3, 4], the roles of the two terms swap: the spread inside a model is aleatoric, and the disagreement across models is epistemic. This new setting provides a Spearman correlation of between the aleatoric and the epistemic estimates, which reflects the expected correlation structure between the outputs of the 6 models. The subtraction identity therefore does not by itself produce a low or negative correlation between the two terms.
Beyond marginal correlations: conditional effects (new result). To test the effect of each axis (i.e. coverage and counts) while holding the other fixed, we report partial Spearman correlations of each uncertainty component with one axis, controlling for the other, on the same 768 records as Table 4. The partial correlations differ from the marginal ones by at most 0.07: the aleatoric term remains strongly associated with interpretation count given coverage ( to ) and unrelated to coverage given count ( to ). The epistemic term remains associated with coverage given count ( to ) and only weakly with count given coverage ( to ). This table is also added to Appendix G2 in the revised paper.
| Model | ||||
|---|---|---|---|---|
| Qwen3.5-4B | +0.70/+0.71 | +0.04/+0.07 | -0.14/-0.15 | -0.37/-0.29 |
| Qwen3.5-9B | +0.70/+0.73 | +0.04/+0.03 | -0.08/-0.10 | -0.35/-0.30 |
| Qwen3.5-27B | +0.71/+0.73 | +0.03/+0.04 | -0.07/-0.08 | -0.36/-0.33 |
| Gemma 4 12B | +0.59/+0.72 | +0.05/+0.08 | -0.14/-0.15 | -0.38/-0.30 |
| Phi-4 14B | +0.58/+0.73 | -0.02/-0.01 | -0.10/-0.11 | -0.33/-0.24 |
Comment 3/5
W3: Method coverage
Following the reviewer's suggestion, we now add two decomposition methods that do not use clarifications and that rely on different decomposition formulas than SD and ICE:
- Cross-model disagreement [5]: for a reference model, the aleatoric term is one minus the mean cosine similarity between its own sampled answers, and the total uncertainty is one minus the mean similarity between its answers and those of a set of models; the epistemic term is the difference. Each of our five open models serves once as the reference, and the model set comprises the five open models plus Llama-3.1-8B, added because it differs in family, architecture and training data.
- Ensemble of ensembles [6]: stochastic decoding of a fixed prompt forms the inner ensemble, and meaning-preserving paraphrases of the prompt form the outer ensemble (10 paraphrases × 10 sampled answers per paraphrase, for each open model); the law of total covariance in embedding space yields the aleatoric/epistemic split (this is contemporary work that will appear at NeurIPS 2026 and was published on arXiv just before the rebuttal period). The table below reports the Table 4 quantities on the same 768 ambiguous questions and 418 controls. Neither new baseline method tracks either axis, and their aleatoric terms do not separate ambiguous records from their controls (AUROC 0.41–0.51). Implementation details can be found in Appendix G.3.
| Method | AUROC | |||||
|---|---|---|---|---|---|---|
| SD | 0.88 – 0.90 | -0.37 – -0.32 | -0.11 – -0.04 | +0.58 – +0.71 | -0.05 – 0.00 | -0.23 – -0.15 |
| ICE | 0.88 – 0.89 | -0.33 – -0.23 | -0.13 – -0.05 | +0.71 – +0.73 | -0.06 – +0.01 | -0.14 – -0.06 |
| Cross-model | 0.43 – 0.51 | -0.01 – +0.06 | 0.00 – +0.12 | -0.20 – +0.10 | -0.09 – +0.02 | -0.54 – -0.35 |
| Ensemble of ensembles | 0.41 – 0.44 | +0.08 – +0.11 | -0.02 – +0.12 | -0.21 – -0.05 | -0.09 – -0.02 | -0.33 – -0.14 |
Furthermore, we already report results for additional methods that provide a single uncertainty estimate, and evaluate SE and P(True), and an ambiguity probe.
Both new baseline methods fail on AUB-wiki as they derive their aleatoric term from the spread of answers sampled for the (ambiguous) question itself. This assumes that ambiguity is expressed as diversity in the sampled answers. Our results in Section 4.3 show that this is not the case: semantic entropy computed on the ambiguous question does not increase with interpretation count and even decreases (Table 2), also when the model answers every clarification correctly (Table 11). Models tend to commit to one interpretation rather than spread their answers across interpretations. This agrees with prior findings that existing uncertainty estimators degrade to near-random performance on ambiguous questions [7], and that approximating model ensembles by varying in-context examples fails to separate ambiguous from unambiguous questions [8]. Paraphrases preserve the ambiguity of the question and therefore do not expose it either.
On whether the results are a property of the method family. We agree with the reviewer that the disentanglement result is specific to clarification-based decompositions, and our new comparison confirms this. On AUB-Wiki, methods that explicitly enumerate the interpretations of a question infer the least entangled uncertainty estimates. We designed AUB-Wiki with explicitly disentangled knowledge and ambiguity axes to uncover such method-specific properties. In the revised manuscript, we now discuss this explicitly in the paper (Appendix G3), report the main caveat of the clarifier-based family and quantify how much of the aleatoric signal is driven by the clarifier (Table 4, last row).
Comment 1/5
We thank the reviewer for the detailed feedback and for recognizing the strengths of our work, in particular that AUB-Wiki is the first benchmark in which the uncertainty components can be checked against independent references on the same questions, a prerequisite for evaluating uncertainty disentanglement. We also appreciate that the reviewer found the motivation well argued.
We address the reviewer's concerns and requests below.
W1: The knowledge proxy degrades for frontier models
We agree that coverage is less discriminative for the strongest models because they answer more low-coverage facts correctly. We therefore now present a new site-link independent evaluation of epistemic uncertainty via answer correctness for additional strong models, and show that AUB-Wiki is not saturated. In addition, we argue that (i) also coverage remains informative for Opus 5 and (ii) a model-independent proxy is needed for disentanglement evaluation. We detail our reasoning below.
A coverage independent evaluation of epistemic uncertainty For evaluations focused on epistemic uncertainty itself rather than on disentanglement, we now present a coverage-independent analysis of the quality of epistemic uncertainty via answer correctness by computing the record-level accuracy of the target model's answers to the ground-truth clarifications, ∣Ir∣1∑i∈IrCM(qr,i). Since individual clarifications are not ambiguous, this fraction of correctly answered questions is a direct measure for the epistemic uncertainty of the evaluated model. We evaluate the epistemic terms against this reference (experiment and results added to Section 4.3). The epistemic term correlates much more strongly with the model's own record accuracy (ρ=−0.40 to −0.68) than with coverage (−0.23 to −0.37), while remaining nearly unrelated to interpretation count (−0.04 to −0.13). AUB-Wiki thus supports two complementary evaluations: a model-independent one (coverage) for disentanglement with decorrelated references, and a model-dependent one (record accuracy) for the epistemic quality of a specific model. We also verify that the model-dependent record accuracy is not saturated for frontier models, as its value ranges from 0 to 1 for Opus 5 and is not concentrated around 1.
Coverage remains informative for Opus 5. Per-clarification accuracy still increases monotonically across coverage bins for Opus 5, from 0.58 for the least-covered clarifications to 0.94 for the most-covered ones (Table 1). This range is narrower than for the open models, but the global trend is clearly present and with Opus 5 still failing on 42% of the least-covered clarifications, the benchmark is far from saturated. Since AUB-Wiki is designed to compare uncertainty methods on the same model, a weaker coverage signal for one model affects all compared methods equally. It only limits comparing a single estimator across model sizes using coverage alone. Coverage is also not specific to Wikipedia: the facts in AUB-Wiki are widely replicated across the web, and prior work shows that LLM factual accuracy tracks how often an entity appears in the whole pretraining corpus [1] and that Wikipedia-based popularity predicts factual knowledge [2]. Importantly, we have revised the paper to suggest correctness-based evaluations, which are independent of coverage, for evaluations focused on epistemic uncertainty rather than on disentanglement.
Why a model-independent proxy is needed. A central goal of AUB-Wiki is to enable disentanglement evaluation, which requires that the ground-truth aleatoric and epistemic signals be decorrelated by construction (see W2). Decorrelation can only be guaranteed at construction time with a model-independent knowledge signal, since a model-dependent one is unknown when the dataset is built. At the same time, epistemic uncertainty is by definition model-dependent, so any model-independent proxy involves a trade-off. We resolve it by using a model-independent proxy that enables decorrelated references through stratified sampling, and by validating that this proxy aligns with each evaluated model's knowledge (V1 check), at the cost of weaker alignment for the most capable models.