acceptodds
Under review as a conference paper at ICLR 2027

AUB-Wiki: Benchmarking Uncertainty Decomposition in Language Models under Controlled Ambiguity

Abstract

Uncertainty in Language Model (LM) answers is commonly decomposed into aleatoric uncertainty, arising from ambiguity in the question, and epistemic uncertainty, arising from missing model knowledge. Evaluating such decompositions requires a benchmark on which the two sources can be measured separately. Existing ambiguous-question datasets do not allow this: at best, they provide ground truth for ambiguity, and they lack an independent measure of knowledge difficulty for each clarification, so a method's aleatoric and epistemic estimates cannot be checked against separate references. We introduce Ambiguity and Uncertainty Benchmarking Wiki (AUB-Wiki), a benchmark of referentially ambiguous questions generated deterministically from Wikidata without any language model in the loop. Each question is paired with the complete set of clarifications implied by the knowledge graph, and each clarification has a verified answer and a model-independent coverage score. Most questions also have a matched unambiguous control. The number of clarifications and their coverage form two near-independent axes of ambiguity and knowledge difficulty. We use AUB-Wiki to benchmark uncertainty quantification and uncertainty decomposition methods across several open-weight language models, evaluating whether they attribute uncertainty to ambiguity or to incomplete knowledge. The benchmark and generation pipeline are released to support controlled evaluation of uncertainty decomposition.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Uncertainty in language models, Uncertainty quantification, Uncertainty decomposition, Aleatoric uncertainty, Epistemic uncertainty, Language models.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ToLbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces AUB-Wiki: a deterministically generated (LLM-free) benchmark with interpretation count as the aleatoric axis and sitelink coverage as the epistemic axis, plus matched unambiguous controls. The authors validate both axes behaviorally and benchmark SE, P(True), an ambiguity probe, and the SD/ICE decompositions. Key findings: total uncertainty scores do not track ambiguity (SE anticorrelates), while the direct probe works; SD/ICE components align with their intended axes.

Critical strengths

Existing ambiguity datasets provide only an aleatoric reference; this is the first benchmark where both components can be checked against independent references on the same questions — a prerequisite for evaluating disentanglement. The motivation is well argued, borrowing from and going beyond Mucsányi et al. (2024) in vision.

Critical weaknesses
  1. The knowledge proxy degrades for frontier models. Coverage AUROC falls with capability (Opus 5: 0.65, still answering 58% of low-coverage clarifications correctly) — sitelinks are a poor knowledge proxy for models trained on corpora far beyond Wikipedia, which is exactly where evaluation matters most.

  2. Weak evidence for "disentanglement". The axes are engineered to be near-orthogonal (ρ≈−0.12) and disentanglement is shown only via marginal Spearman correlations. No calibration, additivity, or covarying-sources test is provided. The ρ=−0.19 between SD components (Fig. 1) may be a mechanical consequence of the entropy-subtraction identity rather than true disentanglement.

  3. Thin method coverage. Only SD/ICE are tested — one clarification-ensembling family whose aleatoric term is structurally aligned with the benchmark's ambiguity axis. The results may be a property of the method family, not of decomposition methods generally.

Questions for authors
  1. Rerun SD/ICE using ground-truth clarifications instead of Sol's: do the aleatoric AUROC and the axis correlations survive?

  2. How robust is Table 4 to the choice of clarifier (e.g., an open-weight model)?

  3. Why does SE anticorrelate with interpretation count even when the model answers all clarifications correctly?

  4. How should the knowledge axis be fixed as coverage becomes uninformative for stronger models?

6 comments
AuthorsOpenReview ↗

Comment 1/5

We thank the reviewer for the detailed feedback and for recognizing the strengths of our work, in particular that AUB-Wiki is the first benchmark in which the uncertainty components can be checked against independent references on the same questions, a prerequisite for evaluating uncertainty disentanglement. We also appreciate that the reviewer found the motivation well argued.

We address the reviewer's concerns and requests below.

W1: The knowledge proxy degrades for frontier models

We agree that coverage is less discriminative for the strongest models because they answer more low-coverage facts correctly. We therefore now present a new site-link independent evaluation of epistemic uncertainty via answer correctness for additional strong models, and show that AUB-Wiki is not saturated. In addition, we argue that (i) also coverage remains informative for Opus 5 and (ii) a model-independent proxy is needed for disentanglement evaluation. We detail our reasoning below.

A coverage independent evaluation of epistemic uncertainty For evaluations focused on epistemic uncertainty itself rather than on disentanglement, we now present a coverage-independent analysis of the quality of epistemic uncertainty via answer correctness by computing the record-level accuracy of the target model's answers to the ground-truth clarifications, 1∣Ir∣∑i∈IrCM(qr,i)\frac{1}{|\mathcal{I}_r|}\sum_{i \in \mathcal{I}_r} C_M(q_{r,i}). Since individual clarifications are not ambiguous, this fraction of correctly answered questions is a direct measure for the epistemic uncertainty of the evaluated model. We evaluate the epistemic terms against this reference (experiment and results added to Section 4.3). The epistemic term correlates much more strongly with the model's own record accuracy (ρ=−0.40\rho = -0.40 to −0.68-0.68) than with coverage (−0.23-0.23 to −0.37-0.37), while remaining nearly unrelated to interpretation count (−0.04-0.04 to −0.13-0.13). AUB-Wiki thus supports two complementary evaluations: a model-independent one (coverage) for disentanglement with decorrelated references, and a model-dependent one (record accuracy) for the epistemic quality of a specific model. We also verify that the model-dependent record accuracy is not saturated for frontier models, as its value ranges from 0 to 1 for Opus 5 and is not concentrated around 1.

Coverage remains informative for Opus 5. Per-clarification accuracy still increases monotonically across coverage bins for Opus 5, from 0.58 for the least-covered clarifications to 0.94 for the most-covered ones (Table 1). This range is narrower than for the open models, but the global trend is clearly present and with Opus 5 still failing on 42% of the least-covered clarifications, the benchmark is far from saturated. Since AUB-Wiki is designed to compare uncertainty methods on the same model, a weaker coverage signal for one model affects all compared methods equally. It only limits comparing a single estimator across model sizes using coverage alone. Coverage is also not specific to Wikipedia: the facts in AUB-Wiki are widely replicated across the web, and prior work shows that LLM factual accuracy tracks how often an entity appears in the whole pretraining corpus [1] and that Wikipedia-based popularity predicts factual knowledge [2]. Importantly, we have revised the paper to suggest correctness-based evaluations, which are independent of coverage, for evaluations focused on epistemic uncertainty rather than on disentanglement.

Why a model-independent proxy is needed. A central goal of AUB-Wiki is to enable disentanglement evaluation, which requires that the ground-truth aleatoric and epistemic signals be decorrelated by construction (see W2). Decorrelation can only be guaranteed at construction time with a model-independent knowledge signal, since a model-dependent one is unknown when the dataset is built. At the same time, epistemic uncertainty is by definition model-dependent, so any model-independent proxy involves a trade-off. We resolve it by using a model-independent proxy that enables decorrelated references through stratified sampling, and by validating that this proxy aligns with each evaluated model's knowledge (V1V_1 check), at the cost of weaker alignment for the most capable models.

epistemic (within clar.)
modelρ\rho(count)ρ\rho(coverage)ρ\rho(accuracy)
SD / ICESD / ICESD / ICE
Qwen3.5-4B-0.10 / -0.12-0.36 / -0.28-0.60 / -0.59
Qwen3.5-9B-0.05 / -0.07-0.34 / -0.29-0.66 / -0.67
Qwen3.5-27B-0.04 / -0.05-0.36 / -0.33-0.67 / -0.68
Gemma 4 12B-0.11 / -0.13-0.37 / -0.29-0.43 / -0.40
Phi-4 14B-0.07 / -0.09-0.32 / -0.23-0.57 / -0.66
AuthorsOpenReview ↗

Comment 2/5

W2: Evidence for disentanglement

Why the axes are near-orthogonal by design. As discussed by Mucsányi et al. [3], many datasets have an inevitable entanglement of aleatoric and epistemic uncertainty as the ground-truth signals themselves are correlated. In image classification, for instance, ambiguous images are undersampled in training sets, so they automatically have a high epistemic uncertainty as well. On such data, a correlation between the estimated components is confounded: it may reflect a failure of the decomposition or simply the correlation already present between the ground-truth quantities. We therefore constructed AUB-Wiki with near-orthogonal ground-truth axes to remove this confounder. If the estimated components are correlated on AUB-Wiki, the correlation can only be attributed to the estimator and is a limitation of the decomposition; if they are not, the decomposition reflects the decorrelation of the ground truth. Near-orthogonality is thus the property that makes the disentanglement test conclusive, not a weakness of it.

Is the low correlation in Figure 1 a consequence of the entropy-subtraction identity? It is not. Mucsányi et al. [3] use the same entropy-subtraction identity, with an ensemble over models instead of clarifications, and find strongly correlated components (rank correlation of at least 0.78 across methods). To test this directly on AUB-Wiki, we kept the SD formula, embedder, kernel and entropy unchanged and changed only the groups. Instead of one group per clarification, the groups are now six models (the five open models and Llama-3.1-8B), each with 10 answers sampled for the ambiguous question. As in the standard ensemble decomposition [3, 4], the roles of the two terms swap: the spread inside a model is aleatoric, and the disagreement across models is epistemic. This new setting provides a Spearman correlation of +0.49+0.49 between the aleatoric and the epistemic estimates, which reflects the expected correlation structure between the outputs of the 6 models. The subtraction identity therefore does not by itself produce a low or negative correlation between the two terms.

Beyond marginal correlations: conditional effects (new result). To test the effect of each axis (i.e. coverage and counts) while holding the other fixed, we report partial Spearman correlations of each uncertainty component with one axis, controlling for the other, on the same 768 records as Table 4. The partial correlations differ from the marginal ones by at most 0.07: the aleatoric term remains strongly associated with interpretation count given coverage (+0.58+0.58 to +0.73+0.73) and unrelated to coverage given count (−0.02-0.02 to +0.08+0.08). The epistemic term remains associated with coverage given count (−0.24-0.24 to −0.38-0.38) and only weakly with count given coverage (−0.07-0.07 to −0.15-0.15). This table is also added to Appendix G2 in the revised paper.

Modelρ(alea,count∣cov.)\rho(alea, \text{count} \mid \text{cov.})ρ(alea,cov.∣count)\rho(alea, \text{cov.} \mid \text{count})ρ(epi,count∣cov.)\rho(epi, \text{count} \mid \text{cov.})ρ(epi,cov.∣count)\rho(epi, \text{cov.} \mid \text{count})
Qwen3.5-4B+0.70/+0.71+0.04/+0.07-0.14/-0.15-0.37/-0.29
Qwen3.5-9B+0.70/+0.73+0.04/+0.03-0.08/-0.10-0.35/-0.30
Qwen3.5-27B+0.71/+0.73+0.03/+0.04-0.07/-0.08-0.36/-0.33
Gemma 4 12B+0.59/+0.72+0.05/+0.08-0.14/-0.15-0.38/-0.30
Phi-4 14B+0.58/+0.73-0.02/-0.01-0.10/-0.11-0.33/-0.24
AuthorsOpenReview ↗

Comment 3/5

W3: Method coverage

Following the reviewer's suggestion, we now add two decomposition methods that do not use clarifications and that rely on different decomposition formulas than SD and ICE:

  • Cross-model disagreement [5]: for a reference model, the aleatoric term is one minus the mean cosine similarity between its own sampled answers, and the total uncertainty is one minus the mean similarity between its answers and those of a set of models; the epistemic term is the difference. Each of our five open models serves once as the reference, and the model set comprises the five open models plus Llama-3.1-8B, added because it differs in family, architecture and training data.
  • Ensemble of ensembles [6]: stochastic decoding of a fixed prompt forms the inner ensemble, and meaning-preserving paraphrases of the prompt form the outer ensemble (10 paraphrases × 10 sampled answers per paraphrase, for each open model); the law of total covariance in embedding space yields the aleatoric/epistemic split (this is contemporary work that will appear at NeurIPS 2026 and was published on arXiv just before the rebuttal period). The table below reports the Table 4 quantities on the same 768 ambiguous questions and 418 controls. Neither new baseline method tracks either axis, and their aleatoric terms do not separate ambiguous records from their controls (AUROC 0.41–0.51). Implementation details can be found in Appendix G.3.
MethodAUROCρ(epi,cov.)\rho(epi, \text{cov.})ρ(epi,count)\rho(epi, \text{count})ρ(alea,count)\rho(alea, \text{count})ρ(alea,cov.)\rho(alea, \text{cov.})ρ(epi,alea)\rho(epi, alea)
SD0.88 – 0.90-0.37 – -0.32-0.11 – -0.04+0.58 – +0.71-0.05 – 0.00-0.23 – -0.15
ICE0.88 – 0.89-0.33 – -0.23-0.13 – -0.05+0.71 – +0.73-0.06 – +0.01-0.14 – -0.06
Cross-model0.43 – 0.51-0.01 – +0.060.00 – +0.12-0.20 – +0.10-0.09 – +0.02-0.54 – -0.35
Ensemble of ensembles0.41 – 0.44+0.08 – +0.11-0.02 – +0.12-0.21 – -0.05-0.09 – -0.02-0.33 – -0.14

Furthermore, we already report results for additional methods that provide a single uncertainty estimate, and evaluate SE and P(True), and an ambiguity probe.

Both new baseline methods fail on AUB-wiki as they derive their aleatoric term from the spread of answers sampled for the (ambiguous) question itself. This assumes that ambiguity is expressed as diversity in the sampled answers. Our results in Section 4.3 show that this is not the case: semantic entropy computed on the ambiguous question does not increase with interpretation count and even decreases (Table 2), also when the model answers every clarification correctly (Table 11). Models tend to commit to one interpretation rather than spread their answers across interpretations. This agrees with prior findings that existing uncertainty estimators degrade to near-random performance on ambiguous questions [7], and that approximating model ensembles by varying in-context examples fails to separate ambiguous from unambiguous questions [8]. Paraphrases preserve the ambiguity of the question and therefore do not expose it either.

On whether the results are a property of the method family. We agree with the reviewer that the disentanglement result is specific to clarification-based decompositions, and our new comparison confirms this. On AUB-Wiki, methods that explicitly enumerate the interpretations of a question infer the least entangled uncertainty estimates. We designed AUB-Wiki with explicitly disentangled knowledge and ambiguity axes to uncover such method-specific properties. In the revised manuscript, we now discuss this explicitly in the paper (Appendix G3), report the main caveat of the clarifier-based family and quantify how much of the aleatoric signal is driven by the clarifier (Table 4, last row).

Reviewer_WDAWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces AUB-Wiki, a Wikidata-derived benchmark designed to test whether uncertainty estimates track referential ambiguity separately from factual knowledge difficulty. Each ambiguous question has a rule-defined set of interpretations with clarifications and gold answers. The benchmark uses interpretation count as a proxy for ambiguity and Wikimedia sitelink coverage as a model-independent proxy for knowledge difficulty, and supplies approximately coverage-matched single-interpretation controls. It releases 2,158 ambiguous records and 1,425 controls; costly decomposition experiments use a 768-record subset. The authors validate the proxies across five open-weight models and one frontier model (Opus 5, used only for the Table 1 validation and its per-template counterpart in Table 10), and evaluate Semantic Entropy, P(True), an ambiguity probe, ICE and SD on the five open models. Total-uncertainty estimators show limited sensitivity to the constructed ambiguity axis, while ICE/SD terms associate with their intended axes. The data construction is useful and carefully audited. However, whether the decomposition methods themselves add independent disentanglement information beyond the upstream clarifier remains unresolved, and the proxy-based, template-limited design supports a narrower conclusion than general uncertainty disentanglement.

Critical strengths
  1. AUB-Wiki is built deterministically from Wikidata, preserves provenance, defines explicit membership rules, performs multiple completeness/precision audits, provides matched controls, and releases the generation/evaluation pipeline and datasheet. The released record counts and major construction statistics are internally consistent. This is stronger data engineering than is common for ambiguity benchmarks and makes the artifact potentially useful even if the current scientific interpretation is narrowed.

  2. The paper asks the right evaluation question. The distinction between ambiguity detection and actual uncertainty disentanglement is important. Testing each estimated component against both an intended and an unintended axis is a useful evaluation principle. The paper also reports potentially informative negative results showing that answer-based total uncertainty measures do not straightforwardly track the constructed ambiguity axis.

Critical weaknesses
  1. Clarifier-count confounding prevents attribution of the headline aleatoric result to the decomposition itself. In Table 4, the aleatoric terms of SD/ICE achieve AUROC 0.88–0.90 for ambiguous-versus-control discrimination on the five open models. Yet clarification count alone achieves 0.90 and correlates with the true interpretation count at +0.79, versus +0.58 to +0.73 for the reported aleatoric estimates. All these estimates depend on the same clarifier-generated variants. The paper does not show whether SD/ICE retain an incremental signal conditional on the number of clarifications, with a fixed K, or using the benchmark’s own ground-truth clarifications. A second independent clarifier is also absent. Appendix E.4 further reports that the clarifier proposed at least two interpretations for 95/418 nominal controls; 216/306 proposed entities in those cases cannot be matched to the benchmark group, although unmatched entities should not automatically be called hallucinations. The current evidence supports alignment with an ambiguity proxy but does not establish that the decomposition itself contributes to the observed effect.

  2. Construct validity and the principal statistical comparisons need stronger controls. Interpretation count is the number of admissible readings, not a probability distribution over them; coverage measures how widely an entity is documented rather than directly measuring a model’s epistemic uncertainty about the particular answer. The paper does validate coverage against clarification correctness (AUROC 0.75–0.81 for open models; 0.65 for the frontier model on a smaller subset), and Appendix A.5 tests alternative popularity measures, but neither establishes a ground-truth uncertainty quantity. The Tier-1 evaluation set comprises only eight templates, 1,375/2,158 (63.7%) People records and predominantly temporal/quantitative questions; as the authors acknowledge, this limits generalization to natural questions. Tier 2 retains 10 template families, two with a single record. Moreover, the code for Table 4 pools Spearman correlations across templates despite using within-template correlations for V2. Table 3 uses 215 high-coverage ambiguous records versus 473 controls, although only 149 of those records have a paired high-coverage control; Table 4 uses all 768 ambiguous records versus 418 controls, rather than limiting the ambiguity side to the 418 paired records. These are valid descriptive group AUROCs but are not strictly matched comparisons. The paper should explicitly distinguish these analyses and show that its findings survive template controls and strict paired-sample sensitivity checks.

Questions for authors
  1. Please compare SD/ICE with clarification count alone under a fixed K or after conditioning on n_variants. Could you also report results with gold-standard benchmark clarifications and a second independent clarifier? Does the purported aleatoric signal remain after these controls?

  2. Please report per-template or fixed-effects estimates of Table 4, with uncertainty intervals that account for template structure. Please also rerun Table 3 on the 149 strictly matched high-coverage pairs and Table 4 on the 418 corresponding pairs (including within-pair concordance if appropriate). Are the conclusions robust?

  3. Can you distinguish interpretation cardinality from aleatoric entropy and report how the results change when epistemic uncertainty is assessed using direct model-specific correctness or gold-answer probabilities? In light of Appendix A.5, what conclusions do the authors consider justified without treating sitelinks as epistemic ground truth?

  4. Can you release raw outputs for Tables 1–4 and exact model/software revisions, and clarify how to reproduce results with the saved proprietary-clarifier outputs? Is there an independent human audit of question naturalness, clarification quality, and correctness-matcher error rates?

6 comments
AuthorsOpenReview ↗

Comment 1/5

We thank the reviewer for the detailed feedback, the precise questions and for recognizing the strengths of our work, in particular the careful, deterministic construction of AUB-Wiki, the importance of the evaluation question we pose, and the informativeness of our negative results. We address the requested experiments and analyses below.

Unless otherwise stated, we use alea and epi to refer to the aleatoric and epistemic terms calculated by the decomposition methods. We expect alea to be correlated to count and epi to coverage.

Q1/W1: The Aleatoric term and the clarifier's performance

You are right, much of the ambiguity discriminative signal is driven by the clarifier itself and the number of generated clarifications by Sol alone achieves an AUROC of 0.90 and correlates with interpretation count at +0.79. This is a property of the uncertainty quantification methods themselves, which was revealed by AUB-Wiki, rather than a limitation of our dataset. We agree that it is an interesting finding that warrants further investigation and ran the additional experiments below to further explain this. In addition, we now analyse the performance of two additional baselines for quantifying aleatoric uncertainty, that do not rely on clarifications.

Q1.1: Further clarifiers and gold clarifications (App. G.1). We reran SD and ICE for all five models with three further clarifiers, Claude Opus 5, the open-weight DeepSeek v4.1 Flash and the smaller GPT-5 mini, and with the benchmark's gold clarifications. The table below shows the results. We report the ranges over the 5 models and 2 methods in the table below and put the per-model results in Appendix G.1.

ρ(alea,⋅)\rho(\text{alea}, \cdot)ρ(epi,⋅)\rho(\text{epi}, \cdot)
ClarifierRecallAUROCcountcov.cov.count
GPT-5.6 Sol0.770.88–0.90+0.58 to +0.73-0.06 to +0.01-0.37 to -0.23-0.13 to -0.04
Claude Opus 50.780.88–0.90+0.54 to +0.750.00 to +0.08-0.37 to -0.24-0.14 to -0.08
DeepSeek v4.10.610.83–0.86+0.50 to +0.62+0.03 to +0.13-0.26 to -0.17-0.22 to -0.16
GPT-5 mini0.510.75–0.77+0.41 to +0.53-0.02 to +0.06-0.29 to -0.18-0.19 to -0.10
Gold1.001.00+0.70 to +0.91-0.04 to +0.03-0.42 to -0.29-0.08 to +0.02

For every tested clarifier, each uncertainty component is correlated to its corresponding axis (aleatoric uncertainty to count, epistemic uncertainty to coverage) and opposite correlation coefficients (aleatoric uncertainty to coverage, epistemic uncertainty to count) are small in absolute terms. Using Opus 5 as clarifier instead of Sol (recall 0.78, compared to 0.77 for Sol) yields very similar results compared to Table 4 (Sol as clarifier). The open-weight DeepSeek (recall 0.61) and GPT-5 mini (recall 0.51) recover fewer interpretations, and the aleatoric term is slightly less predictive of ambiguity (AUROC 0.88–0.90 for Sol and Opus 5, 0.83–0.86 for DeepSeek and 0.75–0.77 for GPT-5-mini).

Q1.2: Fixed KK. We next followed the reviewer's suggestion and computed the performance of SD/ICE under a fixed number of inferred clarifications KK. Within-KK correlation between the aleatoric term and the true number of interpretations ranged −0.08-0.08 to +0.17+0.17 and within-KK AUROC for identifying ambiguous questions ranged from 0.470.47 to 0.550.55, i.e. near chance. To quantify what the aleatoric term adds when controlling for the number of clarifications KK, we computed the partial Spearman correlation between aleatoric uncertainty estimated by SD/ICE and the true interpretation count given KK which ranged from −0.11-0.11 to +0.12+0.12. Ranges across the 2 methods and 5 models are reported in the table below and added to Appendix G.1 in the revised version of the paper.

AuthorsOpenReview ↗

Comment 2/5

KK alonealeatoric term at fixed KK
clarifierAUROCρ(K,count)\rho(K, \text{count})ρ\rho(alea, count)partial ρ∣K\rho \mid KAUROC
GPT-5.6 Sol0.90+0.79+0.02 to +0.12-0.03 to +0.090.47–0.54
Claude Opus 50.90+0.83-0.02 to +0.17-0.06 to +0.120.47–0.55
DeepSeek v4.10.85+0.70-0.08 to +0.09-0.11 to +0.010.48–0.53

Taken together, the observation that the quality of the aleatoric uncertainty term of SD and ICE depends strongly on the quality of the clarifier lies in the clarification step itself that is part of SD and ICE, and AUB-Wiki's gold interpretations allowed us to quantify this. These additional experiments are added to the appendix in full (Appendix G.1).


W1/Q1: Clarifier-free decomposition To further investigate the effect of clarifications, we have followed the suggestion of reviewer ToLb and ran two additional baselines that quantify aleatoric and epistemic uncertainty without relying on clarifiers (for details, see response to ToLb). In brief, we used Cross-model disagreement and an Ensemble of ensembles and found that neither new baseline method tracks either axis, and their aleatoric terms do not separate ambiguous records from their controls (AUROC 0.41–0.51; full results in response to ToLb). We also found that disentanglement is specific to clarification-based decompositions (ρ\rho -0.54 – -0.35 for cross-model disagreement and -0.33 – -0.14 for ensemble of ensembles). That is, methods that explicitly enumerate the interpretations of a question infer the least entangled uncertainty estimates. We designed AUB-Wiki with explicitly disentangled knowledge and ambiguity axes to uncover such method-specific properties. In the revised manuscript, we now discuss this explicitly in the paper (Appendix G3), report the main caveat of the clarifier-based family and quantify how much of the aleatoric signal is driven by the clarifier (Appendix G1).

Q2. Template structure and strictly matched comparisons.

Q2.1: Templates. We recomputed Table 4 separately for each of the six templates with at least 30 ambiguous records (750 of the 768). In every template, the aleatoric term separates ambiguous records from their controls and was correlated with the true interpretation count; the epistemic term decreases with coverage, though only weakly for the smallest template (album, 56 records). Appendix G.2 gives the per-template values for Qwen3.5-4B as an example. We also computed each quantity within templates and averaged it across templates, weighted by each template's number of records, as for validity check V2V_2 in the paper. The table below reports the ranges over the 5 models and the 2 methods.

Overall, within templates, the epistemic–coverage correlation becomes stronger, the aleatoric–count correlation is similar, and ρ(epi,count)\rho(\text{epi}, \text{count}) and ρ(alea,coverage)\rho(\text{alea}, \text{coverage}) stay close to zero as shown in the table below. The conclusions of Table 4 therefore do not depend on the template mix.

Table 4 pooled over all records and within templates (weighted by record count). Ranges over 5 models × 2 methods.

EstimateAUROC (alea)ρ(epi,cov.)\rho(\text{epi}, \text{cov.})ρ(epi,count)\rho(\text{epi}, \text{count})ρ(alea,count)\rho(\text{alea}, \text{count})ρ(alea,cov.)\rho(\text{alea}, \text{cov.})
Pooled (Table 4)0.88–0.90-0.37 to -0.23-0.13 to -0.04+0.58 to +0.73-0.06 to +0.01
Within templates (weighted)0.89–0.91-0.45 to -0.38-0.08 to +0.01+0.64 to +0.73-0.01 to +0.07
AuthorsOpenReview ↗

Comment 3/5

Q2.2: Table 3 on the 149 strict pairs. We have re-generated Table 3 using only the 149 strict pairs as the reviewer suggested. SE, 1−P(True)1 - P(\text{True}) and the overall probe change AUROC by at most 0.04 compared to Table 3. SE and 1−P(True)1 - P(\text{True}) behave as in Table 3 (0.37–0.53 and 0.44–0.60), while the probe still separates the pairs (0.60–0.70), more strongly for records with three or more interpretations.

answer-basedambiguity probe
modelSE1−P(True)1 - P(\text{True})all2≥3
Qwen3.5-4B0.470.490.660.650.72
[0.42, 0.53][0.44, 0.54][0.61, 0.72][0.58, 0.71][0.60, 0.82]
Qwen3.5-9B0.370.440.600.600.63
[0.31, 0.44][0.38, 0.50][0.55, 0.64][0.54, 0.65][0.51, 0.74]
Qwen3.5-27B0.450.600.700.650.84
[0.39, 0.52][0.54, 0.66][0.65, 0.75][0.59, 0.72][0.76, 0.92]
Gemma 4 12B0.460.500.650.600.82
[0.39, 0.52][0.44, 0.56][0.60, 0.70][0.55, 0.66][0.74, 0.90]
Phi-4 14B0.530.530.620.610.67
[0.46, 0.59][0.46, 0.58][0.56, 0.67][0.54, 0.68][0.57, 0.78]

Q2.3: Table 4 on the 418 pairs. We also re-generated Table 4 as suggested and compared each record against its own control. As the results in the table below show, every quantity changes by at most 0.07 from the old Table 4, and the aleatoric AUROC still matches the clarification count alone (0.89), as before.

ρ\rho(epistemic, ⋅\cdot)ρ\rho(aleatoric, ⋅\cdot)
modelAUROCcoveragecountcountcoverage
SD / ICESD / ICESD / ICESD / ICESD / ICE
Qwen3.5-4B0.88 / 0.88-0.34 / -0.28-0.13 / -0.14+0.67 / +0.68+0.03 / +0.05
Qwen3.5-9B0.89 / 0.88-0.34 / -0.31-0.07 / -0.09+0.67 / +0.70+0.01 / +0.01
Qwen3.5-27B0.89 / 0.88-0.38 / -0.35-0.07 / -0.09+0.68 / +0.70+0.02 / +0.02
Gemma 4 12B0.88 / 0.88-0.35 / -0.31-0.12 / -0.13+0.58 / +0.68+0.07 / +0.06
Phi-4 14B0.87 / 0.88-0.30 / -0.25-0.09 / -0.12+0.57 / +0.70-0.01 / -0.02
KK alone0.89——+0.76—

These matched-control results for table 3 and 4 are also now included in Appendix G.2 in the revised paper.

Reviewer_fHQUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a benchmark dataset, named AUB-Wiki, for uncertainty decomposition. Unlike the previous datasets which heavily rely on the human annotation (subjective), AUB-Wiki strictly quantifies the ambiguity and coverage using Wikidata and Wikimedia sitelinks. This benchmark supports a more comprehensive test and evaluation of LLMs uncertainty quantification and decomposition. The experimental results of 6 LLMs on AUB-Wiki show that commonly used total uncertainty measures do not consistently reflect ambiguity, whereas uncertainty decomposition methods ICE and SD exhibit the intended separation.

Critical strengths
  1. AUB-Wiki uses Wikidata and Wikimedia sitelinks to strictly control the ambiguity. The dataset provide a valuable benchmark to evaluation the effectiveness of uncertainty quantification methods and the correctness of uncertainty decomposition methods.

  2. This benchmark dataset has serval advantages: LLM-free with reproducible interpretation sets and a scalable construction pipeline. The ambiguity is objective rather than subjective.

Critical weaknesses
  1. The dataset is heavily focused on QA task. And the evaluation in this work does not compare the uncertainty with/without context (QA vs contextual QA).

  2. The "coverage" is defined by how widely an entity is documented via Wikimedia sitelinks. This might not reflect the model's actual knowledge of a specific fact, since LLMs use other source of training data.

Questions for authors
  1. How well do the findings using this benchmark generalize to other type of queries? Basically queries does not follow the format of the ambiguous question.

  2. Does this benchmark ineffective against more capable models? Since it only test the knowledge (memory) within Wikipedia.

4 comments
AuthorsOpenReview ↗

Comment 1/3

We thank the reviewer for the detailed feedback and for recognizing the strengths of our work, in particular that AUB-Wiki replaces subjective human annotation with an objective, Wikidata-derived definition of ambiguity, and that it provides a valuable benchmark for evaluating both uncertainty quantification and uncertainty decomposition methods. We also appreciate the reviewer highlighting the practical advantages of our design: the construction pipeline is LLM-free, its interpretation sets are reproducible from an explicit membership rule and a fixed Wikidata version, and it scales to new templates. We address the reviewer’s concerns point by point below.

W1/Q1: Contextual QA and queries beyond the QA format

Because AUB-Wiki is built directly on Wikidata, it extends naturally to contextual QA. We can add the corresponding articles to ambiguous and non-ambiguous questions as context in the dataset. However, using context for evaluation makes the clarification task and generating the correct answer much easier, even for small models. A side effect is that the model's internal knowledge about the question (and also the coverage proxy, i.e. sitelinks) don't matter anymore; the model confidently and consistently provides the correct answer to the question if the context containing the answer is provided. This is also reflected in the epistemic uncertainty estimates which become very low, and diverge from the model's internal parametric knowledge on the question and from the coverage proxy. For example, on Qwen3.5-4B, the mean epistemic uncertainty drops from 0.82 to 0.13 on SD after adding context, and from 1.18 to 0.19 on ICE. As a result, the negative Spearman correlation with the coverage proxy −ρ(EU,coverage)-\rho(\text{EU},\text{coverage}) drops from 0.36 to 0.05 for SD and from 0.28 to 0.03 for ICE. On the other hand, the ambiguity detection AUROC increases from 0.89 to 0.95 for SD and from 0.89 to 0.94 for ICE.

Queries beyond question answering, such as summarization, instruction following, or multi-step reasoning, are outside the scope of AUB-Wiki. Our goal is a controlled setting in which the ground-truth interpretation set and a model-independent knowledge reference are both exactly defined, which requires questions whose answers can be verified against a knowledge graph. We state this scope more explicitly in the limitations in the revised paper.

W2/Q2: Coverage as a knowledge proxy, and more capable models

We agree that sitelink coverage measures how widely an entity is documented, and not directly a model's knowledge of a specific fact. We address the two parts of the concern separately: why coverage is a meaningful proxy despite LLMs being trained on more than Wikipedia, and how the benchmark behaves for more capable models. We also report a new orthogonal analysis that is independent of coverage by quantifying the correlation of epistemic uncertainty to record-level accuracy.

Coverage tracks documentation on the web, not only in Wikipedia. The facts in AUB-Wiki (release years, durations, birth years, publication years) are not specific to Wikipedia; they are widely replicated across the web (filmographies, discographies, biographical sites, news). Coverage is therefore best interpreted as a proxy measure of how densely an entity is documented in general, with Wikipedia as a measurable reference point. Wikipedia is also itself a core component of documented pretraining corpora, often upsampled: it is weighted at 3% and seen 3.4 times in GPT-3 [1], sampled at 4.5% for about two epochs in LLaMA [2], included with 3 epochs in the Pile [3], and part of Dolma [4]. Studies of big pre-training corpora [5, 6] also confirm that Wikipedia is overwhelmingly present in these datasets.

AuthorsOpenReview ↗

Comment 2/3

Why a model-independent proxy is needed. In classification, epistemic uncertainty is usually evaluated through in- vs. out-of-distribution labels [7, 8]. For LLMs, such labels are not available: the content of web-scale pretraining corpora is only partially understood [5], and even with full access to the training data, membership inference barely outperforms random guessing, partly because the boundary between members and non-members is inherently fuzzy [9]. Moreover, a central goal of AUB-Wiki is to evaluate disentanglement. [8] notes that on existing datasets part of the correlation between aleatoric and epistemic estimates is inherited from correlated ground-truth signals, so a failure of the decomposition cannot be separated from structure in the data. AUB-Wiki avoids this by stratifying records over interpretation count and coverage, which keeps the two sources of uncertainty (knowledge and ambiguity) nearly decorrelated (ρ=−0.12\rho = -0.12, Appendix B). Such decorrelation can only be guaranteed at construction time with a model-independent knowledge signal. Since epistemic uncertainty is by definition model-dependent, any model-independent proxy involves a trade-off; we resolve it by using a proxy that then is validated against each evaluated model (by computing correlation between epistemic uncertainty and the model-independent proxy).

More capable models. As mentioned by the reviewer, coverage is less discriminative for the strongest models because they answer more low-coverage facts correctly. However, the benchmark does not become ineffective. The table below shows that (i) Accuracy still rises monotonically with coverage for Opus 5, from 0.49 in the lowest bin to 0.94 in the highest, so the spread is narrower but the global trend is clearly present. (ii) Opus 5 still gives a wrong answer to almost 51% of clarifications in the lowest-coverage stratum, so the benchmark is far from saturated. (iii) The ambiguity axis becomes more informative with capability: the enumeration correlation reaches +0.85+0.85 for Opus 5 versus +0.43+0.43 to +0.60+0.60 for the open models. (iv) Since AUB-Wiki is designed to compare uncertainty methods on the same model, a weaker coverage signal for one model affects all compared uncertainty quantification methods equally. It only limits comparing a single estimator across model sizes using coverage alone. To investigate this further, we tested two additional frontier models, GPT-5.6 Sol and DeepSeek v4.1 Flash, on our benchmark to assess their behavior along the coverage axis. The results in the table below (computed on the 768-record subset) show the same overall trend as Opus 5 (points (i) to (iii)), concluding the benchmark is still effective for stronger models. We would also like to emphasize that we foresee the main utility of AUB-Wiki not for evaluating the latest frontier models, but for evaluating the quality of uncertainty estimates of AI systems deployed in safety-relevant settings, which often rely on smaller, locally deployed open-weights models.

Accuracy by coverage (sitelinks)Coverage
Model<1010–1920–2930–3940–4950–59≥60AUROCρ(count,identified)\rho(\text{count},\text{identified})
GPT-5.6 Sol0.670.750.790.860.880.870.950.63+0.84
DeepSeek v4.1 Flash0.550.650.700.790.850.830.930.64+0.77
Opus 50.490.610.670.770.780.800.940.65+0.85
Clarifications3471,059855506256142193
AuthorsOpenReview ↗

Comment 3/3

A model-dependent knowledge reference, and a new result. For evaluations focused on epistemic uncertainty itself rather than on disentanglement, we now present a coverage-independent analysis of the quality of epistemic uncertainty via record accuracy by computing the correctness of the target model's answers to the ground-truth clarifications, 1∣Ir∣∑i∈IrCM(qr,i)\frac{1}{|\mathcal{I}_r|}\sum_{i \in \mathcal{I}_r} C_M(q_{r,i}). Since individual clarifications are not ambiguous, this fraction of correctly answered questions is a direct measure for the epistemic uncertainty of the evaluated model, including frontier models. We now evaluate the epistemic terms against this record accuracy measure, as an analysis that is not based on site-links (new results below, added to Section 4.3). The epistemic term correlates much more strongly with the model's own record accuracy (ρ=−0.40\rho = -0.40 to −0.68-0.68) than with coverage (−0.23-0.23 to −0.37-0.37), while remaining nearly unrelated to interpretation count (−0.04-0.04 to −0.13-0.13). AUB-Wiki thus supports two complementary evaluations: a model-independent one (coverage), which enables disentanglement analysis with decorrelated references, and a model-dependent one (record accuracy), which measures epistemic quality via standard correctness analysis for a specific model, including frontier models. We also verify that this model-dependent record accuracy is not saturated for frontier models, as its value ranges from 0 to 1 for Opus 5 and is not concentrated around 1.

epistemic (within clar.)
modelρ\rho(count)ρ\rho(coverage)ρ\rho(accuracy)
SD / ICESD / ICESD / ICE
Qwen3.5-4B-0.10 / -0.12-0.36 / -0.28-0.60 / -0.59
Qwen3.5-9B-0.05 / -0.07-0.34 / -0.29-0.66 / -0.67
Qwen3.5-27B-0.04 / -0.05-0.36 / -0.33-0.67 / -0.68
Gemma 4 12B-0.11 / -0.13-0.37 / -0.29-0.43 / -0.40
Phi-4 14B-0.07 / -0.09-0.32 / -0.23-0.57 / -0.66

References

  1. T. Brown et al. Language Models are Few-Shot Learners. NeurIPS, 2020.
  2. H. Touvron et al. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971, 2023.
  3. L. Gao et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027, 2020.
  4. L. Soldaini et al. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. ACL, 2024.
  5. Y. Elazar et al. What's in My Big Data? ICLR, 2024.
  6. Y. Liu et al. Datasets for Large Language Models: A Comprehensive Survey. Artificial Intelligence Review, 58(12):403, 2025.
  7. J. Mukhoti et al. Deep Deterministic Uncertainty: A New Simple Baseline. CVPR, 2023.
  8. B. Mucsányi, M. Kirchhof, and S. J. Oh. Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks. NeurIPS, 2024.
  9. M. Duan et al. Do Membership Inference Attacks Work on Large Language Models? COLM, 2024.

Dear Reviewer,

Thank you again for your time and effort. As the end of the rebuttal phase is approaching, we would appreciate hearing whether our responses and the additions to the paper address your concerns. If so, we would be grateful if you would consider updating your assessment. We are happy to answer any further questions.

Sincerely,

 The Authors

Comments on the paper

1 comment

We sincerely thank all reviewers for their time and for their detailed and constructive reviews. We are encouraged that the reviewers found AUB-Wiki a valuable benchmark for evaluating uncertainty quantification and decomposition methods (fHQU, WDAW), and the first where both uncertainty components can be checked against independent references on the same questions (ToLb). They appreciated its deterministic, LLM-free construction from Wikidata, which makes ambiguity objective rather than subjective (fHQU, WDAW, ToLb), its reproducible and scalable released pipeline (fHQU, WDAW), and its matched unambiguous controls (WDAW, ToLb). They described the data construction as carefully audited and stronger than is common for ambiguity benchmarks (WDAW). They also noted that the paper asks the right evaluation question by separating ambiguity detection from true disentanglement (WDAW), found the motivation well argued (ToLb), and considered our results on total-uncertainty measures informative (WDAW).

The reviewers’ comments helped us sharpen the paper, in particular by quantifying how much of the aleatoric signal is driven by the clarifier, adding a sitelink-independent evaluation of epistemic uncertainty against correctness, and further evaluating two very recent baselines on our benchmark. We uploaded a revised PDF with all changes marked in blue, and summarize the main revisions below:

  • Aleatoric term dependency on the clarifier (Reviewers WDAW, ToLb) We reran SD and ICE with three further clarifiers and with the gold clarifications, and added an analysis at a fixed number of clarifications (Appendix G.1). The results show that the performance of these decomposition methods is mainly driven by the quality of the clarifications.

  • Coverage-independent evaluation of epistemic uncertainty (Reviewers fHQU, WDAW, ToLb). Alongside the sitelink coverage proxy, we now evaluate the epistemic term against a model’s correctness on the gold clarifications (section 4.3 of the revised paper). The experiments show a stronger association between epistemic uncertainty and this model-dependent reference, which makes it more suitable for studies focusing on epistemic uncertainty alone instead of disentanglement.

  • Additional frontier models (Reviewers fHQU, ToLb). We evaluated two further frontier models along the coverage axis. Their accuracy still increases with coverage and between 33% and 51% of low-coverage questions are answered incorrectly by frontier models, showing that the benchmark is not saturated (Appendix G4).

  • Clarifier-free baselines (Reviewers ToLb, WDAW, Appendix G.3). We benchmarked two additional recent decomposition methods that do not rely on clarifications: cross-model disagreement and ensemble of ensembles (Appendix G.3). Our findings confirm the results of Table 2, highlighting that detecting ambiguity requires explicitly prompting the model to consider the possible interpretations of a question, and cannot be achieved through answer consistency across paraphrases or across models alone. A novel finding of this analysis is that this limitation also affects the epistemic term and the disentanglement of these decompositions.

  • Additional evidence for disentanglement (Reviewers ToLb, WDAW). We added partial correlations that control for the other axis, analyses per template and within templates, and comparisons restricted to strictly matched pairs. The conclusions of the main tables hold in all of these settings (Appendix G.2).

  • Reproducibility (Reviewer WDAW). We now release all raw model outputs, the exact model revisions and the saved clarifier outputs. With these, all main tables can be recomputed on CPU without calling any proprietary model (supplementary material).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.