AUB-Wiki: Benchmarking Uncertainty Decomposition in Language Models under Controlled Ambiguity

Uncertainty in Language Model (LM) answers is commonly decomposed into aleatoric uncertainty, arising from ambiguity in the question, and epistemic uncertainty, arising from missing model knowledge. Evaluating such decompositions requires a benchmark on which the two sources can be measured separately. Existing ambiguous-question datasets do not allow this: at best, they provide ground truth for ambiguity, and they lack an independent measure of knowledge difficulty for each clarification, so a method's aleatoric and epistemic estimates cannot be checked against separate references. We introduce Ambiguity and Uncertainty Benchmarking Wiki (AUB-Wiki), a benchmark of referentially ambiguous questions generated deterministically from Wikidata without any language model in the loop. Each question is paired with the complete set of clarifications implied by the knowledge graph, and each clarification has a verified answer and a model-independent coverage score. Most questions also have a matched unambiguous control. The number of clarifications and their coverage form two near-independent axes of ambiguity and knowledge difficulty. We use AUB-Wiki to benchmark uncertainty quantification and uncertainty decomposition methods across several open-weight language models, evaluating whether they attribute uncertainty to ambiguity or to incomplete knowledge. The benchmark and generation pipeline are released to support controlled evaluation of uncertainty decomposition.