acceptodds
Under review as a conference paper at ICLR 2027

Epistemic Uncertainty in Large Language Models: Measurement, Evaluation, Mechanisms, and Applications

Abstract

Token-level uncertainty in large language models (LLMs) arises from both knowledge gaps (epistemic uncertainty, EU) and the inherent flexibility of language (aleatoric uncertainty, AU). Accurate token-level EU estimation is fundamental to understanding model reliability at the basic decision unit of generation. However, the lack of a unified and direct testbed leaves estimator effectiveness, EU signal mechanisms, and downstream applicability insufficiently understood. To advance the understanding of uncertainty in LLMs, we make four contributions: 1. We introduce EU and AU proxy labels based on reasonable continuation sets and develop an automated pipeline to construct UncertainBench, comprising 7,216 labeled continuation positions across 15 domains. 2. We systematically evaluate EU across 67 LLMs and benchmark 35 uncertainty estimation methods, revealing that none combines accurate EU estimation with high EU purity. 3. We discover stronger and purer EU signals in the low-probability tail and propose a training-based hypothesis consistent with observed logit patterns. Emphasizing these signals yields a training-based estimator that outperforms baselines and generalizes across domains. 4. We apply EU estimators to five downstream tasks, helping distinguish reliable knowledge from lucky success, assess response quality, predict model capability, and identify critical branching positions. We also find that EU estimators offer limited value for predicting pre-trained model potential, whereas our proposed R\'enyi entropy-based method performs well by emphasizing tokens in head probability buckets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Epistemic Uncertainty, Large Language Model, Uncertainty Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dKnvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper constructs UncertainBench, a token‑level benchmark to separate epistemic(EU) and aleatoric(AU) uncertainty. It evaluates 67 LLMs and 35 uncertainty estimators, discovers strong EU signals in low‑probability token tails, proposes a trained estimator M_{train}, and tests EU estimators on five downstream tasks with mixed results.

Critical strengths
  1. Builds the fine‑grained token‑level UncertainBench benchmark with automatic annotation pipeline and decent quality, filling the gap of direct EU evaluation.

  2. Proposes EU‑purity metric, large‑scale benchmark of existing uncertainty methods; finds tail tokens carry purer epistemic‑uncertainty signals.

  3. Proposed M_{train} outperforms baselines and generalizes across held‑out domains; fairly discusses inconsistent downstream performance.

Critical weaknesses
  1. EU/AU are proxy labels (not ground‑truth oracle values); limitations are buried in appendices and deserve more main‑text emphasis.

  2. The tail‑suppression mechanistic hypothesis is only correlational, lacking causal training‑based verification.

  3. No experiments on zero‑shot transfer of M_{train} to completely unseen LLMs of new architectures.

  4. Benchmark annotations heavily rely on annotator LLMs; human evaluation sample size is small. Downstream task absolute performance remains moderate.

Questions for authors
  1. What controlled training experiment can verify your tail‑suppression hypothesis?

  2. Why do EU estimators fail on reasoning‑pivot branching positions? Is this a fundamental limitation?

  3. How do you mitigate annotation bias brought by annotator LLMs for niche domain inputs?

Reviewer_7f7VOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies token-level epistemic uncertainty in LLMs. It introduces EU/AU proxy labels based on reasonable continuation sets and constructs UncertainBench to evaluate 67 LLMs and 35 uncertainty estimators. The authors analyze uncertainty signals across token probability ranks, propose a learned EU estimator using distributional features, and connect these observations through a Rényi entropy perspective. The methods are evaluated on the proposed benchmark and several downstream tasks.

Critical strengths
  • Comprehensive empirical study and benchmark construction. The large-scale evaluation across models, estimators, and domains provides a useful empirical reference. The automated benchmark construction pipeline is also a meaningful engineering contribution.

  • Interesting probability-distribution analysis. The analysis of uncertainty signals across probability ranks and their connection through Rényi entropy provides a useful perspective on existing estimators. The findings regarding tail probabilities and the effectiveness of head-focused Rényi entropy are particularly interesting. MaxProb, Shannon entropy, and Self-Certainty are shown to be special or limiting cases of Rényi entropy at different orders α. Changing α shifts the weight between head and tail tokens. This explains the ordering of the baselines in one simple framework and gives a simple estimator for the potential-prediction task.

Critical weaknesses
  • Questionable EU/AU proxies and limited novelty. The proposed EU proxy measures the probability of unreasonable continuations which may not be a reliable applixation of AU/EU, where the AU proxy relies solely on the number of reasonable tokens. Yet these formulations are closely related to existing ideas and provide rather coarse approximations that may not preserve the underlying uncertainty decomposition. This raises concerns about the validity of the benchmark for evaluating EU estimators, especially Bayesian methods.
    • EU-hat (probability of an unreasonable token) is the next-step hallucination probability of [1].
    • AU-hat (size of the reasonable set) is similar to lexical-substitution annotations [3].
    • Human cloze data ([2]) gives frequencies over next words. The uniform set used here throws away all relative probabilities within R(c).
    • log|R| also counts tokenizer fragments, e.g., h / hy / hyd / hydr / hydration count as five tokens.
    • The pipeline (candidate generation, LLM voting, adjudication) is a standard LLM-as-judge annotation process.

[1] Xiao & Wang. On Hallucination and Predictive Uncertainty in Conditional Language Generation. EACL 2021.

[2] Evgenia Ilia and Wilker Aziz. Predict the Next Word: Humans exhibit uncertainty in this task and language models _____. EACL2024

[3] McCarthy & Navigli. SemEval-2007 Task 10: English Lexical Substitution Task. SemEval 2007.

  • Insufficient validation of the relationship between probability signals and knowledge gaps. The proposed estimator relies entirely on single-model token probability distributions, which generally cannot uniquely distinguish EU from AU (it mixes up AU and EU). The observed correlations with the proposed proxy labels do not establish that these signals reflect genuine knowledge gaps.
    • A cross-model disagreement baseline is missing, although 67 aligned LLMs are available.
    • "Knowledge gap" is assumed by definition and never tested. At the model level, EU-hat behaves like PPL (|ρ| with the AA index: 0.698 vs. 0.682).
    • In Table6, gpt-oss-120b has 56.3% EU-hat on English Wikipedia, worse than Qwen3-0.6B (41.1%). This shows that gpt-120b has larger knowledge gap than Qwen-0.6B, which dramatically shows that the proposed EU signal is weak to express "Knowledge Gap"
    • The tail finding may be partly built into the label. R(c) has about 50 tokens, so almost all tail mass is counted in EU-hat, while AU-hat does not involve the tail. The reinforcement hypothesis is not tested

This limits the interpretation of the claimed EU estimation improvements.

Questions for authors
  1. Is there direct evidence that EU-hat reflects knowledge gaps? For example, does EU-hat drop, with AU-hat unchanged, when the needed fact is added to the context? Or does it track pretraining frequency in models with open training data?

  2. How does disagreement across the aligned LLMs (e.g., ensemble mutual information QU) compare with M_train on |ρ_EU| and purity?

  3. Can the authors directly validate the proposed connection between tail probability signals and knowledge gaps, for example through controlled knowledge acquisition or forgetting experiments?

  4. see weakness

Reviewer_YNgQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This is an article-like PDF (I hesitate to call it a paper) that tries to claim that they address an important problem in the field of uncertainty quantification in LLMs. The submission has many major flaws, including incorrect citations, incorrect statements, and weird and unjustified proposals.

Critical strengths

This PDF has no strength to be mentioned. No scientific value can be found in this PDF.

Critical weaknesses

Almost everything about this submission supports the rejection. Just to mention a few:

  • The format of the submission looks more like a poorly written technical report rather than a scientific work.
  • Before proposing any benchmarking or any method to improve, aleatoric and epistemic uncertainty require a specific definition in the context of LLMs, especially at the token level. This has never been discussed or mentioned.
  • 54-55: EU is mentioned in those survey papers but not studied in that sense.
  • 59-62: wrong statement. None of the mentioned related work actually does token-level evaluation.
  • 70-72: Hou et al. do not assume such a thing. This reference was used wrongly multiple times.
  • 73: reasonable continuation set R(c), what is reasonable? This has never been explained. There is no clear explanation behind this, and accordingly this makes the entire approach unjustified. This "reasonable" set also connects to the oracle distribution, which by itself is ill-defined (see my further comments).
  • Why is EU purity defined like that? Why does it make sense? There is no explanation for that.
  • 122: denote the distribution of an ideal oracle with complete knowledge relevant to c ==> it is not entirely clear what this oracle is. Considering the case where this prefix c is already a wrong statement, e.g., "Berlin, as a capital of France, is a nice city that", what is the true oracle next-token distribution here? Note that this oracle (unknown) distribution totally makes sense in the context of the reference paper Lahlou et al.
  • In the paragraph "How to select challenging, discriminative continuation positions":
    • What is the seed text? Is it c?
    • what are source models? How are they chosen? why?
    • What are the three representative methods?
    • What are position judges?
    • Why do you need all these complications?
  • In the paragraph "How to improve the coverage of R(c)":
    • Why do you need to improve the coverage of R(c)?
    • What are expansion models? Why are they needed?
  • In the paragraph "How to improve the precision of R(c)":
    • Why do you need to improve the precision of R(c)?
    • What are validation models? Why are they needed?
    • What is the adjudication model?
    • Again, why all these complications?
Questions for authors

I have no questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.