- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies uncertainty quantification for RAG-style use cases, where LLM generations should be faithful to the provided context. They propose two sources of confidence hallucinations: context-lure and prior-dominated errors. They suggest a method called Grounding Profile Contrastive Decoding, which does not require labels or model fine-tuning.
Critical strengths
1) Important problem: The paper studies an important problem. The research question (when does entropy provide meaningful signal for RAG vs. when does it produce confident hallucinations) is meaningful and of high consequence.
2) High-quality figures: The graphics in Figures 1 and 2 are well-made.
Critical weaknesses
1) Quality and clarity of writing: I found this paper incredibly difficult to read. Human-readable research should avoid "mannered prose", including coined jargon that is not explained/defined, forward references to not-yet defined terms, and flashy/handwaivy statements that try to grab reader attention at the expense of clarity and technical precision. Unfortunately, this paper is full of such mannered prose, which may be easy for an LLM to read, but very difficult for a human to read. Just the introduction took me a very long time to read. The purpose of research is to communicate scientitic ideas and concepts to humans, and this paper falls far short of that requirement. A non-exhuastive list of examples:
- "Most label-free remedies treat hallucination as uncertainty" -> imprecise, these papers do not claim this. They use uncertainty signals as a predictor of hallucinations, but do not treat them as the same.
- "These methods fail by construction on confident hallucinations." "such methods cannot see confident hallucinations" -> this is a tautology: confidence scores fail when they predict high confidence on incorrect answers. That is like saying: classifiers fail when they are wrong
- "Detectors flag...test-time methods..." -> the "detector" methods cited are also test-time methods. This distinction is unclear and imprecise
- "take the log-ratio of the two probabilities" -> what are the two probabilities? Unclear
- "On 2WikiMultihopQA it reverses, and on MuSiQue the two groups coincide" -> what is meant by "reverses"? what is meant by "the two groups coincide" Unclear
- "We show that the reversal" -> what "reversal" is this talking about?
- "Entropy bound the full-context term of g (proposition 1)" -> this is a forward reference to a proposition that is not properly explained and cannot be understood in the context of the introduction
- "it scores confident answers with fixed directions separately within the two context-support strata" -> what are "fixed directions"? "context-support strata" is coined jargon that is unexplained
This is just from the introduction. I will not do this for every section of the paper, but it is a persistent problem throughout.
2) Weak experiments:
- Section 3 uses only one model in the experiments. This is far from sufficient to establish any meaningful empirical conclusions
- There are zero baselines from other studies in the detection evaluation in Table 3. Standard UQ baselines can be implemented off-the-shelf in commonly used UQ toolkits for this purpose such as LM-Polygraph and UQLM.
3) Reproducibility:
- The experiment setup description is minimal, including how F1 and EM are calculated in the context/for these datasets.
- No reproducibility materials submitted, despite claiming in the reproducibility packet that they were submitted, worsens the transparency issues
4) Missing related work: The paper is missing important related work for RAG-specific UQ:
- Why Uncertainty Estimation Methods Fall Short in RAG: An Axiomatic Analysis (ACL Findings 2025): https://aclanthology.org/2025.findings-acl.852/
- Sufficient Context: A New Lens on Retrieval Augmented Generation Systems (ICLR 2025): https://openreview.net/forum?id=Jjr2Odj8DJ
- Synchronous Faithfulness Monitoring for Trustworthy Retrieval-Augmented Generation (EMNLP 2024): https://aclanthology.org/2024.emnlp-main.527/
- Uncertainty Quantification in Retrieval Augmented Question Answering (TMLR 2025): https://openreview.net/forum?id=JLkgI0h7wy
- Uncertainty Quantification for Retrieval-Augmented Reasoning (SIRIG 2026): https://dl.acm.org/doi/10.1145/3805712.3809717
- Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval-Augmented Generation: https://aclanthology.org/2026.findings-acl.338/
Questions for authors
- Can you improve the writing so it can be understood by human researchers?
- Why did you not include any baselines from other studies in the detection evaluation?
- Why did you not submit a reproducibility packet given the minimal explanation of how the experiments were conducted in the text?