acceptodds
Under review as a conference paper at ICLR 2027

Read the Evidence Before Seeing the Answer: Hypothesis-Independent Evidence Reading for Detecting Confident Hallucinations

Abstract

Evidence-based fact verification normally asks a leading question: the candidate answer is placed in the verifier's context and the model is asked whether the evidence supports it. We study the confident, self-consistent errors on which such verification is most needed and least trusted. Auditing 9,762 of them across eight benchmarks shows a benchmark-dependent mixture of genuine hallucinations, ambiguous questions and wrong gold labels; on the genuine part the best internal signal we tried reaches 0.69 AUROC, versus 0.90 or more once the same model reads a passage, so evidence is needed. Candidate-conditioned verification has a measurable cost: holding question, evidence and verifier fixed and varying only whether the candidate is the verifier's own output, a model judges its own wrong answer more leniently than another model's equally wrong answer (19,718 matched pairs over five benchmarks, ), increasingly so with its confidence, and not at all when neither candidate is its own. We therefore make the evidence-interpretation stage hypothesis-independent: a reader answers the question from the evidence before seeing the candidate, , and an external matcher compares with the candidate, while the calibrated candidate-conditioned score is retained as a base term. This improves detection on every benchmark tested ( AUROC with benchmarks as units, ), by a similar margin whether or not the verifier produced the answer: what helps is reading the evidence first, not the removal of the ownership bias that motivated it. Routing only the questions that internal signals cannot separate matches the full-retrieval residual error rate at 39% retrieval coverage.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Hallucination Detection, LLM Verification, Self-Verification, Evidence-Based Verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SDmJOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
Critical strengths
Critical weaknesses

The paper is very hard to read. The text reads as largely LLM generated. Almost every sentence is overloaded with numbers, p-values and pointers to the appendix, starting from the abstract. Key terms are used without being introduced.

From my best reading, the story is also inconsistent. The paper starts as an audit of confident errors, then turns into a study of ownership bias, then proposes a verification method, and then adds retrieval routing and a page check on top. These parts are loosely connected.

In its current form I cannot assess the correctness or the significance of the results. I believe the paper needs a full rewrite around one clear story before it can be properly evaluated.

Questions for authors
Reviewer_tCKPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates confident hallucinations and self-anchoring bias in LLM-based fact verification. The authors conduct a large-scale analysis of confident errors and show that LLMs tend to favor their own incorrect answers during verification. They propose BEV, which reads evidence without seeing the candidate answer and then compares the extracted answer with the candidate. Experiments across multiple models and benchmarks show consistent improvements.

Critical strengths
  1. The paper focuses on an important problem in hallucination detection.
  2. The analysis of confident hallucinations and benchmark ambiguity provides some insights.
  3. The experiments cover multiple models and benchmarks, with extensive ablation studies and robustness checks.
Critical weaknesses
  1. The proposed method mainly combines evidence-based answer extraction, string matching, and conventional verification scores. The technical contribution is relatively incremental.
  2. BEV also improves third-party verification, where the reported self-anchoring effect is absent. It is unclear whether the improvements come from reducing anchoring bias or simply combining complementary verification signals.
  3. The average AUROC improvement is only around 0.016, with particularly small gains on several benchmarks.
  4. Since BEV requires an additional inference pass, comparisons with other two-pass verification methods and different score fusion strategies would help better demonstrate its effectiveness.
Questions for authors

What does BEV actually stand for? The acronym should defined when it first appears in the paper.

1 comment

Thank you for the detailed response and additional experiments. The new comparisons with other two-pass verification methods and different fusion strategies partially address my concerns. However, I still have concerns about the limited methodological novelty and the explanation of the performance gains. In particular, the benefit of blind answering over candidate-visible answering is not consistent across different evidence settings, and the statistical significance of their direct comparison remains unclear. Therefore, I will maintain my original score.

Reviewer_Up31OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies confident, self-consistent hallucinations in large language models. It audits such errors across eight benchmarks, distinguishing genuine hallucinations from question ambiguity and incorrect reference labels, and finds that models tend to judge their own incorrect answers more leniently. The authors propose BEV, which independently answers a question from external evidence without seeing the candidate answer, then compares the two answers and combines this signal with conventional verification scores. Experiments show modest improvements in hallucination detection and suggest that selective retrieval can reduce verification costs.

Critical strengths

The paper investigates an important phenomenon in LLM hallucination detection: confident, self-consistent errors that are difficult to identify using internal uncertainty signals. It provides a thorough empirical analysis across multiple models and benchmarks, including an audit of genuine hallucinations versus evaluation artifacts and controlled experiments examining models’ tendency to judge their own incorrect answers more leniently.

Critical weaknesses
  1. Limited methodological novelty. Separating evidence reading from candidate evaluation has clear precedents in QA-based factual consistency evaluation, including FEQA, QAGS, and QAFactEval. The motivation for shielding independent verification from the original answer was also explicitly articulated in CoVe. The authors should clarify BEV’s substantive methodological contribution beyond these established paradigms and provide direct comparisons under matched evidence, models, and computational budgets.
  2. Limited generalizability and mechanistic evidence. The method primarily targets short, entity-valued factual answers, leaving its applicability to long-form generation, complex reasoning, and specialized domains untested. Moreover, while the experiments support the effectiveness of the combined verification signals, they do not establish that removing self-anchoring causes the improvement in detection.
Questions for authors

How does BEV relate to existing agentic systems that retrieve external evidence through tool use? What does it contribute beyond their evidence-gathering and verification workflows, and could the authors provide comparisons under matched retrieval and computational budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.