acceptodds
Under review as a conference paper at ICLR 2027

Answers as Probes: Mitigating LLM Hallucinations with Source-Aware Inference

Abstract

Large language models that answer from a given passage still produce answers the passage does not support. We find that many of these errors are failures of reading, not of knowledge: if Llama-2-7B-chat reads each MRQA prompt in four slightly different ways and the best of its four answers is kept, macro F1 rises from 67.0 to 74.7. Decoding-time methods recover little of this gap, for two reasons. They build the alternative reading by deleting the context, which also changes how the question is encoded, and they commit to one reading token by token, before the answer that would reveal whether the reading was right exists. We propose , a training-free decoder that addresses both. It intervenes only on what the answer may attend to, which leaves every state before the answer intact and lets several readings share one prefill and decode in one batch. Each reading proposes a complete answer; each answer then probes its source, and the unmodified model selects the one with the highest native belief plus source support: exact containment for text sources, an interventional likelihood ratio otherwise. On all 62,316 MRQA questions, improves every domain for each of five models, gaining 3.0–4.2 F1 over greedy decoding and 2.1–2.3 F1 over the strongest published decoder at greedy cost, with no tuned hyperparameters. It also improves TruthfulQA MC1 for all five models tested. Used as an instrument, it shows that hallucinated answers do not read the passage less than correct ones, but read it through shallow layers and punctuation anchors.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Hallucination mitigation, Source-aware inference, Answer-conditioned probing, Training-free inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vgGrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a new probe that attend only on specific parts of the input. They apply it on several settings and get some results. I could not really understand most of the paper, if the probe or results are interesting, how the prove is built etc.

Critical strengths
  • good coverage of tested models
  • the proposed probe seem to do something consistently positive compared to the baselines, but I find it hard to understand what it is
  • There seem to be interesting understandings from the probe, but it is very hard for me to understand them, whether they are really surprising (e.g. "Which reading is right is decidable only after the answer exists" seems trivial).
Critical weaknesses
  • Why using MRQA as the benchmark?

  • It was hard for me to read the paper. Examples: (a) The intro is not clear; (b) The method section conflates between analysis (not clear on which benchmark) and modeling, and it is in general hard to read (all sub sections); (c) answer-row is not defined; (d) I am not sure what do the D4/5/2 markers mean; (e) "for i ∈ T" - what is T? etc.

  • "δencS" in 20% - why is this surprising? ℓ¬S (y | x) encodes unconsistenly - the question with S, the answer without S but over an S-encoded question, which may include internal state aspects that cannot be recovered any more.

  • "the first-token log-probability ranks the better reading higher in only 55% of pairs" - could it be the the first answer token is quite generic and not related to S?, e.g. "the answer is" or "the capital of is" - in both, "the" is a generic starting point and most likely have no strong relatedness to S.

  • In Eq 5, why not also have a way to have only log pRθ(v) - testing directly only the flow in R? This seems a more direct measure than β=1 for the flow via R.

Questions for authors
  • See comments above

  • I strongly encourage the authors to rewrite and resubmit.

5 comments
AuthorsOpenReview ↗

Response to Reviewer vgGr (1/3): the method, MRQA, readability

We thank the reviewer for the frank assessment, which made us restructure the paper (revised PDF, changes in blue). We answer each point below.

What the method is. AnswerProbe is a decoding method: it changes how a model generates an answer so that the answer is correct more often. It is not an analysis probe. "Probe" in the title refers to its last step, in which each candidate answer is checked against the source it should come from.

The problem. When an LLM answers from a passage, many wrong answers are errors of reading, not of knowledge. If the same model reads each MRQA prompt in four slightly different ways and the best of its four answers is kept, F1 rises from 67.0 to 74.7. What fails is how the passage is read, and which reading is trusted.

How existing decoders work, and where they fall short. Most context-faithful decoders follow one template, contrastive decoding: compare the model with a weakened copy of itself that lacks some evidence, and boost the tokens whose probability the evidence raises. They differ only in how the weakened copy, the reference, is built:

  • CAD and COIECD delete the passage from the input;
  • DoLa and SLED read out an earlier layer;
  • DAGCD re-weights passage tokens by attention.

All of them build one such reading and apply it to every token of the answer. This has two consequences, which the paper calls blind spots.

  1. Deleting the passage also changes how the question is understood. The model reads the question after the passage, so without the passage the question itself is encoded differently. CAD's score therefore mixes the effect of the evidence with an unrelated effect of re-encoding the question. We show this exactly (Eq. 3), and the second effect favours the wrong answer.
  2. They commit to the reading before the answer exists. Whether a reading was right is visible only in the answer it produces, for instance whether that answer appears in the passage. Yet a token-level decoder must decide at the first token.

Our method, in the same terms. AnswerProbe keeps the contrastive template but changes three things.

  • How a reading is built. We block the answer positions from attending to one region of the prompt (the passage, or the punctuation tokens), instead of deleting input. Because earlier positions never attend to later ones, the question is encoded exactly as in the unmodified model, which removes the re-encoding effect (blind spot 1). It also lets all readings share one pass over the prompt.
  • How many readings. We use a small family of four (Eq. 5).
  • When to decide. Each reading writes a complete answer. The unmodified model then picks the answer it believes most among those the passage supports (Eq. 6), which addresses blind spot 2.

Prior decoders are special cases of this framework: one reading, one strength, decided per token (Appendix Table 6). Conversely, giving them several strengths and our decision rule lifts every one of them, e.g. CAD from 65.96 to 70.02 F1 (Table 3).

Why MRQA? MRQA is the standard benchmark of context-faithful decoding. CAD, COIECD and DAGCD all report on it, and following the public DAGCD protocol makes their published numbers directly comparable to ours. It spans seven domains, including knowledge conflict (NQ-Swap). The revision adds MRQA's six out-of-domain test sets (+2.9 F1, positive on all), multi-document HotpotQA, abstractive NarrativeQA, and closed-book TruthfulQA.

Readability. Thank you; we agree and revised the paper accordingly:

  • (a) Introduction. It now says what a contrastive decoder does and what we mean by a "reading", explains what the "probe" in the title refers to, and states the method in two steps, propose and decide, alongside the running example of Figure 1 (a wrong answer, "12,000", that a different reading corrects to "over 10,000").
  • (b) Analysis separated from method. The two blind spots of existing decoders now form their own Section 3, which names its data (2,100 fresh MRQA questions; Llama-2-7B-chat). Before the decomposition (Eq. 3), it compares three computations to show which part of CAD's score reflects the passage and which part reflects the re-encoded question. The method is Section 4.
  • (c, e) Answer rows 𝒯. 𝒯 = {n, …, n+m−1} are the positions whose outputs are the answer tokens (in the running example, the positions that produce "over", "10" and ",000"). They are now defined with the example where they first appear, and all symbols are listed in a notation table (Appendix A).
  • (d) D1–D6 are the numbered derivations in Appendix I; every reference now reads "Appendix I, D4".
  • Selection rule. Section 4.3 now opens with the rule in one sentence ("among the answers the passage contains, take the one the unmodified model finds most likely; if none is contained, take the most likely overall") before its probabilistic form (Eq. 6).
AuthorsOpenReview ↗

Response to Reviewer vgGr (2/3): is the main finding trivial?

Is "which reading is right is decidable only after the answer exists" trivial? Put in words, it can sound obvious. Existing decoders, however, are built on the opposite assumption, and our measurements contradict several views held in prior work. That is what makes the findings informative.

1. Prior decoders decide per token; the decision needs the complete answer. CAD, COIECD, DAGCD, DoLa and SLED all fix one adjustment before generating and apply it at every token. They implicitly assume that the next-token distribution already carries the signal of which reading is right. We tested this assumption directly (all 62,316 questions; pairs of candidate answers whose F1 differs by at least 0.5):

How the better of two answers is pickedAccuracy
first-token probability (what a per-token decoder sees)55%
complete answer's likelihood65%
complete answer contained in the passage66%
both together (our rule)75%

The two answers already differ at the first token in 83% of pairs, so a per-token decoder commits at step one, yet its signal there is close to a coin flip. Acting on the complete answer instead is worth +3.0 to +4.2 F1 on five models, and the same step lifts CAD from 65.96 to 70.02 and COIECD from 67.20 to 69.92. It also explains why two of our four readings, each worse than greedy decoding on its own, still add accuracy: a reading only needs to be right where the others are wrong, and only a decision made after the answers exist can use that.

2. Findings that differ from prior views.

Prior viewWhat we find
CAD (Shi et al., 2024): comparing the model with and without the context isolates what the context contributes.Deleting the passage also changes how the question is encoded, and this part of CAD's score ranks the better answer first in only 20% of pairs, below the 50% of chance. The part that reflects the passage itself (ι_S) reaches 87%.
DAGCD (Huang et al., 2025) and Lookback Lens (Chuang et al., 2024): context-faithfulness errors come from insufficient attention to, or use of, the context.Hallucinated answers rely on the passage as much as correct ones: blocking it lowers their log-likelihood by 3.42 vs 3.76 nats per token. What differs is the route: wrong answers depend more on the first four layers (0.87 vs 0.31) and on punctuation tokens (0.34 vs 0.10).
OPERA (Huang et al., 2024): over-attending to a few summary tokens causes hallucination, so it should be penalised.Punctuation tokens do take 21–35% of the answer's attention from layer 5 on, yet blocking them barely changes a correct answer. They are a high-traffic, low-content route that helps on some questions and misleads on others. This is why we use both directions as readings (anchor contrast and anchor removal) instead of penalising one.

3. Implications. Interventions on how a model reads should be judged at the answer level, not per token. Input deletion should not be used as the reference when the goal is to score how much an answer relies on its source. And hallucination can be addressed by changing which route the model reads through, not only how much it attends to the context. Blocking what the answer may attend to is a cheap and exact tool for such analyses: it leaves the prompt's representation untouched, and several interventions share one pass over the prompt.

AuthorsOpenReview ↗

Response to Reviewer vgGr (3/3): δ^enc, first tokens, and log p_R

Why is δ^enc at 20% surprising? The reviewer's reading of the mechanism is correct. Here is why the number is still surprising. δ^enc compares two computations in which the answer cannot see the passage S, and which differ only in whether the question was encoded with S or without it. That difference says nothing about whether a candidate answer is supported by S. A term that carries no information about which answer is better should rank the better one first about half the time, i.e. 50%. It does so in only 20%: it systematically favours the wrong answer. The reason is visible in the data. A question encoded without its passage evokes the model's most typical answer, and contrasting against that penalises every answer the model also knows, often the correct one. This term sits inside every input-deletion score, which is why CAD ranks at chance on TriviaQA and SearchQA (Finding 1). An intervention that keeps the question's encoding fixed removes it by construction (Section 4.1). The revision now states this expectation explicitly (Section 3 and Finding 1).

Are first tokens generic? This is a good alternative explanation, so we tested it. Most first tokens are content-bearing: 86% are content tokens, and only 3.5% are function words such as "the". On the 12,247 pairs where both first tokens are content words, first-token accuracy is still 55.5%, against 72.9% for complete answers. On pairs whose first tokens differ, it is 56.4% against 75.3%.

What this means. The low first-token accuracy is not an artefact of generic tokens. Even when the first token is a content word, such as the start of a name or a number, it does not reveal which reading is right. The information appears only once the answer is complete and can be checked against the passage. This is why the decision has to be deferred to complete answers (the second principle), and why a token-level decoder cannot exploit readings that are right on different questions (Finding 2; Appendix F).

"Only log p_R" in Eq. 5. As a reminder, Eq. 5 builds a reading from two distributions over the next token: the unmodified model p_θ, and p_θ^¬R, the same model with its answer positions blocked from attending to a region R (e.g. the passage). The reading is log p^(R,β) = (1+β) log p_θ − β log p_θ^¬R. With β = +1, it amplifies what flows through R; with β = −1, it removes R. The reviewer proposes the complement: a distribution p_R in which the answer positions attend only to R.

We implemented it. Letting the answer positions see only the passage would also hide the question, so this variant lets them see the passage and the question. Used alone it scores 49.45 F1, and its β = 1 contrast 63.60, against 67.43 for our passage contrast (fresh 7k panel, Llama-2). Ranking candidates by the passage-blocked likelihood alone is worse still (13–33 F1).

Why. Allowing only R removes everything else the answer needs: the instruction, the answer format, and the anchors outside the passage. The result measures a crippled model rather than the flow through R. Blocking R changes one thing and keeps everything else, so the difference between p_θ and p_θ^¬R isolates exactly the contribution of R. This is why Eq. 5 contrasts with p_θ^¬R (Appendix F).

We took this advice seriously and rewrote the paper within the discussion period. In short, it contributes:

  1. A diagnosis of existing decoders. An exact decomposition shows that deleting the passage, as CAD does, mixes the evidence with a re-encoding of the question that systematically favours wrong answers (20% vs 50% chance). We also show that the right reading cannot be identified at the first token (55%) but can be once the answer is complete (75%).
  2. A method. Readings built by blocking only what the answer may attend to, which keeps the question's encoding intact and costs one shared pass over the prompt, plus a decision among complete answers by the unmodified model's belief and the source's support. It is training-free and has no tuned hyperparameters. It gains +3.0 to +4.2 F1 on five models, is the only method that improves every domain on every model, and lifts every prior decoder it wraps.
  3. Findings on how hallucinated answers read. They read the passage as much as correct answers do, but through shallow layers and punctuation "summaries", a high-traffic, low-content channel.

We would be grateful if the reviewer could take another look at the revised version.

Reviewer_NRdoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper points out that existing contrastive decoding methods (e.g., CAD) build the reference distribution by removing the source S (the passage) from the input, which also changes the representation of the question and thus contaminates the contrastive signal with a "question re-encoding" term. To address this, the paper proposes ANSWERPROBE: instead of modifying the input, it adds a −∞ mask to the attention logits of the answer rows, blocking their attention to the passage S or to punctuation/whitespace anchor tokens A. Because attention is causal, all states before the answer remain unchanged, so the prefill can be shared and the readings can be decoded in parallel. Based on this, the paper constructs four readings (native, passage contrast, anchor contrast, anchor removal), each of which generates a complete answer, and then selects the final answer with a set of rules: it first checks whether a candidate is a contiguous substring of the passage (exact containment), and then ranks candidates by the original model's mean log-likelihood ℓ̄ (in the closed-book setting, by the difference between ℓ̄ and the ℓ̄ with the source blocked). On 5 models and 7 MRQA/NQ-Swap domains, the method improves over greedy decoding by about +3.0 to +4.2 F1, and through fairly thorough ablations, the paper argues that the gain comes from the different readings produced by masking certain tokens, as well as from the answer-level selection mechanism itself.

Critical strengths
  1. The method is novel: starting from the observation that input deletion changes the question representation, it replaces input deletion with answer-row attention masking, and provides a clear exact decomposition (Eq. 3) and an argument for prompt invariance. It also reuses the prefill and decodes the four readings in parallel, with a cost of about 1.68× greedy, lower than CAD, while achieving good results.
  2. The ablations are fairly thorough: they separately verify the contribution of each reading and of the two selection factors (belief and support), and include controls such as the length budget and equal-size candidate pools (sampling, random knockouts). This indicates that the gain does not come merely from longer answers or more candidates, but from structured readings that change how the model reads.
  3. The evaluation is broad: it covers 5 models (4 families) and 7 domains with 62,316 questions in total, and extends to closed-book TruthfulQA and general-ability checks on MMLU/ARC/GSM8K; the appendix also provides case studies of both successes and failures.
Critical weaknesses
  1. The comparison with baselines is not fully equal. In its selection stage, ANSWERPROBE uses information from the document (exact containment), whereas baselines such as CAD, COIECD and DAGCD output only a single answer and do not use this answer-level verification. The +3.0 to +4.2 F1 in Table 1 therefore mixes two sources of gain: the family of readings and document-based answer-level verification. The paper's own ablation (belief only 68.01, support only 69.38) also shows that the selection rule itself contributes a lot. A control that gives CAD/DAGCD etc. the same candidate generation and selection pipeline is missing. For example, one could vary the contrast strength of CAD to obtain multiple outputs for CAD, and then apply the same procedure to obtain the evaluation results.
  2. The selection rule relies on the assumptions of extractive QA. In the main experiments the gold answers are contiguous spans of the passage, which is exactly what exact containment matches. For generative QA that requires reasoning, summarization or rephrasing, this check may fail. The failure cases in the appendix also show that it is sensitive to string normalization (e.g., an en dash causes a correct answer to be judged as not in the passage), and that it cannot rank two contained spans by their overlap with the gold answer.
Questions for authors
  1. If baselines such as CAD / DAGCD are also made to generate multiple candidates (e.g., multiple strengths α or multiple variants) and the same containment + ℓ̄ selection rule is applied, how much of the gain remains? Could the authors report such results for a fair comparison?
  2. In the equal-size candidate pool experiment, how does the curve of "temperature sampling + the same selection rule" change when the number of candidates grows to 8, 16 or more? Does the advantage of the structured readings shrink as the number of candidates increases?
  3. When document information is not used and only ℓ̄ is used (Belief only, 68.01), the gap between ANSWERPROBE and greedy is only about 1 point. Does this mean that most of the gain comes from the containment check? In scenarios where the containment check is unavailable (e.g., generative answers, or answers that need to combine information from multiple places), how much of the gain does the method retain?
  4. The anchor definition is purely rule-based (punctuation and whitespace). For models with different tokenization such as Llama-3 and Qwen, are the anchor proportion and its role consistent? Would the results change if it were replaced with sink tokens identified automatically by the model?比如使用attention logits作为判断锚点的方法
4 comments
AuthorsOpenReview ↗

Response to Reviewer NRdo (1/3): W1 and Q1

We thank the reviewer for the precise questions. Each led to a new experiment, all in the revised PDF (changes in blue; Section 5.3, Table 3, Appendices B–E).

W1/Q1: the baselines with the same pipeline. We ran this control as suggested. CAD and COIECD ran at α ∈ {0.5, 1, 2}, and DAGCD at α ∈ {2, 4, 8}. Each was pooled with greedy decoding, so that every pool has 4 candidates like ours, and every pool was selected with our rule (containment, then mean log-likelihood ℓ̄). COIECD uses its official decoding rule and prompt. DAGCD uses its released decoder, with a detector trained by its released script; it reproduces the published gains.

4 candidates, same ruleLlama-2MistralLlama-2, belief onlyMistral, belief only
greedy67.0466.6267.0466.62
greedy + CAD (3 strengths)70.0270.8567.6267.47
greedy + COIECD (3 strengths)69.9270.5467.5667.54
AnswerProbe70.0670.7668.0168.24
greedy + DAGCD (3 strengths)†68.0669.7066.6567.35
AnswerProbe†69.1370.5167.1868.03

†Fresh 7k panel (1,000 held-out questions per domain); the other rows use all 62,316 questions. Belief only ranks candidates by ℓ̄ without checking the passage.

How much of the gain remains. Given our framework, the CAD pool reaches the same accuracy as our readings (70.02 vs 70.06; 70.85 vs 70.76). Our readings stay ahead of the COIECD pool (+0.14, +0.23) and the DAGCD pool (+1.07, +0.82). The CAD pool is no longer CAD as published, however. Decoding several strengths to complete answers and letting the unmodified model choose is exactly our framework, a family of readings (Eq. 5) followed by our decision (Eq. 6). Only the reference of each reading differs: CAD deletes the passage, while we block the answer positions from it. The control therefore answers two questions.

1. Does the framework help other decoders? Yes, every one of them. Over the better of greedy decoding and each method's default, it lifts CAD by +3.0/+4.2, COIECD by +2.7/+3.6 and DAGCD by +0.8/+1.3 F1. No prior contrastive decoder for context faithfulness proposes complete answers and then decides among them by source support.

2. Within the framework, does the reference matter? Four further experiments say it does, and none of them favours input deletion.

(a) A matched 2×2 check. Besides the reference, the CAD pool differs from our readings in one decoding detail: CAD drops the plausibility constraint of contrastive decoding (only tokens with at least 10% of the top token's probability may be chosen). We crossed the reference with the constraint, each cell being greedy plus three strengths (fresh 7k panel):

full rule (belief only)Llama-2Mistral
deletion, no constraint (= CAD pool)69.50 (66.94)70.73 (67.23)
deletion, with constraint68.82 (66.99)70.12 (67.51)
answer rows blocked, with constraint (our reading)68.68 (67.07)70.54 (67.92)
answer rows blocked, no constraint69.41 (66.99)71.36 (67.68)
answer-row minus deletion, same constraint−0.08 / −0.14 (n.s.)+0.62 / +0.42

At matched constraint, the answer-row reference equals input deletion on Llama-2 and beats it on Mistral, under both rules. The CAD pool keeps up only because it lacks the constraint.

(b) Without a string test. Ranked by belief alone (right-hand columns of the first table), our readings beat every equal-size pool on both models, by +0.39/+0.77 against CAD, +0.45/+0.70 against COIECD and +0.54/+0.68 against DAGCD, with every bootstrap interval above zero.

(c) Closed-book QA (TruthfulQA, MC1). Here no passage can be searched. Scoring by input deletion lowers MC1 below the native model (36.7 → 35.6 on Llama-2), whereas scoring by answer-row support raises it on all five models:

MC1, length-normalisedLlama-2-7BLlama-2-13BMistral-7BLlama-3-8BQwen3-1.7B
native36.739.342.543.331.7
AnswerProbe41.942.142.748.136.4

(d) Ten retrieved documents (HotpotQA, 7,405 questions; Q3). When the context is long and noisy, deleting it leaves a question that evokes only memory, so the CAD pool falls below greedy decoding. Our readings gain on both question types:

Llama-2Mistral
greedy50.6657.02
greedy + CAD (3 strengths)49.8657.00
AnswerProbe52.24 (+1.58)59.09 (+2.06)

Finally, the two kinds of candidate complement each other: adding our readings to any baseline pool gains a further +1.0 to +2.0 F1. All of these results are in Section 5.3 and Appendices B and E of the revision.

AuthorsOpenReview ↗

Response to Reviewer NRdo (2/3): Q2 and W2

Q2: larger sampling pools. Prior work shows that the right answer is often already among a model's alternatives, and that the hard part is to pick it out. Repeated sampling raises coverage, the share of questions where some sample is correct, but without a verifier majority voting and reward models plateau (Brown et al., 2024). Branching on the top-k alternatives of the first token recovers reasoning paths that greedy decoding misses (Wang & Zhou, 2024). We therefore measured both accuracy and coverage. Fresh 7k panel, temperature 0.7, same selection rule; cost is wall time per question, with every pool batched, measured in one run on an otherwise idle GPU:

Candidate poolSizeF1, Llama-2F1, MistralCoverage*, Llama-2Coverage*, MistralCost (× greedy)
greedy + 3 samples468.0869.2971.473.33.1
greedy + 7 samples869.0570.5674.376.95.3
greedy + 15 samples1669.6671.6876.680.19.6
AnswerProbe readings469.1370.5173.975.51.7

*Coverage is the F1 of the best candidate in the pool (an oracle), an upper bound on any selection rule.

The advantage does shrink with more samples. At equal size the readings lead by +1.04 and +1.22 F1; sampling ties at 8 candidates and leads at 16. Structure still matters in three ways.

  • Coverage per candidate. Four readings cover as much as eight samples (73.9 vs 74.3), because each reading reads the passage differently rather than adding random variation.
  • Cost. Matching the readings takes about 3× their wall time, and beating them about 6×.
  • Without a string test. Ranked by belief alone, even 16 samples do not beat the four readings (66.99 vs 67.18; 67.98 vs 68.03).

These methods also differ from ours in kind. They propose by sampling or by branching on tokens, and decide by votes or confidence. We propose by changing how the passage is read, and decide by whether the source supports the answer. We added this comparison to Section 2 and Section 5.3.

W2: extractive assumptions, normalisation, and ranking contained spans. We agree that exact containment is an extractive test. The decision rule itself is not tied to it, though: Eq. 6 combines the model's belief with a support score, and containment is only one choice of support. We tested each case the reviewer raises.

Reviewer's concernWhat we testedResult over greedy decoding
Rephrased or summarised answersNarrativeQA, 2,000 questions with answers written by annotators (only 64% appear verbatim in the text)containment: −0.68 F1; entailment support (a): +1.05 F1, +1.07 ROUGE-L (both significant)
Combining several documentsHotpotQA with all 10 paragraphs, 7,405 questions+1.58 / +2.06 F1 (Llama-2 / Mistral)
Computed answersGSM8K, answers not in the problem text (Appendix K)+3.2 accuracy, with the readings' derivations as the source
String normalisationstronger normalisation (b) before containment, all 62,316 MRQA questions262 selections change; F1 70.06 → 69.94
Ranking two contained spansthe 36% of MRQA questions with two or more contained proposalsbelief ranks them: +2.4 F1; adding entailment after containment: 70.06 → 70.60 overall

(a) An entailment (natural-language inference) model predicts whether a text implies a statement. As support we use the probability that the passage entails "⟨question⟩ The answer is ⟨answer⟩", computed with a public DeBERTa-v3-large NLI model without any tuning; the template was fixed before seeing any result. This is an optional plug-in for free-form answers; the default method uses no auxiliary model. (b) NFKC Unicode normalisation (which maps characters that look alike, such as full-width digits, to one form), with every dash and hyphen variant mapped to a space.

Conclusion. None of the three concerns limits the method. The en-dash failure is an isolated case (262 of 62,316 questions, and stronger normalisation does not help), so we keep the standard SQuAD normalisation. Ranking several contained spans is exactly what the belief term does, and it gains +2.4 F1 on those questions. Beyond extractive answers, containment fails as the reviewer expects, but the decision rule does not: replacing containment with an off-the-shelf entailment score turns the gain positive and significant on abstractive QA, and the same rule helps with multiple documents and computed answers. The support term is what adapts to the task; the readings and the decision stay the same.

AuthorsOpenReview ↗

Response to Reviewer NRdo (3/3): Q3 and Q4

Q3: is the gain only containment, and what remains without it?

  • (a) Where containment cannot decide. In 36% of questions, two or more proposals are passage spans, so containment cannot pick one. There, ranking by belief gains +2.4 F1 over greedy decoding (+3.2 with the full rule).
  • (b) Why "+1 point" understates belief. Greedy decoding is already the native model's most likely answer, so a belief-only choice can only improve on it by finding answers the model itself prefers once they are complete. That it does so by +1.0, and that our readings are the best candidates for it (Q1), is the evidence for the second principle.
  • (c) Without any string test. Replacing containment by an off-the-shelf entailment model (W2) keeps +2.07 of the +3.02 F1 on all 62,316 questions. Applying entailment after containment raises the gain to +3.56.
  • (d) Generative answers. NarrativeQA (W2): +1.05 F1 and +1.07 ROUGE-L with entailment support, against −0.68 with containment.
  • (e) Out of domain. The MRQA benchmark comes with an official out-of-domain test split: six datasets held out from its training and development data. They are BioASQ (biomedical), DROP (discrete reasoning), DuoRC (movie plots), RACE (school exams), RelationExtraction and TextbookQA (science), 9,633 questions in all. With the paper's prompt and rule unchanged, AnswerProbe gains +2.94 [2.50, 3.39], and it is positive on every set (from +2.06 on DuoRC to +4.05 on DROP). Entailment after containment gives +3.66.
  • (f) Combining information from several places. We used HotpotQA in its standard distractor setting. Each question comes with the two paragraphs needed to answer it and eight distractor paragraphs retrieved by TF-IDF, all ten placed in the prompt (7,405 questions). Questions are bridge (one paragraph identifies an entity that the other describes) or comparison (two entities must be compared). The four readings gain +1.58 [1.05, 2.08] on Llama-2 and +2.06 [1.58, 2.58] on Mistral, on both types. The equal-size CAD pool falls below greedy decoding (49.86 vs 50.66; 57.00 vs 57.02), because deleting the whole context leaves a question that only evokes memory. (Appendix E).

Q4: anchors across tokenizers, and automatically identified sinks. We ran both checks on the fresh 7k panel.

Llama-2Llama-3Qwen2.5
anchor share of prompt tokens20.7%19.3%18.7%
family F1, rule anchors (paper)69.1468.8866.83
family F1, attention-chosen anchors69.1569.0465.83
anchor contrast alone: rule / attention-chosen65.49 / 67.2864.96 / 66.2563.35 / 64.16
anchor removal alone: rule / attention-chosen64.29 / 28.4164.89 / 34.5157.64 / 25.24

The attention-chosen anchors follow the reviewer's suggestion: the k passage tokens that receive the most attention from the last prompt position, with k equal to the number of rule anchors in the passage.

The results bear out three properties the method relies on.

  1. Anchors are a stable, tokenizer-independent channel. The share of anchors varies by only 2 points across three tokenizers. The rule depends only on characters (punctuation and whitespace), not on the model, so it carries over unchanged.
  2. Anchors carry attention but little content (Finding 3). This is why the two anchor readings work. Attention-chosen keys do mark where the model reads, so contrasting with them strengthens the contrast reading (e.g. 67.28 vs 65.49). But they are content tokens, so removing them removes the answer itself: the removal reading collapses to 25–35 F1, while removing punctuation leaves the answer readable (58–65 F1).
  3. Both directions are useful only on a low-content channel. Anchor removal is meant to make the model read content tokens directly when the aggregated summary misleads it. This works only if the blocked tokens are summaries, not content. The family therefore needs a channel that is high in traffic and low in content, which is what the punctuation rule selects and what an attention criterion does not.

So the family is unchanged on Llama-2 and Llama-3 (±0.2) and 1.0 better with the rule on Qwen2.5. The simple rule is both tokenizer-agnostic and the better choice (Appendix D).

We hope these results address the reviewer's concerns. The questions helped us state the framework more precisely: proposing complete answers under several readings and deciding among them is what lifts every decoder, and building the readings at the answer rows is what keeps the framework working when the source cannot be searched, in closed-book QA, and with many documents.

I thank the authors for conducting the additional experiments in response to my questions. The newly added experiments have satisfactorily addressed my concerns and clarified the points I raised.

Reviewer_eKh6OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that many errors in context-grounded QA are not because of knowledge, but they are the failures of not reading correctly. It proposes AnswerProbe, a training-free decoder built on the answer-row intervention. The method runs four readings as one shared-refill batch, and each produces a complete answer. The main mechanistic finding is that anchors receive much of the answer rows’ attention but carry little information, and wrong answers lean more on shallow layers and anchors.

Critical strengths

The evaluation is very substantial and the method is very straightforward and can be used for almost any model.

Critical weaknesses
  1. Much of the improvements come from a verifier where the baselines do not get.
  2. The length of the answer in the experiments settings are capped short, and in real-world scenarios, things would be more complicated such as abstractive or long-form RAG answers.
Questions for authors

This paper is a solid paper to me, so specific questions to the authors.

4 comments
AuthorsOpenReview ↗

Response to Reviewer eKh6

We thank the reviewer for the positive assessment and for recognising that the evaluation is substantial and that the method applies to almost any model. Both points the reviewer raised led to new results, summarised below (revised PDF, Section 5.3 and Appendices B–E).

W1: the verifier. We thank the reviewer for raising it; we gave every baseline the same verifier: three strengths each, pooled with greedy decoding (4 candidates, like ours), selected by our rule.

4-candidate pool, same verifierLlama-2MistralLlama-2, belief onlyMistral, belief only
greedy decoding67.0466.6267.0466.62
greedy + CAD (3 strengths)70.0270.8567.6267.47
greedy + COIECD (3 strengths)69.9270.5467.5667.54
AnswerProbe (4 readings)70.0670.7668.0168.24
greedy + DAGCD (3 strengths)†68.0669.7066.6567.35
AnswerProbe†69.1370.5167.1868.03

†Fresh 7k panel; other rows use all 62,316 questions.

Several strengths plus a decision among complete answers is precisely the framework of the paper, and giving it to the baselines lifts each of them (CAD +3.0/+4.2, COIECD +2.7/+3.6, DAGCD +0.8/+1.3 F1). So answer-level selection, the "answers as probes" of the title, is a general mechanism that existing decoders do not use. Within it, our readings match the CAD pool when the passage can be searched and beat the COIECD and DAGCD pools. Where the passage cannot be searched, they are the best candidates of all:

Where the source cannot be searchedInput deletionAnswerProbe
belief only, equal-size pools (Llama-2 / Mistral)67.62 / 67.4768.01 / 68.24
TruthfulQA MC1, Llama-2 (native 36.7)35.641.9
HotpotQA, 10 documents (Llama-2 / Mistral; greedy 50.66 / 57.02)49.86 / 57.0052.24 / 59.09

A matched 2×2 control further shows that the CAD pool keeps up under containment only because it drops the plausibility constraint of contrastive decoding; at the same constraint, our reference is equal on Llama-2 and better on Mistral (+0.4 to +0.6 F1). Details are in our reply to Reviewer NRdo.

W2: beyond short extractive answers. Without any re-tuning:

SettingWhat it testsGain over greedy decoding
MRQA out-of-domain test sets (6 datasets, 9,633 q.)*new domains+2.94 F1, positive on all six (DROP +4.05, RACE +2.78, BioASQ +2.50)
HotpotQA, 10 paragraphs per question (8 retrieved distractors)RAG-style noisy context, multi-hop+1.58 / +2.06 (Llama-2 / Mistral); an equal-size CAD pool falls below greedy
NarrativeQA (answers written freely by annotators)abstractive answers+1.05 F1, +1.07 ROUGE-L with an off-the-shelf entailment model as the support term
fixed 48-token budget instead of gold length (paper)longer outputs+2.8 F1
TruthfulQA MC1, 5 models (paper)closed-bookimproved for every model

*The official out-of-domain split of the MRQA benchmark: six QA datasets held out from its training and development data.

Long-form answers with several claims would need claim-level support. This fits the same decision rule, since Eq. 6 only requires a support score for each candidate. We state it in the Limitations.

Insights and outlook. Beyond the gains, the paper offers a way to look at hallucination that we believe will be useful beyond QA.

  • Many errors are errors of reading: the model can produce the right answer, but reads the passage through a misleading route.
  • Which reading is right becomes visible only in the complete answer, which a per-token decoder never sees.
  • Hallucinated answers read the passage as much as correct ones, but through shallow layers and punctuation "summaries", a high-traffic, low-content channel.

Answer-row interventions make these routes measurable at no cost to the prompt's representation. We see two natural next steps: RAG systems with many retrieved sources, where readings can be defined per document, and long-form generation with claim-level support.

Area_Chair_F7ZZOpenReview ↗

Please respond

Hello,

The discussion period ends soon. Please respond to the authors' last comment and note whether it changed your evaluation of the paper.

Thanks,

Your AC

Dear Reviewer eKh6,

Following up on your two weaknesses, we have posted additional results above ("Additional results: AnswerProbe scales to 32B and to long-context RAG at near-greedy cost"), with the method unchanged. In brief:

  • W2, real-world RAG. On LongBench (750 questions with retrieved contexts of 11k tokens on average, up to 18k), AnswerProbe gains +3.01 (Qwen2.5-7B), +2.46 (Llama-3-8B) and +1.99 (Qwen2.5-14B) F1 over greedy decoding, positive on all four tasks for every model. The method is therefore not tied to short extractive passages.
  • W1, the verifier. On the same questions, an equal-size CAD pool under the same verifier gains much less: the readings are 1.1–1.8 F1 above it, and ranked by belief alone, with no string test, they keep 42–79% of their gain. The gain in long-context RAG comes from what the readings propose, not from the verifier.
  • Cost. At 11k tokens AnswerProbe takes 1.08× the time of greedy decoding, because the four readings share one prefill.
  • Scale. On MRQA the gain is +2.73 for Qwen2.5-14B and +5.08 for Qwen2.5-32B, positive in all seven domains: stronger models still misread the passage, and the readings correct it.

The results are added to Appendices C and E of the revised PDF. We hope they address both concerns, and thank you again for your positive assessment.

Thank you for the thorough response. I keep my positive score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading. We ran every requested experiment and several more. All are in the revised PDF (changes in blue): a new Section 5.3 and Table 3 on equal candidate pools, and Appendices A–F. We also improved the readability of the paper. The method, its readings, its hyperparameters and the main results (Table 1) are unchanged.

How AnswerProbe relates to existing decoders. Decoding-time methods such as CAD, COIECD, DAGCD and DoLa all build one modified distribution, a "reading" of the prompt, and commit to it token by token. AnswerProbe has four parts: (i) it builds a reading by blocking what the answer positions may attend to, instead of deleting input; (ii) it uses a small family of readings rather than one; (iii) it lets each reading finish a complete answer; and (iv) it chooses among the answers with the unmodified model's belief plus the source's support.

1. The equal-pool control (NRdo W1/Q1, eKh6 W1). Following NRdo, we ran each baseline at three strengths, pooled with greedy decoding (4 candidates, like ours), and selected with our rule. This places each baseline inside parts (ii)–(iv) of our framework and changes only part (i), how a reading is built.

  • The framework lifts every baseline: CAD by +3.0/+4.2 F1 (Llama-2/Mistral), COIECD by +2.7/+3.6 and DAGCD by +0.8/+1.3, each over the better of greedy decoding and its default. No existing contrastive decoder for context faithfulness proposes several complete answers and decides among them by source support.
  • Within the framework, our answer-row readings match the input-deletion (CAD) pool when the passage can be searched (70.06 vs 70.02; 70.76 vs 70.85), and beat the COIECD, DAGCD and sampling pools. A matched 2×2 check (reference × plausibility constraint) shows that CAD's ability to keep up comes from lacking the constraint, not from deleting the input: at the same constraint, the answer-row reference is equal on Llama-2 and better on Mistral (+0.42 to +0.62 F1, significant).
  • They are the better choice wherever the source cannot be searched. Ranked by belief alone they beat every equal-size pool (+0.3 to +0.8 F1, all significant). On closed-book TruthfulQA, the input-deletion analogue lowers MC1 while ours raises it on all five models. With 10 retrieved documents, the equal-size CAD pool falls below greedy decoding, while ours gains +1.6/+2.1.

2. Beyond single-passage extractive QA (eKh6 W2, NRdo W2/Q3).

  • +2.9 F1 on the six out-of-domain test sets of MRQA, positive on all six.
  • +1.6/+2.1 on multi-document HotpotQA (10 paragraphs, 8 of them retrieved distractors).
  • On abstractive NarrativeQA, exact containment does not apply, but an off-the-shelf entailment model can serve as the support term: +1.1 F1 and +1.1 ROUGE-L.

3. Cost (NRdo Q2). The four readings share one prefill and decode as one batch, at 1.70× greedy wall time. Batched sampling needs about 8 candidates (5.3×) to match them, and a batched CAD pool costs 1.86× (all measured in one run on an otherwise idle GPU).

4. Clarity (vgGr). A separate section on the two blind spots of existing decoders, with its data; a running example; definitions at first use; a notation table (Appendix A).

AuthorsOpenReview ↗

Additional results: AnswerProbe scales to 32B and to long-context RAG at near-greedy cost

Additional results: AnswerProbe scales to 32B and to long-context RAG at near-greedy cost

We thank the reviewers again. Two further checks, with the protocol, readings and selection rule unchanged and no re-tuning, address the two remaining questions about generality: does the gain vanish for stronger models, and does it survive realistic retrieval contexts? Both results are added to the revised PDF (Appendices C and E).

1. The gain holds from 7B to 32B. Fresh 7,000-question MRQA panel (1,000 per domain, disjoint from development), macro F1; ± is the standard error of the gain (context-cluster bootstrap).

ModelGreedyAnswerProbeGainDomains improved
Llama-2-7B-chat66.2569.13+2.88 ± 0.227/7
Mistral-7B-Instruct66.3770.51+4.15 ± 0.257/7
Llama-3-8B-Instruct65.3668.86+3.50 ± 0.227/7
Llama-2-13B-chat69.0572.64+3.59 ± 0.227/7
Qwen2.5-7B-Instruct63.2866.88+3.60 ± 0.227/7
Qwen2.5-14B-Instruct (new)66.3569.08+2.73 ± 0.207/7
Qwen2.5-32B-Instruct (new)60.3665.44+5.08 ± 0.267/7

What this shows. AnswerProbe improves all 49 model-domain pairs, across four families and from 7B to 32B, and the largest model gains the most. Scale does not remove the problem the paper targets: the right answer is often among the 32B model's own readings (oracle 69.73 vs. greedy 60.36), but greedy decoding commits to the wrong one, and deciding among complete answers recovers it. (Under the shared DAGCD prompt, used without a chat template for every model, 32B greedy scores below 14B; we report it as is.)

2. Long-context RAG: the readings, not the verifier, carry the gain (Reviewer eKh6 W2; W1 of NRdo and eKh6). The four English QA tasks of LongBench (multi-document HotpotQA, 2WikiMultihopQA, MuSiQue; single-document MultiFieldQA): 750 questions, 11.4k tokens on average, up to 17.9k. Qwen2.5 reads the full context (32k window); Llama-3-8B has an 8,192-token limit, so its inputs are cut in the middle to 7.5k tokens, following LongBench. The CAD pool is greedy + CAD α ∈ {0.5, 1, 2}: the same size, selection rule and greedy answer as ours.

ModelGreedyCAD poolAnswerProbeGain over greedyGain over CAD pool
Qwen2.5-7B35.4736.8838.48+3.01 ± 0.57+1.60 ± 0.57
Llama-3-8B36.6537.2839.11+2.46 ± 0.72+1.83 ± 0.68
Qwen2.5-14B49.7050.5851.69+1.99 ± 0.60+1.11 ± 0.61

What this shows.

  • Not tied to short extractive passages. On 11k-token contexts full of distractors, the gain is 2.0–3.0 F1 and positive on all four tasks for all three models.
  • The gain is not the verifier's. Under the same verifier and pool size, the readings beat input deletion by 1.1–1.8 F1 (significant on Qwen2.5-7B and Llama-3). Ranked by belief alone, with no string test, they keep 42–79% of their gain and beat the CAD pool by 1.0–1.4 F1 on all three models. In long-context RAG, the gain therefore does not rest on the string test: it comes from what the readings propose.
  • Nearly free at long context. The four readings share one prefill, so on 100 of these questions (Qwen2.5-7B, 10.9k tokens, one 48 GB RTX 4090 used by no other process) AnswerProbe takes 1.08× the time of greedy decoding (CAD pool 1.05×), against 1.7× on short MRQA prompts. The longer the context, the smaller the overhead.

Taken together. The reading errors that motivate the paper persist with stronger models and with long, noisy retrieved contexts, and the answer-row readings correct them without re-tuning and at almost no extra cost at long context. Including the earlier results (MRQA out-of-domain +2.94, ten-paragraph HotpotQA +1.58 / +2.06, NarrativeQA +1.05 with an entailment model as the support term), the method improves every model and domain we have tested, and every task once the support term matches the answer type (containment for extractive answers, entailment for abstractive ones).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.