acceptodds
Withdrawn from ICLR 2027

Why Do Language Models Answer Incorrectly Despite Sufficient Evidence? A Mechanistic Study of Retrieval-Augmented Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieval-Augmented Generation (RAG) can still produce incorrect answers even when sufficient and correct evidence is available. This work studies failures of evidence utilization in the absence of explicit knowledge conflicts. We compare responses to the same question under two conditions: one containing only supporting evidence and the other containing the full context while retaining the same evidence. We combine layer-wise answer scoring with linear probes to track changes in internal representations, and further use bidirectional residual stream patching to test whether the identified states causally affect model responses. Experiments on four open-weight instruction-tuned models across three question-answering benchmarks show that pre-generation states in the middle-to-late layers distinguish answer regression from answer rescue induced by context expansion. When the question and input evidence remain unchanged, patching in the corresponding state from the same question under a correct-answer condition repairs a subset of incorrect responses, whereas patching in the reverse direction degrades a subset of correct responses. After determining and freezing the probe parameters and intervention locations on HotpotQA, we reproduce these effects on MS MARCO and Natural Questions. Across the two types of transfer samples, the repair rates range from 25.4% to 63.7% and from 44.2% to 81.8%, respectively. Equal-norm random-source controls further show that the intervention effect on the overall residual stream depends on the source state, while the effects associated with the attention and multilayer perceptron (MLP) modules vary across models. These findings provide causal intervention evidence for understanding why models sometimes fail to convert correct evidence into correct answers, and provide a basis for studying internal-state-based correction.

Primary area:
interpretability and explainable AI
Keywords:
Retrieval-Augmented Generation, Evidence Utilization, Mechanistic Interpretability, Activation Patching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LQENOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies why RAG models can still produce incorrect answers despite being given sufficient supporting evidence. It compares support-only and full-context conditions and uses probing and hidden-state patching to analyze internal states associated with answer changes. Experiments across multiple models and datasets show that interventions on pre-generation hidden states can repair or disrupt model outputs.

Critical strengths
  1. Broad and systematic experimental design. The paper goes beyond probing and includes layerwise scoring, residual patching, component-level intervention, and free-generation evaluation.
  2. Strong controls and cross-dataset validation. It includes early-layer, question-end, identity write-back, and random-source controls, and evaluates frozen probes and intervention locations on additional datasets.
  3. Careful evaluation and statistical analysis. The paper reports detailed sample filtering, bootstrap confidence intervals, and multiple-testing correction.
Critical weaknesses
  1. The paper does not fully answer its central question of why models fail despite sufficient evidence. The results mainly show that late hidden states differ between correct and incorrect conditions and that replacing these states can change the answer. This identifies where answer-related differences become consequential, but not why evidence utilization fails in the first place.
  2. The main patching result may be partly expected from the intervention design. The intervention is applied to the residual state immediately before answer generation, often at a late layer where substantial information about the upcoming answer is already encoded. Transferring a state from a correct-answer run may therefore amount to transferring an already-formed answer decision rather than revealing a mechanism specific to evidence utilization.
  3. The connection between probing and causal intervention is weak. The final intervention location is selected based on patching effects rather than probe performance alone, and attention/MLP outputs are intervened on without corresponding probing analyses. Moreover, question-end probes achieve relatively high AUROC, while patching the same position yields very low flip rates, showing a clear gap between decodability and causal efficacy.
  4. The Regression-vs.-Rescue probe may largely reflect signed correctness differences. Since the two groups are defined by opposite correctness transitions across the two contexts, a probe on hC−hSh^C-h^S may simply learn the sign of the difference between two correctness-related readouts rather than a distinct transition mechanism.
  5. The novelty over prior latent-intervention work is not sufficiently clear. Giannetti et al. already study RAG failures where correct evidence is present but distractors lead to incorrect answers, and show that latent interventions can recover correct generations. The paper should clarify what new mechanistic insight it provides beyond this prior work.
  6. UNKNOWN responses do not establish the absence of parametric knowledge. Because the closed-book prompt explicitly instructs the model not to guess and to output UNKNOWN when confidence is low, repeated UNKNOWN responses only show that no explicit confident answer is observed. They do not rule out latent or low-confidence parametric knowledge, and therefore do not fully exclude knowledge conflict.
Questions for authors

Please see the weaknesses.

Reviewer_X4ENOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1(−1)
Soundness
2
Read review
Paper summary

The goal of the paper is to try to understand how via model internals or inspections of model internals does a model still produce incorrect answers when given different context types. On HotpotQA, MS Marco, and NQs on 4 small language models, they find that middle-to-late layers tend to distinguish between cases where providing the evidence yields correct but full context yields incorrect vs. the converse. There are a few ablations that are hard to parse, perhaps heavily AI edited, that may indicate causal validation for some of their claims.

Critical strengths
  1. Understanding when and how context is utilized by the model in a variety of scenarios is vital to build better systems.
Critical weaknesses
  1. This work seems very similar to a lot of other work in distinguishing different types of context and correctness prediction from hidden states. LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization (Liu et al. 2025) seems to already do much of the work this paper claims to do. It would be good to discuss this work and explain if your work adds anything substantial to this paper.

  2. I think the LLM rephrasing in parts reduces readability. I found it more difficult than it should be to go through.

  3. When building correctness probes and using those as analysis tools, connections to calibration seem obvious and important. These are entirely missing.

  4. Experiments are completed on very small models, relative to the computational resources available (some experiments where run on "eight NVIDIA A800-SXM4-80GB GPUs"), so you can easily run MUCH larger models, perhaps even up to 100Bish scale.

  5. Minor: some missing literature that should be discussed. There's work on correctness prediction from hidden states and as a result, using model internals for calibration. Here are some of them. InternalInspector I2 (Beigi et al. 2024); MICE for CATs (Subramani et al. 2025); The ACUTE protocol (Subramani et al. 2026), How llms detect and correct their own errors: The role of internal confidence signals (Kumaran et al. 2026).

Questions for authors
  1. How is S_i constructed from C_i?
  2. Did you consider running other probes? Probing is well-studied and there are many drawbacks to each kind of probe.
  3. There's often an inflection in capabilities at higher parameter counts. Did you consider running larger models, perhaps including ones with other architectures (e.g. MoEs?).
  4. How does your paper differ from Model internals-based answer attribution for trustworthy retrieval-augmented generation (Qi et al. 2024), which you cite?
  5. When you patch in the embedding for when it gets the answer correct, shouldn't you often just get the correct answer? What does this prove/show?
Reviewer_C2xeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper explores the problem of why language models give incorrect answers even when sufficient and correct evidence is available in RAG. The paper studies how the model's response to the same question differs with supporting evidence alone versus evidence with additional context and utilizes Logit Lens, linear probes, and state patching to understand the difference inside the model. The conclusion drawn from the analysis is that answer preferences diverge in the middle-to-late layers, and pre-generation hidden states indicate whether additional context will help or harm the answer. State patching, where states from correct-answer runs are inserted into incorrect-answer runs, fixes certain failures, whereas reverse patching turns correct answers into incorrect ones.

Critical strengths
  1. This paper explores a problem that is worth studying. While much prior work focuses on either retrieval errors or conflicting knowledge, this paper specifically looks at instances when there is enough evidence for the model, but it gives wrong answers. Furthermore, the work explores how internal answer preferences change and whether they affect the final output.
  2. Causal interventions performed in this paper are very good. Not only does the paper use probes to analyze hidden states, but it also uses bidirectional patching to evaluate if the generated answers actually change. Appendices H and I provide controls at different layers and token positions, as well as random-source controls.
  3. Cross-dataset evaluation is done well. The probe parameters and the locations of intervention are chosen on the HotpotQA dataset and directly used for MS MARCO and NQ without further adjustment. Table 1 clearly illustrates the effects of repair and damage across all four models and dataset splits.
Critical weaknesses

The patching experiments of Sections 4.3–4.4 confirm that the pre-generation states causally influence the final answers, but they do not provide much insight into the reason why evidence utilization has failed. The patched state is obtained from a run where the output is already known to be the correct answer, so this state may already contain information about the correct answer. The repair effect could be primarily due to transferring the correct answer representation, rather than explaining why the model failed to use the evidence in the first place.

Questions for authors
  1. How would it be possible to separate the transfer of information about correct answers from changes in evidence utilization during patching? For instance, would patching a hidden state from a different question with the same correct answer lead to comparable repair effects?
  2. In Table 8, the question-end and pre-generation probes of Qwen2.5-7B have almost identical AUROC (0.8286 versus 0.8305), while in Table 13, question-end patching hardly ever modifies the final answer. Is this primarily caused by layer selection, or do these states play different roles? What would happen if question-end patching were applied at the layer chosen by its own probe?
  3. In Rescue cases, what makes the addition of context transform an incorrect answer into a correct one? Would it be possible for the paper to provide some examples illustrating whether there are any helpful hints in the added text or whether the modification just influences the model's answer preference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.