Why Do Language Models Answer Incorrectly Despite Sufficient Evidence? A Mechanistic Study of Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) can still produce incorrect answers even when sufficient and correct evidence is available. This work studies failures of evidence utilization in the absence of explicit knowledge conflicts. We compare responses to the same question under two conditions: one containing only supporting evidence and the other containing the full context while retaining the same evidence. We combine layer-wise answer scoring with linear probes to track changes in internal representations, and further use bidirectional residual stream patching to test whether the identified states causally affect model responses. Experiments on four open-weight instruction-tuned models across three question-answering benchmarks show that pre-generation states in the middle-to-late layers distinguish answer regression from answer rescue induced by context expansion. When the question and input evidence remain unchanged, patching in the corresponding state from the same question under a correct-answer condition repairs a subset of incorrect responses, whereas patching in the reverse direction degrades a subset of correct responses. After determining and freezing the probe parameters and intervention locations on HotpotQA, we reproduce these effects on MS MARCO and Natural Questions. Across the two types of transfer samples, the repair rates range from 25.4% to 63.7% and from 44.2% to 81.8%, respectively. Equal-norm random-source controls further show that the intervention effect on the overall residual stream depends on the source state, while the effects associated with the attention and multilayer perceptron (MLP) modules vary across models. These findings provide causal intervention evidence for understanding why models sometimes fail to convert correct evidence into correct answers, and provide a basis for studying internal-state-based correction.