acceptodds
Under review as a conference paper at ICLR 2027

The Attribution Blind Spot: Layerwise Trajectory Diagnostics for Source Reliance in Retrieval-Augmented Language Models

Abstract

A retrieval-augmented model can match a document without relying on it. Controlled knowledge conflicts make source choice observable and let us ask a second question that prediction alone cannot answer: which internal-state properties define useful intervention directions? We study paired hidden-state changes with Latent Trajectory Shift (LTS), a signed projection onto a training-fitted first principal component (PC1), and keep verified training exposure separate from behavioral source choice. Across the evaluated conflicts, state-change magnitude is often the stronger predictor, whereas signed PC1 is the stronger selective controller: equal-norm interventions change source preference while better preserving non-target behavior, and the frozen direction transfers across the tested datasets and aligned model pairs. A same-system OLMo study further combines positive choice and control results with inconclusive exposure detection at the achieved power. The central result is a separation: representations that diagnose what a model will choose need not be the representations that best control that choice.

Primary area:
interpretability and explainable AI
Keywords:
retrieval-augmented generation, source reliance, knowledge conflicts, representation analysis, activation steering, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aneoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the distinction between training exposure, behavioral source choice, and intervention-based control in retrieval-augmented language models. It introduces Latent Trajectory Shift (LTS), a layerwise projection of paired hidden-state differences onto a first principal component fitted exclusively on training data. Controlled conflicts between contextual and parametric answers provide the behavioral endpoint.

Verified-exposure experiments on Pythia and OLMo do not establish reliable exposure detection by the primary LTS feature. For source-choice prediction, state-difference magnitude is often stronger than signed PC1 projection. However, the reported activation interventions favor PC1 for changing source preference while preserving congruent and no-context behavior. The study includes matched-norm controls, candidate-gradient steering, dataset transfer, aligned cross-model transport, and a paragraph-level OLMo experiment.

Critical strengths
  1. The paper makes a useful distinction between exposure detection, behavioral source choice, and controllability. It avoids interpreting answer agreement as evidence of causal source use and does not turn inconclusive membership results into an equivalence claim. The finding that magnitude can outperform PC1 for prediction while PC1 performs better under the tested intervention operator is an informative empirical question for representation analysis.

  2. The intervention evaluation goes beyond demonstrating a change in a candidate score. It includes bidirectional updates, held-out direction fitting, norm-matched controls, discrete answer changes, non-target preservation, and transfer. The candidate-gradient comparison and paragraph-level experiment are particularly relevant to assessing whether the effect extends beyond a local short-answer margin change. These components provide a stronger basis for evaluating a controller than predictive probe accuracy alone.

Critical weaknesses
  1. Several central results cannot currently be reconciled from the reported endpoint definitions. For example, Table 12 gives a Llama middle-block, toward-parametric PC1 effect of 0.314 at unit dose. In Table 15, the corresponding candidate-gradient effect is 0.1407 and the paired LTS-minus-candidate effect is 0.2536, implying an LTS effect of 0.3943. Both sections describe the same 120-item within-dataset setting, but any difference in dose, score normalization, policy aggregation, or cohort is not clearly specified. Table 20 also reports a contextual-direction “target success” of 0.243 over 779 Llama items, although only 43 baseline answers are parametric. This cannot be interpreted as a full-cohort parametric-to-contextual flip rate; a different endpoint would need an explicit definition. These ambiguities affect the central control evidence rather than peripheral reporting.

  2. The controls do not yet establish why PC1 is the preferred learned direction. Random, orthogonal, shuffled-layer, and radial updates are useful controls, and the candidate-gradient baseline is valuable, but the paper does not clearly report the most direct alternative: the normalized mean paired displacement, which is already computed as g_l in Equation (6). A mean-difference steering baseline, following the principle of Contrastive Activation Addition (Rimsky et al., ACL 2024), could be evaluated under exactly the same additive operator, position, layer blocks, and norm budget. A source-choice probe direction would also connect the prediction-versus-control comparison more directly. Without such comparisons, the evidence supports superiority over the tested controls but leaves the specific value of the PC1 construction insufficiently established.

Questions for authors
  1. Please explain the relationship between Tables 12 and 15. For Llama's middle-block toward-parametric condition, why does the candidate effect plus the paired LTS advantage equal 0.3943 rather than the 0.314 reported for PC1? Please identify the exact dose, candidate-score definition, cohort, and aggregation rule for each table.

  2. What precisely is “target success” in Table 20? If it is a flip rate or an increase in target-answer frequency, how is the contextual-direction value of 0.243 compatible with only 43 initially parametric answers among 779 items? If it instead uses margin-based eligibility, a soft score, or a conditional denominator, please provide its formula and distinguish it from generated-answer flips. A clean-versus-steered transition table would resolve this directly.

  3. How does PC1 compare with the normalized mean congruent-minus-conflict displacement under the identical operator and preservation criteria? Please also give the exact mathematical definitions of the signed and unsigned radial updates, including their reference vector and the information available at intervention time. These details are necessary to interpret the claimed separation between a magnitude diagnostic and a directional controller.

  4. How were the 500 members and 500 nonmembers established for OLMo-2-1124-7B-Instruct? Please specify the training-data provenance, deduplication and contamination checks, coverage of relevant training stages, and how these examples were converted into the shared conflict-choice cohort. The Pythia checkpoint construction is substantially clearer than the corresponding OLMo verification.

2 comments

Dear Reviewer aneo,

Thank you again for your time and constructive feedback.

With 1 day remaining in the author–reviewer discussion period, we would greatly appreciate it if you could take a moment to review our rebuttal and revised manuscript. We have addressed your concerns through additional analyses, experiments, and clarifications.

Thank you for your time and consideration.

Best regards,

The Authors

Dear Reviewer aneo,

Thank you again for your time and constructive feedback.

With 1 day remaining in the author–reviewer discussion period, we would greatly appreciate it if you could take a moment to review our rebuttal and revised manuscript. We have addressed your concerns through additional analyses, experiments, and clarifications.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_AnvrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines how a model’s prior knowledge influences retrieval outcomes. To study this, the paper conducts experiments that analyze changes in the model’s hidden states. The findings show that the magnitude of the state change is the strongest predictor of which source the model follows (prior knowledge or presented knowledge) during retrieval.

Critical strengths

The paper examines an interesting question about prior versus presented knowledge in retrieval augmentation.

Critical weaknesses

My biggest concern with this paper is the writing: I found the presentation of the material unclear. Specific issues include:

  1. Vague references: For example, line 179 refers readers to the appendix but does not specify where in the appendix.
  2. Using terms before defining them: For example, Section 2.1 begins with “We need two different objects from the same paired runs: a scalar feature for prediction and a direction that can be applied as an intervention,” but there is no task formalization beforehand to give readers context.
  3. Structure: While the paper does divide its content into subsections, the content within each subsection is scattered. For example, Section 3.1 presents the datasets first, then the models, and then returns to the datasets.

While the paper examines an interesting topic, I believe that clear presentation is the first step in ensuring that the paper receives helpful and meaningful feedback from reviewers. Therefore, I strongly encourage the authors to revise the writing so that they can strengthen the paper for future submissions.

Questions for authors

None

1 comment

Dear Reviewer Anvr,

Thank you again for your time and constructive feedback.

With 1 day remaining in the author–reviewer discussion period, we would greatly appreciate it if you could take a moment to review our rebuttal and revised manuscript. We have addressed your concerns through additional analyses, experiments, and clarifications.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_Tm17OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors show that different features are required for predicting and for controlling the compliance of a language model with the provided context. Specifically, by designing a paired dataset where each item consists of a set of (query, context, context-aligned answer) tuples, they have revealed that the contrast between hidden representations of prompts with two different contexts (e.g., no-context vs. context, congruent context vs. conflicting context) is critical for both prediction and control of a language model's reliance on a given context. While experimental results show that the magnitude of the contrast itself is critical for prediction tasks, the first principal component (PC1) of the contrast between hidden representations of prompts with two different contexts has been found to be effective for controlling the retrieval-augmented language model's generation. The authors have further shown the transferability of the PC1 direction to other datasets and models.

Critical strengths
  • Under the careful experimental design with reasonable ablation studies, the authors have provided statistically valid experimental results. In particular, ablation studies on six different types of contexts and seven different types of intervention directions make it clear for readers to understand the efficacy of the proposed method.

  • The transferability of the PC1 direction to other datasets and models is interesting. If further verified by additional experiments, the finding may provide practical guidelines for controlling the effect of a given context on output generation.

Critical weaknesses
  • The size of the dataset used to empirically verify the efficacy of PC1 for output control is relatively small. (If possible,) additional experiments with larger datasets would further solidify the corresponding claims.

  • The PC1 direction with its sign determined (vℓv_\ell) is used (Line 119) before it is defined in Equation (7).

  • An ablation study on methods such as partial least squares may be interesting, as principal component methods are unaware of the output labels. It would be interesting to see whether such label-aware dimension reduction methods can also transfer to different datasets and models.

Questions for authors

See Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.