acceptodds
Under review as a conference paper at ICLR 2027

Do Context-Reliance Directions Transfer to Hallucination Detection in RAG?

Abstract

Representation engineering identifies directions in transformer activation space that can be used to measure or influence model behavior at inference time. In Retrieval-Augmented Generation (RAG), such a direction would be an attractive hallucination signal: it is intrinsic, costs one dot product per token, and uses information that post-hoc detectors discard by treating the generator as a black box. We ask whether context reliance is encoded as a single, transferable linear direction. We construct 15 contrastive datasets spanning hallucination-prone scenarios — 13 synthetic A/B templates, an external counterfactual QA corpus, and a template-free dialog construction — and extract one Context-Source Vector (CSV) per dataset from Llama-3.2-1B-Instruct and Llama-3.1-8B-Instruct. Each CSV is then scored on RAGTruth against two controls: 100 random directions (lower control) and a CSV fitted in-domain on RAGTruth (upper control). The answer is negative in an informative way. CSVs do transfer above chance, but only modestly: best-layer AUROC is 0.56–0.68 (mean 0.62) for the 1B model and 0.59–0.70 (mean 0.65) for the 8B model, against random-direction means of 0.57 and 0.58 and in-domain upper controls of 0.77 and 0.84. Crucially, CSVs that are nearly orthogonal or even negatively aligned with one another achieve indistinguishable detection performance, yet often peak at entirely different layers. Effective context-reliance signals are therefore not uniquely represented by a single activation-space direction; instead, the extracted directions appear to capture distinct, partly confounded features that reside at different depths in the network and are similarly predictive downstream. The larger model encodes more of these features, yielding a small but consistent improvement in transfer.

Primary area:
interpretability and explainable AI
Keywords:
Retrieval-Augmented Generation, Steering Vectors, Mechanistic Interpretability, Explainable AI, Trustworthy AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ruNNOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks whether context reliance in retrieval augmented generation is encoded as a single, transferable linear direction. The authors build 15 contrastive datasets: 13 synthetic A/B template sets, a ConFiQA derived counterfactual QA set, and a template free dialog construction. They also build a RAGTruth derived set in which an LLM rewrites each annotated hallucinated span into a minimal source supported replacement. For each dataset and each layer of Llama 3.2 1B Instruct and Llama 3.1 8B Instruct they compute a Context Source Vector (CSV) as the mean activation difference between the context reliant and parametric member of each pair (contrastive activation addition). Two controls are used: 100 random directions (lower) and a CSV fitted on RAGTruth itself (upper). In domain probes separate the synthetic classes almost perfectly (LDA up to 0.98 at middle layers of 8B), but best layer transfer AUROC is only 0.56 to 0.68 (mean 0.62) for 1B and 0.59 to 0.70 (mean 0.65) for 8B, against random means of 0.57 and 0.58 (95th percentile 0.634 and 0.645) and upper controls of 0.77 and 0.84. The authors report that vectors that are nearly orthogonal or negatively aligned reach similar best layer AUROC, often at different layers, and conclude that context reliance is not captured by one canonical direction but by several partly confounded features at different depths.

Critical strengths

The evaluation sets a better bar than most steering vector detector papers. Scoring each vector against random directions whose best layer is chosen in the same way, and against an in domain upper control, shows that many "best layer" results are explained by layer selection (the random 95th percentile at 1B, 0.634, is close to the best template, 0.681), and that ten of the 15 templates peak at layers 0 to 3 where the signal is probably lexical. The paper also shows directly that near perfect in domain probe accuracy does not predict transfer (probe F1 near 1.0 while transfer AUROC is 0.59 to 0.69). Recommending these two controls as a minimum bar is useful for the area.

Critical weaknesses

I think the transfer test and controls are too weak to carry the weight placed on them. The differences between vectors are small relative to the noise (best layer AUROCs of 0.59 to 0.70 against a random 95th percentile of 0.645 at 8B on a test set of 1,166 pairs), and no confidence intervals or bootstrap are given for any vector, yet the paper interprets orderings among templates and layers. The "RAGTruth pair" task also differs from the stated goal: the detector scores the final token of a human annotated hallucinated span against the final token of an LLM rewritten correction, so a score could track surprisal, fluency or authorship of the span rather than reliance on context, and no simple baseline (token log probability, entropy, or the paper's own probe) is reported on the same pairs to show how hard the task is. The upper control is a mean difference vector on RAGTruth (0.77 and 0.84), not a trained probe, and the closest prior method (O'Neill et al., a linear probe on an observer model) and white box detectors such as ReDeEP are not compared on the same data, so the reader cannot tell if 0.62 to 0.65 is poor for this family or for the task. No causal test (steering or ablation) is run, though the paper itself notes that comparable vectors can degrade output in both directions. Only two Llama models are tested.

Questions for authors

1/ Please recompute the cosine geometry at each vector's best AUROC layer, and with sign aligned, and report whether the "orthogonal but equally good" pairs survive.

2/What are bootstrap confidence intervals for the best layer and mid layer AUROCs of each vector on the 1,166 RAGTruth test pairs?

3/How was the layer for "Best" chosen for fairness, given the test labels are used to pick it, for the random controls as well as the CSVs?

2 comments

Thank you for the feedback and for recognizing the value of the controls and the negative transfer results.

  • Layer selection and uncertainty: We selected the best layer on the RAGTruth test set for both CSVs and random directions. This makes the comparison useful as an exploratory reference, but does not provide an unbiased estimate for a prespecified layer. We will make this explicit and avoid interpreting small differences as reliable rankings. The random-direction distribution is also distinct from sampling uncertainty; it shows how well random directions perform under the same search, not confidence intervals for the CSVs.
  • Evaluation and controls: We agree that the paired task does not directly establish the causal information source used during generation. Surprisal, fluency, and rewrite artifacts are possible confounds, and simple baselines would help distinguish them. We will correct the source-attribution framing and rename the “upper control” as an in-domain reference. We consider the results mostly negative, but informative—not evidence for a competitive hallucination detector.
  • Geometry: We agree the geometric conclusion was too strong. Comparing vectors at their respective best layers would still mix representations from different depths; the relevant comparison is geometry and performance at the same layer. We will narrow the conclusion to sensitivity to contrastive construction, rather than ruling out a shared direction. Early-layer peaks suggest possible lexical confounds, but do not establish them.

We do not have the capacity to run further experiments for this rebuttal. We will acknowledge the missing uncertainty estimates, baselines, and causal tests, and keep the conclusions within what the existing experiments establish.

Thank you for the clear response, I will keep my score.

Reviewer_SXJpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether context reliance in RAG, i.e., whether a model relies on retrieved context or parametric knowledge, can be represented by a single linear direction in its activation space and used for hallucination detection.

The authors construct 15 contrastive datasets, including 13 synthetic templates, ConFiQA, and a template-free dialog dataset. They extract Context-Source Vectors (CSVs) by averaging activation differences between context-reliant and parametric examples across different layers of Llama-3.2-1B-Instruct and Llama-3.1-8B-Instruct.

These vectors are evaluated on RAGTruth using AUROC, with random directions and an in-domain RAGTruth vector serving as controls. The reported best-layer AUROC ranges from 0.56 to 0.68 for the 1B model and 0.59 to 0.70 for the 8B model, compared with in-domain controls of 0.77 and 0.84, respectively.

The authors observe that vectors extracted from different datasets can be nearly orthogonal or negatively aligned while achieving similar best-layer detection performance. They interpret this as evidence that context reliance is not captured by one stable linear direction, but rather by multiple partly confounded features at different network depths.

Critical strengths

1. Appropriate random-direction controls.

The authors evaluate 100 random directions using the same best-layer selection procedure as the extracted CSVs. This is a useful experimental control because it demonstrates that layer selection alone can raise the apparent AUROC of random directions above 0.5. The random-direction results provide an important reference for interpreting the relatively modest transfer performance of the proposed vectors.

2. Useful empirical investigation of transferability. The paper investigates context-reliance vectors across 15 different data constructions and two model sizes. The observation that strong separation on synthetic data does not necessarily translate into successful hallucination detection on RAGTruth is useful. I also appreciate that the authors explicitly report the limited transferability of their approach rather than presenting the results as a successful hallucination detector.

Critical weaknesses

1. The main geometric conclusion is insufficiently supported, and the reported transfer performance is affected by test-set-based layer selection.

My first major concern is that the geometric analysis and the performance comparison are not conducted at matched layers.

In Section 4.4, the authors compare cosine similarities between CSVs at a fixed layer (layer 8 for the 1B model and layer 16 for the 8B model). However, their claim that orthogonal or negatively aligned vectors achieve similar detection performance relies on each vector’s best AUROC, which is often obtained at a different layer.

When examining performance at the same layer, the results can look quite different. For example, in the 8B model, prompt-7 and prompt-9 are strongly negatively aligned with several other vectors at layer 16. At that same layer, their AUROCs are 0.439 and 0.408, while several of the oppositely aligned vectors achieve AUROCs above 0.6. More generally, all 11 vector pairs with cosine similarity below -0.5 at layer 16 have AUROCs on opposite sides of 0.5 at that layer. This suggests that vectors pointing in opposite directions do not necessarily exhibit similar predictive behavior when evaluated at the same network depth.

The observed pattern is also consistent with an alternative explanation; a shared underlying context-reliance direction combined with template-specific confounding features. For example, two extracted vectors can be orthogonal while both containing a common component representing context reliance (e.g., [-1,1] and [1,1] both have the component of [0,1]: [-1,0]+[0,1] and [1,0]+[0,1]). Consequently, the geometric differences between CSVs do not establish that the underlying concept lacks a shared linear representation. The experiments demonstrate sensitivity to prompt construction, but they do not sufficiently distinguish between multiple underlying features and a shared feature contaminated by other signals.

A related concern is the selection of the best layer. According to Section 3.3, the authors evaluate each CSV across all layers on the RAGTruth test set and report the highest AUROC. This introduces test-set selection bias, since the reported best-layer performance is selected using the same data on which it is evaluated. Although the authors also report fixed-middle-layer results, these are considerably weaker. At the fixed middle layer, only 2 of the 15 vectors for the 1B model and 5 of the 15 for the 8B model exceed the corresponding random-direction 95th percentile. A more appropriate evaluation would select the layer using a validation set and then report its performance on a separate test set.

The paper also does not report confidence intervals for the AUROC results, making it difficult to assess whether differences between vectors are statistically meaningful.

Finally, while the random-direction and in-domain CSV controls are useful, comparisons with existing hallucination-detection methods or transferred linear probes would help establish the practical significance of the findings.

Together, these issues substantially weaken the evidence supporting the paper’s main conclusions about transferability and the geometry of context-reliance representations.

2. The experiments do not directly establish context reliance during generation, and the novelty is insufficiently positioned relative to prior work.

My second major concern is the relationship between what the paper claims to measure and what its experiments actually evaluate.

Hypothesis 1 concerns whether the model internally represents whether a token was derived from contextual information or parametric knowledge. However, in the synthetic datasets, the answer tokens are appended rather than generated by the evaluated models. Similarly, the RAGTruth responses were originally generated by other models, including GPT-3.5, GPT-4, Mistral-7B, and Llama-2. The evaluated Llama-3 models process these existing responses and their corrected versions.

Therefore, the experiments primarily examine whether the activations of a model processing externally provided text can distinguish context-supported from unsupported responses. This is a meaningful observer-based setting, but it does not directly establish whether the extracted directions identify the information source used during the evaluated model’s own generation. The distinction matters because the paper motivates CSVs as intrinsic signals that can monitor context reliance during inference. I would expect either a clearer observer-based framing or additional experiments on responses generated by the evaluated models themselves.

Another concern is the construction of the synthetic datasets. In many templates, context-reliant answers are associated with fictional or counterfactual information, while parametric answers correspond to familiar real-world facts. This introduces a possible confound between context reliance and factuality or entity familiarity. It is therefore unclear whether the extracted vectors primarily encode information-source reliance or other correlated properties of the examples. The paper would benefit from stronger controls that separate context reliance from factual correctness, familiarity, and prompt-specific features.

I also have concerns about the novelty and positioning of the work. Several closely related studies are missing from the Related Work section:

  • Minder et al., “Controllable Context Sensitivity and the Knob Behind It” (ICLR 2025) investigate a one-dimensional subspace controlling the model’s reliance on context versus prior knowledge. This is particularly relevant because it examines a closely related representation hypothesis and reports evidence for a shared direction in the Llama-3.1 model family.
  • “Probing Language Models on Their Knowledge Source” (BlackboxNLP 2024) investigates whether internal activations reveal which knowledge source a language model relies on, overlapping with the motivation and layer-wise analysis in this paper.
  • “Understanding (Un)Reliability of Steering Vectors in Language Models” (2025) investigates how steering vectors constructed from different prompts can have substantially different directions while producing similar effects. This is closely related to the geometric observation presented as a main contribution in Section 4.4.
  • ContextFocus (2026) investigates activation steering for contextual faithfulness on ConFiQA, the same corpus used in this paper, and should be discussed as concurrent work.

These studies do not necessarily invalidate the contribution of the present paper, but they substantially overlap with its motivation, methodology, or main empirical observations. In particular, the observation that steering vectors extracted under different conditions can have different orientations is not entirely new. The most distinctive contribution therefore appears to be the systematic evaluation of these directions on RAGTruth. However, the methodological concerns described above limit the strength of that contribution.

Overall, I find the research question interesting, but I am not convinced that the current experimental evidence supports the proposed interpretation or establishes a sufficiently strong contribution relative to existing work.

Questions for authors
  1. Matched-layer geometric analysis. Could the authors report pairwise cosine similarities across layers and compare vector geometry with detection performance at the same layer? In particular, how does the interpretation change when negatively aligned vectors are compared using their AUROCs at the layer where cosine similarity is measured? It would also be useful to report each CSV’s similarity to the in-domain RAGTruth vector.
  2. Validation-based layer selection. Could the authors report RAGTruth test AUROC when the evaluation layer is selected using the RAGTruth validation split rather than the test split? This would help establish whether the reported transfer performance generalizes beyond test-set-based layer selection.
  3. Context reliance during generation. Is Hypothesis 1 intended to describe the information source used by the generating model or the representations of an observer model processing an existing response? If the former, could the authors provide evidence using responses generated by the evaluated Llama-3 models themselves?
  4. Relationship to previous work. How do the findings relate to Minder et al. (ICLR 2025), who identify a one-dimensional subspace associated with context versus prior knowledge? Could the authors clarify what is conceptually or methodologically new beyond the evaluation on RAGTruth, particularly given prior findings about prompt-dependent steering-vector directions?
  5. Contrastive-pair construction. Section 3.2 states that the correct answer letter is fixed within a contrastive pair, but Table 3 includes examples with different correct letters. Could the authors clarify which construction was used in the actual experiments? This is important because answer-token identity may influence the extracted activation differences.
  6. Reproducibility and dataset construction. Could the authors clarify the apparent discrepancy between the pooled vector results in Section 4.1 and Figure 5? Additionally, which model was used to generate the corrected RAGTruth responses, and were these corrections independently validated?
1 comment

Thank you for the detailed and constructive feedback. We agree with several concerns and will narrow our claims accordingly.

  • Geometric interpretation: The reviewer is correct that middle-layer cosine similarities cannot be directly combined with each vector’s best-layer AUROC to establish similar predictive behavior at the same depth. The shared-direction-plus-confounds explanation is also consistent with our results. We will describe the finding as sensitivity to contrastive construction, rather than evidence ruling out a shared linear representation.
  • Layer selection and transfer: We searched across layers as part of our exploratory analysis, since the location of a transferable signal was unknown. Random directions underwent the same best-layer search; on 8B, 9 of 16 reported cross-dataset vectors exceeded their empirical 95th percentile. We believe this remains informative, but agree it is neither a multiple-testing-adjusted significance result nor an unbiased estimate for a prespecified layer. We will make this distinction explicit. We consider the results mostly negative, but of interest—not evidence for a competitive hallucination detector.
  • What the labels measure: Section 3.2 indeed contained an error. The synthetic labels describe which source the logically correct answer is intended to depend on, not the causal source demonstrably used by the evaluated model. Likewise, the RAGTruth-derived evaluation measures discrimination between hallucinated spans and context-supported rewrites in an observer setting. We will correct the framing and explicitly acknowledge factuality, familiarity, and rewrite artifacts as potential confounds.
  • Positioning and reproducibility: We agree the cited work should be discussed more directly. Our contribution is the systematic transfer evaluation across constructions on RAGTruth, not the first observation of prompt-dependent vector geometry. The appendix examples were outdated and predated the change to fixed answer letters in the extraction data; we will replace them. We will also clarify the pooled-result settings, correction-generation model, and validation procedure.

We do not have the capacity to run further experiments for this rebuttal. Our revisions will focus on correcting the description and keeping the conclusions within what the existing experiments establish.

Reviewer_zb5cOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether reliance on retrieved context versus parametric memory corresponds to a single, transferable linear direction in a transformer’s residual stream. The authors build 15 contrastive datasets: 13 synthetic A/B forced-choice templates (direct extraction, multi-hop, in-context decoys), an A/B version of ConFiQA, and a template-free dialog construction. In each pair, the correct answer comes from the context in one prompt and from world knowledge in the other. Following contrastive activation addition, they take the mean activation difference at the answer-letter position as a per-layer Context-Source Vector (CSV) for Llama-3.2-1B-Instruct and Llama-3.1-8B-Instruct. To test transfer, they pair each annotated hallucinated span in RAGTruth with an LLM-written, context-supported minimal rewrite, replay both versions through Llama, and score the projection of the span’s last token onto each CSV. They compare against 100 random directions (lower control) and a CSV fitted on the RAGTruth training split (upper control). Best-layer AUROC averages 0.62 for 1B and 0.65 for 8B, against random-direction means of about 0.57–0.58 and upper controls of 0.77 and 0.84. The authors also find that some CSVs are nearly orthogonal or negatively aligned with each other yet transfer about equally well, and conclude that context reliance is not carried by a single direction.

Critical strengths
  1. The question matters, and the authors frame it as a falsifiable hypothesis rather than overselling a detector.

  2. The letter-controlled LDA analysis in Section 4.1 is a nice touch. It shows the answer-letter identity dominates the activations and that the source distinction survives once the letter is held fixed.

  3. The authors report a mostly negative result honestly, and they plan to release all datasets, vectors, and scripts.

Critical weaknesses
  1. The RAGTruth evaluation doesn’t measure what the paper claims to study. Section 3.2 stresses that the labels describe how the model computed an answer, not how the text reads. But the RAGTruth responses come from other models, so when the authors replay them through Llama, Llama never generated those tokens, and there is no “source” to recover. The “context-reliant” member of each pair is also an LLM-written rewrite. So the evaluation really asks whether a direction can tell an original hallucinated span from its rewritten correction, a string-level paired discrimination task, which is exactly what the paper says its labels are not. The rewrites may also differ in style or length, which adds another confound.

  2. In the contrastive data, context reliance coincides with factual falsehood. In the forced-conflict templates (Semantic, ConFiQA, Templates 7–11), the context answer is counterfactual (“the capital of France is Berlin”) and the parametric answer is the true fact. Templates 1–4 contrast fictional entities with real ones. The CSVs could therefore encode a known truth direction (Marks & Tegmark) or a fictional-versus-real lexical feature rather than context reliance. On RAGTruth, the corrected span is usually both context-supported and true, so a truth direction would explain the modest transfer equally well. The paper has no control that separates the two, such as pairs where the context answer is true and the parametric answer is false.

  3. The statistics don’t support “above chance.” “Best” takes the maximum AUROC over all layers on the RAGTruth test split, so the authors select layers on the test set. They then compare the best of 16 vectors against the 95th percentile of a single random direction, which invites a multiple-comparisons problem: some vectors would clear that bar even if all of them were random. On 8B, 7 of the 16 vectors don’t exceed the random 95th percentile of 0.645; on 1B, 11 of 16 don’t exceed 0.634. At the fixed middle layer, several vectors fall below 0.5 (e.g., 0.384, 0.408, 0.439). The paper uses one split, with no confidence intervals or seeds. The “upper control” is also just a mean-difference vector rather than a trained probe on RAGTruth, so it isn’t really an upper bound.

  4. The central geometric argument doesn’t hold up. The authors compute the cosine geometry at a fixed middle layer but compare transfer at each vector’s own best layer, so the two analyses don’t line up. At the middle layer of 8B, prompt 7 and prompt 9 point almost opposite to most other vectors, and their AUROCs there are below chance (0.439 and 0.408) while the others sit around 0.6. That pattern is what one would expect from a single axis with flipped sign, which supports the single-direction hypothesis rather than refuting it. More broadly, most vectors perform near the random baseline, and several near-random directions performing “similarly” doesn’t show that multiple valid features exist.

  5. The scope is narrow and the presentation needs work. The paper tests only two Llama models, compares against no existing detector (e.g., the O’Neill et al. probe, ReDeEP, Lookback Lens), and runs no causal interventions. The writing has several inconsistencies: Hypothesis 1 reappears in a different form around line 432; the dataset counts shift between 13 templates, 11 pooled templates, and 15 datasets; prompts 5 and 6 are missing because of merged lists; the exact scoring position for RAGTruth (“the last token of the prompts”) is unclear; and the MDS plot in Figure 2 is hard to read.

Questions for authors
  1. Since Llama didn’t generate the RAGTruth responses, what does “context source” mean for a replayed token? Have you evaluated on Llama’s own generations with hallucination labels?

  2. Can you add contrastive pairs where the context answer is true and the parametric answer is false, to separate context reliance from a truth direction?

  3. If you select the layer on a validation split and report bootstrap confidence intervals with a multiple-comparison correction, which CSVs still beat the random baseline?

2 comments

Thank you for the detailed feedback. We agree with several criticisms and will clarify our claims accordingly.

  • Label interpretation: Section 3.2 contained an error. We did not intend to measure the actual causal source of a token during inference. Our synthetic labels indicate which source the logically correct answer should depend on, not which source the model actually used. Similarly, the RAGTruth-derived evaluation tests discrimination between hallucinated spans and context-supported rewrites, not causal source attribution. We will correct this description and explicitly acknowledge potential shortcuts, factuality differences, and rewrite artifacts.
  • Layer selection and random controls: Searching across layers was part of our exploratory question, since we did not know where a transferable signal would appear. The random directions underwent the same best-layer selection; on 8B, 9 of the 16 reported cross-dataset vectors exceed their empirical 95th percentile. We think this supports the description “above chance, but only modestly.” However, we agree that this is neither a multiple-testing-adjusted significance result nor an unbiased estimate for a prespecified layer. We will make that distinction explicit. We consider the results mostly negative, but still informative.
  • Geometry and remaining limitations: We agree that middle-layer cosine similarities cannot be directly combined with best-layer AUROCs to rule out a shared direction. We will narrow our conclusion to variability in the extracted directions and limited transfer. We will also rename the in-domain “upper control” as an in-domain reference and clarify the dataset counts. The appendix examples were outdated, as they predated the change to fixed answer letters in the extraction data. We will replace them.

We do not have the capacity to run further experiments for this rebuttal. Our revisions will focus on correcting the description and keeping the conclusions within what the existing experiments establish.

Reviewer_zb5cOpenReview ↗

Thank you for the candid response

Thank you for the candid response and the planned clarifications. After considering the rebuttal, I will keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.