acceptodds
Under review as a conference paper at ICLR 2027

Context or Knowledge: How LLMs Attribute Their Own Answers

Abstract

Language models can answer a question from a supplied document or from their parametric knowledge. When later asked where the answer came from, what determines their source report? We study this by intervening on the model's key–value (KV) cache after the answer is generated and before the source question is asked. On a controlled factual task with four models (7B–32B), changing or removing the document's cache never changes the report. In contrast, transplanting the question-and-answer (Q+A) cache between runs with identical question and answer tokens transfers it: every eligible report follows the donor, while the recipient's document is unchanged. On the two smaller models, steering Q+A values along directions fitted before any source question reverses 98.9–100% of reports across four document conditions at a fixed strength, while norm-matched random directions flip at most 1.1%. The components of keys and values along fitted directions carry the transplant effect: copying them alone nearly reproduces the report change; holding them fixed nearly prevents it. Applied during answer generation, the same value directions also change which source supplies the answer, though more weakly and depending on the prompt. Source reporting in this setting thus depends causally on cached state produced while answering. We do not establish general reporting accuracy or a complete interpretation of this state.

Primary area:
interpretability and explainable AI
Keywords:
explanation faithfulness, knowledge tracing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d6pVOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

While lots of prior work focuses on model behavior on knowledge conflict between retrieved text and models' (LLM) internal knoweldge, this paper answers to the question whether which knowledge source does the model refer to, by altering (plugging in and out certain parts) the KV-cache. They provide three hypotheses for this aspect (document re-reading, parametric knowledge comparison, and cached trace), where they verify the hypotheses across 5 experiments (document drop/replace, Q+A KV transplant, qa value steer, projection intervention, and answer source steering). The author shows that LLMs often relies on KV cache made for QA and that these KV cache can affect the model's answer whether which source information the model has referred to.

Critical strengths
  • The idea of using KV cache for the verification of the source report is interesting, where a better analyses with justifications and writing is needed.
Critical weaknesses
  • I think the biggest weakness of the paper is readability as a AI research paper. I believe that re-writing the paper with this in mind is a must for the paper to be published. (1) Detailed related work section in the appendix is doable, however, a bit more in detail how this paper's findings differ from other relevant work should be more addressed in the paper writing, at least as a detailed paragraph; the paper consists of an extremely small number of citations in the main writing. (2) The paper writing seems more of a story telling, which in my opinion is less-favored for a research paper. The writing itself might be easy to follow, but in detail hard to track which the authors are trying to show with rigorousness. (3) Instead of having three similar figures to explain each experiment (fig. 1,2,3), I believe that a figure that explains the whole framework.
  • Most findings are too incremental to the prior work, for instance it is well known that steering vectors exist to guide the model in a certain direction. Plus, besides the quantitative numbers, some qualitative analyses are needed to see how the KV cache are actually behaving.
Questions for authors
  • Are there any qualitative analyses for the KV-cache representation itself that can support the findings?
  • Instead of asking the LLM to answer where they had refer to (which itself can be hallucinated), are there any other ways to check the source reports?
  • Is there a reason the KV cache is the best option for this analysis? Why not other model internal representations?
4 comments

Part 1/2: Weaknesses

We thank the reviewer for the comments on presentation. We accept most of them and list the changes below. We also clarify the main contribution, which our writing did not make prominent enough. We answer the reviewer's questions in Part 2.

W1. Presentation.

(1) Related work. We agree. The revision adds a paragraph to the main text that states the difference from prior work. Attribution work (Rashkin et al., 2023; Cohen-Wang et al., 2024) evaluates whether the context supports an output. Knowledge-conflict work (Longpre et al., 2021; Ortu et al., 2024; Jin et al., 2024; Zhao et al., 2025) studies which source the model selects. We study a different object: what the model's later report about its own source depends on.

(2) Structure. The paper contains several similar interventions, and we ordered them so that each one tests a single hypothesis and motivates the next. Document edits test whether the report depends on the document (Section 3). Transplants test whether it depends on the Q+A state (Section 4). Steering and projection edits locate the components of that state that carry the effect (Sections 5–6). Section 7 links them to source selection. Table 1 maps each hypothesis to its evidence. In the revision, every section will open with the claim under test and the result.

(3) Figures. Figures 2 and 3 show different interventions: one edits the document cache, the other the Q+A cache. We will add a single overview figure of all interventions and shorten these two.

W2. Novelty. We agree that steering vectors are well known. We use steering as a tool. The main finding is in Sections 3–4: after the model has answered, changing or removing the document never changes the source report, while transplanting the Q+A cache always does, with every token fixed. The report depends on state computed before the source question is asked. To our knowledge, no prior work tests this.

Two properties separate our directions from a standard steering result. First, they are fitted on states recorded before any source question, yet they change the later report. Second, without refitting, the same directions change which source supplies the answer during generation (Section 7).

Part 2/2: Questions

Q1. Qualitative analyses of the KV cache. The paper reports content checks after each edit: the model still repeats its answer and quotes the document (Appendix C.7). It also reports one qualitative observation: after a question-cache transplant from an irrelevant-document run, Qwen3-8B often states that the document does not mention the subject (Appendix C.2).

Q2. Checking source reports without asking the model. The report is our object of study; we do not use it as a measurement of provenance. Where provenance must be known, we read it from the answer. In counterfactual runs the answer value shows which source was used, and Section 7 measures source choice from the answer alone, with no source question. In our setting, reports match the expected label in 92–100% of runs in all but one model–condition cell. How the cached state behaves when the report is wrong is an open question that we will state in Section 8.

Q3. Why the KV cache. At report time, later tokens can access earlier positions only through attention, that is, through the keys and values cached at those positions. The cache is therefore the complete interface between the earlier computation and the report. Editing it changes exactly what the report can read. It also lets us intervene after the answer is complete while every token stays fixed. Appendix C.9 compares value steering with residual-stream steering for source selection. We will state this motivation in Section 2.

Thank you for your clarifications and changes. I have checked the revised draft, but no signs of the figure update yet. Can the authors update the figures as stated and add a holistic framework figure?

I believe that the revision makes the paper not as a clear reject, so I raised the scores accordingly, but as the other reviewer mentions, I believe that the work needs some re-write.

Thank you for raising your score. The revision adds the related-work paragraph to the introduction and the motivation for KV-cache interventions to Section 2. We have not yet finalized an overview figure that works well; we will add a single framework figure covering all interventions and shorten Figures 2–3 in the camera-ready version.

Reviewer_wR56OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how a language model decides what to say when asked whether its previous answer came from a supplied document or from its own knowledge. The authors call this reply the source report, and study it by editing the KV cache after the answer is generated and before the source question is asked.

The paper narrative is as follows:

  • Experimental results show that changing or removing the document's cache leaves the report unchanged. In addition, transplanting the cache of the question and answer (Q+A) from another run with identical tokens makes the report follow the donor run.
  • Thus, the paper claims that information retained in the Q+A cache determines the source report.
  • Linear directions fitted in the Q+A keys and values can steer the report, and account for most of the transplant effect.
  • Applied while the model is answering, the same value directions can partly change which source the answer comes from.
Critical strengths

The experimental design is well controlled and careful.

The paper connects source reporting to answer generation. The experiments that apply learned directions during answering suggest that the identified representations may be related not only to how the model describes its answer’s source, but also to which information influences the answer itself.

Critical weaknesses

H1–H3 are phrased as things the model might decide to do: "re-read the document" or "judge the answer against its knowledge." A transformer doesn't pick a strategy. It runs a forward pass, and earlier computation persists in the cache that later tokens attend to. It would be clearer to define the hypotheses by what the report causally depends on:

  • the document's K/V entries,
  • the answer's relation to parametric knowledge, or
  • information retained in the Q+A cache.

This is a presentation issue which can easily be fixed.

In this setting, simply asking the model "did you use the document or your own knowledge?" already gives the right answer almost every time. Eligibility is 92–100%, and the eligible runs are by definition the ones with the expected report. The paper therefore explains how the model performs a task it already performs well, on a task where the right answer is easy to determine.

That can still be interesting from an interpretability perspective. However, the paper does not show that the mechanism helps with anything practical:

  • It does not detect wrong source reports.
  • It does not improve attribution.
  • It does not give a usable way to control which source the model relies on (see W4).

The authors describe their contribution as mechanistic. Even then, the paper should explain why this mechanism matters. What would someone building or auditing a RAG system do differently after reading it? The main finding is hard to connect to any real problem.

A simple alternative to H3 is that the model reports whichever source is consistent with the answer it gave. You can have an experiment where you KV cache looks like:

[document representation encoding answer1] [Q + A representation implicitly encoding answer2] [answer1]

It's likely the model will report "context" in this case as its source, which is against H3.

Runs are kept only if the report matches the label expected for the document type. As a result, every analyzed report is "correct" by construction, and we never see cases where the model misreports its source.

This makes it impossible to tell whether the identified state tracks which source was used or merely whether a relevant document was present. The authors acknowledge this (§8), but it is the central question for a paper titled around a "trace." It is also where practical value would come from (W2). A mechanism that explained or predicted misreports would be far more useful than one that explains correct reports. Studying naturally occurring misreports would help, for example on DynamicQA, where the evidence is less clean.

I also have a concern about mechanism; there is no "intrinsic" difference between the "context" and the "Q+A" token. To the model, they are all tokens encoding information. You could simply learn a steering vector to suppress information from earlier tokens. How do we know this is not the underlying mechanism? During answering, steering the question values changes the downstream queries, and therefore possibly how much attention goes to the document. The direction may simply reduce or increase reliance on the document tokens, rather than switching between "use context" and "use knowledge."

Questions for authors
  1. In this setting, simply asking the model about its source already gives the expected answer almost every time. Is there a task where the cached state, or the fitted directions, gives information that prompting cannot?
  2. How do the identified directions behave on runs where the model reports its source incorrectly? Excluding these through the eligibility filter leaves the most informative cases unexamined. For example, consider runs where the model answers from memory but reports context. Does the projection onto the fitted directions match the report, or the source actually used?
  3. If you transplant the donor's question cache and then recompute the answer cache with a different, forced answer, does the report follow the donor's question state or the new answer? This would directly test the alternative to H3 that the model reports whichever source supports its visible answer.
  4. How do we know the steering vector is not simply reducing or increasing reliance on the early tokens? Evidence that the directions work on multi-document inputs or other slightly more complex settings could help clarify here.

The paper needs to fill in certain gaps - mostly explained in the weaknesses and questions raised above.

4 comments

Part 1/2: Questions

Q1. Does the cached state give anything that prompting cannot?

Yes, two things.

  • Control. The fitted directions override an opposing instruction. They switch the answer's source in 51–85% of cases (Section 7). Prompting usually beats steering (Wu et al., 2025), so this shows how strongly the state drives source selection.
  • Limits of the report. Asking the model returns a report. It cannot reveal that the report can be reversed with every visible token fixed. This matters for anyone who uses self-reported attribution as an audit signal.

Q2. What happens on runs with a wrong report?

Natural misreports are rare in our setting, so we construct the mismatch instead. In cell (c) of the test below, a relevant document is present, but the answer comes from memory. The model then reports knowledge in 91.7% of cases on Qwen3-8B and 68.4% on OLMo-3-7B. The state is therefore more than "a relevant document was present". [We will add the projection analysis on unfiltered runs, including DynamicQA.]

Q3. Forced-answer test.

We ran it on OLMo-3-7B (N = 114) and Qwen3-8B (N = 109). The document always says "Tokyo". The question cache comes from that run or from a neutral-document run. The answer is forced and its cache recomputed. Cells show the share of context reports.

Question stateForced answerOLMo-3-7BQwen3-8B
(a) counterfactualTokyo100%100%
(b) neutralTokyo36.0%0%
(c) counterfactualParis31.6%8.3%
(d) neutralParis0%0%

Cell (b) is the reviewer's layout. The predicted context report does not appear: 0% on Qwen3-8B and 36% on OLMo-3-7B. A visible answer that matches the document is not enough.

The report is reliably context only when the question state and the answer agree (a). Both factors matter about equally, which matches Appendix C.4: only the joint Q+A transplant transfers every report.

Two limits: the recomputed answer cache can read the document, and we did not run the 32B models.

Q4. Is the direction only changing reliance on early tokens?

For the report, no. We steer after the answer is complete, and removing the document never changes the report (Table 2). A report that does not depend on the document tokens cannot be flipped by reweighting attention to them.

For source selection (Section 7), we claim only that the direction changes which source supplies the answer. A shift in attention to the document may be how this happens.

Thanks for your answer. I will elaborate a bit more on why I think the paper needs a bit of re-writing before getting published. I hope this also clarifies my question 4.

The experiments establish causal effects of cached computational history in relatively controlled, short-context settings, but they do not establish that the same effects persist in more "realistic" setups.

An example of a realistic setup is when a long document is followed by a substantial amount of irrelevant tokens before the question is answered. Another example is when the document is injection in the middle of a conversation with the LLM, and a question about the document is asked later. Concrete example: a 1000 token long document, followed by some 1000 tokens of irrelevant questions and answers, followed by a question about a fact in the document.

The paper does not directly test these situations. In particular, it does not test whether the relevant information from the original document survives the intervening computation, whether it remains accessible through attention, or whether the later source report can still be controlled by the earlier cached state. I think the important issue here is that the paper's learned direction is not demonstrated to be a general mechanism for overriding parametric knowledge whenever a document appears in the prompt. The direction may encode a location- and prompt-configuration-dependent signature of relevant document processing, rather than an abstract mechanism for source selection. I think without these experiments, the core findings of the paper are not properly verified.

P.S. I don't think it is necessary to include experiments for all the examples I listed in my second paragraph; that would be exhaustive and difficult. However, I believe making the current setup more "realistic" is needed for the sake of making sure the direction encodes what the paper claims.

Thank you for the follow-up.

Realistic setups. Our findings are not limited to short templated prompts. Appendix E replicates the document edits and cache transplants on DynamicQA, which uses Wikipedia paragraphs with counterfactual variants. We have now also run the setup you proposed:

dialogue with the model → long document containing the needed fact → unrelated multi-turn conversation → question about the document → source question

We applied the paper's interventions to this context. Results on Qwen3-8B:

Intervention (long dialogue)Report changed
Document cache removed0%
Question cache from irrelevant-fact run → knowledge100%
Question cache from true-fact run → context93.1%

As in the paper, removing the document does not move the report, while the transplanted question state does. We will add the full results to the appendix.

Does the direction act through the document? We repeated report steering after removing the document's entire cache, keeping the positions of all other tokens. Qwen3-8B, held-out relations, α = 3:

Steering goalDocument presentDocument removed
Create context (irrelevant / neutral recipients)100%100%
Remove context (true / counterfactual recipients)100%100%

The model reports context even when no document is in the cache. The effect therefore cannot come from re-weighting attention to document tokens.

Scope of our claims. We do not claim that the direction is a general mechanism of source selection, and we do not claim how the model reads it. That may differ across model families and scales, and we leave it to future work. Our claim is narrower: the source report causally depends on state cached while answering, not on the document available when the report is requested. This dependence replicates across two model families, 7B–32B, templated and Wikipedia documents, and now long multi-turn dialogues. Each experiment in the paper is needed to establish it in both directions, by blocking transfer and by reproducing it. A circuit-level account of how the state is used is a separate study that would not fit within the page limit.

We hope this addresses your concerns. If a specific remaining issue would change your assessment, we would be glad to address it.

Projection analysis on unfiltered runs

Setup. We took all runs without the eligibility filter: the templated task (122 facts, five document types) and 500 DynamicQA items, on OLMo-3-7B and Qwen3-8B. For each run, we computed one number with no intervention: the projection of the question and answer values, cached before the source question, onto the Section 5 direction, averaged over positions and layers. Directions were fitted on other relations (for DynamicQA, on templated data), so there is no leakage.

Results.

  • Templated task. The report never disagrees with the actual source, and the projection predicts it almost perfectly (AUROC 0.999 on OLMo-3-7B, 1.000 on Qwen3-8B).
  • DynamicQA. Mismatches between the report and the actual source are rare: 1.2% on OLMo-3-7B and 3.7% on Qwen3-8B. On Qwen3-8B counterfactual runs, where a relevant document is always present, the projection predicts the source actually used better than the report (AUROC 0.988 vs. 0.902). The state therefore tracks document use, not only document presence.

Median projection on DynamicQA, normalized so that 0 is the irrelevant-document level and 1 is the relevant-document level:

Run typeMedian
Used document, reported context0.50
Answered from memory, reported knowledge0.08
Used document, reported knowledge (misreport)0.36
Answered from memory, reported context (misreport)0.11

Misreports that used the document lie between the two correct classes. Memory-answered misreports on Qwen3-8B sit at the memory level. The four such cases on OLMo-3-7B spread over 0.17–0.66.

Scope. This analysis is outside the paper's main claim, which concerns what the report causally depends on, not how it fails. The results are consistent with our interpretation. However, natural misreports are rare in this setting, so the misreport subsets are small and do not support a firm claim. Establishing how the state behaves on misreports would require a dataset designed to elicit them at scale. That is a separate study, beyond the scope of this paper. We will add these results to the appendix as a preliminary analysis.

Reviewer_vetnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how LLMs report whether their answers originate from supplied documents or parametric knowledge, using KV-cache interventions. Experiments cover four models ranging from 7B to 32B parameters, with eligible examples drawn from 155 candidate facts across eight relations and additional evaluations on adapted DynamicQA passages.

Critical strengths

The paper investigates an interesting and underexplored question about LLMs’ ability to identify the sources of their own answers. This question has practical relevance for improving the grounding, transparency, and trustworthiness of LLMs.

Critical weaknesses
  1. Ambiguous provenance ground truth. In the true-fact condition, both context and parametric knowledge support the same answer. Actual source use therefore remains unverified in the transplant experiments.

  2. Unclear interpretation of the cached trace. The transplanted Q+A states already contain document-conditioned information. Their effects on source reports and persistence after document deletion do not distinguish a dedicated provenance trace from document relevance or answer–context compatibility.

Questions for authors

Please find the weakness above.

1 comment

We thank the reviewer. The paper addresses both points, but only briefly and partly in the appendix, because neither is the core claim. The revision will bring them forward.

W1. Provenance ground truth. Document use can be tested directly: replace the document's value and check whether the answer follows. This is our counterfactual condition. When the document states a false value, the model answers with it in at least 92% of runs on every model (Section 2), so under our instruction the answer depends on a relevant document. We agree that this does not settle which source produced the answer when both agree. The main conclusion does not depend on it. We study what the model's report about its source causally depends on, and the report follows the Q+A state in counterfactual runs as well, where provenance is unambiguous (Tables 2 and 11).

W2. Interpretation of the cached trace. We agree that the Q+A state is document-conditioned. Appendix C.8 compares the two readings: a direction fitted on document presence (supporting versus irrelevant document) and one fitted on document use (same document, instruction varied). Both change the report (Table 14), so document relevance alone does not account for the state. A report-time compatibility check does not either: inserting a document that states the answer never creates a context report (Table 2).

Which reading is correct does not affect the main claim, which is causal and makes no assumption about what the state encodes. On true-fact–irrelevant pairs, components of this state are nearly necessary and sufficient for report transfer (Section 6), while the document available at report time has no effect (Section 3). Section 8 lists the semantics of the state as open. We will move the comparison from Appendix C.8 into the main text.

Why the result matters. To our knowledge, no prior work tests what a model's report about its own source depends on. Informally, when we put the paper's question to colleagues who work on LLMs, and to current chat models (Claude Fable, ChatGPT Astra), the first answer was almost always the same: the model checks its answer against the visible document.

Comments on the paper

1 comment

Summary for reviewers and the AC

Main claim. After answering, a model's report about the source of its answer causally depends on state cached at the question and answer positions, not on the document available when the report is requested.

Scope. We study the report as model behaviour, not as ground truth about provenance. We do not give a head- or circuit-level account of how this state is read; that may differ across model families and scales. To our knowledge, what this report depends on has not been tested before.

Discussion experiments. Reviewers proposed several alternative explanations, and every new test of the main claim supports it. A visible answer that matches the document does not determine the report (App. F.1). Report steering works with no document in the cache (App. F.2). The effects persist in a long multi-turn dialogue (App. F.3). The projection analysis of natural misreports (App. F.4) addresses what the state encodes, which lies outside the main claim and remains open; the numbers in Table 27 supersede those in our earlier comment.

Revisions. Clarified the scope in the abstract and introduction, added a related-work paragraph and the motivation for KV-cache interventions to the main text, and updated the limitations. [The overview figure requested by Reviewer d6pV will be added in the camera-ready version]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.