acceptodds
Under review as a conference paper at ICLR 2027

Out of Sight, Still in Mind? Separating Internal and Answer-Level Effects of Retained Key-Value History

Abstract

Cache eviction, prompt compression, and context management change which source tokens remain directly accessible in the key–value (K/V) cache, while later states computed from those tokens can remain. How removed text then influences computation, and whether internal effects translate into answer-level effects, is a measurement question. We study it with a paired cache-interchange design. Two histories specify the same task, by worked examples or by a direct instruction, and are followed by an identical bridge text; we replace the source-prefix cache with a shared neutral cache and swap the retained bridge states between histories under an identical query. We read donor projections at three levels, an intermediate hidden state, the next-token logits, and the correct-minus-incorrect answer margin, with endpoint-specific aggregation, and we report candidate choices wherever candidate scores were archived. Across five arms (Qwen, Mistral, and OLMo; two synthetic rule tasks; 500 indirect-object-identification (IOI) episodes), retained bridge states remain causally active after source-prefix neutralization: mean net donor-axis projections at the intermediate readouts range from 0.59 to 0.97, all 1,780 episodes are positive, and the format difference vanishes when bridge attention is cut. These internal effects do not establish corresponding answer-level effects. Logit-contrast projections are smaller (0.007–0.26), and answer-margin projections range from to . On IOI, which the model solves without the histories, a hidden-state projection near one coexists with a small negative donor-oriented margin shift ( [, ]) and no changed candidate choice in either patch direction. We formalize what donor projections can and cannot establish and give reporting practices that keep internal alignment, answer margins, and candidate choices separate.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation patching, interchange interventions, KV cache, context compression, in-context learning, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_is27OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper swaps K/V bridge states retained after source-history removal between example- and instruction-format histories, then measures effects at hidden-state, logit, and answer-margin levels. Across five arms (Qwen, Mistral, OLMo; synthetic tasks and IOI), large hidden-state effects do not translate into answer-level changes.

Critical strengths
  • The paper is unusually candid about its limitations, failed launches, retrospective analyses, and provenance gaps. It also releases all episode records with CPU scripts that reproduce every reported statistic.
Critical weaknesses
  • No arm tests a setting where the answer depends on the removed history. In the synthetic arms, the query restates the rule and the models are not competent (primary Qwen: exactly 50%, always choosing the second clause). In IOI, the model answers correctly without any history. Since the answer never depends on the history, it is unsurprising that swapping bridge states leaves answers unchanged.

  • The main dissociation has an untested, simpler explanation, and the other findings are largely expected. The hidden state is read at layer 8, just two blocks after the patched layers 0–5, while logits are computed after all layers. The gap may therefore reflect readout depth. The authors acknowledge this, but the inexpensive depth-control experiment was not run. The overall work is trivial.

Questions for authors
  • Can you report results on a source-dependent task, where the answer information appears only in the removed history (high full-history accuracy, low query-only accuracy)? If retained bridge states produce donor-directed answer changes there, my assessment of RQ2 would change substantially. See the weakness above.

  • How do hidden-state projections behave when read at the final layer, or at several depths after the patched layers? If they shrink toward the logit-level values, the dissociation would be largely explained by readout depth.

2 comments

Thank you for identifying the two tests needed to assess the interpretation.

Depth control. The early IOI projection decreases from 0.968 at the original readout to 0.0078 at the final hidden state, close to the centred-logit projection of 0.0072. This supports your explanation of the large intermediate-versus-output contrast. It narrows the interpretation of that result. Middle and late patches produce larger final hidden and logit effects, but candidate choices remain unchanged.

Source-dependent evaluation. The new tasks require information from the passage. When only the generic bridge is retained before the question, source replacement gives near-chance accuracy. When replacement occurs after question processing, accuracy remains high on lookup and adapted SQuAD, and the selected partial swaps increase donor-answer preference by approximately 96–99 percentage points relative to same-answer controls.

This second condition has an important boundary: the retained states have already processed the question and options. It establishes answer-relevant retention under that condition, rather than general recall after removal. The unseen-question experiment finds smaller accuracy gains. For its donor-choice effect, the interval excludes zero in Mistral but not Qwen under the tested swap.

Contribution. The follow-ups provide a more specific empirical account. The original early IOI effect attenuates across depth. Behavioral transfer depends on what the retained segment has processed. The effective patch locations also differ between the two tested models. We believe these comparisons add useful evidence to the measurement contribution, with conclusions limited to the tested settings.

We have also updated our paper as per ICLR 2027 guidelines.

Reviewer_UtLtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper asks whether bridge states in a KV cache continue to affect later computation after the original source prefix has been neutralized, and whether internal representational changes imply changes in the model's answer. The authors pair histories expressed as worked examples or direct instructions, replace the source-prefix cache with a shared neutral cache, and interchange bridge states at selected layers. They measure donor-axis projections in hidden states, logits, and correct-versus-incorrect answer margins, and report candidate choices and task competence where available.

Across five experimental arms using Qwen, Mistral, and OLMo, the hidden-state projection is positive in every episode. Logit projections are smaller, while answer-margin effects vary in sign across tasks. On IOI, the model answers the candidate comparison correctly in nearly all episodes, yet the hidden-state projection is close to one, the donor-oriented margin shift is slightly negative, and no candidate choice changes. The authors argue that internal representation effects, answer-level effects, and task competence should be reported separately.

Critical strengths
  1. It evaluates the same intervention at the hidden-state, logit, and answer-margin levels, and reports candidate choices where available. This directly supports its central methodological point that one internal metric is not enough to establish a behavioral effect.

  2. They report that the synthetic tasks are not solved reliably and avoid presenting positive projections as successful task transfer. They also disclose missing archived outcomes, retrospective analyses, and limits on verifying the stated preregistration chronology.

Critical weaknesses
  1. The intervention patches early layers and the hidden-state readout is close to the last patched layer, with several attention layers able to read donor bridge states before it. Without an early/middle/late patch-location comparison, the near-one projection should not be read as a large or robust retention effect. This weakens the quantitative hidden-vs-logit-vs-margin comparison, though not the qualitative persistence claim.

  2. Examples and direct instructions differ in length, wording, syntax, and repetition; the same-format control changes only a nonce (and IOI source names). The authors acknowledge this, but the framing and practice recommendations should be consistently limited to retained presentation state.

  3. Synthetic arms do not archive patched candidate choices, so the answer-level conclusion is limited to scalar margin movement. The paper also does not evaluate deployed eviction or compression; this is an acceptable scope choice but limits practical implications.

Questions for authors
  1. Have you run controls that vary patch depth or location? If not, what evidence rules out the possibility that the large hidden-state projection is partly due to the readout's proximity to the patched layers?

  2. Can patched candidate choices or equivalent answer-level statistics be recovered for the synthetic tasks from the saved records?

2 comments

Thank you for distinguishing qualitative retained influence from the strength of its quantitative interpretation.

Readout proximity. The depth experiment supports your concern. The early-patch projection falls substantially before the final state. We therefore limit the near-one result to the intermediate readout where it was measured. Middle and late patches show that final effects also depend on intervention location.

Presentation differences. The original examples-versus-instructions comparison does not isolate task information from presentation. We agree that its interpretation should remain limited accordingly.

The new tasks use matched passage formats and token lengths, an identical question within each pair, and different correct answers. Opposite-answer swaps are compared with same-answer donor controls. This tests changes in answer-relevant content while controlling the presentation format.

Patched synthetic choices. We have not recovered patched candidate choices from the original synthetic records or rerun those arms. Their missing endpoints remain unavailable in this response. The follow-up tasks record candidate choices and unrestricted first tokens directly. On Qwen lookup, for example, 1,013 of 1,018 initially correct runs switch to the donor’s answer under the opposite-answer swap, with the same count for unrestricted first-token labels.

Practical scope. These experiments test constructed cache replacements. They do not evaluate a deployed eviction or compression policy. The strongest behavioral effects occur after the retained states have processed the question, and should not be interpreted as general recall for questions asked later.

We have also updated our paper as per ICLR 2027 guidelines.

Reviewer_TY82OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the causal influence of removed prompt tokens (persisting in later "bridge" KV-cache states computed before removal) actually translates from internal representations to final model answers. Using a paired cache-interchange design, the authors prefill two histories specifying the same task (worked examples vs. direct instruction) followed by a shared bridge string, replace the prefix KV cache across all layers with a shared neutral cache, and swap early-layer bridge KV states before evaluating an identical query.

They measure donor-axis projections at three levels -- an intermediate hidden state, next-token logits, and a two-candidate answer margin -- alongside baseline task competence and formal metric limitations (such as unmeasured orthogonal shifts, softmax shift invariance, and surface-format confounding).

Across five arms spanning Qwen2.5-3B, Mistral-7B, and OLMo-2-7B on two synthetic rule tasks and Indirect Object Identification (IOI), early-layer bridge swaps consistently produce large positive hidden-state shifts (0.59–0.97 across all 1,780 episodes), but much smaller logit shifts (0.007–0.26) and task-dependent margin shifts (+0.42 to -0.09).

Notably, synthetic arms show strong internal shifts despite near-chance task competence (50%–63%), while on IOI (where the model is 100% accurate without needing the history), a 0.97 hidden-state shift coexists with a slight negative margin shift (-0.032) and 0/500 changed answer choices, motivating reporting practices that separate internal alignment, output margins, discrete choices, and task competence.

Critical strengths
  1. Evaluating the exact same KV-cache swap across intermediate hidden states, logits, and answer margins/choices cleanly shows that near-complete internal donor alignment (0.97 on IOI; 0.60–0.80 on synthetic tasks) can coexist with 0/500 changed answers (IOI) or near-chance task competence (50%–63%). This offers a concrete, practical audit framework against over-interpreting internal representation shifts in KV-cache and patching studies.

  2. The paper carefully accounts for projection pitfalls (unmeasured orthogonal shifts, softmax shift invariance in raw logits, and surface-marker confounding) and transparently surfaces condition-level heterogeneity and negative findings rather than cherry-picking aggregates.

Critical weaknesses
  1. Bridge KV states are swapped only in early layers (0–5 or 0–4) while all remaining layers still read the recipient's bridge, and hidden states are read just two layers after the swap whereas logits/margins are read at the final layer. The headline internal-vs-answer dissociation may therefore largely reflect readout proximity and downstream overwriting rather than a general property of retained KV history, yet the depth-control experiment in Appendix I was not run.

  2. In every arm, the removed prefix is unnecessary to solve the query (synthetic queries restate the rule inline while models fail at 50%–63% without chat templates; IOI gets 499/500 from the query alone), so the setup only tests surface-format sensitivity rather than retained task knowledge. Furthermore, relying solely on CPU reanalysis of archived scalar logs leaves missing endpoints across arms (e.g., no patched choices or same-format margins in synthetic arms, no margins in OLMo, and no saved vectors for full-distribution metrics).

Questions for authors
  1. On Qwen2.5-3B IOI (500 episodes on a single GPU), if you record Δ^h\hat{\Delta}^h across all layers rather than only two blocks after layers 0–5, how does the hidden-state donor projection evolve through middle and final layers as the query attends to the recipient's unpatched bridge? Furthermore, if you swap an equal-width middle or late 6-layer bridge block instead of layers 0–5, does the dissociation between post-patch hidden states and final answer margins/choices still hold?

  2. Currently, no arm tests a setting where the prefix is required to solve the query (synthetic queries restate the rule inline with 50%–63% competence; IOI is 499/500 query-only). On Qwen2.5-3B with its chat template applied and the rule/mapping omitted from the query (present only in the history), (a) does the neutralized-prefix + intact-bridge cache retain above-chance task accuracy, and (b) when swapping bridges across different task rules (rather than just EE vs. DD formats), do internal shifts still fail to reach candidate choices?

  3. Can you clarify why model inference could not be re-run on the pinned checkpoints to populate the missing cells in Table 1 (OLMo SD margins/competence, synthetic same-format control margins, synthetic patched candidate-choice flips, and full-vocabulary TV/KL on IOI)? Even a single rerun on Qwen2.5-3B (SD, DW, IOI) filling in patched choice-flip counts and full-distribution TV distance would strengthen the empirical completeness of the tables.

2 comments

Thank you for the specific experimental suggestions. They helped us identify which additional measurements were needed.

Depth and patch location. We ran equal-width early, middle and late six-layer comparisons on the original IOI episodes, with hidden readouts throughout the network. For the early patch, the net hidden projection is 0.968 at index 8, 0.410 at index 12, 0.120 at index 20, and 0.0078 at the final index 36. The final centred-logit projection is 0.0072. This attenuation supports your concern. Middle and late patches produce larger final effects, although neither changes candidate choices.

Source dependence and mapping swaps. We did not reuse the original synthetic tasks. Instead, the initial lookup follow-up tests opposite key-to-label mappings using Qwen’s chat template and the paper’s bridge. The mapping appears only in the history.

Across 256 pairs, the model strictly prefers the correct candidate in 511/512 runs with the history intact, and 256/512 after source neutralisation with the generic bridge retained. The latter condition also contains one tie.

Swapping early bridge layers from the opposite mapping, relative to a same-mapping control, gives a hidden projection of 0.60 immediately after the patched layers and 0.004 at the final layer. Net donor-oriented margin movement is +0.0015 logits [−0.0009, 0.0038], with no initially correct choice switching to the donor answer. This addresses the mapping-transfer question through lookup, rather than an exact rerun of the original rule tasks.

Missing measurements. The IOI follow-up includes patched choices, full-vocabulary total variation and KL divergence. The new source-dependent tasks also record margins and unrestricted first tokens. We have not rerun the original synthetic arms or OLMo, so their missing endpoints remain unavailable. We keep these gaps explicit rather than treating the new measurements as replacements for those cells.

We have also updated our paper as per ICLR 2027 guidelines.

Comments on the paper

1 comment

Thank you for the careful reviews. We ran additional experiments addressing the two main concerns: the depth of the hidden-state readout and whether the answer requires information from the removed source.

1. Depth controls on the original IOI task

On the original 500 IOI episodes, we compared early, middle and late six-layer patches and measured hidden states throughout the network. For the early patch, the hidden projection falls from 0.968 at the original readout to 0.0078 at the final hidden state, close to the centred-logit projection of 0.0072. This supports the concern that readout depth explains much of the original contrast in this arm.

Middle and late patches yield final hidden projections of 0.251 and 0.173, with centred-logit projections of 0.246 and 0.182. The gap-weighted net margin projections are +0.339 and +0.046, compared with -0.032 for the early patch. These IOI summaries subtract the corresponding same-format controls.

None of the three patch locations changes a candidate choice. Both baseline formats already support the same correct answer, so this experiment does not test transfer between different correct answers. It does show why changes in answer scores and changes in choices need to be measured separately.

2. Tasks requiring information from the removed source

We added source-dependent tasks on Qwen2.5-3B-Instruct and Mistral-7B-Instruct-v0.3. Paired passages support different answers to an identical question. We apply chat templates, preserve padding masks, and compare opposite-answer donors with same-answer donors. The primary twelve-layer windows were selected using separate pilot items: the last twelve layers for Qwen and the middle twelve for Mistral.

The table reports all tasks, including those below the 90% intact-accuracy criterion. Each cell contains 512 pairs, except adapted SQuAD, which contains 384. “Before” and “After” refer to source replacement before or after question processing. The donor-choice change is the increase in donor preference under an opposite-answer swap relative to a same-answer swap, after question processing.

ModelTaskIntactBeforeAfterDonor-choice change, pp [95% CI]
QwenLookup95.5%49.6%99.4%+99.0 [98.0, 99.8]
QwenTwo-hop tracking48.3%49.6%46.1%-5.8 [-12.3, 0.9]
QwenbAbI one fact90.1%50.4%90.3%+80.1 [75.8, 84.2]
QwenbAbI two facts70.2%49.7%71.3%+42.7 [35.9, 49.0]
QwenAdapted SQuAD98.4%50.0%98.4%+96.9 [94.6, 98.9]
MistralLookup99.5%49.4%99.4%+97.9 [96.7, 98.8]
MistralTwo-hop tracking78.7%49.7%76.2%+50.5 [45.7, 55.1]
MistralbAbI one fact89.1%49.9%89.2%+72.6 [68.3, 76.7]
MistralbAbI two facts65.1%49.9%66.8%+28.0 [22.6, 33.3]
MistralAdapted SQuAD98.8%49.5%98.7%+96.2 [94.3, 97.9]

Here, pp denotes percentage points. Recomputing the same retained text without the passage gives approximately 50% accuracy in every cell. Confidence intervals resample paired items, with bAbI stories and SQuAD articles treated as clusters.

3. A question presented after source replacement

The large effects above concern states that have already processed the question and options. We also tested a question presented only after source replacement.

Retaining a list of source keys without values improves accuracy over matched decoy keys by 7.0 percentage points in Qwen and 4.3 in Mistral. The selected partial swap increases donor preference by 0.2 points [-1.0, 1.4] in Qwen and 7.4 points [4.3, 10.7] in Mistral. Under this swap, the interval excludes zero for Mistral but not Qwen.

4. Scope and reproducibility

The follow-up archive contains the records, inference code and CPU analyses. These experiments concern constructed cache replacements, rather than a deployed eviction or compression policy.

The results support a more precise account of when retained states affect answers, while preserving the paper's central measurement point: an intermediate projection alone does not establish an answer-level effect.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.