- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies whether regional KV-cache replacement effects depend on subsequent continuation content. Using matched correct–incorrect GSM8K solutions across eight checkpoints, it compares thinking modes, early error-related projections, and later error acknowledgment. Natural and controlled prefix interventions show that continuation content can change relative regional sensitivity.
Critical strengths
- The matched examples and fixed-mode prefix interventions provide a controlled setting for testing whether continuation content changes cache-replacement effects.
- Evaluation across multiple model families, together with decoding and judge robustness checks, strengthens the empirical findings within the studied setting.
Critical weaknesses
-
The framing exceeds the demonstrated scope. All experiments concern explicit acknowledgment of localized arithmetic errors, combining error recognition, information use, and reporting behavior. Although acknowledged as a limitation, the connection to general causal attribution and CoT monitoring remains untested. The paper should narrow its framing or validate the finding on a distinct reasoning task without an error-acknowledgment requirement.
-
The incremental contribution and mechanism remain unclear. Prior work studies how intermediate sentences influence subsequent reasoning [1], and Li [2] already shows that CoT changes sensitivity to source-cache edits. The authors should clarify what fixed-mode prefix interventions establish beyond these findings. Regional KV replacement measures intervention sensitivity but does not explain how continuation changes information use. Targeted attention or computational-path interventions would strengthen this interpretation. The early-projection versus behavior comparison also needs a clearer connection to the central question.
[1] Paul C. Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. Thought Anchors: Which LLM Reasoning Steps Matter? 2025. https://arxiv.org/abs/2506.19143
Questions for authors
- Which finding goes beyond the reasoning-dependent cache use demonstrated in [2]?
- What causal quantity does “causal relevance” denote? Which controls distinguish effects of error-related information from other changes introduced by replacing an entire cache region?
- What evidence connects regional sensitivity changes to CoT monitoring reliability, given that predictive monitoring need not rely on causal signals?
[2] Bojie Li. Models Take Notes at Prefill: KV Cache Can Be Editable and Composable. 2026. https://arxiv.org/abs/2606.17107
10 comments
Response to Reviewer 9cPt (2/10)
W1. Generalization beyond arithmetic error acknowledgment (2/2)
Results. Both tasks reproduce the paper's central findings: regional attribution varies across modes and with prefix content within a fixed mode. With an information-insufficient ShARC base and sufficient donor, Gemma reverses from edited-span to downstream dominance. Qwen3.6 shifts similarly but retains stronger edited-span sensitivity; Qwen3 8B shows a smaller, opposite shift. Direction and magnitude vary by checkpoint.
| Checkpoint | : T | : NT | Δ | : T | : NT | Δ |
|---|---|---|---|---|---|---|
| Gemma4 26B-A4B | 84.0 | 0.0 | −84.0 | 9.0 | 89.7 | +80.7 |
| Qwen3.6 27B | 102.8 | 70.2 | −32.6 | 4.9 | 9.2 | +4.3 |
| Qwen3 8B | 97.3 | 98.0 | +0.7 | −6.2 | −12.0 | −5.8 |
| GPT-OSS 20B | 94.4 | Unstable† | Unstable† | 5.6 | Unstable† | Unstable† |
†GPT-OSS NT's joint reference is +0.5 pp [95% CI: −3.5, +5.0], precluding stable normalized NT effects and mode differences.
For XSTest's safe bases with unsafe donors, all four checkpoints show lower normalized edited-span effects and higher downstream effects in non-thinking mode.
| Checkpoint | : T | : NT | Δ | : T | : NT | Δ |
|---|---|---|---|---|---|---|
| Gemma4 26B-A4B | 61.2 | 0.0 | −61.2 | 35.9 | 61.5 | +25.7 |
| Qwen3.6 27B | 67.1 | 3.8 | −63.2 | 10.4 | 19.2 | +8.8 |
| Qwen3 8B | 56.7 | 46.5 | −10.2 | 46.5 | 66.3 | +19.9 |
| GPT-OSS 20B | 88.1 | 10.1 | −78.0 | 51.9 | 92.4 | +40.6 |
Within each fixed mode, prefix content changes regional sensitivity in every profile (BH q < .05):
| Task | Significant raw-L contrasts | Profiles with a significant contrast |
|---|---|---|
| ShARC | 18/24 | 8/8 |
| XSTest | 31/48 | 8/8 |
Overall, these results support the broader task scope of our finding: regional attribution depends on decoding mode and continuation content across distinct tasks and behavioral targets.
Response to Reviewer 9cPt (3/10)
Q1 + W2.1. Distinction from Prior Work (1/2)
Reviewer concern: Given prior work on intermediate-sentence influence and reasoning-dependent cache use, it is unclear what the fixed-mode prefix interventions add.
Summary: We show that continuation content changes the behavioral effect of the same earlier cache replacement and shifts the relative effects of error-span and post-error replacement. We compare each region’s patched–unpatched effect across prefix conditions, testing changes in intervention sensitivity beyond changes in unpatched behavior. A controlled 2×2 manipulation of error restatement and a correct proposition changes this regional balance while the checkpoint, original prompt prefill, target mode, and endpoint remain fixed; natural-prefix replay provides complementary evidence. This establishes continuation dependence of relative regional intervention effects, extending the CoT-dependent source-edit sensitivity demonstrated in [3].
Our contribution concerns instability in regional causal attribution across generation modes, continuation content, and measured endpoints. For each prefix, we measure the behavioral difference between replacing a particular earlier cache region and leaving it unpatched. We then compare these differences across prefixes. The question is whether changing the prefix changes the behavioral effect of the same cache replacement and shifts the balance between error-span and post-error replacement effects.
Cache replacement precedes prefix processing. Within each prefix condition, identical text is processed separately against patched and unpatched caches, with its internal states recomputed in each case. The checkpoint, original prompt prefill, target mode, and behavioral endpoint—error acknowledgment—remain fixed.
Distinction from [4]. [4] measures sentence importance through changes in final-answer distributions after sentence resampling and in subsequent token distributions after attention suppression (§§2–3, 5). The manipulated sentence is itself the object whose influence is measured. In our experiments, the prefix instead provides the conditions under which separate interventions on earlier cached regions are evaluated. A sentence’s influence on subsequent outcomes does not establish that it changes the behavioral effect of intervening on an earlier cached state. Its reasoning/base comparison also concerns receiver-head attention concentration (Appendix K), whereas our mode comparison measures regional KV replacement effects within the same checkpoint.
Distinction from [3] and the central finding. Li [3] studies how cached conclusions support decisions and how caches can be edited and reused, including CoT-dependent source-edit recovery and decision recovery through appended errata (§4; Fig. 2). Our additional finding is that controlled prefix content shifts the balance between error-span and post-error replacement effects, established by comparing the two regions’ patched–unpatched effects across content conditions.
Our controlled prefixes manipulate two content factors—restatement of the erroneous relation and provision of a correct proposition—using short declarative sentences, without explicit correctness judgments or checking/repair instructions. These are controlled propositional manipulations rather than generated or replayed natural CoT. Under each content condition, we separately measure error-span and post-error replacement effects. Their relative effects change across content conditions, with interactions between the two factors in some profiles (§6.2; Fig. 7). This establishes continuation-dependent changes in regional attribution even when the target mode remains fixed.
(Continued in Part 2/2 below.)
Response to Reviewer 9cPt (1/10)
Thank you for taking the time to review our work. We appreciate your emphasis on the contribution and broader relevance of the findings. Your comments motivated the additional experiments below and helped us make the connection to monitoring more concrete.
W1. Generalization beyond arithmetic error acknowledgment (1/2)
Thank you for highlighting the need for evidence beyond arithmetic error acknowledgment. To test the scope of our findings, we extended the Sec. 5.2 mode comparisons and Sec. 6.2 fixed-mode prefix interventions to ShARC-derived rule reasoning [1] and XSTest-derived safety responses [2].
Settings. Each task has 200 token-aligned pairs. From 100 ShARC rules, we built two pairs per rule differing in one Yes/No answer token, yielding decidable versus information-insufficient cases; the outcome is requesting information or expressing an information limitation. XSTest requests were rewritten so a localized edit establishes safe/unsafe status; the outcome is expressed refusal or an intention to refuse. Paired inputs match in token length and all unedited tokens. Pchange denotes the edited span; Pafter change denotes the post-change region, comprising all subsequent prompt tokens through the generation boundary.
We tested Gemma4 26B-A4B, Qwen3.6 27B, Qwen3 8B and GPT-OSS 20B in thinking/non-thinking modes (T/NT), giving eight profiles (GPT-OSS: native/empty analysis). We use the paper's GPT-OSS-120B judge with task-specific definitions and no perception filtering. Both donor directions were evaluated. Matched cohorts use unpatched = 0% and joint replacement of the edited span and post-change region = 100% per checkpoint/mode; Δ = NT − T (normalized pp, before rounding).
Prefix comparisons hold the checkpoint, prefill, and mode fixed. ShARC uses neutral/restatement/inference prefixes with three wordings each. We average across wordings and donor directions, with inference clustered by source rule. XSTest uses four 18-token prefixes (wording/evaluation score/skip review/first conclusion) under a shared evaluation context, with 152 held-out cases, safe donors into unsafe bases and suffix-only judging.
L is the donor-oriented regional-effect difference. Raw contrasts use sign-flip tests with BH correction across all pairwise prefix comparisons: 3 contrasts × 8 profiles = 24 for ShARC, and 6 × 8 = 48 for XSTest.
(Continued in Part 2/2 below.)