- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper asks what information a decoder-only transformer writes into the KV cache at an operation span, as distinct from what the unmodified model later retrieves. It freezes a base model as a "writer," recomputes its cache without gradients, masks state-description spans from a "reader," and trains only a LoRA reader at query/answer positions. On boxes and code state-update tasks, the trained reader can recover information from operation-span KVs that the base reader seldom outputs. Carrier transplants, visibility masks, and token/layer-specific interventions are then used to distinguish a native routing/address record from a payload that becomes directly accessible after isolation training. The paper reports replication across Llama-3.1-8B and Mistral-7B, additional results on ToMi and GSM8K, and several limitations and capacity controls.
Critical strengths
-
A clean experimental separation between writing and reading. Freezing the writer, recomputing its cache with LoRA disabled, and varying only the reader's access is a sensible design. The transplant and masking controls make the routing claim more causal than a standard probe result.
-
Several informative controls and transparent negative results. The paper tests irrelevant donors, a filler span, localized interventions, and multiple reader seeds. It also reports value-range failures and a limited Mistral replication. These checks support the conservative claim that operation-span K/Vs contain exploitable information, although not the stronger payload interpretation.
Critical weaknesses
-
The method does not identify the central claim. My main concern is that the paper defines information as "stored" whenever it is recoverable by a trained reader, although that reader changes approximately 13.6M parameters. Freezing the writer shows that the signal originates in the cache; it does not distinguish an existing payload from a new decoder learned over distributed features. Neither ordinary fine-tuning nor the permutation control resolves this ambiguity. There is no constrained readout or transfer test, and performance collapses when answer values move from 10-25 to 30-45. I therefore do not find the headline payload claim established.
-
The study appears exploratory and insufficiently curated. The pre-specified endpoint was null, and the emphasized contrast was selected afterwards. One reader seed per model family is excluded, while only Mistral passes the screen for the broader model experiment. The authors also disclose generative-AI assistance in experiment and figure code, and the resulting workflow appears insufficiently supervised. Section 2 reads as a checklist of short run-in paragraphs rather than a complete methodology. The Introduction contains five overlapping contribution bullets. Figures 1, 2, and 6 are difficult to parse, and all 14 result tables are placed after the references. Together with the post-hoc analysis and screening decisions, this gives the work the appearance of auto-research rather than a mature, hypothesis-driven study.
Questions for authors
-
How can the current experiments distinguish a pre-existing payload representation from a new computation learned by the LoRA reader? Results with a more constrained readout or transfer to unseen values and operations would materially affect my assessment.
-
The pre-specified endpoint was null, while the reported footprint contrast was selected afterwards. Could the authors clarify how this contrast was chosen and whether it has been confirmed on an independent run or held-out setting?
-
Were the reader-exclusion and model-eligibility thresholds fixed before observing the outcomes? How sensitive are the main conclusions when the failed reader seeds are retained and the unsuccessful model families are considered?
-
Do the token- and layer-localization results remain similar with bf16 writers or with counterfactual patches that stay closer to the model's normal activation distribution?
2 comments
Thank you for the detailed responses and additional experiments. The constrained readers, transfer tests, fresh-seed replication, and bf16 controls substantially address my concerns about recoverability and robustness. However, the evidence supports recoverable operand information rather than an explicit post-operation state, and the methodological presentation still requires substantial improvement. I raise my score to 2.
We thank the reviewer for the detailed critique and concrete questions, which we answer with new experiments.
Q1: Pre-existing payload or new computation?
We trained constrained readers and tested transfer (G2).
We treat this as evidence that the reader exposes a representation already present in the carrier. It does not, however, separate a lookup from a computation by re-weighted attention and frozen MLPs. We will therefore soften “direct payload access” to “payload readout”.
Q2: How was the contrast chosen, and is it confirmed?
+.521,+.674It was chosen after the prespecified endpoint came out null. That endpoint could not succeed under any outcome, because the hypothesis predicts chance on sw (G1); we reported it in Appendix D because it was prespecified. With the contrast locked in advance, five new seeds per model give +.593
on Llama-3.1-8B and +.275
+.174,+.365on Mistral-7B. The Mistral replication is weaker, with one seed below base.
We agree with the broader point behind this question. The prespecified endpoint was mis-specified (G1), and the value-range experiment contained a demonstration confound (G2); we thank the reviewers, whose questions led us to find the second. All new experiments in this response use analyses fixed before training, and the revision holds the paper’s claims to the same standard.
Q3: Sensitivity to excluded seeds and families.
Reader exclusion (≥ .5 on both addressed roles under OPEN) was fixed before running any experiment. All seeds (except one in Mistral) improve: a = b + 1 binding becomes +.493 / +.570 (sx / sy) on Llama-3.1-8B and +.292 / +.347 on Mistral-7B, against ≈ .000 for the uninvolved variable (G1). From Table 5 in paper, the excluded seed binds shows +.305 / +.290 in Mistral and +.365 / +.385 in Llama. They exhibit relatively weakers result comparing to included seeds, but consistently follow the improving pattern over the base model.
The model gate (base OPEN > .5) was fixed before running any experiment. We have now trained readers on every family (G3). All gain payload, including DeepSeek-7B (.042 → .185), whose base model almost never executes the operation. Gains are smaller than for Llama, and Qwen2.5-7B in bf16 gains payload but not accuracy. We will drop the gate and report all families in the revised paper.
Q4: bf16 and on-distribution patches.
In a FRESH overwrite, we replace the keys and values at the operand token with those from the same prompt but the operation reads an unassigned value (Section 4.4 in paper), so they stay within the range of states the model normally produces. Table 9 in paper compares FRESH, MEAN and ZERO, and all three localize to the same band. We also reran localization with a bf16 writer and five bf16 readers:
| +.098
+.039,+.158| | Layers 12–15 | +.223
+.119,+.328| +.162
+.104,+.220| | Other six bands (max) | .000 | +.007
−.005,+.018|
The site is unchanged: only layers 8–15 are sufficient. The effect at 12–15 is somewhat smaller in bf16.
Necessity agrees: in bf16, overwriting the operand token’s K/V at layers 8–11 or 12–15 lowers the readers’ own accuracy by .178 and .217, against at most .013 for any other band. G3 also shows similar payload gains under NF4 and bf16 across five families.
Presentation**.** We agree and will make the following changes: rewrite Section 2 as a continuous methodology section, simplify Figures 1, 2 and 6, and move the central result tables into the main text.
Generative-AI use. As disclosed in our AI-use statement, generative AI assisted with experiment and figure code and with language editing. The authors reviewed all code and are responsible for the design, analyses and claims. We take the reviewer’s concern to rest on the analysis decisions discussed above, which we address directly: all seeds retained, thresholds’ timing, and the independent replication.