acceptodds
Withdrawn from ICLR 2027

From Recurrent Computation to Attention Memory in Hybrid Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hybrid language models store context in recurrent state and an attention key-value (KV) cache. To understand how they use context, we need to trace how computation shapes the memory used for a later answer. Swapping the final recurrent state measures how much it affects the answer while holding the KV cache fixed. This leaves the effects of earlier recurrent computation on the KV cache unchanged. To trace these effects, we replace recurrent-block contributions during prefill, including their feed-forward sublayers in Qwen. The replacements come from a source context that specifies a different answer. We let later layers recompute both memories, then restore either one before the query. Most of the induced change in answer preference remains after restoring the original recurrent state. Across five hybrid models, retaining the modified KV cache preserves 67-93% of the mean shift in the answer-logit margin on exact retrieval. Restoring the original KV cache instead removes or weakens this effect. Layerwise tests locate the effect in the KV caches of deep attention layers. The effect also persists through complete-answer generation. Of 200 paired SQuAD passages with 2-5-token answers, both Qwen3.5-9B and Qwen3.6-27B answer both clean contexts correctly on 162 pairs. After restoring clean recurrent state, the modified KV cache preserves the complete source answer on 98.8% and 98.1% of this shared subset, respectively. These results support a compute-commit-retrieve pattern: Recurrent computation can change an answer through attention memory, even when swapping the final recurrent state has little effect on that answer.

Primary area:
interpretability and explainable AI
Keywords:
hybrid language models, mechanistic interpretability, recurrent computation, key-value cache, causal tracing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VUSqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how recurrent computation contributes to answers in hybrid language models that maintain both recurrent states and attention KV caches. The authors argue that swapping the final recurrent state while holding the KV cache fixed can underestimate recurrence’s contribution: earlier recurrent computation may already have shaped the KV entries written during prefill. To test this pathway, they replace recurrent-block contributions with those from a paired context containing a different answer, allow downstream layers to recompute both memories, and then restore the original recurrent state or KV cache separately before presenting a shared query. The paper’s main contribution is thus a causal-attribution distinction between where answer-relevant computation occurs and where its effects persist. It does not establish a new architecture or a direct efficiency gain, and its conclusions are primarily supported by controlled interventions and tasks.

Critical strengths
  • A well-defined causal question. The paper distinguishes the effect of a final recurrent state from the effect of recurrent computation during prefill. This clarifies why a weak final-state intervention does not necessarily imply that recurrence made little contribution to the answer.
  • A two-stage intervention design. By first altering recurrent-block contributions and then independently restoring the recurrent state or KV cache, the experiments trace where the induced effect persists. This is more informative than swapping either final cache in isolation.
  • Multiple controls and outcome measures. The authors use paired contexts with different answers, no-op and noise controls, answer-logit margins, preference reversals, and complete-answer outcomes. These checks make the interpretation less dependent on a single metric.
  • Evidence across different hybrid architectures. The reported pattern appears across five checkpoints incorporating Gated DeltaNet or Mamba-2 recurrence, including dense and mixture-of-experts models. This makes it less likely to be an artifact of one checkpoint.
Critical weaknesses
  • The SQuAD experiment uses 200 pairs of controlled passages with answers spanning 2–5 tokens, most of which are proper names. This goes beyond single-token labels, but remains substantially different from long-form open-ended generation, multi-turn dialogue, or complex reasoning.

  • The localization of the effect within deeper KV caches remains coarse. Cumulative replacement and restoration show that deep attention caches are important, but the effects of multiple layers may be interdependent. The experiments do not establish which layers or token positions encode specific answer-relevant representations, and the contributions of individual layers cannot simply be added together.

  • The paper should discuss more concretely how its analytical findings could inform model training or inference.

  • My main concern is whether the paper’s novelty is sufficient for a top-tier conference publication.

Questions for authors
  • Could the authors combine head-level analysis with targeted KV interventions at positions containing answer-relevant evidence and distractors to localize this pathway more precisely?
Reviewer_e8bJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how recurrent-block computation during prefill influences later answers in hybrid language models. It replaces block residual contributions using an alternative context, recomputes both memories, and separately restores clean recurrent state or attention KV cache. Across five checkpoints, the modified KV cache retains most of the induced mean answer-margin change. Layerwise tests, source-specific controls, arithmetic and controlled SQuAD generation support this pathway, while probing and suffix-transfer experiments document its boundaries.

Critical strengths
  1. The two-stage intervention addresses a meaningful limitation of final-state swaps: those swaps leave prior KV writes unchanged. Selective restoration directly tests which memory retains the induced prefill effect. No-op, third-source, noise, depth/count and sublayer controls, together with Nemotron’s separately unpatched FFNs, strengthen the bounded block-to-KV interpretation (Section 3, Table 2).

  2. Replication covers GDN and Mamba-2, dense and MoE checkpoints, and complete-answer generation rather than only pairwise logits. On all 200 SQuAD pairs, modified-KV source EM remains close to clean-source EM (84/85% versus 85/87.5%). Reporting residual state effects, the unsuccessful neutral-suffix transfer, and interaction-dependent retention ratios makes the interpretation appropriately bounded (Tables 3, 8 and 9).

Critical weaknesses
  1. The main contribution is a recurrent-state/KV decomposition in pretrained hybrids. Path patching and the prefill write–store–read account already have precedents, including Li (2026). The block-level intervention establishes a pathway but leaves the computation producing the answer-relevant representation unresolved (§§2–3). This limits explanatory depth; a clearer comparison with prior work would help establish the decomposition’s added insight.

  2. The experiments concentrate on controlled retrieval, simple arithmetic and short-answer substitutions (§4). The short, predominantly person-name SQuAD examples leave the pathway’s importance in broader compositional or long-context answering unclear. This limits the breadth of the contribution, while the full-pool results support the stated scope. Paired uncertainty estimates for Table 3’s retention ratios would also strengthen cross-model comparisons.

Questions for authors
  1. How would the authors state the empirical delta relative to Li (2026), Models Take Notes at Prefill? In particular, which conclusions require distinguishing recurrent state from attention KV, beyond the established write–store–read account?

  2. Could Table 3 include paired-bootstrap intervals for C_KV and, if readily available, the corresponding R_S and J/D values? This would clarify uncertainty and interactions without interpreting the retention ratios as natural causal shares.

  3. Which observed boundary most usefully distinguishes settings where the modified recurrent state contributes materially from those where KV nearly suffices? Please synthesize the existing Nemotron-30B and binding-load results.

  4. Beyond the existing retrieval, arithmetic and short-answer tests, how robust is this decomposition on multi-hop question answering? One representative task with evidence distributed across a longer context, retaining the same clean-run and state/KV restoration controls, would help assess the breadth of the finding.

Reviewer_rZf7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The idea in the paper is very interesting: replacing memory modules (recurrent and KV-cache) of a system processing a particular context with those of another system processing a different context or otherwise perturbing them and report on the empirical findings. One of the first observations is that the replacement of the KV cache alone (while keeping the recurrent state) is sufficient to change the answer. In section 4, more such empirical findings are collected and shared.

Critical strengths
  • The idea in the paper of injecting into a model, memory subsystems or signals from a sibling model - which however processes different context of different target is very interesting to gain further insights on the memory-computation interactions in hybrid language systems.

  • The metrics identified in Section 3 are very useful in this type of exploration but could be more broadly applicable (as they stand or generalized) in similar interventions approaches.

Critical weaknesses
  • It is very hard to follow the narrative and extract conclusions, especially across Section 4, which covers a large fraction of the manuscript. In some places the description becomes very high-level or the choice of vocabulary makes the ideas come out unnecessarily complex.
Questions for authors
  • Consider the extensive revision of the presentation part (language, etc) of these interesting ideas and their empirical findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.