- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how recurrent computation contributes to answers in hybrid language models that maintain both recurrent states and attention KV caches. The authors argue that swapping the final recurrent state while holding the KV cache fixed can underestimate recurrence’s contribution: earlier recurrent computation may already have shaped the KV entries written during prefill. To test this pathway, they replace recurrent-block contributions with those from a paired context containing a different answer, allow downstream layers to recompute both memories, and then restore the original recurrent state or KV cache separately before presenting a shared query. The paper’s main contribution is thus a causal-attribution distinction between where answer-relevant computation occurs and where its effects persist. It does not establish a new architecture or a direct efficiency gain, and its conclusions are primarily supported by controlled interventions and tasks.
Critical strengths
- A well-defined causal question. The paper distinguishes the effect of a final recurrent state from the effect of recurrent computation during prefill. This clarifies why a weak final-state intervention does not necessarily imply that recurrence made little contribution to the answer.
- A two-stage intervention design. By first altering recurrent-block contributions and then independently restoring the recurrent state or KV cache, the experiments trace where the induced effect persists. This is more informative than swapping either final cache in isolation.
- Multiple controls and outcome measures. The authors use paired contexts with different answers, no-op and noise controls, answer-logit margins, preference reversals, and complete-answer outcomes. These checks make the interpretation less dependent on a single metric.
- Evidence across different hybrid architectures. The reported pattern appears across five checkpoints incorporating Gated DeltaNet or Mamba-2 recurrence, including dense and mixture-of-experts models. This makes it less likely to be an artifact of one checkpoint.
Critical weaknesses
-
The SQuAD experiment uses 200 pairs of controlled passages with answers spanning 2–5 tokens, most of which are proper names. This goes beyond single-token labels, but remains substantially different from long-form open-ended generation, multi-turn dialogue, or complex reasoning.
-
The localization of the effect within deeper KV caches remains coarse. Cumulative replacement and restoration show that deep attention caches are important, but the effects of multiple layers may be interdependent. The experiments do not establish which layers or token positions encode specific answer-relevant representations, and the contributions of individual layers cannot simply be added together.
-
The paper should discuss more concretely how its analytical findings could inform model training or inference.
-
My main concern is whether the paper’s novelty is sufficient for a top-tier conference publication.
Questions for authors
- Could the authors combine head-level analysis with targeted KV interventions at positions containing answer-relevant evidence and distractors to localize this pathway more precisely?