- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how recurrent models (RMs), which must compress incoming information into a fixed-size state, determine what to retain before knowing what will be queried. To study this, the authors introduce cued key–value recall, a task in which a cue presented before the records provides query-related information to guide retention. They evaluate cue utilization in pretrained, fine-tuned, and scratch-trained models. Among pretrained RMs, only RWKV-7 benefits from the cue, primarily through exact token matching rather than semantic understanding. Fine-tuning improves cue utilization for some models, although the behavior remains largely token-based. When trained from scratch on a synthetic version of the task, RWKV-7 learns to use the cue once its state capacity becomes insufficient to store all records. Further analysis suggests that the cue is retained in the first layer’s state. The authors also investigate whether inference-time memory pressure encourages the learning of generalizable representations. When models are trained solely to recall observed pixels, RWKV-7 learns to reconstruct unobserved portions of digits and characters, whereas a Transformer only recalls the observed pixels. A phase diagram further shows that this effect depends on both state size and cue informativeness. Based on these findings, the authors argue that memory constraints can promote representation learning rather than merely limiting model capacity.
Critical strengths
- S1. Well-motivated problem and systematic evaluation. The paper addresses an underexplored challenge: how recurrent models determine what to retain before the query is known. Cued key–value recall provides a simple and direct testbed, while evaluations across pretrained, fine-tuned, and scratch-trained models help distinguish the effects of pretraining from architectural capabilities.
- S2. An interesting perspective on inference-time memory pressure. The paper suggests that limited inference-time memory can encourage learning generalizable structure rather than simply copying inputs. In pixel recall experiments, RWKV-7 learns to complete unobserved regions. In contrast, a Transformer merely reproduces the observed pixels, supporting the potential role of memory constraints as an inductive bias.
Critical weaknesses
- W1. Limited generalizability across recurrent models. The experiments focus primarily on RWKV-7, making it difficult to generalize the findings to recurrent models (RMs) more broadly. Among the pretrained RMs evaluated, only RWKV-7 benefits from the cue, and most subsequent analyses rely exclusively on RWKV-7 as the representative RM. As a result, the paper's broader claims about RMs appear insufficiently supported by the experimental evidence.
- W2. Lack of analysis explaining RWKV-7's distinctive behavior. The paper does not investigate why RWKV-7 exhibits these behaviors while other RMs do not, particularly in terms of differences in recurrence mechanisms, model scale, or training data. Consequently, despite its broader framing around RMs, the study largely becomes a comparison between RWKV-7 and Transformers. A more detailed analysis of these differences would be valuable, even if other RMs fail to learn the cued key–value recall task, as it could help distinguish whether the observed behaviors arise from general properties of recurrent architectures or characteristics specific to RWKV-7.
Questions for authors
- Clarifications
- Q1. (Related to W1, W2) In Fig. 2f, RWKV-7 is the only pretrained RM that responds to the cue, and Sec. 3.4 notes that Gated DeltaNet and DeltaNet learn the cue policy less reliably even when trained from scratch with matched state capacity. Could the authors clarify what properties of RWKV-7 explain this difference? In particular, is it due to its recurrence design or other factors such as model scale or training data?
- Q2. When the number of records exceeds the state capacity, does recall performance differ between records presented early and late in the sequence, for both cued and uncued queries? For models that erase stale information through decay or the delta rule (e.g., Gated DeltaNet), one might expect a bias toward recent records. Does the cue provide greater benefits for early records, as selective storage would predict?
- Suggestion
- Q3 (Related to W1) Have the authors evaluated other RMs, such as Gated DeltaNet, on the experiments in Sec. 4, including the phase diagram? This would help clarify whether the effect of inference-time memory pressure generalizes across recurrent architectures.
3 comments
Thank you for your thorough review!
We are glad that the reviewer found a "well-motivated problem and systematic evaluation", that cued key–value recall "provides a simple and direct testbed" and "an interesting perspective on inference-time memory pressure".
The reviewer mainly pointed out that most analyses rely on RWKV-7, so it is unclear whether the findings hold for recurrent models in general, and why RWKV-7 behaves differently. Our rebuttals are below.
The experiments focus primarily on RWKV-7, making it difficult to generalize the findings to recurrent models (RMs) more broadly. ... As a result, the paper's broader claims about RMs appear insufficiently supported by the experimental evidence.
Thank you for raising this! Besides RWKV-7, the paper evaluates seven architectures as pretrained and fine-tuned language models and three recurrent architectures trained from scratch. After the same fine-tuning, DeltaNet and Gated DeltaNet also use the cue strongly (70 to 75 points). We have now extended every analysis that used RWKV-7 alone. The mechanism analysis covers Gated DeltaNet (Q1), and recall by position and the phase diagram of Sec. 4 cover Gated DeltaNet and DeltaNet (Q2, Q3). We also added pretrained Mamba-2, which behaves like Gated DeltaNet (App. C.1).
Every analysis that used RWKV-7 alone now covers other recurrent models, and the delta-rule models share RWKV-7's behavior once fine-tuned.
The paper does not investigate why RWKV-7 exhibits these behaviors while other RMs do not, particularly in terms of differences in recurrence mechanisms, model scale, or training data.
Could the authors clarify what properties of RWKV-7 explain this difference? In particular, is it due to its recurrence design or other factors such as model scale or training data?
Thank you for this question! Scale and training data surely matter for the pretrained models, and within RWKV-7 the cue effect grows with size (up to 52 points at 13.3B, Sec. 3.2), but larger checkpoints usually also see more data, so we cannot cleanly separate the two. We therefore focus on the recurrence itself, using from-scratch models matched in data, size and state. There RWKV-7 learns the policy in every seed, while Gated DeltaNet needs a warm start or longer training. The new mechanism analysis (new Fig. 19 in App. D.5) shows that both keep cued records in the same slow memory, the part of the state that forgets slowly, but RWKV-7 can write into it much more precisely, as it controls forgetting and writing per channel (over 300 slow channels) instead of per head (at most 11 slow heads). Per-channel forgetting alone is not enough though, since Kimi Delta Attention forgets per channel but writes per head and learns the policy less readily than Gated DeltaNet (App. D.5).
Scale and data matter but are hard to separate, and with everything else matched the difference lies in the recurrence, as RWKV-7 writes cued records into its slow memory far more precisely.
When the number of records exceeds the state capacity, does recall performance differ between records presented early and late in the sequence, for both cued and uncued queries? ... Does the cue provide greater benefits for early records, as selective storage would predict?
Thank you for this interesting question! We find, interestingly, that once a model has learned the cue policy, the position in the list doesn't really matter (new Fig. 20 in App. D.6, three seeds each). In both RWKV-7 and DeltaNet, uncued recall is about the same for early and late records even beyond capacity limits, so these models don't necessarily forget the oldest records first. Similarly, the cue helps every position pretty equally, and in RWKV-7 a cued record is recalled at 0.95 to 1.00 no matter its location, even with more than 900 records after it. Thus, the cue doesn't help early records more. The recency bias you expected does show up, but only in Gated DeltaNet runs that never learned the cue policy. These mostly keep the last records (0.00 to 0.01 recall for the first tenth of a 1,024-record list vs. 0.05 to 0.14 for the last tenth).
Once the cue policy is learned, position stops mattering and the cue protects a record wherever it is. Recency bias only appears when the policy was not learned.
(1/2)
Have the authors evaluated other RMs, such as Gated DeltaNet, on the experiments in Sec. 4, including the phase diagram? This would help clarify whether the effect of inference-time memory pressure generalizes across recurrent architectures.
Thank you for this suggestion! We ran the phase diagram experiment for Gated DeltaNet and DeltaNet (new Fig. 28 in App. F.3). We trained the four “corners” (512 or 16,384 floats, with an uninformative t = 4 or informative t = 0.25 cue) with three seeds each and the same recipe as Fig. 7. Both models show a similar pattern to RWKV-7 where a small state with an uninformative cue reconstructs unseen cells best, and removing either pressure increases the error. The following table gives the error on unseen cells (RMSE, pooled over seeds).
| Model | State | t = 4 | t = 0.25 |
|---|---|---|---|
| RWKV-7 (paper) | 512 | 0.63 | 1.65 |
| RWKV-7 (paper) | 16,384 | 1.01 | 1.19 |
| Gated DeltaNet | 512 | 0.73 | 1.53 |
| Gated DeltaNet | 16,384 | 1.31 | 1.44 |
| DeltaNet | 512 | 0.65 | 1.38 |
| DeltaNet | 16,384 | 0.95 | 2.21 |
The frozen letter classifier of Fig. 7e reads 85% (Gated DeltaNet) and 89% (DeltaNet) of the small-state, uninformative-cue reconstructions correctly (RWKV-7: 90%).
The inference-time pressure effect of Sec. 4 holds in all three recurrent architectures.
Overall, we have now addressed the reviewer's main concerns that the findings may not hold beyond RWKV-7 and that it is unclear why RWKV-7 differs. Every analysis that used RWKV-7 alone now covers other recurrent models, the effects of Secs. 3 and 4 hold beyond RWKV-7 and with scale and data matched RWKV-7's advantage comes from writing the cued record into its slow memory more precisely. Thank you again for the constructive questions!
These experiments are now in the appendix of the manuscript.
We hope that this addresses the reviewer's concerns, and please consider raising the score if so!
Please let us know if you have additional concerns or questions.
(2/2)
Sorry that our reply is quite late. Thank you for your thorough review, which motivated many new experiments! Here are the main points, and the full rebuttal follows.
If these points address your concerns, we would be grateful if you would consider raising your score!
Thank you very much!