acceptodds
Under review as a conference paper at ICLR 2027

Memory Pressure in Recurrent Models: Behavior, Mechanisms and Representations

56 views

Abstract

While you read this abstract, try to remember how many points fine-tuning adds to cued recall. This instruction is already changing how your brain registers this abstract, and arguably is a major difference between recurrent models (RMs) with finite state and Transformers, which can attend back to any context. Consequently, an RM must commit to its finite memory what it thinks will be relevant in the future and compress information on the go. This intuition behind state and data dependent memory drove recent architectural designs like Gated DeltaNet and RWKV-7. However, we find that language-trained RMs do not actually behave this way. In a task we call cued key–value recall, a cue that names the future query helps mostly through exact matching: the models barely process its meaning, and telling them to ignore a record still helps them recall it. Fine-tuning adds 89 points to RWKV-7’s cued recall (8% to 97% at 256 biographies), yet what the models learn remains largely token matching. Trained from scratch on a synthetic version of the task, RWKV-7 behaves rationally: uncued recall is limited by memory capacity, cued recall by length generalization, and a smaller state starts relying on the cue sooner. But the same limit that hurts recall can help learning: a finite state must compress, and we find that this inference-time compression pressure can actually push a model towards learning better features and concepts. When trained only to recall given pixels of a digit, an RM learns to draw the whole digit, while a Transformer trained on the same data and loss simply copies from its inputs. Finally, we show that these results compile into a phase diagram where task load and memory load both contribute to inference-time pressure, creating better-generalizing features. Now, do you remember how many points fine-tuning added? Perhaps the cue helped, even though you do not have quadratic attention.

Primary area:
interpretability and explainable AI
Keywords:
Memory, Recurrent Models, Transformers, Recall, Concept Learning, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8VonOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how recurrent models (RMs), which must compress incoming information into a fixed-size state, determine what to retain before knowing what will be queried. To study this, the authors introduce cued key–value recall, a task in which a cue presented before the records provides query-related information to guide retention. They evaluate cue utilization in pretrained, fine-tuned, and scratch-trained models. Among pretrained RMs, only RWKV-7 benefits from the cue, primarily through exact token matching rather than semantic understanding. Fine-tuning improves cue utilization for some models, although the behavior remains largely token-based. When trained from scratch on a synthetic version of the task, RWKV-7 learns to use the cue once its state capacity becomes insufficient to store all records. Further analysis suggests that the cue is retained in the first layer’s state. The authors also investigate whether inference-time memory pressure encourages the learning of generalizable representations. When models are trained solely to recall observed pixels, RWKV-7 learns to reconstruct unobserved portions of digits and characters, whereas a Transformer only recalls the observed pixels. A phase diagram further shows that this effect depends on both state size and cue informativeness. Based on these findings, the authors argue that memory constraints can promote representation learning rather than merely limiting model capacity.

Critical strengths
  • S1. Well-motivated problem and systematic evaluation. The paper addresses an underexplored challenge: how recurrent models determine what to retain before the query is known. Cued key–value recall provides a simple and direct testbed, while evaluations across pretrained, fine-tuned, and scratch-trained models help distinguish the effects of pretraining from architectural capabilities.
  • S2. An interesting perspective on inference-time memory pressure. The paper suggests that limited inference-time memory can encourage learning generalizable structure rather than simply copying inputs. In pixel recall experiments, RWKV-7 learns to complete unobserved regions. In contrast, a Transformer merely reproduces the observed pixels, supporting the potential role of memory constraints as an inductive bias.
Critical weaknesses
  • W1. Limited generalizability across recurrent models. The experiments focus primarily on RWKV-7, making it difficult to generalize the findings to recurrent models (RMs) more broadly. Among the pretrained RMs evaluated, only RWKV-7 benefits from the cue, and most subsequent analyses rely exclusively on RWKV-7 as the representative RM. As a result, the paper's broader claims about RMs appear insufficiently supported by the experimental evidence.
  • W2. Lack of analysis explaining RWKV-7's distinctive behavior. The paper does not investigate why RWKV-7 exhibits these behaviors while other RMs do not, particularly in terms of differences in recurrence mechanisms, model scale, or training data. Consequently, despite its broader framing around RMs, the study largely becomes a comparison between RWKV-7 and Transformers. A more detailed analysis of these differences would be valuable, even if other RMs fail to learn the cued key–value recall task, as it could help distinguish whether the observed behaviors arise from general properties of recurrent architectures or characteristics specific to RWKV-7.
Questions for authors
  • Clarifications
    • Q1. (Related to W1, W2) In Fig. 2f, RWKV-7 is the only pretrained RM that responds to the cue, and Sec. 3.4 notes that Gated DeltaNet and DeltaNet learn the cue policy less reliably even when trained from scratch with matched state capacity. Could the authors clarify what properties of RWKV-7 explain this difference? In particular, is it due to its recurrence design or other factors such as model scale or training data?
    • Q2. When the number of records exceeds the state capacity, does recall performance differ between records presented early and late in the sequence, for both cued and uncued queries? For models that erase stale information through decay or the delta rule (e.g., Gated DeltaNet), one might expect a bias toward recent records. Does the cue provide greater benefits for early records, as selective storage would predict?
  • Suggestion
    • Q3 (Related to W1) Have the authors evaluated other RMs, such as Gated DeltaNet, on the experiments in Sec. 4, including the phase diagram? This would help clarify whether the effect of inference-time memory pressure generalizes across recurrent architectures.
3 comments

Sorry that our reply is quite late. Thank you for your thorough review, which motivated many new experiments! Here are the main points, and the full rebuttal follows.

  • Analyses focus on RWKV-7: We extended every RWKV-7-only analysis to Gated DeltaNet (and DeltaNet) and added pretrained Mamba-2.
  • Why RWKV-7 differs: We think scale and data both matter, and with everything else matched the difference is in the recurrence, as RWKV-7 can write the cued record into its slow memory far more precisely.
  • Early vs. late records: We see no recency bias in models that learned the cue policy, and the cue helps at every position.
  • Sec. 4 with other recurrent models: We ran the phase diagram for Gated DeltaNet and DeltaNet, and it holds for both.

If these points address your concerns, we would be grateful if you would consider raising your score!

Thank you very much!

Thank you for your thorough review!

We are glad that the reviewer found a "well-motivated problem and systematic evaluation", that cued key–value recall "provides a simple and direct testbed" and "an interesting perspective on inference-time memory pressure".

The reviewer mainly pointed out that most analyses rely on RWKV-7, so it is unclear whether the findings hold for recurrent models in general, and why RWKV-7 behaves differently. Our rebuttals are below.



The experiments focus primarily on RWKV-7, making it difficult to generalize the findings to recurrent models (RMs) more broadly. ... As a result, the paper's broader claims about RMs appear insufficiently supported by the experimental evidence.

Thank you for raising this! Besides RWKV-7, the paper evaluates seven architectures as pretrained and fine-tuned language models and three recurrent architectures trained from scratch. After the same fine-tuning, DeltaNet and Gated DeltaNet also use the cue strongly (70 to 75 points). We have now extended every analysis that used RWKV-7 alone. The mechanism analysis covers Gated DeltaNet (Q1), and recall by position and the phase diagram of Sec. 4 cover Gated DeltaNet and DeltaNet (Q2, Q3). We also added pretrained Mamba-2, which behaves like Gated DeltaNet (App. C.1).

Every analysis that used RWKV-7 alone now covers other recurrent models, and the delta-rule models share RWKV-7's behavior once fine-tuned.


The paper does not investigate why RWKV-7 exhibits these behaviors while other RMs do not, particularly in terms of differences in recurrence mechanisms, model scale, or training data.

Could the authors clarify what properties of RWKV-7 explain this difference? In particular, is it due to its recurrence design or other factors such as model scale or training data?

Thank you for this question! Scale and training data surely matter for the pretrained models, and within RWKV-7 the cue effect grows with size (up to 52 points at 13.3B, Sec. 3.2), but larger checkpoints usually also see more data, so we cannot cleanly separate the two. We therefore focus on the recurrence itself, using from-scratch models matched in data, size and state. There RWKV-7 learns the policy in every seed, while Gated DeltaNet needs a warm start or longer training. The new mechanism analysis (new Fig. 19 in App. D.5) shows that both keep cued records in the same slow memory, the part of the state that forgets slowly, but RWKV-7 can write into it much more precisely, as it controls forgetting and writing per channel (over 300 slow channels) instead of per head (at most 11 slow heads). Per-channel forgetting alone is not enough though, since Kimi Delta Attention forgets per channel but writes per head and learns the policy less readily than Gated DeltaNet (App. D.5).

Scale and data matter but are hard to separate, and with everything else matched the difference lies in the recurrence, as RWKV-7 writes cued records into its slow memory far more precisely.


When the number of records exceeds the state capacity, does recall performance differ between records presented early and late in the sequence, for both cued and uncued queries? ... Does the cue provide greater benefits for early records, as selective storage would predict?

Thank you for this interesting question! We find, interestingly, that once a model has learned the cue policy, the position in the list doesn't really matter (new Fig. 20 in App. D.6, three seeds each). In both RWKV-7 and DeltaNet, uncued recall is about the same for early and late records even beyond capacity limits, so these models don't necessarily forget the oldest records first. Similarly, the cue helps every position pretty equally, and in RWKV-7 a cued record is recalled at 0.95 to 1.00 no matter its location, even with more than 900 records after it. Thus, the cue doesn't help early records more. The recency bias you expected does show up, but only in Gated DeltaNet runs that never learned the cue policy. These mostly keep the last records (0.00 to 0.01 recall for the first tenth of a 1,024-record list vs. 0.05 to 0.14 for the last tenth).

Once the cue policy is learned, position stops mattering and the cue protects a record wherever it is. Recency bias only appears when the policy was not learned.

(1/2)

Have the authors evaluated other RMs, such as Gated DeltaNet, on the experiments in Sec. 4, including the phase diagram? This would help clarify whether the effect of inference-time memory pressure generalizes across recurrent architectures.

Thank you for this suggestion! We ran the phase diagram experiment for Gated DeltaNet and DeltaNet (new Fig. 28 in App. F.3). We trained the four “corners” (512 or 16,384 floats, with an uninformative t = 4 or informative t = 0.25 cue) with three seeds each and the same recipe as Fig. 7. Both models show a similar pattern to RWKV-7 where a small state with an uninformative cue reconstructs unseen cells best, and removing either pressure increases the error. The following table gives the error on unseen cells (RMSE, pooled over seeds).

ModelStatet = 4t = 0.25
RWKV-7 (paper)5120.631.65
RWKV-7 (paper)16,3841.011.19
Gated DeltaNet5120.731.53
Gated DeltaNet16,3841.311.44
DeltaNet5120.651.38
DeltaNet16,3840.952.21

The frozen letter classifier of Fig. 7e reads 85% (Gated DeltaNet) and 89% (DeltaNet) of the small-state, uninformative-cue reconstructions correctly (RWKV-7: 90%).

The inference-time pressure effect of Sec. 4 holds in all three recurrent architectures.



Overall, we have now addressed the reviewer's main concerns that the findings may not hold beyond RWKV-7 and that it is unclear why RWKV-7 differs. Every analysis that used RWKV-7 alone now covers other recurrent models, the effects of Secs. 3 and 4 hold beyond RWKV-7 and with scale and data matched RWKV-7's advantage comes from writing the cued record into its slow memory more precisely. Thank you again for the constructive questions!

These experiments are now in the appendix of the manuscript.

We hope that this addresses the reviewer's concerns, and please consider raising the score if so!

Please let us know if you have additional concerns or questions.

(2/2)

Reviewer_zJQmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether recurrent models use advance cues to retain relevant information and whether bounded recurrent memory encourages generalizable representations. Biography-recall experiments show that pretrained models make limited use of semantic cues, with the strongest benefits generally associated with exact matches; fine-tuning substantially improves recall and yields some use of non-exact cues in RWKV-7. Synthetic experiments demonstrate learned cue-dependent recall, its dependence on memory and training length, and a causal role for the first-layer cue state. A second set of experiments trains models to recall observed pixels or values and tests prediction of unobserved ones. Recurrent models and some memory-restricted Transformers learn structure that supports completion and downstream readout. A sweep over state size and cue informativeness connects the two themes, presenting memory pressure as a controlled route to structured representations rather than a universal benefit.

Critical strengths
  1. The paper reports interesting observations about recurrent memory. In particular, pretrained models make limited use of semantic cues, and memory constraints can encourage generalization beyond the recall task used for training.
  2. The experiments are well designed. Comparisons between pretrained, fine-tuned, and from-scratch models help separate the effects of training from architectural limitations. State interventions, memory-size sweeps, and comparisons with memory-limited Transformers provide useful controls for the proposed explanations.
Critical weaknesses

The paper shows interesting effects, but does not explain them far enough. The state interventions show where the cue is held and which layers matter, but leave the architectural differences largely unexplained. Gated DeltaNet also learns the policy with a warm start or more training, so the difference is in how readily the models learn it. For pretrained language models, recall accuracy alone does not show whether the cue changes what is stored or how the answer is retrieved. These are useful observations, but I do not find the explanation strong enough to make them a sufficient contribution for acceptance.

Questions for authors
  1. Do the interventions explain why RWKV-7 learns the cue policy more readily than Gated DeltaNet?
3 comments

Sorry that our reply is quite late, and thank you for your review! Here are the main points, and the full rebuttal follows.

  • Why RWKV-7 learns the cue policy more readily: We find that both models use the same policy and the same slow memory. RWKV-7 can write the cued record into its slow memory far more precisely due to per-channel forgetting but this alone is not enough (KDA).
  • Gated DeltaNet learns it with a warm start: We agree. The difference is how easily training finds the policy, not whether the model can support the computation in general.
  • Storage vs. retrieval in pretrained LMs: We ran two new experiments. We find that the cue placed after the records never helps, and replacing the write at the cued fact removes most of the benefit.

If these points address your concerns, we would be grateful if you would consider raising your score!

Thank you very much!

Thank you for your review!

We are glad that the reviewer found our paper reports "interesting observations about recurrent memory", that "the experiments are well designed" and that our state interventions and sweeps "provide useful controls for the proposed explanations".

The reviewer mainly pointed out that the paper shows where the cue is held but does not explain far enough why RWKV-7 learns the cue policy more readily than Gated DeltaNet. Our rebuttals are below.



The state interventions show where the cue is held and which layers matter, but leave the architectural differences largely unexplained.

Do the interventions explain why RWKV-7 learns the cue policy more readily than Gated DeltaNet?

Thank you for this question! We have now measured what the cue does to the write (new Fig. 19 in App. D.5, three seeds per architecture). Both architectures use the same policy. They recognize the cued key while writing and keep cued records in the slow part of the state, since copying the slower half of the state from a cued run into an uncued one restores all of the cue benefit and the faster half restores none. In RWKV-7 a single signal carries the cue. Replacing only the erase strength aa at the cued record drops its recall from 1.00 to 0.13–0.38, while in Gated DeltaNet the cue is spread over several signals. The difference is how precisely each model can write a record into its slow memory, the part of the state that forgets slowly. RWKV-7 controls both forgetting and writing per channel, so on a 256-record list it has over 300 slow channels to write into. Gated DeltaNet controls both per head and has at most 11 slow heads, and we believe this is why training finds the policy less readily. Per-channel forgetting alone is not enough, as Kimi Delta Attention forgets per channel but writes per head and learns the policy less readily than Gated DeltaNet (App. D.5).

Both models use the same policy and slow memory. RWKV-7 finds it more readily because it can write the cued record into its slow memory far more precisely, and per-channel forgetting alone is not enough.


Gated DeltaNet also learns the policy with a warm start or more training, so the difference is in how readily the models learn it.

Thank you, we agree! This is also what Sec. 3.4 and App. D.2 report, where Gated DeltaNet learns the policy in every seed with a warm start. The new analysis supports this. A successful Gated DeltaNet run stores cued records in the same place as RWKV-7, while failed runs never recognize the cued key.

The difference is how easily training finds the policy, not whether the model can hold it.


For pretrained language models, recall accuracy alone does not show whether the cue changes what is stored or how the answer is retrieved.

Thank you for this point! We now test this directly in two ways (new App. C.5). First, we placed the same "remember A" cue after the records and right before the question. This never helps (−10 to 0 points in pretrained RWKV-7 and in fine-tuned RWKV-7, Gated DeltaNet and DeltaNet), while putting it before the records it adds 12 to 76 points. Second, in pretrained RWKV-7 g1 0.4B we replaced the write at the cued fact with the write from a run that cued a different fact. This removes 83% of the cue advantage (53% near the start of the list), and aa is again the largest single part, as in the from-scratch models.

The cue helps only when read before the records and acts through the write, so it changes what is stored, not how it is retrieved.



Overall, we hope to have addressed the reviewer's main concern that the paper does not explain why RWKV-7 learns the cue policy more readily than Gated DeltaNet. We find that Both models use the same policy and slow memory, RWKV-7 finds it more readily because it can write the cued record into its slow memory far more precisely and in pretrained models the cue changes what is stored rather than how it is retrieved. Thank you again for the helpful review!

These experiments are now in the appendix of the manuscript.

We hope that this addresses the reviewer's concerns, and please consider raising the score if so!

Please let us know if you have additional concerns or questions.

Thanks for the detailed response and the new experiments. They address my main concerns, so I’m raising my rating.

Reviewer_6MTKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how fixed-state recurrent models handle memory capacity limits compared to full-attention Transformers across two settings: prospective cued recall (where future relevance is known via an advance cue) and inference-time compression (where future queries are unknown). In the first part, the authors test 15 pretrained LMs on a synthetic biography benchmark with advance cues. Most architectures (DeltaNet, Gated DeltaNet, GLA, RetNet) barely use the cues. While RWKV-7 improves with cues, both pretrained and fine-tuned models mostly rely on surface-level token matching rather than semantic understanding—for example, an adversarial "ignore A" cue actually boosts recall of A.

In the second part, the authors train bounded-state models on literal (x, y, value) pixel recall when future queries are unknown. They show that state compression forces the models to learn structural representations that generalize to unseen MNIST/EMNIST digit completion, held-out Hanzi stroke reconstruction, and few-shot classification, summarized in a 2D phase diagram over state capacity and cue informativeness.

Critical strengths
  • The paper addresses the important problem of the memory mechanism of RMs with fixed-sized states, namely, whether input-dependent gating and decay mechanisms truly enable the model to learn on-demand memory retention during training. It also studies the interplay between the constraints that limited memory imposes on precise recall and how compression during inference can facilitate feature learning.
  • It contains well-designed synthetic experiments with a "prompted key-value recall" task, that showcase the recall curve without prompts is determined by a fixed state capacity, whereas the recall curve with prompts expands as the training length increases.
  • It reveals that inference-time compression pressure encourages feature and concept learning in RMs.

The benchmark effectively exposes the gap between the motivation for input-dependent gating and how pretrained models actually behave—specifically that they mostly act as surface token matchers and fail on semantic or negated cues.

Convincing mechanistic analysis: The synthetic experiments and layer-wise state transplantations nicely explain why selective writing requires at least two layers in linear RNNs (since gates depend on layer inputs rather than the recurrent state, requiring Layer 1 to hold the cue for Layers 2–4).

Interesting perspective on compression vs. generalization: Showing that a purely literal pixel-recall objective under a tight state budget forces the model to learn generalizable spatial features (both in RWKV-7 and KV-budgeted Transformers) is a neat finding.

Thorough controls: The empirical setup includes solid baselines, such as ink-matched wrong-digit controls, OOD EMNIST transfer, and clear reporting of seed counts across experiments.

Critical weaknesses
  • Several core experiments (Section 3.3 and 3.4) relied only on a single seed, while the experiment involving three seeds showed significant instability in the results.

  • The majority of the experiments and all the experiments for model trained from scratch are conducted on very small model sizes (at most 18.2M parameters according to the caption of Figure 17 in the appendix) and synthetic tasks, lacking large-scale, real-world training and validation.

More details: Missing SSM and test-time training baselines: Sections 1 and 2 heavily motivate the paper around selective state-space models and test-time learning architectures, but neither family is evaluated in Sections 3 or 4. Including models like Mamba/Mamba-2 (Gu & Dao, 2024; Dao & Gu, 2024), TTT (Sun et al., 2025), or Titans (Behrouz et al., 2024) in the pretrained or synthetic comparisons would make the empirical claims much more complete.

The pretrained evaluation compares 100B-token base checkpoints (DeltaNet, Gated DeltaNet, GLA, RetNet) against multi-trillion-token instruction-tuned RWKV-7 checkpoints. The fine-tuning setup does not penalize retaining an ignored entity and omits adversarial cues during training. There is no post-document cue control to separate write-time selective storage from query-time readout priming.

Several main-text results—including LM fine-tuning (Fig. 3), the main synthetic cued-recall curves (Fig. 4), and the MNIST/EMNIST experiments (Figs. 5, 6a–b)—only report a single seed (and use unmatched LR schedules for EMNIST classification). Given the high seed and optimizer sensitivity shown in the appendix for Gated DeltaNet (Fig. 16) and Hanzi KV-512 (Fig. 22b), multi-seed runs are needed here.

Evaluating all 384 unseen grid cells from a single cue introduces a test-time distribution shift for models trained to only store a small local radius around the cue. This setup also uses a fixed 192-cell observation mask across scenes and lacks an explicit uncued training baseline.

Questions for authors

address weakness points...

3 comments

Sorry that our reply is quite late. Thank you for your thorough review, which motivated many new experiments! Here are the main points, and the full rebuttal follows.

  • Single seed: We now ran multiple seeds for every main-text result, and all of them replicate.
  • Small synthetic models: We think small models are the right tool to study how an algorithm forms, and the same behavior and mechanism appear in pretrained models at 13.3B.
  • No SSM / TTT: We added pretrained Mamba-2, which behaves like Gated DeltaNet. Linear TTT and Titans also use the delta rule, but we could not run them ourselves in time due to kernel support.
  • 100B vs. 1T+ token checkpoints: We believe retrieval circuits form early in pretraining, and our architecture claim rests on matched comparisons.
  • No adversarial cues in fine-tuning: We made this design choice because the adversarial cues are the test.
  • No post-document cue control: We ran it and placed the same cue after the records, it doesn't help recall.
  • Single cue, fixed mask, no uncued baseline: We use the shift to measure what a model stores beyond its training need, and we added the uncued baseline, which lands where the phase diagram predicts.

If these points address your concerns, we would be grateful if you would consider raising your score!

Thank you very much!

Thank you for your thorough review!

We are glad that the reviewer found that our paper "addresses the important problem of the memory mechanism" of recurrent models, that our mechanistic analysis is "convincing" and that our controls are "thorough".

The reviewer mainly pointed out that some results rest on a single seed and on small models, and that some baselines and controls are missing. Our rebuttals are below.



Several core experiments (Section 3.3 and 3.4) relied only on a single seed, while the experiment involving three seeds showed significant instability in the results.

Several main-text results—including LM fine-tuning (Fig. 3), the main synthetic cued-recall curves (Fig. 4), and the MNIST/EMNIST experiments (Figs. 5, 6a–b)—only report a single seed (and use unmatched LR schedules for EMNIST classification). Given the high seed and optimizer sensitivity shown in the appendix for Gated DeltaNet (Fig. 16) and Hanzi KV-512 (Fig. 22b), multi-seed runs are needed here.

Thank you for raising this, it is a fair point. We have now rerun all main-text experiments which had a single seed, and the seed table is now in App. G. In short, everything replicates and in fact the seed variance is very small for the RWKV-7 results we claim.

For Fig. 3 we re-fine-tuned all models with a new data seed and order. Fine-tuning adds 88.5 and 90.6 points to cued recall in the two runs. Fig. 4 was already covered by App. D.2 (three to nine seeds). For MNIST/EMNIST we trained 3 seeds per model; missing-pixel completion is 0.862 ± 0.001 for RWKV-7 vs. 0.503 ± 0.003 for the Transformer, and EMNIST transfer is 0.802 ± 0.002, i.e. essentially negligible seed dependence. We also redid the EMNIST classification with the same learning rate for both models, and RWKV-7 is still better (0.39 ± 0.06 vs. 0.15 ± 0.01 at 16 labels per class).

There is one place where RWKV-7 itself moves between seeds, namely few-label classification (0.36 to 0.52 at 16 labels per class), but even the worst seed is still around 2x the Transformer. The big seed effects in Fig. 16 and Fig. 22b (now Fig. 24b) come from the comparison models. Gated DeltaNet sometimes finds the cue policy in the budget and sometimes not (that's actually our acquisition result in Sec. 3.4), and the bounded Hanzi Transformer worked in 2 of 3 seeds, which we mention in the main text too.

So overall every main-text result now has multiple seeds and replicates, the big seed effects being in the comparison models, as we claimed. Thanks a lot for pushing us on this, we think the paper got clearly stronger!


The majority of the experiments and all the experiments for model trained from scratch are conducted on very small model sizes (at most 18.2M parameters according to the caption of Figure 17 in the appendix) and synthetic tasks, lacking large-scale, real-world training and validation.

Yes, the from-scratch models are small. We chose this since we want to study how an algorithm forms, in our case whether a model learns to use a cue to decide what to write. We believe big LLM studies are indispensable for “knowledge” studies (Allen-Zhu & Li, 2024), but our task needs almost no knowledge. People have mostly used small synthetic setups for this kind of question, e.g. for induction heads (Olsson et al., 2022), in-context learning (Xie et al., 2021) or mechanistic comparisons of architectures (Arora et al., 2025).

These findings do carry over to real pretrained models. The cue effect actually gets bigger with scale (up to 13.3B), and when we swap the write at the cued fact in pretrained RWKV-7, up to 83% of the cue advantage is gone (new App. C.5), which is close to our small models.

In our view small synthetic models are the right tool to look at how an algorithm forms, and pretrained models up to 13.3B show the same thing.


(1/2)

Missing SSM and test-time training baselines: Sections 1 and 2 heavily motivate the paper around selective state-space models and test-time learning architectures, but neither family is evaluated in Sections 3 or 4. Including models like Mamba/Mamba-2 (Gu & Dao, 2024; Dao & Gu, 2024), TTT (Sun et al., 2025), or Titans (Behrouz et al., 2024) in the pretrained or synthetic comparisons would make the empirical claims much more complete.

Thank you for this suggestion! We have now evaluated pretrained Mamba-2 (1.3B and 2.7B, App. C.1) and it looks a lot like Gated DeltaNet: the cue only helps for short lists (up to 14 points), and "ignore A" still improves recall of A, so Mamba-2 too seems to react to the cue's tokens and not really its meaning. For TTT and Titans with a linear memory, the update is simply the delta rule, as in DeltaNet, Gated DeltaNet and RWKV-7. Unfortunately we couldn't run TTT/Titans themselves in time because of kernel support.

Pretrained Mamba-2 behaves like our other recurrent models. We could not test TTT and Titans themselves, but their linear versions share the delta rule of DeltaNet, Gated DeltaNet and RWKV-7.


The pretrained evaluation compares 100B-token base checkpoints (DeltaNet, Gated DeltaNet, GLA, RetNet) against multi-trillion-token instruction-tuned RWKV-7 checkpoints.

We agree that this comparison alone is not perfectly fair. Our intuition is that a circuit like this forms early in pretraining, similar to induction heads (Olsson et al., 2022), and thus 100B tokens should be enough. To isolate the architecture, we fine-tune one 1.3B, 100B-token suite in exactly the same way (DeltaNet and Gated DeltaNet then gain 70 to 75 points from the cue, GLA at most 26 and RetNet almost nothing), and the from-scratch models are matched in data, size and state.

We believe such circuits form early in pretraining, and we isolate the architecture with matched comparisons (one fine-tuned 1.3B suite and matched from-scratch models).


The fine-tuning setup does not penalize retaining an ignored entity and omits adversarial cues during training.

This was a design choice since the adversarial cues are indeed the test itself. If "ignore A" was followed by a question about A in training, the best strategy would be to just keep whatever the cue mentions, which is exactly the token matching we want to detect. Penalizing retention also wouldn't gain much, as "ignore B" only frees 1 of N records.

Adversarial cues stay out of training since they are our test.


There is no post-document cue control to separate write-time selective storage from query-time readout priming.

Thank you for this suggestion, we ran it! We kept the prompts identical and only moved the "remember A" cue to after the records and right before the question (new App. C.5). In the usual position the cue adds 12 to 76 points, while when placed after the records it doesn't help in any of the four models we tried (−10 to 0 points). This also matches our state experiments, where inserting the cue-carrying state only restores recall if we do it before the records are read (App. D.4), and where swapping just the write at the cued record kills the benefit (in our small models and in pretrained RWKV-7).

The cue has to come before the records and acts via the write gate, i.e. it changes what gets stored rather than the readout.


Evaluating all 384 unseen grid cells from a single cue introduces a test-time distribution shift for models trained to only store a small local radius around the cue. This setup also uses a fixed 192-cell observation mask across scenes and lacks an explicit uncued training baseline.

We actually use this shift to see how much of the field a model keeps beyond what training asks for. None of our models are trained on unseen cells, and the test is indeed unseen generalization. We fixed the mask so that unseen cells never show up in training, and in MNIST, where the mask changes every episode, we see the same effect. We also trained the uncued baseline you asked for (new App. F.4). With 512 floats it gets 0.57 unseen RMSE, so it's close to the uninformative cue (0.63) and nowhere near the informative one (1.65).

The shift is the experiment itself, random masks give the same effect, and the uncued baseline lands right where the phase diagram predicts.



Overall, every main-text result now has multiple seeds, we added the baselines and controls you asked for (pretrained Mamba-2, an uncued baseline, matched learning rates and a post-document cue) and our claims rest on matched comparisons and direct interventions on the write. Thank you again for the detailed and constructive review!

These experiments are now in the appendix of the manuscript.

We hope that this addresses the reviewer's concerns, and please consider raising the score if so!

Please let us know if you have additional concerns or questions.

(2/2)

Comments on the paper

1 comment

Dear Reviewers,

Thank you all for the thoughtful and careful reviews! We are glad that the reviewers found that our paper "addresses the important problem of the memory mechanism" of recurrent models with a "convincing mechanistic analysis" and "thorough controls" (6MTK), that it reports "interesting observations about recurrent memory" with experiments that "are well designed" (zJQm), and that it studies a "well-motivated problem" with a "systematic evaluation" and "an interesting perspective on inference-time memory pressure" (8Von).

The reviewers mainly asked for three items: (1) more seeds, baselines and controls (6MTK), (2) evidence that our findings hold beyond RWKV-7 (8Von), and (3) a better explanation of why RWKV-7 learns the cue more easily than Gated DeltaNet, and whether the cue changes what is stored vs. how it is retrieved (zJQm, 8Von).

To address the comments, we ran:

  • App. G and Figs. 3, 5, 6b: multiple seeds for every main-text result (6MTK). A second independent fine-tuning run replicates Fig. 3, and three seeds per model replicate the MNIST/EMNIST results with very small seed variance.
  • Fig. 23b: matched learning rates for EMNIST classification (6MTK). With the same schedule for both models, RWKV-7 still classifies letters better.
  • Fig. 9 and App. C.1: pretrained Mamba-2 (6MTK, 8Von). Mamba-2 (1.3B and 2.7B) behaves like Gated DeltaNet, with a short-range cue effect that follows the cue's tokens.
  • App. C.5: storage vs. retrieval in language models (6MTK, zJQm). Moving the same cue to after the records removes its benefit in all four models we tested, and in a pretrained RWKV-7 replacing the write at the cued fact removes up to 83% of the cue advantage.
  • Fig. 19 and App. D.5: what the cue does to the write (zJQm, 8Von). RWKV-7 and Gated DeltaNet use the same policy and the same slow memory, but RWKV-7 can write the cued record into it far more precisely. Per-channel forgetting alone is not enough, as a Kimi Delta Attention control shows.
  • Fig. 20 and App. D.6: recall by position (8Von). Once the cue policy is learned, the position of a record in the list doesn't matter and the cue protects it wherever it is.
  • Fig. 28 and App. F.3: the phase diagram for Gated DeltaNet and DeltaNet (8Von). The inference-time pressure effect of Sec. 4 holds in all three recurrent architectures (24 new runs).
  • App. F.4: an uncued phase-diagram baseline (6MTK). A model trained without any cue reconstructs the letters like the uninformative cue, as the phase diagram predicts.

We also updated the main text accordingly, including the seed counts and limitations, a short summary of the new mechanism in Sec. 3.4, and our reasoning for studying algorithm formation in small models.

We believe these additions made the paper clearly stronger, and we thank the reviewers again for their time and constructive feedback!

We address the individual points in the replies below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.