acceptodds
Under review as a conference paper at ICLR 2027

RAM: Routed Associative Memories Learn to Recall in Recurrent Language Models

Abstract

Recurrent sequence mixers such as Mamba-2, Gated DeltaNet, and Kimi Delta Attention offer a more efficient alternative to softmax attention, but lag behind on in-context recall. On synthetic MQAR, an Online Hebbian MLP achieves information-theoretically optimal storage scaling and matches attention KV-cache capacity with up to fewer state scalars. Yet pretrained recurrent language models recall substantially worse than attention. To study this discrepancy, we introduce noisy MQAR, which combines in-context recall with competing next-token supervision. Using a polynomial-kernel memory family, we identify gradient interference from the background objective and greater parameter sensitivity of low-degree recall margins as two obstacles to learning recall circuits. We address these obstacles with Routed Associative Memory (RAM), which upweights recall-relevant tokens and routes computation across multiple recurrent memories. At 1.4B parameters, RAM closes 56.02% of the average-recall gap to attention with 0.51% more parameters than Gated DeltaNet, while its LM-Evals score is within 0.30 percentage points of the Gated DeltaNet baseline. At 360M parameters, adding -gram history to the router closes the average-recall gap to attention entirely with two memories and 94.32% of it with eight.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recurrent language models, linear attention, in-context recall, associative memory, long-context modeling, memory capacity, mixture of memories, efficient sequence modeling

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_3NDtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that recurrent language models have enough memory capacity for recall, but ordinary language training prevents them from learning to use it well. Concretely, the authors find that in multi-query associative recall (MQAR), recurrent language models often have higher memory capacity per parameter then softmax attention models. Through analyses, they identify two causes of this learning failure, which is supervision competition from non-recall tokens and parameter sensitivity of linear attention. Its method, RAM, routes tokens across multiple memories and gives extra training weight to recall examples. At 1.4B parameters, RAM closes 56% of the measured recall gap to attention, with little change in general benchmark scores. Adding n-gram router further closes this recall gap.

Critical strengths
  • Clear problem diagnosis: It separates memory capacity from the ability to learn recall during training.
  • Useful controlled experiments: Synthetic tasks show how competing training objectives disrupt recall.
  • Meaningful improvements: Routing and recall weighting improve several recall tasks, with ablations showing their separate effects.
  • Small parameter overhead: The highlighted 1.4B result adds only 0.51% more parameters.
Critical weaknesses
  • Limited evaluation: The current experiment only verify the proposed method on one single model scale and training data scale, and with limited variety in tasks.
    • I strongly suggest the authors to evaluate on benchmarks such as RULER, LongBench, LongPPL, etc. Moreover, only three recall-intensive tasks is not quite sufficient.
    • I think they should also train smaller models and with varying training data scale to verify that the performance gains are consistent across model scales.
    • Also, in practice, recurrent mixers are often adopted in hybrid models that incorporate both recurrent mixers and softmax attention. Therefore, it is very important to verify that the proposed method is useful in such hybrid architectures as well.
  • The explanation is suggestive, not conclusive: Gradient analysis uses simplified synthetic models rather than directly establishing the same mechanism in pretrained RAM.
  • Efficiency is insufficiently measured: Small parameter overhead does not establish low inference cost, especially with multiple memories and dense reads.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3NDt

Thank you for the detailed review! We're glad you found that our diagnosis clearly separates memory capacity from learning recall, that the controlled experiments were useful, and that RAM's improvements come at a small parameter overhead.

Further, thank you for the thorough feedback leading to the improvement of our work. We address your specific weaknesses and questions below. Please also see the common response for a) RAM on Mamba-2 at 1.4B and broader evaluations, b) efficiency benchmarks, c) the recall mechanism measured in pretrained LMs, and d) the n-gram router, now reported as a diagnostic.

(W1a) Evaluation. Thanks for these important suggestions! We substantially expanded the evaluations (App. B.4):

  • Recall: besides FDA, SWDE and SQuAD, we now report NIAH (single, multi-key, multi-value, multi-query; Tables 15 and 20), the RULER suite at 2K/4K/8K (App. B.4.3, Fig. 15, Table 23), and open-book QA (TriviaQA, NQ, DROP; App. B.4.2, Fig. 14, Table 22).
    • With four memories, RAM improves RULER over GDN at every length. At 2K, RAM M4 scores 28.4 vs. 20.6 for GDN.
    • On open-book QA, RAM M8 4×4\times improves on GDN by +3.3+3.3 points at 1.4B.
  • LongBench and LongPPL: at our scales ($\leq$1.4B parameters, 10B tokens), these benchmarks are too challenging.
    • On LongBench, every model, including attention with YaRN, is near the floor ($\sim$14 on average).
    • On LongPPL, more memories help (perplexity of 102 for M8 vs. 170 for M1; lower is better), but perplexities remain above 100, so these numbers are not very meaningful.
    • Note that all our models were trained at 2K context, while both benchmarks evaluate at 8K, so we expect performance to degrade accordingly.

(W1b) Model and data scale. Thanks for this question. The paper already evaluates two scales: 360M parameters / 3B tokens and 1.4B parameters / 10B tokens (Table 1). We also add the following new results:

  • A second mixer at 1.4B: RAM on Mamba-2 closes 69% of its recall gap to attention (Table 1).
  • More data: for the camera-ready, we commit to train the 1.4B models on 30B tokens.

(W1c) Hybrid models. Thanks for this important question! In this work, our goal is to study how much of the recall gap pure recurrent mixers can close, since recall is exactly where they struggle (Zoology, Arora et al., 2023; Based, Arora et al., 2024; Jelassi et al., 2024). Adding attention layers would mix their recall with the recurrent mixer's, so we keep hybrid architectures out of scope for this paper.

(W2) Mechanism in pretrained models. Thank you for this suggestion. We now test the Sec. 4 account directly in our 360M pretrained models. Please refer to item 3 of the common response for more details.

(W3) Efficiency. We now include training and inference benchmarks at the 1.4B scale (App. A.3), along with pseudocode for the kernel implementations (App. A.2). Please refer to item 2 of the common response for more details.

AuthorsOpenReview ↗

Following Up on Response to Reviewer 3NDt

Thank you again for reviewing our work! As the discussion period ends shortly, we want to check if you have any further questions or found our responses helpful?

Please let us know and thank you for your time!

Reviewer_nFSKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why recurrent mixers (Mamba-2, Gated DeltaNet, KDA) lag behind softmax attention on recall after LM pretraining despite having optimal synthetic state capacity. Attributing this to gradient interference from general next-token prediction and brittle low-degree recall margins, the authors propose RAM, which combines multi-state routing with rare-bigram loss upweighting to narrow the recall gap up to 1.4B scale without hurting general LM quality.

Critical strengths
  1. Framing the recall deficit as a pretraining optimization issue rather than a state-size limit is convincing. Showing that expanding recurrent state by 30× during Pile pretraining barely helps recall, despite strong synthetic capacity, gives clear motivation for looking at training dynamics.
  2. The base RAM recipe is simple and lightweight, adding ~0.5% parameters to Gated DeltaNet at 1.4B. The sweeps over memory count and loss weight nicely show that routing and rare-bigram upweighting complement each other while keeping zero-shot LM-Evals intact.
Critical weaknesses
  1. Several implementation and baseline details are unclear. For example Equation 8 omits how Δθ_m and w_{t,m} modify Gated DeltaNet, why top-4 writes are used when M ≤ 4, and why RAM stays at 389M parameters across M = 2, 4, 8.
  2. Margin sensitivity across polynomial degrees (Figure 3c) is only measured on noiseless runs without reporting how background gradient alignment changes with degree, and in Figure 4a both learned routers (Joint and Pretrained) drop to 0% recall at ρ = 0.98 without loss upweighting.
Questions for authors
  1. Why do the learned Joint and Pretrained routers collapse to 0% recall at ρ = 0.98 in Figure 4a without recall upweighting, and how does gradient cosine alignment vary across polynomial degrees p = 1, 2, 3?
  2. Could you give the exact Gated DeltaNet update for Equation 8 (Δθ_m, w_{t,m}, top-4 routing when M ≤ 4, and why M = 2, 4, 8 all have 389M parameters), and clarify how MoM, GRM, and SSC compare when parameter-matched and evaluated with length-normalized LM-Evals?
2 comments
AuthorsOpenReview ↗

Response to Reviewer nFSK

Thank you for the thorough review! We're glad you appreciated our theoretical analysis of the recall gap in recurrent models, and that you found the recipe behind RAM simple and lightweight.

Further, thank you for the thorough feedback leading to the improvement of our work. We address your specific weaknesses and questions below. Please also see the common response for a) RAM on Mamba-2 at 1.4B and broader evaluations, b) efficiency benchmarks, c) the recall mechanism measured in pretrained LMs, and d) the n-gram router, now reported as a diagnostic.

(W1/Q2a) Implementation details. Thank you for this important question. The new App. A.1–A.2 provides RAM's update rules for GDN and Mamba-2, including pseudocode and the kernel implementations. In short:

  • All mixer parameters are shared across memories; only the recurrent states are per-memory. RAM is native GDN plus a $\sim$3M-parameter router (389M vs. 386M at 360M, 0.8%), and going from M=2M = 2 to M=8M = 8 adds only $\sim$19K parameters.
  • We write to the top-4 memories (for M≤4M \leq 4, every memory is written). An ablation at M=8M = 8 (120M parameters, 10.5B tokens) finds that top-4 writes outperform dense writes on recall at no LM cost: 19.15% vs. 14.52% recall accuracy (on FDA) at equal Pile perplexity (App. A.1, Table 3).

(W2/Q1) Gradient alignment and margin sensitivity across degrees.

Thank you for these great questions.

  • "Margin sensitivity across polynomial degrees (Figure 3c) is only measured on noiseless runs." We now also measure it on noisy runs. We repeated the Fig. 3b experiment at p=2p = 2 and 3: starting from a partly formed recall circuit and continuing with 90% background loss, recall reaches 71 / 98 / 99% for p=1p = 1 / 2 / 3. Along these continuations, relative margin sensitivity stays ordered by degree (median 20.4 / 3.7 / 3.7 after 1,000 updates and 7.7 / 1.8 / 1.6 after 2,000, for p=1p = 1 / 2 / 3), matching the clean setting (App. B.2.2).
  • "In Figure 4a both learned routers (Joint and Pretrained) drop to 0% recall at ρ=0.98\rho = 0.98 without loss upweighting." When recall positions make up a small enough share of the loss, the model receives too little recall signal to learn the task at all. In our revised experiments this happens at ρ=0.98\rho = 0.98 for every configuration, and without upweighting already at ρ=0.92\rho = 0.92 (Fig. 4, Table 8). Recall upweighting restores the signal: together with routing, it retains 96.9% recall at ρ=0.92\rho = 0.92.
  • "How does gradient cosine alignment vary across polynomial degrees p=1,2,3p = 1, 2, 3?" It barely varies. Measuring both terms of the margin decomposition (Eq. 7), the background gradient is about equally misaligned with the recall margin at every degree (median cosine 0.028–0.041), while relative margin sensitivity falls sharply with degree (11.8 / 5.3 / 3.1; App. B.2.2, Table 10, Fig. 7). Gradient interference is thus roughly constant across degrees, but lower-degree circuits are hit harder by it. Sec. 4.2 now points to this analysis.

(Q2b) MoM, GRM and SSC.

  • Length normalization. To clarify, all LM-Evals, baselines included, use the same protocol with length-normalized accuracy on PIQA, HellaSwag, Arc-E and Arc-C (App. A.5). We corrected the wording.
  • Parameter matching. Thanks for this question. The baselines were held to the nominal 360M budget (347–369M) by using fewer heads at the same head dimension, while RAM keeps native GDN's width (389M). We therefore retrained RAM at the GRM/SSC width (15 heads), which matches GRM/SSC in heads, head dimension and total state with 19M fewer parameters (340M vs. 359M). At this width, RAM M8 still has the best average recall (26.8% vs. 21.5% for the best baseline), the best perplexity (10.40 vs. 10.71) and the best LM-Evals (38.8 vs. 38.4). Results for M=2M = 2 and 4 are in Table 16 (recall, perplexity) and Table 17 (LM-Evals).
AuthorsOpenReview ↗

Following Up on Response to Reviewer nFSK

Thank you again for reviewing our work! As the discussion period ends shortly, we want to check if you have any further questions or found our responses helpful?

Please let us know and thank you for your time!

Reviewer_r4ohOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why recurrent sequence mixers, which achieve information-theoretically optimal storage scaling on synthetic MQAR tasks, recall much worse than attention after language model pretraining. Using a noisy version of MQAR (recall is mixed with next-token prediction), the authors attribute this gap to gradient interference from the background objective. To address this, they propose Routed Associative Memory (RAM), which combines routing tokens across multiple recurrent memories and upweighting the loss on rare associative-recall bigrams. Experiments on Gated DeltaNet backbones at 360M and 1.4B parameters show that RAM reduces the recall gap to attention while maintaining competitive general language modeling performance.

Critical strengths
  1. Relevant problem: Understanding why recurrent mixers lose their theoretical recall capacity during pretraining is an important problem. The proposed noisy MQAR task is a simple and useful way to isolate how competing next-token gradients interfere with associative recall.
  2. Clear empirical gains: Combining loss upweighting of rare bigrams with multiple routed memories noticeably improves recall at both 360M and 1.4B scales with minimal parameter overhead, while keeping LM-Evals within 0.3 pp of GDN. The ablations over memory count and loss weighting show the trade-off between recall and perplexity transparently.
Critical weaknesses
  1. Comparisons/Baselines with matched state sizes: Although the main results are framed as parameter-matched, as I understand it the recurrent state in RAM grows linearly with the number of memories MM (so RAM M8 holds 8×8\times the state of the GDN baseline). Since Section 3 argues that recall capacity is governed by state size, I am missing a comparison to a single memory GDN with matched state size (e.g., matching M2–M8 via larger key/head dimensions) and the same 4×4\times loss upweighting. Without this baseline, it is unclear whether the gains come from routing or simply from a larger state, especially since roughly half of the headline improvement at 1.4B already comes from the loss upweighting alone (Table 1).
  2. Single-seed results and missing efficiency comparisons: All LM results appear to come from a single seed and variance is not reported. Several trends in Table 4 are noticeably non-monotonic across memory counts and loss weights. This makes smaller differences (such as M4 vs. M8 at 1.4B) and the claim that recall "strictly improves" with state size hard to interpret. In addition, the paper does not report training/inference throughput or memory overhead of RAM vs. GDN, which is important since efficiency is the main reason to use recurrent mixers.
  3. Limited novelty and unclear advantage of learned routing: The theoretical storage-capacity result (Theorem 3.2) follows directly from Garcia et al. [1], and the routed-memory architecture is very close to Mixture of Memories (MoM) [2], which is only briefly mentioned in the main text without explaining architectural differences. It is also not clear to me why a learned router would separate recall from non-recall tokens without any explicit inductive bias. In fact, in the noisy MQAR experiment (Figure 4a), a jointly trained router (I think this is what RAM uses in practice) collapses at the same noise level as a single memory, and only an oracle or "pretrained" router helps.
  4. Overstated nn-gram router benefits: The claim that the nn-gram router "closes the average-recall gap to attention entirely" feels too strong. Computing its history features (Appendix A.1) appears to require matching against the full prefix, meaning memory grows with sequence length rather than remaining a fixed-state recurrent model. Furthermore, Table 2 shows that a single memory (M1) with the nn-gram router already matches M2–M8, suggesting the gains come from the hand crafted features rather than routing. Finally, the reported average excludes NIAH (where the router hurts M4 and M8), and the router is only evaluated at 360M.
  5. Presentation and structure: The main results are squeezed into ~1.5 pages and the paper ends without a conclusion or limitations section. Moving Section 2.1 (standard attention background) and parts of Section 4 to the appendix would free up space to properly explain the nn-gram router and experiments in the main text. Also, the abstract uses terms that are not introduced ("Online Hebbian MLP", "polynomial-kernel memory family"), and lines 87–90 state that the 1.4B model closes 56% of the gap while the 360M model "fully closes" it, which reads awkwardly without clarifying that the nn-gram router was only used at 360M.

[1] https://arxiv.org/abs/2607.10034
[2] https://arxiv.org/abs/2502.13685

Questions for authors
  1. How does a single-memory GDN with recurrent state matched to RAM M2/M4/M8 (e.g., via larger key/head dimensions) and 4×4\times loss upweighting perform at 360M?
  2. Could you report training/inference throughput and memory overhead of RAM relative to GDN, as well as multi-seed mean/std results?
  3. What are the main architectural differences between RAM and MoM that drive the performance gains, and how is the "pretrained router" in Figure 4 obtained?
  4. Does computing the nn-gram router features gtg_t at inference require memory/compute that grows with sequence length, and why was it not evaluated at 1.4B?
  5. In Section 4.2 (line 331), how does a p=2p=2 or p=3p=3 memory look if the state dimension dd is held fixed? Doesn't a higher polynomial degree pp increase the feature map and state size?
  6. Line 347 states that 4×4\times recall upweighting retains at least 98% recall accuracy at ρ=0.98\rho = 0.98, but in Figure 4(a) the "Recall loss ×4\times4" curve drops to 0% at ρ=0.98\rho = 0.98 (≈98% only holds up to ρ=0.8\rho = 0.8). Could you clarify which is correct?
5 comments
AuthorsOpenReview ↗

Response to Reviewer r4oh (part 1)

Thank you for your detailed review! We are glad you appreciated the relevance of the problem we study, namely, understanding why the maximum storage capacity of recurrent mixers is not achieved during language modeling training, and how the insights we attain studying this problem allow us to develop RAM.

Further, thank you for the thorough feedback leading to the improvement of our work. We address your specific weaknesses and questions below. Please also see the common response for a) RAM on Mamba-2 at 1.4B and broader evaluations, b) efficiency benchmarks, c) the recall mechanism measured in pretrained LMs, and d) the n-gram router, now reported as a diagnostic.

W1 / Q1: Single-memory GDN at matched state size. We agree this is an important control. We ran it and added it to the revision (App. B.3.3, Table 16). Here, we grow a single-memory GDN's state through its key/head dimension, as you suggest. Widening the keys also widens the query/key projections, so we report two versions: a dense one, which has more parameters, and a parameter-matched one, whose query/key projections keep a fixed random subset of their weights so that its parameter count equals RAM's. All models use the same 4×4\times recall upweighting.

  • At 360M parameters, trained at the GRM/SSC width (3B tokens), RAM beats the parameter-matched wide-key GDN at every memory count, and the gap grows with MM: +1.9+1.9 / +4.7+4.7 / +6.2+6.2 points of average recall at M=2M = 2 / 4 / 8 (e.g., 26.8% vs. 20.6% at M=8M = 8), with lower perplexity at every MM. The dense wide-key GDN, with 38% more parameters, is level with RAM M8 (25.7% vs. 26.8%).

More state therefore helps recall, but RAM turns added state into recall better: from M=2M = 2 to M=8M = 8 it gains 5.3 points, the wide-key GDN 1.1 points. Routing therefore helps beyond the state it adds.

W2 / Q2 (single seed): Multi-seed results. We agree multi-seed results would be ideal. Unfortunately, they are out of reach for our compute budget at the time being: for reference, our 1.4B-parameter runs alone cost O(10K USD). Instead, we now report 95% bootstrap confidence intervals over evaluation examples for FDA, SWDE, SQuAD, and average recall, at both the 360M and 1.4B model scales (Figs. 11 and 13, with paired tests in App. B.3.1 and B.4.1). The main trends hold:

  • Recall upweighting: every 1×→4×1\times \rightarrow 4\times step improves average recall, at both scales and every memory count.
  • More memories: at 360M parameters, every memory step improves average recall at both 1×1\times and 4×4\times. At 1.4B parameters, at both weights, $M \in \lbrace 4, 8\rbrace $ improves over $M \in \lbrace 1, 2\rbrace $.

W2 / Q2 (efficiency): Training/inference throughput and memory against GDN. We now include training and inference benchmarks at the 1.4B scale (App. A.3), along with pseudocode for the kernel implementations (App. A.2). Please refer to item 2 of the common response for more details.

W3 / Q3: Relation to MoM. We agree the paper was too brief here. We now provide a more detailed comparison between MoM and RAM in App. A.4.1. Concretely, RAM differs from MoM in three main ways:

  • Shared parameters. MoM gives each memory its own key, value, decay, and update-strength projections. RAM shares all mixer parameters and only the states are memory-specific, so M=8M = 8 adds 0.51% parameters at 1.4B. RAM's parameter count therefore stays nearly constant as the number of memories, and with it the state, grows.
  • Reads decoupled from writes. MoM reads with the same hard top-2 weights it writes with. RAM has a separate dense read router, so a query can read from a memory it would not necessarily write to.
  • Soft writes, no extras. RAM scales the write strength with soft weights, and uses no always-on shared memory and no load-balancing loss.

Moreover, we provide state-matched evaluations comparing MoM and RAM (Fig. 12, Table 16). At 360M parameters, with the same state size, fewer parameters and the same 4×4\times recall upweighting, RAM outperforms MoM on average recall and perplexity at every number of recurrent states $S \in \lbrace 2, 4, 8\rbrace $ (e.g., 26.8% vs. 14.5% average recall at S=8S = 8).

Beyond these empirical gains, part of our contribution is explaining why routing across memories helps: our noisy-MQAR analysis (Sec. 4) traces the recall gap to interference from competing supervision, which motivates recall upweighting and routing, and shows that the two compound (Fig. 4). We now also measure this in pretrained LMs (common response, item 3).

AuthorsOpenReview ↗

Response to Reviewer r4oh (part 2)

W3 / Q3: Learned router in noisy MQAR This is a great observation, and it led us to revamp the experiments of Fig. 4. We now train the 2×22 \times 2 grid of recall-loss weight $\in \lbrace 1\times, 4\times\rbrace $ and memories $\in \lbrace 1, 2\rbrace $, where the two-memory variants use a jointly trained router, as in our LM experiments. Here, our jointly trained router improves recall over the single-memory baseline (73% vs. 29% at ρ=0.80\rho = 0.80), and the gain compounds with 4×4\times recall upweighting, as together they retain 97% at ρ=0.92\rho = 0.92, where neither alone exceeds 38% (Fig. 4, Table 8).

Moreover, we revised Sec. 4.3 to state the router's benefit directly. Concretely, routing across multiple memories improves recall, and the improvement persists when combined with recall-loss upweighting, as we also observe in our language-modeling experiments (see the table in item 3 of the common response).

W4 / Q4: n-gram router. Thanks for bringing these points up, we ran further ablations and agree with them. Please refer to item 4 of the common response for more details.

W5: Presentation and structure. Thank you for the detailed suggestions. In the revision, we added a conclusion that discusses limitations and future work (Sec. 6), and we improved the clarity of the writing throughout.

Q5: What is held fixed when sweeping the polynomial degree in Sec. 4.2? Thank you for raising this. We believe our wording was confusing here and clarified it in Sec. 4.2. What we hold fixed is the query/key/value embedding dimension dd (which we previously called the "state dimension" in Sec. 4.2), not the size of the feature map. For clarity, there is no feature map in the experiments in this section. As noted at the beginning of section 4.2, we use the exact kernel Kp(q,k)=(1+q⊤kpd)pK_p(q, k) = \left(1 + \frac{q^\top k}{p\sqrt{d}}\right)^p, where q,k∈Rdq, k \in \mathbb{R}^d. We evaluate this kernel exactly, so a feature map is never built. We use this exact degree-pp kernel purely as an analysis tool to investigate the failure to learn recall circuits as we interpolate between linear attention (p=1p = 1) and softmax attention.

Q6: Mismatch between the text and Fig. 4(a). Thank you for catching this mismatch. We corrected the text together with the revamped Fig. 4 experiments (see "W3 / Q3: Learned router in noisy MQAR" above).

AuthorsOpenReview ↗

Following Up on Response to Reviewer r4oh

Thank you again for reviewing our work! As the discussion period ends shortly, we want to check if you have any further questions or found our responses helpful?

Please let us know and thank you for your time!

Reviewer_sVgzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why recurrent language models underperform softmax attention on in-context recall despite their theoretical storage capacity. Building on prior Hebbian memory theory, the authors show that an Online Hebbian MLP can store T associations using approximately T log T state coordinates under specific assumptions. They introduce noisy MQAR to study gradient interference and recall-margin sensitivity during training.

To improve recall, the authors propose Routed Associative Memory (RAM), which uses multiple recurrent memories with learned read/write routers and upweights rare repeated-bigram tokens during training.

At 1.4B parameters, RAM closes 56.02% of the average recall gap between Gated DeltaNet and attention while maintaining comparable general language modeling performance. At 360M parameters, adding historical features to the read router further improves recall, reaching attention-level performance on the FDA/SWDE/SQuAD average.

Critical strengths
  • Distinguishing storage capacity from learned recall is a valuable contribution. The contrast between strong synthetic MQAR capacity and weak recall after LM pretraining highlights an important problem. Noisy MQAR and matched-gradient-noise experiments provide useful evidence of interference from competing supervision (Figures 2–3).

  • Memory routing and recall weighting show complementary benefits. At 1.4B, their combination achieves 45.78% average recall, compared with 38.84% for routing alone and 36.23% for weighting alone, while LM-Evals decreases by only 0.30 percentage points relative to GDN (Table 1). Applying the same recall weighting to growing-state baselines further strengthens the experimental comparison.

Critical weaknesses
  • The contribution of multi-memory routing is not clearly isolated. Table 2 shows that adding the n-gram router to a single-memory model improves average recall from 18.60% to 33.49%, nearly matching attention. This suggests that historical features may account for much of the improvement, but their contributions are not adequately ablated. Moreover, the n-gram router reduces M8's NIAH accuracy from 41.88% to 29.70% (Table 8), limiting the generality of the recall improvements.

  • Key state-update details are insufficiently specified. Equation (8) does not clarify how write weights affect GDN's state updates or whether unselected memories decay or remain unchanged. These details affect memory retention and computation, making the method difficult to reproduce.

Questions for authors
  • How much of the n-gram router's improvement comes from historical retrieval features rather than multi-memory routing? Could you provide feature ablations and explain the performance degradation on multi-value and multi-query NIAH tasks?

  • How exactly does the write weight wt,mw_{t,m} affect GDN's state update in Equation (8), and do unselected memories decay or remain unchanged? Could you provide update equations or pseudocode, along with prefill/decode throughput, latency, and peak memory comparisons against GDN under matched conditions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer sVgz

Thank you for the thoughtful review! We are glad you appreciated our contributions on distinguishing the maximum storage capacity of recurrent mixers from their learned storage capacity and how we use our insights from this finding to develop RAM.

Further, thank you for the thorough feedback leading to the improvement of our work. We address your specific weaknesses and questions below. Please also see the common response for a) RAM on Mamba-2 at 1.4B and broader evaluations, b) efficiency benchmarks, c) the recall mechanism measured in pretrained LMs, and d) the n-gram router, now reported as a diagnostic.

W1 / Q1: n-gram router. Thanks for bringing these points up, we ran further ablations and agree with them. Please refer to item 4 of the common response for more details.

W2 / Q2a: State-update details of Eq. 8. We agree Eq. 8 was under-specified, and we have clarified it in the revision. Moreover, we provide pseudocode for RAM (Algorithm 1) and the explicit update equations for Gated DeltaNet and Mamba-2 in App. A.1 (Eq. 11, Table 2). Answering your two questions:

  • How does wt,mw_{t,m} affect the GDN state update? It scales the delta-rule step size βt\beta_t of memory mm:

    St(m)=αtSt−1(m)(I−wt,mβtktkt⊤)+wt,mβtvtkt⊤S_t^{(m)} = \alpha_t S_{t-1}^{(m)}\left(I - w_{t,m}\beta_t k_t k_t^\top\right) + w_{t,m}\beta_t v_t k_t^\top

  • Do unselected memories decay or stay unchanged? They decay: the gate αt\alpha_t is shared and applied to every memory at every token.

Q2b: Throughput, latency, and memory against GDN. We now include training and inference benchmarks at the 1.4B scale (App. A.3), along with pseudocode for the kernel implementations (App. A.2). Please refer to item 2 of the common response for more details.

AuthorsOpenReview ↗

Following Up on Response to Reviewer sVgz

Thank you again for reviewing our work! As the discussion period ends shortly, we want to check if you have any further questions or found our responses helpful?

Please let us know and thank you for your time!

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Response (part 1)

We thank all reviewers for their thoughtful and constructive feedback. We're glad reviewers found that:

  • Separating memory capacity from learning recall is a valuable insight (sVgz, r4oh, nFSK, 3NDt). Our key observation is that recurrent mixers have asymptotically optimal recall capacity on synthetic tasks, yet recall poorly after LM pretraining (Sec. 3).
  • Noisy MQAR and the controlled experiments identify a mechanism that explains the gap (sVgz, r4oh, 3NDt). We attribute it to gradient interference from the competing next-token objective, and show that recurrent architectures are particularly vulnerable to it (Sec. 4.1–4.2).
  • Our method, RAM, is simple, lightweight and effective (sVgz, r4oh, nFSK, 3NDt). RAM-GDN improves recall accuracy by 18.5pp over GDN for only $\sim$0.5% extra parameters at the 1.4B parameter scale while LM-Evals stay within $\sim$0.3pp of base GDN (Sec. 5.2, Table 1).

We summarize the main changes below and address the suggestions brought up by each reviewer in the individual responses. Changes in the revised paper are in blue.

1. Broader evaluation of RAM (suggested by sVgz, r4oh, nFSK, 3NDt). We include additional evaluations on a new base recurrent mixer, more benchmarks, and more thorough parameter-matched baselines at 360M scale:

  • Models: We applied RAM to a second mixer, Mamba-2, at 1.4B parameters / 10B tokens. Here, RAM-Mamba-2 (M=8M = 8, 4×4\times) closes 69% of Mamba-2's recall gap to attention, while its LM-Evals stay within 1.0pp of Mamba-2 across all variants (Table 1).
  • Recall: We ran additional evaluations on NIAH (single, multi-key, multi-value, multi-query), the RULER suite (12 tasks) at 2K/4K/8K, and open-book QA (TriviaQA, NQ, DROP), all with bootstrap 95% CIs (App. B.4). RAM's gains extend to these benchmarks. At 1.4B, RAM (M=8M = 8, 4×4\times) closes 47% of GDN's gap to attention on NIAH, 27% on RULER at 2K, and 65% on open-book QA.
  • Matched baselines at 360M: We retrained RAM at 360M to match GRM/SSC, two of our growing-state baselines, exactly in number of heads, head dimension and total state size, with 5% fewer parameters (nFSK). We compare RAM against two kinds of baselines:
    • Growing-state architectures: RAM M8 reaches 26.8% average recall vs. 21.5% for the best of MoM, GRM and SSC (App. B.3.2, Fig. 12, Table 16). We discuss the differences between RAM and MoM in App. A.4.1.
    • A single memory with the same state: As a state- and parameter-matched baseline, we train single-memory GDNs whose state is grown to match RAM's through wider query/key dimensions, with the query/key projections sparsified to match RAM's parameter count (r4oh). RAM beats them at every memory count, by 1.9 / 4.7 / 6.2 points of average recall at M=2M = 2 / 4 / 8 (App. B.3.3, Fig. 12, Table 16). Memory routing therefore improves recall beyond what the added state alone provides.

2. Efficiency and implementation (suggested by sVgz, r4oh, 3NDt). We added an appendix benchmarking RAM's training, prefill and decoding efficiency. We benchmark RAM-GDN and RAM-Mamba-2 at 1.4B against their base mixers and self-attention (App. A.3, Table 4, Fig. 6). In short, RAM's cost comes from its larger recurrent state, not from routing:

  • Training: With M=2M = 2 / 4 / 8 memories, RAM trains at 0.89 / 0.80 / 0.66$\times$ GDN's throughput (0.89 / 0.78 / 0.60$\times$ Mamba-2's), with at most 1.2$\times$ the peak memory.
  • Decoding: Single-sequence decoding stays within 12% of GDN (0.88–0.90$\times$) and within 17% of Mamba-2 (0.83–0.90$\times$). RAM's state stays fixed as context grows, so beyond 8K–16K tokens of context self-attention decodes more slowly than every RAM variant.

Moreover, we include pseudocode for RAM-GDN and RAM-Mamba-2, including for the fused training and decoding kernels we implement (App. A.1–A.2).

AuthorsOpenReview ↗

Common Response (part 2)

3. The mechanism in pretrained LMs (suggested by nFSK, 3NDt). We now test the Sec. 4 account directly in our pretrained 360M models (App. B.2.3).

Competing supervision erodes recall circuits in LMs too. When we continue training a model on the background loss alone, its recall margins drop by 21–41% within 128 steps, while random updates of the same size cost at most 2.5%. This is the structured interference we found on noisy MQAR, now quantified in pretrained models. Consistent with this account, RAM's recall circuits are more robust to the interference than GDN's.

The two interventions address it in complementary ways.

  • Recall upweighting strengthens the recall training signal against the background gradient. For GDN, 4×4\times upweighting raises the mean recall margin by 1.07 nats while raising background loss by only 0.056 nats.
  • Memory routing reduces crosstalk in the recurrent state. At recall queries, RAM M8's reads overlap 1.7$\times$ more with the write of the kv-pair being recalled than with background writes, and its readout has 1.4$\times$ GDN's signal-to-crosstalk ratio. Upweighting barely changes either quantity. Both patterns hold throughout training, at matched state and parameters (new Fig. 10).

The two gains compound. The table below gives average recall (%) at 360M for RAM M8 and a single-memory GDN with the same state size and parameter count (App. B.3.3, Table 18), each trained with and without 4×4\times recall upweighting:

1×1\times4×4\timesGain from upweighting
Single-memory GDN, matched13.320.6+7.3+7.3
RAM M817.626.8+9.2+9.2
Gain from routing+4.3+4.3+6.2+6.2

Each intervention helps on its own, and each helps more when the other is present; together they double average recall (13.3% →\rightarrow 26.8%).

Following r4oh and nFSK, we also revised the noisy-MQAR experiments: the two-memory models now use a jointly trained router, as in our LMs, and we run the full 2×22 \times 2 grid of upweighting and routing (Sec. 4.3). The same complementarity holds.

4. The n-gram router as a diagnostic (suggested by sVgz, r4oh). Ablations prompted by the reviewers, at 120M parameters, show that the n-gram features' gain comes from the answer they supply, rather than from helping the model read from its memory (App. B.5.1). Concretely:

  • When trained from scratch, providing the model only with whether the current n-gram occurred earlier in the context recovers just 15% of the recall accuracy gain (on FDA) attained when using full n-gram features. Concretely, providing information only of whether the n-gram occurred achieves 6.3% acc vs. 21.5% acc with the full n-gram features and 3.6% acc without any information (Tab. 25).
  • Further, in the model trained with the n-gram features, removing the answer from the document but keeping the n-gram features computed from the original document, retains 59% of the model's FDA score (12.6% vs. 21.5% with the answer present), even though the answer is no longer in the context (Tab. 26).

These ablation results were unexpected to us, and they clarify what the n-gram router was doing. Most of its gain came from supplying the n-gram answer from outside the recurrent state, not from better use of the memory. A signal marking where recall is needed accounts for only 15% of that gain (App. B.5.1). That said, we highlight that this is consistent with our paper's main claim: recurrent models have the storage capacity (Sec. 3) but struggle to learn recall circuits under competing supervision (Sec. 4, App. B.2.3), which is what RAM's interventions target (Sec. 5). We thank the reviewers for the questions that led to this analysis!

We therefore no longer present the n-gram router as part of RAM, and we include these ablations in App. B.5.1 (Tables 25–26).

AuthorsOpenReview ↗

Thank you! Summary of new results in response to reviewer feedback

Dear AC and reviewers,

Thank you again for your feedback. As the discussion period wraps up, we wanted to highlight the experiments we added in response to your questions:

  • New Mamba-2 results. We tested RAM on a second architecture, Mamba-2. At 1.4B parameters, RAM closes 69% of Mamba-2’s recall gap to attention.
  • New recall benchmarks. We evaluated RAM-GDN on needle-in-a-haystack tasks, RULER, and open-book QA. RAM improves on Gated DeltaNet (GDN) across all three, closing 65% of its gap to attention on open-book QA.
  • New matched baselines. We trained a single-memory GDN with the same state size and parameter count as RAM. At 360M parameters, RAM with eight memories scores 6.2 percentage points higher on average recall.
  • New tests in pretrained models. We tested whether training on non-recall tokens interferes with recall, as we saw in our synthetic task. After 128 such training steps, the score advantage for the correct answer fell by 21–41%. Random updates of the same size reduced it by at most 2.5%.
  • New speed measurements. We benchmarked training, prefill, and decoding for both RAM-GDN and RAM-Mamba-2. With eight memories, RAM-GDN trains at 66% of GDN’s speed and decodes within 12% of GDN’s speed for a single sequence.

If you find these additional results convincing, please feel free to update your scores. If you still have concerns, we’d love to hear them so we can address them during the remaining rebuttal period. Thank you again for your time!

Throughout the discussion period, we carefully considered the feedback from all four reviewers, provided detailed responses to their questions and concerns, and incorporated substantial revisions and additional experiments into the paper.

Below is a summary of the reviewers’ initial ratings and their current status following the rebuttal:

ReviewerOriginal RatingAfter Discussion
sVgz2No Reply
r4oh2No Reply
nFSK3No Reply
3NDt2No Reply

While we have not yet received follow-up responses from the reviewers, we believe we have addressed the concerns raised in their reviews. We hope the Area Chair will consider the new experiments and clarifications presented in our rebuttals when making their assessment. We would also appreciate it if the reviewers could take another look at our responses and reconsider their ratings if they find their concerns have been addressed.

We sincerely appreciate the reviewers’ time and constructive feedback, which we believe has been invaluable in strengthening our work. We remain available to clarify any outstanding questions or discuss any remaining concerns during the rebuttal period.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.