- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors argue that recurrent language models have enough memory capacity for recall, but ordinary language training prevents them from learning to use it well. Concretely, the authors find that in multi-query associative recall (MQAR), recurrent language models often have higher memory capacity per parameter then softmax attention models. Through analyses, they identify two causes of this learning failure, which is supervision competition from non-recall tokens and parameter sensitivity of linear attention. Its method, RAM, routes tokens across multiple memories and gives extra training weight to recall examples. At 1.4B parameters, RAM closes 56% of the measured recall gap to attention, with little change in general benchmark scores. Adding n-gram router further closes this recall gap.
Critical strengths
- Clear problem diagnosis: It separates memory capacity from the ability to learn recall during training.
- Useful controlled experiments: Synthetic tasks show how competing training objectives disrupt recall.
- Meaningful improvements: Routing and recall weighting improve several recall tasks, with ablations showing their separate effects.
- Small parameter overhead: The highlighted 1.4B result adds only 0.51% more parameters.
Critical weaknesses
- Limited evaluation: The current experiment only verify the proposed method on one single model scale and training data scale, and with limited variety in tasks.
- I strongly suggest the authors to evaluate on benchmarks such as RULER, LongBench, LongPPL, etc. Moreover, only three recall-intensive tasks is not quite sufficient.
- I think they should also train smaller models and with varying training data scale to verify that the performance gains are consistent across model scales.
- Also, in practice, recurrent mixers are often adopted in hybrid models that incorporate both recurrent mixers and softmax attention. Therefore, it is very important to verify that the proposed method is useful in such hybrid architectures as well.
- The explanation is suggestive, not conclusive: Gradient analysis uses simplified synthetic models rather than directly establishing the same mechanism in pretrained RAM.
- Efficiency is insufficiently measured: Small parameter overhead does not establish low inference cost, especially with multiple memories and dense reads.
Questions for authors
See weaknesses.
2 comments
Following Up on Response to Reviewer 3NDt
Thank you again for reviewing our work! As the discussion period ends shortly, we want to check if you have any further questions or found our responses helpful?
Please let us know and thank you for your time!
Response to Reviewer 3NDt
Thank you for the detailed review! We're glad you found that our diagnosis clearly separates memory capacity from learning recall, that the controlled experiments were useful, and that RAM's improvements come at a small parameter overhead.
Further, thank you for the thorough feedback leading to the improvement of our work. We address your specific weaknesses and questions below. Please also see the common response for a) RAM on Mamba-2 at 1.4B and broader evaluations, b) efficiency benchmarks, c) the recall mechanism measured in pretrained LMs, and d) the n-gram router, now reported as a diagnostic.
(W1a) Evaluation. Thanks for these important suggestions! We substantially expanded the evaluations (App. B.4):
(W1b) Model and data scale. Thanks for this question. The paper already evaluates two scales: 360M parameters / 3B tokens and 1.4B parameters / 10B tokens (Table 1). We also add the following new results:
(W1c) Hybrid models. Thanks for this important question! In this work, our goal is to study how much of the recall gap pure recurrent mixers can close, since recall is exactly where they struggle (Zoology, Arora et al., 2023; Based, Arora et al., 2024; Jelassi et al., 2024). Adding attention layers would mix their recall with the recurrent mixer's, so we keep hybrid architectures out of scope for this paper.
(W2) Mechanism in pretrained models. Thank you for this suggestion. We now test the Sec. 4 account directly in our 360M pretrained models. Please refer to item 3 of the common response for more details.
(W3) Efficiency. We now include training and inference benchmarks at the 1.4B scale (App. A.3), along with pseudocode for the kernel implementations (App. A.2). Please refer to item 2 of the common response for more details.