- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows that when a 2.9B model is trained from scratch using Gated Linear Attention followed by SFT and RL tailored for the BEAM benchmark, it functions as a sparse recovery system within an associative memory matrix. Mathematically, unrolling this multi-timescale layer proves its outputs evaluate a "memory polynomial" whose coefficients represent historical token relevance. Experimental results show that "shape beats size": allocating more heads across diverse timescales outperforms a state with twice the total capacity but fewer heads by over 30% at 500K and 1M token contexts. After just 160 steps of RL with a verifiable recall reward, the model dramatically enhances its fact-locating capabilities.
Critical strengths
-
This paper shows that if we train a small model with gated linear attention, then train it with RL typically for BEAM task, it can perform well on BEAM benchmark.
-
This paper conducts extensive experiments and presents a solid ablation study, offering valuable insights to the research community.
Critical weaknesses
- Since the model's RL phase is tailored specifically for the BEAM benchmark, its high score there is expected. Evaluating the model on broader, general benchmarks is necessary to assess its true versatility.
Questions for authors
-
As shown in weaknesses 1. Please report the general benchmarks of 2.9B model.
-
Please also list the data used in all stages pretain/sft/rl.
-
The background section in the abstract is overly detailed and can be streamlined into two or three sentences to keep the focus on the paper's core contributions.
Other comments:
Line 757: missing refer ids: of the problem, which Proposition ?? and equation ??
2 comments
Follow-up on Author Response
Dear Reviewer 41GN,
Thank you again for your time and valuable comments. As the discussion period nears its end, we would like to follow up on our author response. We would appreciate any further feedback and would be happy to clarify any remaining concerns.
Best regards, The authors
Adds new general benchmarks
We thank the reviewer for the careful summary and for describing the ablations as solid and of value to the community. The paper, also brings new perspectives on the time dimension(context length) for memory recall, carrying interesting parallels with Laplace transform, memory models in neuroscience and the experiments offer valuable insights. Regarding the questions, we address each point below.
W1 / Q1. RL is not tailored to BEAM; general benchmarks. No post-training stage uses BEAM data. SFT and RL train only on the train portions of LoCoMo and HotpotQA. BEAM is a transfer evaluation, not the training task.
We evaluated our final 2.9B model (after RL) and the baselines on general benchmarks (accuracy, %; MMLU scored on answer choice):
Memory training does not degrade general ability: from the pretrained base to the RL model, MMLU moves 29.7→29.4, GSM8K 13.6→12.1 and MATH 5.8→6.6. The gap to the baselines reflects pretraining scale, not the memory recipe: they see 100 to 265× more tokens with about 10× more parameters, roughly three orders of magnitude more pretraining compute. We report these numbers to show that recall is gained without loss of general ability, not to claim parity. We also add a note that training the model till 36T for our settings is expensive and not very relevant.
Q2. Data used in each stage. We will add this table to the appendix:
Neither SFT nor RL uses any BEAM data.
Q3. Abstract. Agreed, we will reduce the background to two or three sentences.
Other comments. Thank you for catching the missing references at L757. Both now resolve in the revision.