- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how small language models acquire and retain individual facts using five LLaMA-style models from 3.2M to 151M parameters. They observe a strongly bimodal memorization-score distribution and model retention as the product of an onset probability and conditional decay after successful memorization. They find that finds encode later in training with smaller learning rates are less durable.
Critical strengths
- They have control experimental setup, in terms of both LMs and data. This provides a clean way to study memorization.
- The most interesting finding is that spacing affects acquisition and later retention in opposite directions.
Critical weaknesses
- In Related Work, the most recent paper that mentioned in the paper from 2025, and most of the references are from 2022–2023. However, there have been many studies on continual learning, memorization, and forgetting in language models from 2025-2026. I'm not sure if the authors are sufficiently familiar with recent research progress of this direction. Also see my question1, I think the two-stage framework already be studied for a long time.
- The model sizes are two small. I don't think it makes sense to use scaling law implications at such scales.
Questions for authors
- Similar two-stage setups were used in papers at least two years ago, such as https://arxiv.org/abs/2411.07175. Can you clarify how your two-stage differs from that work?
- Can you explain how your findings from studying synthetic tasks with small language models might carry over to real-world settings? In other words, what insights does this study offer for larger models or more realistic scenarios?
General comment
We thank the reviewers for their careful reading and in response to their concerns, we have made the changes listed below with details in the individual responses.
What drives the spacing penalty (R1). Two controlled runs show that the penalty grows with the gap between repetitions, not with model size or learning rate. Giving L the gap of M lowered L's penalty to M's level (2.81× → 1.63×; M: 1.74×). Training M at XL's learning rate did not raise it (1.74× → 1.15×). Spacing's retention benefit, by contrast, shrinks mainly with model size. Response to R1; Appendix [D].
Memorization threshold (R2). 0.5 means at least half the target tokens are reproduced exactly; it is not a probability. Because scores are bimodal, all conclusions hold from 0.3 to 0.7. Response to R2; Appendix [G].
Model scale (R2, R3). We scope our claims to this scale, which is standard for controlled studies of memorization: Morris et al. (2025) fit on 100K–20M models, Zucchet et al. (2025) work at 44M, and Zuo et al. (2026) at 100M. In addition, the kernel extrapolates to a held-out larger size (R² = 0.79 / 0.92), and each stage agrees with LLM-scale studies on what it covers. Varying size is also what revealed the effects in point 1. Responses to R2 and R3.
Novelty and related work (R3). Our "two stages" model one fact's trajectory in single-run pretraining; Chen et al. (2024) use the term for sequential fine-tuning. We have added and positioned 2025–26 work. Response to R3.
Presentation (all). We have rewritten the introduction, methods and discussion for clarity, and narrowed two claims to match the evidence.
Thank you once again for improving the quality of this work.