acceptodds
Under review as a conference paper at ICLR 2027

Acquire, Then Retain: A Two-Stage Model of Memorization in Small Language Models

Abstract

The emergence of small agentic models makes the memorization capacity of small language models (SLMs) increasingly important to understand, yet it remains understudied relative to large-scale models. We inject synthetic facts into the training of five models spanning 3M–151M parameters and study memorization as a joint function of model scale, repetition count, exposure pattern, and time since exposure. We find that memorization at this scale is bimodal, i.e, a fact is either memorized or it is not, with little continuum in between. This motivates a two-stage retention kernel that separates whether a fact is acquired from how quickly it is forgotten. This two-stage kernel generalizes to a model size held out from fitting (, versus for a single-stage fit). On the axis of exposure, Spacing a fact's repetitions across training, rather than delivering them back-to-back, makes it harder to acquire at every model size, requiring up to twice as many repetitions. However, once a fact is firmly acquired, spacing slows its forgetting, extending its decay timescale by –. This benefit shrinks with scale though. Spacing thus trades acquisition for durability. Finally, facts acquired late in training, under small learning rates, are forgotten faster, with direct implications for how knowledge introduced during fine-tuning persists.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Memorization, Small Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PNfQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how small language models acquire and retain individual facts using five LLaMA-style models from 3.2M to 151M parameters. They observe a strongly bimodal memorization-score distribution and model retention as the product of an onset probability and conditional decay after successful memorization. They find that finds encode later in training with smaller learning rates are less durable.

Critical strengths
  1. They have control experimental setup, in terms of both LMs and data. This provides a clean way to study memorization.
  2. The most interesting finding is that spacing affects acquisition and later retention in opposite directions.
Critical weaknesses
  1. In Related Work, the most recent paper that mentioned in the paper from 2025, and most of the references are from 2022–2023. However, there have been many studies on continual learning, memorization, and forgetting in language models from 2025-2026. I'm not sure if the authors are sufficiently familiar with recent research progress of this direction. Also see my question1, I think the two-stage framework already be studied for a long time.
  2. The model sizes are two small. I don't think it makes sense to use scaling law implications at such scales.
Questions for authors
  1. Similar two-stage setups were used in papers at least two years ago, such as https://arxiv.org/abs/2411.07175. Can you clarify how your two-stage differs from that work?
  2. Can you explain how your findings from studying synthetic tasks with small language models might carry over to real-world settings? In other words, what insights does this study offer for larger models or more realistic scenarios?
Reviewer_cDnLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper models the acquisition and forgetting of facts in very small language models using a two-stage framework.
  • Based on this framework, the authors quantitatively model how spaced repetition in knowledge injection requires more repetitions for memorization but leads to longer-lasting memories.
  • The authors also demonstrate that facts injected later are forgotten more quickly.
Critical strengths
  • The paper proposes modeling knowledge acquisition and forgetting as the product of two functions: an acquisition function determined by model size and the number of repetitions, and a forgetting function that captures the effect of elapsed time.
  • The authors empirically investigate various phenomena related to knowledge acquisition and forgetting, including analyses under burst and spread conditions.
  • Experiments across models of different sizes provide insights that may support more general conclusions about knowledge acquisition and forgetting.
Critical weaknesses
  • The experiments are conducted on language models that are substantially smaller than modern LLMs. Compared with studies of knowledge acquisition and forgetting in LLMs, such as Chang et al. (2024), this work focuses on substantially smaller models. Although the experimental methodology may be difficult to scale to LLMs, the current results are insufficient to establish that the findings can be directly generalized to LLMs.
  • Chang et al., "How Do Large Language Models Acquire Factual Knowledge During Pretraining?" NeurIPS 2024.
Questions for authors
  • (Figure 2) What is the interpretation of the memorization threshold of 0.5? Does this mean that the model answers a question about the corresponding fact correctly with a probability of 50%?
Reviewer_Y5MNOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how small language models memorize and forget individual facts.They present a model of memorization of an individual fact in said small language models. The authors inject 64 synthetic (what they call) canary facts (made-up facts absent from the training data, as to say any recall comes only from controlled injections) either back-to-back or spaced across training. They then fit a model which measures if a fact is learned at all, and how fast it fades afterward. This model is effective at predicting unseen model sizes. The main findings are that, increasingly with scale, spacing makes facts harder to learn, but, in the end, slows forgetting once they are learned. Facts learned late at low learning rates also fade faster (this could have implications for fine tuning).

Critical strengths
  • The design directly follows a novel empirical finding (memorization is bimodal), giving it strong grounding and motivation. Also, there is what is simply strong validation from the generalization to model sizes it never saw.
  • The tradeoff takeaway is important (and surprising!). The result showing that spacing repetitions makes facts harder to acquire but slows their forgetting once learned (and it quantifies how both sides move with model size) is important and has direct implications for scheduling during fine-tuning (also actionable). This is the opposite of the human spacing effect, which certainly captivates me and will definitely breed interesting discussion at the conference.
Critical weaknesses
  • A headline result is that the spacing penalty on acquisition grows with model size, but step size and spacing gap do as well. Could the growing penalty therefore reflect smaller updates or longer gaps rather than model capacity? I don't think the design could distinguish these. This matters a lot for the grounding of the scaling claim.
Questions for authors
  • Could you test whether the growing spacing penalty is (more) certainly driven by model size (not step size or spacing gap)?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment

We thank the reviewers for their careful reading and in response to their concerns, we have made the changes listed below with details in the individual responses.

  1. What drives the spacing penalty (R1). Two controlled runs show that the penalty grows with the gap between repetitions, not with model size or learning rate. Giving L the gap of M lowered L's penalty to M's level (2.81× → 1.63×; M: 1.74×). Training M at XL's learning rate did not raise it (1.74× → 1.15×). Spacing's retention benefit, by contrast, shrinks mainly with model size. Response to R1; Appendix [D].

  2. Memorization threshold (R2). 0.5 means at least half the target tokens are reproduced exactly; it is not a probability. Because scores are bimodal, all conclusions hold from 0.3 to 0.7. Response to R2; Appendix [G].

  3. Model scale (R2, R3). We scope our claims to this scale, which is standard for controlled studies of memorization: Morris et al. (2025) fit on 100K–20M models, Zucchet et al. (2025) work at 44M, and Zuo et al. (2026) at 100M. In addition, the kernel extrapolates to a held-out larger size (R² = 0.79 / 0.92), and each stage agrees with LLM-scale studies on what it covers. Varying size is also what revealed the effects in point 1. Responses to R2 and R3.

  4. Novelty and related work (R3). Our "two stages" model one fact's trajectory in single-run pretraining; Chen et al. (2024) use the term for sequential fine-tuning. We have added and positioned 2025–26 work. Response to R3.

  5. Presentation (all). We have rewritten the introduction, methods and discussion for clarity, and narrowed two claims to match the evidence.

Thank you once again for improving the quality of this work.

AuthorsOpenReview ↗

Engagement

Dear Reviewers, as the discussion period ends in less than a day,we will appreciate your feedbacks on our new experiments and literature reviews.

Regards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.