- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how capacity affects forgetting during sequential training. The authors use synthetic facts with known information content and estimate capacity under a training protocol. The paper suggests that recall can fail while some information remains recoverable, and that replay focused on useful facts improves retention, reducing forgetting. Capacity experiments use 5M and 11M models, while Pythia-160M experiments test interference and recovery.
Critical strengths
** Controlled experiments**: The synthetic facts experiments make sense in studying this topic, especially as they can vary the amount of information relative to the capacity. This helps test when 'capacity' shapes forgetting in (language) models.
Critical weaknesses
-
Limited advance over existing work: Allen-Zhu and Li (2025) already measure a model's capacity using synthetic random facts and prediction loss. Marek et al. (2026) also studied how capacity affects forgetting. This paper combines these established directions and adds tests of answer ranking, recovery, and relearning. However, the measurement framework builds on Allen-Zhu and Li. I am not convinced that the additional findings establish a substantial advance over these prior studies. This makes it difficult to find the significance of the contribution.
-
Insufficient evidence across model sizes: The capacity study varies information load but uses only two small models, with 5M and 11M parameters. A concurrent work (Marek et al., ) studied a similar problem with more (5+) model sizes. The pythia-160M experiments test interference and recovery without repeating the capacity study. The authors also acknowledge that the two small models have similar storage per parameter. Thus, the results cannot clearly separate the effect of measured load from the number of facts per parameter. A broader study of model sizes and training choices is needed to establish the claimed capacity relationships. The current evidence is insufficient to show how these relationships change as models become larger.
Questions for authors
- Which specific observation provides an advance over the previous works? The idea of studying forgetting from a model capacity perspective has already been studied, with similar synthetic data and thoroughly on real datasets.
- Could the paper's results separate the effect of capacity from different training choices (e.g., data, model, hyperparams)? The authors perform their experiments using a single set of hyperparameters (App. C.). Furthermore, if capacity was indeed the main focus of the paper, it would be sound to vary the amount of pretraining data. I'm also curious if the authors have considered studying the effect of pretraining on forgetting. Learning rates are only tested on values of 1e-4 and 1e-3, which is too narrow of a scope and rather unfit for the training of an LLM.