- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This study investigates how preferred repetition counts change across model scales in data-constrained language model pretraining. The authors demonstrate that repetition counts selected using smaller models may not remain optimal at larger scales. To facilitate repetition selection at larger scales, they propose a candidate selection approach that fits linear regressions of excess loss against log model size and selects three promising repetition counts for evaluation at larger scales. The authors also develop empirical scaling models to describe how loss changes with repetition and estimate loss-minimizing repetition counts.
Critical strengths
- The paper addresses a practically relevant problem in data-constrained pretraining, where selecting an appropriate repetition count can affect both training cost and held-out target loss.
- The proposed candidate selection method identifies a small set of promising repetition counts using smaller-model loss curves, potentially reducing the number of configurations that need to be evaluated at larger scales.
Critical weaknesses
-
The main experiments are limited to models up to 520M parameters within a single Transformer family, using a fixed 100M-token target corpus and a target-data fraction of 0.8. Although the authors acknowledge these limitations, broader evaluations across model scales, architectures, and data configurations would strengthen the empirical evidence for generalizability. Additionally, the evaluation relies primarily on held-out cross-entropy loss without assessing downstream task performance.
-
The proposed candidate selection method is evaluated under a fixed target-data fraction, where increasing repetition also increases generic-data exposure and the total training budget. In contrast, Li et al. (2026) report mildly increasing preferred repetition counts with model size under fixed TPP, where the total token budget is held constant across repetition counts at each model size. Although the authors acknowledge this difference, the proposed candidate selection method has not been evaluated under fixed-TPP settings. Further evaluation would provide stronger evidence for its generalizability.
- Li et al. (2026): Scaling Domain Data Repetition in LLM Pretraining, https://arxiv.org/abs/2608.14071
-
While the empirical scaling models capture the observed repetition-loss trends, their predictions of preferred repetition counts appear less reliable at 520M (Section 6.2 and Appendix E.3.3). Allowing the scaling exponents to vary or introducing saturation does not consistently improve these predictions. Although the authors acknowledge this limitation and report more favorable results for the simple model at 200M, further discussion of the factors affecting cross-scale prediction reliability would be helpful.
-
Given the high proportion of target data in the training mixture, the experimental setup appears relevant to domain-focused pretraining. However, practical domain adaptation often involves continued pretraining of existing LLMs rather than training models from scratch. It would therefore be useful to investigate whether the proposed repetition selection method remains effective when starting from pretrained checkpoints.
Questions for authors
- Is there any additional evidence supporting the generalizability of the proposed method to larger models, different data configurations, or downstream tasks?
- Would the proposed candidate selection method remain effective under fixed-TPP settings, where varying repetition changes the target-data fraction rather than the total training budget?
- What factors might explain the more favorable point predictions at 200M compared with the less reliable extrapolations to 520M?
- Would the proposed repetition selection method remain applicable to continued pretraining from existing pretrained checkpoints?
Thank you for the clarifications and additional experiments. The comparison with Sedova et al. addresses part of my concerns. However, the practical advantage over the fixed candidate set remains unclear, and the evidence is still limited to a narrow setting. I will raise my soundness score to 3, but maintain my overall rating.