- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies several scenarios of data-constrainedness at small scale: a low resource language together with an abundant language; a tunable quality threshold (scarce high-quality data vs. abundant low-quality data); a multi-domain setting. The paper proposes two scaling laws and (one fit per model size, the other fit jointly) to predict optimal repetition counts for data-constrained domains and shows they achieve better fits than some other baselines.
Critical strengths
The paper covers several interesting scenarios representative of real-world challenges: e.g., (simulated) data-constrained low-resource language + unconstrained English text; trade-off between less unique tokens at higher quality thresholds versus more unique tokens at lower quality thresholds; and a multi-domain setting.
Critical weaknesses
The paper throughout studies fairly small models (up to 500M parameters) and fixed training horizons up to 10B tokens. Usually, scaling laws are meant to enable transferring conclusions from smaller proxy models such as these to models at larger scales at, e.g., constant TPP (tokens-per-parameter). However, in this paper, the focus seems to be more on an scaling law that must be fit for each model size separately. (Another scaling law is also presented but with little discussion and results.) The utility of this is a bit unclear to me: for small models and small training horizons, there are usually enough unique tokens and data-constrainedness is not a big concern; but for big models where data-constrainedness is a significant concern, it may be too expensive to fit a scaling law for that model size.
Questions for authors
How do you practically suggest to use your scaling laws for real-world large models trained for large token horizons, where data constrainedness is a real concern? Or else, do you have more to say on the utility of your scaling law?
3 comments
Thank you again for your constructive feedback, we hope our responses and the revised paper have addressed your concerns. As the discussion period is coming to a close, please let us know if any questions remain, and we will be happy to address them.
Thank you for your detailed reply and updating the paper. The material you added to the main text significantly improves the merits of the paper and I have raised my rating to 3.
Thank you for your review! We address the main concerns you raised below.
First, we would like to clarify the scale of our experiments: as stated in Section 3 (L208–210), our models span not up to 500M but up to 1.2B parameters and are trained for up to 54B tokens for the multilingual models and up to 65.5B for the Maths models (see Appendix B). The 10B-token horizons in the main-text examples (e.g., Figure 3) correspond only to our smaller models.
Indeed, fitting
L_fixat the target scale is impractical for large models, andL_sizeis designed exactly for this: it is fitted on a ladder of small models and extrapolated to a larger target. Its data-size coupling termN^δcaptures that larger models overfit faster yet reach lower minima (Figure 5), and at fixed N it reduces exactly toL_fix. Fitted on all but the largest model,L_sizepredicts held-out loss on the unseen largest model with wR² 0.65 on German, 0.73 on Maths (Table 8), 0.93 on French and 0.69 on Swahili (Table 17), higher than all multi-size baselines on German, Maths and French (e.g., 0.65 vs 0.56 for utility decay+N on German). Moreover, fitted on only the three smallest model sizes, it remains predictive, extrapolating 2.7× in parameters to 539M (wR² 0.59 / 0.88 / 0.64 on German / French / Swahili, Table 16). On the unseen model,L_sizeis also the only formula whose recommended mixture avoids catastrophic failure on both German and Maths (Table 15). We used the additional page to bring these results into the main text: Section 6 ("Size extrapolation", L423–457, marked in blue) summarises the held-out-size accuracy (Table 8), the wasted-token comparison (Table 15) and the three-model fitting ladder (Table 16), which remain in the appendix, together with the cost of fitting the law relative to a direct search.To demonstrate the practical usefulness of
L_sizeand address the concern about scale, we validated it at 2B scale:L_sizefitted on 101M–539M models predicted the optimal German weight for a 2B model trained on 100B tokens (h* = 0.199), and we trained 2B models at h = 0.025, 0.199 and 0.625.Table A: Target (German) validation loss:
Table B: Target (German) downstream 0-shot accuracy at the same checkpoints.
The predicted optimum gives the best target-language loss, and downstream accuracy reproduces the loss results, so the effect reported in the paper persists at the larger 2B scale, a practically relevant scale used by widely adopted open model families (e.g., Qwen3.5-2B, Gemma-2-2B) and common in multilingual and on-device settings, where target-language data is scarce and repetition matters most. We have added the 2B validation to Section 6 (paragraph "Validation at 2B scale", L459–469 and Table 3, marked in blue).