- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how often to reuse a high-quality domain corpus during language-model pretraining when the token budget grows with model size. With N parameters, the training budget is D_N = TPP × N. The authors repeat a subset of αD_N unique domain tokens e times and fill the remaining budget with fresh web data. They study Code, Math, Wiki, and Medical data across four model sizes from 348M to 1.85B, three values of α, and a repetition grid from 1 to 7.
The preferred repetition count depends strongly on the domain, increases slightly with model size at fixed TPP, and changes little with α over the tested range. Figure 4 reports correlations of −0.944 with minimum validation loss, 0.400 with model size, and 0.018 with α, based on quadratic fits. A stylized linear-regression analysis links excessive repetition to fitting sample-specific noise. Further experiments compare repeated and fresh domain tokens at fixed mixture fractions and show that the learning-rate schedule also affects how much repetition is beneficial.
Critical strengths
-
Keeping TPP fixed is a useful way to study repetition during model scaling, rather than extrapolating from experiments with a fixed total token budget. The setup in Sections 1 and 2 is clear, and Figure 2 makes the mixture construction easy to follow. Testing four domains, four model sizes, and three unique-data fractions gives a much more informative picture than a single recommended repetition count.
-
The comparison with fresh data is one of the more convincing parts of the paper. Figure 6 keeps the total domain fraction fixed, so repeated tokens replace unique tokens rather than simply displacing web data. Math loses relatively little with up to four presentations, while Wiki degrades much more sharply. This shows why the cost of reuse needs to be assessed separately for each domain.
-
The analysis offers a plausible explanation for the observed domain differences, and the schedule ablation adds an important qualification. Section 3 separates fitting signal from fitting sample-specific noise. In Figure 7, delaying or removing learning-rate decay allows more repetitions. I find this useful because it shows that the best repetition count is not just a property of the data, which matters when transferring a recipe from a proxy model.
Critical weaknesses
-
I am not convinced that minimum validation loss is yet a predictor that generalizes across domains. The points in Figure 4 fall into four visibly separated domain clusters, so much of the pooled correlation could reflect domain identity. Both the minimum loss and the estimated optimum also come from the same curve and quadratic fit. A within-domain analysis or a held-out-domain prediction would help distinguish a transferable relationship from these dependencies. Theorem 3.2 gives a plausible account under fixed N, α, retained support, and signal assumptions, but does not establish that language-model losses measure comparable noise levels across the four domains.
-
The paper may place too much emphasis on the repetition count when the repetition policy itself could be equally important. Figure 7 already shows that changing the learning-rate schedule shifts the point at which repeated data begins to hurt, even when the data recipe is otherwise unchanged. This suggests that an optimal count partly depends on when repeated examples are encountered under a particular optimization regime. The paper does not compare different ways of distributing the same repeated subset through training, such as uniform mixing, early reuse, or later-stage reuse. Without that comparison, it is unclear whether the count is the main transferable quantity or a proxy for a broader repetition schedule.
-
The practical scope of the scaling recommendation remains unclear beyond the small models and data-constrained setting studied here. The experiments cover 348M to 1.85B parameters, so the mild increase in the preferred repetition count does not by itself justify extrapolation to models with tens of billions of parameters. Without uncertainty estimates, even the stability of the trend within this range is difficult to assess. It is also unclear when repetition remains the right choice if more high-quality unique domain data can be collected or produced through quality-controlled synthesis. Figure 6 already shows that fresh domain tokens are generally preferable to repeated ones at a fixed domain fraction, but it does not establish the conditions under which reuse remains worthwhile once corpus expansion is feasible. Repetition may still be useful when domain data is genuinely scarce. The broader recommendation needs clearer boundaries on model scale and data availability, along with domain-task results showing that the selected counts improve useful capabilities rather than validation loss alone.
-
Although I can follow the paper’s main ideas, the main figure appears to have been generated using GPT-Image. This presentation weakens my confidence in the paper and raises concerns about whether the manuscript itself may also be AI-generated. I encourage the authors to redraw the figure in a more precise and conventional academic style in a future revision.
Questions for authors
-
How much of the −0.944 correlation in Figure 4 remains after accounting for domain identity? Could you report a within-domain or domain-adjusted analysis and test prediction on a held-out domain? Please also assess the dependence introduced by deriving both axes from the same validation-loss curve.
-
Could you report repeated runs and uncertainty intervals for the loss curves and fitted optima, along with downstream domain results for the selected repetition counts?
3 comments
The author responded to my concerns quite well, candidly admitted the shortcomings of some of the discussions, and promised to make revisions. In the latest revised version, the main image I suggested has also been modified. Therefore, I raise my rating to 3.
Thank you for your feedback! We will incorporate the relevant experiments and additional discussions into the next revision of the paper.
We sincerely thank the reviewer for the comments and suggestions. We address the questions below.
A1: We thank the reviewer for this comment. On the 48 settings in Figure 4, we report (i) the partial correlation controlling for domain, (ii) the within-domain correlation, and (iii) the held-out-domain error of a linear fit from minimum loss to optimal count on the other three domains, compared with predicting their mean optimal count. To address the dependence between the two axes, we also replace the minimum loss with the loss at e=1, which does not depend on where the curve reaches its minimum.
The correlation (−0.944 pooled) remains strongly negative after controlling for domain (−0.837) and within each domain (−0.860 on average), and the loss-based prediction has a much lower average held-out error than the baseline (0.32 vs. 0.82), suggesting that the relationship is not merely driven by domain identity. The two predictors are comparable only on Code, whose mean optimal count is close to that of the other domains. Using the loss at e=1 gives a similar correlation (−0.945), so this reduces the concern that the correlation is induced by extracting both quantities from the same fitted minimum. This predictor is also more practical, as it requires only one model per domain.
Within each domain, the relationship mainly comes from model size, as larger models reach lower loss and tolerate more repetitions (correlation of the optimal count with log model size: 0.86 to 0.92; with α: 0.02, pooled). Increasing α lowers the loss without changing the optimal count (Figure 3), consistent with Theorem 3.2: since UNlog(⋅) in the denominator is approximately constant for large UN, the leading term of eN∗(σ2) depends on σ2 but not on the amount of unique data.
Finally, we agree that Theorem 3.2 does not directly show that the losses of different domains reflect comparable noise levels; we use the loss as an empirical proxy and will clarify this in the next version.
A2: We thank the reviewer for this insightful comment. Our paper studies the repetition count under one fixed policy, and Section 4.2 shows that changing the learning-rate schedule substantially shifts the optimal count, which supports the reviewer's point. A direct comparison of policies, e.g., spreading repeated data uniformly versus concentrating it early or late, would be valuable; we cannot run it within the discussion period but will add it in the next version.
A3: We thank the reviewer for this comment. We did not train the same configuration with multiple seeds, since each run is costly at these scales. As an alternative, the three unique-data fractions at each model size correspond to three independently trained models, and Theorem 3.2 predicts the optimal count to be independent of α. The fitted optimal repetition counts across the three fractions (mean ± std) are:
The spread at a fixed model size (mean std of 0.14 over the 16 configurations) is much smaller than the increase across model sizes, and the monotone increase holds in all four domains. In the next version, we will add explicit seed repetitions with uncertainty intervals, as well as downstream evaluation, which we agree would be more decisive than validation loss.
A4: We appreciate the suggestion. As stated in our AI use statement, we used generative AI tools to assist with the visualization of the experimental framework in Figure 2, while all experiments and results were produced by the authors. We have redrawn Figure 2 in a conventional academic style. You can see it in the paper.
We thank the reviewer once again for the valuable and helpful suggestions. We are happy to provide further clarifications if the reviewer has any additional questions.