- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how reusing data affects language-model pretraining and whether total token counts adequately describe the value of repetition. The authors distinguish between adding more passes over a fixed dataset and replacing fresh data with repeats under the same training budget. They examine how the distribution and order of repetitions, model size, and tokenization affect held-out language-modeling loss. Experiments cover multiple data sources and models up to 2B parameters. The experiments show that additional passes over the same data can reduce loss, although fresh data generally gives better results at the same training budget. Spreading repetitions across samples and over time also works better than concentrating them, even when the underlying data and total token counts remain unchanged.
Critical strengths
- The distinction between additional training and replacing fresh data is useful. These comparisons answer different practical questions about data reuse. Separating them helps explain why additional passes can improve a model even when fresh data would provide a better use of the same training budget.
- The comparisons with identical data and token counts help isolate the effects of repetition patterns. Changing which samples receive additional exposures, or when those exposures occur, provides direct evidence that aggregate token counts do not fully describe how data reuse affects training.
- The main comparison is supported by checks across runs and data sources. Matching the initialization across conditions helps reduce unrelated training variation. The consistent direction of the effect across sources gives some confidence that concentrating repetitions produces a small but reproducible increase in loss.
Critical weaknesses
- The paper lacks a sufficiently focused central argument. It moves between several questions about data reuse and measures such as fresh-token equivalence, loss penalties, and optimal repeat counts. Their connections are not explained well enough to show how the findings build toward a common conclusion. This makes the main contribution difficult to identify beyond the broad statement that token counts alone are insufficient.
- The presentation makes the experiments difficult to follow and assess. The reader must repeatedly check what is held fixed, how model sizes are counted, and whether results use the same evaluation setup. Important explanations are spread across the main text, figure captions, and appendices. This makes it difficult to understand individual comparisons and judge how strongly they support the conclusions.
- The additional insight beyond existing accounts of data reuse is limited. The comparison with identical data and token counts shows a small cost from concentrating repetitions, but Appendix F explains its direction and approximate size using a per-sample version of an existing diminishing-returns model. These measurements are useful, but the paper does not clearly establish what deeper understanding they add beyond confirming the expected effect of repeated exposure.
- The scaling analysis offers limited guidance for new training settings. The larger-model results helped determine the form of the fitted model, so they do not provide an independent test of its predictions. At smaller data budgets, the fit predicts that more repetition should help when the measured loss instead increases. Its usefulness for choosing how many passes to use in a new setting therefore remains unclear.
Questions for authors
- What is the paper’s central claim, and which results most directly support it? The paper covers several questions using different experimental setups and measures of repetition value, making it difficult to distinguish the main contribution from the additional observations. Could you clarify the most important finding, how the individual analyses support it, and which results can be compared directly?
- Does the explanation in Appendix F also account for the other allocation results? Appendix F shows that diminishing returns at the sample level can explain the direction and approximate size of the main concentration effect. Could you compare the same model’s predictions with other allocation results already reported in the paper? This would help clarify what these experiments teach us beyond that existing explanation.
- Can the scaling model predict a useful number of training passes in a new setting? The larger-model results helped choose the formula, while the smaller-budget tests also use a different experimental setup. Could you keep the current fit fixed and test one new data budget with a small model, using the original training and evaluation setup? A few measurements around the predicted best number of passes would help assess whether the model can guide repetition choices.
6 comments
Response to Reviewer pQzV (Part 2/2)
Re: Weakness 3 / Q2.
We tested it on all 27 nonfresh source–allocation means from the original study, keeping the original parameters fixed. The underlying experiment fixes the 45M model and targets and varies the repeated-budget fraction and subset multiplicity across nine allocations on CommonCrawl, GitHub and Books. These are or with ; with ; and . Each allocation is compared with fresh training at the same model size and budget. The single-pair calculation is now in Appendix J; this wider check is reported here and has not yet been added to the PDF.
With the fraction of the training budget assigned to the repeated subset, its multiplicity and the total budget, the original benefit-only extension is
D_{\mathrm{eff}}=D(1-f)+\frac{fD}{r}g(r).$$ It predicts excess loss $B(D_{\mathrm{eff}}^{-\beta}-D^{-\beta})$, using $B=4450$, $\beta=0.485$ and $R_D=4.545$. We omit the full-corpus penalty, as in the original calculation, because its partial-repetition extension was not specified. | Saturation constant | Predicted concentration difference | MAE across 27 means | RMSE across 27 means | |---|---:|---:|---:| | $R_D=4.545$ | +0.00460 bpb | 0.02399 bpb | 0.04603 bpb | | $R_D=15.4$ sensitivity check | +0.00158 bpb | 0.03598 bpb | 0.06329 bpb | The original parameters still give the right order of magnitude for the +0.0039-bpb concentration contrast, but transfer less well across allocations. For CommonCrawl at $f=1/2$, $r=32$, the prediction is 0.0630 bpb and the measured penalty is 0.1607 bpb. This limits the quantitative reach of that frozen explanation. It does not invalidate all sample-wise models: the coefficients were estimated under a different protocol and metric, the full-corpus penalty is absent, and the 27 means share fresh baselines rather than constituting independent replications. > Weakness 4 / Q3 The requested test has not been completed. The larger-model results informed the choice of functional form, even though they were excluded from coefficient fitting; they are therefore not a fully independent prediction test. The smaller-budget check also changes protocol. Neither the new measurement study nor the 7B extension answers your specific request to freeze the original model and test a new $U$ under the original protocol. We retain the small-budget and 64-epoch failures in Appendices J and K. Section 4.4 limits the newer allocation analysis to a fitted description: its predicted advantage is at most 0.002 bpb, and proximity to $R_c$ is not a validated stopping rule. Testing pass selection would require separately annealed runs around the frozen prediction at a new $U$, including simpler epoch choices as baselines. We hope the clearer separation of comparisons and the frozen-model check address your comments on the argument and its supporting evidence. The independent pass-selection test remains open. We would appreciate a careful reassessment of the empirical contribution and consideration of a higher rating if the revisions make that contribution more convincing.Dear Reviewer pQzV,
Thank you again for your time and thoughtful feedback. We have also revised the manuscript to improve clarity and address the issues raised. We hope these revisions, together with our responses and additional evidence, have helped address your concerns. We welcome any follow-up questions and would be happy to clarify further. We would appreciate your reconsideration of your assessment and rating in light of these updates.
Thank you for considering our work.
Response to Reviewer pQzV (Part 1/2)
Thank you for recognizing the usefulness of separating additional training from fresh-data replacement, and the value of the matched-content comparisons and their consistency across sources. We agree that the submission did not connect its questions clearly enough.
The first claim is that conclusions about repetition depend on the comparison baseline and the resource constraint. Repeating data can improve on stopping after one pass, yet yield higher loss than fresh-data training at the same budget. Across model sizes, fixing unique data U, unique tokens per parameter U/N, or training tokens per parameter D/N also answers different questions. The second claim is that matching these aggregate counts still leaves exposure structure unspecified.
The revised manuscript follows that argument:
The new 45M CommonCrawl concentration control described in our response to 5jzj matches marginal adjacent replay-gap distributions and the global first/repeat event calendar. It yields a small positive difference of +0.00101 bpb (95% paired-t CI [0.00050, 0.00152]; six paired training replicates), with full recency distributions and higher-order histories still different. This result is not yet in the PDF and does not constitute the independent pass-selection test requested below.
The fresh references also expose a limit of benefit-only models. Of 80 repeated configurations with a directly measured one-epoch reference at the same U, 21 end above that reference. A model based on a monotone fresh-data curve with 0≤η≤1 and no degradation term cannot represent those endpoints. Most configurations have one seed, so this is an observed pattern, not evidence of significant degradation at every point.
We have organized Sections 3–6 around the comparison baseline and the quantity held fixed, integrated short definitions into the prose, and simplified result descriptions and captions. Appendix A consolidates the experiment families and parameter conventions. These changes are intended to make each comparison easier to identify before interpreting its result.
Direct comparisons stay within each model, source, recipe and evaluation family. We compare directions across families without pooling absolute bpb values. Tables 1–4 in Appendix A distinguish exact non-embedding sizes, the total-parameter count used in z, and the proxy labels of the legacy sweeps. Budgets in the newer study are nominally matched, with integer update rounding. The coordinate z and sample-wise saturation build on prior work; our contribution is to measure how the conclusions change across conditions and where those descriptions fall short.