- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This papers studies the effect of data repetition on LLM pre-training. Under fixed 90% non repeat training budget and fixed 10% repeat training budget, we can repeat 1% of data 10 times, or 0.1% of data 100 times, or 0.01% of data 1000 times. This paper's main finding is that repeating a moderately sized subset a moderate number of times damages performance more than repeating a large subset a few times or a small subset many times. The paper then discusses the most damaging repetition frequency's relation to model's parameter counts, and how the observed intermediate peak observation in data repetition can be explained by a misspecified linear regression model.
Critical strengths
- The paper is well written and easy to follow.
Critical weaknesses
It is unclear what new knowledge / findings this paper presents, especially considering Hernandez et al. (2022)
- The main finding of this work, that there is an intermediate peak of performance cost due to data repetition, is the same from Hernandez et al. (2022), which the authors acknowledged.
- While this paper measures it in compute-equivalent gains & losses and Hernandez et al. (2022) uses model size as one example to illustrate the degree of data repetition cost, the change of measurement does not warrant new knowledge / findings. The observed phenomenon has been consistent of Hernandez et al. (2022)
- Both Hernandez et al. (2022) and this works reports validation loss on a held out validation set. compute-equivalent gains & losses, and model parameter based comparisons are just ways to interpret the magnitude of performance degradation.
The range of experiments conducted has been smaller than Hernandez et al. (2022)
- Hernandez et al. (2022) trained models from 1M to 800M, this paper trained models from 34M to 344M.
- Hernandez et al. (2022) trained all models for 100B tokens. this paper trained the largest 344M models for 6.88B tokens.
- Therefore, Hernandez et al. (2022) reported the same observation on a strictly larger sweep of pre-training experiments. It is challenging to see the findings from this work as generalizing beyond Hernandez et al. (2022).
Questions for authors
I find the title "INTERNAL DATA REPETITION DESTROYS LANGUAGE MODELS" misleading and should be made more accurate.
- "DESTROYS" is a strong word that the main finding of this paper does not support.
- What is "internal" data repetition? could you define what internal means? It is unclear.
- The current title is a strong and general claim. The core observation/finding from the main paper is narrower and more concrete. The gap is misleading.
"We demonstrate that these phenomena are not language model specific, and can be analytically understood in a simple statistical model"
- This is slightly overclaim to me. When the authors say "We demonstrate that these phenomena are not language model specific", I would expect experiments of other models, such as vision models, diffusion models, etc, that also shows the intermediate peak of data repetition to performance degradation.
- It would be more clear and consistent to say that we could explain XXX from a simple statistical model.
2 comments
Part 2
[Continued from the previous comment.]
2. Experimental scope: more tokens do not subsume different training regimes
The review compares parameter ranges and token counts, then concludes that Hernandez conducted a “strictly larger sweep.” That conclusion overlooks the training-budget dimension.
Hernandez covers a wider parameter range and uses more tokens per natural-language model. It does not cover our model-size-dependent training allocations. For example, 100B tokens for its approximately 805M model corresponds to roughly 124 tokens per parameter, compared with our OT=1 reference of 20. Its smaller models have much higher ratios still.
Our 6.88B-token budget at 344M is deliberate: it follows the allocation being tested. Increasing it to 100B tokens would move the experiment into a substantially different overtraining regime. More training tokens would not substitute for measuring repetition damage at the intended allocation.
Our completed grid contains 25 model-size/overtraining combinations. The existing experiments have already cost upwards of $40,000 in compute. Within our academic budget, we prioritized coverage across repetition structure, model size, and training duration. We do not claim a wider parameter range than Hernandez; we investigate training-budget regimes that its fixed-token design does not resolve. Hernandez ran their experiments at Anthropic, which is as of Nov 18 2026 planned to IPO at a ~1.5T USD valuation -- we don't have access to that level of capital resourcing.
3. The statistical model adds an explanation, not another observation of the same curve
The regression analysis identifies how literal duplication changes the statistical problem. Copies of an example share the same unobserved features; they do not provide independent observations. We account for that dependence explicitly and derive conditional training and test errors.
When pool size and repetition vary jointly, the resulting simulations reproduce a peak and recovery. This establishes that the qualitative pattern can arise without attention, model depth, or transformer-specific optimization. It supplies an explicit statistical mechanism rather than relying solely on an informal memorization hypothesis.
The regression result does not claim to predict transformer losses or establish the same behavior in every model family. Its contribution is an example that explains how the pattern can arise.
4. Title and the meaning of “internal”
“Internal” refers to repetition within a single model’s pretraining stream: selected documents recur alongside otherwise non-repeated training documents. We will define this directly to address your feedback -- thanks for your suggestion!
5. Scope of “not language-model-specific”
Our linear-regression result demonstrates the qualitative behavior in a model outside language modeling. The intended claim is that transformer-specific ingredients are not necessary for such a pattern to arise. We're not trying to imply that we tested vision or diffusion models, but we're happy to make that clarification more explicit!
Thank you again for the review. We hope this helps address your concerns, and if it does, that you'll consider raising our score.
Part 1
Thank you for the review and for noting that the paper is clear and easy to follow. The review treats the intermediate loss peak as our primary contribution. Our paper instead asks how much compute repetition wastes and which repetition structures waste the most as models scale. We clarify these distinctions below.
1. Contribution and novelty
The central result is that repetition structure can waste a substantial share of a training budget even when the resulting loss increase looks small. At 344M parameters, allocating 10% of training tokens to the most damaging repetition structure produces a loss that our fitted no-repetition reference reaches with approximately 33% less compute.
This is a quantitative finding about training efficiency, supported by a calibrated loss/compute relationship. It cannot be inferred from the existence of an intermediate loss peak alone.
We make three contributions:
Why compute-equivalent cost is more than relabeling validation loss
Both studies evaluate held-out loss, but this does not make their quantitative conclusions interchangeable. A small loss increase can correspond to a large amount of wasted compute because further improvements in loss become increasingly expensive. Measuring that cost requires an appropriate reference relationship.
Hernandez expresses damage through effective parameter count along a fixed-token reference. We measure the compute required by a fitted no-repetition reference to reach the same loss. These answer different resource-allocation questions. Our approximately 33% compute-equivalent loss at 344M is a measured consequence under that calibration, not simply another name for a few-percent loss increase. We also report its calibration sensitivity.
Why the training-budget distinction matters
Hernandez trains every natural-language model on 100B tokens. The tokens-per-parameter ratio therefore changes substantially across model sizes. We instead scale tokens with model size and sweep training allocations, allowing us to study repetition at controlled overtraining multipliers.
Hernandez explicitly identifies fixed-token training as its first limitation and a compute-efficient-frontier scan as its first future direction. Our experiments address that unanswered question using Chinchilla-style budgets. We use 20 tokens per parameter as a reference convention.
The scaling measurements also differ: Hernandez fits boundaries of a harmful-repetition region under fixed-token training; we estimate peak locations under model-size-dependent budgets. These are different quantities. The earlier boundary fits do not establish the peak-location relationship measured here.
The contribution therefore does not rest on discovering that repetition can hurt. It lies in measuring its compute cost, characterizing the worst repetition structures under controlled budget scaling, and deriving a statistical description of its impact.
[Continued in the next comment.]