acceptodds
Under review as a conference paper at ICLR 2027

Internal Data Repetition Destroys Language Models

Abstract

Language models are running out of high-quality training data, and even aggressively deduplicated corpora retain some amount of repetition. Earlier controlled studies predated Chinchilla-style scaling laws and could only measure the cost of repetition indirectly. We revisit repetition in the Chinchilla era, using a fitted no-repetition scaling law to report Compute-Equivalent Gain and Compute-Equivalent Loss. We show that under this modernized paradigm, repetition damage is systematic in three ways. First, when holding the amount of compute allocated to repeated data constant, eval loss peaks at an intermediate repeat count R; repeating a moderately sized subset a moderate number of times damages performance more than repeating a large subset a few times or a small subset many times. Second, the peak location is well described by a power law in model size; within the range studied, this trend shows that the most damaging repeated-data pool grows faster than the total training-token budget at a fixed overtraining multiplier. Finally, when repeated documents consume 10% of the FLOPs budget in a controlled exact-document repetition setting, the compute-equivalent loss can be large: on FineWeb-Edu-Dedup, the fitted peak for a Qwen3-style 344M-parameter model at OT=1 matches the loss predicted by a no-repetition calibration using about 67% of the FLOPs. We demonstrate that these phenomena are not language model specific, and can be analytically understood in a simple statistical model: a misspecified linear regression with verbatim duplicates reproduces the same qualitative loss peak, quantifying how such peaks can arise from a statistical tradeoff between memorization and generalization. Our findings add precision to the study of duplication in language models, allowing practitioners to quantify the wasted compute incurred by the presence and repeat structure of duplicates in pretraining corpora.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language model pretraining, training data repetition, data deduplication, memorization and generalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QKFAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This papers studies the effect of data repetition on LLM pre-training. Under fixed 90% non repeat training budget and fixed 10% repeat training budget, we can repeat 1% of data 10 times, or 0.1% of data 100 times, or 0.01% of data 1000 times. This paper's main finding is that repeating a moderately sized subset a moderate number of times damages performance more than repeating a large subset a few times or a small subset many times. The paper then discusses the most damaging repetition frequency's relation to model's parameter counts, and how the observed intermediate peak observation in data repetition can be explained by a misspecified linear regression model.

Critical strengths
  • The paper is well written and easy to follow.
Critical weaknesses

It is unclear what new knowledge / findings this paper presents, especially considering Hernandez et al. (2022)

  • The main finding of this work, that there is an intermediate peak of performance cost due to data repetition, is the same from Hernandez et al. (2022), which the authors acknowledged.
  • While this paper measures it in compute-equivalent gains & losses and Hernandez et al. (2022) uses model size as one example to illustrate the degree of data repetition cost, the change of measurement does not warrant new knowledge / findings. The observed phenomenon has been consistent of Hernandez et al. (2022)
  • Both Hernandez et al. (2022) and this works reports validation loss on a held out validation set. compute-equivalent gains & losses, and model parameter based comparisons are just ways to interpret the magnitude of performance degradation.

The range of experiments conducted has been smaller than Hernandez et al. (2022)

  • Hernandez et al. (2022) trained models from 1M to 800M, this paper trained models from 34M to 344M.
  • Hernandez et al. (2022) trained all models for 100B tokens. this paper trained the largest 344M models for 6.88B tokens.
  • Therefore, Hernandez et al. (2022) reported the same observation on a strictly larger sweep of pre-training experiments. It is challenging to see the findings from this work as generalizing beyond Hernandez et al. (2022).
Questions for authors

I find the title "INTERNAL DATA REPETITION DESTROYS LANGUAGE MODELS" misleading and should be made more accurate.

  • "DESTROYS" is a strong word that the main finding of this paper does not support.
  • What is "internal" data repetition? could you define what internal means? It is unclear.
  • The current title is a strong and general claim. The core observation/finding from the main paper is narrower and more concrete. The gap is misleading.

"We demonstrate that these phenomena are not language model specific, and can be analytically understood in a simple statistical model"

  • This is slightly overclaim to me. When the authors say "We demonstrate that these phenomena are not language model specific", I would expect experiments of other models, such as vision models, diffusion models, etc, that also shows the intermediate peak of data repetition to performance degradation.
  • It would be more clear and consistent to say that we could explain XXX from a simple statistical model.
2 comments

Thank you for the review and for noting that the paper is clear and easy to follow. The review treats the intermediate loss peak as our primary contribution. Our paper instead asks how much compute repetition wastes and which repetition structures waste the most as models scale. We clarify these distinctions below.

1. Contribution and novelty

The central result is that repetition structure can waste a substantial share of a training budget even when the resulting loss increase looks small. At 344M parameters, allocating 10% of training tokens to the most damaging repetition structure produces a loss that our fitted no-repetition reference reaches with approximately 33% less compute.

This is a quantitative finding about training efficiency, supported by a calibrated loss/compute relationship. It cannot be inferred from the existence of an intermediate loss peak alone.

We make three contributions:

  • Quantifying wasted compute. We fit a no-repetition loss–compute relationship and translate repetition damage into compute-equivalent cost. A percentage change in validation loss does not directly specify this cost: its compute implications depend on the reference curve and training scale.
  • Characterizing repetition under controlled budget scaling. We scale training tokens with model size and sweep overtraining multipliers. Within these budgets, we identify the most damaging repetition structures and estimate how their locations change with scale.
  • Deriving a statistical explanation. We derive conditional training and test errors for misspecified linear regression with literal duplicates. The analysis and simulations show how duplication can produce a peak and recovery without attention or other transformer-specific mechanisms.

Why compute-equivalent cost is more than relabeling validation loss

Both studies evaluate held-out loss, but this does not make their quantitative conclusions interchangeable. A small loss increase can correspond to a large amount of wasted compute because further improvements in loss become increasingly expensive. Measuring that cost requires an appropriate reference relationship.

Hernandez expresses damage through effective parameter count along a fixed-token reference. We measure the compute required by a fitted no-repetition reference to reach the same loss. These answer different resource-allocation questions. Our approximately 33% compute-equivalent loss at 344M is a measured consequence under that calibration, not simply another name for a few-percent loss increase. We also report its calibration sensitivity.

Why the training-budget distinction matters

Hernandez trains every natural-language model on 100B tokens. The tokens-per-parameter ratio therefore changes substantially across model sizes. We instead scale tokens with model size and sweep training allocations, allowing us to study repetition at controlled overtraining multipliers.

Hernandez explicitly identifies fixed-token training as its first limitation and a compute-efficient-frontier scan as its first future direction. Our experiments address that unanswered question using Chinchilla-style budgets. We use 20 tokens per parameter as a reference convention.

The scaling measurements also differ: Hernandez fits boundaries of a harmful-repetition region under fixed-token training; we estimate peak locations under model-size-dependent budgets. These are different quantities. The earlier boundary fits do not establish the peak-location relationship measured here.

The contribution therefore does not rest on discovering that repetition can hurt. It lies in measuring its compute cost, characterizing the worst repetition structures under controlled budget scaling, and deriving a statistical description of its impact.

[Continued in the next comment.]

[Continued from the previous comment.]

2. Experimental scope: more tokens do not subsume different training regimes

The review compares parameter ranges and token counts, then concludes that Hernandez conducted a “strictly larger sweep.” That conclusion overlooks the training-budget dimension.

Hernandez covers a wider parameter range and uses more tokens per natural-language model. It does not cover our model-size-dependent training allocations. For example, 100B tokens for its approximately 805M model corresponds to roughly 124 tokens per parameter, compared with our OT=1 reference of 20. Its smaller models have much higher ratios still.

Our 6.88B-token budget at 344M is deliberate: it follows the allocation being tested. Increasing it to 100B tokens would move the experiment into a substantially different overtraining regime. More training tokens would not substitute for measuring repetition damage at the intended allocation.

Our completed grid contains 25 model-size/overtraining combinations. The existing experiments have already cost upwards of $40,000 in compute. Within our academic budget, we prioritized coverage across repetition structure, model size, and training duration. We do not claim a wider parameter range than Hernandez; we investigate training-budget regimes that its fixed-token design does not resolve. Hernandez ran their experiments at Anthropic, which is as of Nov 18 2026 planned to IPO at a ~1.5T USD valuation -- we don't have access to that level of capital resourcing.

3. The statistical model adds an explanation, not another observation of the same curve

The regression analysis identifies how literal duplication changes the statistical problem. Copies of an example share the same unobserved features; they do not provide independent observations. We account for that dependence explicitly and derive conditional training and test errors.

When pool size and repetition vary jointly, the resulting simulations reproduce a peak and recovery. This establishes that the qualitative pattern can arise without attention, model depth, or transformer-specific optimization. It supplies an explicit statistical mechanism rather than relying solely on an informal memorization hypothesis.

The regression result does not claim to predict transformer losses or establish the same behavior in every model family. Its contribution is an example that explains how the pattern can arise.

4. Title and the meaning of “internal”

“Internal” refers to repetition within a single model’s pretraining stream: selected documents recur alongside otherwise non-repeated training documents. We will define this directly to address your feedback -- thanks for your suggestion!

5. Scope of “not language-model-specific”

Our linear-regression result demonstrates the qualitative behavior in a model outside language modeling. The intended claim is that transformer-specific ingredients are not necessary for such a pattern to arise. We're not trying to imply that we tested vision or diffusion models, but we're happy to make that clarification more explicit!

Thank you again for the review. We hope this helps address your concerns, and if it does, that you'll consider raising our score.

Reviewer_djZwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors isolate the controlled case of "exact document-level replay of a selected repeated pool" (about 10%) and precisely quantify the cost incurred by exact repeats during pre-training.
Across six sizes of Qwen3-style models, the study finds that the worst case repetition structure is a function of the model size and there exists an intermediate-repetition regime where damage peaks. Power laws fit to these peaks allow translation of losses into compute equivalent units. Further statistical analysis reproduces the non-monotonic behavior in closed form suggesting that the observed behavior of peak and recovery may have statistical explanations that are not specific to attention, model depth or optimizer specifics.

Critical strengths
  1. This work is clear, well motivated and addresses a real gap regarding data repetition with a focus on exact repeats of documents in pre-training. This is a very useful result in the guidance of data mixtures in pre-training, especially in scarce data domains.
  2. The experiments are controlled, substantial and well designed and the accompanying statistical analysis offers a clean interpretation of observed test risk.
Critical weaknesses
  1. The peak location scaling law fits are not definitive, it uses single seeds and may be sensitive to the shape of the peak enforced (log Gaussian). Predicting the worst repetition structure from model size is very valuable and needs to be supported by robust estimates.
  2. As acknowledged, experiment grid is incomplete, the largest run at 344 M is only populated at OT=1.
  3. The training recipe is not completely specified, dropout is not specified and the exact learning rate adjustment is under specified.
Questions for authors

Clarifications:

  1. Please clarify the exact learning rate formula used, maybe a table with actual learning rate and effective batch sizes
  2. Per appendix I.4, what does the disagreement between R=193 and R=2682 mean for the confidence in the peak-location scaling ? Maybe controlled replicated at a few model sizes would help answer this question
  3. Relatedly, what happens if peaks are not log-Gaussian, are the scaling exponents still stable?
  4. Appendix C mentions the gap in the realized vs nominal budgets. Does correcting it change peak locations?

Future Work:

  1. Where does regularization fit in? Weight decay is 0.01 and dropout is not specified. There is some evidence for regularization helping mitigate degradation of repeated tokens. How does this change the peak location and scaling?
  2. What does extending it to multiple repeated token fraction(s), f look like?
3 comments

Thank you for the careful review and for recognizing the value of quantifying repetition costs, the controlled experimental design, and the statistical analysis. We address your questions about robustness, experimental coverage, and the training recipe below.

1. Peak-location uncertainty and the seed replication

Our additional experiments support the qualitative peak and recovery, while leaving uncertainty in the exact peak location.

In the 93M replication, we changed the training seed, data shuffle, and train/test split together. Both settings show an intermediate maximum followed by lower loss at repeat count 10000. Their sampled maxima occur at 193 and 2682, with loss increases of 2.74% and 1.39%.

Limited compute prevented us from evaluating as many intermediate repeat counts for the second seed, so we may have missed a higher peak between its sampled points. We therefore cannot interpret the difference between the sampled maxima as the exact shift in the true peak, but more a reflection of our compute constraints and inability to sweep the full latter.

We conducted additional sensitivity analyses:

  • Resampling the fitted sweep cells gives a model-size exponent of −0.872, with a conditional bootstrap interval of approximately [−0.962, −0.773].
  • Removing one fitted cell at a time gives exponent estimates between −0.900 and −0.849.
  • Three no-repetition runs at each model size show stable reference losses. At 344M, the sample standard deviation is approximately 0.00057 loss units.

These checks answer different questions. The baseline replications measure variability in the clean reference; the resampling analyses measure sensitivity to the fitted cells. We will distinguish these uncertainties and describe the fitted relationship as a trend within the measured runs.

2. Dependence on the log-Gaussian peak shape

The log-Gaussian fit captures the peak already visible in the measured losses; it does not create the non-monotonic pattern. The fitted exponent remains between −0.900 and −0.849 when we remove individual sweep cells, supporting the consistency of the trend across the experimental grid. We believe that this effectively tests sensitivity to grid composition.

3. Experimental grid coverage

Our existing experiments cover 25 model-size/overtraining combinations and have already cost upwards of $40,000 in compute. Our academic compute budget limited coverage at the largest sizes: 153M includes four overtraining multipliers through OT=2, while 344M includes OT=1. The smaller models cover all five multipliers. This provides evidence across training durations and a larger-scale measurement at 344M.

[Continued in the next comment.]

4. Learning rate, effective batch size, and dropout

Thank you for suggesting that we include the exact learning-rate formula and effective batch sizes. We provide both below and would be happy to add them to the paper as well.

We compute the peak learning rate as follows (which leverages Shuai et, al -- Scaling Law for Language Models Training Considering Batch Size)

target_training_tokens = int(20 × OT × N)
target_tokens_per_step = int(3240 × target_training_tokens^0.264)
peak_learning_rate = 10^-6 × sqrt(target_tokens_per_step)

N is the exact total parameter count, OT is the overtraining multiplier, and int truncates to an integer. We use exact parameter counts rather than rounded model-size labels.

At sequence length 2048, we compute gradient accumulation and effective batch size as follows:

accumulation_steps = ceil(
    target_tokens_per_step / (GPUs × sequences_per_device × 2048)
)
effective_sequences_per_step = GPUs × sequences_per_device × accumulation_steps
effective_tokens_per_step = effective_sequences_per_step × 2048

The learning rate uses the target tokens per step before accumulation rounding.

The table reports recorded values across the main experimental grid. Multiple batch sizes represent actual differences between runs and appear in corresponding order.

ModelOTPeak learning rateEffective sequences/stepEffective tokens/step
34M0.250.0006946978256524,288
34M0.50.0007612575288589,824
34M10.0008341954352720,896
34M20.0009141209412 / 416 / 432843,776 / 851,968 / 884,736
34M40.001001705504 / 5121,032,192 / 1,048,576
48M0.250.0007268851288589,824
48M0.50.0007965293320655,360
48M10.0008728465384786,432
48M20.0009564753448917,504
48M40.0010481175441,114,112
63M0.250.0007532410288589,824
63M0.50.0008254108352720,896
63M10.0009044949416851,968
63M20.0009911559480983,040
63M40.0010861206081,245,184
93M0.250.0007932660320655,360
93M0.50.0008692704384786,432
93M10.0009525566448917,504
93M20.0010438235441,114,112
93M40.0011438336401,310,720
153M0.250.0008471434352720,896
153M0.50.0009283098432 / 448884,736 / 917,504
153M10.0010172535121,048,576
153M20.001114718608 / 6401,245,184 / 1,310,720
344M10.001131978640 / 7041,310,720 / 1,441,792

These counts describe full optimizer steps at sequence length 2048, rather than independently measured counts for any final partial step.

We use linear warmup over 20% of optimizer steps followed by cosine decay. We use no dropout. The optimizer is AdamW with beta1=0.9, beta2=0.95, weight decay=0.01, and gradient clipping at 1.0.

[continued in next comment]

5. Realized versus nominal token budgets.

The difference comes from selecting whole documents: a document can push the repeated pool above its target, and repetition amplifies that excess. Correcting the compute accounting changes CEG/CEL, but does not change the measured losses or their sampled maxima. Our peak-location fits use nominal budgets; determining whether unequal realized budgets affected the fitted trend requires checking the tokens actually processed.

6. Regularization and the repetition peak

We use weight decay 0.01 and no dropout, consistent with the Qwen3 architecture we study, whose released configuration also sets attention dropout to zero. We keep regularization fixed across sweeps to isolate repetition structure.

7. Extending to other repeated-token fractions

We ran additional sweeps at a 5% repeated-token fraction and OT=1 for 34M, 63M, 93M, and 153M models. The sampled loss increases are approximately:

  • 34M: 1.32% above the no-repetition baseline.
  • 63M: 1.45% above the lowest measured repeat count, 13. This sweep lacks a completed no-repetition baseline.
  • 93M: 1.27% above the no-repetition baseline.
  • 153M: 0.11% above the no-repetition baseline, giving a nearly flat curve.

These results broaden the evidence beyond the main 10% setting. Small percentage changes in loss can still represent meaningful compute costs: our main 344M experiment illustrates how a few-percent loss increase corresponds to approximately 33% compute-equivalent loss under the fitted reference.

Thank you again for the thoughtful suggestions. We hope these additional details and analyses help address your concerns, and we would appreciate your reconsidering the rating if our work addresses your concerns.

Reviewer_qGD8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the cost of repeated data in pretraining, in the spirit of Hernandez et al. (2022). The authors train Qwen3-style models on FineWeb-Edu-Dedup under Chinchilla-style budgets. In each run, 10% of the training tokens come from a pool of documents that is repeated R times, and the rest are unique. Varying R at fixed compute, they find that held-out loss peaks at an intermediate R, and that the peak moves to smaller R (i.e. a larger repeated pool) as model size grows, following a fitted power law. Finally, they show that a misspecified linear regression with duplicated samples reproduces a similar loss peak, and offer this as a simple statistical explanation.

Critical strengths
  • This paper studies an important question.
  • This paper conducts an in-depth study on the effects of data duplication in LLMs.
Critical weaknesses

Signifance of the paper

  • Imporvement over Hernandez (2022) seems incremental. I understand that the authors claim their work “predated the widespeard use of compute-optimal scaling”, but it seems that this paper follows Hernandez spiritually, which is follow a scaling law and vary the repeat count varied. Also, the author did not compare the findings of this paper and Hernandez (2022), and it is clear which are new findings, and which are conflicting with previous findings.

  • Application / implications of the findings. The main setting has 10% of the training data duplicated, which seems unrealistic for modern pretraining. As the authors themselves point out, people now go to great lengths to deduplicate their corpora. The paper’s own f = 0.05 results also suggest the effect fades quickly with less duplication; for the 153M model the curve is almost flat. The authors say the findings let “practitioners quantify the wasted compute” caused by duplicates in pretraining corpora. However, I did not see this applied to any real corpus. Could the authors show a real case? For example, Pythia was trained on both the original and the deduplicated Pile. It would be convincing if the authors could use their method to predict how much deduplicating the Pile should help, and then check that prediction against the actual Pythia models.

Evaluation

  • Possible overlap between the evaluation split and the repeated pool. The paper states that the held-out split is drawn before the repeated pools are built, so no evaluation document appears in training, however duplication could happen at a finer-grained level, e.g.sentence-level. This may bias results — Could the authors report the n-gram or MinHash overlap between the evaluation split and the repeated pools, or show that results hold after removing the overlapping eval documents?
  • What is suprising to me is that CEG recovers when R is large. This deserves an explanation, and the paper currently gives none for the language-model runs.
  • Robustness of the ``peak location''. Most sweeps are single runs. The one replication (in Appendix I.4) found its peak at R = 193 with one seed and R = 2682 with another, with loss increases of 2.74% and 1.39%. For the second seed, the loss barely moves until R = 2682. Since the peak location can shift by more than a factor of ten within a single model size, I am not sure how much to trust the power law proposed in main text. More seeds, at least at a few sizes, would make this result much more convincing.

Presentation

The paper needs significantly improve on presentation.

  • I am concerned that the title fails to provide new information to LLM memorization commuity, and does not highlight the research of this paper — “duplication hurts LLM” is well known, and what this paper do is to quantify the effects.
  • The analysis of the simple statistical model feels unnecessary, since the main text is about LLMs. Alternatively, it could be placed right after Section 2, as a motivating example before the LLM experiments.
  • In the abstract, the "three ways" are not parallel. The first two describe where the damage peaks, while the third is experiment results. The linear-regression result then comes outside the list. Reorganizing this would make the contributions easier to follow.
  • n the introduction (line 45), the meaning of "cost" is not clear to the reader; it is only defined later, when CEG/CEL is introduced.
Questions for authors
  1. Which of your findings are new compared with Hernandez et al. (2022), which confirm theirs, and which disagree with them?

  2. Can the method be applied to a real corpus? For example, can it predict the gap between Pythia models trained on the original and on the deduplicated Pile?

  3. What is the n-gram or MinHash overlap between the evaluation split and the repeated pools? Do the results change if overlapping evaluation documents are removed?

  4. Why does CEG recover at large R? Could you show training loss on the repeated pool as a function of R, to check whether small pools are simply memorized early?

  5. Please demonstrate the stability of the fitted exponent of R_peak across seeds (and propobaly across different model sizes)?

I am happy to raise my rating if my concerns got solved.

2 comments

Thank you for taking the time to read and review our paper, and for your thoughtful comments. We’re happy to address your questions and thoughts below. The central result is that repetition structure can waste a substantial share of a training budget even when the resulting loss increase looks small. At 344M parameters, allocating 10% of training tokens to the most damaging repetition structure produces a loss that our fitted no-repetition reference reaches with approximately 33% less compute. Measuring that cost and how the worst repetition structure changes with scaleis the focus of our paper.

We make three contributions:

  • Quantifying wasted compute. We fit a no-repetition loss–compute relationship and use it to translate repetition damage into compute-equivalent cost. Raw loss changes do not directly reveal this cost: the same percentage increase can have different compute implications depending on the reference curve and training scale.
  • Characterizing repetition under controlled budget scaling. We scale training tokens with model size and sweep overtraining multipliers. Within these budgets, we identify the most damaging repetition structures and estimate how their locations change with scale. We report this relationship as a within-range empirical trend.
  • Deriving a statistical explanation. We derive training and test errors for misspecified linear regression with literal duplicates. The analysis and simulations show how duplication can produce a peak and recovery without attention or other transformer-specific mechanisms.

Our qualitative and quantitative finding confirms Hernandez. The fitted exponents concern different quantities and budget regimes, so their difference does not establish disagreement.

Relevance of 10% repetition and smaller fractions. The 10% setting is a controlled intervention, not an estimate of residual duplication in modern corpora. It matches a Hernandez setting and isolates repetition structure while holding the repeated-token allocation fixed. At 5%, sampled loss increases are approximately 1.32% at 34M and 1.27% at 93M, while the 153M curve is nearly flat. However, small percentage changes in loss can conceal substantial compute costs: their significance depends on how much additional compute is needed to achieve that improvement without repetition. Our main experiments illustrate this directly– a few-percent loss increase corresponds to roughly 33% compute-equivalent loss at 344M. Thus, a 1% loss increase should not automatically be interpreted as negligible.

Real corpus We have begun experiments to test whether repetition measurements on the Pile can predict the gap between Pythia models trained with and without deduplication. We will report the comparison when it is complete. We did run an additional 34M experiment on C4-en. Loss increases from 4.8899 without repetition to 4.9500 at R=2682, then decreases to 4.9322 at R=10000. We therefore observe the same peak and partial recovery on a second corpus. We also evaluated perplexity on benchmark text containing correct answers. Several domains show degradation. These evaluations measure how well the models predict benchmark text; they do not measure task accuracy. We will include these results.

Evaluation overlap We agree that document-level separation does not rule out shared passages or near-duplicates. We are checking overlap between the evaluation split and repeated pools and will test whether excluding overlapping evaluation documents changes the results. We will report the findings once the analysis is complete.

[continued below in another comment]

AuthorsOpenReview ↗

response continued

[continued from above]

Recovery at large R.

Thank you for raising this. We hope the following helps address your question, and we’re happy to add additional clarification in the paper.

We hold the total number of repeated training tokens fixed. Increasing the repeat count therefore means repeating a smaller set of documents more often. A plausible explanation is that the model can memorize a very small pool while retaining its ability to learn from the unique data. An intermediate-sized pool may create more competition: the model sees it often enough to encourage memorization, but memorizing it may interfere more with generalization. Hernandez also discusses this hypothesis. Our statistical model shows how this tradeoff can produce a peak and recovery without attention or other transformer-specific mechanisms. Section 4.4 derives training and test errors for linear regression with duplicated examples. Section 4.5, particularly Figure 7, shows efficiency declining at intermediate repeat counts and recovering at larger counts. This recovery depends on shrinking the repeated pool as the repeat count increases; simply repeating the same fixed pool more often does not produce recovery in the stated formula.

Stability Thank you for highlighting this distinction. Our additional experiments provide evidence for the qualitative peak and recovery. In the 93M replication, we changed the training seed, data shuffle, and train/test split together. Both settings show an intermediate maximum followed by lower loss at repeat count 10000. The sampled maxima occur at repeat counts 193 and 2682, with loss increases of 2.74% and 1.39%. Limited compute prevented us from evaluating as many intermediate repeat counts for the second seed, so we may have missed a higher peak between the sampled points, which is where this instability in this additional analysis roots from.

Presentation Thanks for your suggestions about the presentation -- we will reorganize the abstract around the distinct contributions, and define “cost” better. We will also clarify the statistical model’s purpose and consider introducing it earlier as motivation.

Thanks so much for your thoughtful review, and we hope that if this information addresses your concerns, that you will consider raising our score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.