acceptodds
Under review as a conference paper at ICLR 2027

What Is a Repeated Token Worth? Beyond Token Counts in Language Model Pretraining

Abstract

Data reuse in language-model pretraining is usually summarized by token counts: processed tokens, unique tokens, passes and tokens per parameter. We ask whether these counts determine what a repeated token is worth, and find that they do not, using single-seed sweeps over model sizes up to 2B parameters and pre-registered multi-seed paired studies on three data sources. First, the value of a repeat depends on the comparison: relative to stopping after one pass, a second pass is worth most of a fresh one (an estimated , and at least within measurement resolution), yet at a fixed compute budget, replacing fresh tokens with repeats raises loss for every model in a five-size sweep and mean loss for every pre-registered allocation on all three sources. Second, equal counts do not give equal loss: with all counts matched, concentrating repeats on fewer samples raises loss slightly but consistently, and replaying each shard consecutively rather than in spaced passes raises loss at all three sizes tested, by up to bpb. Third, scaling depends on which count is held fixed: at fixed unique data, across eight model sizes, larger models reach their loss minimum after fewer passes (about 15 down to 4 over a 16-fold size range), whereas at matched tokens per parameter the larger of two models loses less to repetition. A count-based loss surface fitted to smaller models predicts larger ones where reuse helps, but below its fitted range of unique data it predicts a benefit from repetition where every measured case gets worse. Finally, in a 60-model study, re-tokenizing repeats with BPE-dropout lowers loss at high repetition but not at matched compute. Reuse claims should therefore report the comparison, the allocation and order of repeats, and the tokenization, not only aggregate counts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pretraining

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_pQzVOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how reusing data affects language-model pretraining and whether total token counts adequately describe the value of repetition. The authors distinguish between adding more passes over a fixed dataset and replacing fresh data with repeats under the same training budget. They examine how the distribution and order of repetitions, model size, and tokenization affect held-out language-modeling loss. Experiments cover multiple data sources and models up to 2B parameters. The experiments show that additional passes over the same data can reduce loss, although fresh data generally gives better results at the same training budget. Spreading repetitions across samples and over time also works better than concentrating them, even when the underlying data and total token counts remain unchanged.

Critical strengths
  1. The distinction between additional training and replacing fresh data is useful. These comparisons answer different practical questions about data reuse. Separating them helps explain why additional passes can improve a model even when fresh data would provide a better use of the same training budget.
  2. The comparisons with identical data and token counts help isolate the effects of repetition patterns. Changing which samples receive additional exposures, or when those exposures occur, provides direct evidence that aggregate token counts do not fully describe how data reuse affects training.
  3. The main comparison is supported by checks across runs and data sources. Matching the initialization across conditions helps reduce unrelated training variation. The consistent direction of the effect across sources gives some confidence that concentrating repetitions produces a small but reproducible increase in loss.
Critical weaknesses
  1. The paper lacks a sufficiently focused central argument. It moves between several questions about data reuse and measures such as fresh-token equivalence, loss penalties, and optimal repeat counts. Their connections are not explained well enough to show how the findings build toward a common conclusion. This makes the main contribution difficult to identify beyond the broad statement that token counts alone are insufficient.
  2. The presentation makes the experiments difficult to follow and assess. The reader must repeatedly check what is held fixed, how model sizes are counted, and whether results use the same evaluation setup. Important explanations are spread across the main text, figure captions, and appendices. This makes it difficult to understand individual comparisons and judge how strongly they support the conclusions.
  3. The additional insight beyond existing accounts of data reuse is limited. The comparison with identical data and token counts shows a small cost from concentrating repetitions, but Appendix F explains its direction and approximate size using a per-sample version of an existing diminishing-returns model. These measurements are useful, but the paper does not clearly establish what deeper understanding they add beyond confirming the expected effect of repeated exposure.
  4. The scaling analysis offers limited guidance for new training settings. The larger-model results helped determine the form of the fitted model, so they do not provide an independent test of its predictions. At smaller data budgets, the fit predicts that more repetition should help when the measured loss instead increases. Its usefulness for choosing how many passes to use in a new setting therefore remains unclear.
Questions for authors
  1. What is the paper’s central claim, and which results most directly support it? The paper covers several questions using different experimental setups and measures of repetition value, making it difficult to distinguish the main contribution from the additional observations. Could you clarify the most important finding, how the individual analyses support it, and which results can be compared directly?
  2. Does the explanation in Appendix F also account for the other allocation results? Appendix F shows that diminishing returns at the sample level can explain the direction and approximate size of the main concentration effect. Could you compare the same model’s predictions with other allocation results already reported in the paper? This would help clarify what these experiments teach us beyond that existing explanation.
  3. Can the scaling model predict a useful number of training passes in a new setting? The larger-model results helped choose the formula, while the smaller-budget tests also use a different experimental setup. Could you keep the current fit fixed and test one new data budget with a small model, using the original training and evaluation setup? A few measurements around the predicted best number of passes would help assess whether the model can guide repetition choices.
6 comments
AuthorsOpenReview ↗

Response to Reviewer pQzV (Part 1/2)

Thank you for recognizing the usefulness of separating additional training from fresh-data replacement, and the value of the matched-content comparisons and their consistency across sources. We agree that the submission did not connect its questions clearly enough.

Re: Weakness 1 / Q1.

The first claim is that conclusions about repetition depend on the comparison baseline and the resource constraint. Repeating data can improve on stopping after one pass, yet yield higher loss than fresh-data training at the same budget. Across model sizes, fixing unique data UU, unique tokens per parameter U/NU/N, or training tokens per parameter D/ND/N also answers different questions. The second claim is that matching these aggregate counts still leaves exposure structure unspecified.

The revised manuscript follows that argument:

  1. Section 3 defines the two baselines. Sections 4.1–4.3 then measure repeat value and cost over five sizes and 5–300 TPP, with the full budget range shown in revised Figure 4. At each model size and nominal training budget, full-corpus passes vary over R=1/2/4/8/16, with U=D/RU=D/R; measured one-epoch curves provide the references. The measured second-epoch equivalence is approximately 0.87–1.02.
  2. Sections 5.1–5.3 compare the three data constraints. The fixed-UU study keeps the unique corpus at 1B tokens while varying model size and passes; its measured loss minimum shifts to fewer passes as size increases. The fixed-U/NU/N study instead gives each size U=2.5NU=2.5N unique tokens and varies passes, with CommonCrawl fitted minima at 3.7–4.4 epochs across seven sizes. The fixed-D/ND/N study holds approximately 20 TPP and the repetition allocation constant while varying model size; heavy partial-repetition penalties fall with size in this five-size, three-replicate comparison. These results help reconcile differing size trends, without isolating model size from every recipe change.
  3. Section 6.2 tests what counts omit. The original concentration comparison changes frequency and timing together and produces a small loss difference. The order experiment preserves shard-level multiplicities while changing temporal arrangement, with much larger differences in the single-seed runs.

The new 45M CommonCrawl concentration control described in our response to 5jzj matches marginal adjacent replay-gap distributions and the global first/repeat event calendar. It yields a small positive difference of +0.00101 bpb (95% paired-tt CI [0.00050, 0.00152]; six paired training replicates), with full recency distributions and higher-order histories still different. This result is not yet in the PDF and does not constitute the independent pass-selection test requested below.

The fresh references also expose a limit of benefit-only models. Of 80 repeated configurations with a directly measured one-epoch reference at the same UU, 21 end above that reference. A model based on a monotone fresh-data curve with 0≤η≤10\le\eta\le1 and no degradation term cannot represent those endpoints. Most configurations have one seed, so this is an observed pattern, not evidence of significant degradation at every point.

Re: Weakness 2.

We have organized Sections 3–6 around the comparison baseline and the quantity held fixed, integrated short definitions into the prose, and simplified result descriptions and captions. Appendix A consolidates the experiment families and parameter conventions. These changes are intended to make each comparison easier to identify before interpreting its result.

Direct comparisons stay within each model, source, recipe and evaluation family. We compare directions across families without pooling absolute bpb values. Tables 1–4 in Appendix A distinguish exact non-embedding sizes, the total-parameter count used in zz, and the proxy labels of the legacy sweeps. Budgets in the newer study are nominally matched, with integer update rounding. The coordinate zz and sample-wise saturation build on prior work; our contribution is to measure how the conclusions change across conditions and where those descriptions fall short.

AuthorsOpenReview ↗

Response to Reviewer pQzV (Part 2/2)

Re: Weakness 3 / Q2.

We tested it on all 27 nonfresh source–allocation means from the original study, keeping the original parameters fixed. The underlying experiment fixes the 45M model and D=805,306,368D=805{,}306{,}368 targets and varies the repeated-budget fraction and subset multiplicity across nine allocations on CommonCrawl, GitHub and Books. These are f=1/8f=1/8 or 1/21/2 with r=2,8,32r=2,8,32; f=1f=1 with r=2,8r=2,8; and f=1/3,r=4f=1/3,r=4. Each allocation is compared with fresh training at the same model size and budget. The single-pair calculation is now in Appendix J; this wider check is reported here and has not yet been added to the PDF.

With ff the fraction of the training budget assigned to the repeated subset, rr its multiplicity and DD the total budget, the original benefit-only extension is

D_{\mathrm{eff}}=D(1-f)+\frac{fD}{r}g(r).$$ It predicts excess loss $B(D_{\mathrm{eff}}^{-\beta}-D^{-\beta})$, using $B=4450$, $\beta=0.485$ and $R_D=4.545$. We omit the full-corpus penalty, as in the original calculation, because its partial-repetition extension was not specified. | Saturation constant | Predicted concentration difference | MAE across 27 means | RMSE across 27 means | |---|---:|---:|---:| | $R_D=4.545$ | +0.00460 bpb | 0.02399 bpb | 0.04603 bpb | | $R_D=15.4$ sensitivity check | +0.00158 bpb | 0.03598 bpb | 0.06329 bpb | The original parameters still give the right order of magnitude for the +0.0039-bpb concentration contrast, but transfer less well across allocations. For CommonCrawl at $f=1/2$, $r=32$, the prediction is 0.0630 bpb and the measured penalty is 0.1607 bpb. This limits the quantitative reach of that frozen explanation. It does not invalidate all sample-wise models: the coefficients were estimated under a different protocol and metric, the full-corpus penalty is absent, and the 27 means share fresh baselines rather than constituting independent replications. > Weakness 4 / Q3 The requested test has not been completed. The larger-model results informed the choice of functional form, even though they were excluded from coefficient fitting; they are therefore not a fully independent prediction test. The smaller-budget check also changes protocol. Neither the new measurement study nor the 7B extension answers your specific request to freeze the original model and test a new $U$ under the original protocol. We retain the small-budget and 64-epoch failures in Appendices J and K. Section 4.4 limits the newer allocation analysis to a fitted description: its predicted advantage is at most 0.002 bpb, and proximity to $R_c$ is not a validated stopping rule. Testing pass selection would require separately annealed runs around the frozen prediction at a new $U$, including simpler epoch choices as baselines. We hope the clearer separation of comparisons and the frozen-model check address your comments on the argument and its supporting evidence. The independent pass-selection test remains open. We would appreciate a careful reassessment of the empirical contribution and consideration of a higher rating if the revisions make that contribution more convincing.

Dear Reviewer pQzV,

Thank you again for your time and thoughtful feedback. We have also revised the manuscript to improve clarity and address the issues raised. We hope these revisions, together with our responses and additional evidence, have helped address your concerns. We welcome any follow-up questions and would be happy to clarify further. We would appreciate your reconsideration of your assessment and rating in light of these updates.

Thank you for considering our work.

Reviewer_Ga9VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper systematically investigates the value of data repetition in language model pretraining, examining the effects of repetition frequency, data allocation, training order, model scale, and tokenization. The main finding is that token counts alone are insufficient to characterize the value of repeated training data.

Strengths

  1. Important research question. As high-quality pretraining data becomes increasingly limited, understanding the value of data repetition is important for future language model training.

  2. Comprehensive and carefully designed experiments. The paper covers different model sizes, data sources, and repetition strategies. The authors also use controlled comparisons, multiple random seeds, and statistical analyses to validate several key findings.

  3. Valuable empirical insights. In particular, the paper shows that different repetition allocations and training orders can lead to different model performance even when token counts are identical. These findings provide useful insights for modeling data reuse and designing pretraining strategies.

Weaknesses

  1. Some important findings lack sufficient validation. For example, the observation that repetition order can cause up to a 0.46 bpb difference in loss is one of the most interesting findings, but it is primarily based on single-seed experiments. In contrast, the more rigorously validated repetition-allocation experiments show a relatively small difference of approximately 0.0039 bpb, raising questions about its practical significance.

  2. Limited generalizability of the conclusions. Although the experiments cover models up to 2B parameters, the multi-seed controlled studies mainly focus on smaller models. Moreover, the fitted scaling law exhibits substantial prediction errors outside the evaluated range, suggesting that a reliable generalizable relationship has not yet been established.

  3. Lack of downstream evaluation. The paper primarily evaluates the effects of data repetition using held-out loss. It remains unclear whether the observed differences translate into meaningful improvements or degradations in downstream task performance.

Overall Assessment:

Overall, this is a solid empirical study. Although the paper does not propose a new training algorithm and some findings are consistent with existing intuitions, it provides a systematic investigation of important factors affecting data repetition, offering useful empirical evidence and insights for the research community.

Critical strengths

Important research question. As high-quality pretraining data becomes increasingly limited, understanding the value of data repetition is important for future language model training.

Critical weaknesses

None

Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ga9V

Thank you for recognizing the importance of the problem and the value of the controlled experiments. We appreciate your assessment of the paper as a solid empirical study.

Re: Weakness 1

The original three-source concentration contrast is approximately +0.0039 bpb, with a 95% interval of [0.0033, 0.0045] over six source-averaged paired training replicates. It is reproducibly positive, but the entire interval is below 0.005 bpb in magnitude. We interpret it as a small difference between two schedules with equal aggregate counts, not a large practical gain. Those schedules also differ in replay timing, so the result does not isolate frequency alone (Section 6.2; Appendix E, Figure 18d; Appendix B, Table 6).

We have also completed six paired training replicates of a CommonCrawl concentration control using the same 45M architecture. Each replicate compares a larger subset presented twice with a nested, smaller subset presented four times, under the same data/initialization seed configuration and matched model size, unique content and training budget. The control matches marginal adjacent replay-gap distributions and the global first/repeat event calendar. The new difference is +0.00101 bpb, with a 95% paired-tt interval of [0.00050, 0.00152]; all six differences are positive. This is a small effect under additional controls, with full recency distributions and higher-order histories still unmatched. These completed results are reported in our response to 5jzj and are not yet in the manuscript.

The order difference of up to 0.46 bpb is much larger, but remains supported by single-seed comparisons. The added per-epoch shard-shuffle arm differs from spaced replay by less than 0.006 bpb across five settings (Section 6.2, Figure 8). It clarifies which order changes matter in these runs; it is not a multi-seed replication of the consecutive-replay result.

Re: Weakness 2.

The revised Section 5.3 adds a study at approximately 20 TPP over five sizes, 45M–1.2B, two sources and three paired training replicates. Heavy-repetition penalties decrease by about one-third on CommonCrawl and one-fifth on GitHub, but remain positive (Figure 7; Appendix F, Table 7). The fixed-U/NU/N study covers seven sizes (Section 5.2), and the five-size study in Section 4 covers 5–300 TPP (revised Figure 4); these latter studies are predominantly single-seed.

We also report 15 completed CommonCrawl tests at 7.014B non-embedding parameters, covering 5/10/20 TPP and R=1/2/4/8/16. At 20 TPP, eight and sixteen epochs incur excess losses of 0.0444 and 0.3843 bpb. These results are not yet in the PDF. They extend the observed heavy full-corpus reuse penalty beyond 2B, but use one data/initialization seed configuration, with changing model proportions and recipes.

The old surface's failures at small data budgets and at 64 epochs remain in Appendices J and K. The newer fitted allocation predicts at most a 0.002-bpb advantage near its optimum, and the correspondence with RcR_c is empirical (Section 4.4). We therefore treat the fits as descriptions of the tested conditions, not independently validated rules for selecting passes in new settings.

Re: Weakness 3. Downstream evaluation

We have not established whether these loss differences translate into downstream capability changes. This applies to the small concentration contrast and to the larger order effects. Section 7 states this limit. The present evidence concerns held-out loss and the conditions needed to compare data-reuse results.

We hope the expanded paired study and the clearer account of effect size, replication and generality address your comments. We would appreciate your careful reconsideration of the paper and, if the added evidence warrants it, a higher rating.

Dear Reviewer Ga9V,

Thank you again for your insightful comments. We hope our responses have addressed your concerns and clarified the contribution and supporting evidence. We welcome any follow-up questions and would be glad to discuss them further. We would appreciate your reconsideration of your assessment and rating in light of our responses.

Thank you for considering our work.

Dear Reviewer Ga9V,

As the discussion period draws to a close on November 18, we would welcome any further feedback on our responses and manuscript revisions. Please let us know whether there are remaining concerns you would like us to address. We would be happy to discuss them.

Thank you for your time and consideration.

Reviewer_mrznOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This work scrutinizes aggregate token counts (total processed tokens, unique tokens, passes, and tokens per parameter) as sufficient measurements to report for model training. Through systematic experiments, the authors show that the way in which tokens are repeated (adding extra compute or not), the way repeats are allocated (e.g., concentrating repeats on only a subset of tokens), and the order of repeats (e.g., shuffled or sequential), all substantially change held-out loss. Further, model scale and stochastic re-tokenization between repeats can also affect the usefulness of these measurements. As a result, training runs that report the same counts for these measurements can lead to different losses.

Critical strengths
  1. The authors present a systematic and exhaustive study, showcasing how different design decisions during training can compound to make reported aggregate token counts insufficient in predicting held-out loss.
  2. The paired experiment setups provide compelling evidence for the authors’ claims and include confidence intervals where possible.
  3. The authors are transparent about negative results and failed runs.
Critical weaknesses
  1. The results for larger model size (e.g., Figs. 2, 4) rely on a single seed, and the largest models also double the width-to-depth ratio, if I understand correctly. That makes me question how reliable these results are. Especially in comparison between model scales (e.g., Fig. 5), it is unclear to me how much of the reported difference might be due to architecture changes and/or noise.
  2. The matched TPP, e.g., in Fig. 6, is vastly below Chinchilla scaling. I'm not sure how meaningful these results are.
  3. All evaluation seems to be performed on quantized models, making me wonder whether that plays a role.
  4. Large parts of the paper's result sections and figure captions are very terse and dense, to a point where they are hard to follow and understand and give the impression of being fully LLM written. I encourage the authors to conduct a serious editorial pass.
  5. If I understand correctly, BPE dropout in Sec. 7 also increases token counts and fragments documents, which might be a confounder.
Questions for authors
  1. 2B models are still relatively small by current standards. While I understand the scale limitations for a study like this, do you expect your results to hold up at larger scales?

Nits: 2. In Fig. 6b, what are the colors corresponding to?

6 comments
AuthorsOpenReview ↗

Response to Reviewer mrzn (Part 1/2)

Thank you for recognizing the systematic study, the paired comparisons with confidence intervals, and our reporting of negative results.

Re: Weakness 1

The original proxy-size family does change shape: several intermediate models have width-to-depth ratios near 50, while the largest approach 100. Its fixed-UU and order sweeps remain single-seed, so they do not establish a pure parameter-count effect.

The two scale studies added to the current PDF use a fixed width-to-depth ratio of 64 (Appendices A and F). The partial-repetition study covers five sizes from 45M to 1.2B non-embedding parameters, two sources and three paired training replicates (Section 5.3, Figure 7; Appendix F, Table 7). This adds controlled scale evidence. Batch size, update count and some recipe choices still vary with scale, and the order study has not gained additional seeds.

Re: Weakness 2

We added the approximately 20-TPP study to extend the original comparison at about 4.5 TPP. At each size, the comparison holds the model and nominal total training budget fixed and contrasts all-fresh training with two allocations: half the training targets come from a subset presented twice or eight times, and the rest are presented once. With eight presentations of the repeated subset, the excess loss falls from approximately 0.032 to 0.022 bpb on CommonCrawl, and from 0.034 to 0.027 bpb on GitHub, between 45M and 1.2B. It remains positive. Twofold repetition incurs a much smaller penalty.

A paired comparison of the heavy-repetition penalties at the largest and smallest sizes, averaged across the two sources, gives a difference of −0.00881 bpb, with a pointwise 95% paired-tt interval of [−0.01049, −0.00713]. The three paired training replicates are the independent units. The corresponding interval for light repetition crosses zero. This cross-size summary is reported here; the per-source results are already in Appendix F, Table 7.

The five-size study in Section 4 also covers budgets from 5 to 300 TPP, as shown in revised Figure 4. TPP uses non-embedding parameters in these studies, so 20 TPP should not be read as a verified compute optimum for every architecture.

Re: Weakness 3

The paired allocation/concentration studies, CommonCrawl order comparisons and newer scale studies, including the 7B extension, were evaluated without weight quantization. Thank you for raising this concern. All results reported in the revised manuscript are now evaluated without post-training weight quantization.

Re: Weakness 4

We have simplified the captions and Related Work, removed repeated discussion, and organized Sections 3–6 around the comparison baseline and the quantity held fixed. These changes make the experimental comparisons and the role of the fits easier to follow.

Re: Weakness 5.

The tokenization comparison holds the 45M model, source bytes and their order fixed. At R=1/2/4/8/16, it compares deterministic BPE throughout, BPE-dropout throughout, and BPE-dropout switched off for the final 20% or 50% of updates. The dropout rate is 0.05, with three paired training replicates. BPE-dropout changes target counts and context segmentation even under these byte-matched conditions. Valid prediction targets increase by up to approximately 5% and padded positions by up to approximately twofold (Appendix L, Table 12).

The supported result is a heavy-reuse benefit under this byte-matched protocol (Section 6.3, Figure 10). It is not evidence of improved compute efficiency or an isolated segmentation mechanism. Section 7 states the compute limitation; no compute-matched BPE comparison has been completed.

AuthorsOpenReview ↗

Response to Reviewer mrzn (Part 2/2)

Re: Q1. Evidence beyond 2B

Yes, completed full-corpus repetition experiments reach 7.014B non-embedding parameters, or 7.219B total parameters. On CommonCrawl, they cover nominal budgets of 5, 10 and 20 TPP, each at R=1, 2, 4, 8 and 16: 15 completed training runs with final test evaluations. These experiments are reported here and are not yet in the current PDF.

Within each 7B comparison, model size and nominal total training tokens per parameter are fixed. Changing the number of full-corpus passes changes the unique-data budget according to U=D/RU=D/R. The one-pass condition uses fresh data throughout and is the reference. At approximately 20 TPP:

Epochs RTest loss, bpbExcess over fresh training at the same nominal budget
10.70180.0000
20.7033+0.0015
40.7084+0.0066
80.7462+0.0444
161.0861+0.3843

The heavy-reuse penalty therefore persists in these 7B settings. The two-epoch difference is close to the operational resolution threshold; the eight- and sixteen-epoch differences are much larger. Completed results also exist at approximately 2.10B and 3.74B.

The 7B study uses one data-seed/initialization-seed configuration. Its width is 4096 and depth is 38, so it does not retain the ratio of 64 used in the smaller scale studies. These runs extend the observed full-corpus replacement penalty; they do not validate every fitted constant, the order and concentration interventions, or downstream transfer at 7B. Model shape and recipe changes also limit causal comparisons across sizes.

Re: Q2.

In the archived source of the two-size comparison, light coral circles denote 45.1M, dark red squares denote 184.6M, and red open diamonds mark the development-selected epoch counts. The size legend was shared with panel c. This comparison is now Appendix F, Figure 19, with the size and selected-epoch encodings labeled. The new main-text scale studies are Figures 6 and 7.

We hope these revisions and the additional scale evidence address your comments. We would be grateful for a careful reassessment of the paper and consideration of a higher rating if these additions strengthen your evaluation.

Dear Reviewer mrzn,

Thank you again for your careful review and constructive comments. We hope the clarifications and additional evidence in our responses have addressed your concerns. Please feel free to raise any follow-up questions. We would appreciate your reconsideration of your assessment and rating in light of these responses.

Thank you for your time and consideration.

Reviewer_5jzjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how repeated data affects language-model pretraining beyond aggregating token counts. It examines the comparison baseline, allocation and order of repetitions, model scale, and stochastic tokenization. Sweeps up to approximately 2B parameters and paired multi-seed studies show that identical aggregate counts can produce different held-out losses.

Critical strengths

Useful evidence that exposure structure matters. The matched-content comparison (Section 5.1, Fig. 3d) and order experiment (Section 5.2, Fig. 4) demonstrate that aggregate token counts do not fully describe data reuse. This is useful when comparing pretraining recipes.

Critical weaknesses
  1. Replay spacing is not sufficiently quantified. Section 5.2 demonstrates an order effect, but the shard sizes and per-example replay-gap distributions are not clearly reported, particularly for the matched-content comparison in Section 5.1. For fixed-order full passes, the gap is already determined by the corpus size UU; the unresolved issue is how partial-repetition schedules change spacing and recency. Reporting these gaps in both processed tokens and optimizer updates would clarify the comparisons. Earlier memorization-window work, already cited in Section 2, motivates examining these quantities. The results establish that counts alone are insufficient, but provide limited guidance on how replay timing determines the value of repetition.

  2. The additional insight and practical guidance remain limited. Several conclusions are expected from existing work on data reuse (Section 2). The controlled concentration effect is consistent but small: approximately 0.00390.0039 bpb (Section 5.1), and Appendix F shows that per-sample diminishing returns account for its sign and approximate magnitude. The larger order effects rely on single-seed sweeps, while the tokenization gains are not compute-matched (Section 7). The fitted surface in Eq. (2) also provides limited predictive guidance outside its fitted range. The paper therefore needs to establish more clearly what substantial new understanding follows from these measurements.

Questions for authors
  1. Clarification: What are the shard sizes, shuffling rules, and gaps between repeated examples, in both processed tokens and optimizer updates? Which spacing quantities remain fixed across the allocation and scale comparisons?

  2. Targeted validation: Does the concentration effect in Section 5.1 persist when replay gaps are matched? A representative comparison would help distinguish the exposure-count distribution from the timing of those exposures.

  3. Clarification: Which result constitutes the main advance beyond existing repetition studies and the per-sample effective-data explanation considered in Appendix F?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5jzj (Part 1/2)

Thank you for recognizing the value of the matched-content and order comparisons. We have completed the requested representative control: a small positive concentration penalty remains under the timing constraints specified below.

Re: Weakness 1 / Q1

The original order study uses ten shards of approximately 100M tokens each, totaling U=1U=1B. Within-shard order is fixed. Spaced replay visits each shard once per pass; consecutive replay repeats a shard before moving to the next. Their nominal start-to-start revisit distances are therefore approximately 1B and 100M tokens. At a batch of 524,288 targets, these correspond to about 1,907 and 191 updates; at 1,048,576 targets, about 954 and 95 updates. Loader boundaries, context-only tokens and rounding affect the exact distances.

The original global shuffle permutes the full multiset of repeated shards. The added per-epoch shuffle permutes shards independently within each pass. Neither shuffles individual tokens. At fixed UU, full passes preserve nominal token distances, but changing batch size changes update distances. The fixed-U/NU/N and fixed-D/ND/N scale studies do not match replay gaps across sizes.

The concentration study uses fixed-context samples rather than shard-level replay. The original comparison globally shuffles the multiset of sample presentations, without constraining replay gaps. In the original CommonCrawl schedules, repeating a larger subset twice and a smaller subset four times gives mean adjacent gaps of 268.43M and 161.14M prediction targets, or 2047.97 and 1229.43 updates. Pooling adjacent replay intervals across the six CommonCrawl replicates, the 10th percentile, median and 90th percentile are 41.31M, 235.98M and 550.77M targets (315, 1800 and 4202 updates) for the larger subset presented twice, and 20.91M, 128.21M and 352.62M targets (160, 978 and 2690 updates) for the smaller subset presented four times. These are descriptive interval statistics, not additional independent training observations. The original schedules were not gap-matched, so describing them as differing only in repeat frequency was too strong.

The new control matches the entire empirical marginal distribution of adjacent same-sample start-to-start gaps: 134,217,728, 201,326,592 and 268,435,456 prediction targets, each with probability one-third. At 131,072 targets per update, these are exactly 1,024, 1,536 and 2,048 updates. Each sample's first presentation and the complete global first-versus-repeat event calendar also match, so extra exposures occur at the same learning-rate phases. Mean terminal recency, the distance from a sample's last presentation to the end of training, also matches when averaged over all unique samples. Its full distribution, recency among repeated samples alone, and the assignment of gaps to particular repeat events remain different.

Re: Q2.

Yes, in the tested CommonCrawl setting. Using the same 45M non-embedding-parameter model, we compared a larger subset presented twice with a nested, smaller subset presented four times, holding unique content, total training budget and the optimization recipe fixed. Across six paired training replicates, the latter increased final test loss by 0.00101 bpb (95% paired-tt CI [0.00050, 0.00152]). All six differences were positive.

The comparison matches the marginal adjacent-gap distribution and global first/repeat calendar described in Q1. The effect is small, and full recency distributions and higher-order histories remain unmatched. It therefore establishes persistence under these controls, not a timing-independent frequency effect.

AuthorsOpenReview ↗

Response to Reviewer 5jzj (Part 2/2)

Re: Weakness 2 / Q3

We agree that sample-wise diminishing returns can explain the direction of the original concentration contrast. The additional contribution is to quantify how repetition behaves across budgets and comparison conditions, and to test the reach of that explanation. We distinguish the benefit of continuing on fixed data from the cost of replacing fresh data at a fixed training budget.

  1. Quantifying repeat value across training budgets. The expanded CommonCrawl study covers five model sizes from 23M to 361M and training budgets from 5 to 300 TPP, using measured one-epoch curves as references. Revised Figure 4 shows this full budget range together with the average half-value crossings. Second-epoch fresh-token equivalence is approximately 0.87–1.02, while heavier reuse loses value faster at low TPP. The crossing at which repeated tokens are worth half a fresh token on average moves from about four to five epochs at budgets up to 20 TPP to about ten at 300 TPP (Sections 4.1–4.3). This quantifies the budget dependence of reuse; the crossing is not a validated stopping rule. These measurements are predominantly single-seed.

  2. Distinguishing fixed-total-token from fixed-TPP size trends. For the same heavy partial-repetition allocation, with half the budget drawn from a subset presented eight times, the historical CommonCrawl sweeps at a fixed nominal D=8D=8B show a penalty that increases with size across the 252M, 1B and 2B proxy models (Appendix H). By contrast, at fixed 20 TPP, the penalty decreases across the five-size, 45M–1.2B study with three paired training replicates on CommonCrawl and GitHub, while remaining positive (Section 5.3), extending the original approximately 4.5-TPP comparison. In both cases, the penalty is measured against fresh training of the same model at the same nominal budget. Fixed DD gives larger models fewer training tokens per parameter; fixed TPP increases DD with model size. The fixed-U/NU/N study separately finds CommonCrawl fitted minima at 3.7–4.4 epochs across seven sizes, unlike the decline at fixed UU (Sections 5.1–5.2). These observations clarify the resource conditions behind differing size trends. The fixed-DD sweeps are single-seed, and the studies differ in model family and protocol, so this is not a pure causal model-size comparison.

  3. Checking the quantitative reach of the existing explanation. We kept the original benefit-only sample-wise model fixed and applied it to all 27 existing nonfresh source–allocation means. It predicts the original concentration contrast at the right order of magnitude, but its MAE is 0.0240 and RMSE is 0.0460 bpb across the wider allocations. Agreement on one contrast therefore does not establish quantitative transfer to other allocations. This additional analysis is reported in our responses, not yet in the PDF. Its coefficients were transported across protocols and the means share fresh baselines, so the result limits this particular frozen explanation rather than rejecting diminishing returns generally.

  4. Testing concentration under tighter temporal controls. The new experiment in Q2 shows that the small concentration penalty persists when the marginal adjacent-gap distribution, per-sample first presentations and the global first/repeat calendar are matched. This answers a control question left open by the original experiment. Full recency distributions and higher-order histories still differ, and the result remains compatible with diminishing returns; it does not establish a timing-independent mechanism or a large practical gain.

Together, these studies provide a measured account of repeat value under explicit budget, scale and scheduling conditions, and identify where simple descriptions or transferred fits are insufficient. The practical contribution is to specify the references and controls needed to interpret a reuse result. The larger shard-order effects still require multi-seed replication, and the BPE study remains byte-matched rather than compute-matched. The new concentration control does not resolve either limitation.

We also agree that the fitted surface offers limited guidance outside its tested range. Its small-data and 64-epoch failures remain in Appendices J and K. The newer allocation analysis predicts an advantage of at most 0.002 bpb near its optimum (Section 4.4), and neither the fitted optimum nor the average half-value crossing is an independently validated pass-selection rule. We therefore present these fits as conditional empirical descriptions, while the requested original-protocol test at a new data budget remains incomplete.

We hope the expanded measurements and targeted control address your comments. We would appreciate a careful reassessment of the contribution and consideration of whether these additions warrant a higher rating.

Dear Reviewer 5jzj,

Thank you again for your thoughtful feedback and suggestions. We have revised the manuscript and provided additional analyses and a targeted experiment in our responses. We hope these updates have addressed your concerns and clarified the paper’s contribution. We welcome any follow-up questions and would be glad to discuss remaining concerns. We would appreciate your reconsideration of your assessment and rating in light of these updates.

Thank you for your time and consideration.

Dear Reviewer 5jzj,

With the discussion period closing on November 18, we would appreciate any feedback on our responses and manuscript revisions. Please let us know if any concerns remain or if further clarification would be helpful. We would be happy to continue the discussion.

Thank you for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response to Reviewers

Thank you for the careful reviews. We appreciate 5jzj's recognition of the exposure-structure comparisons, mrzn's acknowledgment of the paired evidence and transparent reporting, Ga9V's emphasis on the importance of data reuse, and pQzV's recognition of the baseline distinction and consistency across sources.

The reviews raise three common concerns: the focus of the central argument, the strength and scope of the experimental evidence, and the additional understanding beyond existing accounts of repetition. We address them as follows.

Central contribution and clarity. We have organized the manuscript around the comparison baseline and the quantity held fixed. Sections 3–5 distinguish the value of continuing on a fixed corpus from the cost of replacing fresh data, and examine how size trends change under different data constraints. In particular, fixed total training tokens and fixed tokens per parameter are different scale comparisons and can show different repetition-penalty trends. Section 6 then tests what aggregate counts leave unspecified. We have also simplified the captions and Related Work and removed repeated discussion, so that the experimental comparisons and their connection to the central claim are easier to follow.

Experimental support and generality. To address concerns about low training budgets and limited scale coverage, the revision extends the original approximately 4.5-TPP paired comparison to 20 TPP and adds measurements across 5–300 TPP, explicitly shown in revised Figure 4, which tracks repetition cost and the budget dependence of the average half-value crossing. It also expands the separate fixed-U/NU/N and fixed-D/ND/N comparisons across model sizes (Sections 4–5; Appendix F), and adds a per-epoch shard-shuffle control. Completed 7B experiments and the new concentration control with matched marginal replay gaps and first/repeat calendars are reported in the individual responses; these two extensions are not yet in the PDF. We specify which results have paired replication, which remain single-seed, and what architecture or protocol differences limit their interpretation.

Additional insight and practical meaning. We distinguish measured differences from their mechanistic explanations and predictive uses. In the replies, we assess the reach of the existing sample-wise explanation and retain the fitted surface's failures outside its fitted range. The new control is compatible with diminishing returns, and its small loss difference is not presented as a large practical gain. Our guidance concerns how to define and report a reuse comparison; the evidence does not establish a universal replay schedule, a validated stopping rule or downstream improvements.

The reviewer-specific replies address both the stated weaknesses and the questions, grouping overlapping concerns together and giving the supporting evidence and remaining limitations. Some completed supplementary results and related wording corrections have not yet been incorporated into the current PDF; we distinguish them from changes already present in the manuscript. We hope these revisions and clarifications address the shared concerns raised in the reviews. We would appreciate a careful reassessment of the paper and, if the revised evidence strengthens your assessment, consideration of a higher rating.

AuthorsOpenReview ↗

Request for engagement before the rebuttal deadline

Dear Reviewers and Area Chair,

Thank you for the feedback and reassessments during the discussion. We have responded to the original questions and the follow-up concerns about practical relevance and the scope of our contribution. We have also substantially revised the manuscript to sharpen the central argument, clarify the experimental comparisons and controls, and simplify the presentation and figure captions.

We would like to clarify a possible misunderstanding regarding the scope of our contribution and its relationship to prior work on data reuse. Our central question is how repetition value changes with the comparison baseline, training budget, and model-size constraint. The concentration experiment is one supporting comparison within this broader study. Its consistency with diminishing returns does not, by itself, settle the contribution of the other empirical findings.

We would be grateful if you could review our latest responses and follow-up clarifications alongside the revised manuscript. These explain how our findings build on the existing literature, including the 5–300 TPP measurements and 20-TPP paired scale study in the manuscript, as well as the additional experiments extending to 7B parameters at 5/10/20 TPP and targeted controls reported in our responses, together with their limitations. We hope the manuscript revisions and this broader evidence will be considered together in the overall assessment.

As the rebuttal deadline of November 18 approaches, we would appreciate any remaining feedback as soon as possible. If concerns remain after considering our responses, please indicate the specific unresolved points so that we have an opportunity to address them before the discussion closes. We remain available and would be glad to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.