acceptodds
Under review as a conference paper at ICLR 2027

Future Data Allocation Shapes Repetition-History Persistence in Language Model Pretraining

Abstract

High-quality pretraining data are finite, so repeated exposure is increasingly common. Prior work has mainly examined how repetition affects the final model. We instead ask whether predictive differences caused only by when repeated data appear persist under later training. We train matched model pairs on the same document-occurrence multiset using different repetition schedules, then continue both models on identical ordered batches. Under a fixed continuation budget, the remaining difference changes systematically with support–reuse allocation: concentrating tokens on repeated visits to narrow support can increase disagreement, whereas distributing them over broader support produces greater contraction. Token count alone does not explain this response. At 124M parameters, 15M tokens over broader support produce more contraction than 30M tokens of restricted-support reuse. A shared-first-update intervention localizes the same behavior: after an identical first update, every tested revisit increases disagreement, while matched distinct batches yield a larger one-step decrease. The allocation contrast replicates in a document-disjoint FineWeb-Edu construction and on C4, and remains positive across model scales and optimizer controls. These results show that the fate of a repetition-induced predictive difference depends on both the amount of continued pretraining and the allocation of its future data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model pretraining, data repetition, data reuse, data allocation, training dynamics, continued pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_1ZRMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks a simple question: if two models see the exact same documents the exact same number of times during pretraining, just in a different order, does that leave a permanent mark? Both models then continue training on identical data, and the authors check whether their predictions converge. Divergence is symmetric KL between next-token distributions on held-out contexts, normalized to 1 when continuation starts.

The headline result, at least for me: how you spend the continuation budget matters more than how big it is. At 124M, 15M tokens spread over broad document support closes divergence faster than 30M tokens drawn from a narrow subset. At a fixed 30M budget, going from 25% to 100% support gives a graded reduction, and at 25% divergence actually goes up. The pattern repeats on a document-disjoint FineWeb-Edu split, on C4, at 22M/124M/355M, and under the optimizer controls. The update-level experiment is the part I found most convincing: steps on previously seen batches push divergence up, steps on fresh batches push it down, and validation loss improves for both models either way.

Critical strengths

The question is interesting, and the experimental setup isolates it well. Holding the document occurrence counts constant while changing only their order is a good way to test something that would otherwise be difficult to separate from other training effects.

The 15M-versus-30M result is particularly useful. It makes a convincing case that continuation token count alone doesn't tell us enough about what the models are learning from. I think studies of continued pretraining would benefit from reporting document coverage and revisit counts alongside token budgets.

The results are reasonably consistent across settings. All 12 trajectories in the document-disjoint experiment (across seeds, strata, and metrics) show the same qualitative pattern. Seeing it again on C4 and at three model sizes makes the finding more convincing.

Critical weaknesses

The biggest weakness is that the paper never shows the divergence matters. Validation NLL is basically identical between the two models, and with no downstream evaluation, memorization test, or extraction probe, there's no way to tell whether the measured difference changes anything in practice. The persistence result is interesting, but it is effectively independent of other claims, so I don't know how much weight to give it. One experiment showing that training history affects behavior in a way would actually make someone care would greatly benefit the paper. Without one, I can't give this paper a high significance score.

The main sweep confounds two variables. Broader support means fewer revisits by construction, so the sweep can't separate their effects. The update-level experiment helps locate where the divergence shows up, but it doesn't touch the mechanism. The experiments also stop at 355M parameters, with just two runs at that size. From that, I can't say whether the effect holds at realistic pretraining scale or is mostly a small-model artifact.

Questions for authors
  • Can you vary revisit frequency at fixed support, e.g., the same support set at several continuation budgets? This seems like the most direct way to separate the two.
  • Is there any evidence the divergence affects downstream behavior (memorization or extraction of repeated documents, calibration, fine-tuning)?
  • At 25% support, divergence increases. Does it keep increasing, or does it eventually turn around? Some of the 355M curves look like they flatten between 30M and 90M tokens.
  • Do you have a hypothesis for the ~5× gap between C4 and FineWeb-Edu? Differences in within-corpus redundancy, perhaps?

Minor Suggestions

  • Define the three data categories from §3.3 in one table.
  • Split Figure 5, or give the main comparison its own panel.
  • Report revisit counts per condition alongside token counts.
  • Report the number of seeds in each figure caption, not only in the text.
3 comments

Thank you again for your constructive feedback. As the discussion period is approaching its end, we would be grateful if you could let us know whether our responses and additional experiments have addressed your main concerns, or if any points still require clarification. We would be happy to provide further clarification promptly.

Thank you for your responses. The main concerns are mostly resolved, and while the gap does not merit a score change from my recommendation to accept, I believe the paper is going to be much stronger as a result.

Thank you for following up and for your constructive suggestions. The fixed-support revisit analysis and memorization probe you requested helped us clarify the findings, and we appreciate your assessment of the revised paper.

Reviewer_4j88OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper trains pairs of GPT-2 style models (22M-355M) from the same initialization on the same documents, which differ only in whether a repeated subset (about 5% of tokens) appears early or late in training. The ask whether further training on the same data makes the two models agree on held-out context. Persistence is tracked as the symmetric KL between next-token distributions on held-out, normalized by value at onset C=1-D_t/D_0.

At a fixed token budget, continuing on many distinct documents shrinks the difference more, while continuing on fewer documents that are more repeated keeps or grows the difference. At 124M, 15M tokens of distinct documents shrink it more than 30M tokens of reuse (5.15% vs. 1.68%). Varying the share of distinct documents over 5 levels gives a monotone trend. The contrast holds for both FineWeb-Edu and C4 across a couple model scales.

Critical strengths
  • The design is clean. Same initialization, same document multiset, and continuation batches. The controls (AdamW, optimizer state, LRs) and the fact that every seed agrees in sign make me trust the direction of the effect.
  • The papers asks a novel analysis question about what later training does to a difference caused by the effect of repetition, which is more nuanced than just the repetition effect itself. The four-step experiment (same first batch then varies) is a nice way to see the long-run trend one update at a time.
Critical weaknesses
  • I'm not sure the effect is about repetition history. Moving the repeated windows also shifts the position of every other document, so the two models see different batches almost from the first step. Because a change in the data order makes 2 models diverge, would D0 mix this generic divergence with whatever the repetition timing adds? Figure 2 hints that the generic part is large (D0 is already at about 0.18 when the windows are 5 points apart, more than half of the 0.325 at 50 points apart).
  • It is hard to see why the disagreement matter. After 30M tokens, contraction is 1-10%, so over 90% of the difference survives under every allocation. The difference also does not show up in the NLL loss either (from Table 5)?
Questions for authors
  • In Figure 4, the distinct arm's one-step change is close to 0, while revisits increase the disagreement by about 0.009. So is the story mostly that reuse amplifies the diff, rather than broad data removes them? Also, reuse improves NLL less than broad data (Table 5), so part of the contrast could come from reuse simply teaching the models less. Plotting C against each branch's NLL improvement for every condition would may show whether broad vs. reused data gives different C at the same NLL improvement?
  • Continued pretraining usually rewarms the LR. Here it is constant and far below the peak. At 335M, the contrast doubles from LR 5e-6 vs. 1-5. Wonder how the contrast changes under such a schedule?
3 comments

Thank you again for your constructive feedback. As the discussion period is approaching its end, we would be grateful if you could let us know whether our responses and additional experiments have addressed your main concerns, or if any points still require clarification. We would be happy to provide further clarification promptly.

Thank you for your thorough response and for running the experiments I asked you. I am raising my score to borderline

Thank you for reviewing the additional results and reconsidering your assessment. Your questions about learning progress and learning-rate rewarming helped us strengthen the analysis and clarify the interpretation of the allocation response.

Reviewer_bv29OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether predictive differences created by early versus late repetition persist during subsequent pretraining. Matched models see the same document occurrences in different schedules, then receive identical continuation batches. Persistence is measured through normalized predictive disagreement, CH=1−DH/D0C_H=1-D_H/D_0. Experiments on FineWeb-Edu and C4, using 22M–355M models, show that broader continuation coverage generally contracts these differences more than restricted-support reuse.

Critical strengths

A controlled experimental design. : The matched-history construction preserves document identities, occurrence counts, initialization, and training settings while changing repetition timing. Common continuation batches and optimizer-state controls help isolate the measured response (§3.1\S 3.1, §4.5\S 4.5).

Consistent empirical evidence.: The allocation contrast appears across support levels, independent data constructions, two corpora, and multiple model sizes. Agreement between symmetric KL and top-1 disagreement strengthens the evidence beyond a single observable (Figs. 3–5\mathrm{Figs.}\ 3\text{--}5, Table 1\mathrm{Table}\ 1).

Critical weaknesses

The relevance of the measured disagreement is insufficiently established.: The observable in Eqs. (2)–(3)\mathrm{Eqs.}\ (2)\text{--}(3) measures whether two models become more similar, rather than whether either retains earlier knowledge or capabilities. Greater agreement is not inherently desirable: both models could forget an earlier capability while becoming more similar. Conversely, both can improve validation NLL while diverging (§4.4\S 4.4). The authors acknowledge this distinction, but do not establish what meaningful consequence follows from preserving or reducing the measured difference. Without that connection, or stronger explanatory insight, the study answers a narrow question with limited demonstrated relevance to training decisions.

The additional insight specific to repetition history remains unclear.: The experiments show that varied versus repeatedly reused continuation data changes the distance between predictors with different repetition histories. However, there is no comparison with model pairs differing through ordinary data shuffling or initialization, with comparable initial disagreement D0D_0. Such a control would clarify whether the result reveals something distinctive about repetition timing or a general property of training different predictors on the same future data. The shared-first-update experiment (Fig. 4\mathrm{Fig.}\ 4) supports the observed effect but does not resolve this question, limiting the demonstrated contribution.

Questions for authors
  1. Clarification: What concrete aspect of learned information does the persistent disagreement capture? Can the authors provide evidence connecting it to identifiable prediction changes, example retention, or another meaningful consequence beyond the disagreement statistic itself?
  2. Targeted validation: Does the same broad-versus-restricted continuation effect occur for model pairs differing only through ordinary data shuffling or initialization, with comparable initial disagreement D0D_0? This would clarify what additional insight comes specifically from repetition history.
2 comments

Thank you again for your constructive feedback. As the discussion period is approaching its end, we would be grateful if you could let us know whether our responses and additional experiments have addressed your main concerns, or if any points still require clarification. We would be happy to provide further clarification promptly.

AuthorsOpenReview ↗

Follow-up on the two central concerns

Before the discussion closes, we would appreciate your reassessment of the evidence addressing your two central concerns.

Behavior and persistence. In the FineWeb-Edu reconstruction, late-repetition models exactly reproduce a 32-token continuation from a 32-token prefix for 41.5–48.7% of repeated documents, versus 0% for early-repetition models, despite identical occurrence counts. After 30M held-out continuation tokens, exact reproduction is nearly gone under both allocations, but signed held-out prediction differences retain correlations of 0.936–0.958 with onset. Their symmetric-KL magnitude increases under 25% support and decreases under 100% support (three seeds; Appendices B.3–B.4). The behavioral probe identifies a consequence of the schedule; the pattern analysis identifies a predictive trace that survives after exact recall has nearly disappeared.

The requested alternative-history controls. On C4, ordinary reshuffling and different-initialization pairs have initial disagreements of approximately 0.324 and 0.249, bracketing the timing pair's 0.287. Their held-out allocation contrasts are 41.57 and 56.30 percentage points, compared with 49.37 for timing, with every seed positive (Appendix B.7). Thus the response extends across the tested sources of initial difference. Repetition timing supplies a controlled history with identifiable behavioral consequences; future allocation governs how the predictive difference evolves.

Would you reconsider your recommendation in light of these additions? We would also appreciate knowing which of the two concerns remains unresolved after considering the revised evidence.

Reviewer_Q4UbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors study how the timing of repeated training data in pretraining affects a language model that's obtained under further training. They find that further training doesn't necessarily erase the effect of the differences in earlier stage training (specifically, the timing at which it sees repeated data). More specifically, they find that further training on diverse data tends to reduce the differences, while repeating on a smaller subsampled set can amplify the differences. Contrary to most of the previous literature which studies how repeated data affects validation loss, memorization, and training efficiency, the authors measure the differences in the two models' next token predictive distributions (and not downstream performance).

Critical strengths

S1. The authors measure disagreement in the next token prediction distributions, rather than just things like validation loss or memorization as done in prior works. This allows for more a fine-grained look at the effect of repetition timings on model differences.

S2. The authors perform several experiments to disambiguate multiple potential causes for their reported phenomenon. For instance, they compare (in table 4) broader coverage with each restricted-support set separately, rather than just calculating their mean. Section 4.5 also test a shared AdamW second-moment denominator across the two continuations, and investigate keeping the endpoint optimizer and RNG states.

Critical weaknesses

W1. It's unclear to me what exactly is the motivation behind the paper's chosen scope, given how oddly narrow it is. First, the authors do not sufficiently explain what their observed predictive contraction / divergence tells us scientifically or practically. Lower disagreement between two models (in terms of predictive distributions) is not inherently a better outcome (practically speaking), and how it relates to consequential aspects of model behavior remains unclear to me. Second, the timing of repetition seems like a narrow choice of independent variable, given that there can be other axes like the amount of repetition, etc. It's unclear to me why the authors fixate on this setting. Perhaps this is related to prior works, and I simply do not have enough knowledge of the prior litertaure. In either case, the authors should clarify in their manuscript what their exact motivation is for studying this setting.

W2. Although the paper contains a lot of empirical analysis and control expeirments to make sure that the evidence supports their claim, they do not offer explanation of why restricted-support reuse during continuation amplifies the disagreement (or why broader coverage leads to contraction). The control experiments do not provide an account of why the effect occurs in the first place. Moreover, although the authors have a related-work section that surveys the prior works, they do not offer much interpretation of their findings in therms of the broader context of the field. It's thus unclear to me what scientific understanding (or practical lessons) can be gained from this study.

W3. The persistence ratio PHP_H or contraction CHC_H can only measure the magnitude of predictive disagreement. THey do not measure whether the differences from the onset keep their patterns throughout continuation. For instance, the temporal correlation of token wise log probability differences could serve as a measure of whether those patterns persist.

Questions for authors

Q1. What scientific or practical insights should we draw from the paper? e.g. Is there a practical setting where this would inform our training decisions?

Q2. Does the same allocatino effect for model pairs occur when we create model pairs simply by ordinary data reshuffling (instead of different repetitition timings), or other changes in training history? i.e. Is there evidence that suggests anything specific to repetition timing?

Q3. (This is related to Q1.) Can you provide a testable explanation for why restricted-support reuse amplifies disagreement while broader coverage leads to contraction ?

Q4. Using your saved checkpoints and the same validation contexts, can you examine the temporal correlation in the (signed) log probability differences? By "temporal", I mean throughout continuation. This could clarify whether the repetition history difference's patterns are retained or not.

1 comment

Thank you again for your constructive feedback. As the discussion period is approaching its end, we would be grateful if you could let us know whether our responses and additional experiments have addressed your main concerns, or if any points still require clarification. We would be happy to provide further clarification promptly.

Comments on the paper

3 comments

Dear Reviewers, thank you for your reviews. Please read the authors' rebuttal and engage with them to clarify any remaining concerns before the discussion ends on Nov 18 AoE.

Dear reviewers, as we approach the close of the author–reviewer discussion, please keep the November 18 (AoE) deadline in mind.

Q4Ub, thank you for following up. Please feel free to share any additional comments before the deadline.

bv29, you asked what predictive disagreement reveals about learned behavior and why it matters for training decisions. You also asked whether the effect is specific to repetition timing, compared with reshuffling or changing initialization at comparable initial disagreement.

4j88, you asked whether initial disagreement mixes the effects of repetition timing with more general data-order effects, and why it matters given modest contraction and similar validation NLL. You also asked how reuse, broader coverage, and NLL improvement relate to changes in disagreement, and whether the allocation contrast is sensitive to learning-rate rewarming.

1ZRM, you asked what disagreement reveals about behavior beyond validation NLL and noted that the main sweep couples support breadth with revisit frequency. You also questioned the mechanism behind the divergence in the update-level experiments and how much the two 355M runs establish about generalization to larger models.

Do the authors’ responses adequately address these concerns? If any remain, please let us know before the discussion deadline.

AuthorsOpenReview ↗

Closing clarification and evidence references

Thank you for coordinating the discussion. For ease of final assessment, we would like to highlight our clarification to Q4Ub and the locations of the requested controls.

What persists after continuation. In the same FineWeb-Edu reconstruction, after 30M tokens of continuation on held-out documents, exact reproduction is at most 0.11% under either allocation. Yet signed held-out prediction differences retain correlations of 0.936–0.958 with their onset pattern, while symmetric-KL contraction is −6.60% under 25% support and +4.18% under 100% support (three initialization seeds; Appendices B.3–B.4). Near-zero verbatim reproduction therefore does not establish that the earlier schedule's predictive effect has disappeared. The scientific finding concerns continued sensitivity to training history; it does not require treating greater agreement as better model quality.

Where the requested controls are addressed. Appendix B.7 reports the reshuffling and initialization comparisons, showing that the allocation response extends beyond repetition timing. Appendix B.2 provides the fixed-support revisit trajectories and the NLL-progress analysis, including the descriptive comparison at matched NLL progress obtained by interpolation. Appendix B.5 tests learning-rate rewarming, and Appendix B.6 reports the update-component and optimizer-reset interventions.

Together, these results characterize how future support–reuse allocation attenuates or amplifies an earlier schedule's predictive effect under common continuation. Matching continuation-token budgets alone does not ensure that subsequent training removes sensitivity to earlier schedules.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.