- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks a simple question: if two models see the exact same documents the exact same number of times during pretraining, just in a different order, does that leave a permanent mark? Both models then continue training on identical data, and the authors check whether their predictions converge. Divergence is symmetric KL between next-token distributions on held-out contexts, normalized to 1 when continuation starts.
The headline result, at least for me: how you spend the continuation budget matters more than how big it is. At 124M, 15M tokens spread over broad document support closes divergence faster than 30M tokens drawn from a narrow subset. At a fixed 30M budget, going from 25% to 100% support gives a graded reduction, and at 25% divergence actually goes up. The pattern repeats on a document-disjoint FineWeb-Edu split, on C4, at 22M/124M/355M, and under the optimizer controls. The update-level experiment is the part I found most convincing: steps on previously seen batches push divergence up, steps on fresh batches push it down, and validation loss improves for both models either way.
Critical strengths
The question is interesting, and the experimental setup isolates it well. Holding the document occurrence counts constant while changing only their order is a good way to test something that would otherwise be difficult to separate from other training effects.
The 15M-versus-30M result is particularly useful. It makes a convincing case that continuation token count alone doesn't tell us enough about what the models are learning from. I think studies of continued pretraining would benefit from reporting document coverage and revisit counts alongside token budgets.
The results are reasonably consistent across settings. All 12 trajectories in the document-disjoint experiment (across seeds, strata, and metrics) show the same qualitative pattern. Seeing it again on C4 and at three model sizes makes the finding more convincing.
Critical weaknesses
The biggest weakness is that the paper never shows the divergence matters. Validation NLL is basically identical between the two models, and with no downstream evaluation, memorization test, or extraction probe, there's no way to tell whether the measured difference changes anything in practice. The persistence result is interesting, but it is effectively independent of other claims, so I don't know how much weight to give it. One experiment showing that training history affects behavior in a way would actually make someone care would greatly benefit the paper. Without one, I can't give this paper a high significance score.
The main sweep confounds two variables. Broader support means fewer revisits by construction, so the sweep can't separate their effects. The update-level experiment helps locate where the divergence shows up, but it doesn't touch the mechanism. The experiments also stop at 355M parameters, with just two runs at that size. From that, I can't say whether the effect holds at realistic pretraining scale or is mostly a small-model artifact.
Questions for authors
- Can you vary revisit frequency at fixed support, e.g., the same support set at several continuation budgets? This seems like the most direct way to separate the two.
- Is there any evidence the divergence affects downstream behavior (memorization or extraction of repeated documents, calibration, fine-tuning)?
- At 25% support, divergence increases. Does it keep increasing, or does it eventually turn around? Some of the 355M curves look like they flatten between 30M and 90M tokens.
- Do you have a hypothesis for the ~5× gap between C4 and FineWeb-Edu? Differences in within-corpus redundancy, perhaps?
Minor Suggestions
- Define the three data categories from §3.3 in one table.
- Split Figure 5, or give the main comparison its own panel.
- Report revisit counts per condition alongside token counts.
- Report the number of seeds in each figure caption, not only in the text.
3 comments
Thank you for your responses. The main concerns are mostly resolved, and while the gap does not merit a score change from my recommendation to accept, I believe the paper is going to be much stronger as a result.
Thank you for following up and for your constructive suggestions. The fixed-support revisit analysis and memorization probe you requested helped us clarify the findings, and we appreciate your assessment of the revised paper.
Thank you again for your constructive feedback. As the discussion period is approaching its end, we would be grateful if you could let us know whether our responses and additional experiments have addressed your main concerns, or if any points still require clarification. We would be happy to provide further clarification promptly.