- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that duplication is model-dependent, not an intrinsic dataset property. It shows that alignment of semantic variants emerges with capability and training steps, via per-document gradient cosines under seven semantics-preserving transformations. It also finds nearest-neighbor cosine gaps deviating from an isotropic null above ~10M documents in the 192M-doc FineWeb-Edu-Dedup corpus. The authors provide 34M–344M Chinchilla-optimal ladders on streams sampled with replacement from 10⁵–10⁸-document pools, finding penalties growing with compute.
Critical strengths
-
Definition 1 makes duplication θ-dependent, yielding a testable prediction that dedup thresholds should tighten with target scale.
-
Ablation analysis is substantive and well-executed: all 192M documents are embedded and the dimensionality ablation is obtained for free by slicing and renormalizing Matryoshka prefixes.
Critical weaknesses
-
The fitted scaling law appears to have no held-out test. Section 5.4 reports error "on the runs used to fit the law," so the 0.77% / 0.90% figures are in-sample. No value or uncertainty for is given anywhere, nor the fitting objective or the number of points entering the fit.
-
Only 34M–344M models are trained, which is small relative to the pretraining regime.
-
The theory names intervention points but no method is built. Given this scaling ladder and Proposition 1, the natural question is how a data pipeline should act differently: what dedup threshold, selection rule, or reweighting does.
Questions for authors
See Weaknesses
Rebuttal to Reviewer 3vUn
We thank the reviewer for raising this point. We agree that data repetition is not universally harmful, and that deliberately upsampling high-quality data can be beneficial. However, we respectfully disagree that the harmful effects of duplication are merely an assumption of our work. A substantial body of empirical literature has demonstrated that data duplication can degrade generalization, increase memorization, and reduce pretraining efficiency.
For example:
Importantly, these findings are not inconsistent with the reviewer's observation that frontier-model developers intentionally upsample high-quality data. Indeed, Fang et al. (2025) [6] show that repeating carefully filtered high-quality data can outperform training on a substantially larger, lower-quality dataset. This establishes that the optimal allocation of training tokens depends on both quality and redundancy, not that repetition is costless.
Our paper does not assume that all duplication is harmful. Rather, the cost of exact repetition is already well documented, and we directly measure such penalties in our controlled pretraining experiments. Our central contribution is to investigate how effective redundancy changes with model capability and corpus scale. In particular, we provide evidence that semantically equivalent but surface-distinct documents produce increasingly aligned training gradients as models become more capable, suggesting that conventional surface-level deduplication may underestimate effective redundancy at scale.
We agree that directly measuring the training penalties of semantic duplicates, as distinct from exact repeats, is an important direction for further validation. Nevertheless, the empirical evidence that excessive repetition can harm pretraining is extensive and extends to modern, multi-billion-parameter language models. We will clarify the distinction between beneficial intentional upsampling and harmful redundancy and incorporate these references into the manuscript.
References
[1] Lee et al. (2022). Deduplicating Training Data Makes Language Models Better. ACL 2022.
https://aclanthology.org/2022.acl-long.577/
[2] Tirumala et al. (2023). D4: Improving LLM Pretraining via Document De-Duplication and Diversification. NeurIPS 2023.
https://arxiv.org/abs/2308.12284
[3] He et al. (2024). SoftDedup: An Efficient Data Reweighting Method for Speeding Up Language Model Pre-training. ACL 2024.
https://aclanthology.org/2024.acl-long.220/
[4] Muennighoff et al. (2025). Scaling Data-Constrained Language Models. Journal of Machine Learning Research, 26(53). Originally presented at NeurIPS 2023.
https://jmlr.org/papers/v26/24-1000.html
[5] Chudnovsky et al. (2026). Internal Data Repetition Destroys Language Models. arXiv preprint.
https://arxiv.org/abs/2606.24998
[6] Fang et al. (2025). Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality. NeurIPS 2025.
https://proceedings.neurips.cc/paper_files/paper/2025/hash/2aba6ec20299931d46ddeefd5ddcb442-Abstract-Conference.html