acceptodds
Under review as a conference paper at ICLR 2027

Scale Dependent Data Duplication

Abstract

Data duplication during pretraining can degrade generalization and lead to memorization, motivating aggressive deduplication pipelines. However, at web scale, semantically equivalent documents (e.g. translations) may induce redundant training signals once models become sufficiently capable. We present evidence that duplication is scale-dependent in two ways. First, as model capability increases, cross-entropy loss gradients for semantically equivalent documents become more aligned. Smaller models produce gradients that reflect surface similarity rather than semantic similarity. Second, we embedded all 192 million FineWeb-Edu-Dedup documents using EmbeddingGemma-300m. For moderate corpus sizes, nearest-neighbor cosine gap follows an isotropic power law baseline, but deviates sharply as corpus size grows to hundreds of billions of tokens, indicating accelerated semantic collisions. Finally, controlled pretraining on data sampled with replacement from finite pools of unique documents shows mild degradation for small models but rapidly increasing loss penalties for larger models, breaking naive scaling extrapolation. We derive scaling laws that account for limited semantic uniqueness, allowing for more accurate prediction at scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language model pretraining, data deduplication, semantic duplication, synthetic training data

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Xv9rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that duplication is model-dependent, not an intrinsic dataset property. It shows that alignment of semantic variants emerges with capability and training steps, via per-document gradient cosines under seven semantics-preserving transformations. It also finds nearest-neighbor cosine gaps deviating from an isotropic null above ~10M documents in the 192M-doc FineWeb-Edu-Dedup corpus. The authors provide 34M–344M Chinchilla-optimal ladders on streams sampled with replacement from 10⁵–10⁸-document pools, finding penalties growing with compute.

Critical strengths
  • Definition 1 makes duplication θ-dependent, yielding a testable prediction that dedup thresholds should tighten with target scale.

  • Ablation analysis is substantive and well-executed: all 192M documents are embedded and the dimensionality ablation is obtained for free by slicing and renormalizing Matryoshka prefixes.

Critical weaknesses
  • The fitted scaling law appears to have no held-out test. Section 5.4 reports error "on the runs used to fit the law," so the 0.77% / 0.90% figures are in-sample. No value or uncertainty for a,β,γa,\beta,\gamma is given anywhere, nor the fitting objective or the number of (C,K)(C,K) points entering the fit.

  • Only 34M–344M models are trained, which is small relative to the pretraining regime.

  • The theory names intervention points but no method is built. Given this scaling ladder and Proposition 1, the natural question is how a data pipeline should act differently: what dedup threshold, selection rule, or reweighting does.

Questions for authors

See Weaknesses

Reviewer_3vUnOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides an interesting analysis into effects of deduplication on language model pretraining. They show that semantically equivalent documents result in similar cross-entropy loss gradients sooner in training for large models. Moreover, they find that scaling corpus size results in semantic collisions. The authors find that any power law fitted to small corpus sizes (on the mean cosine gap) decreases when scaling to larger sizes. Lastly, the authors investigate practical implications of these two findings. They use exact document repeats as a proxy for semantic duplicates and fit a scaling law that models the effect of training on these proxy semantic duplicates.

Critical strengths

Novelty: I think the investigations of the authors into dependence of data deduplication on model and dataset scale is quite novel. I think that the paper's novelty is a critical strength, and lays the groundwork for more followup work into this question, potentially at larger scale.

Clarity: I think the paper was arranged in a very logical fashion and even the theoretical discussions were written in a clear and readable format.

Critical weaknesses

I think the main weakness is the authors' assumption that data duplication is bad for language model training. At current industry scale, model creators often retain duplicates or even upsample high quality data. I wonder if there is more nuance to the authors' assumption: perhaps deduplication was seen as essential in the early days of LM training because all the models were of smaller scale, thus gradients of semantic duplicate documents were misaligned?

Questions for authors

n/a

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer 3vUn

We thank the reviewer for raising this point. We agree that data repetition is not universally harmful, and that deliberately upsampling high-quality data can be beneficial. However, we respectfully disagree that the harmful effects of duplication are merely an assumption of our work. A substantial body of empirical literature has demonstrated that data duplication can degrade generalization, increase memorization, and reduce pretraining efficiency.

For example:

  • Lee et al. (2022) [1] demonstrate that deduplicating pretraining data reduces memorized generation by approximately 10× and enables models to achieve equivalent or better accuracy with fewer training steps.
  • Tirumala et al. (2023) [2] show that randomly repeating pretraining data degrades performance relative to baseline training, whereas intelligently selected repetition improves it, including at the 6.7B-parameter scale.
  • He et al. (2024) [3] demonstrate that downweighting redundant examples achieves comparable perplexity with at least 26% fewer training steps and improves few-shot accuracy, even on previously deduplicated datasets.
  • Muennighoff et al. (2025) [4] study repetition across approximately 400 training runs, with models up to 9B parameters. They find that while a few epochs of repetition incur negligible penalties, the marginal value of repeated tokens eventually approaches zero.
  • Chudnovsky et al. (2026) [5] provide further controlled evidence that exact data repetition can impose substantial compute-equivalent losses, including a setting in which allocating 10% of training FLOPs to repeated documents yields the same validation loss as a non-repeating run using only 67% of the compute.

Importantly, these findings are not inconsistent with the reviewer's observation that frontier-model developers intentionally upsample high-quality data. Indeed, Fang et al. (2025) [6] show that repeating carefully filtered high-quality data can outperform training on a substantially larger, lower-quality dataset. This establishes that the optimal allocation of training tokens depends on both quality and redundancy, not that repetition is costless.

Our paper does not assume that all duplication is harmful. Rather, the cost of exact repetition is already well documented, and we directly measure such penalties in our controlled pretraining experiments. Our central contribution is to investigate how effective redundancy changes with model capability and corpus scale. In particular, we provide evidence that semantically equivalent but surface-distinct documents produce increasingly aligned training gradients as models become more capable, suggesting that conventional surface-level deduplication may underestimate effective redundancy at scale.

We agree that directly measuring the training penalties of semantic duplicates, as distinct from exact repeats, is an important direction for further validation. Nevertheless, the empirical evidence that excessive repetition can harm pretraining is extensive and extends to modern, multi-billion-parameter language models. We will clarify the distinction between beneficial intentional upsampling and harmful redundancy and incorporate these references into the manuscript.

References

[1] Lee et al. (2022). Deduplicating Training Data Makes Language Models Better. ACL 2022.
https://aclanthology.org/2022.acl-long.577/

[2] Tirumala et al. (2023). D4: Improving LLM Pretraining via Document De-Duplication and Diversification. NeurIPS 2023.
https://arxiv.org/abs/2308.12284

[3] He et al. (2024). SoftDedup: An Efficient Data Reweighting Method for Speeding Up Language Model Pre-training. ACL 2024.
https://aclanthology.org/2024.acl-long.220/

[4] Muennighoff et al. (2025). Scaling Data-Constrained Language Models. Journal of Machine Learning Research, 26(53). Originally presented at NeurIPS 2023.
https://jmlr.org/papers/v26/24-1000.html

[5] Chudnovsky et al. (2026). Internal Data Repetition Destroys Language Models. arXiv preprint.
https://arxiv.org/abs/2606.24998

[6] Fang et al. (2025). Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality. NeurIPS 2025.
https://proceedings.neurips.cc/paper_files/paper/2025/hash/2aba6ec20299931d46ddeefd5ddcb442-Abstract-Conference.html

Reviewer_WNk8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the scale dependence of data duplication in pretraining, in three parts. First, using per-document full-parameter gradient cosine similarity of the cross-entropy loss, the paper quantifies the emergence of semantic sensitivity. For more capable models, gradients of semantic-preserving transformations (translation, random character swaps, word drops, alternating capitalization) align more with the original document than an unrelated-document baseline, and larger models reach a given separation earlier in training. Second, embedding 192M FineWeb-Edu-Dedup [3] documents with EmbeddingGemma-300m [5], they find that nearest-neighbor cosine statistics across subset sizes from 10^4 to 10^8 follow the power law of an isotropic null at moderate scales and deviate sharply beyond roughly ten million documents, with the fully synthetic Recycling-the-Web corpus [4] deviating about an order of magnitude earlier. Third, in controlled pretraining of Chinchilla-optimal [6] models from 34M to 344M parameters, streams sampled with replacement from pools of K unique documents (exact repeats) induce validation-loss penalties that grow with compute and break naive extrapolation. The theory models semantics as hierarchical latents, defines an effective pool size K_eff via the Simpson collision probability, fits a three-parameter plane law for the fractional loss increase, and estimates K_eff from the mean nearest-neighbor cosine of the training stream. On the runs used to fit the law, the mean absolute relative error is about 0.77% with the true K and 0.90% with the estimated K_eff.

Critical strengths

1. A well-executed, multi-angle measurement study of an important, under-quantified question. Whether deduplication should be defined relative to model capability is a valuable question. The gradient-alignment diagnostic is cleanly designed, covering 25 models and training checkpoints with a negative baseline and AUC, and the trend direction is credible. The nearest-neighbor collision analysis over 192M embedded documents is a substantial measurement effort, the synthetic-versus-real comparison is concrete, and the correlation between embedding and gradient cosine is checked with a second embedding model on 3,000 document pairs.

2. The effective-uniqueness viewpoint yields a cheap, computable diagnostic. The hierarchical-latent framing with the Simpson effective count leads to an estimator of K_eff from a single stream statistic, the mean nearest-neighbor cosine, with the derivation and its approximation conditions clearly laid out. For corpus builders this is a testable route to quantifying effective diversity, even though validation is currently incomplete.

Critical weaknesses

1. The core mechanistic claim has no training-side validation. The paper argues that semantic duplicates increasingly behave like exact duplicates as models become more capable, and thereby harm training. The link "semantic duplicates harm training" is supported only by static gradient geometry of pretrained models. Around ten billion parameters in Figure 1, translation pairs have gradient cosine of about 0.1 to 0.2, but the training penalty associated with this partial alignment has not been established. All training experiments use exact repeats, a setting studied by repeated-data scaling laws [1][2], and the authors acknowledge in the main text and Appendix C that exact repeats serve as a proxy. At fixed K, increasing compute also increases the number of epochs over the pool, so this experiment alone does not establish that more capable models are more sensitive to semantic duplicates. The middle link of the paper's causal chain is currently an assumption, not a finding.

2. Restored scaling lacks held-out extrapolation validation, and the estimator has not been validated on non-exact semantic duplicates. The 0.77% and 0.90% errors are reported on the runs used to fit the law, with no held-out extrapolation, for example fitting only on mild-reuse runs and predicting the K=10^5 breakdown. The K_eff estimator is validated only for exact repeats with m+ = 1, where duplicates are countable by hashing anyway. For semantic collisions it requires a calibrated m+ and a high-uniqueness reference stream m0(N), neither of which is demonstrated, and the paper gives no K_eff estimate for a real corpus and no concrete prediction for production-scale training. The claim of restored predictability is missing key validation before it can be applied.

Questions for authors
  1. The bridging experiment. Could you mix surface-distinct semantic duplicates into training data, for example controlled fractions of translation pairs or same-language paraphrase pairs, and compare their loss penalty against matched counts of exact repeats, for both small and large models.
  2. Held-out extrapolation. Could you fit the plane law only on mild-reuse runs and then predict the K=10^5 breakdown regime, reporting extrapolation error rather than error on the fitting set.
  3. K_eff for a real corpus. Using your existing 192M embeddings, could you estimate K_eff of FineWeb-Edu-Dedup. How would m0(N) and m+ be obtained in this setting. Substituted into the plane law, what loss penalty does this predict for a compute-optimal production-scale run.
  4. Are high-cosine nearest neighbors actually semantic duplicates. Could you sample high-similarity nearest-neighbor pairs at the largest scale (10^8 documents) and verify the fraction that is semantically equivalent with a strong model or human check. Representative case studies across similarity ranges, with document excerpts and qualitative analysis, would help distinguish semantic equivalence from topical similarity or shared templates and stock phrases. How much of the deviation from the isotropic null is explained by semantic equivalence versus generic topical clustering.
  5. How do you explain the differences across models. In Table 1, Pythia-14M shows correlation 0.57 while Pythia-31M to 160M are near zero, and in Figure 8 gradient alignment in the Meta-Llama family does not increase with size. How does each metric relate to model capability. Is semantic sensitivity driven mainly by size, training data, or tokenizer language coverage.
  6. Predictive improvement over existing repeated-data scaling laws [2]. In the exact repetition experiments of Section 4, does the proposed plane law provide more accurate loss predictions on held-out runs than existing repeated-data scaling laws [2]?

References:

[1] Hernandez et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487, 2022.

[2] Muennighoff et al. Scaling data-constrained language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.

[3] Penedo et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024.

[4] Nguyen et al. Recycling the web: A method to enhance pre-training data quality and quantity for language models. arXiv preprint arXiv:2506.04689, 2025.

[5] Vera et al. EmbeddingGemma: Powerful and lightweight text representations. arXiv preprint arXiv:2509.20354, 2025.

[6] Hoffmann et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer WNk8

Response to Reviewer WNk8

Thank you for reviewing our paper and identifying areas for improvement. We address your questions below.

The Bridging Experiment

We tried a preliminary version: 100 rewrites per document generated by a 4B Qwen model. We found no clear semantic-repetition penalty in our small models, consistent with their weak semantic sensitivity. Improved data quality or distillation could also offset a penalty, so this does not establish the proposed mechanism. Our training already cost approximately USD 20,000; with T≈20PT \approx 20P, compute grows as C≈6PT≈120P2C \approx 6PT \approx 120P^2. A comparable experiment at much larger scales is beyond our academic lab's budget.

Held-Out Extrapolation and Predictive Improvement

We now hold out the largest available model: 344M parameters, or 153M for the 100M pool. Because the 34M setup differed slightly, we report corrected fits excluding all 34M runs (-34M) and including them (+34M).

KObservedOriginal-34M+34M
100K7.7512.6165.2755.126
300K3.6432.5073.6333.605
600K3.1582.4793.2603.252
1M3.0362.4893.1193.117
10M2.9412.4692.9482.949
100M3.3263.0343.3243.324

With 34M included, five of six predictions are within 3%. At K=105K = 10^5, error falls from 66.2% to 33.9%; the correction helps substantially but still underpredicts the breakdown. Fitting penalties are at most 7.2%, versus 164.3% at this held-out endpoint. The corrected fit retains the measured same-scale clean baseline, so this tests prediction of the duplication penalty.

The corrected fit improves held-out predictions at every pool size compared with ordinary power-law extrapolation (“Original”). Both methods exclude the tested loss.

K_eff for a Real Corpus

Our exact-repeat experiments already validate this approach: substituting estimated KeffK_{\mathrm{eff}} into the loss law gives 0.90% mean relative fitting error, compared with 0.77% using the known pool sizes. We now extend the estimation to FineWeb-Edu-Dedup.

Our new Qwen3-Embedding-0.6B analysis covers 103.95M documents, with mean nearest-neighbor cosine 0.83305. We obtain m+=0.85544m_+ = 0.85544 from 1,000 original–rewrite pairs. Taking each query's highest cosine below m+−0.05m_+ - 0.05 gives the background proxy m0=0.77170m_0 = 0.77170. These yield K^eff=78.8×106\widehat{K}_{\mathrm{eff}} = 78.8 \times 10^6.

Using this embedding-based estimate, extrapolating the corrected fit to a 7B model trained on 140B tokens projects a 24–25% loss penalty.

Are High-Cosine Nearest Neighbors Semantic Duplicates?

At 103.95M documents, a model audit judged 22/30 randomly sampled Gemma nearest-neighbor pairs above cosine 0.99 semantically equivalent (73%; approximate 95% Wilson interval 56–86%). The other eight contained substantive differences.

Examples from the two audits distinguish the cases:

  • Gemma, 0.993: “a specific concept for a limited time” / “a particular idea for a constrained time”—the same article with synonym substitutions.
  • Qwen, 0.999: A 1,893-word essay matches the opening of a 17,317-word compilation: partial duplication.
  • Gemma, 0.921: Two biographies give birth years of “1565” and “1563,” with different scope: topical similarity, not equivalence.

We inspected 24 Qwen nearest-neighbor pairs sampled across four similarity ranges. Extrapolating the small-corpus power law predicts a mean cosine of 0.804 at 103.95M documents, compared with the observed 0.833. In this sample, weighting by the prevalence of each range, semantically equivalent pairs accounted for 24% of positive excess cosine, max⁡(s−0.804,0)\max(s - 0.804, 0), partial duplicates for 54%, and topical matches for 22%.

How Do You Explain the Differences Across Models?

These metrics measure particular invariances, not capability in general. The tested Llama models may already recognize the transformations, leaving little additional alignment to gain with size. The small embedding model also limits the embedding–gradient comparison. We are not sure why Pythia-14M has such a high correlation. Shared lexical features are one possibility. Size, training data, and tokenizer coverage were not independently controlled, so we present these as possible explanations rather than established causes.

Thank you for the detailed and candid responses. The held-out evaluation, the K_eff estimate for FineWeb-Edu-Dedup, the nearest-neighbor audit with case studies, and the clear statement that the preliminary rewrite experiment does not establish the proposed mechanism are all helpful.

However, there is still no training-side evidence that semantic duplicates incur a penalty. The corrected fit improves on plain power-law extrapolation, but it still underpredicts the K = 100K breakdown by about one third, and it is not compared with existing repeated-data scaling laws, so its increment over them remains unmeasured. The projected 24 to 25% penalty at 7B therefore extends beyond what the fit has been shown to predict. The audit also indicates that semantically equivalent pairs account for 24% of the excess nearest-neighbor cosine, against 54% for partial duplicates.

These results make the paper more informative, but since the semantic-duplication mechanism and the breakdown regime remain unvalidated, I keep my rating at 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.