- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how naturally occurring AI-generated web text affects language model pretraining when mixed with human-written data. Through controlled experiments across model sizes and data mixtures, the authors show that AI-generated and human-written tokens have different effects on validation loss and propose a scaling law to characterize these differences. They find that adding AI-generated text can improve loss on AI-written text while worsening loss on human-written text, an effect that mixed validation sets may obscure. However, these loss differences do not consistently translate into differences in downstream benchmark performance.
Critical strengths
The paper addresses an important and timely problem: how the growing prevalence of AI-generated web content may affect the pretraining of future language models. Studying naturally occurring AI-generated text makes the investigation particularly relevant to practical data collection.
The authors provide a systematic empirical analysis showing that AI-generated and human-written tokens are not interchangeable in their effects on human-text validation loss. The proposed scaling law offers an interesting framework for characterizing pretraining on mixtures of these two data sources.
Critical weaknesses
The implications for model capabilities remain insufficiently established. Lower next-token prediction loss does not necessarily imply stronger capabilities, and higher loss on human-written text does not by itself establish broader capability degradation. Indeed, Appendix G reports similar improvements in CORE scores from adding AI-generated and fresh human-written text, despite their different effects on human-text loss.
One possible explanation is that AI-generated text has a narrower distribution, as suggested by prior work on generative monoculture [1]. Related work also shows that data repetition can affect memorization and generalization [2]. These findings motivate further investigation into whether the observed loss patterns reflect distributional differences, redundancy, or broader changes in model capabilities. More informative capability evaluations, together with a deeper analysis of diversity and redundancy beyond the reported topic and format comparisons, would strengthen the paper’s conclusions.
The robustness of the findings to provenance-labeling errors needs further validation. The paper uses EditLens for initial selection and Pangram for final labeling, and reports a false-positive check on pre-ChatGPT web text. These are useful safeguards, but they do not fully establish labeling reliability on contemporary web content, particularly partially edited or mixed human–AI text.
The authors should report agreement and disagreement between detectors and examine whether the main findings remain stable under alternative detectors, confidence thresholds, or a manually audited subset. Such analyses would help assess whether systematic labeling errors or detector-induced selection biases materially affect the estimated scaling behavior.
References [1] Wu et al. Generative Monoculture in Large Language Models. ICLR 2025. [2] Hernandez et al. Scaling Laws and Interpretability of Learning from Repeated Data. 2022.
Questions for authors
Relationship between validation loss and model capabilities. Appendix G reports similar CORE improvements from adding AI-generated and fresh human-written text, despite their different effects on human-text loss. How should this discrepancy affect the interpretation of “harm” from AI-generated data? Could the authors provide evidence distinguishing broader capability degradation from changes in distributional fit, for example through diversity-sensitive evaluations or comparisons that control for topic, format, and redundancy?
Robustness to provenance labeling. Given that the data construction relies on EditLens for initial selection and Pangram for final labeling, how sensitive are the main findings and fitted scaling laws to detector choice and labeling errors? Could the authors provide an audit of contemporary web samples, including detector disagreements and partially AI-edited text, or repeat key experiments using alternative labels or confidence thresholds?
Independence of the held-out evaluation. Were the 477M and 973M models excluded from both coefficient fitting and functional-form selection? Appendix D.1 and Table 8 compare candidate benefit and penalty terms using errors on these held-out sizes, making their role in model selection unclear. If these results informed the final functional form, could the authors provide an evaluation with a separate model-selection set and a genuinely untouched test set?
2 comments
As the discussion period nears its end, we wanted to check in and see if our response has addressed your concerns, particularly about including more discussion of model capabilities. Please let us know there is anything else we can clarify further and we would appreciate it if you would consider updating your assessment if your concerns have been resolved. Thank you!
Thank you for your insightful review, we hope we can address your questions below and will incorporate them into the final version of the paper!
Capabilities and diversity
Capabilities: We read the harm as harm to human-text loss rather than broad capability loss, and will word it that way in the paper: it shows up on specific tasks, not in aggregate CORE. We find that capabilities differ by task, which we will add an analysis of in the final version of the paper. AI text helps most on how-to and explanatory tasks, matching the formats it over-represents, and least on narrative and entity-knowledge tasks. For example, on HellaSwag, AI text (+4.09) improves more than human text (+2.19), but only on the ⅔ of the dataset that are wikihow items (details and table in our reply to RxJ6). Our topic/format experiment in that reply is the controlled comparison you suggest: given the human topic/format mix, AI text still does far worse on human-text loss than fresh human text (C4 +0.32% vs. −1.79% at 135M), and topic and format explain only 8–17% of the gap.
Diversity: AI text is narrower than human text, even within matched topic × format cells: fewer distinct n-grams in all 44 of the largest cells, and lower embedding (Vendi) diversity in 79 of 80. Matching topic and format closes only 10 to 25% of the lexical gap and none of the embedding gap. Single AI documents are as varied as human ones, the narrowness is across documents, the generative monoculture of Wu et al. (2025) showing up in wild web text. We also note that our law fits the loss better than any of the laws for repeating training data (Table 1). We will add discussion of both of these into the paper for reasons why the human and AI models may perform differently.
Robustness to detector
Four independent detectors on 11,000 documents all separate the pools (AUROC 0.94 to 0.98) and find the human pool indistinguishable from 2021 text. Their misses within the AI pool barely overlap (κ ≤ 0.12), so there is no shared systematic error, and Pangram–EditLens agreement is flat across all 23 crawl months. Refitting under the noise rates the detectors imply leaves held-out error and costs essentially unchanged (C4 0.81 vs. 0.83, 1.34× either way). Mixed documents (4.8%) are excluded from both pools; every detector scores them between the two pools, more AI-like as more of their windows are AI. We will note that lightly edited AI text is under-represented and leave its effects to future research.
11,000 documents, official thresholds, every detector reading the same 512-token excerpt.
Held-out independence
Thanks for checking this. The 477M and 973M models were excluded from every coefficient fit, but Table 8 does compare candidate forms on them, and we will say so in Appendix D. We will update the appendix to make this clear, and we also reran selection with 268M as a separate model-selection set, as you suggested: we fit all 27 candidates (Table 8's 16 forms plus the 11 benchmarked laws) through 135M, picked the form with the lowest 268M error, refit it through 268M, and scored it once on 477M and 973M. Selection picks our harm term with a unit AI-token value (η = 1), which still beats all 11 benchmarked laws on every human-text target (C4 0.97 vs. 1.41 for Shukor et al.).
We also fit every law on nested subsets of the development models, through 35.8M, 86.2M, 135M and 268M, and scored each fit on the same 477M and 973M models, which enter no fit. Our law has the lowest error of all 12 laws in 15 of these 16 cells, at extrapolations of up to 27×:
Paired RMSE ×10⁻³ on the 477M and 973M models, ours / Shukor et al. joint; each row fits every law only on models from 19.9M up to the listed size.
The one exception is Paloma fit through 35.8M, where our law is third. On C4, its lead over Shukor et al. is significant at every cutoff (95% bootstrap intervals over control groups exclude zero). We will add both protocols next to Table 1 in the final version.