- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors argue that since the “evidence” produced by AI models can themselves get used in the later stages of training that builds a co-dependence between current model capabilities and future data quality. In that regard, the core question authors study is whether scaling the model at the right time can alleviate the pollution caused by poorer quality evidence, or more broadly whether scaling can help with this problem at all in the first place. Under their analysis assumption, the system can support training of both fully clean and persistently degraded equilibria given a large model is introduced earlier since delaying its introduction can worsen the model. The authors present numerical simulation as well as controlled experiments on a Qwen2.5 model to suggest replacing clean data with model-generated outputs degrades subsequent models, while auditing and repairing the data reduces this effect. Crucially, these experiments seem to hold the model size fixed.
Critical strengths
I think the paper studies a timely problem and nicely identifies the coupling between the current model capacbility and future data quality which is currently underlooked in many scaling analysis.
Critical weaknesses
-
The key experiment done doesn’t seem to test the model’s central scaling claim as the central theoretical claim probes whether the same larger model earlier or later leads to different outcomes in data pollution regimes; whereas the language model experiment holds the model scale fixed. The data generation share is varied but that also seems more pre-determined than basis of the learning dynamics.
-
Additionally, empirical setting makes several results predictable. For instance, if the generator produces ~17% correct facts, while independent production is correct by construction then replacing 20% of corpus with 75% model generated replacements predicts ~12.43% false data which matches reported first generation results. I am not convinced whether these experiment setup provide value in understanding real world large scaling language mdoelling experiments and how should data curation be modulated there.
-
I also found the presentation of the paper to be unnecessarily abstract with substantial notations and terminology that make the underlying mechanism harder to follow. For instance, “evidence production” here largely seems to refer synthetic data generation and evolving the pool of overall data. I think simpler terminology with qualitative example trajectories to motivate would greatly help with the presentation of the paper.
Questions for authors
- Could you directly test the timing result by comparing a larger model used from the very beginning with a smaller model followed by larger model; evertyhign else being equal.
- How do the abstract scale parameter correspond to model size, training compute or training duration in modern large language model training? Can you provide some insights for practical pretraining/midtraining/posttraining regimes.
4 comments
Part 2 of 3
Additionally, empirical setting makes several results predictable. ... reported first generation results. ...
We agree that the first-generation fraction of incorrect evidence follows directly from the replacement mechanism. With 20% corpus turnover, 75% model-produced replacements, and producer correctness , the error fraction contributed by model production is .
For the 1.5B producer, , giving an error fraction of approximately 12.43%, as the reviewer observes. This calculation, however, only serves as a verification of the construction of the evidence pool; it is not by itself the empirical learning result.
The additional experiment reported in Response (1) measures what happens when a new learner is trained on the resulting evidence. Although both final learners use the same 3B architecture and training protocol, early scaling improves external accuracy by 6.10 and 10.70 percentage points across the two training seeds. This performance difference is not determined by the above arithmetic computation using the replacement fraction.
We additionally extended the curation experiment through P2, applying at both generations with a fixed pseudorandom audit mask across generations, as detailed in Appendix E.2. Relative to no curation, the P2 corpus contains 11.21% rather than 22.50% incorrect evidence, while a fresh 1.5B learner achieves 34.54% rather than 32.91% external accuracy, an improvement of 1.63 percentage points (paired 95% CI ).
The replacement arithmetic accounts for the immediate evidence-error fraction under the controlled protocol; it does not determine the downstream learner's response to the resulting evidence distribution. The latter is what our learning experiments measure.
The revised manuscript makes this distinction explicit. Equation (65) in Appendix E.1 describes the immediate replacement arithmetic, while Tables 7–9 report downstream learning and evidence-quality outcomes. Appendix E.3 discusses the limitations of the finite-generation evidence. These additions are highlighted in blue.
I also found the presentation of the paper to be unnecessarily abstract ...
Thank you for this suggestion. We agree that the mechanism can be motivated more concretely before introducing the dynamical system.
Consider two organizations that start with the same collection of independently produced records. Both begin using AI assistance to produce new records that will subsequently enter their training data. One deploys a stronger model immediately; the other initially uses a weaker model and upgrades later. Even if both eventually train the same larger model, the evidence available to them may differ because of the records produced during the intervening period.
This is the feedback mechanism studied in the paper. By evidence production, we mean the creation of new documents, labels, code, reports, or other records that can subsequently be used for learning. AI assistance can change this process by replacing some records that would otherwise have been produced independently. The issue is therefore not simply the addition of synthetic data to an unchanged training corpus.
This example illustrates the mechanism behind the scaling-timing result. The new experiment in Appendix E.1 implements the comparison directly, using two production histories while holding the final learner architecture fixed. It also makes clear that the effect concerns changes to the evidence subsequently available for learning, rather than parameter inheritance between successive models.
We will incorporate a concise version of this example into the Introduction in the final version, before presenting the formal model, to make the motivation and feedback mechanism more concrete.
Part 3 of 3
Q1: Could you directly test the timing result by comparing a larger model used from the very beginning with a smaller model followed by larger model; evertyhign else being equal.
Yes. Response (1) reports exactly this controlled comparison, including two downstream training seeds and a second generation of evidence feedback.
Q2: How do the abstract scale parameter correspond to model size .. pretraining/midtraining/posttraining regimes.
The parameter represents effective model capability, which can depend on parameter count, training compute, and training duration. In our new experiment, we operationalize it through model size (1.5B versus 3B parameters).
A practical calibration procedure is to construct evidence corpora with different controlled pollution levels , train a fresh learner for each model configuration and pollution level, and measure production accuracy against an independent reference. This distinguishes baseline capability from sensitivity to evidence degradation.
For example, under a normalized extension of our canonical specification in which clean-data production quality may depend on scale, we may write
Under this specification, we can estimate the coefficient from the slope of normalized accuracy against , and obtain by taking its negative logarithm. With multiple pollution levels, the coefficient can be fitted by least squares. This is an effective-scale estimate conditional on the chosen functional form, not a universal conversion from parameter count to .
To illustrate, if a configuration achieves 50% production accuracy on clean evidence and 48% at pollution level , the normalized sensitivity is , corresponding to an effective under this specification.
Adoption and curation effectiveness can be estimated separately from deployment records and data audits. Given these quantities, a practitioner can simulate different upgrade times and curation levels and compare their predicted evidence-quality trajectories.
For example:
- Pretraining: Assess whether increasingly AI-mediated web data require stronger provenance filtering or independently verified sources.
- Midtraining: Determine whether upgrading deployed coding assistants before collecting specialized code and documentation is preferable to upgrading only the subsequent learner.
- Posttraining: Determine whether upgrading AI-assisted annotators or strengthening human verification before collecting new labels can protect future training data quality.
The common issue is that improving the learner after data collection need not undo changes in the evidence produced earlier.
In the experiments, model size provides one operational distinction between scales, while the theoretical parameter represents effective capability more generally. Appendix E.1 provides the controlled comparison between 1.5B and 3B models. A broader empirical calibration would require measurements across multiple model configurations and controlled evidence-quality levels; the proposed calibration above is illustrative rather than an estimate obtained from our experiments.
Thank you again for your time and constructive feedback. We hope you have had a chance to review our rebuttal and revised manuscript, and would appreciate it if you could let us know of any remaining concerns while there is still time for us to respond before the discussion period closes.
Part 1 of 3
We thank the reviewer for the careful reading and constructive comments. We appreciate the recognition that the paper addresses a timely problem and identifies a coupling between model capability and future data quality that is often overlooked in scaling analyses. We address the empirical concerns with new scaling-timing and sustained-curation experiments, and clarify the mechanism and interpretation of these results in the revised manuscript. The additions are highlighted in blue.
To address this concern, we have completed a controlled experiment comparing early and delayed scaling while holding the eventual learner scale fixed.
We compare two schedules:
Both schedules begin with the same evidence corpus and use identical production requests, corpus turnover (20%), model-produced share (75% of replacements), and curation rate (zero). The final learner architecture is the same in both arms, and no parameters are inherited from the evidence-producing model.
On 23,250 matched production requests, the 3B producer achieves 35.15% factual accuracy compared with 17.15% for the 1.5B producer. The resulting evidence pools contain 9.73% and 12.43% incorrect slots, respectively.
We then train fresh 3B learners on these evidence pools using two downstream training seeds. On the same 18,750 externally grounded questions, we obtain:
The respective paired 95% confidence intervals are [5.31,6.89] and [9.91,11.49] percentage points, conditional on the trained checkpoints. The mean early-scaling advantage across the two seeds is 8.40 percentage points. The advantage persists through another generation of evidence feedback: at P2, the corresponding differences are 7.38 and 5.77 percentage points across the two seeds.
Thus, even when both schedules reach the same final model scale, their different evidence-production histories lead to substantially different external performance.
The new experiments are presented in Appendix E.1 of the revised manuscript, highlighted in blue. Tables 7–8 report the results across both generations and training seeds, including evidence-quality measurements and paired confidence intervals.
This is correct for the controlled experiment. We fix the share of model-produced replacements to isolate the effect of the quality of newly produced evidence on subsequent learning. If the production share changed simultaneously with model scale, the two effects would be difficult to separate.
The theoretical model allows adoption to evolve endogenously, whereas the experiment deliberately holds the production mixture fixed. The experiment therefore tests the evidence-feedback mechanism under a controlled production policy, rather than attempting to estimate the adoption dynamics.
This distinction is now made explicit in the interpretation and limitations discussion in Appendix E.3 of the revised manuscript, highlighted in blue.