acceptodds
Withdrawn from ICLR 2027

Model Collapse in Closed Form: Exact Gaussian Laws and Their GPT-2 Signatures

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Models increasingly learn from data generated by other models, yet the mechanism of the resulting is contested—mean drift or vanishing tails?—and mitigations such as data mixing are designed without an exact law to target. We answer exactly in the Gaussian testbed that both sides of the debate use as their reference model. Under recursive maximum-likelihood training, mean drift and variance contraction contribute to the forward KL—the objective that maximum likelihood actually optimises—doubling the collapse rate; the quadratic cliff that remains is exactly half the blind spot of the plug-in variance estimator, while the reverse KL cancels it entirely, a direction asymmetry between the two divergences. Mixing real data into each generation admits an exact steady state; any schedule whose mixing fraction decays to zero diverges; and constant mixing is the best non-increasing schedule at matched real-data budget, and the only one that bounds every generation. In recursive GPT-2 training these laws appear as token-level signatures: a forward/reverse ratio near two at the first generation, sustained thereafter while both divergences saturate; extinction concentrated in a frequency band above the plug-in window; and mixing schedules that set the level of the degradation plateau—with a decaying schedule that looks safe at short horizons becoming the worst choice at triple the horizon.

Primary area:
generative models
Keywords:
Recursive Generative Training, Model Collapse, Mean Drift, Variance Contraction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_idztOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies model collapse in the Gaussian setting. They compute the expected KL divergence between the original Gaussian and the estimated one after t rounds. They also investigate decaying real-data mixing and show that, in this setting, it cannot prevent collapse. The authors also track KL and reverse KL for GPT-2 training initially on WikiText-2, and then on synthetic data.

Critical strengths
  • The idea of clearing up some disagreements regarding model collapse is well motivated, and the Gaussian MLE setting is very clean. The analysis is straightforward.
Critical weaknesses
  • Scope of the theory: Much of the analysis is known. In fact, in talks and presentations on model collapse that I gave in the past, I presented both Theorem 1 in this paper and the variance contraction in Theorem 2. The Gaussian case has been analyzed since the first paper on model collapse by [1] and in subsequent works such as [2, 3].

  • Relating results to existing ones: There is essentially no discussion of how the results relate to existing ones, despite this already being a very crowded space. Connecting to my previous point, much of this analysis is not novel, so careful discussions that specify what exactly the theorems contribute beyond what is already known are needed.

  • Data Mixing and Accumulation: The entire part of the paper around mixing somewhat feels disconnected from a long line of works that already showed that model collapse does NOT occur in natural mixing regimes (e.g. [4, 5, 6]). The authors make statements such as “Can the mixing fraction decrease over time while still preventing collapse? No” which feel detached from the current state of the field. Once again, the authors do not discuss the relation to any of these, the role of the decaying mixing factor, and the role of data replacement instead of the (perhaps more natural) data accumulation setting.

  • Inconsistency of Results: The regime theorem 2 needs for equation (3) is t≪nt \ll n (so that the order 3 remainder is small), but in such a regime, much of the behavior the authors discuss breaks. First, the authors say “the strictly positive 3t2/n23t^2 / n^2 correction is the quadratic cliff that compounds collapse faster than linearly in tt”, which is false if you are in the t≪nt \ll n regime. Moreover, there is no real collapse in the t≪nt \ll n regime, since the KL stays relatively close. The regime the paper requires for collapse is t≫nt \gg n, but much of the discussion and interpretation of the results does not hold in that regime.

  • Experiments: It's not clear what the authors hope to accomplish with the figure. The figure doesn’t actually show collapse by standard definitions, as the KL divergence stabilizes, and does not continue to grow linearly or to diverge to infinity as the Gaussian theory suggests.

[1] - The curse of recursion: Training on generated data makes models forget, Shumailov et al., 2023

[2] - Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World, Kazdan et al., 2024

[3] - Rate of Model Collapse in Recursive Training, Suresh et al., 2024

[4] - On the stability of iterative retraining of generative models on their own data, Bertrand et al., 2023

[5] - Universality of the π2/6\pi^2/6 Pathway in Avoiding Model Collapse, Dey and Donoho, 2024

[6] - When Models Don't Collapse: On the Consistency of Iterative MLE, Barzilai and Shamir, 2025

Questions for authors

Can you address the weakness above?

Reviewer_4fZiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper observes forward KL and reverse KL when recursively training with data produced by the model. It also considers the setting where real data is mixed in at each recursion and provides GPT-2 experiments.

Critical strengths

Since in this era we are running out of internet corpus, we might have to use the data produced by models. In this view, I think this paper is handling a timely subject.

Critical weaknesses

In Remark 5 and Appendix B.4, the authors say that μt→μ0\mu_t \to \mu_0 almost surely due to the Robbins–Monro theorem, but it is hard to understand why the conditions for the Robbins–Monro theorem are satisfied.

Questions for authors

I have written in the weakness section that the authors say that μt→μ0\mu_t \to \mu_0 almost surely due to the Robbins–Monro theorem. Since there is a possibility that I am misunderstanding or that the explanation in the paper is too compact to understand, it would be helpful if the authors could clarify this.

Reviewer_h4ZPOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the model collapse under recursive maximum-likelihood training. It focuses on the Gaussian models and derives the expected forward and reverse KL divergences between the true data distribution and model distribution at the generation tt. Its analysis separates the contribution of mean drift and variance mismatch. It also analyzes the real data mixing and does experiments on recursive GPT-2 training.

Critical strengths
  1. This paper derives the expected forward and reverse KL divergences. These formulas quantify and separate the contributions from mean drift and variance mismatch.

  2. This paper also compares the maximum-likelihood and bias-corrected variance estimators. It shows that the bias correction halves the quadratic cliff coefficient but leaves the leading rate unchanged.

Critical weaknesses
  1. The motivation/significance of the KL divergences is not sufficiently clear. Authors should explain what additional insight they can add into model collapse.

  2. The mixing analysis only considers the known-variance setting and does not involve the variance mismatch. Hence, it is still not clear how the real data mixing affects the variance mismatch.

  3. The connection between the theories and the GPT-2 experiments is not clear. The authors should specify the clear correspondence.

Questions for authors

See the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.