acceptodds
Under review as a conference paper at ICLR 2027

Model Collapse Is Avoidable Even as Real Data Vanishes: A Sharp Criterion for Iterative Gaussian Estimation

Abstract

Large generative models are commonly trained on data collected from the open web, which may contain data produced by previous generations of the same model. Previous work has shown that recursively training on model-generated data can cause a progressive degradation in model performance, a phenomenon known as *model collapse*. Yet, currently, generative models continue to improve dramatically over time. Existing theoretical research suggests that this is enabled by the availability of new real data at each iteration, kept separate from model-generated data. However, exact separation is not realistic, and uncurated data is still commonly used. We bridge this gap by considering a less restrictive and more realistic setup in which real and generated data are not handled differently during training. Specifically, we present a comprehensive theoretical study of iterative estimation of the mean and covariance of a multivariate Gaussian data distribution. At each iteration, samples are drawn from a mixture of the real data distribution and the estimated distribution from the previous iteration, but are used indistinguishably. We provide a single equivalent condition for consistency of the mean and covariance estimators in Euclidean norms, and convergence to zero of the expected squared -Wasserstein distance between the estimated and true distributions. Using this result, we then show that consistency holds, i.e., model collapse is avoided, for any fixed real data fraction above zero, and even for a fraction that vanishes across iterations, provided it does not vanish too quickly. However, more real data (or data curation) remains important at practical finite times: we show that the convergence rate of the mean estimator deteriorates as the real data fraction approaches zero. We support our results with an empirical investigation of the theoretical setup, and complement our theoretical work with experiments on iterative training of a diffusion model on CIFAR-10 that correlate with our theory.

Primary area:
learning theory
Keywords:
model collapse, self-consuming loops, iterative retraining, iterative estimation, generative models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AWBCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies iterative estimation of a multivariate Gaussian when each round adds an indistinguishable mixture of fresh real samples and samples from the previously fitted model, while retaining all historical data. It establishes a necessary-and-sufficient condition for consistency of the mean, covariance, and fitted distribution in expected squared Wasserstein distance. For a fixed synthetic fraction and batch size, it characterizes the mean-estimation rate, showing a transition at a synthetic fraction of one half. The results are supported by Gaussian simulations and supplementary DDPM experiments on CIFAR-10.

Critical strengths
  1. The necessary-and-sufficient criterion connects the injection of fresh real information with consistency under data accumulation. The joint treatment of the mean, covariance, and fitted distribution makes the result more informative than a stability statement or a sufficient condition alone.

  2. A useful distinction between consistency and efficiency. The convergence-rate analysis shows that avoiding collapse asymptotically can coexist with very slow improvement at finite sample sizes. This provides a meaningful perspective on when identifying or curating synthetic data may be valuable.

Critical weaknesses
  1. The diffusion evidence is limited relative to the broader claims. The CIFAR-10 evaluation relies exclusively on FID and does not show generated-image examples or separate measures of quality and coverage. It uses fixed mixing fractions over a short horizon, and Figure 6 displays a single training seed. Moreover, some settings receive larger optimization budgets, while the high-synthetic setting switches to warm-start training with a lower learning rate. These choices complicate comparisons across mixing fractions. The real-only and synthetic-only controls are useful, but there is no matched comparison with relevant prior retraining or stabilization strategies. The evidence therefore supports qualitative trends under the chosen protocol more strongly than general conclusions about recovery in practical generative models.

  2. The novelty needs a clearer comparison with closely related work. “Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training” (arXiv:2602.16065) also studies fresh real and synthetic data accumulated across rounds. “From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources” (arXiv:2602.10531) studies source-agnostic mixture training and decaying real-data fractions, although its main theoretical workflow differs. These works do not establish that the present results are redundant, but they make the broad first-work claims difficult to assess. The contribution should be positioned around the precise new results, such as the necessary-and-sufficient criterion, covariance analysis, or matching rates.

Questions for authors
  1. Could you provide representative, uncurated CIFAR-10 samples from early and late generations to help interpret the FID trends? If complementary measurements of sample quality or diversity are available, do they support the same interpretation?

  2. For the synthetic batches generated by the previous model, could you clarify the DDIM stochasticity setting and whether class imbalance, repeated images, or visible artifacts were examined? Could these properties influence the observed degradation in addition to the synthetic-data fraction itself?

  3. How comparable are initialization procedures and optimization budgets across mixing fractions and against the real-only controls? Could you clarify how the changes in training settings may affect the comparisons, and whether matched comparisons with relevant prior retraining strategies are available?

  4. Could you summarize the variability in the duplicate runs mentioned in Appendix D? In particular, were the ordering of the mixing fractions and the late-stage plateau reproduced across runs?

  5. Should the agreement between the diffusion experiments and the Gaussian theory be interpreted qualitatively, or is a quantitative correspondence in convergence rates intended? Since the diffusion experiments use fixed mixing fractions, could you also clarify the scope of the recovery claim when the real-data fraction decreases toward zero?

5 comments
AuthorsOpenReview ↗

Response to Reviewer AWBC (1/3)

We thank the reviewer for their thorough review and valuable suggestions, and are pleased that they find our necessary and sufficient criterion and the joint treatment of the mean, covariance and fitted distribution informative, and the distinction between consistency and efficiency meaningful. We address their concerns below.

W1 — The diffusion evidence is limited relative to the broader claims.

We thank the reviewer for this comment. We agree: the diffusion experiments support qualitative trends rather than general conclusions about practical generative models, and we have toned down our claims accordingly in the revision (Section 4 and Appendix D). Our main result, consistency, is asymptotic in the number of iterations, and is verified under the theoretical setting (Figure 1). The number of iterations in the practical diffusion model experiments is limited by the dataset size. To strengthen the empirical evidence, we reran all experiments under a unified protocol and extended them:

  • Uncurated sample grids, and precision and recall as separate measures of quality and coverage (Q1).
  • A unified protocol: every model is trained from scratch, for at most 1M steps (no run reached this maximum), with early stopping on a validation set (Q3).
  • Three seeds for every setting, with the same ordering in α\alpha in every seed (Q4).
  • Two sequences αt\alpha_t with a vanishing fraction of real data, for which our theory predicts consistency and collapse, respectively (Q5).
  • An additional fixed value, α=0.5\alpha = 0.5.

All of these are included in the revision (Appendix D, Figures 6 and 7 and Tables 1 and 2; summary in Section 4).

W2 — Relation to closely related work. We thank the reviewer for pointing out these recent works [1, 2], the first of which is not yet peer-reviewed. We agree that our contribution should be positioned around the precise new results. In the revision, we add both works, briefly in the related work (Section 2) and in detail in a new appendix (Appendix F), and remove the broad "first-work" claims. In short:

  • [1] studies the same accumulation setting with fresh real data and generated data, for general generators, with a fixed fraction of real data and fixed batch sizes. It provides an upper bound on the error of order t−min⁡(p,αreal)t^{-\min(p, \alpha_{\mathrm{real}})}, which for the Gaussian mean matches the upper bound of our Theorem 3.6. Our results add: matching lower bounds, which show that this upper bound is tight and that the transition at a synthetic fraction of 12\tfrac12 is real; an if-and-only-if criterion for consistency with time-varying αt\alpha_t and nb(t)n_b(t), which covers a vanishing fraction of real data; exact finite-tt results (the bias of the covariance estimator and the recursion for Ct\boldsymbol{C}_t); and the joint treatment of the mean, the covariance, and W2W_2.
  • [2] mainly studies the replacement setting for categorical models fitted by maximum likelihood, and gives sufficient conditions for decaying real-data fractions. Our criterion is necessary and sufficient, in the accumulation setting. It is also consistent with their empirical finding that accumulation fails when the real fraction decays as 1/t1/t with polynomially growing batches, since then our series converges. Moreover, our results extend to categorical data in the accumulation setting (new Appendix E.1; see our reply to Reviewer iWF2, W3).

[1] Wang, Niu & Li, "Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training", arXiv:2602.16065, 2026.

[2] Bakshi & Chakraborty, "From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources", NeurIPS 2026.

AuthorsOpenReview ↗

Response to Reviewer AWBC (2/3)

Q1 — Samples, and measures of quality and diversity. We thank the reviewer for this suggestion. In the revision, we add uncurated samples from the shared initial model and from the last iteration of each fixed α\alpha, generated from the same initial noise (Appendix D, Figure 7): the final samples become sharper the more real data each iteration contains, whereas for α=1\alpha=1 they do not improve on the initial model and lose detail, consistent with the precision and recall reported below. We also measure precision and recall [1], which separate sample quality (precision) from coverage of the real distribution (recall). Each model is evaluated on 3,750 samples against 3,750 CIFAR-10 test images (k=3k=3). The table reports the values at the first iteration (shared by all settings) and the last iteration, averaged over 3 seeds; standard deviations across seeds are at most 0.020.02:

SettingPrecisionRecall
Initial model (shared)0.6140.443
α=0\alpha = 00.6730.595
α=0.25\alpha = 0.250.6420.577
α=0.5\alpha = 0.50.6130.550
α=0.75\alpha = 0.750.5900.501
α=1\alpha = 10.5550.412
αt=1−1/ln⁡(t+3)\alpha_t = 1-1/\ln(t+3)0.6180.549
αt=1−1/(t+1)\alpha_t = 1-1/(t+1)0.5760.516

They support the FID interpretation. For α=1\alpha=1, precision and recall both decrease (0.614→0.5550.614\to0.555 and 0.443→0.4120.443\to0.412): the model loses quality and diversity, the typical signature of collapse. For every α<1\alpha<1, recall increases (precision decreases slightly for large α\alpha), and at the last iteration both metrics are ordered by α\alpha. Additionally, for the two varying {αt}\lbrace\alpha_t\rbrace sequences, the schedule satisfying the criterion (αt=1−1/ln⁡(t+3)\alpha_t=1-1/\ln(t+3)) ends above the one violating it (αt=1−1/(t+1)\alpha_t=1-1/(t+1)). We include this table in the revision (Appendix D, Table 1).

[1] Kynkäänniemi et al., "Improved Precision and Recall Metric for Assessing Generative Models", NeurIPS 2019.

Q2 — Generation of synthetic images, and analysis of class imbalance, repeated images and visible artifacts. We thank the reviewer for this question. We use DDIM with 200 steps and η=0\eta=0 (no stochastic noise added during the diffusion steps) to sample from the model selected in the previous iteration. Sampling is therefore deterministic given the initial Gaussian noise, and every batch uses fresh noise with its own seed. Generated images are added to the pool without any filtering. To check whether class imbalance, repeated images or memorization could contribute to the degradation, we classify 3,750 samples per model with a pretrained ResNet-56 (94.4% CIFAR-10 test accuracy) and measure near-duplicates in Inception features, with distance thresholds calibrated to flag 1% of real held-out images (within a sample, and relative to the 50,000 training images). Values are at the first (shared by all settings) and last iterations, averaged over 3 seeds; standard deviations across seeds are at most 0.020.02 for class entropy and largest-class share, and at most 0.4% for the near-duplicate and training-image columns:

SettingClass entropyLargest classNear-duplicatesNear a training image
Real test images0.99950.1061%1%
Initial model (shared)0.9710.1590.5%0.1%
α=0\alpha = 00.9960.1221.3%0.2%
α=0.25\alpha = 0.250.9890.1421.0%0.1%
α=0.5\alpha = 0.50.9900.1400.8%0.1%
α=0.75\alpha = 0.750.9900.1460.1%0.0%
α=1\alpha = 10.9840.1550.1%0.0%

Class entropy is normalized (1 = uniform). According to the pretrained classifier, the initial model (shared by all settings) is mildly imbalanced. With real data in every batch (α<1\alpha<1), this imbalance is largely corrected, whereas for α=1\alpha=1 it changes only slightly. Near-duplicates stay close to or below the real-data level and do not increase with α\alpha, and near-copies of training images are at most 0.2% for every setting. Hence, the degradation at α=1\alpha=1 is not driven by class imbalance, repeated images or memorization: its FID, precision and recall worsen over iterations, while its class imbalance changes only slightly and its duplicate rate is among the lowest of all settings. The uncurated samples (Q1) show no systematic visual artifacts. We include these checks and the sampling details in the revision (Appendix D, Table 2).

AuthorsOpenReview ↗

Response to Reviewer AWBC (3/3)

Q3 — Comparability of training settings, and prior retraining strategies. We thank the reviewer for this question. In the revised experiments (Appendix D), all models, including the real-only control (α=0\alpha=0), are initialized and trained identically: every model in each iteration is trained from scratch with the same schedule, for at most 1M steps, using early stopping on a validation set. The previous warm-start training is removed. Hence, the comparison across mixing fractions does not depend on the training budget or initialization (see also our reply to W1).

Regarding prior strategies: our experiments are designed to test a prediction of our theory, namely the effect of the fraction of fresh real data when real and generated samples are pooled indistinguishably, rather than to propose a new method. Most stabilization strategies, such as filtering, verification or reweighting, require distinguishing real from generated samples, which our setting deliberately excludes. It is thus fair to leave such empirical investigations out of the scope of our theoretical paper.

Q4 — Variability across runs. We thank the reviewer for this question. In the revision (Appendix D, Figure 6), all experiments are repeated with 3 seeds under a unified protocol, which allows us to measure the run-to-run variation directly. At every iteration after the first, the gaps between consecutive values of α\alpha exceed the standard deviations across 3 seeds by a factor of at least 3.5, indicating that the trends do not depend on the random seeds. The ordering of the mixing fractions holds at every iteration t≥2t \ge 2 and in every seed. The late-stage behavior of each curve is also reproduced in every seed: for large α<1\alpha<1, the FID levels off after a few iterations, while for α=1\alpha=1 the FID never improves but remains bounded.

Q5 — Qualitative or quantitative agreement, and the scope as the real-data fraction vanishes. We thank the reviewer for this question. The agreement should be interpreted qualitatively: we do not claim a quantitative correspondence in convergence rates, and we have removed from the revision the claims on the rates, the phase transition at α=12\alpha=\tfrac12, and the value of detection for diffusion models (Section 4 and Appendix D).

Regarding a vanishing fraction of real data, our theory gives the exact scope of the recovery claim: recovery (i.e., consistency) holds if and only if ∑t=1∞(1−αt)(1−ρt)=∞\sum_{t=1}^\infty(1-\alpha_t)(1-\rho_t)=\infty (Theorem 3.3). Hence, the fraction of real data may vanish, but only slowly. To examine this setting with diffusion models, we added two sequences with a vanishing fraction of real data: αt=1−1/ln⁡(t+3)\alpha_t = 1-1/\ln(t+3), for which our theory predicts consistency, and αt=1−1/(t+1)\alpha_t = 1-1/(t+1), for which it predicts collapse (i.e., bounded degradation). The FID of the former is below that of the latter at every iteration t≥2t \ge 2 and in every seed. Within 9 iterations, this shows the predicted ordering, while the asymptotic separation between the two regimes is shown in the Gaussian experiments, which use the same sequences (Section 3.3, Figure 1(d)–(f)).

We hope that these answers and the revised paper address the reviewer's concerns, and we would be glad to answer any additional questions. We would be grateful if the reviewer would consider raising their score.

Reviewer_zP6LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies iterative Gaussian estimation where new batches mix fresh real data and generated samples from previous models, with all past data retained. The main result provides a sharp necessary and sufficient condition for consistency, backed by a Gaussian mean estimation rate and a transition at α=1/2\alpha=1/2. The authors also include Gaussian simulations and CIFAR-10 diffusion experiments.

Critical strengths
  • Theorem 3.3 gives a useful, precise necessary and sufficient condition for consistency that handles varying batch sizes and synthetic fractions.

  • Theorem 3.6 nicely separates eventual consistency from practical finite-sample quality, showing that a vanishing real-data fraction can lead to extremely slow convergence.

  • The theoretical analysis and covariance recursions are interpretable and well-supported by Gaussian simulations.

Critical weaknesses
  • The CIFAR-10 experiments vary training budgets (up to 800k vs 200k steps) and learning-rate warm-starts across different synthetic fractions (α\alpha), making it hard to isolate the effect of the data mixture.

  • The paper extends Gaussian convergence rate claims to diffusion models and FID scores too loosely. Nine iterations with a single seed do not prove a theoretical phase transition.

  • The consistency proof requires a continuing, unbounded supply of fresh real data (e.g., t/log⁡tt/\log t), meaning a fixed real dataset cannot be diluted indefinitely. Broader claims about web-scale training need stronger qualifications.

Questions for authors
  • Do the qualitative trends in Figure 6 hold under a strictly matched training budget, learning rate, and initialization across all α\alpha? If not, can these empirical claims be narrowed?

  • Can you explicitly separate what is proven by the Gaussian theory from what is merely suggested by diffusion analogies, especially regarding the α=1/2\alpha=1/2 rate transition?

  • Can you clarify the required growth of cumulative real data in the discussion and distinguish true consistency from bounded degradation?

6 comments
AuthorsOpenReview ↗

Response to Reviewer zP6L (1/2)

We thank the reviewer for their thorough review and valuable suggestions, and are pleased that they find our necessary and sufficient criterion (Theorem 3.3) useful and precise, appreciate the separation between consistency and finite-sample quality (Theorem 3.6), and find the analysis interpretable and well supported by the Gaussian simulations. We address their concerns below.

W1 / Q1 — DDPM experimental protocol consistency across values of α\alpha. We thank the reviewer for this comment. We agree, and we reran every DDPM experiment under one protocol. Every model at each iteration tt and in every seed is trained from scratch (no warm starts) with the same schedule and a maximum of 1M optimization steps. Every 50k optimization steps, we evaluate the FID of the model on a validation set of 5,000 held-out real images, select the checkpoint with the lowest validation FID, and stop training once later checkpoints no longer improve (early stopping). In practice, no run reached the 1M maximum. We report the FID of the selected checkpoint on the 10,000-image test set. Hence, the comparison does not depend on the training budget. We use 3 seeds, 9 iterations, and DDIM sampling with 200 steps and η=0\eta = 0. Under this protocol, we ran three additional experiments: fixed α=0.5\alpha = 0.5, and two varying sequences, αt=1−1/ln⁡(t+3)\alpha_t = 1 - 1/\ln(t+3), for which the theory predicts consistency, and αt=1−1/(t+1)\alpha_t = 1 - 1/(t+1), for which it predicts collapse (i.e., bounded degradation).

The FID is ordered as α=0<0.25<0.5<0.75<1\alpha = 0 < 0.25 < 0.5 < 0.75 < 1, and αt=1−1/ln⁡(t+3)\alpha_t = 1 - 1/\ln(t+3) is below αt=1−1/(t+1)\alpha_t = 1 - 1/(t+1), at every iteration t≥2t \ge 2 and in every seed. The test FID at α=1\alpha = 1 rises from 34 to about 39 and overall the increase appears bounded. Within 9 iterations, the experiments are aligned with the predicted ordering between the two sequences {αt}\lbrace \alpha_t \rbrace (the asymptotic separation is shown in the Gaussian experiments).

We report the unified protocol, the additional experiments, and the final results and figures in the revision (Appendix D and Figure 6; summary in Section 4).

W2 / Q2 — More accurate statements on what the DDPM experiments show. We agree, and we have toned down claims. The DDPM experiments now use 3 seeds and a unified protocol (see W1/Q1), and we restrict the claims to what they support:

  • The FID is ordered by α\alpha, and αt=1−1/ln⁡(t+3)\alpha_t = 1 - 1/\ln(t+3) is below αt=1−1/(t+1)\alpha_t = 1 - 1/(t+1), at every iteration t≥2t \ge 2 and in every seed.
  • For every fixed α<1\alpha<1, the FID improves upon its initial value.
  • For α=1\alpha=1, the FID does not improve and overall remains bounded, rising from 34 to about 39. This is in line with our theory, which shows that without fresh real data, the error remains bounded but does not vanish (Proposition 3.5).

The first two trends are in line with our theory, which establishes them asymptotically, in the number of iterations. In practice, we observe them already within a finite number of iterations. We removed the claims related to the convergence rates (Theorem 3.6), the phase transition at α=12\alpha=\tfrac12, and the value of detection. Our main result, consistency, is asymptotic in the number of iterations and is verified in the Gaussian experiments (Figure 1); with diffusion models, the number of iterations is limited by the dataset size. We provide the revised claims in the revision (Section 4 and Appendix D).

AuthorsOpenReview ↗

Response to Reviewer zP6L (2/2)

W3 / Q3 — Unbounded supply of fresh real data. We thank the reviewer for this comment. The reviewer is correct: consistency requires an unbounded supply of fresh real data, and we prove in the revision that this is necessary under our setup (Corollary 3.3.1). In web-scale training, fresh human data is continuously produced, so this condition is plausible. If it stops, our results predict bounded degradation rather than divergence.

Indeed, a bounded amount of real data leads to bounded degradation. Let

R(t)=nr+∑s=1t(1−αs)nb(s)R(t) = n_r + \sum_{s=1}^{t}(1-\alpha_s) n_b(s)

be the expected number of real samples in the pool after tt iterations. Then,

∑t=1∞(1−αt)(1−ρt)=∑t=1∞(1−αt)nb(t)Nt+1≤1nr∑t=1∞(1−αt)nb(t)=1nrlim⁡t→∞R(t)−1,\sum_{t = 1}^\infty (1 - \alpha_t)(1 - \rho_t) = \sum_{t = 1}^\infty (1 - \alpha_t)\frac{n_b(t)}{N_{t + 1}} \le \frac{1}{n_r}\sum_{t = 1}^\infty (1 - \alpha_t) n_b(t) = \frac{1}{n_r}\lim_{t\to\infty}R(t) - 1,

where we used Nt+1≥nrN_{t + 1} \ge n_r. Hence, if R(t)R(t) is bounded, the criterion of Theorem 3.3 fails, and, as we prove in the revision (Proposition 3.5), the error remains bounded, i.e., bounded degradation. Therefore, R(t)→∞R(t) \to \infty is necessary for consistency.

Moreover, we precisely characterize the required growth of cumulative real data. Summation by parts shows that the criterion of Theorem 3.3 (i.e., consistency) holds if and only if

∑t=1∞R(t−1)Nt(1−ρt)=∞,\sum_{t=1}^{\infty} \frac{R(t-1)}{N_{t}}(1-\rho_t) = \infty,

where R(t−1)/NtR(t-1)/N_t is the expected fraction of real data in the pool before iteration tt. For example, for a fixed batch size nbn_b, R(t)≍t/ln⁡tR(t) \asymp t/\ln t yields consistency even though the fraction of real data in the pool vanishes (R(t)/Nt+1≍1/ln⁡t→0R(t)/N_{t+1} \asymp 1/\ln t \to 0), whereas R(t)≍t/(ln⁡t)2R(t) \asymp t/(\ln t)^2 yields bounded degradation, since then the series above behaves as ∑t1/(tln⁡t)=∞\sum_t 1/(t\ln t)=\infty and ∑t1/(t(ln⁡t)2)<∞\sum_t 1/(t(\ln t)^2)<\infty, respectively (Appendix A.6). Thus, the fraction of real data may vanish, but R(t)R(t) must grow almost linearly. We added this result to the revision as Corollary 3.3.1, right after Theorem 3.3 (proved in Appendix A.6), and qualified our claims on web-scale training accordingly (after Lemma 3.4, and in the contributions in Section 1).

We hope that these answers and the revised paper address the reviewer's concerns, and we would be glad to answer any additional questions. We would be grateful if the reviewer would consider raising their score.

I thank the authors for the revisions and would welcome a real-only baseline with a matched real-data budget. The main theoretical advance is the necessary and sufficient condition for consistency under time-varying mixing ratios, while the rate result refines prior work.

I have one follow-up question about diffusion models. Could the authors suggest an experiment or analysis to assess whether their consistency criterion offers insight into diffusion distillation [1,2] or the choice between teacher forcing and self forcing [3] in autoregressive video generation?


[1] Yin, Tianwei, et al. "One-step diffusion with distribution matching distillation." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Luo, Weijian, et al. "Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models." Advances in Neural Information Processing Systems 37 (2023): 76525-76546.

[3] Huang, Xun, et al. "Self forcing: Bridging the train-test gap in autoregressive video diffusion." Advances in Neural Information Processing Systems 39 (2025): 167283-167308.

Reviewer_iWF2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on theoretically understanding the model collapse problems that happen in recursive training on model-generated data. Assuming the data distribution is Gaussian, the work provides a precise characterization of the model's learning dynamics (under the assumption that the model learns perfectly in each step). They show that when providing sufficiently many new data in each step, the new data will correct the bias introduced by the generated samples from the model in the current iteration. Empirical studies have been conducted in a numerical toy setup. They also provide real diffusion model training experiments and show that the results correlate with their theoretical claim.

Critical strengths
  1. This work focuses on an important problem that frequently happens in foundational model training nowadays. A good understanding of the issue will definitely help people develop tools to alleviate it and have a real impact on large-scale training.

  2. The writing is easy to follow.

  3. The empirical results support the claim and the diffusion model example is interesting.

Critical weaknesses
  1. It appears to me that the definition of collapse in this work is different from the general understanding or definition we see in the foundational model training. In particular, in this work, collapse appears to me to simply mean that it cannot converge to the ground-truth distribution. However, in the real empirical setup, collapse means whether the performance of the model will degrade during iterative training. In other words it appears to me that the definition used in this work is weaker than existing literature.

  2. There's a related paper Strong Model Collapse (ICLR 2025) claiming that even with very few synthetic data we cannot converge to the ground truth value; And the result seems to conflict with the result in this work. I was wondering if the authors could provide further details and explanations on these differences.

  3. The paper assumes a very simple setup that the data distribution is Gaussian. While I acknowledge that in theoretical work it's usually a common practice to assume a simple setup for clear characterization, it's a bit hard to tell whether this simple setup could provide enough insights into why collapse happens in real foundational model training.

Questions for authors

I have mentioned several points in the critical weakness section. I'm especially interested in the answers to the first two points.

4 comments
AuthorsOpenReview ↗

Response to Reviewer iWF2 (1/2)

We thank the reviewer for their thorough review and valuable suggestions, and are pleased that they find the problem important, the writing easy to follow, and the diffusion experiments interesting, and that the empirical results support our claim. We address their concerns below.

W1 — Definition of collapse. We thank the reviewer for this comment. We agree that our notion of collapse, defined as failure to converge to the true distribution, counts more cases as collapse than divergence does. The important point is that divergence is not possible under our setting. We provide a complete proof in the revision (Proposition 3.5, proved in Appendix A.8). Note that we have already proved this for the mean in the original submission (Lemma 3.2). This is consistent with prior work, which shows under multiple setups that the worst case in the accumulation setting is bounded degradation, not divergence [1, 2, 3]. In contrast, divergence is possible under the replacement setting, where past data are discarded in every iteration [3, 4, 5]. Hence, under our setup, the distinction is between bounded degradation, termed collapse in our paper, and consistency, termed non-collapse in our paper. Theorem 3.3 characterizes exactly when each one occurs. We provide a clear definition of collapse in the revision (Section 1).

[1] Gerstgrasser et al., "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data", COLM 2024.

[2] Seddik et al., "How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse", COLM 2024.

[3] Kazdan et al., "Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World", ICML 2025.

[4] Shumailov et al., "AI models collapse when trained on recursively generated data", Nature 2024.

[5] Dohmatob et al., "Model Collapse Demystified: The Case of Regression", NeurIPS 2024.

W2 — Relation to Strong Model Collapse. We thank the reviewer for this comment. The setups of our paper and [1] (Strong Model Collapse) differ, and there is no contradiction between the results. The main result of [1] (Theorem 1) is in a non-iterative setup: a single model is trained once on a mixture of real and synthetic data, where the synthetic data come from a fixed, misspecified external model whose error does not decrease with the amount of data. In our setting, the process is iterative, and the synthetic data are generated by the current model. In [1], collapse (i.e., non-convergence) is shown under any fixed and positive proportion p2∈(0,1)p_2 \in (0,1) of synthetic data (as the number of samples and the dimension grow). Specifically, Theorem 1 in [1] shows that the test error has an additional term ζ>0\zeta > 0 that does not vanish as the number of samples grows and is determined by the fixed error of the synthetic data source; hence model collapse (non-convergence). Our paper shows consistency, i.e., non-collapse, asymptotically in the number of iterations, for any fixed and positive fraction of real data 1−α∈(0,1]1 - \alpha \in (0,1] in every iteration. Intuitively, in our paper, the error of the synthetic data source is not fixed, and decreases with the number of samples. We added this comparison to the revision, briefly in the related work (Section 2) and in detail in a new appendix (Appendix F).

[1] Dohmatob et al., "Strong Model Collapse", ICLR 2025.

AuthorsOpenReview ↗

Response to Reviewer iWF2 (2/2)

W3 — "Gaussian estimation is too simple." We thank the reviewer for this comment. The Gaussian model is a standard setting in theoretical work on model collapse [1, 2, 3, 4]. Its tractable structure allows an exact, if-and-only-if characterization of when collapse is avoided, which is our main contribution. Following the reviewer's comment, we examined where our proofs use Gaussianity: only through the first two moments of a new sample, the fourth moments of the samples, and the closed form of W2W_2. Consequently, in the revision (Section 3.2.4 and new Appendix E, with the setup, formal statements and proofs), our results extend as follows:

  • Arbitrary real data distribution with finite variance, and any fitted distribution whose mean and covariance are the pooled mean and covariance of the accumulated data (e.g., a Gaussian fitted to non-Gaussian data, or a categorical distribution fitted by its empirical frequencies, i.e., a unigram language model): our criterion for the mean, its convergence rates, and the bias of the covariance estimator hold verbatim (Appendix E.1).
  • Location–scale families with a finite fourth moment (e.g., Laplace, logistic, uniform, Student-tt) [5]: all our results hold (Appendix E.2).

Beyond these families, the qualitative predictions also carry over to iterative training of a DDPM on CIFAR-10, where real and generated images are pooled indistinguishably: the FID is ordered by α\alpha at every iteration t≥2t \ge 2 and in every seed, and for α=1\alpha=1 it never improves but remains bounded, in line with our theory (Section 4 and Appendix D, Figure 6).

[1] Shumailov et al., "AI models collapse when trained on recursively generated data", Nature 2024.

[2] Alemohammad et al., "Self-Consuming Generative Models Go MAD", ICLR 2024.

[3] Kazdan et al., "Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World", ICML 2025.

[4] He et al., "Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework", Journal of the Royal Statistical Society Series B, 2026.

[5] Casella & Berger, "Statistical Inference" (2nd ed.), Section 3.5, Duxbury 2002.

We hope that these answers and the revised paper address the reviewer's concerns, and we would be glad to answer any additional questions. We would be grateful if the reviewer would consider raising their score.

I thank the authors for the thorough response and revision. The comparison in W2 with Strong Model Collapse now explains why the results differ, and the additional experiments strengthen the paper. I have not found technical issues, and I appreciate the necessary and sufficient criterion and the α=1/2\alpha = 1/2 transition as clean contributions that future theoretical work can build on.

My remaining reservation concerns significance. A precise characterization of a simplified model is most valuable when the model captures the mechanism behind the real phenomenon, or when its distinctive predictions are validated. Here, the idealized learner abstracts away systematic estimation error, which may be a main driver of collapse in practice. Moreover, the trends in the diffusion experiments are largely what one would expect from the amount of real data each setting receives, while the truly valuable points, such as the exact criterion and the transition at alpha equal to 1/2, remain untested. It remains unclear to me whether the theory’s critical predictions (such as the exact criterion and the transition between the two regimes) carry over to neural networks, although I recognize that such asymptotic predictions are difficult to test within a few iterations.

I am therefore leaning towards keeping my score. My concern is about the degree of significance rather than correctness, and given my limited confidence, I would not object to acceptance if other reviewers find the theoretical contribution more compelling.

AuthorsOpenReview ↗

Response to follow-up comment

We thank the reviewer for engaging with our responses, and are glad that the comparison with Strong Model Collapse and the additional experiments address their concerns.

Systematic estimation error. Our analysis includes a biased estimator (the covariance estimator, whose exact bias we characterize), and the fitted model need not be correctly specified (Appendix E.1, e.g., a Gaussian fitted to non-Gaussian data). A persistent bias of the learner in every iteration is outside our setting, and we agree that it is an important direction for future work.

Significance. Under accumulation, the error never diverges (Proposition 3.5), so the question is whether fresh real data suffices, which our criterion answers exactly. Consequently, a vanishing fraction of real data can suffice, but only if the cumulative amount of real data grows almost linearly (Corollary 3.3.1). Moreover, the transition at a synthetic fraction of 12\tfrac12 quantifies when identifying synthetic data becomes valuable: below it, the convergence rate is of the same order as with real data only, while above it, convergence slows sharply (Theorem 3.6 and Appendix B). These predictions are verified in the Gaussian experiments (Section 3.3); as the reviewer notes, testing them asymptotically with neural networks requires many more iterations than are feasible on CIFAR-10.

Beyond the theoretical setup. Nevertheless, the DDPM experiments include a prediction specific to our criterion: two sequences in which the fraction of real data vanishes, αt=1−1/ln⁡(t+3)\alpha_t=1-1/\ln(t+3), which satisfies the criterion, and αt=1−1/(t+1)\alpha_t=1-1/(t+1), which violates it. From the second iteration on, the FID of the former decreases, whereas that of the latter increases (Figure 6(b)), in line with the proven asymptotic behavior.

We hope this addresses the reviewer's remaining concerns, and we would be glad to answer any additional questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful and constructive reviews. We respond to each reviewer's comments point by point below, and we have uploaded a revised version of the paper, in which changes are highlighted in blue. The main changes are:

  • Theory. We show that the error remains bounded for every schedule {αt}\lbrace \alpha_t \rbrace (Proposition 3.5), so the only two regimes under accumulation are consistency and bounded degradation; we characterize the required growth of the cumulative amount of real data (Corollary 3.3.1); and we extend our results beyond the Gaussian distribution (new Appendix E).
  • Experiments. We reran all DDPM experiments under a unified protocol with 3 seeds, added α=0.5\alpha=0.5 and two sequences with a vanishing fraction of real data, and added sample grids, precision and recall, and checks for class imbalance and repeated images (Section 4 and Appendix D).
  • Claims. We restrict the claims on the DDPM experiments to qualitative trends, and position our contribution with respect to closely related work, including two recent works (Section 2 and Appendix F).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.