- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies iterative estimation of a multivariate Gaussian when each round adds an indistinguishable mixture of fresh real samples and samples from the previously fitted model, while retaining all historical data. It establishes a necessary-and-sufficient condition for consistency of the mean, covariance, and fitted distribution in expected squared Wasserstein distance. For a fixed synthetic fraction and batch size, it characterizes the mean-estimation rate, showing a transition at a synthetic fraction of one half. The results are supported by Gaussian simulations and supplementary DDPM experiments on CIFAR-10.
Critical strengths
-
The necessary-and-sufficient criterion connects the injection of fresh real information with consistency under data accumulation. The joint treatment of the mean, covariance, and fitted distribution makes the result more informative than a stability statement or a sufficient condition alone.
-
A useful distinction between consistency and efficiency. The convergence-rate analysis shows that avoiding collapse asymptotically can coexist with very slow improvement at finite sample sizes. This provides a meaningful perspective on when identifying or curating synthetic data may be valuable.
Critical weaknesses
-
The diffusion evidence is limited relative to the broader claims. The CIFAR-10 evaluation relies exclusively on FID and does not show generated-image examples or separate measures of quality and coverage. It uses fixed mixing fractions over a short horizon, and Figure 6 displays a single training seed. Moreover, some settings receive larger optimization budgets, while the high-synthetic setting switches to warm-start training with a lower learning rate. These choices complicate comparisons across mixing fractions. The real-only and synthetic-only controls are useful, but there is no matched comparison with relevant prior retraining or stabilization strategies. The evidence therefore supports qualitative trends under the chosen protocol more strongly than general conclusions about recovery in practical generative models.
-
The novelty needs a clearer comparison with closely related work. “Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training” (arXiv:2602.16065) also studies fresh real and synthetic data accumulated across rounds. “From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources” (arXiv:2602.10531) studies source-agnostic mixture training and decaying real-data fractions, although its main theoretical workflow differs. These works do not establish that the present results are redundant, but they make the broad first-work claims difficult to assess. The contribution should be positioned around the precise new results, such as the necessary-and-sufficient criterion, covariance analysis, or matching rates.
Questions for authors
-
Could you provide representative, uncurated CIFAR-10 samples from early and late generations to help interpret the FID trends? If complementary measurements of sample quality or diversity are available, do they support the same interpretation?
-
For the synthetic batches generated by the previous model, could you clarify the DDIM stochasticity setting and whether class imbalance, repeated images, or visible artifacts were examined? Could these properties influence the observed degradation in addition to the synthetic-data fraction itself?
-
How comparable are initialization procedures and optimization budgets across mixing fractions and against the real-only controls? Could you clarify how the changes in training settings may affect the comparisons, and whether matched comparisons with relevant prior retraining strategies are available?
-
Could you summarize the variability in the duplicate runs mentioned in Appendix D? In particular, were the ordering of the mixing fractions and the late-stage plateau reproduced across runs?
-
Should the agreement between the diffusion experiments and the Gaussian theory be interpreted qualitatively, or is a quantitative correspondence in convergence rates intended? Since the diffusion experiments use fixed mixing fractions, could you also clarify the scope of the recovery claim when the real-data fraction decreases toward zero?
5 comments
Response to Reviewer AWBC (2/3)
Q1 — Samples, and measures of quality and diversity. We thank the reviewer for this suggestion. In the revision, we add uncurated samples from the shared initial model and from the last iteration of each fixed , generated from the same initial noise (Appendix D, Figure 7): the final samples become sharper the more real data each iteration contains, whereas for they do not improve on the initial model and lose detail, consistent with the precision and recall reported below. We also measure precision and recall [1], which separate sample quality (precision) from coverage of the real distribution (recall). Each model is evaluated on 3,750 samples against 3,750 CIFAR-10 test images (). The table reports the values at the first iteration (shared by all settings) and the last iteration, averaged over 3 seeds; standard deviations across seeds are at most :
| Setting | Precision | Recall |
|---|---|---|
| Initial model (shared) | 0.614 | 0.443 |
| 0.673 | 0.595 | |
| 0.642 | 0.577 | |
| 0.613 | 0.550 | |
| 0.590 | 0.501 | |
| 0.555 | 0.412 | |
| 0.618 | 0.549 | |
| 0.576 | 0.516 |
They support the FID interpretation. For , precision and recall both decrease ( and ): the model loses quality and diversity, the typical signature of collapse. For every , recall increases (precision decreases slightly for large ), and at the last iteration both metrics are ordered by . Additionally, for the two varying sequences, the schedule satisfying the criterion () ends above the one violating it (). We include this table in the revision (Appendix D, Table 1).
[1] Kynkäänniemi et al., "Improved Precision and Recall Metric for Assessing Generative Models", NeurIPS 2019.
Q2 — Generation of synthetic images, and analysis of class imbalance, repeated images and visible artifacts. We thank the reviewer for this question. We use DDIM with 200 steps and (no stochastic noise added during the diffusion steps) to sample from the model selected in the previous iteration. Sampling is therefore deterministic given the initial Gaussian noise, and every batch uses fresh noise with its own seed. Generated images are added to the pool without any filtering. To check whether class imbalance, repeated images or memorization could contribute to the degradation, we classify 3,750 samples per model with a pretrained ResNet-56 (94.4% CIFAR-10 test accuracy) and measure near-duplicates in Inception features, with distance thresholds calibrated to flag 1% of real held-out images (within a sample, and relative to the 50,000 training images). Values are at the first (shared by all settings) and last iterations, averaged over 3 seeds; standard deviations across seeds are at most for class entropy and largest-class share, and at most 0.4% for the near-duplicate and training-image columns:
| Setting | Class entropy | Largest class | Near-duplicates | Near a training image |
|---|---|---|---|---|
| Real test images | 0.9995 | 0.106 | 1% | 1% |
| Initial model (shared) | 0.971 | 0.159 | 0.5% | 0.1% |
| 0.996 | 0.122 | 1.3% | 0.2% | |
| 0.989 | 0.142 | 1.0% | 0.1% | |
| 0.990 | 0.140 | 0.8% | 0.1% | |
| 0.990 | 0.146 | 0.1% | 0.0% | |
| 0.984 | 0.155 | 0.1% | 0.0% |
Class entropy is normalized (1 = uniform). According to the pretrained classifier, the initial model (shared by all settings) is mildly imbalanced. With real data in every batch (), this imbalance is largely corrected, whereas for it changes only slightly. Near-duplicates stay close to or below the real-data level and do not increase with , and near-copies of training images are at most 0.2% for every setting. Hence, the degradation at is not driven by class imbalance, repeated images or memorization: its FID, precision and recall worsen over iterations, while its class imbalance changes only slightly and its duplicate rate is among the lowest of all settings. The uncurated samples (Q1) show no systematic visual artifacts. We include these checks and the sampling details in the revision (Appendix D, Table 2).
Response to Reviewer AWBC (3/3)
Q3 — Comparability of training settings, and prior retraining strategies. We thank the reviewer for this question. In the revised experiments (Appendix D), all models, including the real-only control (), are initialized and trained identically: every model in each iteration is trained from scratch with the same schedule, for at most 1M steps, using early stopping on a validation set. The previous warm-start training is removed. Hence, the comparison across mixing fractions does not depend on the training budget or initialization (see also our reply to W1).
Regarding prior strategies: our experiments are designed to test a prediction of our theory, namely the effect of the fraction of fresh real data when real and generated samples are pooled indistinguishably, rather than to propose a new method. Most stabilization strategies, such as filtering, verification or reweighting, require distinguishing real from generated samples, which our setting deliberately excludes. It is thus fair to leave such empirical investigations out of the scope of our theoretical paper.
Q4 — Variability across runs. We thank the reviewer for this question. In the revision (Appendix D, Figure 6), all experiments are repeated with 3 seeds under a unified protocol, which allows us to measure the run-to-run variation directly. At every iteration after the first, the gaps between consecutive values of exceed the standard deviations across 3 seeds by a factor of at least 3.5, indicating that the trends do not depend on the random seeds. The ordering of the mixing fractions holds at every iteration and in every seed. The late-stage behavior of each curve is also reproduced in every seed: for large , the FID levels off after a few iterations, while for the FID never improves but remains bounded.
Q5 — Qualitative or quantitative agreement, and the scope as the real-data fraction vanishes. We thank the reviewer for this question. The agreement should be interpreted qualitatively: we do not claim a quantitative correspondence in convergence rates, and we have removed from the revision the claims on the rates, the phase transition at , and the value of detection for diffusion models (Section 4 and Appendix D).
Regarding a vanishing fraction of real data, our theory gives the exact scope of the recovery claim: recovery (i.e., consistency) holds if and only if (Theorem 3.3). Hence, the fraction of real data may vanish, but only slowly. To examine this setting with diffusion models, we added two sequences with a vanishing fraction of real data: , for which our theory predicts consistency, and , for which it predicts collapse (i.e., bounded degradation). The FID of the former is below that of the latter at every iteration and in every seed. Within 9 iterations, this shows the predicted ordering, while the asymptotic separation between the two regimes is shown in the Gaussian experiments, which use the same sequences (Section 3.3, Figure 1(d)–(f)).
We hope that these answers and the revised paper address the reviewer's concerns, and we would be glad to answer any additional questions. We would be grateful if the reviewer would consider raising their score.
Response to Reviewer AWBC (1/3)
We thank the reviewer for their thorough review and valuable suggestions, and are pleased that they find our necessary and sufficient criterion and the joint treatment of the mean, covariance and fitted distribution informative, and the distinction between consistency and efficiency meaningful. We address their concerns below.
W1 — The diffusion evidence is limited relative to the broader claims.
We thank the reviewer for this comment. We agree: the diffusion experiments support qualitative trends rather than general conclusions about practical generative models, and we have toned down our claims accordingly in the revision (Section 4 and Appendix D). Our main result, consistency, is asymptotic in the number of iterations, and is verified under the theoretical setting (Figure 1). The number of iterations in the practical diffusion model experiments is limited by the dataset size. To strengthen the empirical evidence, we reran all experiments under a unified protocol and extended them:
All of these are included in the revision (Appendix D, Figures 6 and 7 and Tables 1 and 2; summary in Section 4).
W2 — Relation to closely related work. We thank the reviewer for pointing out these recent works [1, 2], the first of which is not yet peer-reviewed. We agree that our contribution should be positioned around the precise new results. In the revision, we add both works, briefly in the related work (Section 2) and in detail in a new appendix (Appendix F), and remove the broad "first-work" claims. In short:
[1] Wang, Niu & Li, "Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training", arXiv:2602.16065, 2026.
[2] Bakshi & Chakraborty, "From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources", NeurIPS 2026.