- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies recursive generative-model training through repeated density estimation and sampling. It compares replacement, accumulation, and partial retention of past data, arguing that retention is the main factor determining whether recursive learning remains stable. Experiments cover KDEs, GMMs, a normalizing flow, small autoregressive sequence models, and a VAE on digit images. The theoretical section derives moment recursions for recursive Gaussian estimation and compares them with simulations and selected VAE statistics.
Critical strengths
The retention-probability experiment is useful. Moving beyond the two endpoints of replacement and full accumulation gives a more informative picture of how memory affects recursive training. The consistent direction of the effect across several estimators is also worth documenting.
The paper makes an effort to connect controlled experiments with an analytically tractable model. Multiple trials, explicit sampling parameters, and comparisons with small neural estimators make the observations easier to assess than an isolated demonstration of degradation.
Critical weaknesses
Proposition 2 is not exact under the stated variance estimator. Let N_t = N_{t-1}+n. The pooled sum-of-squares identity used in Proposition 3 gives
\widehat s_{t-1}^2 \left(1-\frac{n}{N_t(N_t-1)}\right). ]
Thus the expected variance decreases multiplicatively and does not converge to the original variance as claimed. Correspondingly, the exact mean-variance expression must retain (\mathbb E[\widehat s_{i-1}^2]) inside the sum. The expression stated in Proposition 2 is an upper bound, not the claimed exact limit. This correction preserves boundedness, but exact recursions are a central contribution of the paper. The distinction between Bessel-corrected estimation and Gaussian MLE also needs to be corrected.
The empirical interpretation has a related gap. Repeated fixed-bandwidth KDE fitting introduces smoothing even in the population limit, so the observed broadening does not by itself identify finite-sample resampling noise as the dominant mechanism. Likewise, forty generations of increasing error do not establish asymptotic divergence for every positive real-data injection rate.
The novelty assessment omits a directly relevant prior study: Kazdan et al., “Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World,” ICML 2025, PMLR 267:29469–29494. That work already studies replacement and accumulation in Gaussian estimation, KDE, and language-model training, including an accumulation–subsampling workflow. The submission needs to establish what its retention sweep and moment calculations add beyond these results. As currently framed, much of the claimed statistical reinterpretation is already established, while the proposed new exact theory contains the issue above.
Questions for authors
Can you reconcile Proposition 2 with the conditional pooled-variance recursion above? Please provide the corrected exact limit and indicate which theoretical and numerical claims change.
What is the precise contribution beyond Kazdan et al. (ICML 2025), particularly for KDE, Gaussian estimation, and fixed-size subsampling of accumulated data?
What bandwidth rules are used in Sections 4–7? Can you separate finite-sample resampling error from repeated kernel smoothing using a population baseline and sample-size/bandwidth controls? What supports asymptotic divergence rather than a biased plateau under positive real-data injection?
Under the 4000-sample cap, are original real observations protected or randomly discarded? Can retention be compared at matched training sample sizes? Please also clarify whether “identical batches” means identical observations or shared random numbers.
What supports the “faster than the idealization’s worst case” interpretation of the VAE result? The propositions do not establish a worst-case bound for neural estimators. Please distinguish typical variance trajectories from ensemble moments and provide uncertainty estimates for the across-trial variance comparisons.
4 comments
Response (Cntd)
VAE "worst case" is withdrawn; the propositions do not bound neural estimators. With 95% bootstrap intervals over 20 trials: accumulation growth 1.17 [0.57, 3.09] vs. theory 1.20 (consistent, weak test; the VAE's also falls 0.47 to 0.16); replacement 1.68 [0.60, 5.67] vs. theory 9.26. collapses to about in 20/20 trials. The bias explanation is now a hypothesis.
Response to the Authors
Thank you for the detailed rebuttal and for carefully addressing my concerns. I appreciate the authors' corrections, additional control experiments, and clarification of the paper's claims. My main concerns have been largely addressed, although some limitations remain. I will therefore raise my rating.
Dear reviewer.
Thanks a lot for taking the time to go through our reply. It seems you may have forgotten to raise your rating. Best,
Response
Comment 1: Proposition 2 and the MLE label (you are right)
Thank you for the careful reading. We confirmed both points.
Proposition 2. The pooled sum of squares gives E[s^t2∣Ft−1]=s^t−12(1−Nt(Nt−1)n), your expression. The exact statements are E[s^t2]=σ02∏i≤t(1−Ni(Ni−1)n) and Var(μ^t)=n0σ02+∑i≤tNi2nE[s^i−12]. The limit displayed in Prop. 2 is an upper bound, and E[s^t2]→σ02Π∞, not σ02. For n0=2000, n=500: Π∞≈0.99956 and limVar(μ^t)≈0.000942 (bound 0.000943).
What changes. The statement of Prop. 2, the claim E[s^t2]→σ02 in the text and proof, and the "MLE" label. What does not. Boundedness, Prop. 1, and Prop. 3, whose product form already gives the correct E[s^t2]. Our code used the exact Qt-recursion, which matches the product to seven digits, and the correction (0.04%) is far below Monte Carlo resolution, so no figure or comparison moves.
MLE. We analyze the sample mean with the Bessel-corrected variance, which is not the MLE. The setting is now called "Gaussian estimation with an unbiased variance" throughout.
Comment 2: KDE smoothing, injection rate, and Section 6 controls (you are right)
Bandwidth. Section 3 uses fixed bandwidths (0.5, 0.1); Sections 4-7 use
scipy.stats.gaussian_kde(Scott's rule), refit every generation. A KDE refit to its own samples broadens the density even with infinitely many samples (Scott's rule multiplies the variance by 1+n−2/5 per generation), so the KDE results alone do not identify sampling noise.Population baseline. The infinite-sample recursion pt+1=qt∗N(0,n−2/5Var(qt)), with no sampling, reproduces our results:
It also reproduces Table 3. So the KDE experiments measure deterministic smoothing bias, and the paper no longer claims otherwise (new Sec. 3.2).
Sampling-noise control. A 1-D two-component GMM fitted by EM contains p0, so any drift is sampling noise (15 trials). KL after 25 generations: 0.155 (replacement), 0.021 (size-matched), 0.002 (accumulation). The retention ordering holds against noise, and the size-matched control is clean.
Injection rate (your 40-generation point). Withdrawn. In the KDE, forgetting with injection α diverges exponentially below α∗=n−2/5/(1+n−2/5)≈0.077 and plateaus above it, a bandwidth-rule artifact. With noise only (GMM, T=40), α=0 keeps growing (KL 0.074, 0.135, 0.225 at generations 10, 20, 40), but α=0.05 plateaus (0.039, 0.038, 0.044). We no longer claim "no safe α".
Section 6 controls (8 trials, final KL at α = 0.02 / 0.05 / 0.10).
For the KDE the retention advantage survives size-matching. For the GMM, sample size explains most of the gap, about 5x remains at small α, and at α=0.1 matched forgetting is indistinguishable from accumulation. We have qualified the paper's claim accordingly. The leaky-memory sweep repeated with the GMM is also smooth but roughly geometric rather than concave, so "concave" is removed as a general claim.
Comment 3: Kazdan et al., the cap, "identical batches", and the VAE
Kazdan et al. (ICML 2025) and three related papers are now cited, and the abstract, Introduction and Related Work are rewritten. We no longer claim novelty for the replace/accumulate separation, the KDE replication, or the 1/N2 mechanism for the mean. What we did not find in that work: the fixed-α retention comparison, leaky-memory interpolation, the exact product form for Var(s^t2), and the recursion-vs-neural-estimator test. Our cap in Sections 6-7 is accumulate-subsample; leaky memory generalizes it.
The cap. It is a uniformly random subsample of the whole pool, so real observations are not protected (control above).
"Identical batches" was inaccurate: the arms use different seeds and their synthetic batches come from different models, so they are matched in size and injection rate only. Corrected in Section 6 and the Reproducibility Statement.