acceptodds
Under review as a conference paper at ICLR 2027

Generative collapse does not imply discriminative collapse

Abstract

Recursive training on synthetic data can lead to model collapse, in which learned generative distributions progressively lose diversity or degenerate across training generations. In this paper, we ask whether such generative degeneration necessarily deteriorates the downstream task performance. We distinguish generative collapse, meaning collapse of the learned data distribution, from discriminative collapse, referring to degradation in performance on downstream tasks such as classification or regression. We theoretically investigate this distinction in a solvable setting: the iterative maximum likelihood estimation (MLE) for the symmetric binary Gaussian mixture model under the discard workflow, where each generation is trained only on synthetic samples from the previously fitted model. In line with a widely known covariance collapse phenomena under the multivariate Gaussian distribution, the fitted covariance matrix also vanishes to zero almost surely, so the trained generative model becomes degenerate under the symmetric binary GMM. Our main finding is that this generative collapse does not necessarily imply collapse of the performance on the downstream classification task. Although the covariance collapses, the fitted mean vector converges almost surely to a non-degenerate random vector that retains directional information about the population signal. Consequently, the regularized plug-in linear discriminant analysis (LDA) classifier has the limiting expected risk strictly smaller than , i.e., it is superior to random guessing. Moreover, in the high-SNR regime, its limiting expected risk approaches to the Bayes risk at a quadratic rate keeping the condition number of the ground truth covariance matrix constant. We further show that within the large-sample regime as , the terminal mean vector converges weakly to the multivariate -distribution with degrees of freedom , centered at the true mean vector. These results show that generative collapse does not always lead to discriminative collapse.

Primary area:
generative models
Keywords:
generative model, model collapse, LDA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_F2ZpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether generative collapse under repeated training on synthetic data necessarily reduces downstream classification performance to random guessing. The authors consider a symmetric binary Gaussian mixture model, where each generation is trained only on samples generated by the previous model. They show that the estimated covariance converges to zero, while the estimated mean converges to a non-degenerate random vector. Despite the covariance collapse, the limiting expected risk of a regularized plug-in LDA classifier remains strictly below random guessing. The paper also provides an upper bound on additive excess risk at high SNR. After taking infinitely many training generations, the terminal mean converges in distribution to a multivariate Student's tt distribution with four degrees of freedom as the per-generation sample size increases. Simulations illustrate the classification results under different covariance structures and the distribution of the terminal mean.

Critical strengths
  • The main strength is the clear distinction between generative collapse and classification performance. The paper gives an explicit setting in which the learned distribution collapses while the resulting classifier still performs better than random guessing in expectation. This result holds without retaining real data or using an external verifier. For a nonzero population mean, Theorem 4.1 allows arbitrary positive-definite population covariance matrices and finite sample sizes satisfying n≥d+1n \geq d+1. Preserving the mean in expectation alone would not establish this classification guarantee. I therefore find the result a useful contribution to understanding what information can survive repeated training on synthetic data.

  • The analysis also explains why classification information can survive. As the covariance decreases, the mean updates become less noisy, allowing the estimated mean to stabilize. The comparison with the fixed-covariance training process helps explain the role of this mechanism. In addition, the multivariate t4t_4 limit characterizes the estimation error that remains after infinitely many generations. It shows that subsequently increasing the sample size does not remove the randomness of the terminal mean. These results help explain both the preservation of classification information and its limitations.

Critical weaknesses
  • The quantitative interpretation of the classification guarantee could be developed further. Theorem 4.1 establishes that the limiting expected risk is below 1/21/2, and Theorem 4.2 provides an excess-risk bound that becomes informative at sufficiently high SNR. Figure 1 complements these results by showing that the remaining advantage over random guessing depends substantially on the covariance structure. However, the experiments keep the SNR fixed, leaving the connection between the observed performance at moderate SNR and the high-SNR guarantee less clear. A discussion of this connection would help readers understand when the surviving information supports substantial classification performance. This is a limitation of the quantitative interpretation rather than a concern about the validity of the main separation result.
Questions for authors
  • Theorem 4.2 provides an excess-risk bound with dependence on dimension, covariance conditioning, and SNR. How should this bound be interpreted alongside the results in Figure 1, particularly at moderate SNR? Based on the current analysis, what can be said about the size of the advantage over random guessing in this regime? An explanation for an isotropic or one-dimensional setting would help clarify when the advantage is substantial and when it may be small.
2 comments

We thank the reviewer for the useful and insightful question. The comparison between Theorem 4.2 and Figure 1 should be understood with the covariance structure, and hence its condition number, held fixed within each panel. For fixed dimension and condition number, Theorem 4.2 gives

= \mathcal O\left( \frac{d\kappa(\Sigma_\star)^2}{\Delta_\star^2} \right), \qquad \text{where }\overline R_n:= \lim_{T\to\infty} \mathbb{E}\left[\mathcal R \\{\widehat f_T^\delta(n)\\} \right].$$ Thus, increasing the SNR while keeping the covariance structure fixed makes the limiting expected risk approach the Bayes risk. Figure 1 illustrates the evolution across training generations at the fixed, moderate SNR $\Delta_\star=1.5$. Its plateaus correspond to $\overline R_n$, and their non-negligible separation from the Bayes risk is consistent with Theorem 4.2, whose near-Bayes guarantee concerns the high-SNR regime. Regarding the size of the advantage over random guessing, define $$A_n:=\frac12-\overline R_n.$$ Theorem 4.1 guarantees $A_n>0$ for every nonzero population signal. However, the general bound does not necessarily certify a substantial advantage at moderate SNR. More explicitly, the proof of Theorem 4.2 gives $$A_n \ge \frac12 -\Phi\left( -\frac{\Delta_\star}{3\kappa(\Sigma_\star)} \right) -\frac{4d\kappa(\Sigma_\star)^2}{\Delta_\star^2}.$$ At the parameters of Figure 1, $d=10$ and $\Delta_\star=1.5$, the final term is at least $40/2.25\approx17.78$, even for isotropic covariance. Consequently, this bound is vacuous for quantifying the advantage in these experiments. The observed plateaus suggest advantages of approximately $0.07$ to $0.29$ across the displayed configurations, but these are numerical observations rather than quantitative guarantees supplied by Theorem 4.2. We will include a one-dimensional case study, which is a special case of isotropic covariance, to clarify when the advantage is substantial and when it may be small. In this setting, let $\Sigma_\star=\sigma^2$, assume $\mu_\star>0$ without loss of generality, and write $p_n=\mathbb{P}\\{\widehat\mu_\infty(n)<0\\}$. The limiting classifier either agrees with the Bayes classifier or reverses its labels, giving the exact identity $$A_n = \big[\Phi(\Delta_\star)-\frac{1}{2}\big](1-2 p_n).$$ This separates the effect of class separation from the probability of retaining the correct signal direction. In particular, a substantial advantage requires both factors to be appreciable, while $$0<A_n\le\Phi(\Delta_\star)-\frac12 \longrightarrow0 \qquad\text{as }\Delta_\star\downarrow0.$$ We will include this calculation and quantitative examples in the revised manuscript to clarify the distinction between strictly positive and substantial improvement over random guessing.

Thank you for the helpful clarification. The response addresses my questions, and I will maintain my rating of weak acceptance.

Reviewer_p2SVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether generative degeneration necessarily leads to a deterioration in downstream task performance when models are trained recursively on synthetic data. The theoretical analysis focuses on Gaussian mixture models (GMMs). The main finding is that generative collapse does not necessarily imply a corresponding collapse in downstream classification performance. Although the covariance matrix collapses, the estimated mean vector converges almost surely to a non-degenerate random vector that preserves directional information about the population signal. The authors also conduct experiments to validate their theoretical findings.

Critical strengths
  1. The paper addresses an important topic, as developing a deeper understanding of the relationship between model collapse and generative degeneration is valuable.

  2. The theoretical analysis is comprehensive and appears to be sound.

Critical weaknesses
  1. The theoretical analysis focuses on GMMs, and it is unclear whether the results can be extended to more general and realistic settings.

  2. It is unclear how the proof techniques differ from those in existing work and what the main technical challenges are in establishing the theoretical results.

Questions for authors

See Weaknesses.

1 comment

We thank the reviewer for the useful and insightful comment.

  • To answer your first question, we use the symmetric binary GMM to study whether generative collapse necessarily implies classification collapse. Its closed-form MLE and classification risk allow us to isolate the effect of recursive sampling from optimization error and model misspecification. The model reveals a specific mechanism: covariance collapse reduces the noise in subsequent mean updates, allowing information relevant to classification to survive. Our results establish this separation within the GMM setting. The quantitative guarantees rely on Gaussian-specific properties, and extensions to non-Gaussian distributions or nonlinear classifiers require further analysis. We will clarify this motivation and scope in the revision.

  • To answer your second concern, covariance collapse and martingale methods already appear in earlier work, including Alemohammad et al. (2024) and Bertrand et al. (2024). Our additional analysis addresses three issues:

    • Classification after collapse. Unbiasedness of the terminal mean does not imply risk below 1/21/2. We derive its Gaussian scale-mixture representation and use conditional sign comparisons to establish a strict classification advantage (Lemma D.1 and Theorem 4.1).

    • Quantitative excess risk. We control the normalized classification margin despite persistent randomness in the terminal mean. Spectral and second-moment bounds yield the dependence on dimension, covariance conditioning, and SNR in Theorem 4.2.

    • The terminal distribution. For Theorem 5.1, we analyze the accumulated noise through a random perpetuity. A functional central limit theorem handles bounded time intervals, while uniform tail control justifies passage to infinitely many generations, yielding the Student-tt limit.

We will explain these steps after the corresponding results and distinguish them from the established covariance-collapse arguments.

Reviewer_iyuCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines the differences between model collapse in the generative sense (where generative modeling diverges from the real distribution) and the discriminative sense (where the statistics obtained via MLE may still be useful for classification). Overall, the observation in the paper is interesting, but there are several issues I believe to be major (see below).

I want to emphasize that my score is largely based on my belief that the revisions that the paper would need to resolve the issues I raised may be significant, and warrant a new review process. I do however believe that an edited version of this paper can be accepted in the future.

Critical strengths
  • The difference between generative and discriminative collapse is interesting in my opinion, and I think it would interest some people in the community.
  • I didn't check proofs too closely, but at least at a glance, everything seems correct and sensible.
  • Overall the paper is clear and easy to follow.
Critical weaknesses
  • Essentially all results up to and including section 3 are either very well known results, or immediate consequences of well known results. These formula for the MLE, the (1−1/n)T(1-1/n)^T rate and the Wishart distribution formulation are all well known. Moreover, because the authors are assuming that the learner sees the labels, the mixture MLE that the authors analyze is actually equivalent to standard Gaussian MLE on zi:=yixiz_i := y_i x_i.

  • The framing around theorem 4.2 is unjustified and misleading, and in many cases of interest, the bound on the risk that it establishes (in equation 5.4) is vacuous (meaning not better than a trivial predictor). There is generally no reason to expect Δ∗→∞\Delta_* \to \infty, especially in the κ=O(1)\kappa = O(1) regime that the theorem assumes. It is not true that the risk “approaches the Bayes risk” in the way that most people would interpret this statement (a theorem about asymptotics in the number of samples nn). The bound in equation 4.5 does not approach 00 as n→∞n\to\infty, and in many cases it is completely trivial and bigger than 11. For example, in equation 4.5, if we take Σ=I\Sigma = I, then the risk bound is O(d/∥μ∥2)\mathcal O(d / \lVert \mu\rVert^2).

  • I worry that in perhaps the most natural MLE cases, the entire narrative of the paper is misleading, and the MLE does suffer from collapse, in the sense that its improvement over random guessing is almost 0. The paper does not demonstrate (at least explicitly) a non-negligible advantage of the risk beyond random guessing. E.g. continuing from the previous point, In the natural setting of Σ=I\Sigma = I and ∥μ∥=1\lVert \mu\rVert=1, theorem 4.2 is vacuous, and theorem 4.1 only says the risk is ,1/2, 1/2, but does not rule out that it is >1/2−od(1)>1/2 - o_d(1).

Less significant weaknesses

  • The use of Theorem 5.1 is very strange. On one hand, the authors have a formula for the asymptotic predictor, but the authors don’t actually use that formula in any way, and don’t show us what this means for the excess risk in the n→∞n\to \infty limit. My recommendation, is that it should be presented earlier (at the end of section 3), and the asymptotic result for the Bayes risk should be presented in section 4.
  • There is some AI-style writing that would benefit from editing. E.g. the first paragraph in section 3
Questions for authors
  • What is the asymptotic risk implied by theorem 5.1, and how does it compare to that of Theorem 4.2?
  • What conditions are needed to ensure a that the excess risk is meaningfully bounded away from 1/21/2?
1 comment

We thank the reviewer for the useful and insightful comments.

  • To answer your first question, Theorem 5.1 characterizes the asymptotic distribution of the terminal fitted mean vector: μ^∞(n)⇒M,M∼td(μ⋆,Σ⋆2,4).\widehat\mu_\infty(n)\Rightarrow M, \qquad M\sim t_d\left(\mu_\star,\frac{\Sigma_\star}{2},4\right). To obtain the corresponding risk limit, we combine this result with equation (4.2). Define
\right),\qquad m\ne 0.$$ Since $r$ is continuous away from zero and $M$ has no atom at zero, the continuous mapping theorem gives $R_\infty(n)\Rightarrow r(M)$. Boundedness of the risk then yields $$\lim_{n\to\infty}\mathbb{E}[R_\infty(n)] = \mathbb{E}[r(M)] =:\overline R_\infty.$$ This is a consequence of Theorem 5.1, rather than its direct statement. The order of limits is first $T\to\infty$, then $n\to\infty$. We will state this implication explicitly. - To answer your second concern, Theorem 4.1 establishes $\mathbb{E}[R_\infty(n)]<1/2$ for any nonzero population signal under the stated assumptions. Thus, the limiting expected classification risk has a positive gap below random guessing. However, we are unsure what is intended by ``meaningfully'' bounded away from $1/2$. Could the reviewer clarify whether this refers to an explicit quantitative lower bound on that gap?
Reviewer_nCDqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the discard-workflow of iterative MLE for the symmetric binary Gaussian mixture model. In this setting, the fitted covariance matrix converges to zero, and hence the learned generative distribution becomes degenerate. In contrast, the fitted mean vector converges to a nondegenerate d-dimensional random vector, rather than being forced to vanish together with the covariance matrix. The covariance collapse makes the learned generative distribution degenerate, but it also reduces the conditional noise in later updates of the fitted mean vector.

Critical strengths

The authors provide rigorous proofs showing under the discard-workflow of iterative MLE for Gaussian mixture model, although the covariance matrix converges to zero the fitted mean vector converges to a nondegenerate d-dimensional random vector.

Critical weaknesses
  1. One main weakness regarding the paper's novelty is the high correlation of Theorem 3.1 and its proof, with Theorem 3.4.1 of the thesis paper:"Context Collapse: In-Context Learning and Model Collapse" (https://arxiv.org/abs/2601.00923), which authors did not cite. Both theorems show that under the discard workflow and Gaussian distribution assumption, the estimated mean converges to a finite random limits.

  2. One additional weakness is that the discarded workflow regime is also considered already in Section 3.4 of the same thesis paper. So the formulation itself does not seem to be new as well.

Questions for authors
  1. Could the authors illustrate the relation between the current manuscript and thesis paper:"Context Collapse: In-Context Learning and Model Collapse" (https://arxiv.org/abs/2601.00923) and add the citation?

  2. Could the authors add more detailed discussions after relevant propositions and theorems so as to highlight what's the main technical challenges and significance compared to existing works?

1 comment

We thank the reviewer for this reference. We will cite Ott (2026). Section 3.4 of the thesis uses Gaussian-Wishart dynamics and martingale arguments to establish covariance collapse and almost-sure convergence of the fitted mean. Our analysis shares these tools and studies the classification performance retained after collapse, establishing a strict advantage over random guessing and a high-SNR excess-risk bound.

A technical difference is that the thesis uses unbiased covariance estimation, while our MLE uses normalization 1/n1/n. The resulting contraction in expected covariance gives L2L^2 convergence of the mean at fixed sample size. We also derive the terminal mean's large-sample Student-tt distribution using a perpetuity representation, a martingale functional central limit theorem, and uniform tail control.

We will add discussions after the relevant results to clarify these connections and technical steps. For Theorems 4.1 and 4.2, we will explain how the Gaussian scale-mixture representation, conditional sign comparisons, and normalized-margin bounds yield the classification guarantees. For Theorem 5.1, we will explain why uniform tail control is needed to pass from finite-time convergence to the terminal-vector limit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.