acceptodds
Under review as a conference paper at ICLR 2027

Understanding Mode Collapse in Diffusion Models under Classifier-Free Guidance

Abstract

The use of classifier-free guidance in diffusion and flow models has proven critical to their empirical success, as it allows for the controllable generation of lower-temperature and higher-fidelity samples. Despite its widespread use, classifier-free guidance remains both empirically and theoretically poorly understood, owing to the difficulty of solving the associated transport equations for arbitrary normalizing flows. In this paper, we observe that the effects of classifier-free guidance arise from a mismatch between the representation capabilities of the conditional and unconditional distributions, wherein the conditional distribution exhibits higher complexity and fidelity. We propose an analytical model to capture this observation: the conditional distribution is a Gaussian mixture whose means lie on an ellipsoid, and the unconditional distribution is a single Gaussian approximating the mixture. The flows arising from this mixture are closed under scaling, enabling us to derive analytic closed-form solutions for the guided distribution at all noise scales and guidance levels. Our analytic solution shows that classifier-free guidance induces (a) a shift along a mean-axis direction and (b) a persistent skew toward, and concentration around, a singular point corresponding to the conditional sample starting from zero noise. We validate our theoretical predictions on pretrained, large-scale image diffusion models.

Primary area:
generative models
Keywords:
diffusion, classifier free guidance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wuWFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new model for studying classifier-free guidance. Instead of modelling the unconditional data-generating distribution p0(x)p_0(x) as a mixture of Gaussian with the conditional distributions p0(x∣c)p_0(x\mid c) as mixture components, they reverse the modelling assumption and assume that p0(x∣c)p_0(x\mid c) is a mixture of Gaussian and p0(x)p_0(x) Gaussian; both density are tied together through a common-radius conditioning property. The author study the resulting classifier-free flow and derive explicit expression for the time and scale dependent probabilities. The theory predicts two phenomena observed in practice: The exaggeration of class-specific features and the loss of sample diversity. Predictions are verified on pre-trained latent image diffusion models.

Critical strengths

Theorem 3.4 derives closed form solutions for the time- and scale dependent probability flow of the underlying model. This form predicts the behavior of classifier-free guidance for class-conditional strength γ>1\gamma>1. It offers a potential explanation for mode-collapse in classifier-free guidance: samples of the vector field vγ,tv_{\gamma,t} concentrate around solutions of the corresponding ODE with initial value 00 (the mean value of the noise distribution p1p_1). Theoretical predictions are supported by experiments with pre-trained latent image diffusion models for both text and class conditioning.

Critical weaknesses

Theoretical toy models are important to understand behavior and make predictions. The authors did a great job in understanding current behavior but concrete predictions are only touched upon. However, I am not familiar with the specific literature and cannot assess the paper's contribution into the field properly.

Questions for authors

Questions whose answers could materially affect my assessment:

  1. Your theory nicely explain mode-collapse towards yγ,0(0)y_{\gamma,0}(0). Can this be exploited to guide the collapse to a specific sample beyond the one corresponding to yγ,0(0)y_{\gamma,0}(0)?

  2. Does your theory make any predictions which could be useful for practioners? Like an optimal γ\gamma subject to a specific objective?

Clarifying Questions:

  1. Figure 3 actually supports the classicial assumption (p0p_0 is a mixture and the conditional one is a component of it). Can you comment on it?

  2. In the experiments, you choose p(x∣c)p(x|c) based on K-means clustering and μˉ\bar{\mu} the sample mean. Does this fulfill Definition 3.1? If not, please comment. If yes, please provide prove.

  3. Could you quantitatively measure the observed concentration around yγ,0(0)y_{\gamma,0}(0)?

  4. You say that you confirm the degree of mode collapse is bounded in the experiments. I don't quite see that, can you clarify please? I see the that the ske coefficient saturates (Figure 5) but this is fidelity not collapse?

Here some minor questions/remarks:

  • Line 140: you write "(2.5) simlpifies to the ODE" but (2.5) is a reverse ODE which is slightly confusing and is resolved only by the "integrated from t=1t=1 toward t=0t=0" addition. I wonder with stating (2.5) in the same manner (from t=1 to t=0) avoids this confusion and does not require adding the reverse subscript which is not used in the paper

  • Secion 3: I suggest sticking to p0p_0 for the data distribution throughout (line 198, Def 3.2, line 454 and 458). In any case: I would not use pp for Definition 3.1; better use qq.

  • Proposition 3.3: a common-radius mixture is a density not a random variable as by definition 3,2

  • Property (1), line 261 is true for any γ\gamma not only for γ>1\gamma>1, isn't ? Is it fair to say that this essentially says: guidance provides fidelity in general?

  • Line 263: "...the resulting scaled distribution " p and not p^\hat{p}?

  • Corollary 3.5: your choices of p(x∣c)p(x|c) and p(x)p(x) do not fulfill 3.2. Please comment

  • Line 351: subscript of yy omitted

  • Theorem 3.7: what is R?

  • line 429: subscript for time missing for pp in the expectation.

  • Why did you stop at γ=12\gamma=12?

  • Line 430-431: How does yγ,0(0)y_{\gamma,0}(0) relate the x∼pγ,0(x∣c)x\sim p_{\gamma,0}(x|c)? I think it requires a justifying comment. I appreciate the Figure in the appendix though.

  • I was struggling to understand the x-axis of Figure 5 and 6. The way I understood it: take each image of cluster c, project towards random or principal direction, figure 5 and 6 display the resulting distributions. For high γ\gamma, cluster becomes more aligned with this direction. Then x-axis measure alignment with direction and is expressed in units of cluster-SD?

2 comments

We thank the reviewer again for the time and effort that went into their review. The author-reviewer discussion period ends November 18th. If our answers have sufficiently addressed your concerns, please consider updating your score. If there is anything that we have not addressed or you have additional questions, please be sure to respond before the deadline.

Thank you for your response. My questions were well addressed and I will consider this in my final assessment.

Reviewer_yjPnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to establish a mathematical model and explanation for the behaviour of classifier-free guidance, and establishes analysis on mode collapse and the feature magnification for CFG. The paper establishes a simplification of transport velocity based on some assumptions, and derives the probability density after CFG is performed. Analytical solution for the density is established in special cases, and quantification of"collapse" is derived for more generalized situation. Authors also show experiments that support their theoretical prediction.

Critical strengths
  1. The paper proposes an analytic model that tries to quantify the effect of collapse and in general how and why CFG works. I have no expertise in this particular problem at all, but some theoretical results are still interesting to me.
  2. The paper provides empirical evidence that matches their theoretical predictions. Very useful to decide whether further real-world behaviour should be explained by the proposed model.
Critical weaknesses
  1. The paper in general has less than optimized presentation. Something like a roadmap that helps preview the analysis is immesely helpful. I personally couldn't really tell the centerpiece is essentially Theorem 3.4 until I actually finished the whole paper.
  2. The paper relies on the assumption that conditional and unconditional distributions all share the same covariance. I understand the technical limitation without such assumption, but especially the part where the unconditional distrubiton has the same covariance as the conditional is too limiting. I appreciate the related author's statement in the limitation on this.
  3. At the current presentation, the analysis carries no insight that can be used to improve conditional generation. I understand that this is likely not the focus, but practicality is still limited.
Questions for authors
  1. Is it possible for authors to provide analysis, either analytical or empirical, when we relax the assumption to allow unconditional distribution to have a constantly scaled covariance? From my limited probing it seems like empirical analysis using ODE is possible for this situation.
  2. Theorem 3.4 suggests a mismatch between noise scaled by γ\gamma, and effort has been spent to analyze the mismatch. Have the authors considered sampling initial noise from a normal distribution with γI\gamma \mathbf I variance instead? The results would be very indicative of the theoretical model proposed in the paper, provided the actual generator is not too sensitive to the noise scale.
  3. The RHS bound of Theorem 3.4 seems vacuous for γ\gamma that matters. Can the authors provide their insight on this?
  4. What are the practical guidelines that can be extracted from the theoretical model? Any insight from the authors is appreicated.
1 comment

We thank the reviewer again for the time and effort that went into their review. The author-reviewer discussion period ends November 18th. If our answers have sufficiently addressed your concerns, please consider updating your score. If there is anything that we have not addressed or you have additional questions, please be sure to respond before the deadline.

Reviewer_1gAGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies mode collapse in diffusion conditional sampling (classifer-free setting). It assumes the underlying conditional is Gaussian mixture with common Mahalanobis radius and shared covariance, and the unconditional is simply Gaussian. Under this setting, the authors show that the guided density with some γ\gamma velocity is the mixture p^γ,t(x∣c)\hat p_{\gamma, t}(x|c), whose means and covariance are scaled by γ\gamma and reweighted by some WW.

Then the authors proceed to show the analytical form of WW for the 1-D case, and generalizes it to n-dimensional case under some very specific conditions. Lower and upper bounds on WW are also provided.

For experiments, the authors use Stable diffusion 1.5 and Flux-Klein 9B. The experiments project clusters along the direction of samples mean. Then demonstrate that the guidance effect is most prevalent along this direction.

Critical strengths

The theory in general is nice and fresh. While the setting is rather simple (unconditional Gaussian and conditional mixture Gaussian), the result is clean and is, in my opinion, novel. I find Theorem 3.7 most interesting to me as it potentially shows how this guidance behaves at the long tail.

Critical weaknesses

My main criticism is with regard to the paper title/framing and the experiment setup. The paper title is about understanding mode collapse with classifier-free guidance, however, while the math is sound, I'm not certain that it actually explains the mode collapse well. The authors construct a very specific scenario where these theorems and bounds happen, whether or not this is close to be true in practice remains open, for example, the shared covariance in the conditional density setup is likely problematic. I also think it is quite misleading to say the result is closed-form when:

    1. It's only true for 1-D
    1. It's only available for n-D under a very strict setting in an already a very specific scenario.

Moreover, the experiments also do not provide strong evidence to support the paper framing:

  • In Figure 5, this experiment doesn't prove anything. Do other CFG models observe the same result or not?. And for the random control, since it's defined orthogonal to the direction, won't it make any mean shift 0?
  • Also in Figure 5, Theorem 3.4 predicts the shift will grow approximately linearly with γ\gamma, which seems to only be true for SD 1.5.
  • Similarly for Figure 7, especially the bottom one, I'm not sure it's proving the paper's point. To me it's showing that the guidance is working, not that it's collapsing to something. Even if the claim is true, showing the zero-initialized sample doesn't prove distributional collapse.
  • Figure 6 is better, but it still doesn't show that other guidance models don't produce the same skewness.
Questions for authors
  1. I probably need some clarification on Figure 5 & 7 experiments. Please refer to the Weakness list above.
  2. How does Theorem 3.4 actually explains mode collapse in practice? Since this phenomenon can also depend on training setup and discretization scheme.
2 comments

We thank the reviewer again for the time and effort that went into their review. The author-reviewer discussion period ends November 18th. If our answers have sufficiently addressed your concerns, please consider updating your score. If there is anything that we have not addressed or you have additional questions, please be sure to respond before the deadline.

I appreciate the authors' comments. While it addresses most of my questions, I'm keeping the rating the same as it doesn't substantially change the story for a strong accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.