- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a new model for studying classifier-free guidance. Instead of modelling the unconditional data-generating distribution as a mixture of Gaussian with the conditional distributions as mixture components, they reverse the modelling assumption and assume that is a mixture of Gaussian and Gaussian; both density are tied together through a common-radius conditioning property. The author study the resulting classifier-free flow and derive explicit expression for the time and scale dependent probabilities. The theory predicts two phenomena observed in practice: The exaggeration of class-specific features and the loss of sample diversity. Predictions are verified on pre-trained latent image diffusion models.
Critical strengths
Theorem 3.4 derives closed form solutions for the time- and scale dependent probability flow of the underlying model. This form predicts the behavior of classifier-free guidance for class-conditional strength . It offers a potential explanation for mode-collapse in classifier-free guidance: samples of the vector field concentrate around solutions of the corresponding ODE with initial value (the mean value of the noise distribution ). Theoretical predictions are supported by experiments with pre-trained latent image diffusion models for both text and class conditioning.
Critical weaknesses
Theoretical toy models are important to understand behavior and make predictions. The authors did a great job in understanding current behavior but concrete predictions are only touched upon. However, I am not familiar with the specific literature and cannot assess the paper's contribution into the field properly.
Questions for authors
Questions whose answers could materially affect my assessment:
-
Your theory nicely explain mode-collapse towards . Can this be exploited to guide the collapse to a specific sample beyond the one corresponding to ?
-
Does your theory make any predictions which could be useful for practioners? Like an optimal subject to a specific objective?
Clarifying Questions:
-
Figure 3 actually supports the classicial assumption ( is a mixture and the conditional one is a component of it). Can you comment on it?
-
In the experiments, you choose based on K-means clustering and the sample mean. Does this fulfill Definition 3.1? If not, please comment. If yes, please provide prove.
-
Could you quantitatively measure the observed concentration around ?
-
You say that you confirm the degree of mode collapse is bounded in the experiments. I don't quite see that, can you clarify please? I see the that the ske coefficient saturates (Figure 5) but this is fidelity not collapse?
Here some minor questions/remarks:
-
Line 140: you write "(2.5) simlpifies to the ODE" but (2.5) is a reverse ODE which is slightly confusing and is resolved only by the "integrated from toward " addition. I wonder with stating (2.5) in the same manner (from t=1 to t=0) avoids this confusion and does not require adding the reverse subscript which is not used in the paper
-
Secion 3: I suggest sticking to for the data distribution throughout (line 198, Def 3.2, line 454 and 458). In any case: I would not use for Definition 3.1; better use .
-
Proposition 3.3: a common-radius mixture is a density not a random variable as by definition 3,2
-
Property (1), line 261 is true for any not only for , isn't ? Is it fair to say that this essentially says: guidance provides fidelity in general?
-
Line 263: "...the resulting scaled distribution " p and not ?
-
Corollary 3.5: your choices of and do not fulfill 3.2. Please comment
-
Line 351: subscript of omitted
-
Theorem 3.7: what is R?
-
line 429: subscript for time missing for in the expectation.
-
Why did you stop at ?
-
Line 430-431: How does relate the ? I think it requires a justifying comment. I appreciate the Figure in the appendix though.
-
I was struggling to understand the x-axis of Figure 5 and 6. The way I understood it: take each image of cluster c, project towards random or principal direction, figure 5 and 6 display the resulting distributions. For high , cluster becomes more aligned with this direction. Then x-axis measure alignment with direction and is expressed in units of cluster-SD?
2 comments
Thank you for your response. My questions were well addressed and I will consider this in my final assessment.
We thank the reviewer again for the time and effort that went into their review. The author-reviewer discussion period ends November 18th. If our answers have sufficiently addressed your concerns, please consider updating your score. If there is anything that we have not addressed or you have additional questions, please be sure to respond before the deadline.