acceptodds
Withdrawn from ICLR 2027

Disentangling Density from Semantics in Classifier-Free Guidance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Classifier-Free Guidance (CFG) is essential for conditional diffusion and flow-matching models. However, operating at high guidance scales, where prompt alignment is strongest, causes two distinct failure modes: visual oversaturation (color burnout) and an unintended coupling between prompt alignment and sample density, which forces trajectories into high-density mode centers and suppresses fine structural details. Because the clean image estimate and noise prediction directions are not aligned during generation, previous methods that correct only one of these two directions leave the other distortion uncorrected. To resolve both issues simultaneously, we introduce _Density-Decoupled CFG_ (-CFG), a training-free framework that projects guidance updates orthogonally to both directions. By decoupling semantic steering from typicality control, -CFG mitigates density drift into mode centers. This preserves the high-dimensional typical-set structure of the latent space, generating highly detailed images without the color oversaturation induced by standard high guidance. This allows increasing guidance without visual burnout and gives independent, calibrated control over sample density via a quantile parameter . Across SDXL and SD3.5, -CFG consistently achieves superior text alignment, natural contrast, and improved distribution fidelity when using high guidance.

Primary area:
generative models
Keywords:
Classifier Free-Guidance, Diffusion models, Density control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DB1SOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the coupling between semantic guidance, sample density, and image amplitude in classifier-free guidance (CFG), and proposes D2-CFG, a training-free modification that aims to disentangle these effects through geometric projection and explicit density control. The method is evaluated on synthetic distributions and text-to-image generation benchmarks, where the authors report improved stability and reduced saturation in high-guidance regimes while maintaining prompt alignment.

Critical strengths
  1. Clear geometric perspective on CFG failure modes: The paper provides a relatively clean decomposition of CFG into components associated with density change and amplitude growth, which helps organize two commonly observed high-guidance failure modes within a unified geometric framework.

  2. Simple training-free modification: The proposed method only modifies the inference-time guidance rule and does not require additional training or auxiliary networks. The additional computation beyond standard CFG is limited to low-dimensional projection and rescaling operations.

Critical weaknesses
  1. Limited methodological novelty beyond adaptive coefficient reweighting: Although the paper provides a geometric interpretation of CFG, the resulting guidance vector ϵD2\epsilon_{\mathrm{D^2}} still lies in the span of ϵc\epsilon_c, ϵu\epsilon_u, and xtx_t, and can therefore be written as a state-dependent linear combination of these three quantities. The effective combination coefficients are controlled by the hyperparameters η(q)\eta(q), ww, and γ2\gamma_2, while the remaining geometric quantities are deterministically induced by the current state and do not introduce additional degrees of freedom. Since these hyperparameters still require manual selection in practice, the proposed geometric structure does not by itself uniquely determine the guidance rule, and much of the final behavior can be viewed as an adaptive, manually calibrated reweighting of existing directions. This weakens the methodological novelty of the proposed formulation and makes it unclear how much of the observed behavior is attributable to the claimed geometric disentanglement rather than to coefficient tuning.

  2. Limited empirical evidence and insufficient comparison with closely related guidance methods: The experimental evaluation omits several closely related methods, including "Manifold Preserving Guided Diffusion (ICLR 2024)", "Angle Domain Guidance: Latent Diffusion Requires Rotation Rather Than Extrapolation (ICML 2025)", and "TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling (ICML 2026)". This makes it difficult to assess the contribution relative to the most relevant prior work. Moreover, under the more common setting of w=7.5w=7.5, the proposed method shows essentially no improvement over the reported baselines in the main quantitative metrics, and the qualitative results likewise do not demonstrate a clear advantage. The gains become more visible mainly at w=20w=20, but this is an unusually large guidance scale and is better interpreted as a stress-test regime than a typical operating point. Even in this regime, the improvements over the closest baselines are relatively modest. Overall, the current experiments do not provide sufficiently strong evidence that the proposed method yields a practically meaningful advantage under commonly used guidance settings.

Questions for authors
  1. Since the proposed guidance vector can be rewritten as a state-dependent linear combination of ϵc\epsilon_c, ϵu\epsilon_u, and xtx_t, can the authors clarify what additional modeling capability is introduced by the proposed geometric formulation beyond selecting the corresponding mixing coefficients? In particular, how would D2-CFG compare against a generic adaptive baseline of the form Atϵc+Btϵu+CtxtA_t\epsilon_c+B_t\epsilon_u+C_tx_t under a comparable hyperparameter tuning budget?
  2. The main advantages appear substantially more pronounced at very large guidance scales such as w=20w=20, whereas the improvements at the more common w=7.5w=7.5 setting are limited. Can the authors provide stronger quantitative and qualitative evidence that the proposed method offers a meaningful advantage in practically relevant guidance regimes?
Reviewer_uNs5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors analyze failure modes of CFG, i.e. oversaturation and loss of detail at high guidance strength. They argue that these are caused by the guidance term having components in the direction of the score and of the predicted clean sample. It is shown that by projecting the guidance term onto the space orthogonal to both the score and clean data prediction, one can improve cfg at high guidance strengths. The proposed method is evaluated on large scale diffusion models.

Critical strengths
  1. Empirically, the method seems to improve upon different CFG variants.
  2. The method does not introduce any extra cost at inference time
Critical weaknesses
  1. Presentation - the paper is not written well. It contains a lot of undefined jargon (please see the examples in the questions section), which makes it difficult to read.
  2. Many moving parts. CFG only has w. The proposed method has w, gamma_2 and eta. From table 3 it seems like the method is sensitive to gamma_2 which trades off e.g. HPS vs saturation
  3. I feel like important baselines are missing from the comparison, which are
  • Sadat et al. "Guidance in the Frequency Domain Enables High-Fidelity Sampling at Low CFG Scales", arxiv 2025
  • Kynkäänniemi et al. "Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models", NeurIPS 2024
Questions for authors
  1. Acronym TSR is undefined
  2. What do authors mean by "Because conditional and unconditional clean estimates share a similar coarse composition"? I also don't think I understand how the consequence of this is " their difference is typically positively aligned with the image itself"
  3. In equation 7, I think it would help to make it explicit that u is the conditional drift (I mean it can be inferred from eq4, but I did not register upon first reading)
  4. Line 238 - can the authors explain why this is the intended likelihood shift?
  5. "Because R(xt, t; w) is linear in w, increasing guidance directly corrupts the target density rate unless 〈c, ∆〉 = 0." I don't think linearity is relevant here. Any non-constant function of w would be a problem, no?
  6. The "Linear diffusion model. " paragraph is too difficult for me to understand.
    1. In this gaussian example, I guess "image" is not really an image and I don't know what "amplitude" means in "The shifted mean makes the guidance update point along the clean estimate, so guidance inflates the image amplitude"
    2. what is "score alignment " ?
    3. what is " clean-space alignment" ?
  7. Line 275 - "The solution of (10) and its relaxation read" - which relaxation? OK, I guess it's explained later. Why would we be interested in this relaxation?
  8. I guess the role of gamma_1 could be changed if the first basis vector was x_0^\hat rather than eps_c - could the authors comment on this choice?
  9. I don't think I understand what "the clean estimate stops inflating at low noise; empirically, the projection alone suffices there" means. Neither of the sentence components
  10. "Figure 2: Class-A samples" - how is class A defined? The text says "classes (A, B) share a trunk but diverge at sub-branches (details in Appendix C.3)", but the appendix does not define the classes.
  11. I am not sure it makes sense to compare at matched cfg weight (w) (e.g. in Table 1). All these methods offer a tradeoff between different metrics, so we should primarily be looking at pareto curves. A different w maybe a better choice for a baseline.
  12. I think some text was truncated here: "our gains coincide with lower saturation (Fig. 3), the opposite pattern."
  13. "unconstrained latent growth causes over-saturation" - what is latent growth?
  14. "Table 9 (Appendix F) separates the two ingredients: the retraction alone lifts standard CFG" - what retraction?
  15. " without retraction every variant burns out" - what does it mean for a variant/baseline to burn out?
  16. what are "mode-center trajectories" ?
Reviewer_1kMpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates two undesirable effects of classifier-free guidance (CFG): clean-image amplitude inflation and density drift along the conditional score direction. Based on a geometric and probability-density analysis, the authors propose Density-Decoupled CFG (D²-CFG), a training-free guidance method that projects the guidance update away from these two directions and introduces an additional mechanism for controlling density. The method is evaluated on synthetic distributions and text-to-image generation using SDXL and SD3.5-Medium.

Critical strengths
  1. The paper provides an interesting analysis of CFG from both geometric and probability-density perspectives. The distinction between amplitude inflation and density drift offers a useful interpretation of the artifacts associated with strong guidance.

  2. The proposed method is simple and training-free, requiring only modifications to the sampling procedure. This makes it relatively easy to integrate into existing diffusion and flow-matching models.

  3. The experiments cover both synthetic distributions and practical text-to-image generation, and demonstrate improvements in generation quality under high guidance scales.

Critical weaknesses
  1. Limited methodological novelty. The proposed approach largely builds upon existing ideas of guidance projection and density control. Although the joint projection is motivated by the theoretical analysis, its advantages over existing methods such as APG and DGS are not sufficiently established. The paper would benefit from a clearer explanation of what fundamentally new capability the joint formulation provides.

  2. Limited and inconsistent empirical improvements. The benefits are primarily observed under relatively high guidance scales. Under more conventional settings, the proposed method does not consistently outperform standard CFG or existing alternatives. For example, on SDXL at a guidance scale of 7.5, D²-CFG achieves an FID of 23.35 compared with 23.23 for standard CFG, while the CLIP scores are essentially identical. Even at a guidance scale of 20, the improvements remain relatively modest.

  3. Insufficient isolation of the key contributions. On SD3.5-Medium, a substantial portion of the reported improvement appears to come from spherical retraction rather than the proposed joint projection. At a guidance scale of 15, spherical retraction alone achieves an ImageReward of approximately 1.12, compared with 1.17 for the full method. This raises questions about the practical contribution of the main projection mechanism.

  4. The theoretical claims require stronger empirical validation. The paper emphasizes decoupling semantic guidance from density control, but the experiments do not fully establish that these factors can be independently controlled in realistic high-dimensional generation. In addition, the two-dimensional synthetic experiments do not directly validate the proposed joint two-direction projection, since only the score-direction projection is applied in that setting.

Questions for authors
  1. What is the main advantage of the proposed joint projection over existing guidance correction methods such as APG and DGS? Since both projection-based guidance correction and density control have been explored in prior work, it would be helpful to clarify what additional benefits arise specifically from their joint formulation.

  2. How much of the performance improvement can be attributed to the joint projection itself? In the SD3.5-Medium experiments, spherical retraction alone already achieves an ImageReward of approximately 1.12, compared with 1.17 for the full method. Could the authors further clarify the independent contribution of the joint projection, particularly under practical guidance settings?

  3. Can the authors provide stronger empirical evidence for the claimed decoupling between semantic guidance and density control? The theoretical analysis motivates independent control of these two factors, but it remains unclear how well this property holds in high-dimensional image generation. In addition, the two-dimensional synthetic experiments only apply the score-direction projection rather than the full joint projection. How can the effectiveness of the proposed decoupling be more directly validated?

  4. Can the authors evaluate the proposed method on image editing tasks? The current experiments focus primarily on text-to-image generation. It would be interesting to investigate whether D²-CFG can also improve instruction following, preservation of non-edited regions, and visual quality in image editing, thereby providing further evidence of its generalizability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.