acceptodds
Under review as a conference paper at ICLR 2027

CACFG: Curvature-Aware Classifier-Free Guidance and Optimal Control

Abstract

Diffusion models generate samples by learning to reverse a fixed corruption process, and classifier-free guidance (CFG) is the standard mechanism for conditioning this process on a desired class or prompt. CFG can be applied at varying guidance strengths, and while higher strengths improve image quality and conditional alignment, too high a guidance strength can degrade image quality and diversity. Furthermore, CFG violates principled diffusion sampling dynamics, and existing explanations for why it works despite the violation disagree on the underlying theory or do not extend to deterministic samplers used in practice. We address both these issues. We first frame CFG sampling as a continuous-time optimal control problem, treating the sampling trajectory as a sequence of controls chosen to maximise the probability of the desired condition. Solving the resulting Hamilton–Jacobi–Bellman equation shows that CFG is recovered under specific path costs when using an unconstrained control set. We argue this lack of constraint is responsible for CFG's failure at high guidance strengths, since it permits the sampling path to move arbitrarily far from the current image estimate. To fix this, we propose curvature-aware CFG (CACFG), which constrains the control set to a hypersphere informed by the Gaussian regularisation used when training variational autoencoders. We show that the control inputs produced by CFG sampling routinely violate this bound, and that across diffusion models, datasets, and guidance schedules, CACFG achieves superior generative quality at mid-to-high guidance strengths with a less severe quality-diversity tradeoff than regular CFG.

Primary area:
generative models
Keywords:
classifier-free guidance (CFG), optimal control, Hamilton–Jacobi–Bellman (HJB) equation, curvature-aware diffusion guidance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uKK6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Curvature-Aware Classifier-Free Guidance (CACFG), which reformulates CFG as a continuous-time optimal control problem using the Hamilton–Jacobi–Bellman (HJB) equation. The authors show that conventional CFG can be recovered under specific path costs and an unconstrained control set. Based on this formulation, they introduce a hyperspherical constraint derived from the Gaussian regularization of VAE latent representations to mitigate high-guidance degradation. Experiments on SD1.5 and SDXL demonstrate improvements in image quality and diversity, particularly at high guidance scales.

Critical strengths

S1. Optimal control interpretation of CFG

Although prior studies have explored diffusion guidance from an optimal control perspective [1,2], this paper presents a formulation that recovers conventional CFG through a deterministic HJB framework and provides a unified interpretation of existing guidance scheduling methods.

S2. Improved generation quality with negligible computational overhead

The proposed CACFG can be applied without additional training or significant computational overhead. Experimental results demonstrate improvements in FID, perceptual quality, and diversity-related metrics over standard CFG, particularly at high guidance scales.

[1] K. Pandey, F. M. Sofian, F. Draxler, T. Karaletsos, and S. Mandt, "Variational Control for Guidance in Diffusion Models," Proceedings of the 42nd International Conference on Machine Learning (ICML), https://arxiv.org/pdf/2502.03686.

[2] I. Azangulov, P. Potaptchik, Q. Li, E. Aamari, G. Deligiannidis, and J. Rousseau, "Adaptive Diffusion Guidance via Stochastic Optimal Control," Proceedings of the 29th International Conference on Artificial Intelligence and Statistics (AISTATS), 2026, https://arxiv.org/pdf/2505.19367.

Critical weaknesses

W1. Insufficient validation of the proposed failure mechanism

The authors argue that unconstrained control is a primary cause of high-guidance degradation. However, CACFG still exhibits increasing FID as the guidance scale increases (Appendix Tables 4–5). While the results demonstrate mitigation of degradation compared to standard CFG, they do not sufficiently establish that the proposed Gaussian norm constraint addresses the underlying failure mechanism. In particular, the relationship between norm-bound violations and actual generation degradation remains unclear.

W2. Unclear quality–alignment tradeoff

Although CACFG improves FID over standard CFG at high guidance scales, it remains unclear whether these improvements are achieved without sacrificing prompt alignment. The paper reports both FID and CLIP Score, but does not directly establish whether CACFG achieves better generation quality at comparable levels of text alignment. This is important for distinguishing a genuine improvement in the quality–alignment tradeoff from simply reducing the effective guidance strength.

W3. Limited evaluation on modern generative models

The experiments are limited to SD1.5 and SDXL. Since the proposed projection-based method does not directly depend on a specific UNet architecture, its effectiveness on modern diffusion transformers and flow-matching models, such as FLUX, remains unexplored. Furthermore, the validity of the Gaussian norm constraint across different latent representations is unclear.

Overall Assessment: Borderline Reject

The paper provides an optimal control interpretation of CFG and proposes a computationally inexpensive guidance correction. However, the experimental evidence primarily demonstrates mitigation rather than elimination of high-guidance degradation. The relationship between the proposed constraint and the underlying failure mechanism is not sufficiently validated, and the practical quality–alignment advantage and generalizability of the method remain unclear.


I would consider raising my rating to Borderline Accept if the authors could address the following points:

  • Failure mechanism validation: Provide an analysis relating Gaussian norm-bound violations to generation quality degradation, ideally including comparisons with simpler norm clipping or guidance rescaling baselines, to clarify whether the proposed constraint specifically addresses the claimed failure mechanism.

  • Quality–alignment tradeoff: Present an FID–CLIP Score Pareto frontier across guidance scales to demonstrate whether CACFG achieves better generation quality at comparable levels of prompt alignment than standard CFG.

  • Generalization: Provide evidence of applicability to a modern diffusion transformer or flow-matching model, or clarify the assumptions and limitations that prevent such an extension.

Questions for authors

Please refer to the Weaknesses section for my questions and concerns.

7 comments

(1/2)

We appreciate the reviewer's feedback and thank them for their comments. We are pleased that the reviewer found our formulation to provide "a unified interpretation of existing guidance scheduling methods", and we are glad they appreciated that CACFG "can be applied without additional training or significant computational overhead". We also thank the reviewer for clearly outlining the three points that would address their concerns, and we respond to each below.

Weaknesses

(W1): The reviewer notes that the relationship between norm-bound violations and generation degradation remains unclear.

We thank the reviewer for this suggestion, and have improved Appendix C.1 to address it. Our optimal control formulation shows that CFG places no constraint on the control (the Tweedie estimate), which permits the sampling path to move arbitrarily far from the current image estimate (Section 4). We then note that the VAE of a latent diffusion model is trained to regularise its encodings towards an isotropic Gaussian, which gives a theoretical upper bound on the magnitude of valid encodings, and hence of the Tweedie estimate (Section 5). In Appendix C.1 we show that as guidance strength increases, the proportion of SD1.5 CFG sampling paths that violate this bound rises from under 10%10\% to over 90%90\% (Figure 4), and that FID, NIQE, BRISQUE, TReS, Density, and Coverage all degrade as the proportion of violating paths grows (Figure 5). Furthermore, CACFG only alters a sampling path when the Tweedie estimate leaves the curvature-aware control set. Thus, at a fixed guidance strength, the improvements of CACFG arise solely from correcting violating paths, and these improvements are clear both qualitatively (Figure 3 and Appendix D.3) and quantitatively (Section 6.1, Figure 2b, and Appendices D.1 and D.2).

Regarding comparisons with norm clipping and guidance rescaling baselines, clipping methods such as the dynamic thresholding of Saharia et al. [2] bound the x0\boldsymbol{x}_0-prediction by the known range of the data ([−1,1][-1, 1] in pixel space). Latent spaces have no such known range, so these methods do not directly transfer to latent diffusion [3, 4]. Supplying this range for latent models is what drives our VAE-informed control set. We therefore compare against the two closest pre-existing latent-space analogues, CFG-Rescale (RECFG) [3] and APG [5], in Appendix D.5 (Figure 19). The curvature-aware correction can be applied on top of both methods, and it improves the FID-CLIP tradeoff of each at high guidance strengths (Figure 19b). In particular, applying the correction to CFG-Rescale (RECACFG) achieves the best FID-CLIP tradeoff of all methods tested. This further supports that bound violations are a source of degradation shared across guidance methods.

We agree that CACFG mitigates, rather than eliminates, degradation at high guidance strengths. Increasing guidance strength trades diversity for prompt alignment (which degrades FID at high guidance strengths). However, CACFG removes the additional degradation caused by controls that violate the upper bound.

(W2): The reviewer notes that it is unclear whether CACFG improves the quality-alignment tradeoff, or simply reduces the effective guidance strength.

We thank the reviewer for this suggestion. We have added FID-CLIP Pareto plots to the main paper (Figure 2b, lines 512-515) and Appendix D.2. At low CLIP scores, the curvature-aware and uncorrected curves coincide; however, as guidance increases, the uncorrected schedules incur a sharp FID penalty for minimal CLIP gains, whereas the curvature-aware variants achieve higher CLIP scores with a smaller FID penalty. Thus, at matched prompt alignment CACFG achieves better FID, and a reduction in effective guidance strength alone would not reproduce this result.

(2/2)

(W3): The reviewer notes that the experiments are limited to SD1.5 and SDXL, and questions the validity of the Gaussian norm constraint across different latent representations.

Our derivation assumes the variance-preserving flow ODE dynamics of Eq. 2, so extending the optimal control result to flow-matching models requires re-deriving the HJB solution under flow-matching dynamics. Furthermore, FLUX.1 [dev] is guidance-distilled [1], and does not use CFG at inference. However, we agree that evaluating CACFG on a diffusion transformer would strengthen the paper, and we are currently running experiments to this end. Due to hardware constraints these will be provided later in the week, and we will post an additional comment alongside them.

Regarding the validity of the Gaussian norm constraint, we agree that SD latents are not isotropic Gaussian, and our method does not require them to be strictly Gaussian. As reviewer GJw1 notes, the VAE is trained so that its latents have approximately unit variance. This forces most encodings to lie within a shell, and the Gaussian hypothesis provides an estimate of where this shell is located. In Appendix C.2 we compare the empirical distribution of SD1.5 and SDXL encodings with the theoretical chi density: although the empirical distribution is broader and right-skewed, rur_u falls just past its mode for both models, i.e., the estimated shell captures a large proportion of embeddings. Furthermore, we ablate rur_u against the empirical maximum observed encoding magnitude (CACFG* in Appendix C.2), and find that the Gaussian estimate outperforms the empirical bound on FID, NIQE, BRISQUE, TReS, Density, and Coverage at moderate-to-high guidance strengths for both models. We hypothesise this is a result of the Gaussian bound constraining the sampling trajectory to high-density embedding regions.

References

[1] Black Forest Labs. FLUX. https://github.com/black-forest-labs/flux, 2024.

[2] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems 35, NeurIPS, 2022.

[3] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV, 2024.

[4] Candi Zheng and Yuan Lan. Characteristic guidance: Non-linear correction for diffusion model at large guidance scale. In International Conference on Machine Learning, ICML, 2024.

[5] Seyedmorteza Sadat, Otmar Hilliges, and Romann M. Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In 13th International Conference on Learning Representations, ICLR, 2025.

Reviewer_uKK6OpenReview ↗

Response to Authors

Thank you for the thoughtful response and the substantial revisions, which have clarified several aspects of the work.

W2 has been addressed by the additional experiments and comments. The following remain partially addressed or unresolved:

  • W1: The current evidence does not yet fully establish that bound violations are the primary cause of degradation, or that the proposed constraint specifically targets this cause. Moreover, the CACFG* results (Table 1) suggest that the gains depend strongly on how tightly the radius is set, leaving open the possibility that the effect stems from norm-shrinking regularization rather than from preventing violations.

  • W3: Thank you for pointing out that FLUX.1-dev is guidance-distilled. Models such as DiT-XL/2 or PixArt-α/Σ use VP diffusion with SD-family VAEs, so CACFG should apply to them without re-deriving the HJB solution.

I look forward to further discussion on these points.

Reviewer_G763OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

Starting from the concern that CFG violates principled diffusion sampling dynamics, the paper re-examines CFG sampling as a continuous-time optimal control problem. Viewing the sampling trajectory as a sequence of controls and solving the HJB equation, it shows that CFG is recovered when the control set is unconstrained and the traversal penalty is on the same scale as the denoising training loss. The authors take this absence of constraint to be the cause of failure at high guidance, and propose CACFG, which restricts the control set to a hypersphere whose radius is estimated from the Gaussian regularisation of the VAE. Generation quality at medium-to-high guidance strength improves on SD1.5 and SDXL.

Critical strengths
  1. Whereas prior work in the guidance line has relied considerably on intuition (e.g., adjusting the weight or steering the guidance direction) this paper views the problem from a different angle, that of optimal control, and derives its solution within that framework. That existing guidance scheduling methods (CFG++, IntCFG) are derived from the same formulation under different choices of traversal penalty, and the observation that this penalty weighting coincides with the ELBO weighting, serve as evidence that this viewpoint has explanatory power.
  2. The results in the high guidance regime are convincing. As guidance is increased, FID degrades less while NIQE, BRISQUE, and TReS improve together, and Density and Coverage rise alongside FID rather than trading off against it. Since quality and diversity are obtained simultaneously, it is hard to view this as a simple shift along a trade-off.
Critical weaknesses
  1. The optimal control background is absent from the preliminaries. Section 4 begins by presupposing the Bolza form, the value function, the HJB equation, and the control set, while the preliminaries in Section 2 cover only diffusion models. LL, ϕ\phi, and bb appear in Eq. (10) without explanation, and the structure by which these are undetermined symbols that are then filled in successively in Eqs. (13), (14), and (16) is not made apparent in the main text, which makes the derivation hard to follow on a first reading. Given that a substantial portion of readers in this area are not familiar with optimal control, a minimum of background explanation should be in the main text.
  2. The non-uniqueness of inverse optimal control is not addressed. In Eq. (15) the value function is assumed to equal ϕ\phi, and then the hh that makes this hold is solved for backwards. However, the cost function that makes a given control optimal is in general not unique. There may be several control problems that have CFG as their optimal solution, and Eq. (16) is only one of them. In that case, on what basis do we judge that this particular formulation is a good one?
Questions for authors
  1. Given that the inverse optimal control solution is not unique, what is the basis for justifying the choices in Eqs. (13) and (16)?
  2. In Fig. 2, saturation increases monotonically for the comparison methods as guidance grows, whereas the CACFG variants barely change and in some ranges actually decrease. If the operation is clipping the norm, it is understandable that saturation would be capped at an upper limit, but becoming lower than the comparison methods requires further explanation. Could you provide an explanation or analysis of this phenomenon?
  3. If the control set UU is changed, does hh in Eq. (16) not change along with it through Υ\Upsilon? If so, does the path cost interpretation derived for CFG still hold for CACFG?
  4. Can this constraint also be used in the training or inference of flow matching models?
4 comments

(1/2)

We appreciate the reviewer's feedback and thank the reviewer for their comments. We are thrilled that the reviewer appreciated our optimal control perspective and that they found our curvature-aware correction to be more than "a simple shift along a trade-off".

Weaknesses

(W1): The reviewer notes that the optimal control background is absent from the preliminaries.

We thank the reviewer for this suggestion, and acknowledge some readers may be unfamiliar with optimal control. We have added a "Continuous-time optimal control" paragraph to the preliminaries (lines 175-193) that introduces the Bolza form, the value function, the HJB equation, and the inverse problem we solve in Section 4. We have also restructured Section 4 so that each component is introduced and then explained (see Eq. 15, lines 256-277, Eq. 16, lines 278-283, Eqs. 17-18, lines 284-301).

(W2): The reviewer notes that the non-uniqueness of inverse optimal control is not addressed.

We thank the reviewer for raising this point, and agree that the cost that makes a given control optimal is, in general, not unique. To address this, we now characterise every running reward under which CFG is optimal (Proposition E.13 and Corollary E.14, summarised on lines 345-357). Using an unconstrained control set and fixing the control cost as an ℓ2\ell_2 penalty with arbitrary scale, we find that CFG is the optimal control if and only if V^=κϕ+C\hat{V} = \kappa \phi + C and h=κhω+κ′ϕ+C′h = \kappa h_\omega + \kappa' \phi + C', for some positive, time-dependent κ\kappa and time-dependent CC. This gives the value function two degrees of freedom, and neither changes the guidance: CC adds the same reward to every trajectory, and κ\kappa rescales the penalty. Furthermore, κ≡1\kappa \equiv 1 if and only if the ℓ2\ell_2 penalty is the SNR weighting of [1] scaled by the guidance weight (Corollary E.14). Thus, the hh of Eq. 18 (Eq. 16 in the original submission) is the unique reward function that expresses CFG in terms of the SNR scale multiplied by a user-defined movement penalty. Any other reward function introduces either an arbitrary scaling that exists only to be cancelled by a corresponding scaling of the value function, or an additive term that uniformly moves the cost surface up or down.

(2/2)

Questions

(Q1): Given that the inverse optimal control solution is not unique, what is the basis for justifying the choices in Eqs. (13) and (16)?

The choice of hh in Eq. 18 (previously Eq. 16) is addressed in (W2). For the running reward of Eq. 15 (previously Eq. 13), we have restricted hh to depend on the state (xt\boldsymbol{x}_t) only, whereas in the original submission it could also depend on the control (this does not change hh, but is necessary for the uniqueness result above). The ℓ2\ell_2 penalty of the control (ut\boldsymbol{u}_t) is then justified as follows (lines 261-277): (i) the unconditional Tweedie estimate minimises the denoising objective, so the penalty scores a candidate control using the same loss the denoiser is trained on; (ii) a quadratic control cost is the canonical choice in optimal control [2], and with affine dynamics it yields a unique optimal control; and (iii) it makes the inverse problem well-posed, since if hh could depend on the control, any control could be made optimal for any value function (as the reviewer points out).

(Q2): In Fig. 2, saturation increases monotonically for the comparison methods as guidance grows, whereas the CACFG variants barely change and in some ranges actually decrease...

We thank the reviewer for this insightful question. Sadat et al. [3] show that the component of the CFG update parallel to the denoised estimate acts as a "gain" on that estimate, increasing its saturation. The curvature-aware projection bounds the magnitude of the Tweedie estimate, and thus directly caps this gain. As guidance strength increases, the projection becomes active on a larger proportion of sampling paths (Appendix C.1), which we believe explains the difference in saturation at high guidance strengths. Furthermore, the rising saturation of the comparison methods at high guidance strengths is oversaturation, a known unwanted artefact [3].

(Q3): If the control set U is changed, does h in Eq. (16) not change along with it through Υ? If so, does the path cost interpretation derived for CFG still hold for CACFG?

Yes, hh changes with UU through Υ\Upsilon (Eq. 19). However, Theorem E.11 holds for any nonempty, closed, and convex control set that is either unconstrained, or compact and continuous in time. Thus, CACFG is the optimal control of the same problem with Uγ(t)U_\gamma(t), under the same reward with the projected Υ\Upsilon. The path cost interpretation in the main paper is actually presented for arbitrary UU, and therefore still holds: the conditional score alignment and early-convergence terms are unchanged, while Υ\Upsilon, which we interpret as the maximum allowable CFG movement (line 332), is now bounded by the control set.

(Q4): Can this constraint also be used in the training or inference of flow matching models?

Yes, in theory the curvature-aware constraint could be used during the training of flow-matching models, and this is an exciting future research direction! However, training a flow-matching model using CACFG is beyond the scope of this study. It is also technically possible to apply CACFG to flow-matching models at inference; however, our derivation assumes the variance-preserving probability-flow ODE of Eq. 2. Extending the optimal control result to flow matching at inference requires re-deriving the HJB solution under flow-matching dynamics.

References

[1] Diederik P. Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. In Advances in Neural Information Processing Systems 36, NeurIPS, 2023.

[2] Daniel Liberzon. Calculus of Variations and Optimal Control Theory: A Concise Introduction. Princeton University Press, 2012.

[3] Seyedmorteza Sadat, Otmar Hilliges, and Romann M. Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In 13th International Conference on Learning Representations, ICLR, 2025.

Thank you for the revision. Both of my concerns are addressed; the new preliminaries paragraph supplies the optimal-control background, and Proposition E.13 with Corollary E.14 answers the non-uniqueness question.

I am keeping my rating at 3.

We thank the reviewer for their engagement throughout the discussion period, and we are glad that the additions to the preliminaries and the characterisation of the path costs addressed their concerns. We believe these changes have strengthened the paper, and we appreciate the reviewer’s suggestions.

Reviewer_GJw1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new way of looking at classifier-free guidance (CFG). The authors treat sampling as an optimization problem and derive the cost for which CFG is the optimal solution. They argue that CFG breaks down at high guidance strengths because nothing in this formulation limits how far each step can push the image, and they propose adding such a limit. Latent diffusion models operate in a compressed space learned by an autoencoder trained with a standard Gaussian regularization on its latents. The authors therefore cap the size of the model’s predicted clean latent at the size of a typical Gaussian latent, which in practice amounts to clipping its norm at each step. Experiments on Stable Diffusion 1.5 and SDXL show better image quality, diversity, and prompt alignment at medium-to-high guidance strengths, at almost no extra cost.

Critical strengths
  • Good presentation: The paper is easy to follow and well written.
  • Elegant unifying view: CFG, CFG++ and IntCFG are expressed as one family that differs only in the cost scale. The link between the CFG cost scale and the ELBO weighting is a cool observation.
  • Simple method: CACFG is just one norm and one rescale per step, with no retraining, plus it can be combined with any guidance schedule.
Critical weaknesses
  • What does the optimal-control result actually explain? The derivation seems sound, but it works backwards. The authors fix the value function, the form of LL, the dynamics, and the cost scale, then solve for whatever hh makes CFG the optimal control. With different choices, CFG would be optimal for a different hh. And because the guidance schedule ω(t)\omega(t) is free, any schedule is optimal under this framework. The central choice, V=ϕV = \phi, is justified by CFG already working well, which assumes the very thing the section claims to explain! The result shows that CFG can be written as the solution to an optimal control problem, not why it works despite violating sampling dynamics. The interpretations of the terms of hh inherit the same issue. I’d ask the authors to frame this as an interpretive reformulation rather than an explanation, or to give an independent justification for their modeling choices.

  • Disconnect between theory and method: My biggest concern is the fact that the theory section, which despite its flaws is rather insightful, seems mostly disconnected from the proposed approach that follows (Section 5). There, the constraint comes entirely from the VAE prior. CACFG could be motivated and derived without most of Section 4.

  • Method amounts to norm clipping: I don't see any notion of curvature in the method. Plus, the N(0,I)\mathcal{N}(0, I) assumption is questionable for SD latents. My understanding is that these VAEs use a very small KL weight and a post-hoc scaling factor to get roughly unit variance, which is not the same as being isotropic Gaussian.

  • Missing baselines: All comparisons are between a guidance scheme X and X + Curvature Aware. The method, as proposed, belongs to a family of clipping and rescaling methods which are not compared against (for exmaple Saharia et al., 2022).

  • Missing Pareto fronts: The main claim is about the quality–diversity tradeoff, so it calls for Pareto fronts or best ω\omega per-method comparisons, not comparisons at matched ω\omega.

Questions for authors
  • Can you acknowledge the non-uniqueness of the inverse construction? Is there an independent justification for V=ϕV = \phi and the L2 form of LL ? Does the framework make any non-obvious prediction that you verify empirically?

  • How does CACFG compare with dynamic thresholding?

  • See other weaknesses.

5 comments

(1/3)

We appreciate the reviewer's feedback and thank them for their consideration. We are pleased that the reviewer found the paper "easy to follow and well written", and that they found our treatment of CFG, CFG++, and IntCFG to be an "elegant unifying view", with the link between the CFG cost scale and the ELBO weighting being "a cool observation".

Weaknesses

(W1): The reviewer questions what the optimal control result explains, noting that the derivation works backwards and that the choice V^=ϕ\hat{V} = \phi is justified by CFG already working well.

We agree that our derivation works backwards. Section 4 is an inverse optimal control problem [1]: we fix the dynamics and the control cost, and solve for the cost under which an observed control is optimal (lines 191-193). Inferring the objective implied by an observed controller is the purpose of inverse optimal control, and Kalman [1] posed this problem using the same linear dynamics and quadratic control cost. Our goal in this work was to identify the cost implied by CFG.

We also acknowledge that the cost that makes a given control optimal is, in general, not unique. To address this, we now characterise every running reward under which CFG is optimal (Proposition E.13 and Corollary E.14, summarised on lines 345-357). Fixing the dynamics, an unconstrained control set, and an ℓ2\ell_2 control cost with arbitrary scale, we find that CFG is the optimal control if and only if V^=κϕ+C\hat{V} = \kappa \phi + C and h=κhω+κ′ϕ+C′h = \kappa h_\omega + \kappa' \phi + C', for some positive, time-dependent κ\kappa and time-dependent CC. Neither κ\kappa nor CC changes the guidance: CC adds the same reward to every trajectory, and κ\kappa rescales the movement penalty. Furthermore, κ≡1\kappa \equiv 1 if and only if the ℓ2\ell_2 penalty is the SNR weighting of [2] scaled by the guidance weight (Corollary E.14). Thus, the hh of Eq. 18 (Eq. 16 in the original submission) is the unique reward function that expresses CFG in terms of the SNR scale multiplied by a user-defined movement penalty. Any other reward function introduces either an arbitrary scaling that exists only to be cancelled by a corresponding scaling of the value function, or an additive term that uniformly moves the cost surface up or down. Furthermore, this means that V^=ϕ\hat{V} = \phi is no longer justified only by CFG working well: up to a time-only constant, it is the only value function under which CFG is optimal with scaling coming from the SNR-weighted penalty and movement penalty (lines 290-291).

The remaining modelling choice is the ℓ2\ell_2 control cost, which we have now fixed (in the original submission hh could also depend on the control; restricting it to the state does not change our hh). We justify this as follows (lines 261-277): (i) the unconditional Tweedie estimate minimises the denoising objective, so the penalty scores a candidate control using the same loss the denoiser is trained on; (ii) a quadratic control cost is the canonical choice in optimal control [1, 3], and with affine dynamics it yields a unique optimal control; and (iii) it makes the inverse problem well-posed, since if hh could depend on the control, any control could be made optimal for any value function. Finally, the framework does not fix the guidance schedule ω(t)\omega(t), and we view this as a strength, as it allows CFG, CFG++, and IntCFG to be expressed as differing only in their proposed movement penalties.

(W2): The reviewer notes that the theory section seems disconnected from the proposed approach, as the constraint comes entirely from the VAE prior.

We respectfully disagree. From a control perspective, an unrestricted control set is the first thing one would correct, before attempting to modify the cost surface. Section 4 shows that CFG imposes no constraint on the control set, which permits the sampling path to move arbitrarily far from the current image estimate given a sufficiently low movement penalty (lines 366-376). The VAE discussion of Section 5 then facilitates this correction, as it provides an estimate of where valid controls lie. Without Section 4, it is not clear which quantity should be constrained (e.g., xt\boldsymbol{x}_t, the score, or ϵ\boldsymbol{\epsilon}), nor is there justification for why a projection is the appropriate correction. Most importantly, as CFG breaks diffusion sampling dynamics [4], its guided estimate has no inherent theoretical bound, and it only inherits a constraint set once it is treated as a control.

(2/3)

(W3): The reviewer notes that the method amounts to norm clipping, and that the N(0,I)\mathcal{N}(\boldsymbol{0}, I) assumption is questionable for SD latents.

We agree that SD latents are not isotropic Gaussian, and our method does not require them to be. As the reviewer notes, the VAE is trained so that its latents have approximately unit variance, which packs most encodings densely within a specific shell. The Gaussian hypothesis provides an estimate of where this shell is located. In Appendix C.2 we compare the empirical distribution of SD1.5 and SDXL encodings with the theoretical chi density. The empirical distribution is broader and right-skewed; however, rur_u falls just past its mode for both models and thus captures the region where the bulk of the density has accumulated. Furthermore, we ablate rur_u against the maximum observed encoding magnitude (CACFG* in Appendix C.2), and find that the Gaussian estimate outperforms the empirical bound on FID, NIQE, BRISQUE, TReS, Density, and Coverage at moderate-to-high guidance strengths for both models. We hypothesise this is a result of constraining the sampling trajectory to high-density embedding regions. Regarding curvature, the name refers to the curvature of the control set: CACFG constrains the control to a hypersphere, rather than leaving it unconstrained.

(W4): The reviewer notes that all comparisons are between a guidance scheme and its curvature-aware variant.

We thank the reviewer for this suggestion. Clipping methods such as the dynamic thresholding of Saharia et al. [5] bound the x0\boldsymbol{x}_0-prediction by the known range of the data ([−1,1][-1, 1] in pixel space). Latent spaces have no such known range, so these methods do not directly transfer to latent diffusion [6, 7]. Supplying this range for latent models is what drives our VAE-informed control set and the subsequent curvature-aware correction. We therefore compare against the two closest pre-existing latent-space analogues, CFG-Rescale (RECFG) [6] and APG [8], in Appendix D.5 (Figure 19). As both methods modify the Tweedie estimate rather than the guidance schedule, the curvature-aware correction can also be applied on top of each. We find that the curvature-aware correction improves the FID-CLIP tradeoff of all three methods at high guidance strengths (Figure 19b). The best FID-CLIP tradeoff overall is achieved by RECACFG, i.e., by applying the curvature-aware correction to the strongest existing baseline, with the greatest improvement appearing at high guidance strengths. While RECFG alone outperforms CACFG, it relies on a tuned blending parameter, whereas the curvature-aware bound is derived from the VAE, and the two corrections are complementary. We also note that combining the correction with APG, which already rescales its update, degrades no-reference quality metrics at the highest guidance strengths (despite improving FID and CLIP), which we hypothesise results from the combined corrections over-shrinking the guided estimate, producing a washed-out colour palette.

(W5): The reviewer notes that the main claim calls for Pareto fronts, rather than comparisons at matched ω\omega.

We thank the reviewer for this suggestion. We have added FID-CLIP Pareto plots to the main paper (Figure 2b, lines 512-515) and Appendix D.2. At low CLIP scores, the curvature-aware and uncorrected curves coincide; however, as guidance increases, the uncorrected schedules incur a sharp FID penalty for minimal CLIP gains, whereas the curvature-aware variants achieve higher CLIP scores with a smaller FID penalty.

(3/3)

Questions

(Q1): Can you acknowledge the non-uniqueness of the inverse construction? Is there an independent justification for V=ϕV = \phi and the L2 form of LL? Does the framework make any non-obvious prediction that you verify empirically?

We acknowledge the non-uniqueness of the inverse construction, and address it alongside the justification for V^=ϕ\hat{V} = \phi and the ℓ2\ell_2 penalty in (W1). Regarding predictions, the control set constrains the control at every time in [tl,T][t_l, T], and thus the framework predicts that a violation at any point along the sampling trajectory is harmful, not only a violation at the terminal sample. CACFG enforces the constraint along the entire trajectory, and Figure 4 (Appendix C.1) illustrates this: CFG paths violate the bound throughout sampling, while CACFG paths remain within the time-varying bound. The improvements in Section 6.1, and the relationship between violation rate and image quality in Appendix C.1 (added in the updated manuscript), support this prediction. Importantly, outside of the control formulation there is no theoretical reason to bound the CFG estimate at all. Lemma E.4 bounds the Tweedie estimate of a true posterior, but CFG breaks the diffusion sampling dynamics, so its guided estimate is not the Tweedie estimate of any distribution and inherits none of its properties. Approaches such as dynamic thresholding [5] are therefore motivated heuristically. Only by treating the CFG estimate as a control, as in our framework, does it inherit a constraint set, and the projected control is then the optimal control at every time.

(Q2): How does CACFG compare with dynamic thresholding?

As discussed in (W4), dynamic thresholding is designed for pixel-space models, where the data range is known [5], and does not directly transfer to latent diffusion [6, 7]. However, we provide comparisons with the closest latent-space analogues in Appendix D.5.

References

[1] Rudolf E. Kalman. When is a linear control system optimal? Journal of Basic Engineering, 86(1):51-60, 1964.

[2] Diederik P. Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. In Advances in Neural Information Processing Systems 36, NeurIPS, 2023.

[3] Daniel Liberzon. Calculus of Variations and Optimal Control Theory: A Concise Introduction. Princeton University Press, 2012.

[4] Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. In Advances in Neural Information Processing Systems 37, NeurIPS, 2024.

[5] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems 35, NeurIPS, 2022.

[6] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV, 2024.

[7] Candi Zheng and Yuan Lan. Characteristic guidance: Non-linear correction for diffusion model at large guidance scale. In International Conference on Machine Learning, ICML, 2024.

[8] Seyedmorteza Sadat, Otmar Hilliges, and Romann M. Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In 13th International Conference on Learning Representations, ICLR, 2025.

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your time and thoughtful reviews. With the author–reviewer discussion period ending on November 18 (AoE), please review the authors' rebuttal and share your updated assessment. Specifically, let us know whether your main concerns have been resolved, and if not, which points remain open.

Your input before the deadline is greatly appreciated!

AC

General Response

We present CACFG, which frames classifier-free guidance (CFG) as a continuous-time optimal control problem whose path costs are expressed using the SNR weighting of the denoising objective [1]. Our framework characterises modern guidance schedules as differing only in the movement penalties placed on the control, and it shows that CFG and its scheduled variants use an unbounded control set. Reviewer GJw1 found our treatment of CFG to be an "elegant unifying view", reviewer uKK6 highlighted its "unified interpretation of existing guidance scheduling methods", and reviewer G763 noted that our optimal control viewpoint "has explanatory power".

Additionally, we introduce a curvature-aware correction that leverages a property of latent diffusion models: their VAE regularises encodings towards unit variance, concentrating them within a hypersphere. We show that if latent encodings are bounded, then so too is the Tweedie estimate of the diffusion posterior (Lemma E.4). This means that the control, which takes the place of the Tweedie estimate, should inherit this bound, a fact that is ignored by CFG. We hypothesise that controls exceeding this bound contribute to the degradation of image quality in high-CFG regimes. To address this, we propose a convex, time-varying control set that interpolates between an SNR-dependent bound and the raw upper bound. Our experimental results show that the correction improves the FID-CLIP Pareto frontier of every guidance technique it is applied to at mid-to-high guidance strengths, including existing rescaling methods. Reviewer G763 found the curvature-aware correction to be more than "a simple shift along a trade-off", while reviewer uKK6 noted that it "can be applied without additional training or significant computational overhead".

During the rebuttal we made the following additions:

  • Section 2 and Section 4 (Rev. GJw1, G763): A new optimal control paragraph in the preliminaries introduces the value function, the HJB equation, and the inverse control problem in Section 4. Section 4 is restructured to justify each modelling choice, including the ℓ2\ell_2 control cost.
  • Proposition E.13 and Corollary E.14 (Rev. GJw1, G763): We characterise every running reward under which CFG is optimal, and show that our reward is the unique one that expresses CFG in terms of the SNR-weighted denoising objective scaled by a user-defined movement penalty.
  • Appendix C.1 (Rev. GJw1, uKK6): The proportion of CFG sampling paths that violate the curvature-aware bound rises with guidance strength, and FID, NIQE, BRISQUE, TReS, Density, and Coverage degrade as this proportion grows.
  • Figure 2b and Appendix D.2 (Rev. GJw1, uKK6): FID-CLIP Pareto plots show that, at high guidance strengths, the curvature-aware (CA) variants achieve higher CLIP scores at a lower FID, which distinguishes them from a mere reduction in effective guidance strength.
  • Appendix D.5 (Rev. GJw1, uKK6): Comparisons with CFG-Rescale [2] and APG [3], the closest latent-space analogues of dynamic thresholding, show that the curvature-aware correction improves the FID-CLIP tradeoff of both, with RECACFG achieving the best tradeoff of all methods tested.
  • Appendix D.7 (Rev. uKK6): Applying CACFG to the PixArt-α\alpha diffusion transformer [4] improves the FID-CLIP tradeoff at all tested guidance strengths, demonstrating that the correction transfers to transformer-based architectures.
  • Abstract and Section 1 (Rev. uKK6): We have amended our wording to state more precisely that the unconstrained control set is a key contributor to, rather than the sole cause of, CFG's degradation at high guidance strengths.

In summary, we show that CFG is the optimal control of an unconstrained problem, and that constraining its control set improves generative quality at mid-to-high guidance strengths across diffusion models, architectures, datasets, guidance schedules, and existing rescaling methods, at negligible additional computational cost. Most importantly, our control framework provides a principled lens through which CFG and its variants can be interpreted and corrected.

References

[1] Diederik P. Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. In Advances in Neural Information Processing Systems 36, NeurIPS, 2023.

[2] Shanchuan Lin et al. Common diffusion noise schedules and sample steps are flawed. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV, 2024.

[3] Seyedmorteza Sadat et al. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In 13th International Conference on Learning Representations, ICLR, 2025.

[4] Junsong Chen et al. PixArt-α\alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In 12th International Conference on Learning Representations, ICLR, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.