acceptodds
Withdrawn from ICLR 2027

CtrlGen: Optimal Control for Text-to-Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Flow-based text-to-image models achieve strong generation quality and efficiency, yet conditional generation still relies heavily on classifier-free guidance. Existing guidance methods mainly adjust when or how strongly the conditional correction is applied, while its direction and magnitude are usually regulated together. This coupling limits local strength control, and unconstrained spatial and temporal magnitude variations can produce excessive responses during sampling. We identify guidance magnitude as the key variable for controlling strength while preserving the model-predicted direction and suppressing excessive local magnitude variations. Based on this observation, we propose CtrlGen, a training-free framework that decomposes the conditional correction into local direction and magnitude and treats the latter as a state-dependent control variable. At each sampling state, CtrlGen solves a regularized convex control problem that balances guidance fidelity, control energy, spatial smoothness, and state consistency. The resulting optimality condition is a screened elliptic equation with an explicit spectral solution, enabling frequency-selective magnitude control without iterative optimization. The optimized magnitude is recombined with the preserved direction to form a feedback guidance rule without modifying the pretrained model. Extensive experiments show consistent improvements over existing guidance methods in generation quality, text alignment, and preference-oriented metrics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Classifier-Free Guidance, Flow Matching, Optimal Control, Training-Free Guidance, Elliptic Regularization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Tak1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on guidance control for flow-based text-to-image generation. The authors find that coupling guidance direction and magnitude limits local strength control and allows excessive spatial and temporal variation. They therefore propose CtrlGen, a training-free method that preserves guidance direction while regulating its magnitude using an explicit spectral solution. To validate their method, they conduct several experiments, including comparisons on MS-COCO across three generators, component ablations, and guidance-scale robustness tests.The experimental results demonstrate that CtrlGen improves text alignment and preference scores across the evaluated models.

Critical strengths
  1. This paper proposes CtrlGen, which decomposes guidance into direction and magnitude and treats the local magnitude as a control variable.

  2. The authors formulate the magnitude update as a convex control problem that admits an explicit spectral solution without iterative optimization.

  3. They evaluate the method on three different flow-based text-to-image models, and it introduces negligible computational overhead without requiring retraining.

Critical weaknesses
  1. My main concern is that every baseline entry in Table 1 is identical to Table 2 of "CFG-Ctrl: Control-Based Classifier-Free Diffusion Guidance" (Wang et al., 2026), which introduced SMC-CFG. The paper does not state whether these values were reused, although Section 4.1 claims shared sampling settings and seeds for all methods. In Figure 3, the baselines' w=3w = 3 points match these values and show a sharp drop in MPS, whereas their w=2w = 2 and w=4w = 4 points approach CtrlGen. Notably, CtrlGen's large gains in Table 1 are concentrated in MPS, the very metric showing this drop. If the baselines come from a different pipeline, the margins reported in Table 1 and Figure 1(c) are not directly comparable.

  2. Even within a single pipeline, Table 1 compares all methods at only one guidance scale per backbone, and SMC-CFG and APG are not swept in Figure 3. Moreover, Eq. (9) shrinks the zero-frequency magnitude by 1/(1+λ+μ)1/(1+\lambda+\mu), roughly sixfold on FLUX.1-dev and Qwen-Image and twofold on SD3.5L under the default settings in Section 4.1. Equal nominal ww therefore does not imply equal guidance strength, and the reported margins depend on the chosen operating point.

  3. The training-free feedback contribution is not clearly positioned against existing control-based guidance. CFG-Ctrl, cited only as a baseline, already treats CFG in flow models as feedback control with separate gain and direction operators. Section 2 does not clearly distinguish CtrlGen's contribution, namely a spatially varying magnitude with a closed-form solution, from this setting.

  4. Section 1 claims that guidance strength is difficult to control without changing the model-predicted correction orientation. However, positive scalar CFG already preserves local correction directions. This motivation therefore does not establish direction preservation as a distinguishing contribution.

  5. The title frames CtrlGen as an optimal-control method, yet the optimality applies only to a hand-designed per-step objective with no term tied to generation quality, and Section 3.4 concedes that it is not trajectory-level. Its solution is a fixed spectral filter, so the framing overstates the theoretical contribution.

Questions for authors
  1. Could the authors state whether the baseline results in Table 1 were reused from CFG-Ctrl and confirm that all methods share the same prompts, seeds, sampler, step count, and metric implementations? If reused, please rerun the baselines in the same pipeline and explain the w=3w = 3 drop in Figure 3.

  2. Could the authors compare all methods, including SMC-CFG and APG, at their best-tuned or effective-strength-matched guidance scales across all three backbones?

  3. Could the authors clarify which aspects of the training-free feedback formulation are new relative to CFG-Ctrl? Please distinguish their objectives, control variables, and solvers.

  4. Could the authors revise this motivation and distinguish spatial magnitude regulation from CFG's existing direction preservation? Please specify which compared methods actually alter local correction directions.

  5. Could the authors clarify how optimality under Eq. (4) relates to generation quality, or otherwise qualify the optimal-control framing?

Reviewer_MgkQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CtrlGen modifies classifier-free guidance by separating each spatial guidance vector into its direction and magnitude. It preserves the direction while optimizing the magnitude through a quadratic objective combining fidelity, shrinkage, spatial smoothness, and consistency with the previous sampling step. The resulting elliptic system admits an efficient frequency-domain solution. Experiments on SD3.5 Large, FLUX.1-dev, and Qwen-Image examine generation quality, component contributions, computational cost, and compositional performance.

Critical strengths
  1. The controller is mathematically coherent and practical. The positive-definiteness and non-negativity arguments support the stated per-state solution and direction-preserving reconstruction. The paper appropriately limits its optimality claims, and the controller adds no network evaluations relative to the tested explicit-CFG setup.

  2. The experiments address a meaningful alternative explanation. Table S2 shows improvements beyond uniform magnitude attenuation, while Table S3 reports positive paired confidence intervals against SMC-CFG on SD3.5 and FLUX. The additional compositional evaluation strengthens the evidence beyond the main COCO results.

Critical weaknesses
  1. The particular elliptic formulation is not sufficiently distinguished from simpler magnitude filtering. Its solution combines spatial smoothing with a temporal reference. The attenuation control is valuable, but comparison with a direction-preserving spatial filter plus EMA would clarify whether the specific objective contributes beyond a convenient formulation of these operations.

  2. The practical quality trade-off is incompletely documented. The guidance sweep omits SMC-CFG, the strongest baseline in Table 1, and hyperparameter-selection procedures need clarification. Qwen also exhibits an FID regression relative to SMC-CFG, while its reported unguided reference lacks a complete metric row.

Questions for authors
  1. How does CtrlGen compare with spatial magnitude smoothing plus EMA at matched guidance magnitude or text alignment?

  2. Please clarify baseline tuning and report the complete Qwen reference metrics together with measured wall-clock overhead.

Reviewer_8dtyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CtrlGen is a training-free modification of classifier-free guidance (CFG) for flow-based text-to-image models. At each spatial location, it splits the guidance vector gt=vc−vug_t = v_c - v_u into a unit direction dtd_t and a magnitude rtr_t. It keeps the direction and replaces the magnitude with the solution m∗m^* of a quadratic problem that combines fidelity to rtr_t, an L2 energy term (λ\lambda), a gradient-smoothness term (γ\gamma), and coupling to the previous step's magnitude (μ\mu).

The optimality condition is a screened Poisson equation, solved in closed form using an FFT. The paper proves strong convexity, uniqueness, and non-negativity of m∗m^*.

On 5k MS-COCO prompts with SD3.5-L, FLUX.1-dev, and Qwen-Image, CtrlGen is compared against CFG, APG, CFG-Zero*, Rect-CFG++, and SMC-CFG on FID, CLIP, Aesthetic, PickScore, HPS, and MPS. The paper also reports ablations, a hyperparameter sweep, guidance-scale robustness curves, and computational cost.

Critical strengths

Cheap, simple and broadly applicable. The method needs one FFT per step, no extra network evaluations, and negligible memory and FLOPs. It works across three modern flow backbones and is easy to adopt.

Critical weaknesses
  1. Much of the gain comes from uniform guidance attenuation, baselines are not compared at matched strength, and the "optimal control" framing is overstated
  • Eq. 9 is a fixed linear spatio-temporal filter on the magnitude: m^t⋆=(r^t+μm^tref)/(1+λ+μ+γΛ)\hat m^\star_t = (\hat r_t + \mu \hat m^{\mathrm{ref}}_t)/(1+\lambda+\mu+\gamma\Lambda). With the FLUX/Qwen defaults (λ=5\lambda=5, μ=0.5\mu=0.5), the per-step DC gain is 1/6.51/6.5 and the steady-state gain is 1/(1+λ)≈0.171/(1+\lambda)\approx 0.17. CtrlGen therefore runs at an effective guidance scale of roughly w/6w/6 (about 0.5 instead of 3.0 on FLUX), plus spatial smoothing of the magnitude field.
  • The authors' own control in Table S2 confirms this matters. The "magnitude-only" variant, which they note is exactly CFG with weff=w/(1+λ)w_{\mathrm{eff}} = w/(1+\lambda), already improves MPS from 7.84 to 8.55 and HPSv2.1 from 0.283 to 0.297 over CFG at w=3w=3. That is roughly half of CtrlGen's total gain on FLUX. The spatial and state terms add the rest (MPS 8.55→9.35), so the filter does contribute, but this decomposition is shown only on FLUX and only at λ=5\lambda=5.
  • All competing methods (CFG, APG, CFG-Zero*, Rect-CFG++, SMC-CFG) are run at w=3w=3, while CtrlGen effectively operates at about one sixth of that strength. The fair comparison is each baseline at its best ww, or at weffw_{\mathrm{eff}}, ideally plus a simple Gaussian blur of ∣g∣|g|. Without it, the gains in Table 1 and the "robustness to large ww" in Fig. 3 cannot be attributed to the proposed controller rather than to weaker guidance.
  • The objective has no dynamics, no terminal cost and no trajectory-level coupling. It is a per-step Tikhonov smoothing of ∣g∣|g|, as the authors acknowledge (App. A.3, Limitations). The "optimal control" and "feedback control law" framing overstates the contribution.
  1. The evaluation does not fully support the size of the claims
  • The paired bootstrap analysis (Table S3) shows the gains over SMC-CFG are statistically reliable, but they are small for most metrics (e.g. HPSv2.1 +0.0025 and CLIP +0.0018 on FLUX).
  • MPS is the exception: it rises by about 1.1–1.4 points (e.g. 7.57→9.01 on SD3.5-L) while every other preference metric moves by about 1%. This discrepancy needs an explanation, such as a scale difference or a different evaluation protocol. As it stands, MPS dominates the headline improvements.
  • Hyperparameters are set separately per model (SD3.5-L uses different λ\lambda, μ\mu, γ\gamma). It is not stated whether they, and the sensitivity sweep in Table 3, were selected on the same 5k COCO prompts used for reporting.
  • On Qwen-Image, CtrlGen is not best on FID, Aesthetic or PickScore. On T2I-CompBench (Table S4), it is below SMC-CFG on color and texture binding.
Questions for authors
  1. Since CtrlGen effectively operates at weff≈w/(1+λ)w_{\mathrm{eff}} \approx w/(1+\lambda) (about 0.5 on FLUX), could you report CFG, APG, CFG-Zero*, Rect-CFG++ and SMC-CFG each tuned over ww, including w≈weffw \approx w_{\mathrm{eff}}, on all three backbones?
  2. Could you report the magnitude-only / spatial-only / state-only / full ablation on SD3.5-L and Qwen-Image as well, and for more than one value of λ\lambda? This would show whether the spatial and state terms consistently add value beyond uniform attenuation.
  3. How does CtrlGen compare with CFG at weffw_{\mathrm{eff}} plus a fixed Gaussian blur of ∣g∣|g| with a matched frequency response? This would isolate whether the per-step optimization formulation matters, or only the resulting low-pass filter.
  4. Were λ\lambda, γ\gamma, μ\mu (and the sweep in Table 3) selected on the same 5k MS-COCO prompts used for the reported results? If so, could you report results on a held-out prompt set with hyperparameters fixed in advance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.