acceptodds
Under review as a conference paper at ICLR 2027

Uncertainty-adaptive Feedback Guidance for Improved Image Generation with Diffusion Model

Abstract

The success of modern diffusion models heavily depends on the guidance mechanisms that align generated images with conditions. However, the widely adopted classifier-free guidance (CFG) applies a global guidance scale across the entire image. Hence, it can overlook the complex dynamics of the generative process, and may result in misaligned features or visual artifacts. To overcome this limitation, we introduce a novel guidance mechanism named Uncertainty-adaptive Feedback Guidance (UFG) that leverages pixel-wise uncertainty as an adaptive scale. By utilizing uncertainty estimates, our method adaptively determines the pixel-wise guidance scale by minimizing the one-step marginal variance at each denoising step. This guidance affects the next-step uncertainty estimation, and hence establishes a feedback loop. Through empirical evaluations across various image generation tasks, we demonstrate that scaling guidance based on pixel-wise uncertainty enhances overall generation quality and effectively mitigates visual artifacts. Consequently, UFG achieves strong performance compared to standard CFG, recent guidance variants, and existing uncertainty-aware baselines.

Primary area:
generative models
Keywords:
Image Generation, Diffusion, Guidance, Uncertainty Quantification, Bayesian inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sgrGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Uncertainty-adaptive Feedback Guidance (UFG), a guidance mechanism for conditional diffusion models. Its core contribution is to replace the global, fixed classifier-free guidance (CFG) scale with a pixel-wise adaptive guidance scale derived from Bayesian variance estimation (BayesDiff). Specifically, at each denoising step, the pixel-wise scale is given by the closed-form solution (Proposition 3.1, Eq. 19) of a constrained one-step marginal variance minimization problem (Eq. 18); since the guidance modifies the uncertainty estimate of the next step, a feedback loop emerges across steps (Fig. 2). Method variants include UFG-M, UFG-accel (approximately 1.2× the cost of CFG), and a step-skipping strategy. For validation, the paper reports FID, Precision/Recall, CLIP-score, and VQAScore on class-conditional ImageNet (256/512, U-ViT and DiT-XL/2) and text-to-image generation (SDXL, Z-Image, SD1.5), together with an uncertainty-dispersion analysis and qualitative comparisons.

Critical strengths
  1. A theoretically grounded closed-form pixel-wise guidance rule. The variance-minimization problem (Eq. 18) is cleanly formulated and admits a unique closed-form solution (Proposition 3.1), with a complete KKT derivation in Appendix C; the decomposition of the one-step variance into pi+ℓisi+qisi2p_i + \ell_i s_i + q_i s_i^2 is concise and highly interpretable.

  2. A conceptual feedback-loop framework with direct diagnostic evidence. The paper clearly articulates how guidance feeds back into the next-step uncertainty estimate (Fig. 2), and introduces the uncertainty-dispersion metric ψ(x0)\psi(x_0) as a per-sample diagnostic (Fig. 3), supported by visualizations of artifact suppression in high-uncertainty regions (Figs. 4, 19–22). This provides a testable mechanistic explanation and is valuable for understanding CFG-like guidance behavior.

Critical weaknesses
  1. The empirical gains are small and inconsistent, leaving the central claim unsubstantiated. In several settings the baselines actually win: on ImageNet 512×512, CADS achieves the best FID (4.3587, vs. 4.3728 for UFG-M and 4.5532 for UFG-accel); on SDXL, CFG++ achieves the best FID (13.9986, vs. 14.0395 for UFG-M); on DiT-XL/2, CADS has the lowest FID (1.9697, vs. 2.2492 for UFG-accel), and UFG (MC) is even slightly worse than CFG on FID (2.3552 vs. 2.2863). The claimed superiority simply does not hold across backbones and metrics, which is the most direct factor affecting my recommendation.
  2. Insufficient analysis of the computational overhead trade-off, raising doubts about practical usability. UFG (MC) is about 2.9× slower than CFG (Table 6), yet the paper neither specifies when the accuracy gains justify this overhead, nor explains why UFG-accel sometimes underperforms UFG (MC) (Table 2, 512×512). For a practically oriented guidance method, the lack of a clear decision guideline is a substantive deficiency.
  3. Reliance on BayesDiff's Laplace approximation, with no quantitative analysis of how its bias is mitigated. The method directly inherits BayesDiff's approximation and Monte Carlo machinery; the paper itself acknowledges the bias in variance estimation (Sec. 3.2), but relies solely on "independent coupling" as a remedy without quantifying how much of the bias this inflation compensates. Since the closed-form solution (Eq. 19) takes this variance estimate as input, estimation errors propagate into the guidance scale, undermining confidence in the core mechanism.
  4. The causal chain "variance reduction → quality improvement" is never directly verified. This assumption is largely transplanted from BayesDiff's observations (Sec. 3.1); no controlled experiment is conducted (e.g., reducing variance through an alternative mechanism and checking whether quality improves accordingly); and the dispersion metric ψ(x0)\psi(x_0) measures the spatial uniformity of variance rather than image quality, with its correlation to FID/CLIP unquantified. The core mechanism is asserted more than demonstrated.
  5. Failure modes in which variance minimization may harm diversity or alignment are not discussed. Reducing one-step variance is essentially a contraction toward low-entropy solutions; the paper does not analyze whether this contraction produces side effects for highly diverse prompts, long-tail concepts, or scenarios requiring precise semantic alignment (the compositional prompts in Appendix E.9 cover only limited cases).
  6. Notation and derivation inconsistencies. x∘2:=xx⊤x^{\circ 2} := xx^\top (outer product) is defined below Eq. 15, yet the notation is used elementwise in Eq. 15, creating ambiguity; the derivation from Eq. 12 to Eq. 15 mixes ⊙\odot and ∘2\circ 2 inconsistently, and the derivation of Σpred∗\Sigma_{pred}^* should be checked for dimensional consistency; in Appendix A.1, Eq. 24 omits the g(t)2/σtg(t)^2/\sigma_t factor relative to Eq. 23 without stating the neglected terms and their order. The rigor of the core probabilistic derivation directly affects trust in the variance-minimization objective.
  7. The stability and convergence of the feedback loop lack theoretical or experimental analysis. The feedback mechanism means the guidance scale depends on uncertainty estimates from previous steps, yet the paper does not discuss whether this closed loop remains stable as the number of sampling steps or the guidance strength varies, nor does it provide an analysis of error propagation across steps. An uncharacterized feedback loop may introduce unpredictable behavior.
  8. The error accumulation of the step-skipping strategy, and variance sources beyond the MC sample size MM (e.g., parameter sensitivity of the Laplace approximation), are not ablated.
  9. The paper highlights only the winning settings in its conclusions and offers no mechanistic explanation for the cases where CADS (Tables 2, 4) and CFG++ (Table 3) dominate (e.g., backbone architecture, sampler interaction, guidance-scale sensitivity), making it difficult for readers to judge the boundary of UFG's applicability.
Questions for authors
  1. Are there specific scenarios (datasets, prompt types, guidance-strength ranges) in which UFG (MC) can be clearly recommended over UFG-accel or the baselines? Under what accuracy gains is the 2.9× additional overhead justified?
  2. Have you quantified the correlation between ψ(x0)\psi(x_0) and FID/CLIP? If not, is there direct experimental evidence for the mechanism "reducing one-step variance improves quality" (e.g., reducing variance through an alternative means and observing whether quality changes)?
  3. How large is the impact of the overestimated-variance bias introduced by the Laplace approximation on the closed-form solution (Eq. 19)? How much does the independent-coupling inflation (Sec. 3.2) numerically compensate for?
3 comments

We thank the reviewer for the very detailed review. The comments led to new analyses on the stability of the feedback loop and the sensitivity to the Laplace variance, and to a section that extends the combination with APG in the original appendix. Revisions made in response to the reviewer's comments are marked in blue in the manuscript.

Weakness 1

  • We agree that UFG alone does not outperform CADS and CFG++ in every setting. We think this is because they act on different parts of the guidance. CADS perturbs the condition to improve diversity, and CFG++ modifies the renoising step, whereas UFG decides how a given guidance budget is distributed over pixels.
  • Since these mechanisms are compatible, UFG is guidance-agnostic and can be combined with them. We thank the reviewer for this comment, which led us to move the combination with APG in Appendix E.6 of the original submission to Section 4.5 and add CFG++. Adding UFG to APG on Z-Image and to CFG++ on SDXL further improves the FID of both while CLIP-score and VQAScore stay at the same level, as the table below shows (Table 4 in Section 4.5).

Combining UFG with existing guidance methods on MS-COCO-30K.

BackboneMethodFID ↓\downarrowCLIP-score ↑\uparrowVQAScore ↑\uparrow
Z-ImageAPG11.30090.31850.9029
Z-ImageAPG + UFG-M10.26080.31800.8996
SDXLCFG++13.99860.31970.8723
SDXLCFG++ + UFG-M13.59300.31990.8724
  • Compared with CFG under the same guidance budget, UFG-accel improves FID in all five settings of backbone and resolution in Tables 2, 3 and 6, and on ImageNet 256×256256\times256 the gain is well above the variation across generation seeds (Appendix E.1.3).

Weakness 2

  • We recommend UFG-accel as the default. It costs about 1.2 times CFG, as the table below shows, and UFG (MC) is needed only when the Monte Carlo uncertainty estimate itself is of interest. Both measurements are reported in Appendix E.3.1.

Sampling time per image relative to CFG.

CFGUFG-accelUFG (MC)
SDXL1.00×1.00\times1.23×1.23\times2.91×2.91\times
DiT-XL/21.00×1.00\times1.23×1.23\times3.05×3.05\times
  • To see why UFG-accel sometimes underperforms UFG (MC), we performed a simple experiment on DiT-XL/2, comparing both estimators with a Monte Carlo reference (M=128M=128, 256×256256\times256). Both match the reference for μ^t\hat\mu_t and Σ^t\hat\Sigma_t (correlation above 0.99), and differ only in the cross-covariance C^t\hat C_t, which UFG (MC) estimates with sampling noise and UFG-accel with an approximation error. Hence neither estimator is uniformly better, and which one gives a lower FID depends on the setting, as in Tables 2, 3 and 6.

Weakness 3

  • We thank the reviewer for this comment, which led to Appendix E.4.2. There we run two tests. We multiply the Laplace variance Σpred\Sigma_{\mathrm{pred}} by 0.50.5 and 22 to mimic an underestimate and an overestimate. We also replace the independent coupling with a perfectly correlated one, which treats ϵtc\epsilon_t^c and ϵtu\epsilon_t^u as fully correlated at each pixel.
  • We compare each setting with the original UFG on the same initial noise, using three measures.
    • Map similarity: cosine similarity between the two guidance maps γt\gamma_t.
    • Variance ratio: tr(Var(x0))\mathrm{tr}(\mathrm{Var}(x_0)) of the setting divided by that of the original UFG.
    • Image change: mean absolute pixel difference between the two generated images.

Sensitivity to the Laplace variance (DiT-XL/2, 500 samples).

settingmap similarityvariance ratioimage change
variance ×0.5\times0.50.999×0.50\times0.500.5/255
variance ×2\times20.999×2.00\times2.000.6/255
correlated coupling0.990×0.62\times0.621.9/255
  • As the table above shows, rescaling changes the variance accordingly, but barely changes the guidance map or the samples. This is because γt\gamma_t depends only on how the variance compares across pixels, which rescaling does not change.
  • The coupling mainly changes the reported variance and changes the guidance map only slightly. This test does not cover a bias that varies across pixels, but Appendix E.4.1 replaces the Laplace approximation by an ensemble and still improves FID over CFG, which suggests that UFG does not rely on the specific form of the variance estimate.

Weakness 4

  • We would like to clarify that ψ(x0)\psi(x_0) is not meant to score image quality. Section 4.4 uses it to check whether UFG works as designed, that is, whether any region of the image remains far more uncertain than the rest. Image quality is measured separately by FID and CLIP score, and we relate the two below.
  • Per sample, ψ\psi is indeed only weakly correlated with the CLIP score and the Inception log-probability of the target class on 1,000 DiT-XL/2 samples, and the correlation vanishes within each class, as the table below shows. We think this is because both are global scores of the whole image and barely respond to a defect in a small region. We added this role of ψ\psi to Section 4.4.

Spearman correlation of ψ(x0)\psi(x_0) with per-sample quality proxies on DiT-XL/2.

all sampleswithin class
CLIP-score0.10−0.06-0.06
Inception log-probability of target class0.090.06
  • We did not test this by reducing the variance through another mechanism. However, we can notice the link from the comparison between BayesDiff+CFG and UFG, which is already included in the paper. With the same initial noise and only the allocation of the guidance changed, UFG lowers the uncertainty dispersion on nearly every sample (Figure 3), and UFG-accel also attains a lower FID (Table 2).

Weakness 5

  • We agree that variance minimization acts as a mild contraction. On ImageNet 256×256256\times256 in Table 2, UFG-accel has slightly lower recall and slightly higher precision than CFG.
  • Regarding alignment, UFG keeps the prompt alignment of CFG on 500 highly compositional prompts with SDXL (Appendix E.2.1) and on DPG-Bench with UFG-M (accel) on Z-Image (Appendix E.2.2), as the table below shows. On DPG-Bench, the gap is not significant when paired by prompt.

Prompt alignment of CFG and UFG.

CFGUFG
compositional prompts, CLIP-score0.37360.3748
compositional prompts, VQAScore0.60000.6060
DPG-Bench86.7686.57

Weakness 6

  • We thank the reviewer for the careful reading. We intended x∘2:=xx⊤x^{\circ2}:=xx^\top, so a∘2⊙Σ=diag(a),Σ,diag(a)a^{\circ2}\odot\Sigma=\mathrm{diag}(a)\\,\Sigma\\,\mathrm{diag}(a) is the covariance of diag(a),ϵ\mathrm{diag}(a)\\,\epsilon, and Eq. (15) holds as an identity between d×dd\times d matrices. Our Laplace approximation is diagonal, so in practice Eq. (15) reduces to an elementwise square. We added this explanation below Eq. (15).
  • The factor g(t)2/σtg(t)^2/\sigma_t was missing from Eq. (24) by a typo, not by an approximation. We restored it there and in Eq. (23), and also restored the factor 2 of the cross-covariance term in Eqs. (24) and (30).

Weakness 7

  • We thank the reviewer for this comment, which led to Appendix E.4.3. We tracked the loop at every step on DiT-XL/2 over 200 samples, varying the number of steps TT and the guidance strength λ=1d∑i=1dγt,i\lambda=\frac{1}{d}\sum_{i=1}^{d}\gamma_{t,i}.
  • Under the default λ\lambda and T=250T=250, the loop is stable. All quantities stay finite, tr,Vt\mathrm{tr}\\,V_t grows smoothly, and γt\gamma_t settles after about 40% of the steps.
  • With λ\lambda doubled and T=250T=250, some samples reach max⁡iγt,i>20λ\max_i\gamma_{t,i} > 20\lambda, which we call concentrated. This is possible because λ\lambda fixes only the mean of γt\gamma_t. With the same λ\lambda and T=100T=100, the loop stays stable.
  • To prevent this, we replace γt,i\gamma_{t,i} by γ~t,i=min⁡(γt,i+c,;kλ),\tilde\gamma_{t,i}=\min\left(\gamma_{t,i}+c,\\;k\lambda\right), where the constant c≥0c\ge0 is set so that 1d∑iγ~t,i=λ\frac{1}{d}\sum_{i}\tilde\gamma_{t,i}=\lambda. With k=5k=5, the largest variance drops sharply, as the table below shows.
  • Image change is 1D∥x0(k)−x0(∞)∥1\frac{1}{D}\lVert x_0^{(k)}-x_0^{(\infty)}\rVert_1, where x0(k)x_0^{(k)} is the image under cap level kk, x0(∞)x_0^{(\infty)} is the image without the cap, and DD is the number of pixel values.

Effect of the cap.

λ\lambdacap level kkconcentrated samples (%)largest tr,Var(x0)\mathrm{tr}\\,\mathrm{Var}(x_0)image change
defaultnone0.0290-
default20.02902.0/255
2×2\times defaultnone11.518,130-
2×2\times default50.05181.6/255
  • If max⁡iγt,i≤kλ\max_i\gamma_{t,i}\le k\lambda, then c=0c=0 and γ~t=γt\tilde\gamma_t=\gamma_t, so the cap changes nothing unless some pixel exceeds kλk\lambda. Even with k=2k=2, the images barely change, as the last column shows.
  • Without the cap, the most concentrated sample is visibly degraded, and with the cap it looks normal. The cap can therefore stay on as a safeguard without changing the default behavior of UFG.

Weakness 8

  • Skipping drops the variance terms at the skipped steps, so its error can accumulate. All uncertainty-based methods we compare use the same schedule, so this error does not favor UFG in our paired comparisons, and BayesDiff reports that skipping keeps the images ranked most uncertain (Kou et al., 2024, Figure 2). We leave a direct measurement of the accumulated error to future work.
  • For variance sources beyond MM, changing the Laplace variance or the coupling (Appendix E.4.2) or replacing the Laplace approximation by an ensemble (Appendix E.4.1) does not remove the effect of UFG.

Weakness 9

  • As in our response to W1, CADS and CFG++ act on different parts of the guidance from UFG. CADS improves diversity and attains the highest recall in Table 2, and CFG++ modifies the renoising step. We think this explains where they dominate. It is also why UFG, being guidance-agnostic, can be added on top of them. We revised the abstract and the conclusion to state that UFG improves CFG and can be combined with existing guidance methods.

Question 1

  • As in our response to W2, we recommend UFG-accel in general, since it costs about 1.2 times CFG. UFG (MC) can be preferred when the Monte Carlo uncertainty estimate itself is needed, for example to report the pixel-wise uncertainty of each generated image.

Question 2

  • As in our response to W4, ψ\psi checks whether UFG works as designed rather than image quality. For quality, we compare UFG with a baseline on the same initial noise. On ImageNet, UFG lowers ψ\psi on nearly every sample compared with BayesDiff+CFG (Figure 3), and UFG-accel attains a lower FID (Table 2). On SDXL with MS-COCO-30K, UFG attains a higher CLIP score than CFG on the same initial noise, and the paired difference is significant, as the table below shows (two-sided paired tt-test over the 30K captions).

CLIP-score paired with CFG on SDXL.

CLIP-scorepaired difference vs. CFG (95% CI)pp-values
CFG0.3197--
UFG-accel0.3203+0.00061+0.00061 [0.000500.00050, 0.000730.00073]2×10−262\times10^{-26}
UFG-M (accel)0.3203+0.00063+0.00063 [0.000520.00052, 0.000740.00074]1×10−271\times10^{-27}

Question 3

  • As in our response to W3, a global bias of the Laplace variance has little effect on γt\gamma_t. Rescaling the variance by 0.5 or 2 leaves γt\gamma_t almost the same, because γt\gamma_t depends only on the relative variance across pixels. The independent coupling makes the variance about 1.6 times larger than a perfectly correlated coupling, and γt\gamma_t again barely changes.

Final remark

We thank the reviewer for the thorough comments. One concern was that a biased Laplace variance could undermine the guidance, and our new analysis shows that such a bias barely changes γt\gamma_t. Our contribution is to decide where to apply the guidance from pixel-wise uncertainty. This choice is independent of how the guidance update is formed, so UFG can be added to other guidance methods, and it lowers the FID of APG on Z-Image and of CFG++ on SDXL. We are grateful for the reviewer's thoughtful advice.

Reviewer_KKu8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Uncertainty-adaptive Feedback Guidance (UFG) for improving diffusion-based image generation. Instead of using a single global CFG scale, UFG assigns pixel-wise guidance strengths based on estimated uncertainty and optimizes them to reduce the marginal variance at each denoising step. The updated uncertainty is then used to determine the guidance at the following step, forming a feedback process during sampling. The method is evaluated on class-conditional and text-to-image generation, and the experiments show that it generally improves generation quality and reduces localized artifacts compared with existing guidance methods.

Critical strengths

This paper proposed a novel guidance mechanism for diffusion models, so-called Uncertainty-adaptive Feedback Guidance (UFG) . In contrast to commonly used classifier-free guidance (CFG) , it can adaptively determines the pixel-wise guidance scale to capture the complex dynamics during the generation process. Therefore, it can improve the image generation quality and mitigates visual artifacts.

Critical weaknesses

One concern is the choice to directly set the UFG budget parameter  to the CFG scale . These two parameters play different roles in the two guidance schemes, so it is not obvious that they should use the same value. Since the performance of UFG depends on, the paper would benefit from a clearer justification for this design choice.

Questions for authors

The paper directly sets the UFG budget parameter  equal to the CFG scale , although these two parameters serve different roles in the method. The justification for this choice is currently limited. Could the authors provide empirical evidence showing that setting = is consistently reasonable across different backbones and tasks?

1 comment

We thank the reviewer for the positive evaluation and for the question on the choice of λ\lambda, which led to a new sweep of λ\lambda on a text-to-image backbone. Revisions made in response to the reviewer's comments are marked in blue in the manuscript.

Weakness

  • We thank the reviewer for this question. We would like to clarify that λ=ω\lambda=\omega is the natural choice under our formulation. The constraint in Eq. (18) bounds the average guidance over pixels by λ\lambda, and CFG with scale ω\omega corresponds to the uniform allocation s=ω1s=\omega\mathbf{1}. Hence λ=ω\lambda=\omega compares UFG and CFG under the same total amount of guidance, and the only difference is how UFG distributes it across pixels.
  • We use λ=ω\lambda=\omega as a default that needs no extra tuning. On U-ViT in Appendix D.1, λ=ω\lambda=\omega is already close to the best value at both resolutions.

Question

  • We thank the reviewer for this suggestion, which led to a new sweep of λ\lambda on Z-Image in Appendix D.1. With the same prompts and initial noise for all runs, λ\lambda moves UFG along a trade-off between fidelity and prompt alignment, in the same way as the CFG scale, as the table below shows.

Effect of λ\lambda on Z-Image with ω=4\omega=4 (5K MS-COCO prompts).

λ/ω\lambda/\omegaFID-5KVQAScoredifference in VQAScore vs. λ=ω\lambda=\omega (95% CI)
UFG-M0.5015.890.8981−0.0078-0.0078 [−0.0109-0.0109, −0.0047-0.0047]
UFG-M0.7517.780.9038−0.0021-0.0021 [−0.0045-0.0045, 0.00030.0003]
UFG-M1.0018.860.9059-
UFG-M1.2519.650.9081+0.0022+0.0022 [0.00020.0002, 0.00430.0043]
CFG-19.790.9071+0.0012+0.0012 [−0.0010-0.0010, 0.00350.0035]
  • At λ=ω\lambda=\omega, UFG attains a lower FID than CFG at the same VQAScore, and a slightly larger λ\lambda outperforms CFG on both metrics. Hence λ=ω\lambda=\omega is a reasonable default across backbones and tasks, and λ\lambda can be adjusted when fidelity or alignment is preferred.

  • We thank the reviewer again for the constructive comments. We hope that our responses have addressed the reviewer's concerns, and we would be happy to answer any further questions.

Reviewer_hHRfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes to adopt a local scheduling of CFG based on a measure of uncertainty. The uncertainty is approximated by a Gaussian model of the diffusion estimate and then incorpora ted into CFG. Minimizing this uncertainty leads to a closed form solution of the guidance weight. The Gaussian approximation can be accelerated using a Taylor expansion of the noise predictor rather than Monte Carlo estimation. Experiments are carried out on class conditional and text-to-image generation.

Critical strengths

While the idea to link CFG with uncertainty is not new, the proposed approximation is sound and the derivation gets rid of the costly MC sampling.

Critical weaknesses

The experiments are unconvincing. First the models used are not on par with the state of the art. I see no reason why the same modeling could not be made for more recent flow matching methods (you can recover a noise-based SDE) which would make the setup more up-to-date. Second, the scores are well within the noise level of sampling effects on FID. The teaser looks good compared to the baseline, but Fig.4 shows no significant difference compared to a related method and Fig.11 is not dramatic either. For the T2I images in appendix, the effect are subtle. I am not sure a user study would always favor the method. Finally, for numerical scores, given that the method are more degrees of freedom, it may just be over-fitting can can happen with parametrized CFG as shown in [34].

Questions for authors

Could you update the method and the scores for more recent models (typically JiT for class condition and SD3.5 for T2I) and show that the method is a significant improvement over other CFG variants? Also, if this fixes artifacts, then scores on benchmarks like geneval or DPGBench should be improved as well. Since this method does not require training, I expect that it would be easy to do.

3 comments

We thank the reviewer for the careful review and for the suggestions on recent backbones and other CFG variants. Revisions made in response to the reviewer's comments are marked in blue in the manuscript.

Weakness 1

  • We would like to note that our text-to-image experiments already include Z-Image (2025), a flow-matching model released after SD3.5 (Section 4.3, Table 3). UFG keeps the diffusion model frozen and changes only the guidance term, with a last-layer Laplace approximation fitted once, so it is not tied to a particular model or sampler. Appendix A.2 derives the moment recursions for several samplers, and Z-Image is sampled with a flow-matching Euler sampler.
  • In addition, we show that UFG can be combined with recent guidance methods and further improves them, as detailed in our response to Q1.

Weakness 2

  • We thank the reviewer for this concern, which led to Appendix E.1.3. We repeated the class-conditional experiment on ImageNet 256×256256\times256 with three generation seeds. The gap between CFG and UFG is about three times the standard deviation of CFG across seeds, as the table below shows, so the improvement is not due to sampling noise.

FID over three generation seeds on ImageNet 256×256256\times256.

seed 1seed 2seed 3mean ±\pm std
CFG3.0743.1683.2133.152 ±\pm 0.071
UFG-accel2.8832.9152.9342.911 ±\pm 0.026
UFG-M (accel)2.8902.9152.9272.911 ±\pm 0.019
  • On Z-Image, we also compared UFG and CFG on 5K MS-COCO prompts with the same initial noise. At λ=ω\lambda=\omega, UFG lowers FID from 19.79 to 18.86, while the change in VQAScore is not significant. Details are in Appendix D.1.
  • We agree that the visual differences are often local, since UFG mainly suppresses artifacts in highly uncertain regions, as in Figure 4. We appreciate the reviewer's point on a user study, which would give a useful complementary view of such local differences. In this response, we focused on strengthening the quantitative evidence, with multiple seeds and paired comparisons under the same initial noise.

Weakness 3

  • We would like to clarify that UFG does not add parameters tuned on the evaluation metric. The pixel-wise scale γt\gamma_t is the closed-form solution of Proposition 3.1 at every step, and its only user-set input, the budget λ\lambda, is set to the CFG scale ω\omega. Hence UFG has the same tuning burden as CFG, and the Monte Carlo size MM and the step-skipping schedule follow BayesDiff without tuning on FID.
  • We believe that [34] in the review refers to a work on parametrized CFG schedules such as Wang et al. (2024), since [34] of our paper is Sohl-Dickstein et al. (2015).

Question 1

  • As in our response to W1, Z-Image is a recent flow-matching model released after SD3.5.
  • Regarding other CFG variants, we thank the reviewer for this question, which led us to move the combination with APG in Appendix E.6 of the original submission to Section 4.5 and add CFG++. Since UFG only decides where the guidance is applied over pixels, it is guidance-agnostic and can be combined with methods that change how the guidance update is formed. Adding UFG to APG on Z-Image and to CFG++ on SDXL further improves the FID of both while CLIP-score and VQAScore stay at the same level, as the table below shows (Table 4 in Section 4.5).

Combining UFG with existing guidance methods on MS-COCO-30K.

BackboneMethodFID ↓\downarrowCLIP-score ↑\uparrowVQAScore ↑\uparrow
Z-ImageAPG11.30090.31850.9029
Z-ImageAPG + UFG-M10.26080.31800.8996
SDXLCFG++13.99860.31970.8723
SDXLCFG++ + UFG-M13.59300.31990.8724

Question 2

  • We thank the reviewer for this suggestion, which led to Appendix E.2.2. We evaluated DPG-Bench with its official scorer on Z-Image, with the same prompts and initial noise for CFG and UFG. As the table below shows, the paired difference over prompts is not significant.

DPG-Bench on Z-Image.

DPG-Benchpaired difference vs. CFG (95% CI)
CFG86.76-
UFG-M (accel)86.57−0.19-0.19 [−0.51-0.51, 0.120.12]
  • DPG-Bench asks whether the objects, attributes and relations in a prompt appear in the image, and does not directly measure the local artifacts that UFG suppresses. Hence the result shows that UFG keeps the dense prompt following of CFG while it lowers FID in Table 3.

Final remark

We thank the reviewer for the constructive comments. Our aim is to decide where to apply the guidance according to pixel-wise uncertainty. Since this choice does not depend on how the guidance update is formed, UFG can be used on top of existing and future guidance methods, which we believe is its main practical strength. We are grateful for the reviewer's thoughtful advice.

Thanks for the answers. I'll discuss this with the other reviewers and the AC and then modify my score accordingly.

We sincerely appreciate your time and careful review. Your comments helped us improve the paper, and we have reflected them in the revision. Please let us know if anything remains unclear.

Comments on the paper

1 comment

We thank all reviewers for the careful reviews and constructive comments. We have uploaded a revised manuscript in which all changes are marked in blue. All section, appendix, table, figure and equation numbers in our responses refer to the revised manuscript. The main changes are summarized below.

  • Combination with existing guidance methods (Section 4.5). Since UFG only decides where the guidance is applied over pixels, it is guidance-agnostic and can be combined with other guidance methods. Adding UFG to APG on Z-Image and to CFG++ on SDXL further improves their FID while keeping their prompt alignment. The APG result was in Appendix E.6 of the original submission, and we strengthened it with CFG++ and moved it to the main text. (Reviewers hHRf, sgrG)
  • Multiple seeds (Appendix E.1.3). The FID gain of UFG over CFG on ImageNet 256×256256\times256 is well above the variation across generation seeds. (Reviewer hHRf)
  • Choice of λ\lambda (Appendix D.1). A sweep of λ\lambda on Z-Image shows that λ=ω\lambda=\omega improves FID over CFG at the same prompt alignment. (Reviewer KKu8)
  • DPG-Bench (Appendix E.2.2). UFG keeps the dense prompt following of CFG. (Reviewer hHRf)
  • Stability of the feedback loop (Appendix E.4.3). The loop is stable at the default guidance strength, and a simple cap keeps the guidance from concentrating at a larger strength. (Reviewer sgrG)
  • Sensitivity to the Laplace variance (Appendix E.4.2). A global bias of the variance barely changes the guidance map. (Reviewer sgrG)
  • Notation and typos (Eq. (15), Appendix A.1). We clarified the notation and fixed the typos in the derivation. (Reviewer sgrG)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.