- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Uncertainty-adaptive Feedback Guidance (UFG), a guidance mechanism for conditional diffusion models. Its core contribution is to replace the global, fixed classifier-free guidance (CFG) scale with a pixel-wise adaptive guidance scale derived from Bayesian variance estimation (BayesDiff). Specifically, at each denoising step, the pixel-wise scale is given by the closed-form solution (Proposition 3.1, Eq. 19) of a constrained one-step marginal variance minimization problem (Eq. 18); since the guidance modifies the uncertainty estimate of the next step, a feedback loop emerges across steps (Fig. 2). Method variants include UFG-M, UFG-accel (approximately 1.2× the cost of CFG), and a step-skipping strategy. For validation, the paper reports FID, Precision/Recall, CLIP-score, and VQAScore on class-conditional ImageNet (256/512, U-ViT and DiT-XL/2) and text-to-image generation (SDXL, Z-Image, SD1.5), together with an uncertainty-dispersion analysis and qualitative comparisons.
Critical strengths
-
A theoretically grounded closed-form pixel-wise guidance rule. The variance-minimization problem (Eq. 18) is cleanly formulated and admits a unique closed-form solution (Proposition 3.1), with a complete KKT derivation in Appendix C; the decomposition of the one-step variance into is concise and highly interpretable.
-
A conceptual feedback-loop framework with direct diagnostic evidence. The paper clearly articulates how guidance feeds back into the next-step uncertainty estimate (Fig. 2), and introduces the uncertainty-dispersion metric as a per-sample diagnostic (Fig. 3), supported by visualizations of artifact suppression in high-uncertainty regions (Figs. 4, 19–22). This provides a testable mechanistic explanation and is valuable for understanding CFG-like guidance behavior.
Critical weaknesses
- The empirical gains are small and inconsistent, leaving the central claim unsubstantiated. In several settings the baselines actually win: on ImageNet 512×512, CADS achieves the best FID (4.3587, vs. 4.3728 for UFG-M and 4.5532 for UFG-accel); on SDXL, CFG++ achieves the best FID (13.9986, vs. 14.0395 for UFG-M); on DiT-XL/2, CADS has the lowest FID (1.9697, vs. 2.2492 for UFG-accel), and UFG (MC) is even slightly worse than CFG on FID (2.3552 vs. 2.2863). The claimed superiority simply does not hold across backbones and metrics, which is the most direct factor affecting my recommendation.
- Insufficient analysis of the computational overhead trade-off, raising doubts about practical usability. UFG (MC) is about 2.9× slower than CFG (Table 6), yet the paper neither specifies when the accuracy gains justify this overhead, nor explains why UFG-accel sometimes underperforms UFG (MC) (Table 2, 512×512). For a practically oriented guidance method, the lack of a clear decision guideline is a substantive deficiency.
- Reliance on BayesDiff's Laplace approximation, with no quantitative analysis of how its bias is mitigated. The method directly inherits BayesDiff's approximation and Monte Carlo machinery; the paper itself acknowledges the bias in variance estimation (Sec. 3.2), but relies solely on "independent coupling" as a remedy without quantifying how much of the bias this inflation compensates. Since the closed-form solution (Eq. 19) takes this variance estimate as input, estimation errors propagate into the guidance scale, undermining confidence in the core mechanism.
- The causal chain "variance reduction → quality improvement" is never directly verified. This assumption is largely transplanted from BayesDiff's observations (Sec. 3.1); no controlled experiment is conducted (e.g., reducing variance through an alternative mechanism and checking whether quality improves accordingly); and the dispersion metric measures the spatial uniformity of variance rather than image quality, with its correlation to FID/CLIP unquantified. The core mechanism is asserted more than demonstrated.
- Failure modes in which variance minimization may harm diversity or alignment are not discussed. Reducing one-step variance is essentially a contraction toward low-entropy solutions; the paper does not analyze whether this contraction produces side effects for highly diverse prompts, long-tail concepts, or scenarios requiring precise semantic alignment (the compositional prompts in Appendix E.9 cover only limited cases).
- Notation and derivation inconsistencies. (outer product) is defined below Eq. 15, yet the notation is used elementwise in Eq. 15, creating ambiguity; the derivation from Eq. 12 to Eq. 15 mixes and inconsistently, and the derivation of should be checked for dimensional consistency; in Appendix A.1, Eq. 24 omits the factor relative to Eq. 23 without stating the neglected terms and their order. The rigor of the core probabilistic derivation directly affects trust in the variance-minimization objective.
- The stability and convergence of the feedback loop lack theoretical or experimental analysis. The feedback mechanism means the guidance scale depends on uncertainty estimates from previous steps, yet the paper does not discuss whether this closed loop remains stable as the number of sampling steps or the guidance strength varies, nor does it provide an analysis of error propagation across steps. An uncharacterized feedback loop may introduce unpredictable behavior.
- The error accumulation of the step-skipping strategy, and variance sources beyond the MC sample size (e.g., parameter sensitivity of the Laplace approximation), are not ablated.
- The paper highlights only the winning settings in its conclusions and offers no mechanistic explanation for the cases where CADS (Tables 2, 4) and CFG++ (Table 3) dominate (e.g., backbone architecture, sampler interaction, guidance-scale sensitivity), making it difficult for readers to judge the boundary of UFG's applicability.
Questions for authors
- Are there specific scenarios (datasets, prompt types, guidance-strength ranges) in which UFG (MC) can be clearly recommended over UFG-accel or the baselines? Under what accuracy gains is the 2.9× additional overhead justified?
- Have you quantified the correlation between and FID/CLIP? If not, is there direct experimental evidence for the mechanism "reducing one-step variance improves quality" (e.g., reducing variance through an alternative means and observing whether quality changes)?
- How large is the impact of the overestimated-variance bias introduced by the Laplace approximation on the closed-form solution (Eq. 19)? How much does the independent-coupling inflation (Sec. 3.2) numerically compensate for?
3 comments
Weakness 4
- We would like to clarify that is not meant to score image quality. Section 4.4 uses it to check whether UFG works as designed, that is, whether any region of the image remains far more uncertain than the rest. Image quality is measured separately by FID and CLIP score, and we relate the two below.
- Per sample, is indeed only weakly correlated with the CLIP score and the Inception log-probability of the target class on 1,000 DiT-XL/2 samples, and the correlation vanishes within each class, as the table below shows. We think this is because both are global scores of the whole image and barely respond to a defect in a small region. We added this role of to Section 4.4.
Spearman correlation of with per-sample quality proxies on DiT-XL/2.
| all samples | within class | |
|---|---|---|
| CLIP-score | 0.10 | |
| Inception log-probability of target class | 0.09 | 0.06 |
- We did not test this by reducing the variance through another mechanism. However, we can notice the link from the comparison between BayesDiff+CFG and UFG, which is already included in the paper. With the same initial noise and only the allocation of the guidance changed, UFG lowers the uncertainty dispersion on nearly every sample (Figure 3), and UFG-accel also attains a lower FID (Table 2).
Weakness 5
- We agree that variance minimization acts as a mild contraction. On ImageNet in Table 2, UFG-accel has slightly lower recall and slightly higher precision than CFG.
- Regarding alignment, UFG keeps the prompt alignment of CFG on 500 highly compositional prompts with SDXL (Appendix E.2.1) and on DPG-Bench with UFG-M (accel) on Z-Image (Appendix E.2.2), as the table below shows. On DPG-Bench, the gap is not significant when paired by prompt.
Prompt alignment of CFG and UFG.
| CFG | UFG | |
|---|---|---|
| compositional prompts, CLIP-score | 0.3736 | 0.3748 |
| compositional prompts, VQAScore | 0.6000 | 0.6060 |
| DPG-Bench | 86.76 | 86.57 |
Weakness 6
- We thank the reviewer for the careful reading. We intended , so is the covariance of , and Eq. (15) holds as an identity between matrices. Our Laplace approximation is diagonal, so in practice Eq. (15) reduces to an elementwise square. We added this explanation below Eq. (15).
- The factor was missing from Eq. (24) by a typo, not by an approximation. We restored it there and in Eq. (23), and also restored the factor 2 of the cross-covariance term in Eqs. (24) and (30).
Weakness 7
- We thank the reviewer for this comment, which led to Appendix E.4.3. We tracked the loop at every step on DiT-XL/2 over 200 samples, varying the number of steps and the guidance strength .
- Under the default and , the loop is stable. All quantities stay finite, grows smoothly, and settles after about 40% of the steps.
- With doubled and , some samples reach , which we call concentrated. This is possible because fixes only the mean of . With the same and , the loop stays stable.
- To prevent this, we replace by where the constant is set so that . With , the largest variance drops sharply, as the table below shows.
- Image change is , where is the image under cap level , is the image without the cap, and is the number of pixel values.
Effect of the cap.
| cap level | concentrated samples (%) | largest | image change | |
|---|---|---|---|---|
| default | none | 0.0 | 290 | - |
| default | 2 | 0.0 | 290 | 2.0/255 |
| default | none | 11.5 | 18,130 | - |
| default | 5 | 0.0 | 518 | 1.6/255 |
- If , then and , so the cap changes nothing unless some pixel exceeds . Even with , the images barely change, as the last column shows.
- Without the cap, the most concentrated sample is visibly degraded, and with the cap it looks normal. The cap can therefore stay on as a safeguard without changing the default behavior of UFG.
Weakness 8
- Skipping drops the variance terms at the skipped steps, so its error can accumulate. All uncertainty-based methods we compare use the same schedule, so this error does not favor UFG in our paired comparisons, and BayesDiff reports that skipping keeps the images ranked most uncertain (Kou et al., 2024, Figure 2). We leave a direct measurement of the accumulated error to future work.
- For variance sources beyond , changing the Laplace variance or the coupling (Appendix E.4.2) or replacing the Laplace approximation by an ensemble (Appendix E.4.1) does not remove the effect of UFG.
Weakness 9
- As in our response to W1, CADS and CFG++ act on different parts of the guidance from UFG. CADS improves diversity and attains the highest recall in Table 2, and CFG++ modifies the renoising step. We think this explains where they dominate. It is also why UFG, being guidance-agnostic, can be added on top of them. We revised the abstract and the conclusion to state that UFG improves CFG and can be combined with existing guidance methods.
Question 1
- As in our response to W2, we recommend UFG-accel in general, since it costs about 1.2 times CFG. UFG (MC) can be preferred when the Monte Carlo uncertainty estimate itself is needed, for example to report the pixel-wise uncertainty of each generated image.
Question 2
- As in our response to W4, checks whether UFG works as designed rather than image quality. For quality, we compare UFG with a baseline on the same initial noise. On ImageNet, UFG lowers on nearly every sample compared with BayesDiff+CFG (Figure 3), and UFG-accel attains a lower FID (Table 2). On SDXL with MS-COCO-30K, UFG attains a higher CLIP score than CFG on the same initial noise, and the paired difference is significant, as the table below shows (two-sided paired -test over the 30K captions).
CLIP-score paired with CFG on SDXL.
| CLIP-score | paired difference vs. CFG (95% CI) | -values | |
|---|---|---|---|
| CFG | 0.3197 | - | - |
| UFG-accel | 0.3203 | [, ] | |
| UFG-M (accel) | 0.3203 | [, ] |
Question 3
- As in our response to W3, a global bias of the Laplace variance has little effect on . Rescaling the variance by 0.5 or 2 leaves almost the same, because depends only on the relative variance across pixels. The independent coupling makes the variance about 1.6 times larger than a perfectly correlated coupling, and again barely changes.
Final remark
We thank the reviewer for the thorough comments. One concern was that a biased Laplace variance could undermine the guidance, and our new analysis shows that such a bias barely changes . Our contribution is to decide where to apply the guidance from pixel-wise uncertainty. This choice is independent of how the guidance update is formed, so UFG can be added to other guidance methods, and it lowers the FID of APG on Z-Image and of CFG++ on SDXL. We are grateful for the reviewer's thoughtful advice.
We thank the reviewer for the very detailed review. The comments led to new analyses on the stability of the feedback loop and the sensitivity to the Laplace variance, and to a section that extends the combination with APG in the original appendix. Revisions made in response to the reviewer's comments are marked in blue in the manuscript.
Weakness 1
Combining UFG with existing guidance methods on MS-COCO-30K.
Weakness 2
Sampling time per image relative to CFG.
Weakness 3
Sensitivity to the Laplace variance (DiT-XL/2, 500 samples).