acceptodds
Under review as a conference paper at ICLR 2027

Let Attention Flow: Energy-Dissipating Guidance for Diffusion Models

Abstract

Diffusion models achieve remarkable generative performance by iteratively refining noisy samples through a sequence of denoising operations. An important component driving this refinement is the attention mechanism within the learned parameterization, which governs how information flows across spatial locations, semantic tokens and intermediate features. Existing training-free guidance methods often rely on score-space corrections or heuristic changes to internal activations, such as blurring attention maps. We instead guide diffusion by evolving attention through a Partial Differential Equation (PDE) inspired by Landau–de Gennes (LdG) dynamics. This energy-dissipating flow suppresses unstable attention patterns while preserving directional structure, providing a principled way to modulate the model’s internal information flow without retraining. The resulting guidance integrates cleanly with modern text-to-image samplers, requires no architectural modification, and improves generation quality both qualitatively and quantitatively. On MS-COCO, our method improves over SDXL and prior attention-based guidance methods. Under 4-bit model quantization, it improves generation quality and outperforms the unguided FP16 SDXL baseline across all reported metrics.

Primary area:
generative models
Keywords:
Diffusion Models, PDE

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9TPrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Existing interventions choose the perturbation operator by trial-and-error such as blurring, masking, shuffling, or replacing attention. Instead, this paper investigates the statement 'Can training-free diffusion guidance be improved by replacing heuristic attention perturbations with principled, energy-minimizing dynamics?' They propose to use a PDE inspired by Landau–de Gennes (LdG) theory a free-energy framework for fields with directional order. On MS-COCO, the method improves over SDXL and SEG.

Critical strengths
  • the paper proposes a new theory, possibly more principled, the method is complicated, but the authors do reasonable job in motivating and explaining the proposed method. The division in anisotropic and isotropic parts which are separately operated on by the PDE seems plausible.

  • results on COCO and especially those on quantized networks show improved results with respect to SEG (NeurIPS 2024) (I need however more clarifications (see below) to value these results)

Critical weaknesses
  • The quantitative results are few, the authors only compare with SEG; in the introduction the method is explained to compete with self-attention guidance, perturbed-attention guidance , smoothed energy guidance and more recent token-level perturbation methods (TPG) but in the end they only compare to SEG. Can the authors explain why, and/or add more results. Could a comparison with just CFG be added? And similarly as the TPG paper, could you include results for conditioned and unconditioned (which showed superiority of CFG for conditioned) ?

  • The work is written for people with knowledge of SEG, PAG research line. However these are not highly cited works. For the more general DM-research community, I think the paper presentation could still be improved of multiple points (see comments below). Clear impact statement, scope (models trained unconditionally), further analysis of limitations of current methods (problem statement).

Questions for authors

In the SEG paper it is clearly stated that the focus is on unconditional models; in your paper the word unconditional only shows up in the conclusion. This should be clear from early on in the paper if that is the scope.

The motivation why 'Classifier-free guidance' remains an important research topic could be improved.

I think that a better explanation or analysis how 'Gaussian blur used in prior attention-smoothing approaches treat all directions uniformly and can attenuate useful anisotropic structure, producing overregularization, flat texture, and weaker spatial coherence when guidance is strong. ' could still strengthen the paper. The paper says a few times it is doing similar as other methods but more principled, which I think is fine, but I think it is stronger if you can also improve the analysis why other methods potentially fail (problem-> solution structure).

minor: The authors say 'We propose to act on the attention tensor....' , this statement seems to suggest that also other choices could have been made. What other locations could have been chosen ?

Could the authors elaborate on this 'This view also aligns with recent stochastic-thermodynamic perspectives on generative modeling...'.

5 comments

Dear Reviewer,

We thank you for taking the time and effort to review our work and providing valuable feedback. We appreciate the positive view of the formulation and of the quantized results, and seek to clarify the concerns. A summary of all new results, with the exact references for every baseline, is in our general response. Unless stated otherwise, all new results are on MS-COCO with 10K images, with identical captions and seeds across methods.

1. Comparison beyond SEG, including CFG

Thank you for the suggestion. We now compare against SAG, PAG, SEG, TPG and CFG, with every method's scale chosen from the same 1K sweep.

Without CFG, LdG is best on CLIP, ImageReward, Aesthetic, PickScore and HPSv2. For example, its ImageReward is 0.239, against -0.088 (SEG), -0.170 (PAG), -0.129 (TPG) and -0.703 (SAG). Under 4-bit quantization it is best on all six metrics. With CFG (w=3):

SDXL, CFG onFID↓CLIP↑ImageReward↑Aesthetic↑PickScore↑HPSv2↑
CFG only15.710.64360.4275.42021.870.2489
+ SEG16.550.64020.4825.54822.080.2621
+ PAG16.350.64310.4895.52122.160.2615
+ TPG15.650.64580.5115.50122.120.2613
+ SAG17.650.64020.4515.55921.980.2625
+ LdG16.480.64310.5435.55122.240.2683

LdG gives the best ImageReward, PickScore and HPSv2. More importantly, CFG and LdG are two different forms of guidance, one along the text condition and one along the spatial structure of attention, and they combine well. For every CFG scale from 1.5 to 7.5, adding LdG raises ImageReward, Aesthetic, PickScore and HPSv2. For w ≤ 3 it also lowers FID, from 63.00 to 51.71 at w=1.5 and from 54.65 to 50.24 at w=2.

2. Conditional and unconditional results

Following the TPG protocol, we also ran the unconditional setting with an empty prompt:

UnconditionalnoneSEGPAGTPGLdG
FID↓126.7386.98103.1279.2782.24
Aesthetic↑5.1945.7635.6195.6555.805

LdG gives the best Aesthetic score and reduces FID by 35% over no guidance, second only to TPG.

With this, LdG now covers conditional generation without CFG, conditional generation with CFG, and unconditional generation. As the TPG paper also observed, CFG remains strong for conditional FID. LdG adds to CFG rather than competing with it.

3. Scope and Contributions of the paper

We agree, and thank the reviewer for this. We have made the scope and the contributions explicit at the end of the introduction and in Sec. 6. The paper makes two contributions. First, LdG is a principled guidance mechanism: it perturbs the attention of a pretrained diffusion model along the descent direction of a Landau–de Gennes free energy, rather than by a heuristic blur, mask or shuffle. It applies to text-conditional generation with and without CFG and to unconditional generation, on UNet, MMDiT and guidance-distilled models. Second, as a practical application, LdG is especially effective for 4-bit quantized models: under INT4 it is the best method on all six metrics, and it loses the least quality from FP16 to INT4 on every metric. We have also corrected the phrase "unconditional guidance strategies" in the conclusion (L477), which should read "attention-perturbation guidance strategies".

4. Why CFG-related guidance remains important

Thank you for the suggestion. CFG requires a conditional model and a second, unconditional pass, it degrades at high guidance scales, and it is not available for guidance-distilled models. Attention-perturbation guidance avoids all three: it needs no conditioning, it adds to CFG at every scale, and it works on guidance-distilled models. On FLUX.1-dev, where true CFG is not used, LdG lowers FID from 26.62 to 25.88 and improves ImageReward, Aesthetic, PickScore and HPSv2. We have added this motivation to the introduction.

5. Why Gaussian blur falls short

This is a good point, and we have restructured the introduction and Sec. 4. SEG blurs the attention queries, and its guidance is strongest when σ→∞. In that limit every query is replaced by the global mean, which removes all spatial structure in the perturbed branch, not only the unstable part. The LdG step, by contrast, is local: it damps high-curvature components and keeps the large-scale layout. The data support this directly. With the same extra branch, replacing SEG's blur by the LdG step improves every metric in every setting we tested:

SEG → LdGFID↓ImageReward↑Aesthetic↑
without CFG27.64 → 26.72-0.088 → 0.2395.704 → 5.791
with CFG (w=3)16.55 → 16.480.482 → 0.5435.548 → 5.551
INT4, without CFG27.10 → 23.93-0.215 → 0.1795.575 → 5.721
unconditional86.98 → 82.24–5.763 → 5.805

We have added this comparison next to Fig. 4.

6. Other locations where guidance could act

Thanks for the question. Guidance can act at several places: on the score or latent (CFG), on intermediate features (SAG, which blurs the regions selected by its attention mask), on the attention map (PAG, which replaces it with the identity), on the tokens (TPG, which shuffles them), or on the attention queries (SEG and LdG). We act on the attention field because it is a spatial field over image positions. The Laplacian and the energy then have a direct meaning there, and the update is cheap. We have added this to Sec. 3.

7. The stochastic-thermodynamics sentence

We thank the reviewer for asking us to elaborate. Recent work (e.g. Welling et al., 2026) views diffusion sampling as a non-equilibrium process organized by free-energy dissipation and gradient-flow structure. Our update is a gradient flow of a free energy on the attention field, so it fits this view at the level of internal representations. The method does not depend on this connection, and we have rephrased the sentence to present it as context rather than a claim.

8. Presentation and impact statement

We appreciate the reviewer's comment about presentation. We have added a short list of contributions and the scope at the end of the introduction, and a brief impact paragraph in the conclusion. We have also made the problem statement on isotropic smoothing (point 5 above) explicit. We believe this will make the paper more accessible to the broader diffusion-model community.

If the clarification(s) above address the reviewer's concerns, we will promptly upload the revised PDF, which incorporates all of these changes marked in blue.

Dear Reviewer, we thank you again for your time and valuable feedback. As the discussion period is coming to an end, we wanted to check whether our responses have addressed all your concerns. We would be grateful for any further feedback, and we are happy to clarify anything that remains unclear.

Reviewer_tUDWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free guidance method for diffusion models by evolving attention maps through a PDE inspired by Landau–de Gennes dynamics. The proposed energy-dissipating flow aims to suppress unstable attention patterns while preserving directional structure, providing a principled alternative to heuristic attention manipulation. The method requires no retraining or architectural modification and can be integrated with existing text-to-image samplers. Experiments on MS-COCO and under 4-bit quantization show improvements over SDXL and prior attention-based guidance methods.

Critical strengths
  1. The approach can be directly applied to pretrained models and improves generation through inference-time guidance without updating model parameters.

  2. In the 4-bit SDXL experiments, it improves multiple metrics over both the unguided baseline and SEG, demonstrating its potential for deployment in memory-constrained settings.

Critical weaknesses
  1. The main text defines an energy over the attention matrix, the quantization experiments modify the attention logits, while the appendix ablation applies the update to the Query tensor. A unified explanation of these different operations is missing, making it difficult to assess whether the theoretical formulation truly matches the actual implementation.

  2. A decrease in the internal attention energy does not directly guarantee that the overall denoising trajectory becomes more stable or that generation quality improves. The paper does not provide sufficient justification for the connection between these two aspects.

  3. The main experiments focus primarily on SDXL and COCO, while the quantitative comparisons are largely against SEG. Comparisons with well-tuned CFG, PAG, and other guidance methods, as well as evaluations on additional model architectures, would make the empirical study more convincing.

  4. The method requires both the original and perturbed prediction branches, together with additional attention updates, yet the paper does not report inference latency, memory consumption, or performance under a matched computational budget.

  5. The fact that INT4 + LdG outperforms the unguided FP16 baseline may mainly reflect the benefit of the guidance itself. Results for FP16 + LdG should be included to determine whether the method provides a specific advantage in mitigating quantization errors.

Questions for authors

Please refer to weakness.

4 comments

Dear Reviewer,

We thank you for taking the time and effort to carefully review our work and providing valuable feedback. We appreciate the positive comments on the training-free nature of LdG and its value in memory-constrained settings, and seek to clarify the concerns. A summary of all new results, with the exact references for every baseline, is in our general response. Unless stated otherwise, all new results are on MS-COCO with 10K images, with identical captions and seeds across methods.

1. Unified explanation of the attention energy, the logits and the Query

Thank you for pointing this out, as it helps us state the method more precisely. These are not three different operations; they are one operation viewed at three levels, and we have now said this once, explicitly, in Sec. 5.

In the implementation, the update acts on the query field QQ of the perturbed branch (SDXL mid-block self-attention), and the keys and values are unchanged. The attention logits S=QK⊤/dS = QK^\top/\sqrt{d} are linear in QQ, so a Laplacian step on QQ over query positions is exactly the same step on the logits: (ΔQ)K⊤/d=ΔqS(\Delta Q)K^\top/\sqrt{d} = \Delta_q S. This is the quantity the quantization experiments modify. The attention matrix is then the row-wise softmax of these logits.

The bulk terms use a=c=10−6a = c = 10^{-6} and are negligible in practice, so the step is the elastic term of Eq. 22. The theory and the implementation therefore describe the same update. All new results use the pointwise operator of Eq. 22, and we have updated L300 and Table 1 accordingly.

2. Lower attention energy and better sampling

We agree with the reviewer that a lower attention energy does not by itself guarantee a better denoising trajectory. Eq. 11 is a first-order link, and we now present it as motivation rather than a guarantee. To make the connection concrete, we added two pieces of evidence.

First, we measured what the update does to the energy at every call during sampling. The energy decreases exactly when the step satisfies ηλ≤h2/4\eta\lambda \le h^2/4, which we now state as a proposition. Second, we measured what the update does to the samples, directly and broadly: without CFG, with CFG, unconditionally, under quantization, and on two additional architectures (SD3.5 and FLUX). In all of these settings LdG improves the preference metrics over the respective baseline. We believe this is a more honest and more useful link than a theoretical guarantee would be.

3. Comparison with well-tuned baselines and other architectures

This is a great suggestion. We added tuned CFG, PAG, TPG and SAG, with every method's scale chosen from the same 1K sweep. With CFG (w=3):

SDXL, CFG onFID↓CLIP↑ImageReward↑Aesthetic↑PickScore↑HPSv2↑
CFG only15.710.64360.4275.42021.870.2489
+ SEG16.550.64020.4825.54822.080.2621
+ PAG16.350.64310.4895.52122.160.2615
+ TPG15.650.64580.5115.50122.120.2613
+ SAG17.650.64020.4515.55921.980.2625
+ LdG16.480.64310.5435.55122.240.2683

With CFG, LdG gives the best ImageReward, PickScore and HPSv2, and over three seeds its gains are far larger than the seed variation (ImageReward +0.104 ± 0.002). Without CFG, LdG is best on CLIP, ImageReward, Aesthetic, PickScore and HPSv2.

We also ran two additional architectures. On SD3.5-medium (MMDiT), LdG keeps FID at the CFG level (20.86 vs 20.83) and improves every other metric, e.g. ImageReward from 0.991 to 1.044 and HPSv2 from 0.2837 to 0.2925, whereas PAG raises FID to 45.00. On the guidance-distilled FLUX.1-dev, LdG lowers FID from 26.62 to 25.88 and improves ImageReward, Aesthetic, PickScore and HPSv2. So the gains are not specific to SDXL.

4. Latency, memory, and matched compute

Thanks for the question. LdG uses the same single perturbed branch as SEG, PAG and TPG, so its cost is in the same range. On one GPU (batch 4, 1024², 25 steps), we measured the time per image:

Seconds per imageSEGPAGTPGLdGLdG vs SEG
without CFG0.6840.6780.6820.750+9.7%
with CFG (w=3)0.9310.914–0.954+2.5%
INT4, without CFG1.870––1.890+1.1%

The overhead is small, and it shrinks further with CFG and under INT4. Peak memory is unchanged in all cases (22.74 GiB at FP16).

For a matched computational budget, we ran unguided SDXL with 50 steps, which uses the same number of network passes as 25 guided steps. It reaches FID 44.35 and ImageReward -0.855, against 26.72 and 0.239 for LdG. So spending the same compute on more sampling steps does not come close to what LdG gives. We have added a dedicated subsection reporting runtime and memory to the manuscript.

5. FP16 + LdG and the specific benefit under quantization

This is an important question, and we thank the reviewer for it. We have added FP16 + LdG to the main table. At FP16, LdG helps strongly on its own, reducing FID from 52.71 to 26.72 and raising ImageReward from -1.030 to 0.239. So, as the reviewer suspected, part of the INT4 gain is the benefit of guidance itself.

What is specific to quantization is how much quality each method loses when going from FP16 to INT4:

Drop from FP16 to INT4 (lower is better)SEGPAGLdG
CLIP0.00810.00840.0029
ImageReward0.1270.1480.060
Aesthetic0.1290.1240.070
PickScore0.230.240.08
HPSv20.00500.00670.0018

LdG loses the least of all methods on every metric, roughly half or less of what SEG and PAG lose.

Moreover, INT4 + LdG is better than every FP16 guidance baseline (SEG, PAG, TPG and SAG) on all five per-image metrics. This suggests that the LdG update is robust to quantization noise in the attention, which is consistent with its role of damping high-curvature components. We have revised the quantization claim in the paper to state exactly this.

If the clarification(s) above address the reviewer's concerns, we will promptly upload the revised PDF, which incorporates all of these changes marked in blue.

Dear Reviewer, we thank you again for your time and valuable feedback. As the discussion period is coming to an end, we wanted to check whether our responses have addressed all your concerns. We would be grateful for any further feedback, and we are happy to clarify anything that remains unclear.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. I will consider changing my score after discussion with other reviewers.

Reviewer_8FxWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes guiding the denoising diffusion process via the evolving attention mechanism using energy-minimizing dynamics inspired by Landau-de Gennes (LdG) theory. The method tends to stabilize attention patterns during sampling, improving visual quality over the baseline SDXL model and the SEG method. LdG, when applied to the correct layer in the diffusion architecture, can produce higher-quality generations with larger LdG steps. Finally, LdG can also benefit diffusion models that are quantized.

Critical strengths

I believe that the authors' approach to the problem is an interesting, physics-motivated formulation that is novel and sounds promising. The idea of posing diffusion guidance as energy-minimizing dynamics is elegant, and I highly appreciate the mathematical foundation of your method.

Also, the experimental setting that measures generation quality for quantized generative models is practically relevant but is rarely examined in other works.

Critical weaknesses

W1. Generalization The method is evaluated only on the SDXL model and is referenced as a "drop-in" guidance step for text-to-image samplers. It remains undetermined how the method could be adapted to more recent architectures, such as Flux. Precisely:

  1. Joint attention mechanism in MMDiTs text tokens that have no spatial coordinates, and I am curious how energy would be defined over such a process.
  2. Flux is a guidance-distilled model, but this work does not report how LdG interacts with classifier-free guidance. It is unclear whether LdG would work in such a setup.
  3. Given the huge dependence of quality gains on the choice of layers, I'm concerned about how the optimal set of layers would be chosen in bigger architectures that do not have a simple block-level structure.

W2. Impact of step size on quality In the analysis of LdG step size (Fig. 6, A.7), after η=0.5η=0.5, the quality for the mid block monotonically improves, and the best values are reported at the end of the plot for η=30η=30. Conversely, authors claim (l175-l177) that a decrease in energy is guaranteed only for sufficiently small ηη. I am not sure whether the gains come from the structure-preserving relaxation or from a stronger perturbation of attention.

Questions for authors

Q1. Could you analyze whether your method works on FLUX to address W1?

Q2. Could you analyze how your method performs with CFG enabled?

Q3. Could you extend your mid-block analysis beyond η=30η=30 reporting the quality of your method, and measure whether the energy you minimize still decreases? It would directly address W2.

Q4. Why do the authors not compare against other cited works, such as PAG and SAG? This is concerning, as the authors claim their approach improves over prior attention-based guidance methods.

5 comments

Dear Reviewer,

We thank you for taking the time and effort to review our work and providing valuable feedback. We appreciate the positive assessment of the formulation and of the quantized setting, and seek to clarify the concerns. To be responsive, we ran all the experiments requested in the review; a summary of all new results, with the exact references for every baseline, is in our general response. Unless stated otherwise, all results are on MS-COCO with 10K images, with identical captions and seeds across methods.

1. Extension to MMDiT architectures and text tokens

Thanks for the question. LdG is applied only to the queries of the image tokens. In MMDiTs, these image tokens still live on the latent grid, so the Laplacian and the energy are defined over image positions exactly as in the UNet. The text tokens carry no spatial coordinates and we leave them unchanged. The update then reaches the joint attention through the image queries, and the text-image interaction is preserved.

To check that this works in practice, we ran LdG on SD3.5-medium (MMDiT, CFG 4.5):

SD3.5-mediumFID↓CLIP↑ImageReward↑Aesthetic↑PickScore↑HPSv2↑
CFG only20.830.66470.9915.31422.440.2837
+ PAG45.000.63520.0995.38321.220.2382
+ LdG20.860.66581.0445.39822.490.2925

LdG keeps FID at the level of CFG alone and improves every other metric, while PAG on the same model doubles FID and loses most of the ImageReward. This suggests that acting on the image queries transfers cleanly to joint attention, while a hard replacement of the attention map does not.

2. Guidance-distilled models such as Flux

This is an important question. LdG does not need an unconditional branch. It needs one extra forward pass with the perturbed attention, and the guided prediction is ϵ+s(ϵ−ϵptb)\epsilon + s(\epsilon - \epsilon_{\text{ptb}}). It therefore sits on top of FLUX's distilled guidance (3.5) and does not interfere with it.

On FLUX.1-dev (10K images):

FLUX.1-devFID↓CLIP↑ImageReward↑Aesthetic↑PickScore↑HPSv2↑
distilled guidance only26.620.64511.0735.71423.040.3033
+ LdG25.880.64161.0885.72123.090.3046

LdG lowers FID and improves the four preference metrics, with CLIP staying close. In other words, LdG works in exactly the setting where classifier-free guidance is not available. We have added both models to the paper.

3. Choice of layers in larger architectures

We appreciate the reviewer raising this practical point. We use a simple protocol that does not require manual search: we split the transformer blocks into four contiguous groups, apply LdG to each group on 1K images, keep the group with the best FID, and then tune the scale. This costs four short runs:

FID↓ (1K images)group 0group 1group 2group 3
SD3.5-medium112.1392.84173.78197.69
FLUX.1-dev130.3481.5487.2298.07

The second group wins clearly on both models. This matches our finding on SDXL that the middle of the network is the right place to act. The SD3.5 and FLUX results above both use this choice. We have added the protocol and the group results to the appendix.

4. Impact of step size, and whether the energy still decreases

Thank you for this question, which we found very useful, as it helps us state precisely what the LdG step does. The update is one explicit step Q←Q+ηλ ΔQQ \leftarrow Q + \eta\lambda\,\Delta Q, i.e. a move along the energy-descent direction −∇E-\nabla E. The direction is set by the energy. The step size ηλ\eta\lambda only sets how far the attention is moved, and therefore how strong the guidance is. For the 5-point Laplacian, this step is guaranteed to lower EE when ηλ≤h2/4\eta\lambda \le h^2/4. We have replaced "for sufficiently small η" (L175–176) with this precise statement as a proposition.

To verify this and to extend the analysis beyond η=0.5 as requested, we measured the energy before and after every LdG call during sampling (SDXL INT4, 1K images, 25 steps, 10 mid-block layers, i.e. 4000 calls per setting). Within the bound (ηλ=0.25\eta\lambda = 0.25), the energy decreases in all 4000 calls, exactly as the proposition states, but the guidance is weak (FID 100.39). At the step size used in the paper (ηλ=2\eta\lambda = 2), the step moves further along the same descent direction and overshoots, so the energy after the step is higher, and the guidance becomes much stronger (FID 56.22, ImageReward 0.164). This mirrors other perturbation-based guidance: SEG is strongest when its blur goes to σ→∞, and PAG replaces the attention map altogether. In all of these methods, the size of the perturbation sets the strength of the guidance.

The key question is therefore whether the direction matters, i.e. whether moving along −∇E-\nabla E is better than a generic strong perturbation. The comparison with SEG isolates exactly this. SEG uses the same extra branch with its strongest perturbation (σ→∞). Replacing it with the LdG step improves all six metrics in every setting; we show three here:

SEG → LdGFID↓ImageReward↑HPSv2↑
without CFG27.64 → 26.72-0.088 → 0.2390.2455 → 0.2722
with CFG (w=3)16.55 → 16.480.482 → 0.5430.2621 → 0.2683
INT4, without CFG27.10 → 23.93-0.215 → 0.1790.2405 → 0.2704

With the same extra branch and the same cost, the energy-descent direction is what improves the samples. In other words, the gains come from the structure of the perturbation, not only from its size. We have rewritten Sec. 3 and Sec. 5 to present the update in these terms.

5. Performance with CFG enabled

Thank you for the suggestion. CFG and LdG are two different forms of guidance. CFG guides along the text condition, through the difference between the conditional and unconditional predictions. LdG guides along the spatial structure of attention, through the difference between the prediction and its attention-perturbed version. Both act on the same noise prediction, ϵu+w(ϵc−ϵu)+s(ϵc−ϵptb)\epsilon_u + w(\epsilon_c - \epsilon_u) + s(\epsilon_c - \epsilon_{\text{ptb}}), so they interact: with CFG on, LdG needs a smaller scale s than without it. We added all methods on top of CFG (w=3), with every method's scale chosen from the same 1K sweep:

SDXL, CFG onFID↓CLIP↑ImageReward↑Aesthetic↑PickScore↑HPSv2↑
CFG only15.710.64360.4275.42021.870.2489
+ SEG16.550.64020.4825.54822.080.2621
+ PAG16.350.64310.4895.52122.160.2615
+ TPG15.650.64580.5115.50122.120.2613
+ SAG17.650.64020.4515.55921.980.2625
+ LdG16.480.64310.5435.55122.240.2683

With CFG, LdG gives the best ImageReward, PickScore and HPSv2 of all methods. To make sure this is not noise, we repeated CFG alone and CFG + LdG with three generation seeds. The gains are far larger than the seed-to-seed variation (ImageReward +0.104 ± 0.002, HPSv2 +0.0188 ± 0.0001).

The gain also does not depend on the choice of w. We swept the CFG scale from 1.5 to 7.5 (1K images):

CFG only → + LdGFID↓ImageReward↑HPSv2↑
w = 1.563.00 → 51.71-0.320 → 0.0790.2079 → 0.2403
w = 254.65 → 50.240.100 → 0.3110.2273 → 0.2527
w = 350.21 → 50.040.435 → 0.5230.2484 → 0.2670
w = 550.31 → 51.390.699 → 0.7240.2680 → 0.2809
w = 7.550.93 → 52.490.790 → 0.8210.2788 → 0.2883

At every scale, adding LdG raises ImageReward, Aesthetic, PickScore and HPSv2. The sweep also shows how the two forms of guidance interact. At low CFG scales (w ≤ 3), LdG supplies guidance that CFG alone lacks and also lowers FID, while at high CFG scales it still improves all preference metrics at a small cost in FID. Overall, these results provide good evidence that LdG is complementary to CFG rather than a substitute for it.

6. Comparison with PAG and SAG

We agree that these comparisons were needed to support our claim, and thank the reviewer for pointing this out. We have added PAG, SAG and TPG, without and with CFG, with every method's scale chosen from the same 1K sweep. Without CFG:

SDXLMethodFID↓CLIP↑ImageReward↑Aesthetic↑PickScore↑HPSv2↑
FP16none52.710.5074-1.0305.28319.970.1761
SEG27.640.5894-0.0885.70421.320.2455
PAG22.670.5937-0.1705.55921.380.2304
TPG19.510.5977-0.1295.52721.300.2300
SAG34.970.5436-0.7035.43520.380.2026
LdG26.720.60830.2395.79121.800.2722
INT4none80.080.4780-1.4214.90319.370.1535
SEG27.100.5814-0.2155.57521.090.2405
PAG23.990.5854-0.3195.43521.140.2237
LdG23.930.60540.1795.72121.730.2704

Without CFG, LdG is best on five of the six metrics, and it beats SAG on all six. Under 4-bit quantization, LdG is best on all six. Against PAG, LdG is better on five of six metrics both with and without CFG, and on all six under INT4 and on SD3.5. Together with the CFG results above, the claim that LdG improves over prior attention-based guidance is now supported against SEG, PAG, TPG and SAG. We have added these tables to Sec. 6.

If the clarification(s) above address the reviewer's concerns, we will promptly upload the revised PDF, which incorporates all of these changes marked in blue.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We appreciate the time and effort put in to understand the work. We are encouraged that the formulation was found novel and principled, the quantized setting practically relevant, and the method easy to apply to pretrained models without retraining. The paper has two contributions. First, LdG is a principled guidance mechanism that perturbs attention along the descent direction of a Landau–de Gennes free energy, rather than by a heuristic blur, mask or shuffle. Second, as a practical application, LdG is especially effective for 4-bit quantized models.

To be responsive, we ran all the requested experiments. In addition to SEG [1], we now compare against CFG [2], SAG [3], PAG [4] and TPG [5]. For a fair comparison, every method's guidance scale is chosen from the same sweep. All experiments use MS-COCO with 10K images, and we additionally evaluate LdG on two newer architectures, SD3.5 [6] and FLUX.1-dev [7]. The main points addressed are:

  1. Baselines. Against all baselines, LdG achieves the best ImageReward, PickScore and HPSv2, both with and without CFG, and under 4-bit quantization it is the best method on all six metrics.
  2. Compatibility with CFG. CFG and LdG are two different forms of guidance, and they are complementary: adding it improves the preference metrics at every CFG scale from 1.5 to 7.5, and these gains are consistent across three random seeds (e.g. ImageReward +0.104 ± 0.002).
  3. Other architectures. LdG transfers to newer architectures. On SD3.5 (MMDiT), it matches the FID of CFG and improves the other five metrics, and on the guidance-distilled FLUX, where CFG is not available, it lowers FID and improves all four preference metrics.
  4. Quantization. LdG is the most robust method under quantization: going from FP16 to INT4, it loses the least quality of all methods on every metric.
  5. Unconditional generation. In the unconditional setting, LdG achieves the best Aesthetic score of all methods and reduces FID by 35% over unguided sampling.
  6. Cost. LdG uses the same single extra forward branch as SEG, PAG and TPG, adds only 1.1% to 9.7% time over SEG, and requires no extra memory.
  7. Theory. We replace the informal step-size condition with a precise proposition, verify it with energy measurements during sampling, and present the attention, logit and query updates as one unified operation.

We have incorporated all of these changes into the manuscript, and we will upload the revised PDF, with all changes marked in blue, once all the reviewers' concerns are fully addressed.

References

[1] S. Hong. Smoothed energy guidance: Guiding diffusion models with reduced energy curvature of attention. NeurIPS 2024.

[2] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv:2207.12598, 2022.

[3] S. Hong, G. Lee, W. Jang and S. Kim. Improving sample quality of diffusion models using self-attention guidance. ICCV 2023.

[4] D. Ahn, H. Cho, J. Min, W. Jang, J. Kim, S. Kim, H. H. Park, K. H. Jin and S. Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. ECCV 2024.

[5] J. Rajabi, S. Mehraban, S. Sadat and B. Taati. Token perturbation guidance for diffusion models. NeurIPS 2025.

[6] P. Esser et al. Scaling rectified flow transformers for high-resolution image synthesis. ICML 2024.

[7] Black Forest Labs. FLUX.1. https://github.com/black-forest-labs/flux, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.