acceptodds
Under review as a conference paper at ICLR 2027

Perturbing Text Attention for Guiding Diffusion Models

Abstract

Training-free guidance methods steer the sampling process of a diffusion model by extrapolating the prediction of the model away from a weak prediction. The weak prediction is obtained by perturbing intermediate computations of the model at inference time, such as self-attention weights or the arrangement of visual tokens. However, their performance gains over unguided sampling remain marginal for text-conditional generation. To understand why, we present an in-depth analysis of how diffusion models behave under conditions of varying complexity. Our analysis reveals that the primary bottleneck for text-conditional generation lies in aligning generated samples with the given condition, rather than in refining spatial details among visual tokens, which existing methods mainly focus on. Based on this, we introduce text attention guidance (TAG), a simple yet effective training-free guidance method that perturbs the interaction between visual and textual tokens. To this end, we propose to degrade the attention weights between the two modalities, yielding a weak prediction that particularly struggles to align samples with the given condition. This provides a targeted negative signal that mitigates the core bottleneck of text-conditional diffusion models. Extensive experiments on both U-Net and transformer-based models, including SDXL, SD3.5, Flux.1-dev, and Qwen-Image, demonstrate the effectiveness of our approach, which outperforms existing training-free guidance methods significantly and achieves results comparable to or even better than those of classifier-free guidance without any training. We will release the code upon acceptance.

Primary area:
generative models
Keywords:
Diffusion models, Guidance, Training-free

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_j83zOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce a new training-free guidance metric for diffusion models. The authors separate two sources of error, condition guidance and target guidance. The authors show that the main bottleneck of text-conditional generation is in the conditioning method with a toy experiment, and based on this observation, they propose tho flatten the attention logits from visual-to-textual tokens with temperature scaling, so that this could serve as a weak prediction in guidance. The layers with the lowest attention entropy are selected to be flattened. The method is applicable to diverse TTI diffusion models, with consistently good performance.

Critical strengths
  • The method is applicable to a diverse models, from U-Net based Diffusion models to recent MM-DiT architectures, whereas many of the recent studies are specifically designed for one type of architecture.
  • The method is incredibly simple and seems robust, with an approach well-motivated from a good observation.
Critical weaknesses
  • Empirical Analysis with Toy Experiments: the toy experiment is interesting and greatly supports the method. However, I have a few concerns: i) I think CG and TG should be a bit more clearly defined. Although I can grasp the overall concept, but I think it is hard to be articulated precisely (especially TG). This is a concern in connection to the following concern ii) How can CG and TG be separated in practical TTI setting, as I believe it is difficult whether it's CG or TG in real use case. iii) does the observation still hold in the practical TTI setting? I wonder since this could be something which holds under a conceptual toy experiment.

  • Potential contradiction in arguments: the paper's main argument and motivation is on providing a better conditioning to the model. Based on this intuition, I expect the CLIP scores to also rise greatly. However, the CLIP scores fall behind against CFG, or scores in par, not as much as expected. I think this could potentially be a contradictory to the main argument. Could the authors provide an explanation on this, or clarifications in case I have misunderstood anything?

Questions for authors
  • Have the authors considered analyzing the entropy in terms of the attention heads, instead of layers, since each head could differ within a layer? A head in a layer could already have a flat attention distribution, whereas another head in that specific layer could have a very peaky distribution.

  • Study on 'w' for extrapolation (guidance): I wonder how the performance changes with different 'w' values; the approach showed to be robust with different temperatures, how is it with different 'w's? Each baseline guidance method use different 'w' values, so I wonder how they each perform under same/different 'w's.

  • Could the authors provide an explanation on why TAG outperforms baselines in IR by a huge margin, whereas gains in other metrics are somewhat marginal?

6 comments
AuthorsOpenReview ↗

Response to Reviewer j83z (1)

We thank the reviewer for the positive assessment and the thoughtful questions. Our responses follow the order of the review, and the changes are marked in blue in the revised manuscript.

W1 : CG and TG should be defined more clearly, it is unclear how they are separated in the practical setting, and whether the observation holds there.

  • Thank you for the constructive comments on our analysis. (i) We have clarified in Sec. 3.2 that CG and TG are categorized by what is degraded in the weak model, which we summarize in Table R1. A diffusion model has two types of operations, those injecting the condition into visual tokens and those among visual tokens. CG degrades the former, or the condition itself as in CFG, and its weak model fails to align samples with the condition. TG degrades the latter, or the visual tokens themselves, and its weak model fails to predict the target even if the condition is injected properly. In our analysis, the encoder corresponds to the former and the denoiser to the latter. We obtain the weak model of CG with a narrower encoder, and that of TG with a narrower denoiser. (ii) The same criterion applies to the practical setting. Existing training-free methods perturb visual tokens or the operations among them (e.g., SXXS_{XX}), which corresponds to TG, whereas TAG perturbs SXCS_{XC}, where the condition is injected into visual tokens, which corresponds to CG. (iii) To verify that the observation holds in the practical setting, we have added in Table 7 (Table R2) the results of applying the same flattening in Eq. (4) to SXXS_{XX}. Flattening SXCS_{XC} improves IR over unguided sampling by 0.79, whereas flattening SXXS_{XX} improves it by 0.20. This suggests that the observation of our analysis holds in the practical setting as well.

Table R1 : Comparison of condition guidance (CG) and target guidance (TG) in terms of what is degraded in the weak model, with the corresponding weak models in our analysis and examples in the practical setting.

\hline & \textbf{Degraded in the weak model} & \textbf{Our analysis} & \textbf{Practical examples} \newline \hline \text{CG} & \text{the condition, or its injection into visual tokens} & \text{narrower encoder} & \text{CFG, TAG} \newline \text{TG} & \text{visual tokens, or the operations among them} & \text{narrower denoiser} & \text{SEG, TPG, } S_{XX} \text{ in Table R2} \newline \hline \end{array}$$ **Table R2** : Quantitative results on 5,000 COCO prompts with Flux.1-dev for dividing different blocks of the attention logits in Eq. (2) by the temperature $\tau$. Bold indicates the best result in each column. $$\begin{array}{l c c c c c c} \hline \textbf{Perturbed block} & \textbf{IR}\uparrow & \textbf{PickScore}\uparrow & \textbf{HPSv2}\uparrow & \textbf{Aesthetic}\uparrow & \textbf{FID}\downarrow & \textbf{CLIP}\uparrow \newline \hline \text{None (w/o guidance)} & 0.269 & 21.36 & 0.242 & 5.50 & \textbf{30.07} & 29.53 \newline S_{XX} & 0.472 & 21.79 & 0.270 & 5.77 & 36.31 & 29.80 \newline S_{XC}\ \text{(TAG)} & \textbf{1.062} & \textbf{23.09} & \textbf{0.290} & \textbf{5.85} & 30.61 & \textbf{31.04} \newline \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer j83z (2)

W2 : CLIP scores do not rise as much as expected, which may contradict the main argument.

  • Thank you for raising this point, which we would like to clarify. The CLIP score of TAG does rise over unguided sampling on all four models (e.g., from 29.53 to 31.04 on Flux.1-dev), and it is the highest among training-free methods on Flux.1-dev, SDXL, and SD3.5 (Tables 2 and 10). It is also comparable to that of CFG (e.g., 31.04 vs. 31.44 on Flux.1-dev).

  • The CLIP score is, however, known to be unreliable for precise prompt alignment, since CLIP behaves like a bag of words and does not capture how attributes and relations are bound to objects [1, 2]. We have added in Fig. 15 of Appendix D examples where CFG obtains higher CLIP scores than TAG, although it misses parts of the prompt.

  • VQAScore, which agrees with human judgments better [1], is more reliable. In terms of VQAScore, TAG outperforms all training-free methods on all four models, and even CFG on Flux.1-dev and SDXL (Tables 3 and 11). To verify prompt alignment more directly, we have also added in Table 8 (Table R3) a human evaluation, where 35 raters choose the image that follows the prompt best among those of CFG, SEG, TPG, and TAG for 40 random COCO prompts with Flux.1-dev. TAG receives 50.6% of the votes, compared with 30.4% for CFG, suggesting that TAG improves prompt alignment in terms of human judgments, although its CLIP score is lower than that of CFG.

Table R3 : Human and VLM evaluation of guidance methods on 40 prompts randomly selected from the COCO validation split with Flux.1-dev. We report the percentage of votes from 35 raters, each of whom chooses the best image among the four methods for each criterion, and the percentage of 5,000 prompts for which Claude Opus 5.5 chooses each method (VLM). Bold indicates the best result.

\hline \textbf{Criterion} & \textbf{CFG} & \textbf{SEG} & \textbf{TPG} & \textbf{TAG (ours)} \newline \hline \text{Prompt adherence} & 30.4 & 13.6 & 5.3 & \textbf{50.6} \newline \text{Image quality} & 27.3 & 16.4 & 3.1 & \textbf{53.2} \newline \text{Overall preference} & 26.9 & 11.8 & 2.7 & \textbf{58.6} \newline \hline \text{Overall preference (VLM)} & 28.1 & 8.0 & 0.5 & \textbf{63.5} \newline \hline \end{array}$$ [1] Evaluating text-to-visual generation with image-to-text generation, ECCV 2024. [2] When and why vision-language models behave like bags-of-words, and what to do about it?, ICLR 2023. **Q1 : Head-level approach instead of layer-level approach used in TAG.** - Thank you for the constructive comment. Following the suggestion, we have added in Table 13 of Appendix D (Table R4) a head-level variant of TAG. It computes the entropy in Eq. (5) for each attention head, and perturbs the 384 heads of lowest entropy across all layers, which matches the number of heads in the 16 layers of TAG. The two provide similar results (e.g., IR of 1.030 vs. 1.062), and both outperform CFG and existing training-free methods (0.949 and at most 0.752 in Table 2, respectively). This suggests that the key is that TAG is designed as CG, rather than the granularity of the selection. We keep the layer-level selection, since it is simpler. **Table R4** : Quantitative results of TAG on 5,000 COCO prompts with Flux.1-dev for selecting attention heads or layers to perturb. The head-level variant perturbs the 384 heads of lowest entropy across all layers. $$\begin{array}{l c c c c c c} \hline \textbf{Selection} & \textbf{IR}\uparrow & \textbf{PickScore}\uparrow & \textbf{HPSv2}\uparrow & \textbf{Aesthetic}\uparrow & \textbf{FID}\downarrow & \textbf{CLIP}\uparrow \newline \hline \text{Head-level} & 1.030 & 22.91 & 0.286 & 5.81 & 31.06 & 30.84 \newline \text{Layer-level (TAG)} & 1.062 & 23.09 & 0.290 & 5.85 & 30.61 & 31.04 \newline \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer j83z (3)

Q2 : Study on the guidance weight ww.

  • Thank you for the suggestion. We have added in Table 15 of Appendix D (Table R5) results of SSG, SEG, CFG, and TAG on Flux.1-dev under the same guidance weights. TAG outperforms the training-free baselines for all weights, and performs best for a sufficiently large weight (w≥3.5w \geq 3.5). SSG degrades as ww increases and performs worse than unguided sampling (0.269) for all weights.

Table R5 : Quantitative results of guidance methods on 5,000 COCO prompts with Flux.1-dev in terms of IR for different guidance weights ww. Methods and weights are sorted by the weight selected for each method, such that the underlined numbers on the diagonal correspond to the settings in Table 9. The first row shows unguided sampling for reference, which does not depend on ww. Bold indicates the best result in each column.

\hline \textbf{Method} & w = 2 & w = 3 & w = 3.5 & w = 5.5 \newline \hline \text{w/o guidance} & 0.269 & 0.269 & 0.269 & 0.269 \newline \hline \text{SSG} & \underline{-2.080} & -2.228 & -2.252 & -2.277 \newline \text{SEG} & 0.708 & \underline{0.752} & 0.736 & 0.495 \newline \text{CFG} & \textbf{0.878} & \textbf{0.948} & \underline{0.949} & 0.603 \newline \text{TAG (ours)} & 0.732 & 0.924 & \textbf{0.966} & \underline{\textbf{1.062}} \newline \hline \end{array}$$ **Q3 : Why does TAG outperform baselines in IR by a large margin, whereas gains in other metrics are marginal?** - Thank you for the question. We conjecture that this is because IR reflects prompt alignment more than PickScore and HPSv2. IR agrees with human judgments of alignment better than them [1] (e.g., pairwise accuracy of 67.3% vs. 59.4% and 55.2% on TIFA160). Note that IR builds on BLIP [2], where textual features attend to image features through cross-attention, while PickScore and HPSv2 build on CLIP, which compares a single embedding of an image with that of a text. Since TAG targets condition alignment, its gains are reflected in IR more clearly. - Note that the gains in the other metrics are not marginal either, considering that the metrics have different scales. We show in Table R6 the gains of TAG over the best training-free baseline on Flux.1-dev, normalized by those of CFG over unguided sampling, where the results are from Tables 2 and 3. The normalized gains are 46% (IR), 67% (PickScore), and 71% (VQAScore). The gain is small for HPSv2 only (6%), where SEG already exceeds CFG (0.288 vs. 0.278). TAG also outperforms all training-free methods in PickScore and HPSv2 on all four models (Tables 2 and 10), and it receives 53.2% of the votes for image quality in the human evaluation (Table 8), compared with 16.4% for SEG. **Table R6** : Gains of TAG over the best training-free baseline on Flux.1-dev, normalized by those of CFG over unguided sampling, i.e., (TAG $-$ baseline) / (CFG $-$ w/o guidance). IR, PickScore, and HPSv2 are from Table 2 (COCO), and VQAScore is the overall score in Table 3 (GenAI-Bench). $$\begin{array}{l c c c c c} \hline \textbf{Metric} & \textbf{w/o guidance} & \textbf{Best training-free baseline} & \textbf{CFG} & \textbf{TAG} & \textbf{Normalized gain (\\%)} \newline \hline \text{IR} & 0.269 & 0.752\ \text{(SEG)} & 0.949 & 1.062 & 46 \newline \text{PickScore} & 21.36 & 22.25\ \text{(SEG)} & 22.62 & 23.09 & 67 \newline \text{HPSv2} & 0.242 & 0.288\ \text{(SEG)} & 0.278 & 0.290 & 6 \newline \text{VQAScore} & 0.680 & 0.713\ \text{(PAG)} & 0.750 & 0.763 & 71 \newline \hline \end{array}$$ [1] Evaluating text-to-visual generation with image-to-text generation, ECCV 2024. [2] BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation, ICML 2022.
Reviewer_4t2tOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes TAG, a training-free method that perturbs attention between visual and textual tokens to improve text-to-image generation. The authors use an entropy-based criterion to select the layers to perturb and evaluate the method across several diffusion models.

Critical strengths
  1. The paper addresses an important problem of improving text-image alignment without additional training.

  2. The authors provide experiments across different diffusion architectures and compare the proposed approach with several existing training-free guidance methods.

Critical weaknesses

1. Limited contribution. There are already many approaches to training-free guidance. The paper does not provide sufficiently novel insights or a compelling methodological contribution compared with existing work. It remains unclear why practitioners should adopt this method over existing alternatives.

2. The analysis in Section 3.2 seems far-fetched. The main conclusion of the analysis is that diffusion models struggle with condition alignment when conditions become complex. However, this observation is already intuitive and does not require the complicated and difficult-to-follow synthetic analysis presented by the authors.

3. Additional model-specific hyperparameter tuning. The proposed method requires additional hyperparameter tuning for each new model. In particular, the authors argue that it is important to perturb text attention in specific layers, which are selected using normalized attention entropy. This entropy must be computed for each new model. Together with the perturbation temperature and number of selected layers, this introduces additional complexity and limits the practical convenience of the method.

4. Lack of human evaluation. Human evaluation is important to confirm that the proposed approach genuinely improves image quality and text-image alignment, as automatic metrics do not capture the full picture.

5. Incomplete ablation of entropy-based layer selection. In Table 5, the authors show that selecting the 16 layers with the highest entropy leads to worse results. However, this comparison is insufficient to demonstrate the effectiveness of the proposed selection criterion. I would also consider selecting 16 random layers and testing a wider range of numbers of randomly selected layers across more models.

6. More visual results are needed. First, some results appear oversaturated, for example, in Figure 7. It would be important to provide more qualitative comparisons to assess the visual quality of the generated images. Second, it would be interesting to visualize how the generated images change for different values of the temperature and different selected layers. Such visualizations would provide better intuition about the method, whereas the authors currently present only numerical results for these ablations.

Questions for authors
  1. Why does the proposed approach improve aesthetic quality when it directly modifies text attention and primarily targets text-image correspondence?
  2. Does the proposed method provide additional improvements when applied on top of CFG?
  3. Does the proposed approach work for few-step models such as FLUX.1-schnell and other distilled diffusion models?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 4t2t (1)

We thank the reviewer for the detailed comments, which helped us clarify our contributions and add the requested experiments. We address each weakness and question below, and the changes are marked in blue in the revised manuscript.

W1 : Limited contribution, and it is unclear why practitioners should adopt TAG over existing alternatives.

  • Thank you for the comment. We respectfully disagree that the contribution is limited and would like to clarify it. (1) We present an analysis on why guidance should target condition alignment. Existing training-free methods propose various perturbations among visual tokens, without analyzing which errors guidance should suppress. Our analysis in Sec. 3.2 shows that suppressing errors in condition alignment becomes more important as the condition becomes more complex, as in text-conditional generation. (2) Based on this, we propose TAG, a training-free guidance method that targets condition alignment specifically by perturbing the attention from visual to textual tokens, together with an entropy-based criterion that selects the layers to perturb without a search. (3) We show that TAG works across different architectures. It outperforms all training-free methods in IR, PickScore, and VQAScore on SDXL, SD3.5, Flux.1-dev, and Qwen-Image (Tables 2, 3, 10, and 11), whereas TPG and SSG perform worse than unguided sampling on Flux.1-dev and SD3.5. This makes TAG a simple and practical choice.

W2 : The analysis in Sec. 3.2 seems far-fetched, and its conclusion is intuitive.

  • Thank you for the candid feedback on our analysis. We would respectfully like to clarify its conclusion. The analysis does not conclude that diffusion models struggle with complex conditions. We have shown in Fig. 2 that the more effective type of guidance depends on the complexity of the condition. Target guidance (TG) provides better precision than condition guidance (CG) for simple conditions (e.g., P=1P = 1 in Fig. 2(a) and ∣c∣=64|c| = 64 in Fig. 2(d)), while CG outperforms TG as the condition becomes more complex. Existing training-free methods do not provide such an analysis, and their guidance is not targeted at improving condition alignment, since they perturb operations among visual tokens. To make the analysis easier to follow, we have also revised Sec. 3.2 to define CG and TG more clearly. On this point, we would also kindly refer you to the comment of Reviewer j83z, who finds our approach well motivated by the observation.

W3 : Additional model-specific hyperparameter tuning (entropy, temperature, and the number of layers).

  • Thank you for raising this concern on practicality. TAG does not particularly require more hyperparameters than existing training-free methods, and the hyperparameters of TAG are easy to tune. The number of layers kk is not an additional hyperparameter of TAG, since existing methods also set the layers to perturb for each model (See Table 9). It rather removes the extensive layer selection that they require. Existing methods are developed for U-Net, where they select the layers empirically by exploiting its hierarchical structure (e.g., the layers in the mid-block). This does not apply to transformer-based models, where all the layers are structurally identical. Applying them to a transformer-based model with LL attention layers requires searching for the layers among 2L2^L possible subsets with generation and evaluation, or among L(L+1)/2L(L+1)/2 candidates even when the search is restricted to consecutive layers (e.g., 1,653 for Flux.1-dev with L=57L = 57). In contrast, TAG ranks the layers by the entropy in Eq. (5), measured once per model while generating images for 8 prompts only, such that the only choice left is kk, i.e., LL candidates. TAG is also robust to kk. We have shown in Table 5 that IR varies by less than 0.05 over k∈[12,20]k \in [12, 20] on Flux.1-dev, outperforming CFG for all these values.

  • The temperature τ\tau is the only additional hyperparameter of TAG, similar to the blur strength σ\sigma of SEG and the swap ratio of SSG (See Table 9). It is also easy to set, since TAG is robust to τ\tau when it is sufficiently large. We have shown in Table 4 that IR varies by less than 0.06 over τ∈[1.3,10]\tau \in [1.3, 10] on Flux.1-dev, outperforming CFG for all these values.

AuthorsOpenReview ↗

Response to Reviewer 4t2t (2)

W4 : Lack of human evaluation.

  • Thank you for the suggestion. We agree that human evaluation is important. We have added in Table 8 (Table R1) a human evaluation on Flux.1-dev, where 35 raters choose the best image among those of CFG, SEG, TPG, and TAG for 40 random COCO prompts in terms of prompt adherence, image quality, and overall preference. TAG receives 50.6%, 53.2%, and 58.6% of the votes, respectively, outperforming CFG, the second best, by at least 20.2%. We have also added a VLM-as-a-judge evaluation, where Claude Opus 5.5 chooses TAG for 63.5% of all 5,000 COCO prompts and CFG for 28.1%. This indicates that TAG improves both prompt alignment and visual quality in terms of human judgments as well. We have described the protocols in Appendix B.

Table R1 : Human and VLM evaluation of guidance methods on 40 prompts randomly selected from the COCO validation split with Flux.1-dev. We report the percentage of votes from 35 raters, each of whom chooses the best image among the four methods for each criterion, and the percentage of 5,000 prompts for which Claude Opus 5.5 chooses each method (VLM). Bold indicates the best result.

\hline \textbf{Criterion} & \textbf{CFG} & \textbf{SEG} & \textbf{TPG} & \textbf{TAG (ours)} \newline \hline \text{Prompt adherence} & 30.4 & 13.6 & 5.3 & \textbf{50.6} \newline \text{Image quality} & 27.3 & 16.4 & 3.1 & \textbf{53.2} \newline \text{Overall preference} & 26.9 & 11.8 & 2.7 & \textbf{58.6} \newline \hline \text{Overall preference (VLM)} & 28.1 & 8.0 & 0.5 & \textbf{63.5} \newline \hline \end{array}$$ **W5 : Incomplete ablation of the entropy-based layer selection (random layers, a wider range of numbers of layers, and more models).** - Thank you for suggesting this ablation. Following the suggestion, we have added in Table 6 (Table R2) random selection of 16 layers to the ablation, and compared in Table 12 of Appendix D (Table R3) random selection with our entropy-based selection for different numbers of layers $k$ across two different models, Flux.1-dev and Qwen-Image. With 16 layers, random selection degrades IR by 0.22 on average (0.842 $\pm$ 0.100 vs. 1.062) and performs similarly to the reverse selection (0.850). Our selection also outperforms random selection for all values of $k$ on both models, by 0.17 to 0.22 in IR on Flux.1-dev ($k$ from 12 to 20) and by 0.04 to 0.07 on Qwen-Image ($k$ from 22 to 38), where every random draw performs worse than ours. This indicates that which layers are perturbed matters and that the entropy identifies them effectively. **Table R2** : Quantitative results of TAG on 5,000 COCO prompts with Flux.1-dev for different layer selection strategies with $k = 16$. Random selection is averaged over three seeds (mean $\pm$ standard deviation), and the reverse selection perturbs the 16 layers of highest entropy. $$\begin{array}{l c c c c c c} \hline \textbf{Selection} & \textbf{IR}\uparrow & \textbf{PickScore}\uparrow & \textbf{HPSv2}\uparrow & \textbf{Aesthetic}\uparrow & \textbf{FID}\downarrow & \textbf{CLIP}\uparrow \newline \hline \text{Random} & 0.842 \pm 0.100 & 22.75 \pm 0.28 & 0.279 \pm 0.011 & 5.76 \pm 0.12 & 32.52 \pm 0.93 & 30.75 \pm 0.18 \newline \text{Reverse} & 0.850 & 22.85 & 0.284 & 5.76 & \textbf{30.50} & 30.46 \newline \text{Ours} & \textbf{1.062} & \textbf{23.09} & \textbf{0.290} & \textbf{5.85} & 30.61 & \textbf{31.04} \newline \hline \end{array}$$ **Table R3** : Quantitative results of TAG on COCO with Flux.1-dev (5,000 prompts) and Qwen-Image (a subset of 2,500 prompts) in terms of IR for different numbers of perturbed layers $k$. Random selection is averaged over three seeds (mean $\pm$ standard deviation). $$\begin{array}{l c c c c c} \hline \textbf{Flux.1-dev} & k = 12 & k = 14 & k = 16 & k = 18 & k = 20 \newline \hline \text{Random} & 0.805 \pm 0.124 & 0.826 \pm 0.100 & 0.842 \pm 0.100 & 0.845 \pm 0.140 & 0.875 \pm 0.103 \newline \text{Ours} & \textbf{1.017} & \textbf{1.030} & \textbf{1.062} & \textbf{1.063} & \textbf{1.049} \newline \hline \textbf{Qwen-Image} & k = 22 & k = 26 & k = 30 & k = 34 & k = 38 \newline \hline \text{Random} & 1.194 \pm 0.023 & 1.209 \pm 0.021 & 1.208 \pm 0.019 & 1.189 \pm 0.019 & 1.176 \pm 0.026 \newline \text{Ours} & \textbf{1.247} & \textbf{1.249} & \textbf{1.253} & \textbf{1.245} & \textbf{1.246} \newline \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer 4t2t (3)

W6 : More visual results for main experiments and ablations.

  • Thank you for the helpful suggestions on visual results. (1) For more qualitative comparisons, we have added more samples to Figs. 1 and 5, and provided in Figs. 10-13 of Appendix C qualitative results on GenAI-Bench for all four models. For the concern on oversaturation, we would like to note that TAG provides better visual quality than other methods. In the human evaluation in Table 8 (Table R1), TAG receives 53.2% of the votes for image quality, compared with 27.3% for CFG, 16.4% for SEG, and 3.1% for TPG. TAG also outperforms all training-free methods in terms of PickScore and HPSv2 on all four models (Tables 2 and 10). (2) We have also added in Fig. 14 of Appendix D qualitative results for different temperatures τ\tau and numbers of perturbed layers kk. The images follow the prompts with high visual quality for all values of τ\tau and kk, indicating that TAG is robust to both hyperparameters.

Q1 : Why does TAG improve aesthetic quality when it modifies text attention and targets text-image correspondence?

  • Thank you for the interesting question. Guidance suppresses the errors that the weak model makes more severely than the main model. TAG thus suppresses errors in condition alignment, since its weak model fails to align samples with the prompt. We have shown in Fig. 3 that, for complex conditions, CG keeps most of the samples on the support of the target distribution (i.e., the data manifold), while TG generates many samples off the support. The visual quality of a sample is closely related to how close it is to the manifold of real images [1], and keeping samples on the manifold during guided sampling reduces visual artifacts [2]. This suggests that TAG, which belongs to CG, improves aesthetic quality together with text-image correspondence by keeping samples on the data manifold. This is also supported by Autoguidance [3], which attributes the improvement of image quality by guidance to pushing samples toward the data manifold and eliminating outliers.

[1] Improved precision and recall metric for assessing generative models, NeurIPS 2019.

[2] CFG++: Manifold-constrained classifier free guidance for diffusion models, ICLR 2025.

[3] Guiding a diffusion model with a bad version of itself, NeurIPS 2024.

Q2 : Does TAG provide additional improvements on top of CFG?

  • Thank you for the question. Yes. We have added in Table 17 of Appendix D (Table R4) results of combining each training-free method with CFG on Flux.1-dev, where we add the two guidance terms as in PAG and select the weights on 200 training prompts (Table 18). CFG+TAG outperforms both CFG and TAG alone in terms of IR (1.113 vs. 0.949 and 1.062) and HPSv2 (0.296 vs. 0.278 and 0.290), suggesting that TAG is complementary to CFG, although both belong to condition guidance (CG). CFG+PAG and CFG+SEG, in contrast, provide marginal gains over CFG alone (IR of 0.959 and 0.956 vs. 0.949), suggesting that errors in visual refinement may not be the primary bottleneck once CFG is applied. Combining TPG or SSG with CFG even degrades IR.

  • We conjecture that TAG is complementary to CFG because the two methods degrade the injection of the condition in different ways. CFG removes the condition itself, and its weak model fails to reflect the prompt as a whole. TAG instead keeps the condition and prevents visual tokens from attending to textual tokens selectively, such that its weak model misses the details of the prompt (Fig. 6). This allows the two methods to suppress different errors in condition alignment.

Table R4 : Quantitative results of combining CFG with training-free guidance methods on 5,000 COCO prompts with Flux.1-dev. "+ CFG" denotes the combination with CFG, and the other rows are from Table 2. Bold indicates the best result in each column.

\hline \textbf{Method} & \textbf{IR}\uparrow & \textbf{PickScore}\uparrow & \textbf{HPSv2}\uparrow & \textbf{Aesthetic}\uparrow & \textbf{FID}\downarrow & \textbf{CLIP}\uparrow \newline \hline \text{CFG} & 0.949 & 22.62 & 0.278 & 5.51 & 28.73 & \textbf{31.44} \newline \hline \text{PAG} & 0.643 & 22.08 & 0.275 & 5.82 & 30.27 & 29.99 \newline \quad \text{+ CFG} & 0.959 & 22.74 & 0.282 & 5.55 & \textbf{27.36} & 31.25 \newline \text{SEG} & 0.752 & 22.25 & 0.288 & 5.78 & 34.55 & 30.03 \newline \quad \text{+ CFG} & 0.956 & 22.53 & 0.292 & 5.70 & 33.29 & 31.00 \newline \text{TPG} & 0.083 & 21.21 & 0.240 & 5.72 & 44.98 & 29.68 \newline \quad \text{+ CFG} & 0.328 & 21.41 & 0.236 & 5.35 & 55.90 & 30.83 \newline \text{SSG} & -2.080 & 18.48 & 0.104 & 3.90 & 163.43 & 22.41 \newline \quad \text{+ CFG} & -0.387 & 20.96 & 0.195 & 4.71 & 74.26 & 28.57 \newline \text{TAG (ours)} & 1.062 & 23.09 & 0.290 & \textbf{5.85} & 30.61 & 31.04 \newline \quad \text{+ CFG} & \textbf{1.113} & \textbf{23.10} & \textbf{0.296} & 5.76 & 31.44 & 31.29 \newline \hline \end{array}$$
Reviewer_yCYsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Text Attention Guidance (TAG), a training-free guidance method for improving text-to-image generation. The authors argue that existing training-free guidance methods primarily perturb visual representations, whereas the major limitation in text-conditional generation is aligning the generated image with the given prompt. Motivated by a synthetic analysis of conditional complexity, TAG constructs a weak prediction by flattening the attention logits between visual and textual tokens. The original prediction is then extrapolated away from this weak prediction to improve generation.

Critical strengths
  1. Paper is well written. The flow of the paper is clear. After reading introdcution I am able to understand the problem statement and the proposed solution.

  2. The authors have tried to cover the breadth of the evaluations like multiple models, multiple baselines, multiple benchmarks, multiple evaluation metrics provide a extensive evalaution.

Critical weaknesses
  1. In introduction, authors said that they conjecture that '''for practical tasks with complex conditions (e.g., text-conditional generation), the primary performance bottleneck lies in aligning the generated samples with the given condition, rather than merely refining fine-grained visual details'''. I think this conjecture is not well supported by the evidences by the experiments because what if the both tasks are entangled.

  2. Authors have reported multiple automatic metrics but the human evalaution is missing from the paper. I will appreciate if authors can conduct a human study on a small set. I will also suggest authors to perform vlm as a judge evalaution because that will be scalable.

  3. TAG requires an additional perturbed forward pass during sampling. The authors should report inference latency, memory consumption, and computational overhead relative to CFG and existing training-free methods.

  4. Some baselines, particularly SSG, produce severely degraded or blurry results (Tables 2 and 7). Although the authors report hyperparameter tuning, additional sensitivity analysis would help determine whether these failures are inherent to the methods or influenced by the selected configurations.

  5. In Equation (4), modifying visual-to-text attention logits in MM-DiT may affect both attention selectivity among textual tokens and the relative attention allocated to textual versus visual tokens. The authors should clarify which effect contributes to the observed improvements.

Questions for authors

I think the paper is well written and of good quality. The authors have made a commendable effort to provide extensive experimental evaluations across multiple models, baselines, and benchmarks. I would be happy to reconsider my score if the authors adequately address the concerns highlighted in the weaknesses section during the rebuttal.

5 comments
AuthorsOpenReview ↗

Response to Reviewer yCYs (1)

We thank the reviewer for the careful reading and the encouraging assessment of our work. We respond to each weakness below, and the corresponding changes are marked in blue in the revised manuscript.

W1 : The conjecture that the bottleneck lies in condition alignment is not well supported, since condition alignment and visual refinement may be entangled.

  • Thank you for the constructive comment. We agree that condition alignment and visual refinement are entangled, and we do not claim that they can be disentangled completely. There are, however, operations that are more specialized for each of them. The operations that inject the condition into the features of the noisy sample (the condition encoder in Sec. 3.2 and SXCS_{XC} in TAG) are more specialized for condition alignment, while those among the features (the denoiser in Sec. 3.2 and SXXS_{XX}) are more specialized for visual refinement. We conjecture that the former are more important for text-to-image generation, such that degrading them (CG) provides more effective guidance than degrading the latter (TG). We have revised Sec. 1 and Sec. 3.2 to make this clear.

  • To verify this conjecture, we have added in Table 7 (Table R1) the results of applying the same flattening in Eq. (4) to SXXS_{XX}. Flattening SXCS_{XC} (TAG) improves IR over unguided sampling by 0.79, while flattening SXXS_{XX} improves it by 0.20. This suggests that the two operations play different roles even though the two tasks are entangled, and that the one for condition alignment is more important.

Table R1 : Quantitative results on 5,000 COCO prompts with Flux.1-dev for dividing different blocks of the attention logits in Eq. (2) by the temperature τ\tau. Bold indicates the best result in each column.

\hline \textbf{Perturbed block} & \textbf{IR}\uparrow & \textbf{PickScore}\uparrow & \textbf{HPSv2}\uparrow & \textbf{Aesthetic}\uparrow & \textbf{FID}\downarrow & \textbf{CLIP}\uparrow \newline \hline \text{None (w/o guidance)} & 0.269 & 21.36 & 0.242 & 5.50 & \textbf{30.07} & 29.53 \newline S_{XX} & 0.472 & 21.79 & 0.270 & 5.77 & 36.31 & 29.80 \newline S_{XC}\ \text{(TAG)} & \textbf{1.062} & \textbf{23.09} & \textbf{0.290} & \textbf{5.85} & 30.61 & \textbf{31.04} \newline \hline \end{array}$$ **W2 : Human evaluation is missing, and a VLM-as-a-judge evaluation is suggested.** - Thank you for the helpful suggestions. We have added in Table 8 (Table R2) a human evaluation on Flux.1-dev, where 35 raters choose the best image among those generated by CFG, SEG, TPG, and TAG for 40 randomly selected COCO prompts in terms of prompt adherence, image quality, and overall preference. TAG receives about half of the votes for prompt adherence and image quality and 58.6% for overall preference, outperforming CFG, the second best, by at least 20.2% in all criteria. Following the suggestion, we have also added a VLM-as-a-judge evaluation, where Claude Opus 5.5 chooses the image it prefers overall among the four methods for all 5,000 COCO prompts. TAG is chosen for 63.5% of the prompts, compared with 28.1% for CFG, 8.0% for SEG, and 0.5% for TPG. This indicates that the gains of TAG in the human preference metrics agree with both human and VLM judgments. We have described the protocols in Appendix B. **Table R2** : Human and VLM evaluation of guidance methods on 40 prompts randomly selected from the COCO validation split with Flux.1-dev. We report the percentage of votes from 35 raters, each of whom chooses the best image among the four methods for each criterion, and the percentage of 5,000 prompts for which Claude Opus 5.5 chooses each method (VLM). Bold indicates the best result. $$\begin{array}{l c c c c} \hline \textbf{Criterion} & \textbf{CFG} & \textbf{SEG} & \textbf{TPG} & \textbf{TAG (ours)} \newline \hline \text{Prompt adherence} & 30.4 & 13.6 & 5.3 & \textbf{50.6} \newline \text{Image quality} & 27.3 & 16.4 & 3.1 & \textbf{53.2} \newline \text{Overall preference} & 26.9 & 11.8 & 2.7 & \textbf{58.6} \newline \hline \text{Overall preference (VLM)} & 28.1 & 8.0 & 0.5 & \textbf{63.5} \newline \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer yCYs (2)

W3 : Inference latency, memory consumption, and computational overhead relative to CFG and existing training-free methods.

  • Thank you for pointing this out. TAG requires two forward passes per sampling step, exactly as CFG and all training-free baselines, since every guidance method in Eq. (3) needs a weak prediction in addition to the main one. The only additional computation at inference time is an element-wise division of the attention logits in Eq. (4), since the layers to perturb are selected once per model before sampling. We have added in Table 19 of Appendix D (Table R3) the latency and peak memory of all methods for generating a single image with Flux.1-dev on an NVIDIA RTX PRO 6000 GPU. All guidance methods show similar latency (14.45 to 15.05 s) and peak memory (32.71 to 32.72 GiB).

Table R3 : Latency and peak memory of guidance methods for generating a single image of 512×512512 \times 512 with Flux.1-dev on an NVIDIA RTX PRO 6000 GPU. We report the median latency over two rounds of 10 images, each after 3 images for warm-up, where the main and weak predictions are computed in a single batch for all methods.

\hline \textbf{Method} & \textbf{Latency (s)} & \textbf{Peak memory (GiB)} \newline \hline \text{w/o guidance} & 7.75 & 32.08 \newline \text{PAG} & 14.87 & 32.71 \newline \text{SEG} & 14.49 & 32.71 \newline \text{TPG} & 14.92 & 32.71 \newline \text{SSG} & 15.05 & 32.71 \newline \text{CFG} & 14.75 & 32.72 \newline \text{TAG (ours)} & 14.45 & 32.71 \newline \hline \end{array}$$ **W4 : Whether the severe degradation of some baselines, particularly SSG, is inherent to the methods or due to the selected configurations.** - We appreciate your comment. We tune all baselines on 200 prompts from the training split of COCO, searching over the guidance weight, the perturbed layers, and the method-specific hyperparameter of SEG and SSG (Table 9). SSG and TPG perform well on SDXL with the settings of the original papers (Table 2), whereas they perform worse than unguided sampling on Flux.1-dev and SD3.5 (Tables 2 and 10). To verify that this is not due to the configuration, we have added in Table 15 of Appendix D (Table R4) results for different guidance weights $w$ on Flux.1-dev. IR of SSG is lower than that of unguided sampling (0.269) for all weights, and it decreases monotonically as $w$ increases. SSG performs better as the guidance is applied less, indicating that the perturbation designed for U-Net does not always work well for transformer-based models. **Table R4** : Quantitative results of guidance methods on 5,000 COCO prompts with Flux.1-dev in terms of IR for different guidance weights $w$. Methods and weights are sorted by the weight selected for each method, such that the underlined numbers on the diagonal correspond to the settings in Table 9. The first row shows unguided sampling for reference, which does not depend on $w$. Bold indicates the best result in each column. $$\begin{array}{l c c c c} \hline \textbf{Method} & w = 2 & w = 3 & w = 3.5 & w = 5.5 \newline \hline \text{w/o guidance} & 0.269 & 0.269 & 0.269 & 0.269 \newline \hline \text{SSG} & \underline{-2.080} & -2.228 & -2.252 & -2.277 \newline \text{SEG} & 0.708 & \underline{0.752} & 0.736 & 0.495 \newline \text{CFG} & \textbf{0.878} & \textbf{0.948} & \underline{0.949} & 0.603 \newline \text{TAG (ours)} & 0.732 & 0.924 & \textbf{0.966} & \underline{\textbf{1.062}} \newline \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer yCYs (3)

W5 : Eq. (4) may change both the attention selectivity among textual tokens and the attention allocated to textual versus visual tokens, and it is unclear which effect contributes to the improvements.

  • Thank you for the precise observation. We agree that Eq. (4) changes both the selectivity among textual tokens and the attention allocated to textual versus visual tokens, since the softmax in MM-DiT runs over the concatenated tokens. To separate the two effects, we have added in Table 14 of Appendix D (Table R5) results of a variant of TAG that changes the selectivity only, which we call TAG-LSE. It replaces SXC/τS_{XC} / \tau in Eq. (4) with SXC/τ+LSE(SXC)−LSE(SXC/τ)S_{XC} / \tau + \mathrm{LSE}(S_{XC}) - \mathrm{LSE}(S_{XC} / \tau), where LSE denotes the log-sum-exp function applied to each row. This preserves the sum of the exponentiated logits in each row of SXCS_{XC}, keeping the attention allocated to textual tokens unchanged.

  • TAG-LSE improves IR over unguided sampling from 0.269 to 1.024, and TAG improves it further to 1.062. This suggests that both effects contribute, and that the gains mainly come from the selectivity. Note that for SDXL, where the softmax of cross-attention runs over textual tokens only, Eq. (4) does not change the allocation, and TAG is still effective (Tables 2 and 3).

Table R5 : Quantitative results of TAG-LSE on 5,000 COCO prompts with Flux.1-dev. The other rows are from Table 2. Bold indicates the best result in each column.

\hline \textbf{Method} & \textbf{IR}\uparrow & \textbf{PickScore}\uparrow & \textbf{HPSv2}\uparrow & \textbf{Aesthetic}\uparrow & \textbf{FID}\downarrow & \textbf{CLIP}\uparrow \newline \hline \text{w/o guidance} & 0.269 & 21.36 & 0.242 & 5.50 & 30.07 & 29.53 \newline \text{CFG} & 0.949 & 22.62 & 0.278 & 5.51 & \textbf{28.73} & \textbf{31.44} \newline \text{TAG-LSE} & 1.024 & 22.67 & 0.286 & \textbf{5.88} & 30.75 & 30.85 \newline \text{TAG (ours)} & \textbf{1.062} & \textbf{23.09} & \textbf{0.290} & 5.85 & 30.61 & 31.04 \newline \hline \end{array}$$

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

Dear Area Chair and Reviewers,

We thank the area chair and the reviewers for the time and care they put into reviewing our submission.

In this work, we propose text attention guidance (TAG), a training-free guidance method that perturbs the attention from visual to textual tokens, based on our analysis that aligning samples with the condition is the main bottleneck of guidance for complex conditions. The reviewers find the paper clear and easy to follow (yCYs), the method simple, robust, and motivated by a good observation (j83z), and the evaluation broad in terms of architectures, baselines, and benchmarks (yCYs, 4t2t, j83z).

We have addressed every comment below and uploaded a revised manuscript, where all changes are marked in blue so that they are easy to locate. Table and figure numbers in our responses follow the revised version. The main revisions are summarized as follows:

  • Clearer definitions of condition guidance (CG) and target guidance (TG) (Sec. 1 and Sec. 3.2).
  • Random layer selection (Table 6 in Sec. 4.3) and results for different numbers of layers on Flux.1-dev and Qwen-Image (Table 12 in Appendix D).
  • Comparison of perturbing different blocks of the attention logits (Table 7 in Sec. 4.3).
  • Human and VLM-as-a-judge evaluations (Table 8 in Sec. 4.3).
  • Head-level approach (Table 13 in Appendix D).
  • Variant of TAG that preserves the attention allocated to textual tokens (Table 14 in Appendix D).
  • Results for different guidance weights (Table 15 in Appendix D).
  • Results for different numbers of sampling steps (Table 16 in Appendix D).
  • Combination with CFG (Tables 17 and 18 in Appendix D).
  • Computational cost (Table 19 in Appendix D).
  • More qualitative results (Figs. 1 and 5, and Figs. 10-13 in Appendix C), including different temperatures and numbers of layers (Fig. 14 in Appendix D) and examples for the CLIP score (Fig. 15 in Appendix D).

We believe that the revision resolves the concerns raised, and we would be glad to answer any further questions during the discussion period.

Best regards,

The authors

Dear Reviewers,

The authors have posted their responses. Could you please read the rebuttal and let us know whether it resolves the questions and concerns raised in your review? If any point remains unclear, feel free to follow up with the authors during the discussion period.

Please update your review and final rating at your earliest convenience.

Thank you very much for your time and contribution.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.