- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduce a new training-free guidance metric for diffusion models. The authors separate two sources of error, condition guidance and target guidance. The authors show that the main bottleneck of text-conditional generation is in the conditioning method with a toy experiment, and based on this observation, they propose tho flatten the attention logits from visual-to-textual tokens with temperature scaling, so that this could serve as a weak prediction in guidance. The layers with the lowest attention entropy are selected to be flattened. The method is applicable to diverse TTI diffusion models, with consistently good performance.
Critical strengths
- The method is applicable to a diverse models, from U-Net based Diffusion models to recent MM-DiT architectures, whereas many of the recent studies are specifically designed for one type of architecture.
- The method is incredibly simple and seems robust, with an approach well-motivated from a good observation.
Critical weaknesses
-
Empirical Analysis with Toy Experiments: the toy experiment is interesting and greatly supports the method. However, I have a few concerns: i) I think CG and TG should be a bit more clearly defined. Although I can grasp the overall concept, but I think it is hard to be articulated precisely (especially TG). This is a concern in connection to the following concern ii) How can CG and TG be separated in practical TTI setting, as I believe it is difficult whether it's CG or TG in real use case. iii) does the observation still hold in the practical TTI setting? I wonder since this could be something which holds under a conceptual toy experiment.
-
Potential contradiction in arguments: the paper's main argument and motivation is on providing a better conditioning to the model. Based on this intuition, I expect the CLIP scores to also rise greatly. However, the CLIP scores fall behind against CFG, or scores in par, not as much as expected. I think this could potentially be a contradictory to the main argument. Could the authors provide an explanation on this, or clarifications in case I have misunderstood anything?
Questions for authors
-
Have the authors considered analyzing the entropy in terms of the attention heads, instead of layers, since each head could differ within a layer? A head in a layer could already have a flat attention distribution, whereas another head in that specific layer could have a very peaky distribution.
-
Study on 'w' for extrapolation (guidance): I wonder how the performance changes with different 'w' values; the approach showed to be robust with different temperatures, how is it with different 'w's? Each baseline guidance method use different 'w' values, so I wonder how they each perform under same/different 'w's.
-
Could the authors provide an explanation on why TAG outperforms baselines in IR by a huge margin, whereas gains in other metrics are somewhat marginal?
6 comments
Response to Reviewer j83z (2)
W2 : CLIP scores do not rise as much as expected, which may contradict the main argument.
-
Thank you for raising this point, which we would like to clarify. The CLIP score of TAG does rise over unguided sampling on all four models (e.g., from 29.53 to 31.04 on Flux.1-dev), and it is the highest among training-free methods on Flux.1-dev, SDXL, and SD3.5 (Tables 2 and 10). It is also comparable to that of CFG (e.g., 31.04 vs. 31.44 on Flux.1-dev).
-
The CLIP score is, however, known to be unreliable for precise prompt alignment, since CLIP behaves like a bag of words and does not capture how attributes and relations are bound to objects [1, 2]. We have added in Fig. 15 of Appendix D examples where CFG obtains higher CLIP scores than TAG, although it misses parts of the prompt.
-
VQAScore, which agrees with human judgments better [1], is more reliable. In terms of VQAScore, TAG outperforms all training-free methods on all four models, and even CFG on Flux.1-dev and SDXL (Tables 3 and 11). To verify prompt alignment more directly, we have also added in Table 8 (Table R3) a human evaluation, where 35 raters choose the image that follows the prompt best among those of CFG, SEG, TPG, and TAG for 40 random COCO prompts with Flux.1-dev. TAG receives 50.6% of the votes, compared with 30.4% for CFG, suggesting that TAG improves prompt alignment in terms of human judgments, although its CLIP score is lower than that of CFG.
Table R3 : Human and VLM evaluation of guidance methods on 40 prompts randomly selected from the COCO validation split with Flux.1-dev. We report the percentage of votes from 35 raters, each of whom chooses the best image among the four methods for each criterion, and the percentage of 5,000 prompts for which Claude Opus 5.5 chooses each method (VLM). Bold indicates the best result.
\hline \textbf{Criterion} & \textbf{CFG} & \textbf{SEG} & \textbf{TPG} & \textbf{TAG (ours)} \newline \hline \text{Prompt adherence} & 30.4 & 13.6 & 5.3 & \textbf{50.6} \newline \text{Image quality} & 27.3 & 16.4 & 3.1 & \textbf{53.2} \newline \text{Overall preference} & 26.9 & 11.8 & 2.7 & \textbf{58.6} \newline \hline \text{Overall preference (VLM)} & 28.1 & 8.0 & 0.5 & \textbf{63.5} \newline \hline \end{array}$$ [1] Evaluating text-to-visual generation with image-to-text generation, ECCV 2024. [2] When and why vision-language models behave like bags-of-words, and what to do about it?, ICLR 2023. **Q1 : Head-level approach instead of layer-level approach used in TAG.** - Thank you for the constructive comment. Following the suggestion, we have added in Table 13 of Appendix D (Table R4) a head-level variant of TAG. It computes the entropy in Eq. (5) for each attention head, and perturbs the 384 heads of lowest entropy across all layers, which matches the number of heads in the 16 layers of TAG. The two provide similar results (e.g., IR of 1.030 vs. 1.062), and both outperform CFG and existing training-free methods (0.949 and at most 0.752 in Table 2, respectively). This suggests that the key is that TAG is designed as CG, rather than the granularity of the selection. We keep the layer-level selection, since it is simpler. **Table R4** : Quantitative results of TAG on 5,000 COCO prompts with Flux.1-dev for selecting attention heads or layers to perturb. The head-level variant perturbs the 384 heads of lowest entropy across all layers. $$\begin{array}{l c c c c c c} \hline \textbf{Selection} & \textbf{IR}\uparrow & \textbf{PickScore}\uparrow & \textbf{HPSv2}\uparrow & \textbf{Aesthetic}\uparrow & \textbf{FID}\downarrow & \textbf{CLIP}\uparrow \newline \hline \text{Head-level} & 1.030 & 22.91 & 0.286 & 5.81 & 31.06 & 30.84 \newline \text{Layer-level (TAG)} & 1.062 & 23.09 & 0.290 & 5.85 & 30.61 & 31.04 \newline \hline \end{array}$$Response to Reviewer j83z (3)
Q2 : Study on the guidance weight .
- Thank you for the suggestion. We have added in Table 15 of Appendix D (Table R5) results of SSG, SEG, CFG, and TAG on Flux.1-dev under the same guidance weights. TAG outperforms the training-free baselines for all weights, and performs best for a sufficiently large weight (). SSG degrades as increases and performs worse than unguided sampling (0.269) for all weights.
Table R5 : Quantitative results of guidance methods on 5,000 COCO prompts with Flux.1-dev in terms of IR for different guidance weights . Methods and weights are sorted by the weight selected for each method, such that the underlined numbers on the diagonal correspond to the settings in Table 9. The first row shows unguided sampling for reference, which does not depend on . Bold indicates the best result in each column.
\hline \textbf{Method} & w = 2 & w = 3 & w = 3.5 & w = 5.5 \newline \hline \text{w/o guidance} & 0.269 & 0.269 & 0.269 & 0.269 \newline \hline \text{SSG} & \underline{-2.080} & -2.228 & -2.252 & -2.277 \newline \text{SEG} & 0.708 & \underline{0.752} & 0.736 & 0.495 \newline \text{CFG} & \textbf{0.878} & \textbf{0.948} & \underline{0.949} & 0.603 \newline \text{TAG (ours)} & 0.732 & 0.924 & \textbf{0.966} & \underline{\textbf{1.062}} \newline \hline \end{array}$$ **Q3 : Why does TAG outperform baselines in IR by a large margin, whereas gains in other metrics are marginal?** - Thank you for the question. We conjecture that this is because IR reflects prompt alignment more than PickScore and HPSv2. IR agrees with human judgments of alignment better than them [1] (e.g., pairwise accuracy of 67.3% vs. 59.4% and 55.2% on TIFA160). Note that IR builds on BLIP [2], where textual features attend to image features through cross-attention, while PickScore and HPSv2 build on CLIP, which compares a single embedding of an image with that of a text. Since TAG targets condition alignment, its gains are reflected in IR more clearly. - Note that the gains in the other metrics are not marginal either, considering that the metrics have different scales. We show in Table R6 the gains of TAG over the best training-free baseline on Flux.1-dev, normalized by those of CFG over unguided sampling, where the results are from Tables 2 and 3. The normalized gains are 46% (IR), 67% (PickScore), and 71% (VQAScore). The gain is small for HPSv2 only (6%), where SEG already exceeds CFG (0.288 vs. 0.278). TAG also outperforms all training-free methods in PickScore and HPSv2 on all four models (Tables 2 and 10), and it receives 53.2% of the votes for image quality in the human evaluation (Table 8), compared with 16.4% for SEG. **Table R6** : Gains of TAG over the best training-free baseline on Flux.1-dev, normalized by those of CFG over unguided sampling, i.e., (TAG $-$ baseline) / (CFG $-$ w/o guidance). IR, PickScore, and HPSv2 are from Table 2 (COCO), and VQAScore is the overall score in Table 3 (GenAI-Bench). $$\begin{array}{l c c c c c} \hline \textbf{Metric} & \textbf{w/o guidance} & \textbf{Best training-free baseline} & \textbf{CFG} & \textbf{TAG} & \textbf{Normalized gain (\\%)} \newline \hline \text{IR} & 0.269 & 0.752\ \text{(SEG)} & 0.949 & 1.062 & 46 \newline \text{PickScore} & 21.36 & 22.25\ \text{(SEG)} & 22.62 & 23.09 & 67 \newline \text{HPSv2} & 0.242 & 0.288\ \text{(SEG)} & 0.278 & 0.290 & 6 \newline \text{VQAScore} & 0.680 & 0.713\ \text{(PAG)} & 0.750 & 0.763 & 71 \newline \hline \end{array}$$ [1] Evaluating text-to-visual generation with image-to-text generation, ECCV 2024. [2] BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation, ICML 2022.
Response to Reviewer j83z (1)
We thank the reviewer for the positive assessment and the thoughtful questions. Our responses follow the order of the review, and the changes are marked in blue in the revised manuscript.
W1 : CG and TG should be defined more clearly, it is unclear how they are separated in the practical setting, and whether the observation holds there.
Table R1 : Comparison of condition guidance (CG) and target guidance (TG) in terms of what is degraded in the weak model, with the corresponding weak models in our analysis and examples in the practical setting.
\hline & \textbf{Degraded in the weak model} & \textbf{Our analysis} & \textbf{Practical examples} \newline \hline \text{CG} & \text{the condition, or its injection into visual tokens} & \text{narrower encoder} & \text{CFG, TAG} \newline \text{TG} & \text{visual tokens, or the operations among them} & \text{narrower denoiser} & \text{SEG, TPG, } S_{XX} \text{ in Table R2} \newline \hline \end{array}$$ **Table R2** : Quantitative results on 5,000 COCO prompts with Flux.1-dev for dividing different blocks of the attention logits in Eq. (2) by the temperature $\tau$. Bold indicates the best result in each column. $$\begin{array}{l c c c c c c} \hline \textbf{Perturbed block} & \textbf{IR}\uparrow & \textbf{PickScore}\uparrow & \textbf{HPSv2}\uparrow & \textbf{Aesthetic}\uparrow & \textbf{FID}\downarrow & \textbf{CLIP}\uparrow \newline \hline \text{None (w/o guidance)} & 0.269 & 21.36 & 0.242 & 5.50 & \textbf{30.07} & 29.53 \newline S_{XX} & 0.472 & 21.79 & 0.270 & 5.77 & 36.31 & 29.80 \newline S_{XC}\ \text{(TAG)} & \textbf{1.062} & \textbf{23.09} & \textbf{0.290} & \textbf{5.85} & 30.61 & \textbf{31.04} \newline \hline \end{array}$$