- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies compositional failures in text-to-image diffusion models and proposes Mordant, a training-free guidance method. Experiments show effective results across several datasets and diffusion models.
Critical strengths
1.This paper studies compositional failures from the perspective of attention allocation. The research motivation is clear and provides a useful perspective on the missing object problem.
2.This method combines object presence, spatial relations, and attribute binding in a unified constrained optimization framework. It uses dual ascent to adjust constraint multipliers during sampling, which offers some methodological novelty.
Critical weaknesses
-
The information-theoretic decomposition describes the statistical relationship between attention tokens and spatial positions. But, it does not directly measure how semantic information affects the final generated image.
-
The authors pointed out that existing spatial guidance methods mainly affect localization rather than attention allocation. However, both factors depend on the same attention matrix. Existing methods may also change attention allocation indirectly, so this claim is not fully justified.
Questions for authors
-
Please clarify the relationship between the information-theoretic decomposition and the actual generation process.
-
Please discuss whether existing spatial guidance methods can also change attention allocation. This would be helpful for clarifying the contribution of Mordant.
2 comments
Dear Reviewer 7HPX,
Thank you again for your time and your feedback on our paper. As the discussion period is about to close, we would be grateful to know whether our rebuttal and the revised paper address your concerns, including the questions of how the decomposition relates to the generation process and whether existing spatial guidance can change allocation indirectly. If anything is still unclear, or if further analysis would help, we are happy to provide it in the remaining time.
Best regards, The authors
Reply to Reviewer 7HPX
We sincerely thank the reviewer for the valuable feedback and questions, which helped us improve the paper. In short, we added a proposition and an intervention experiment that connect the decomposition to the generated image, and we measured how much existing spatial guidance changes allocation. The revised paper is uploaded (changes in blue), and the references are at the end of the general response.
W1 / Q1. How does the decomposition relate to the generation process? We agree that eq. 3 by itself describes attention statistics. The revised paper now connects it to the generated image in two ways.
First, we show how the two factors enter the output of the layer. The cross-attention output is o(u)=n∑tEtpt(u)Vt. Summed over the positions, a token therefore adds nEt∥Vt∥, and its map pt decides where this goes (new Prop. 4, App. S). A word whose share Et goes to zero adds nothing through the layer, wherever its map lies.
Second, we tested the effect on the final image with an intervention (new Fig. 2b, App. P). With no loss, we edited the attention of one object in the forward pass during steps 0-24. The test uses SD 1.4 on 250 prompts of the VISOR benchmark (two objects and a spatial relation) with two seeds each, which gives 500 images per arm, and every image is paired with the unguided run on the same prompt and seed. An object counts as found when VISOR's detector (OWL-ViT) finds it. When we multiply the attention of the second object by 0 / 0.25 / 0.5 / 2 / 4 and leave its map unchanged, the rate at which it is found goes from .614 to .206 / .310 / .420 / .744 / .810. The first object, which we did not edit, moves the opposite way (from .694 to .896 / .846 / .790 / .574 / .518). We repeated the test on SD 2.1, and the effect is the same and equally strong (App. P). We also replaced the map by a uniform one at an unchanged share, and this lowers the rate to .432.
These results show that, at a fixed map, the share of an object changes whether it is drawn, that the effect grows with the size of the change, and that the two objects compete for the share. Raising the share of one object alone does not help both objects to be drawn (.340 / .354 at ×2 / ×4, compared with .326), whereas the re-allocation, which raises the share of every object at its own positions, gives .426. The decomposition also predicts presence in the unguided images of this test. The term of an object in eq. 3 is its share times the concentration of its map, E⋅KL(p∥U), and it predicts detection with AUC .835, compared with .752 for the share alone and .789 for the concentration alone (.847 on SD 2.1).
W2 / Q2. Can existing spatial guidance change allocation indirectly? Yes, and we agree with the reviewer on this point. The revised paper makes clear that spatial guidance does not act only on localization, and Prop. 5 gives the precise statement. A loss that is computed only from the normalised maps does not change when the attention of a token is rescaled, so it has no direct reason to raise allocation. It can still change allocation through the shared latent, as the reviewer says, and we measured by how much. We used the same VISOR prompts and seeds as above. Each method is paired with the unguided run, and the two numbers are for the first / second object (Table 23, App. O).
Existing methods therefore do change allocation, but as a side effect whose size their losses do not control. Mordant differs in that it acts on allocation by design. The forward re-allocation gives every object more of the start token's attention at its own positions (+143% / +151% in share after the edit), and the floor keeps the share from decaying during sampling. On these prompts, the full method finds both objects in 80% of the images, compared with 70% for relation guidance alone and 33% for the unguided model (Table 23). App. R gives the gain of each component on the benchmarks of the paper (the re-allocation, for example, adds .022 on GenEval and 3.1 points of VISOR object accuracy), and App. A.1 now separates the two kinds of loss.
We hope this addresses the reviewer's concerns, and we are happy to clarify anything further.