acceptodds
Under review as a conference paper at ICLR 2027

Lost in Attention: Understanding and Correcting Compositional Failures in Diffusion Models

Abstract

Text-to-image diffusion models remain unreliable at simple compositional requests: objects go missing, counts are wrong, attributes attach to the wrong object and spatial relations are ignored. We trace these failures to how cross-attention allocates its budget: the start token absorbs most of it, the concept named first is favoured, and a concept whose share falls early never appears. The information the text transmits to the image factorises exactly into an allocation and a localization factor, so stating where an object should be cannot make it appear. We propose Mordant, a training-free guidance method that states compositional requirements as constraints on the frozen model's attention, with multipliers found by dual ascent along the sampling trajectory and a relation margin derived from the frame. On Stable Diffusion 1.4, Mordant raises GenEval from 0.45 to 0.70, T2I-CompBench colour binding from 0.37 to 0.69 and 2D spatial relations from 0.13 to 0.39, and VISOR object accuracy from 33.4% to 79.9%; with its constants unchanged it also raises GenEval on SD 2.1 (0.50 to 0.72), SDXL (0.58 to 0.71), SD3-medium (0.70 to 0.83) and SD3.5-medium (0.70 to 0.83).

Primary area:
generative models
Keywords:
Diffusion Models, Inference-time Guidance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7HPXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies compositional failures in text-to-image diffusion models and proposes Mordant, a training-free guidance method. Experiments show effective results across several datasets and diffusion models.

Critical strengths

1.This paper studies compositional failures from the perspective of attention allocation. The research motivation is clear and provides a useful perspective on the missing object problem.

2.This method combines object presence, spatial relations, and attribute binding in a unified constrained optimization framework. It uses dual ascent to adjust constraint multipliers during sampling, which offers some methodological novelty.

Critical weaknesses
  1. The information-theoretic decomposition describes the statistical relationship between attention tokens and spatial positions. But, it does not directly measure how semantic information affects the final generated image.

  2. The authors pointed out that existing spatial guidance methods mainly affect localization rather than attention allocation. However, both factors depend on the same attention matrix. Existing methods may also change attention allocation indirectly, so this claim is not fully justified.

Questions for authors
  1. Please clarify the relationship between the information-theoretic decomposition and the actual generation process.

  2. Please discuss whether existing spatial guidance methods can also change attention allocation. This would be helpful for clarifying the contribution of Mordant.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer 7HPX

We sincerely thank the reviewer for the valuable feedback and questions, which helped us improve the paper. In short, we added a proposition and an intervention experiment that connect the decomposition to the generated image, and we measured how much existing spatial guidance changes allocation. The revised paper is uploaded (changes in blue), and the references are at the end of the general response.

W1 / Q1. How does the decomposition relate to the generation process? We agree that eq. 3 by itself describes attention statistics. The revised paper now connects it to the generated image in two ways.

First, we show how the two factors enter the output of the layer. The cross-attention output is o(u)=n∑tEtpt(u)Vto(u) = n \sum_t E_t p_t(u) V_t. Summed over the positions, a token therefore adds nEt∥Vt∥n E_t \lVert V_t \rVert, and its map ptp_t decides where this goes (new Prop. 4, App. S). A word whose share EtE_t goes to zero adds nothing through the layer, wherever its map lies.

Second, we tested the effect on the final image with an intervention (new Fig. 2b, App. P). With no loss, we edited the attention of one object in the forward pass during steps 0-24. The test uses SD 1.4 on 250 prompts of the VISOR benchmark (two objects and a spatial relation) with two seeds each, which gives 500 images per arm, and every image is paired with the unguided run on the same prompt and seed. An object counts as found when VISOR's detector (OWL-ViT) finds it. When we multiply the attention of the second object by 0 / 0.25 / 0.5 / 2 / 4 and leave its map unchanged, the rate at which it is found goes from .614 to .206 / .310 / .420 / .744 / .810. The first object, which we did not edit, moves the opposite way (from .694 to .896 / .846 / .790 / .574 / .518). We repeated the test on SD 2.1, and the effect is the same and equally strong (App. P). We also replaced the map by a uniform one at an unchanged share, and this lowers the rate to .432.

These results show that, at a fixed map, the share of an object changes whether it is drawn, that the effect grows with the size of the change, and that the two objects compete for the share. Raising the share of one object alone does not help both objects to be drawn (.340 / .354 at ×2 / ×4, compared with .326), whereas the re-allocation, which raises the share of every object at its own positions, gives .426. The decomposition also predicts presence in the unguided images of this test. The term of an object in eq. 3 is its share times the concentration of its map, E⋅KL(p∥U)E \cdot \mathrm{KL}(p \parallel \mathcal{U}), and it predicts detection with AUC .835, compared with .752 for the share alone and .789 for the concentration alone (.847 on SD 2.1).

W2 / Q2. Can existing spatial guidance change allocation indirectly? Yes, and we agree with the reviewer on this point. The revised paper makes clear that spatial guidance does not act only on localization, and Prop. 5 gives the precise statement. A loss that is computed only from the normalised maps does not change when the attention of a token is rescaled, so it has no direct reason to raise allocation. It can still change allocation through the shared latent, as the reviewer says, and we measured by how much. We used the same VISOR prompts and seeds as above. Each method is paired with the unguided run, and the two numbers are for the first / second object (Table 23, App. O).

  • A centroid hinge and the order surprisal (the statistic of PSG), each used alone in our pipeline, raise the share by 13% / 18% and 19% / 19%.
  • Attend-and-Excite (diffusers implementation, 500 images) raises it by 19% / 19% over its 25 update steps. Its loss is on a renormalised attention peak, so it depends on both factors.
  • The effective-area term of Mordant is also a loss on maps, and it changes the share in the other direction (-9% / -3%).

Existing methods therefore do change allocation, but as a side effect whose size their losses do not control. Mordant differs in that it acts on allocation by design. The forward re-allocation gives every object more of the start token's attention at its own positions (+143% / +151% in share after the edit), and the floor keeps the share from decaying during sampling. On these prompts, the full method finds both objects in 80% of the images, compared with 70% for relation guidance alone and 33% for the unguided model (Table 23). App. R gives the gain of each component on the benchmarks of the paper (the re-allocation, for example, adds .022 on GenEval and 3.1 points of VISOR object accuracy), and App. A.1 now separates the two kinds of loss.

We hope this addresses the reviewer's concerns, and we are happy to clarify anything further.

Dear Reviewer 7HPX,

Thank you again for your time and your feedback on our paper. As the discussion period is about to close, we would be grateful to know whether our rebuttal and the revised paper address your concerns, including the questions of how the decomposition relates to the generation process and whether existing spatial guidance can change allocation indirectly. If anything is still unclear, or if further analysis would help, we are happy to provide it in the remaining time.

Best regards, The authors

Reviewer_YN28OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the compositional generation problem with diffusion models. The key observation is that the attention scores assigned to each pixel-token pair can be factorized into an allocation term and a localization term. The authors further propose a method to directly modulate cross-attention scores to fix the unsatisfying score assignment.

Critical strengths
  1. The compositional failure is a true problem in diffusion generation.
  2. The overall experimental gain is persuasive.
Critical weaknesses
  1. The writing is hard to understand. An obvious problem is that many common words are given specific meanings without explicitly explaining the meanings. For example, "the token’s share of the budget" is unclear since a reader might not know what "budget" actually means. This is a typical pattern of AI writing. I'm not against the use of AI tools, but they should be used to improve readability, not decrease it.
  2. I couldn't learn much from this paper. To be specific, the discussion of the mechanism observation in section 4 is too brief. The authors do draw some observations and conclusions, but the connection between them is not fully discussed. Furthermore, although the method also adopts the allocation-localization factorization, its link to the observation is not clear enough. Therefore, what I can take away is a rather complicated method without clear empirical/mechanism motivation, and some disconnected observations. Nevertheless, my first feeling is slightly better on this paper than most of the other submissions in my batch.
Questions for authors
  1. How should a reader combine the information from this paper with other studies addressing the compositional failure? I'm not questioning the novelty here, but trying to figure out how this study can fit into the bigger community picture and thus offer meaningful new knowledge.
  2. <sos> token taking a large propotion of the attention scores is a known phenomenon. For example, it might be the result of attention sink in LLMs, or because this token encodes the semantic of the whole sentence in certain text encoders. What do the authors add on top of this information?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer YN28

We sincerely thank the reviewer for the valuable feedback and questions, which helped us improve the paper. We agree with both weaknesses and revised the paper accordingly (changes in blue).

W1. Writing. We agree that we used several everyday words as technical terms without explaining them. They are now defined where they first appear (Sec. 1 and Sec. 4), and App. N is a glossary. For the reviewer's example, "the token's share of the budget", the attention at each image position is a probability distribution over the tokens of the prompt, so the tokens divide one unit of probability among them. We call this unit the attention budget, and a token's share is the part of it that this token receives. Its map is its attention normalised over the image, which shows where the token attends. The three conditions of Sec. 5 are now also stated in words.

W2. Section 4 and the method. We agree that Sec. 4 did not say how its observations are connected, or which part of the method follows from which one. The end of Sec. 4 now joins the measurements into one mechanism, and Sec. 5 opens by saying how the method follows from it. In short:

  1. At the positions of an object, the start token takes 84-89% of the attention, but it accounts for only 38-48% of what the layer writes there, because its value vector is about ten times shorter than a word's. This is why the method moves part of that attention to the object (the forward re-allocation).
  2. The objects divide what is left, and the second named one gets less. It is drawn less often (61.7% vs 69.1%), and its share falls faster during sampling (to .21 of its initial value, vs .29). An object that ends up missing has lost its share early (Fig. 15). The method therefore keeps every object's share above a minimum (the allocation floor).
  3. We then tested whether the share is a cause. On VISOR prompts and with no loss, we scaled the attention of the second object and left its map unchanged. The rate at which a detector finds it goes from .61 to .21 when the attention is set to zero and to .81 when it is multiplied by 4. The first object moves the opposite way (new Fig. 2b, App. P; the same on SD 2.1). A flat map also lowers the rate, so the method also keeps each map compact (the effective-area term).
  4. Guidance on the relation alone places the pair correctly whenever both objects are drawn (1.000 vs .663 unguided), but both are drawn in only 70% of the images, so the method keeps the terms above next to the relation.
  5. A spatial loss changes the share only as a side effect (+13-19% for relation guidance), which is why the re-allocation and the floor act on the share directly.

Beyond these points, the apart term keeps two objects apart when the prompt does not relate them, and without it the second object forms inside or beside the first (Fig. 11). Binding keeps an attribute on its noun, and it raises the rate at which both colours are right (Table 26). App. R gives the gain of each component. The effective-area term, for example, adds .037 on GenEval, and the floor and the separation condition add .01-.03 on the attribute sets.

Q1. How this fits with other studies. Earlier studies explain these failures in the training data, in the text encoder, in cross-attention and in the sampling process (references in the general response). In the text encoder, CLIP favours the first-named object (Abbasi et al. 2025). In cross-attention, an object can be neglected (Attend-and-Excite), or its attribute can leak to another object (SynGen). Our paper is at the cross-attention level, and its finding that the share of the second-named object falls faster agrees with the first-mention bias of the text encoder. What it adds is that the attention to an object consists of two quantities, how much it receives and how concentrated it is, and that these can be measured and changed one at a time (Fig. 2b, App. P). Sec. 2 now cites these studies, and we will add this discussion to the final version.

Q2. The start token as a sink is known. What is new? We agree that the sink is known, and the paper now cites it (Xiao et al. 2024 for language models and Darcet et al. 2024 for vision transformers). We add three things. First, the attention that the start token receives is much larger than what it writes into the image (point 1), so moving it to the object changes what is drawn. On SD 1.4 the re-allocation adds .022 on GenEval and 3.1 VISOR points. Second, the start token of CLIP does not encode the sentence. The text encoder is causal, so this token's vector is the same for every prompt, and a summary of the sentence can sit only at the end-of-text and padding tokens (App. S). Third, the effect depends on the backbone. On SDXL the start token's value vector is about 16 times longer than a word's, so we do not use the re-allocation there.

We hope this answers the reviewer's questions and are happy to discuss them further.

Dear Reviewer YN28,

Thank you again for your time and your feedback on our paper. As the discussion period is about to close, we would be grateful to know whether our rebuttal and the revised paper address your concerns, including the definitions of our terms and the connection between Sec. 4 and the method. If anything is still unclear, or if further analysis would help, we are happy to provide it in the remaining time.

Best regards, The authors

Based on the response and the revised manuscript, my major concern now is still the readability.

  1. The "budget" issue is specifically tackled. However, on one hand, the way to tackle it is not what I expect. The ideal solution should be to choose a suitable way that's commonly used in this research field to represent this concept, not to explain the word with the original word unchanged. On the other hand, I mention "budget" only as an example, and there're many similar issues in the manuscript, which are not fully fixed. The revision even introduces more of them, e.g., "The margin comes from the frame, which is not itself a condition of the program".
  2. Now that I understand how the method connects with the observation, a remaining problem is the disconnection between the key factorization of allocation-localization and the observation. The key factorization has only two terms, but there are many observations. They should be connected, too. I feel they can indeed be logically connected, but the revision has to be done explicitly.

In general, the manuscript cannot go published in such a readability state. It requires significant writing overhaul through human writing or an AI agent with proper writing skills installed. Hence, I tend to keep my original score.

By the way, I've read the novelty/contribution discussion of other reviewers. My feeling is that, supposing the writing is fixed, the potential contribution is moderate. Some findings in this submission are new while the others are not, but both are unified under a new factorization perspective. So, this can make a contribution that is meaningful but not substantial. Again, this shows that writing can really play an important role in the judgment of this manuscript.

We are grateful for the reviewer's feedback on the revised paper and for the suggestions to further improve it. We have uploaded an updated PDF that already contains the changes described below (the changes requested by the reviewers are marked in blue), and we will complete them in the final version.

Terminology. We thank the reviewer for this suggestion. We used words such as "budget" as an intuitive analogy for the mathematical quantities. As suggested, the updated PDF replaces them with concrete terms, for example:

beforein the updated PDF
budgetattention weights, which sum to one over the tokens at each image position (eq. 2)
frameimage borders
constrained programconstrained optimisation
"Diffusion models play favorites" (title of Sec. 4.2)"Attention is unevenly divided among tokens"
the start token's hold on the budgetthe attention that the start token receives
hooked layersselected layers

Given the limited time before the discussion closes, we will go through the rest of the text in the same way for the final version, with one standard term for each quantity.

The sentence on the margin. The updated PDF rewrites it (Sec. 5.2). To clarify, take the relation "A to the left of B". We ask that A's centre is at least mm to the left of B's centre, and that neither centre is closer than mm to the side of the image. These three gaps have to fit in the image width, so 3m≤13m \le 1, and 1/3 is the largest margin for which all three can hold. We only use the image border to get this number, and it is not one of the constraints in eqs. 5-7.

Connecting the factorization and the observations. As the reviewer notes, the observations can be logically connected to the two factors of eq. 3. The updated PDF states this connection at the start of Sec. 4.2, and we summarise it here.

  • Four observations concern allocation. First, the start token receives 84-89% of the attention weight at the positions of an object, so the words of the prompt divide the rest. Second, the allocation at the first 16×16 layer already predicts whether an object is drawn. Third, the object named second is drawn less often, and its allocation decays during sampling. Finally, the objects compete. When we scale up the attention of the second object and the start token supplies the difference, the first object is found less often (.69 to .52 at a factor of 4).

  • One observation concerns localization. A flat map at an unchanged allocation lowers presence from .61 to .43.

  • The last observation links the two factors. Guidance that only states where an object should be acts on localization, yet it also raises the allocation of the object by 13-19%, although nothing in its loss asks for this (Table 23).

Contribution. We thank the reviewer for also reading the discussion on the contribution, and we appreciate the remark that the factorization unifies the findings. To clarify what we consider new, we highlight three points.

  • A causal test shows that the first factor of eq. 3 changes whether an object is drawn. With no loss and no method involved, scaling the attention of an object at an unchanged map moves the rate at which it is drawn from .21 to .81 (the same holds on SD 2.1).

  • We measured how much existing spatial guidance changes allocation as a side effect (Table 23).

  • The conditions are weighted by multipliers that are found during sampling, in place of hand-set weights.

Mordant improves GenEval on five backbones, including the recent SD3-medium and SD3.5-medium. Each cell of the table below is the score of the unguided model → the score with Mordant (Table 1).

GenEvaloverallpositioncolour attribution
SD 1.4.45 → .70.06 → .71.08 → .40
SD 2.1.50 → .72.07 → .78.15 → .42
SDXL.58 → .71.11 → .68.23 → .42
SD3-medium.70 → .83.28 → .79.57 → .66
SD3.5-medium.70 → .83.26 → .78.56 → .70

On SD 1.4, the overall score is also above the best of the eight training-free methods that we ran (.57).

We sincerely appreciate the time that the reviewer spent on the revised paper. We hope that the updated PDF, especially the start of Sec. 4.2 and Sec. 5.2, addresses the concern on readability, and we are happy to clarify or revise anything further. We thank the reviewer again and will incorporate all answers into the final version of the paper.

Reviewer_JxyxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the missing objects, wrong binding attribute and mismatched spatial relationship for composable diffusion in text-to-image models. The authors divide the attention into the quota gained by each concept and normalized spatial distribution and propose a training-free guidance method Mordant. This method uses effective area, apart from different concepts and attention allocation floor to build the constraint with the augmented lagrangian and dynamic multipliers to guide the sampling process. For U-Net based baselines, authors also re-distribute the attention of the start token to the object token.

Critical strengths
  1. The problem is important and clearly presented. Attribute binding, missing objects and spatial relationships are the bottlenecks of text-to-image models, the separation of attention quota distribution and spatial distribution are important perspectives of those failures.

  2. Performance Improvement: The proposed Mordant enhances performance on GenEval using SD 1.4 from 0.45 to 0.70 and also improves four additional backbones, with particular gains on spatial benchmark.

  3. Extensive supplementary analysis: The paper includes evaluation with multiple detectors, comparisons against fixed-weight objectives, and failure cases

Critical weaknesses
  1. The theoretical analysis does not establish the claimed explanations: Equation 3 is an identity obtained by interpreting the attention as a probabilistic channel. It does not directly measure semantic information transmitted from text to the final image. Moreover, spatial guidance operating through shared latent variables may change both attention allocation and localization.

  2. The task-level justification for (m=1/3) is insufficient: Projecting the positions onto the relation direction is reasonable. However, theorem 2 relies on the additional assumption that the minimum gaps between object centers and between the centers and frame boundaries should be maximized. This assumption ignores the object size and spatial extent.

  3. The ablations do not clearly identify the main sources of improvement in the final method: For the ablation on the color category, removing binding reduces the color score from 0.69 to 0.62, since the metric depends on both object presence and attribute correctness, this does not isolate improved binding from reduced object omission.

  4. Evaluation on stronger backbones is incomplete: Comprehensive T2I-CompBench results are missing for stronger models such as SDXL and SD3. The evidence therefore does not establish broad transfer across shape, texture, and complex composition.

Questions for authors
  1. Can the authors provide sensitivity experiments for (m) reporting the spatial relationship performance change?

  2. Can the authors provide comprehensive T2I-CompBench results on stronger backbones?

  3. Can the authors report object presence with and without binding, and evaluate color assignment, color swapping, and attribute leakage?

  4. Can authors provide some qualitative metrics comparison and more detailed explanations for which term of equation 4 plays the most important roles?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer Jxyx

We sincerely thank the reviewer for the detailed feedback and questions, which helped us improve the paper. The revised PDF is uploaded, with the changes in blue.

W1. Eq. 3 is an identity, and spatial guidance may change both factors. We agree with both points.

  • On the link to the image, a token adds nEt∥Vt∥n E_t \lVert V_t \rVert to the output of the layer, and its map ptp_t decides where (new Prop. 4). With no loss, we scaled the attention of the second object by 0 / 0.5 / 2 / 4 and left its map unchanged. The rate at which a detector finds it goes from .61 to .21 / .42 / .74 / .81, and the first object, which we did not edit, goes the opposite way (new Fig. 2b, App. P; same slope on SD 2.1).
  • On spatial guidance, the reviewer is right that it also changes allocation, and the revised paper makes this clear. A loss on normalised maps has no direct incentive to raise EE (new Prop. 5), but it changes EE through the latent, and we measured by how much (Table 23, App. O). Relation-only guidance raises an object's share by 13-19%, and Attend-and-Excite by 19%. Mordant's floor alone adds 10-11%, which is not more than these side effects. The forward re-allocation raises the share far more (+143-151% after the edit).

W2 / Q1. m=1/3m = 1/3 and object size. Thm. 2 assumes the same gap between the two centres and from each centre to the frame. This premise is now stated in Sec. 5.2, and App. Q presents 1/3 as a default and not as an optimum. For the object size, we added Thm. 6 with the objects' half-widths and implemented it (last row). Here is the scan on SD 1.4 (App. Q; OA is the % of images with both objects found by OWL-ViT, cond the % of those with the relation correct).

mmGenEval all / positionVISOR OA / condT2I 2D
0.63 / .4876.4 / 98.20.34
0.25.68 / .6681.0 / 99.81.38
0.33.70 / .7179.9 / 99.97.39
0.50.72 / .7678.2 / 99.97.39
0.70.70 / .7373.2 / 99.97.37
size-aware.70 / .7578.3 / 99.90.40

With m=0m = 0, every benchmark is worse, and 0.25 does not differ significantly from 1/3. Above 1/3 the benchmarks disagree. GenEval's position task prefers a larger margin, whereas VISOR object accuracy falls and T2I 2D does not improve. The size-aware margin is not better overall (higher on GenEval position, lower on VISOR OA).

W3 / Q3. Binding vs presence. We agree that the colour score mixes the two. We graded every colour image again with OWL-ViT and GenEval's colour classifier, independently of BLIP (App. R). On the images where both runs draw both objects (n = 1,450), binding raises the rate at which both colours are right from .29 to .34 and reduces swaps from .03 to .02. Leakage is .46 in both runs (Table 26). Binding also costs some presence (both found .60 vs .68), so on SD 1.4 the two runs are level end to end (.20 vs .19; unguided .05).

W4 / Q2. Stronger backbones. We ran all eight T2I-CompBench categories on SDXL and SD3-medium (10 images per prompt; six categories are now in Table 2). Each cell is unguided → Mordant, with the gain in brackets.

colourshapetexture2D3Dnumeracy
SDXL.63→.74 (+.11).51→.56 (+.05).57→.66 (+.09).22→.35 (+.13).38→.38 (.00).52→.51 (-.01)
SD3-m.81→.83 (+.02).59→.60 (+.01).73→.75 (+.02).31→.47 (+.16).40→.43 (+.03).61→.64 (+.03)

Colour, shape, texture and 2D improve significantly on both backbones, and 3D and numeracy also improve on SD3-medium. Non-spatial (CLIP similarity) and complex change by less than .01.

Q4. Which term matters most, and a qualitative comparison. We considered the effect of each component on SD 1.4 (App. R). The table gives what each component adds (the full method minus the method without it).

componentGenEvalVISOR OA (OWL-ViT / Mask2Former / Grounding DINO)T2I 2DT2I colour
effective area+.04+9.6 / +1.5 / +2.3+.02+.03
multipliers (vs held at 0).00+9.0 / +11.5 / +7.8.00+.03
forward re-allocation+.02+3.1 / +2.8 / +1.6+.02-.01
apart term+.02n/an/a+.04
binding.00n/an/a+.08
floor-.01+0.6 / -0.2 / +0.9-.01+.01

(n/a means that the term cannot act there. Binding needs an attribute and the apart term needs a pair of objects that the prompt does not relate, and VISOR and T2I 2D prompts have neither.)

Among the terms of eq. 4, the effective area matters most (on VISOR mainly with OWL-ViT), then the apart term, and the floor least. Outside eq. 4, holding the multipliers at zero lowers VISOR cond from 99.97 to 89.76. For a qualitative comparison, Figs. 11-13 (App. M) show the same seed with and without the apart term and binding.

We hope this addresses the reviewer's concerns, and we are happy to clarify anything further.

Dear Reviewer Jxyx,

Thank you again for your time and your feedback on our paper. As the discussion period is about to close, we would be grateful to know whether our rebuttal and the revised paper address your concerns, including the results on SDXL and SD3-medium and the separation of binding from presence. If anything is still unclear, or if further analysis would help, we are happy to provide it in the remaining time.

Best regards, The authors

I thank the authors for the response and the new experiments. The results for SDXL and SD 3.0 solve my concerns regarding generalization, but my concerns over which term plays the most important role still remains. Your results indicate that simple heuristic terms such as pushing centers apart, shrinking covariance area etc are the main contributors to the performance. These heuristics are not new and in my experience don't scale to prompts with complex concept interactions. Further, forward re-allocation hardly affects T2I-CompBench attributes. So, the results show a disconnect between the main narrative and the components that drive actual performance. For these reasons I would like to keep my original score.

We sincerely thank the reviewer for the feedback on our response, and we are glad that the SDXL and SD3-medium results address the concern on generalization. The remaining concern is drawn from the component table that the reviewer asked for (Q4, Table 25), and we would like to clarify how this table relates to the analysis of the paper.

1. The effective area is the second factor of our decomposition. Eq. 3 says that the information a token sends to the image is its share times the concentration of its map, Et⋅KL(pt∥U)E_t \cdot \mathrm{KL}(p_t \Vert \mathcal{U}), and Sec. 4 tests both factors with no loss. Scaling the share at an unchanged map moves presence from .21 to .81, and a flat map at an unchanged share lowers it from .61 to .43. The effective area is the entropy power of the map, so lowering it raises the second factor (+0.22 nats, Table 23), and the floor and the forward re-allocation raise the first. The effective area, which adds most among the terms of eq. 4, therefore comes directly from the analysis.

2. The multipliers have the largest effect on VISOR. The contributions listed in Sec. 1 are the analysis, the constrained program with its multipliers, the derived margin and the forward re-allocation. When the multipliers are held at zero, so that the same conditions act as fixed penalties, object accuracy falls by 9.0 points and the relation accuracy falls from 99.97 to 89.76. Together, the rate at which an image has both objects in the asked relation falls by 16 points, which is the largest effect of any component. We found the same on SD 2.1 (7.6 points, 99.91 to 91.72).

3. The forward re-allocation acts on presence, and colour attribution improves on every backbone. The forward re-allocation gives the start token's attention to the object nouns and leaves the attribute words as they are, so it acts on presence. Alone, with no loss, it raises the rate at which both objects are drawn from .33 to .43 (Table 23), and in the full method it adds 3.1 points of VISOR object accuracy (Table 25). Attribute binding is the role of the binding condition. For prompts where concepts interact, the closest test in the paper is GenEval's colour-attribution task, which asks for two objects, each with its own colour. There Mordant improves on all five backbones with the same constants, for example from .08 to .40 on SD 1.4 and from .57 to .66 on SD3-medium (Table 1).

We are grateful for the reviewer's questions, which led to the new experiments in the revised paper (Fig. 2b, App. Q, Tables 2, 25 and 26). We hope this clarifies the role of each component, and we would be happy to provide any further analysis or clarification that would help. We thank the reviewer again for the time and the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of the changes

We sincerely thank all three reviewers for their valuable feedback, which helped us improve the paper. The revised paper is uploaded as the current PDF of the submission. The changes are in blue and are limited to what the reviews asked for, and most of the new material is in App. N-S. The papers that we cite in the replies are listed at the end of this response.

What was done

  1. We connected the decomposition (eq. 3) to the generated image. New Prop. 4 shows how each factor enters the output of the layer, and a causal test with no loss (new Fig. 2b) shows that, at an unchanged map, the share of an object changes whether it is drawn (Reviewers Jxyx and 7HPX).
  2. We clarified what spatial guidance changes. As both reviewers point out, it also changes allocation, through the shared latent. New Prop. 5 states this precisely, and Table 23 gives the size of the change (Reviewers Jxyx and 7HPX).
  3. We added four experiments. These are a scan of the margin with a size-aware version of Thm. 2, T2I-CompBench on SDXL and SD3-medium (the main categories are now in Table 2), presence and colour binding graded separately, and what each component adds on the reported benchmarks (Reviewer Jxyx).
  4. We improved the writing and the motivation of the method. The terms such as budget are defined where they are first used, with a glossary in App. N, and the three conditions of Sec. 5 are stated in words. The end of Sec. 4 now connects the measurements into one mechanism, and the opening of Sec. 5 says how the method follows from it (Reviewer YN28).
  5. We related the paper to earlier work. Sec. 2 now cites the studies of these failures in the text encoder. Sec. 4 cites the work on attention sinks, and App. S states what the paper adds about the start token (Reviewer YN28).

Summary of the results. When we scale the attention of the second object by 0 / 0.5 / 2 / 4, with no loss and an unchanged map, the rate at which a detector finds it goes from .61 to .21 / .42 / .74 / .81. Relation guidance alone raises the share of an object by 13-19%, as a side effect. The margin 1/3 is on the plateau of every benchmark, and a size-aware margin is not better overall. On SDXL and SD3-medium, Mordant significantly improves colour, shape, texture and 2D spatial. On the images where both objects are drawn, binding raises the rate at which both colours are right from .292 to .335. The details are in the replies below.

Papers cited in the replies

cited aspaper
Attend-and-ExciteChefer et al., "Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models", ACM TOG 2023
SynGenRassin et al., "Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment", NeurIPS 2023
PSGRezaei et al., "Why Settle for Mid: A Probabilistic Viewpoint to Spatial Relationship Alignment in Text-to-image Models", TMLR 2025
Abbasi et al. 2025"CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation", CVPR 2025
Xiao et al. 2024"Efficient Streaming Language Models with Attention Sinks", ICLR 2024
Darcet et al. 2024"Vision Transformers Need Registers", ICLR 2024

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.