acceptodds
Withdrawn from ICLR 2027

Filling the Slots: Repeated Mentions Get Their Attributes Before Their Instances in MMDiT Text-to-Image Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-image counting benchmarks phrase multiplicity almost exclusively with numerals ("three bicycles”). We show that enumerating the same objects as repeated mentions ("a bicycle, a bicycle and a bicycle”) instead lowers counting accuracy by 44.3 points in SD3.5-medium and 59.2 points in FLUX.1-dev, a failure that numeral-only evaluation cannot detect. The penalty is not explained by same-category numerosity: repeated mentions lose instances, and cues that distinguish them, such as contrastive determiners and distinct colors, recover far more instances in FLUX.1-dev than in SD3.5-medium, which instead renders the requested colors on too few objects. Scoring counting and color binding separately shows that both models bind colors once the count is right, so the asymmetry lies in instances. Distinct colors raise the separability of the mentions' attention maps early in both models, but this predicts success only in FLUX.1-dev. In SD3.5-medium, the two ways a mention loses its instance have distinct early signatures: an omitted mention's image-to-text attention decays on its own, whereas fused mentions keep their attention but are never given separate regions. Intervening on joint attention shows that restoring early attention to a mention causally restores its color, through the early image layout, but has no specific effect on its instance. A rendering correction built on this mechanism, EMAR, restores missing color bindings and, across six prompt structures, raises counting accuracy by 7–15 points while lowering omission and, wherever a noun repeats, fusion.

Primary area:
generative models
Keywords:
Text-to-Image Generation, Multimodal Diffusion Transformer, Failure Diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yCC6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies counting failures in text-to-image models when objects are described through repeated noun mentions rather than numerals. Specifically, the paper designs controlled prompts to separate counting from color binding and conducts experiments on SD3.5-medium and FLUX.1-dev. The paper then uses attention interventions to analyze the failures and finds that restoring the attention of a mention can recover its color but not its instance. Finally, the paper proposes EMAR, a correction method that improves results under a selective application protocol.

Critical strengths
  1. The paper identifies a counting gap that numeral-based benchmarks may miss, which is an interesting and important finding.
  2. The paper designs controlled prompts to separate different factors, such as noun repetition, color differences, counting, and binding, which makes the analysis clear.
  3. The paper conducts attention experiments with several controls and acknowledges the limitation that the mechanism does not directly transfer to FLUX.
Critical weaknesses
  1. The reported EMAR gains rely on applying the method only to images where the baseline undercounts, so they do not reflect performance in a single generation pass. The paper should report results without this selection, as well as the additional generation.
  2. The main results rely on a single VLM to judge object counts and color combinations, which may affect the distinction between fusion and omission. The paper is suggested to add human evaluation on the main results.
  3. The description of EMAR is inconsistent between Section 6 and Appendix, such as the choice of tokens and the value of b. The paper should specify which version is used for each result.
Questions for authors
  1. How does EMAR perform without selecting images based on completed baseline outputs?
  2. What is the full computational cost?
  3. Can a human review confirm the main counting and failure type results?
Reviewer_aCmPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores MMDiT diffusion models and reveals a mechanism that binds an instance and its attribute in a special case in which instances are repeated, each mentioned separately, potentially with a binded attribute. Behavioral study shows that this prompt setting is particularly challenging for the model to generate. The authors identify a potential cause of failure through attention decay and propose a simple correction through attention reinforcement that improves performances.

Critical strengths
  • Interpreting the mechanism of DiT and MMDit specifically is still underexplored, and I’m overall enthusiastic about the topic

  • The authors made an effort to go beyond observation and analysed an attention mechanism that potentially causes the failure, and suggested a simple approach to try to correct failures.

Critical weaknesses
  • While I’m supportive of interpretability research on DiT, the motivation for studying this specific failure mode is unclear to me. I’m not surprised by the drop in performance when translating numericity to repeated instances; the models were simply not trained for this format of prompting, as it is not the natural way to give instructions. I don’t see why any user will use this prompt formatting, as it is very unnatural. If I’m missing any specific use case or motivation, please elaborate on it.

  • In general the paper seems to be very technical with very little motivation, even the abstract dives into technical details immediately and the intro is non-existent. It feels more like a technical report than a paper.

  • I’m not surprised by the fact that matching a different attribute to each instance improves performance, as in this case it becomes the natural way to describe the prompt, and therefore this case is probably well represented in the training set.

  • Although the improvement with Emar is real, it recovers only a small margin of failures and therefore doesn’t explain the mechanism causing the other large portion of failures. Importantly, compared to the baseline, Emar improvement is mainly for color attribution and not for counting! Which, at least, seems to me like the main goal of the paper and the mechanism to reveal.

Questions for authors
  • What is a real use case for using a repeated mention prompt? Please explain the motivation for using the prompt setting and revise the manuscript to explain it carefully.

  • What is your explanation for the failure cases that the EMAR doesn't correct?

  • Compared to the baseline, EMAR mainly improves color binding and not numerosity. How does that align with the claims and main motivation of the paper?

Reviewer_j84GOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper is testing whether the counting ability of diffusion models is tied to using numerals or with multiplicity of the mention. They use the diffusion models SD3.5-medium and Flux.1-dev and scored by Qwen 2.5 VL is used as a scoring model. According to the testing the enumerated form costs 44.3 and 59.2 points of counting accuracy. They use a factorial design (noun duplication and color structure) splits failures into omission, fusion and overcount. A correction built on this named EMAR (Early Mention Attention Reinforcement), is reported to raise counting accuracy by 7-15 points across six prompt structures.

Critical strengths
  • The authors explore a robust behavioral finding that could matter for benchmark design.
  • The statistics and intervention controls are thorough in this paper and do back up the authors finding.
Critical weaknesses
  • EMAR's reported gains across prompt structures appear to depend on an oracle. EMAR is applied only to images that the evaluator has already scored as undercounted; all other images keep their baseline generation. This is not in the abstract. Are we sure the model is just getting better results do to a second attempt?
    • Table 15 seems to show EMAR applied to all images. At N1×A3 it breaks 41% of already-correct images.
  • The fusion/omission split and the causal account rest on an unvalidated judge and a small, narrow set of interventions. All grid results come from one 7B VLM run on downscaled 448 pixel images, with no human agreement study. The causal analysis uses the consensus of two VLM judges (κ = 0.75 between them, A10), but neither is compared with human labels. The author inspected only the 23 single-omission cases by hand. The causal claims rest on 19 SD3.5 cases at N1 which itself has its limits.
  • The contribution is narrow, one failure mode, without explanation for why it happens or how often it does in practice.
Questions for authors
  • Do you have unoracle gated EMAR results on all Table 12 structures?
  • How do fusion rates change if only perceptually distinct color triplets are used? Navy vs blue is very similar.
  • Where does the failure come from? Why is it happening? How often do public caption datasets express multiplicity with repeated mentions rather than numerals, and how common is this phrasing in real user prompts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.