- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies counting failures in text-to-image models when objects are described through repeated noun mentions rather than numerals. Specifically, the paper designs controlled prompts to separate counting from color binding and conducts experiments on SD3.5-medium and FLUX.1-dev. The paper then uses attention interventions to analyze the failures and finds that restoring the attention of a mention can recover its color but not its instance. Finally, the paper proposes EMAR, a correction method that improves results under a selective application protocol.
Critical strengths
- The paper identifies a counting gap that numeral-based benchmarks may miss, which is an interesting and important finding.
- The paper designs controlled prompts to separate different factors, such as noun repetition, color differences, counting, and binding, which makes the analysis clear.
- The paper conducts attention experiments with several controls and acknowledges the limitation that the mechanism does not directly transfer to FLUX.
Critical weaknesses
- The reported EMAR gains rely on applying the method only to images where the baseline undercounts, so they do not reflect performance in a single generation pass. The paper should report results without this selection, as well as the additional generation.
- The main results rely on a single VLM to judge object counts and color combinations, which may affect the distinction between fusion and omission. The paper is suggested to add human evaluation on the main results.
- The description of EMAR is inconsistent between Section 6 and Appendix, such as the choice of tokens and the value of b. The paper should specify which version is used for each result.
Questions for authors
- How does EMAR perform without selecting images based on completed baseline outputs?
- What is the full computational cost?
- Can a human review confirm the main counting and failure type results?