- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a benchmark to evaluate the value of clarifying ambiguous instructions for text-to-image models and how that can help improve the alignment of the model achieve the outcome intended by the user. To do this, ambiguous statements are created (e.g., "the red pitcher and glass" which could mean the glass should be red or not) and sent to different T2I models. After that, the intention is clarified (should the glass be red or not) and the T2I model is queried again with the intention made clear (all of this can also happen with a prompt rewriter in the loop). The benchmark is run with several T2I models and evaluates the gain (how much the outcome improves if the intent is made clear vs not) and executability (how much a T2I model can stick to the revealed intention).
Critical strengths
Figuring out the intent of a user to make sure the produced outcomes are useful is of high value. This benchmark provides a controlled setting to evaluate exactly that.
Critical weaknesses
The approach seems to be pretty artificial to me, given that the prompts need to be artificially created to be ambiguous which is not how most users usually prompt. Even then, once you have an ambiguous prompt, checking how good different models are at "guessing" the user's intent (not even really guessing, at that point it's mostly a question of the training data as the paper also notes) is interesting but not sure what is gained by this.
Then, once the prompt is "fixed to make the actual response clear it's just a matter of the model to stick to that prompt. I'm not sure how much is gained by measuring the executability since this is just evaluating basic text-image alignment which many previous approaches already evaluate. I.e., I assume executability is directly correlated to how well a model can follow an input prompt, which is something that's already studied a lot for most of the models.
The approach claims that only in some eval settings a rewriter is used but for many close-source models or APIs there is no control over this. E.g., when calling GPT-Image I assume a rewriter will always be called, regardless of what the prompt is.
Questions for authors
It's not really clear to me what to take from the evaluation and conclusion as (I think) it's not really surprising or novel I think: if the prompt is ambiguous the model may or may not get it "right", once you make the prompt clear the model is much more likely to get it right, depending on its capability of accurately following the textual instruction. Is the conclusion, therefore, just that people should not prompt T2I models with ambiguous instructions?
We thank the reviewer for the feedback. As context for our responses: when an interactive system should ask a clarifying question before acting, instead of how users should phrase their requests, is an established research question in both language and text-to-image generation [2–7]. A2I-Bench adds a paired, per-generator measurement of how much the user’s answer is worth, and shows that this value differs across generators. We address each concern below.
A1: 301 of the 527 prompts are built on real user prompts in DiffusionDB [1]. Underspecified requests are a routine part of real interaction with deployed text-to-image systems: users rarely state attribute scope, exact counts, or spatial layout, which is why interactive clarification has become an active research topic [2, 3]. The synthetic cases add controlled coverage of ambiguity types that are sparse in the wild, following prior ambiguity benchmarks such as TAB [2].
A2: No. Users often do not realize their prompt is ambiguous, so the decision falls on the system: ask, or draw? Our finding is that this cannot be decided from the prompt alone.
Prior work decides whether to ask from the request or the model's uncertainty about it [3, 4, 5, 6]. A2I-Bench measures what has been missing: how much the answer is worth to a particular generator.
A3: Es explains a result that alignment scores alone cannot: clarification is nearly worthless for FLUX.1-dev on quantity and spatial prompts, because the model cannot reliably render the intent even once it is known. The measurement also differs from standard alignment benchmarks, which assume one correct reading per prompt. We score the same image against two competing intent contracts, which separates what a system draws by default from what it can draw when told.
A4: This does not affect our conclusions. Direct means only that we add no external rewriter, and Rewrite is an explicit external baseline that we control. As stated in Section 4.1, the unit of evaluation is the generation stack, i.e., the model together with its interface, so any internal processing is part of the system being measured. This is the quantity that matters in deployment: the benefit of explicit user intent over what the whole pipeline would otherwise produce. Internal processing may also contribute to why defaults differ across systems, which is why we estimate them per stack.
References
[1] Wang et al. DiffusionDB: A Large-Scale Prompt Gallery Dataset for Text-to-Image Generative Models. ACL 2023.
[2] Mehrabi et al. Resolving Ambiguities in Text-to-Image Generative Models. ACL 2023.
[3] Hahn et al. Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty. ICML 2025.
[4] Zhang, Knox, and Choi. Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions. ICLR 2025.
[5] Zhang and Choi. Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs. Findings of NAACL 2025.
[6] Dong et al. Value of Information: A Framework for Human–Agent Communication. ACL 2026.
[7] Rao and Daumé III. Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information. ACL 2018.