acceptodds
Under review as a conference paper at ICLR 2027

A2I-Bench: Measuring the Value of Clarification in Interactive Text-to-Image Generation

Abstract

Interactive text-to-image systems ask for clarification when a user prompt admits multiple plausible interpretations. Yet ambiguity alone does not determine whether clarification is useful: the same user intent may already match one generator's behavior under the original prompt while conflicting with another's. To formalize this issue, we introduce A2I-Bench, a new benchmark of 527 ambiguous text-to-image prompts for measuring the value of clarification (VoC). Each case contains two linguistically plausible, visually distinguishable user intents together with intent-specific visual contracts, enabling matched counterfactual evaluation while holding the prompt, generation system, and image budget fixed. We operationalize clarification as revealing the user's intended interpretation. For each generation system, we estimate a Direct-based default interpretation from independent direct generations and measure the gain from intent revelation relative to a frozen no-question rewrite. Across four generation systems, revealing the intended interpretation improves contract satisfaction by 0.158 when the user's intent conflicts with the Direct-based default, a 0.092 larger gain than when they are aligned. These results show that clarification value depends on whether the no-question generation behavior already realizes the user's intent, rather than on prompt ambiguity alone. Because generator defaults and clarification gains are estimated separately for each system, A2I-Bench provides a reusable protocol for evaluating clarification value in new text-to-image generators. We will release the benchmark and code.

Primary area:
datasets and benchmarks
Keywords:
Interactive Text-to-Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MEDdOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a benchmark to evaluate the value of clarifying ambiguous instructions for text-to-image models and how that can help improve the alignment of the model achieve the outcome intended by the user. To do this, ambiguous statements are created (e.g., "the red pitcher and glass" which could mean the glass should be red or not) and sent to different T2I models. After that, the intention is clarified (should the glass be red or not) and the T2I model is queried again with the intention made clear (all of this can also happen with a prompt rewriter in the loop). The benchmark is run with several T2I models and evaluates the gain (how much the outcome improves if the intent is made clear vs not) and executability (how much a T2I model can stick to the revealed intention).

Critical strengths

Figuring out the intent of a user to make sure the produced outcomes are useful is of high value. This benchmark provides a controlled setting to evaluate exactly that.

Critical weaknesses

The approach seems to be pretty artificial to me, given that the prompts need to be artificially created to be ambiguous which is not how most users usually prompt. Even then, once you have an ambiguous prompt, checking how good different models are at "guessing" the user's intent (not even really guessing, at that point it's mostly a question of the training data as the paper also notes) is interesting but not sure what is gained by this.

Then, once the prompt is "fixed to make the actual response clear it's just a matter of the model to stick to that prompt. I'm not sure how much is gained by measuring the executability since this is just evaluating basic text-image alignment which many previous approaches already evaluate. I.e., I assume executability is directly correlated to how well a model can follow an input prompt, which is something that's already studied a lot for most of the models.

The approach claims that only in some eval settings a rewriter is used but for many close-source models or APIs there is no control over this. E.g., when calling GPT-Image I assume a rewriter will always be called, regardless of what the prompt is.

Questions for authors

It's not really clear to me what to take from the evaluation and conclusion as (I think) it's not really surprising or novel I think: if the prompt is ambiguous the model may or may not get it "right", once you make the prompt clear the model is much more likely to get it right, depending on its capability of accurately following the textual instruction. Is the conclusion, therefore, just that people should not prompt T2I models with ambiguous instructions?

1 comment

We thank the reviewer for the feedback. As context for our responses: when an interactive system should ask a clarifying question before acting, instead of how users should phrase their requests, is an established research question in both language and text-to-image generation [2–7]. A2I-Bench adds a paired, per-generator measurement of how much the user’s answer is worth, and shows that this value differs across generators. We address each concern below.

Q1: The prompts are artificially constructed, which is not how users usually prompt.

A1: 301 of the 527 prompts are built on real user prompts in DiffusionDB [1]. Underspecified requests are a routine part of real interaction with deployed text-to-image systems: users rarely state attribute scope, exact counts, or spatial layout, which is why interactive clarification has become an active research topic [2, 3]. The synthetic cases add controlled coverage of ambiguity types that are sparse in the wild, following prior ambiguity benchmarks such as TAB [2].

Q2: The result is unsurprising. Is the conclusion just that users should avoid ambiguous prompts?

A2: No. Users often do not realize their prompt is ambiguous, so the decision falls on the system: ask, or draw? Our finding is that this cannot be decided from the prompt alone.

  • On prompts with a stable default for all four systems, at least two systems prefer different readings in 37.3% of cases.
  • Revealing a conflicting intent improves contract satisfaction by 0.279 for GPT-Image-2 but only 0.023 for FLUX.1-dev. Averaged over the two equally weighted intents, the gains are 0.202 and 0.013; at an interaction cost of 0.05, the net values are +0.152 and −0.037 (Table 8).
  • A prompt-only predictor explains almost none of the clarification value on an unseen generator (R² = 0.009); adding observed generator behavior raises this to 0.179.

Prior work decides whether to ask from the request or the model's uncertainty about it [3, 4, 5, 6]. A2I-Bench measures what has been missing: how much the answer is worth to a particular generator.

Q3: Executability is just text-image alignment, which is already well studied.

A3: EsE_s explains a result that alignment scores alone cannot: clarification is nearly worthless for FLUX.1-dev on quantity and spatial prompts, because the model cannot reliably render the intent even once it is known. The measurement also differs from standard alignment benchmarks, which assume one correct reading per prompt. We score the same image against two competing intent contracts, which separates what a system draws by default from what it can draw when told.

Q4: Closed-source APIs such as GPT-Image may always apply an internal rewriter.

A4: This does not affect our conclusions. Direct means only that we add no external rewriter, and Rewrite is an explicit external baseline that we control. As stated in Section 4.1, the unit of evaluation is the generation stack, i.e., the model together with its interface, so any internal processing is part of the system being measured. This is the quantity that matters in deployment: the benefit of explicit user intent over what the whole pipeline would otherwise produce. Internal processing may also contribute to why defaults differ across systems, which is why we estimate them per stack.

References

[1] Wang et al. DiffusionDB: A Large-Scale Prompt Gallery Dataset for Text-to-Image Generative Models. ACL 2023.

[2] Mehrabi et al. Resolving Ambiguities in Text-to-Image Generative Models. ACL 2023.

[3] Hahn et al. Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty. ICML 2025.

[4] Zhang, Knox, and Choi. Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions. ICLR 2025.

[5] Zhang and Choi. Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs. Findings of NAACL 2025.

[6] Dong et al. Value of Information: A Framework for Human–Agent Communication. ACL 2026.

[7] Rao and Daumé III. Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information. ACL 2018.

Reviewer_QMK1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces A2I-Bench, a benchmark of 527 ambiguous text-to-image prompts. Each prompt comes with two human-validated interpretations and an intent-specific visual contract with positive and negative checks. The authors measure the value of clarification as the gain in contract satisfaction when the intended interpretation is revealed (Reveal), compared with a no-question rewrite (Rewrite). Users are labeled aligned or conflict depending on whether their intent matches the generator’s default interpretation under Direct generation. Across four generators, the gain is 0.158 in the conflict state, 0.092 higher than in the aligned state. The paper also shows that generators differ in their default interpretations and in how well they execute a resolved intent. The appendix adds analyses of interaction cost, stronger no-question baselines, and a prospective ask/no-ask policy based on generator behavior.

Critical strengths

S1. The counterfactual design is clean. Because the same images can be scored against both interpretations, the comparison between user states holds the prompt, generator, and image budget fixed. The statistical treatment is careful (cross-fitted defaults, a two-level bootstrap over construction groups), and the human validation of intents, with unambiguous and implausible controls, is thorough. I find the measurements credible.

S2. The leave-one-generator-out result in Appendix E.9 is the most useful finding to me. Prompt-side features alone barely predict clarification value on an unseen generator (R² = 0.009), while adding generator behavior raises this to 0.179. This is a meaningful negative result for approaches that decide whether to ask from the prompt alone, and I think it deserves a more prominent place in the main text.

Critical weaknesses

W1. I am not sure what the benchmark is meant to evaluate. If it is for comparing generators, a larger clarification gain is not obviously good: it may just mean the generator does worse without clarification. If it is for evaluating clarification policies, it needs comparisons with existing methods (see W3). If it is a diagnostic benchmark, the findings should go beyond known value-of-information principles (see W6).

W2. Only four systems are evaluated, and two of them are from the same model family. This makes it hard to draw general conclusions about how clarification value varies across generators. Native also uses a different conditioning interface for each system, so model capability and interface effects are confounded in E_s.

W3. Appendix E.9 only compares the authors’ own predictors. I would expect at least some standard policy baselines: always-ask, an LLM-based ambiguity detector, IntentSim (Zhang & Choi, Findings of NAACL 2025), and an adapted value-of-information method (Dong et al., ACL 2026). Without these, it is unclear whether the benchmark can separate good policies from bad ones. As the authors acknowledge, the reported utility also ignores the compute cost of Direct profiling and image self-checking.

W4. The prospective analysis assumes a uniform prior over intents, even though Appendix B.1 already estimates empirical intent distributions. It would be easy to rerun the analysis with these.

W5. Gain is measured against Rewrite, but alignment is defined using Direct. The two disagree on about 20% of untied units, and on that subset Δ flips to −0.328 (Appendix E.5). In other words, an intent labeled Direct-conflict may already be what Rewrite produces. Since a cross-fitted Rewrite analysis is already available, I do not see why Direct labels are used for the main results.

W6. In Appendix E.8, the conflict coefficient drops from 0.092 to −0.025 once Rewrite satisfaction is controlled for, and is close to zero with further controls. So the main contrast seems to mostly reflect how much headroom the no-question baseline leaves. That is what one would expect from value-of-information principles (Rao & Daumé III, ACL 2018; Dong et al., ACL 2026). I would like the authors to explain what the aligned/conflict distinction adds beyond this.

Questions for authors

See weaknesses

4 comments

We thank the reviewer for the careful reading. We agree that a larger clarification gain is not itself a ranking of generator quality, and that the aligned/conflict contrast is not a new value-of-information principle. We have therefore made the transfer result highlighted in S2 more prominent (Section 5.5: on an unseen generator, observing generator behavior improves policy utility over the prompt-side policy by 0.0138 [0.0094, 0.0186] at λ = 0.10), and added reference policies (W3), a fifth generator (W2), the three alignment definitions side by side (W5), and a decomposition of the remaining execution gap (W6).

W1: What is the benchmark meant to evaluate?

A1: Ask/no-ask policies, conditioned on a generator. For every prompt, generator, and candidate intent, A2I-Bench records the outcome of both branches (asking and not asking), so any policy can be scored offline without new generation. The default and executability measurements explain why the value of a policy differs across generators.

W2: Only four systems, two from one family; Native confounds capability and interface.

A2: We added Stable Diffusion 3.5 Medium as a fifth system from a different family, using the same frozen inputs (18,972 new images). It has the lowest resolved-intent executability of the five systems (0.783) and, under the original Direct-defined labels, the smallest conflict–aligned contrast (0.008; table under W6), close to FLUX.1-dev (0.020). Across the five systems, the Direct-defined contrast follows the executability ranking exactly (SD3.5 Medium 0.008, FLUX.1-dev 0.020, Qwen-Image 0.056, GPT-Image-1.5 0.139, GPT-Image-2 0.154). Under Rewrite-defined labels, SD3.5 Medium has a conflict gain of 0.085 and a contrast of 0.130. Pooled over the five systems, the contrast is 0.075 under Direct-defined labels (0.092 for the original four) and 0.188 under Rewrite-defined labels (0.203). Its defaults differ from those of the other four systems on 15–25% of jointly fold-stable prompts. Applied without retraining, the policy trained on the original four systems ranks clarification value better on this unseen system than the prompt-side policy (rank correlation 0.26 vs. 0.10) and achieves higher utility. On the shared family: the two GPT systems still disagree on 16.0% of fold-stable defaults (Table 4).

W3: No comparison with standard policy baselines; compute cost is ignored.

A3: We added the requested baselines, evaluated on the same held-out groups. The LLM-based policies use an open-weight model (Qwen3-VL-32B, text-only); 4B and 8B variants give the same ordering. Intent-Sim [1] and VoI [2] are adapted to our setting: the clarifying question and its two answers are fixed by the benchmark, so Intent-Sim becomes the entropy of the model's intent distribution, and the VoI rule asks when (1 − max_z q(z)) · D exceeds the interaction cost, with D the penalty for rendering the wrong reading, estimated on training folds.

PolicyUtility at λ = 0.10% of oracle
Always ask0.01216%
LLM ambiguity detector (zero-shot)0.00912%
VoI rule (adapted)0.01217%
Intent-Sim (adapted, with generator identity)0.04359%
Generator identity only0.04359%
Strongest prompt-side predictor0.04764%
Self-check (2 no-question images)0.05171%
Direct profile + self-check0.05272%
Oracle0.072100%

The benchmark separates these policies, and the generator-conditioned ones rank highest. On cost: the self-check scores two no-question images, which are delivered when the system does not ask and are extra only when it asks (1.0 image per unit on average at λ = 0.10), plus judge calls. On an unseen generator, the generator-conditioned policy improves utility over the strongest prompt-side policy by 0.014 at λ = 0.10, with the same direction on each of the four held-out systems.

References

[1] Zhang and Choi. Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs. Findings of NAACL 2025.

[2] Dong et al. Value of Information: A Framework for Human–Agent Communication. ACL 2026.

W4: The prospective analysis assumes a uniform intent prior.

A4: We agree that population-specific intent priors provide an important complementary prospective estimand. We re-ran the analysis using prompt-specific priors estimated from blind intent elicitation, replacing the balanced 1/2–1/21/2–1/2 mixture. With the E.9 feature sets unchanged, the generator-conditioned predictor improves held-out R2R^2 relative to the strongest prompt/context baseline from 0.295 to 0.376 under construction-group cross-validation and from 0.014 to 0.143 under leave-one-generator-out evaluation, with utility gains of +0.0076 and +0.0131 at λ=0.10\lambda=0.10, respectively. As an additional sensitivity analysis, the advantage persists across six fixed-prior settings from 0.6/0.4 to 0.9/0.1, skewed either toward or against the cross-generator Direct consensus: the held-out R2R^2 advantage ranges from +0.031 to +0.096 under construction-group cross-validation and from +0.050 to +0.161 under leave-one-generator-out evaluation, with positive utility gains in every setting.

W5: Gain is measured against Rewrite but alignment is defined by Direct.

A5: Following the reviewer's suggestion, we now report Rewrite-defined labels in the main text alongside the original Direct-defined analysis, with all three definitions side by side:

Label / comparatorConflict gainAligned gainΔ
Rewrite / Rewrite (cross-fitted)0.2130.0100.203
Direct / Rewrite0.1580.0660.092
Direct / Direct0.2000.0420.158

The contrast is positive under all three. The subset behavior in Appendix E.5 follows from a simple identity. If the no-question route guesses the intent and uses the same compiler as Reveal, the gain is zero when the guess is right and about 0.214 when it is wrong, equally in both states. The Direct-labelled contrast is then about 0.214 · (2h − 1), where h is how often the no-question guess agrees with the Direct default. With an LLM intent predictor, h = 0.69 and the identity gives 0.081, against a measured 0.085. Direct preference is thus a proxy for no-question behavior (the two agree on about 80% of units), which is why the Rewrite-defined contrast is larger.

W6: The contrast mostly reflects baseline headroom, as VoI principles predict. What does aligned/conflict add?

A6: As a predictor of gain, the aligned/conflict label adds nothing beyond headroom; we agree. We do not claim a departure from value-of-information principles. By VoI, the value of asking is the expected improvement in the final image once the answer is known. What A2I-Bench contributes is a measurement of the two generator-specific quantities this requires, and evidence that neither can be read off the prompt.

The first is headroom: how far the no-question output is from the user's intent. The second is recovery: how much of that headroom the generator realizes once told the intent. An exact decomposition of Δ under the original Direct-defined labels separates them:

No-question term (W_A − W_C)Reveal term (R_C − R_A)Δ
Pooled+0.181−0.0890.092
FLUX.1-dev+0.145−0.1250.020
Qwen-Image+0.170−0.1150.056
GPT-Image-1.5+0.208−0.0690.139
GPT-Image-2+0.201−0.0460.154
SD3.5 Medium (new)+0.132−0.1240.008

Both terms vary across systems, and they move together: the GPT systems leave the most headroom and also recover the most of it. Every system renders its non-default reading worse even when asked for it explicitly, by 0.046 for GPT-Image-2 and 0.125 for FLUX.1-dev. This shortfall is tied to the system’s own default, not just to how hard a reading is: with prompt fixed effects, identified from the 204 prompts on which systems disagree about the default, a system scores 0.083 higher on a reading when it is that system’s own default (0.086 when system fixed effects are added as well). Because both quantities vary by generator, a prompt-side estimate of the value of asking does not transfer: on an unseen generator, prompt features explain almost none of the clarification value (R² = 0.009), while the generator’s own no-question behavior explains R² = 0.179, the result the reviewer highlights in S2.

Thank you for the additional experiments. I found the cross-generator result informative: observing a new generator’s outputs helps predict clarification gains beyond the prompt alone. My main concern remains that the larger gains on conflict cases mostly reflect how poorly the no-question baseline already performs on those cases. So I’m keeping my score at 2.

In a future revision, I’d make the cross-generator result the main takeaway and use the conflict analysis to explain where that room for improvement comes from.

Thank you for engaging with our response and for the concrete suggestion. We agree on both points: the larger gain in conflict cases mostly reflects the headroom left by the no-question route, and the cross-generator result is the more informative finding. Following your suggestion, we have revised the abstract, the summary of findings in the introduction, the contribution list and the conclusion, so that the cross-generator result is the main takeaway and the conflict analysis explains where that room for improvement comes from (blue text in the uploaded revision). We would be grateful if you would take this into account in your final assessment.

Reviewer_CfsCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces A2I-Bench to study when asking clarifying questions actually helps in text-to-image generation. The main argument is that prompt ambiguity alone does not mean a system should ask a question; instead, the value of clarification depends on what the specific generator produces by default and whether it can actually render the resolved intent. The benchmark has 527 ambiguous prompts, each has two plausible user intents evaluated by atomic visual predicates (visual contracts). Testing across 4 models (GPT-Image-1.5, GPT-Image-2, FLUX.1-dev, and Qwen-Image), the authors show that different generators often prefer different interpretations of the same prompt, and revealing user intent gives a much larger gain when the intent conflicts with the generator's default.

Critical strengths
  1. Conditioning clarification value on the downstream generator makes a lot of sense. I especially like how the protocol seperates estimating the default preference (using 2-fold cross-fitting on 16 Direct images to avoid selection bias) from measuring clarification gain and native execution. It clearly shows why weaker compositional models like FLUX.1-dev barely benefit from clarification on spatial or quantity prompts, even when the ambiguity is resolved, the model still struggles to draw it.

  2. Benchmarks using LLM-generated intents and VLM judges can easily suffer from bias, so the human studies in Appendix B.1 and F are very reassuring. Showing that both intents naturally come up during blind human elicitation (65.2% of cases) and that the local Qwen3-VL-4B judge has ~90% agreement with human annotators makes the empirical findings convincing.

Critical weaknesses
  1. In the main text, aligned and conflict states are defined using the generator's Direct outputs, but the clarification gain is measured against a frozen Rewrite baseline (where Claude Sonnet 5 is forced to commit to one reading at temperature 0). This feels a bit inconsistent because the LLM rewriter imposes its own prior that can override the generator's native default. In fact, Table 6 (Appendix E.5) shows that when Direct and Rewrite disagree on the preferred intent (~19% of untied cases), the conflict-aligned gap basicly flips. Using this hybrid setup in the main paper makes the core metric harder to interpret than a pure Direct or pure Rewrite analysis.

  2. The intro asks "Should a T2I system ask before it draws?", but the main paper only does retrospective analysis where the user intent is already known. When looking at the actual prospective policy in Appendix E.9, the best predictor requires a "self-check" and "Direct profile" meaning the system has to generate images and run the VLM judge before deciding whether to ask. In practice, if we already spend compute and latency to generate images upfront, the system could just generate 2 images for z1 and 2 for z2 (within the same 4-image budget) and let the user pick, rather than asking a text question.

Questions for authors
  1. Why did you choose to pair Direct-defined alignment labels with the Rewrite comparator as the primary metric in the main paper, rather than directly reporting Reveal - Direct (or using the Rewrite-defined alignment from Table 6 as the main setup)?

  2. In Table 10 (Appendix F), the automatic judge has much lower negative sensitivity (.667 / .762) compared to positive sensitivity (.897 / .940). Since 73.4% of contracts rely on negative predicates to rule out the competing interpretation, does this lower sensitivity biases the scores for images that accidentally contain elements of both intents?

  3. For the prospective policy in Table 9, how many images need to be generated at test time before making an ask decision? Also, under the same 4-image budget, how would asking compare against a non-interactive "diverse rewrite" baseline that simply generates 2 images for each candidate interpretation?

1 comment

We thank the reviewer for the constructive feedback and the specific questions. We answer each below.

W1 / Q1: Alignment is defined by Direct, but gain is measured against Rewrite. Why not report Reveal − Direct, or use Rewrite-defined alignment?

A1: We now report Rewrite-defined alignment in the main text alongside the original analysis, with all three definitions side by side:

Label / comparatorConflict gainAligned gainΔ
Rewrite / Rewrite (cross-fitted)0.2130.0100.203
Direct / Rewrite0.1580.0660.092
Direct / Direct0.2000.0420.158

The contrast is positive under all three. The subset behavior in Table 6 follows from a simple identity. If the no-question route guesses the intent and uses the same compiler as Reveal, the gain is zero when the guess is right and about 0.214 when it is wrong, equally in both states. The Direct-labelled contrast is then about 0.214 · (2h − 1), where h is how often the no-question guess agrees with the Direct default. With an LLM intent predictor, h = 0.69 and the identity gives 0.081, against a measured 0.085. Direct preference is thus a proxy for no-question behavior (the two agree on about 80% of units), which is why the Rewrite-defined contrast is larger.

W2 / Q3: The best prospective predictor must generate images before deciding. How many? And how does asking compare with generating 2 images per interpretation?

A2: Images needed before the ask decision. The self-check scores two no-question images, which are the images the system delivers if it decides not to ask; they are extra only when it asks, 1.0 image per unit on average at λ = 0.10, plus judge calls. At interaction cost 0.10, policy utility is 0.047 for the strongest prompt-side predictor and 0.051 with the self-check alone (oracle: 0.072). The Direct profile (16 additional images) is needed only for transfer to an unseen generator, where it raises utility from 0.026 to 0.038.

The 2+2 baseline. We computed it from the existing Reveal images, averaging over all ways of choosing two images per interpretation, and scored against the user's contract with both users equally likely:

4-image budgetDirectRewrite2+2 diverseAsk, then 4 images
Mean of the 4 images0.7110.7200.7240.832
Best of the 4 images0.8020.7860.8820.890
At least one fully satisfying image0.4850.4500.6630.687

Asking gives the highest score on every measure. Its margin is largest when every image counts (0.832 vs. 0.724), since half of the diverse images render the other reading; this is the setting of a single delivered output, as in an API call or an automated pipeline. When the user selects one of four images, diversifying comes close to asking (0.882 vs. 0.890) and is the better choice at any interaction cost above 0.008, which includes every λ we report.

Diversifying is therefore a useful third action alongside committing and asking, and the benchmark evaluates all three offline. Its value is as generator-dependent as the value of asking: the best-of-4 gain over Rewrite is 0.029 for FLUX.1-dev and 0.163 for GPT-Image-2.

Q2: The judge has lower sensitivity on negative predicates. Does this bias scores for images containing elements of both intents?

A3: We tested this in three ways.

  • Few images receive full credit under both contracts: among no-question images, only 0.21% (Direct) and 0.06% (Rewrite) do.
  • Re-scoring every image with positive predicates only, the conflict gain is 0.139 and the conflict–aligned contrast is 0.065. The effect keeps its sign, and the ordering of the four systems is unchanged.
  • On the 240 prompts whose two contracts have the same number of negated checks, the contrast is 0.080, against 0.092 on the full benchmark.

These checks support the direction of the effect under alternative scoring choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.