acceptodds
Under review as a conference paper at ICLR 2027

When Chekhov's Gun Fires Anyway: Over-Realization in Text-to-Image Models

15 views

Abstract

In text-to-image generation, a target that the prompt mentions often appears in the generated image in a way the prompt does not call for. We call this failure over-realization. Mental-spaces theory points to the situations where it can happen: under a negation, inside what a character believes or imagines, as the vehicle of a figurative comparison, or behind a viewpoint that hides the target. Prior work has studied only negation in depth, and current evaluation cannot see the failure. We build OverReal to measure it: OverReal-Gen, 698 prompts covering the four situations, on which we measure each generator's over-realization rate; and OverReal-Det, 2,841 generated images human-labeled for whether and how the target is over-realized, on which we verify the automatic judge that produces this rate. Our evaluation of eight text-to-image systems reveals that all of them over-realize, and that conventional metrics such as CLIPScore reward the failure rather than penalize it. Prompt expansion only partly mitigates the failure. Inside the generator, the diffusion backbone attends to a mentioned target almost as much as to a requested one. The cue words that should hold the target back receive less attention than ordinary words.

Primary area:
datasets and benchmarks
Keywords:
text-to-image generation, over-realization, prompt faithfulness, benchmark, figurative language, negation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KXAWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces over-realization, a failure mode in text-to-image generation where a concept mentioned in the prompt appears in the generated image in a way that was not requested. The authors categorize this phenomenon into four families (existence-canceling, mental-state, figurative, and perspectival) and distinguish between disruptive and silent over-realization.

They introduce OverReal-Gen, containing 698 prompt-target pairs, and OverReal-Det, containing 2,841 human-annotated images. Experiments across eight text-to-image models show that over-realization is common, while conventional alignment and preference metrics often fail to penalize it. The paper also investigates prompt expansion and attention patterns to better understand this behavior.

Critical strengths
  1. Interesting problem formulation. The paper highlights a limitation of existing text-to-image evaluation: not every concept mentioned in a prompt needs to be visually realized. Extending this observation beyond negation to mental-state, figurative, and perspectival expressions provides a useful perspective on prompt faithfulness.

  2. Comprehensive empirical analysis. The evaluation covers eight generators, multiple prompt families, and several conventional metrics. I particularly appreciate the finding that existing alignment metrics can favor over-realized images, as well as the analyses of prompt expansion and attention patterns.

Critical weaknesses
  1. The definition of silent over-realization is not sufficiently justified. I am not convinced that naturally incorporating an unrequested object should necessarily be considered a generation failure. For example, in Figure 2, the prompt describes a coach as "fierce as a tiger," and the generated image includes a tiger on the team's emblem. While the prompt does not explicitly request a tiger emblem, it does not prohibit one either. Text-to-image models necessarily fill in many unspecified visual details, and it is unclear why this particular detail should be considered incorrect. This is especially important because silent over-realization accounts for a substantial portion of the reported failures. The paper would benefit from stronger evidence that such images are actually less faithful to the prompt.

  2. Potential selection bias in benchmark construction. For three of the four prompt families, the authors retain prompts only when Gemini produces at least one over-realized image among three generations. This creates a challenge set enriched for failures of a particular model family, potentially affecting comparisons across generators. Although the in-the-wild evaluation helps address this concern, the benchmark's construction limits how broadly its reported failure rates can be interpreted.

Questions for authors
  1. What makes silent over-realization a genuine failure rather than an acceptable visual detail? For example, if the prompt says a coach is "as fierce as a tiger," why should a tiger emblem be considered incorrect when the prompt does not explicitly prohibit it? Have the authors considered a human study comparing the perceived prompt faithfulness of images with and without such details?

  2. Would the conclusions change if the benchmark focused on explicit constraints? For example, adding "no tiger should appear in the image" to the coach prompt would make the violation unambiguous. It would be interesting to know whether models still fail under such conditions, particularly for the figurative and mental-state families.

  3. How sensitive are the main results to the Gemini-based prompt selection process? In particular, how do the relative model rankings and the proportions of disruptive versus silent over-realization compare between OverReal-Gen and the in-the-wild evaluation?

4 comments

We have uploaded a revised PDF that includes the changes promised in our response. For your convenience, here is where each one is:

  • Why silent over-realization counts (Q1/W1): §3.2 now states that disruptive over-realization is an error and silent over-realization a warning, and that the two shares are reported separately so that the disruptive share can be read alone.
  • Explicit constraints (Q2): the experiment is now Appendix D.11, with the per-generator results in Table 23.
  • Sensitivity to the prompt selection (Q3/W2): Appendix D.6 now compares the two sets in more detail: the generator rankings (Spearman correlation per family), a separate paragraph on Nano Banana and the screening, and the disruptive share of over-realization on the two sets (Table 14) with the two causes of the figurative gap. Appendix B.1 now also states that the annotators were asked not to reuse one template.

Thank you again for the careful review. We are happy to discuss further.

AuthorsOpenReview ↗

Revised PDF updated

Dear Reviewer KXAW,

We have updated the revised PDF, and the changed text is now marked in blue. One update since our last note relates to your Q1/W1: §3.2 now states that disruptive over-realization is the error and that silent over-realization is a neutral diagnostic, and Figure 2 marks the two levels differently. As the discussion period ends on November 18, we would be grateful if you could let us know whether our response and the revision address your concerns. We are happy to clarify anything further.

Thank you again for your time.

Thank you for the detail experiments. I appreciate the authors' efforts to address my concerns. I raised my rating to 3.

I have one remaining question regarding the benchmark construction. The disruptive share of over-realization in the figurative family differs substantially between OverReal-Gen (35%) and the in-the-wild set (8%), and the model rankings for figurative and perspectival prompts show little correlation across the two datasets. While the differences in prompt composition provide a plausible explanation, could the authors clarify whether the main conclusions, particularly the comparisons across generators, remain consistent when controlling for figurative prompt subtypes (e.g., appearance-based versus quality-based comparisons)?

Thank you very much for the thoughtful comments throughout the discussion and for raising your rating. We truly appreciate it.

We classified every figurative prompt of OverReal-Gen by hand as appearance-based (32%) or quality-based (68%). The numbers differ from those in our earlier response, which came from a keyword rule. The table gives the disruptive and silent over-realization rates of each generator. The full table and the analysis are now in Appendix D.6.

GeneratorAppearance-based: disruptiveAppearance-based: silentQuality-based: disruptiveQuality-based: silent
GPT-Image13%28%28%30%
Nano Banana2%20%24%31%
Ideogram11%31%5%22%
FLUX.1-dev5%19%11%37%
Qwen-Image23%31%32%35%
OmniGen222%31%17%25%
SD3.5-Medium6%27%11%33%
SD3.5-Large16%26%22%47%

The main conclusions hold within each subtype. Every generator over-realizes, and silent over-realization exceeds the disruptive form. The generators differ in which subtype they handle better: six of the eight are disruptive more often on the quality-based prompts, most clearly Nano Banana (24% against 2%), while Ideogram and OmniGen2 are disruptive more often on the appearance-based ones.

On OverReal-Gen, the appearance-based prompts are disruptive less often than the quality-based ones (12% against 18% over all generators), in line with our earlier response. In the wild, the appearance-based prompts are disruptive even less often (2%). This is likely because the appearance-based prompts of OverReal-Gen also passed the selection with Gemini, which keeps only the prompts prone to over-realization. The wild set has only 9 quality-based prompts, too few for a meaningful analysis.

Reviewer_HxxWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Current T2I models tend to render every thing a prompt mentions, even when the prompt does not mean to call for it. The authors propose and define the task of OverReal, consisting of a challenge set of curated prompts for measuring over-realization rates, and a human-labeled image set for validating an LLM annotator. Evaluation results on eight T2I systems show conventional alignment and preference metrics still reward what the authors deem as over-realization failures. The author also find that LLM-assisted prompt rewriting helps mainly when the rewriter is able to detect and drop the uncalled for items in the prompts.

Critical strengths

I like the direction of this paper. Something that is mentioned in the prompt is not always supposed to appear in the generation.

  • A true blind spot in T2I faithfulness evaluation. The authors show that common alignment and preference metrics indeed favor the over-realized images.
  • The four over-literation families extend beyond the existing works on negation. I like the framework in Section 3.1, where esp. mental states, figurative comparisons, and hidden viewpoints are called out and treated individually. The form of the failures also differs by family in a consistent way.
Critical weaknesses

Still, I find that any literal appearance of the uncalled for target object tends to be treated as a total failure according to the authors’ standards. Following this observation, I decide to frame my issues with this paper on top of the task’s definition and the evaluation standards.

Weakness #1 - The Over-Realization Failure Is Too Excessive.

I find this work’s over-literalization criterion way too excessive, as are shown in the two examples in Figure 1. To my understandings, both failure showcases in Figure 1 should be at least ‘boardline acceptable’.

  • In Figure 1 (left), showing the contents of the letter facing the camera is a common convention for showing the written contents to the audience, such as how characters read newspaper headlines in early-day classic cartoons like in Tom & Jerry films.
  • In Figure 1 (right), the football coach character has already been demonstrating excessive anger, which is satisfactory enough. Literalizating the tiger team emblem does not necessarily destroy the whole rendition.

Based on the observations above, I find the authors’ over-literalization definition is way too strict that eventually misses the mark. A satisfying generation should be judged by if it correctly renders the implicated meaning behind the implicit prompt. I am afraid that the overall baseline results, likewise, have been deflated for the same reasons.

Weakness #2 - The Success Criterion Is Too Loose.

Beyond weakness #1, I also find the two seccess modes - integrated realization and withheld realization - too loosely defined.

  • Integrated realization is ill-defined. Although it allows the target to appear, the human annotators do not share a concrete standard as the authors point out.
  • Withheld realization only requires the target to be absent, so a target-less image that also drops the prompt’s implicated meaning may still count as correct.

Especially for withheld realization, per the authors’ standards, a calm-and-stern-looking coach might still count for “as fierce as a tiger”, while the paper’s own withheld example in Table 2 of a woman “imagining her deceased friend” shows few signs of her actually imagining anything.

Questions for authors

Suggestions

  • On W1, I would rather redefine the definition of over-realization in terms of whether the target’s appearance harms the rendition prompt’s intended meaning.
  • On W2, I would instead annotate each prompt with multiple must-have contents for a correct image, and count any image that miss them as an error.

Comments

  • Please correct me if I’m wrong, but I believe Chekhov’s gun is more about ‘foreshadowing devices’ in theater, whereas the main theme of this work is better aligned with unearthing ‘the true intentions behind the literal prompt’. I feel there is a subtle difference there.
2 comments

We have uploaded a revised PDF that includes the changes promised in our response. For your convenience, here is where each one is:

  • Opening sentence (your comment on Chekhov's gun): the introduction now says that we borrow the principle in the reverse direction.
  • Two-level reading of over-realization (W1): §3.2 now states that disruptive over-realization is an error and silent over-realization a warning, and that the disruptive share can be read alone. §6.1 now also reports the disruptive-only rates: every generator still over-realizes at least 29% of the mental-state and 54% of the perspectival images, while the figurative rates fall to 7% to 28%.
  • Withheld realization (W2): §3.2 now states that an image that is severely corrupted or contradicts the prompt is labeled other and excluded, rather than counted as withheld.
  • Integrated realization (W2): Appendix C.3 now reports the agreement between the two human annotators on whether an image is integrated (κ = 0.52).

Thank you again for the careful review. We are happy to discuss further.

AuthorsOpenReview ↗

Revised PDF updated

Dear Reviewer HxxW,

Following our discussion, we have uploaded a revised PDF with the changes in blue (§3.2, Figure 2, §6.1 and the Limitations). As the discussion period ends on November 18, we would be grateful if you could let us know whether these changes are appropriate and resolve your concerns. We are happy to clarify anything further.

Thank you again for your time.

Reviewer_3P6sOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies over-realization, a text-to-image failure in which a target mentioned in the prompt is rendered despite the prompt not requesting its presence. It organizes the phenomenon across existence-canceling, mental-state, figurative, and perspectival prompts and distinguishes disruptive and silent failures from integrated and withheld realization. The authors introduce curated prompts and human-annotated generated images, evaluate a broad range of proprietary and open generators, and additionally study conventional metrics, prompt expansion, text-encoder representations, diffusion-backbone attention, and prompts collected from real usage.

Critical strengths

The paper establishes a consequential and previously underexamined failure mode through complementary evidence rather than a curated failure gallery alone. Controls for unprompted target appearance, evaluation on real-user prompts, and analysis of repeated generation all support the conclusion that over-realization is systematic. The distinction between disruptive and silent outcomes is also practically useful because it reveals failures that ordinary evaluation can conceal.

The matched-pair metric analysis is especially compelling. It shows that commonly used similarity and preference metrics frequently favor over-realized images over correct images from the same prompt and generator, whereas question-answering and language-model evaluation behave differently. This demonstrates that existing evaluation practices can actively reward the failure, giving the work significance beyond the proposed benchmark.

Critical weaknesses

Many large-scale conclusions depend on an automatic judge whose development and test partitions are not clearly disjoint by prompt and target. Agreement is weakest for perspectival examples, which also underlie some of the strongest claimed effects, and the real-user evaluation is entirely automatically labeled. Consequently, the precision of the headline failure rates and some generator comparisons remains uncertain.

A substantial portion of the human annotations is marked ambiguous, including direct disagreement over whether an image is over-realized or correct. Several prompt families are also challenge-selected using failures from a particular generator. Although the authors disclose this selection and partially address it through real-user prompts, the curated rates should not be interpreted as generator-wide failure probabilities, and some outcome distinctions remain sensitive to subjective interpretation.

Questions for authors

Are the automatic judge’s development and test images strictly disjoint by prompt and target, and can agreement be recomputed using grouped splits if necessary?

Can you human-validate a stratified sample of the real-user predictions, particularly the perspectival examples underlying the strongest effects?

How stable are the headline rates and generator rankings when ambiguous images are included under alternative reasonable label assignments?

2 comments

We have uploaded a revised PDF that includes the changes promised in our response. For your convenience, here is where each one is:

  • Development and test splits (Q1): Appendix C.3 now states that the splits were drawn over images and reports the agreement on the 148 test images whose prompt never appears in the development split (κ = 0.54 over the four labels, 0.66 for over-realized against correct).
  • Human validation of the in-the-wild predictions (Q2): Appendix D.6 now reports the human check of 202 wild images, with Cohen's κ per family and the over-realization rates after reweighting by the human labels (Table 15). §6.1 points to it.
  • Stability under the ambiguous images (Q3): Appendix B.2 now reports the rates and rankings on OverReal-Det with the ambiguous images included under each annotator's labels.

Thank you again for the careful review. We are happy to discuss further.

AuthorsOpenReview ↗

Revised PDF updated

Dear Reviewer 3P6s,

We have updated the revised PDF, and the changed text is now marked in blue. As the discussion period ends on November 18, we would be grateful if you could let us know whether our response and the revision address your questions. We are happy to clarify anything further.

Thank you again for your time.

Comments on the paper

3 comments

I appreciate the follow-up details. That being said, I only have two smaller requests for the final revision.

On W1, now that the silent over-literalization becomes a neutral diagnostic, the abstract and Figure 3 should now lead with the disruptive rate (the true error rate that is).

On W2, I suggest write in explicit terms in Section 6.2 and alongside Table 1 that withheld images are not checked for completeness, since the prompt-expansion gains and the metric comparison both rely on that label.

With all have been much clarified, I'm happy to raise my rating accordingly.

Thank you very much for the thoughtful discussion throughout the rebuttal and for raising your rating. We have made both changes in the revised PDF (in blue).

W1. The abstract now leads with the disruptive rate (25% to 53% of the images). Figure 3 now prints the disruptive rate in bold on each bar, and the Introduction and §6.1 give the disruptive rate separately.

W2. §6.2 and the paragraph alongside Table 1 now state that withheld images are not checked for completeness.

Your comments on the silent label and on completeness have made the paper clearer and more precise. We are very grateful for the time and care you put into this discussion.

Dear reviewers,

The author-reviewer discussion period is ending soon. If you have not done so yet, please read the authors’ rebuttal and reply to it. Tell the authors whether it addresses your concerns, and update your score if your assessment has changed. With the high number of submissions this year, your input matters more than ever.

Thank you for your time and help.

Best regards,


Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.