- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces over-realization, a failure mode in text-to-image generation where a concept mentioned in the prompt appears in the generated image in a way that was not requested. The authors categorize this phenomenon into four families (existence-canceling, mental-state, figurative, and perspectival) and distinguish between disruptive and silent over-realization.
They introduce OverReal-Gen, containing 698 prompt-target pairs, and OverReal-Det, containing 2,841 human-annotated images. Experiments across eight text-to-image models show that over-realization is common, while conventional alignment and preference metrics often fail to penalize it. The paper also investigates prompt expansion and attention patterns to better understand this behavior.
Critical strengths
-
Interesting problem formulation. The paper highlights a limitation of existing text-to-image evaluation: not every concept mentioned in a prompt needs to be visually realized. Extending this observation beyond negation to mental-state, figurative, and perspectival expressions provides a useful perspective on prompt faithfulness.
-
Comprehensive empirical analysis. The evaluation covers eight generators, multiple prompt families, and several conventional metrics. I particularly appreciate the finding that existing alignment metrics can favor over-realized images, as well as the analyses of prompt expansion and attention patterns.
Critical weaknesses
-
The definition of silent over-realization is not sufficiently justified. I am not convinced that naturally incorporating an unrequested object should necessarily be considered a generation failure. For example, in Figure 2, the prompt describes a coach as "fierce as a tiger," and the generated image includes a tiger on the team's emblem. While the prompt does not explicitly request a tiger emblem, it does not prohibit one either. Text-to-image models necessarily fill in many unspecified visual details, and it is unclear why this particular detail should be considered incorrect. This is especially important because silent over-realization accounts for a substantial portion of the reported failures. The paper would benefit from stronger evidence that such images are actually less faithful to the prompt.
-
Potential selection bias in benchmark construction. For three of the four prompt families, the authors retain prompts only when Gemini produces at least one over-realized image among three generations. This creates a challenge set enriched for failures of a particular model family, potentially affecting comparisons across generators. Although the in-the-wild evaluation helps address this concern, the benchmark's construction limits how broadly its reported failure rates can be interpreted.
Questions for authors
-
What makes silent over-realization a genuine failure rather than an acceptable visual detail? For example, if the prompt says a coach is "as fierce as a tiger," why should a tiger emblem be considered incorrect when the prompt does not explicitly prohibit it? Have the authors considered a human study comparing the perceived prompt faithfulness of images with and without such details?
-
Would the conclusions change if the benchmark focused on explicit constraints? For example, adding "no tiger should appear in the image" to the coach prompt would make the violation unambiguous. It would be interesting to know whether models still fail under such conditions, particularly for the figurative and mental-state families.
-
How sensitive are the main results to the Gemini-based prompt selection process? In particular, how do the relative model rankings and the proportions of disruptive versus silent over-realization compare between OverReal-Gen and the in-the-wild evaluation?
4 comments
Revised PDF updated
Dear Reviewer KXAW,
We have updated the revised PDF, and the changed text is now marked in blue. One update since our last note relates to your Q1/W1: §3.2 now states that disruptive over-realization is the error and that silent over-realization is a neutral diagnostic, and Figure 2 marks the two levels differently. As the discussion period ends on November 18, we would be grateful if you could let us know whether our response and the revision address your concerns. We are happy to clarify anything further.
Thank you again for your time.
Thank you for the detail experiments. I appreciate the authors' efforts to address my concerns. I raised my rating to 3.
I have one remaining question regarding the benchmark construction. The disruptive share of over-realization in the figurative family differs substantially between OverReal-Gen (35%) and the in-the-wild set (8%), and the model rankings for figurative and perspectival prompts show little correlation across the two datasets. While the differences in prompt composition provide a plausible explanation, could the authors clarify whether the main conclusions, particularly the comparisons across generators, remain consistent when controlling for figurative prompt subtypes (e.g., appearance-based versus quality-based comparisons)?
Thank you very much for the thoughtful comments throughout the discussion and for raising your rating. We truly appreciate it.
We classified every figurative prompt of OverReal-Gen by hand as appearance-based (32%) or quality-based (68%). The numbers differ from those in our earlier response, which came from a keyword rule. The table gives the disruptive and silent over-realization rates of each generator. The full table and the analysis are now in Appendix D.6.
| Generator | Appearance-based: disruptive | Appearance-based: silent | Quality-based: disruptive | Quality-based: silent |
|---|---|---|---|---|
| GPT-Image | 13% | 28% | 28% | 30% |
| Nano Banana | 2% | 20% | 24% | 31% |
| Ideogram | 11% | 31% | 5% | 22% |
| FLUX.1-dev | 5% | 19% | 11% | 37% |
| Qwen-Image | 23% | 31% | 32% | 35% |
| OmniGen2 | 22% | 31% | 17% | 25% |
| SD3.5-Medium | 6% | 27% | 11% | 33% |
| SD3.5-Large | 16% | 26% | 22% | 47% |
The main conclusions hold within each subtype. Every generator over-realizes, and silent over-realization exceeds the disruptive form. The generators differ in which subtype they handle better: six of the eight are disruptive more often on the quality-based prompts, most clearly Nano Banana (24% against 2%), while Ideogram and OmniGen2 are disruptive more often on the appearance-based ones.
On OverReal-Gen, the appearance-based prompts are disruptive less often than the quality-based ones (12% against 18% over all generators), in line with our earlier response. In the wild, the appearance-based prompts are disruptive even less often (2%). This is likely because the appearance-based prompts of OverReal-Gen also passed the selection with Gemini, which keeps only the prompts prone to over-realization. The wild set has only 9 quality-based prompts, too few for a meaningful analysis.
We have uploaded a revised PDF that includes the changes promised in our response. For your convenience, here is where each one is:
Thank you again for the careful review. We are happy to discuss further.