acceptodds
Under review as a conference paper at ICLR 2027

Presence Is Not Faithfulness: Figurative Vehicle Intrusion in Text-to-Image Generation

Abstract

Text-to-image (TTI) models increasingly generate high-quality images from natural-language prompts, yet figurative language exposes a failure: a vehicle that should guide the depiction of a tenor may instead be rendered as a visible object. We call this failure **Figurative Vehicle Intrusion**: the intruding content is textually licensed, but it is assigned the wrong visual role, showing that visual presence is not always faithfulness and that presence-oriented evaluation can miss such errors. To study it systematically, we introduce **V**ehicle **I**ntrusion and **S**emantic **T**enor **A**ssessment (**VISTA**), a multilingual benchmark of figurative prompts organized by Figurative Form and Mapping Mechanism. We further propose **V-Score**, a diagnostic question-answering metric that evaluates role-aware figurative faithfulness in generated images. Evaluations on recent high-performing TTI models show that vehicle intrusion persists across languages and figurative categories. As a lightweight mitigation, we introduce **VISTA-Guard**, which partially reduces vehicle intrusion and suggests a practical path toward more figuratively faithful TTI generation. All resources will be released publicly.

Primary area:
datasets and benchmarks
Keywords:
Text-to-Image Generation, Benchmark, Figurative Vehicle Intrusion, Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RegWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates Figurative Vehicle Intrusion (FVI), a failure mode in text-to-image (TTI) generation where models incorrectly render figurative expressions as literal visual objects. The authors argue that conventional image-text faithfulness metrics primarily reward the presence of prompt-mentioned concepts, potentially overlooking cases where figurative language is assigned an incorrect visual role. To study this problem, they introduce VISTA, a multilingual benchmark containing 1,930 figurative prompts across Chinese, English, Japanese, and Arabic, annotated by Figurative Form and Mapping Mechanism. They further propose V-Score, a question-answering-based metric that evaluates tenor preservation, vehicle avoidance, scene coherence, and figurative meaning. Experiments on five TTI models demonstrate that vehicle intrusion remains common, while conventional presence-oriented evaluation can fail to identify these errors. Finally, the authors introduce VISTA-Guard, a lightweight LLM-based prompting strategy that explicitly distinguishes tenor and vehicle roles before generation. The method improves V-Score by 5.2–9.9 points across the evaluated generators without modifying their underlying models.

Critical strengths
  • The paper identifies a meaningful and relatively underexplored limitation of existing TTI evaluation. The distinction between literal object presence and semantic faithfulness is important, particularly for applications involving narrative prose, metaphors, and figurative descriptions. The examples in Figures 1 and 2 effectively illustrate why rendering every mentioned concept does not necessarily produce a faithful image. The problem formulation is intuitive and provides a useful perspective beyond conventional object-presence and attribute-binding evaluation.

  • The multilingual benchmark and role-aware evaluation framework are valuable contributions. VISTA covers four languages and organizes figurative expressions using complementary linguistic and semantic taxonomies. V-Score evaluates not only vehicle intrusion but also tenor preservation, scene coherence, and figurative meaning, which helps discourage trivial solutions that simply remove figurative content. The experiments demonstrate that the identified failure occurs across multiple TTI generators. The human evaluation also provides encouraging preliminary evidence for V-Score, with a reported Spearman correlation of 0.939 between MLLM-based scores and mean human judgments.

Critical weaknesses
  • The reliability and generalizability of V-Score are not sufficiently established. Although the authors construct 1,930 benchmark prompts, the main experiments evaluate only 193 samples, while the comparison against TIFA-like evaluation uses just 40 prompts and the human validation involves only 30 images. These subsets are relatively small for supporting broad conclusions about multilingual figurative understanding and metric reliability. More importantly, V-Score relies heavily on an MLLM evaluator answering questions generated specifically around the proposed tenor–vehicle framework. The high correlation with human judgments is encouraging, but the human participants answer the same predefined questions using the same scoring criteria. This primarily validates agreement within the proposed evaluation framework rather than demonstrating that V-Score captures broader human perceptions of image faithfulness. A larger independent human preference study, ideally involving direct comparisons of generated images without exposing the predefined evaluation questions, would provide stronger evidence.

  • The comparison with existing TTI evaluation methods is relatively limited. The paper argues that conventional faithfulness metrics can reward figurative vehicle intrusion, but its primary quantitative comparison uses a custom TIFA-like protocol on only 40 prompts. This is not equivalent to evaluating the full official TIFA implementation or other recent evaluation frameworks such as DSG and Gecko. Consequently, the observed disagreement between TIFA-like scores and V-Score demonstrates that the two particular evaluation protocols measure different properties, but does not fully establish the limitations of existing TTI evaluation methods in general. More comprehensive comparisons with established metrics and human judgments are necessary to demonstrate the incremental value of V-Score. Furthermore, the authors should clarify how the proposed metric differs from existing figurative-language and visual-metaphor evaluation approaches beyond explicitly penalizing literal vehicle realization.

  • The effectiveness of VISTA-Guard is not sufficiently isolated from straightforward role-aware prompt rewriting. The proposed mitigation identifies the tenor and vehicle, instructs the generator to preserve the tenor, and explicitly discourages literal realization of the vehicle. While the resulting improvements are substantial, this intervention is closely aligned with the criteria directly rewarded by V-Score, especially the vehicle-avoidance question. The current baselines include ordinary prompting and generic anti-hallucination instructions, but do not include a simple, explicitly role-aware prompt rewriting baseline without the proposed agent or skill framework. Therefore, it is difficult to determine whether the improvements arise from the proposed skill-based mechanism or merely from providing clearer instructions about figurative interpretation. An ablation comparing direct tenor–vehicle prompt rewriting, automatic role extraction, and the complete VISTA-Guard pipeline would better establish the necessity of the proposed design.

  • The treatment of figurative meaning as a fixed visual role assignment may oversimplify legitimate artistic interpretations. The paper generally assumes that the vehicle should not appear as an independent visual entity, but figurative image generation can intentionally combine tenor and vehicle through surrealism, symbolic imagery, or visual metaphors. In such cases, literalizing part of the vehicle may be a valid artistic interpretation rather than an error. This distinction is especially important for metaphors and idioms, where the intended visual realization can be subjective and culturally dependent. Although the benchmark excludes cases requiring literal vehicle presence, the paper does not sufficiently quantify annotation ambiguity or agreement on these distinctions. Additional analysis of ambiguous cases, inter-annotator agreement on tenor–vehicle roles, and sensitivity to alternative valid visual interpretations would strengthen the conceptual validity of the benchmark.

Questions for authors
  1. Can the authors expand the human evaluation beyond the current 30 images and include direct human judgments of overall semantic faithfulness without providing the predefined V-Score questions? This would help establish whether V-Score generalizes beyond its own evaluation criteria.

  2. Can the authors compare V-Score against additional established TTI metrics, such as official TIFA, DSG, and Gecko, on a larger subset of VISTA? In particular, how does V-Score correlate with human judgments compared with these alternatives?

  3. How does VISTA-Guard compare against a simple LLM-based prompt rewriting baseline that explicitly identifies the tenor and vehicle and instructs the generator to avoid literal vehicle rendering? Can the authors isolate the contributions of role identification, skill invocation, and guarded-context generation?

  4. How were ambiguous figurative interpretations handled during benchmark construction? Did multiple annotators independently verify tenor–vehicle assignments, and what was their agreement? How would V-Score handle visually creative interpretations in which a vehicle is intentionally represented through symbolic or surreal imagery?

1 comment

Thank you for the thorough rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns.

The expanded comparisons with TIFA, DSG, and Gecko, along with the independent human preference study, have substantially addressed my concerns about the validity of V-Score and its distinction from existing metrics. The additional annotation details and discussion of ambiguous figurative interpretations also help clarify the intended scope of the benchmark.

I still see two limitations: the human preference agreement remains relatively modest (59.63%), and VISTA-Guard performs nearly identically to direct role-aware prompting on VISTA (86.62 vs. 86.44). I encourage the authors to acknowledge these limitations and avoid overstating the generality of V-Score or the methodological novelty of VISTA-Guard. Reporting the pre-adjudication inter-annotator agreement for tenor–vehicle annotations would also further strengthen the benchmark's reliability.

Overall, I find that the rebuttal has meaningfully addressed my main concerns, and I appreciate the authors' careful responses. I will raise my rating from 2 to 3.

Reviewer_JW7VOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies cases where an image generation model uses figurative language more literally and generates an extra object instead of using it to describe something else. It introduces VISTA, a test set in four languages, and V-Score to check whether images preserve the intended meaning. It also conducts tests on five image models, showing this error is common, while the VISTA-Guard prompting method reduces it.

Critical strengths

The paper points out an error that simple object checks can miss. A word can describe the subject without naming something that should appear as a separate object. When judging images made from stories or other prose, the paper shows that these errors are common. It then proposes VISTA as a test set covering four languages and several types of figurative language.

The paper's score checks more than whether an unwanted object is gone. It also checks that the subject remains, the scene makes sense, and the image keeps the figurative meaning. This is a holistic way of evaluating this error case, preventing a model from scoring well by ignoring the comparison.

Also, VISTA-Guard improves scores on all five models and beats general instructions to avoid adding objects.

The agreement with human ratings and a second scoring model provides some support for V-Score's validity.

Critical weaknesses

I lean toward acceptance because the paper identifies a useful problem and provides a way to test and reduce it. Mentioning something does not always mean it should be drawn as an object. I would focus the claims on that insight rather than present the results ranking of image models.

It is not always clear what counts as an error. Equation 1 focuses on unwanted objects. Yet Figure 2 also penalizes a dragon-shaped island, even though the prompt asks for an island shaped like a dragon. This is less clear than a picture with an extra dragon beside the island. The paper needs clear rules for extra objects, replacement of the subject, and valid use of a comparison's shape. The human study uses the same questions and scoring rules on 30 images. The agreement under those rules does not show that the chosen interpretation is the only reasonable one.

The tests are smaller than the paper's broad claims suggest. The main study uses 193 of the 1,930 prompts, with one image per prompt and model. Some language and category groups have very few examples. The score comparison uses just 40 prompts and a TIFA-like method, and not the original scoring tools. The different model rankings also do not show which score better matches human judgment. These tests do, however, show that the error occurs across models. But they say less about exact model rankings, differences between languages, or weaknesses in existing scoring methods.

Questions for authors

What rule separates a valid dragon-shaped island from a dragon that replaces the island? How often do independent raters disagree? Please report separate results for extra objects and replacement of the subject.

How were the 30 images for human review chosen? Which languages did they cover, and how well did the raters know those languages? Is agreement still high for the unwanted-object question alone and for unclear shape comparisons?

On the same images, does V-Score agree with independent human judgments better than the original TIFA or DSG tools? Do the main results hold when each model generates several images per prompt?

Reviewer_R1vYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines Figurative Vehicle Intrusion, a failure where a figurative vehicle that should only shape the depiction of a tenor is rendered as a visible object. It introduces VISTA, a multilingual benchmark of 1930 figurative prompts in Chinese, English, Japanese and Arabic, and V-Score, a four-question multiple-choice metric that checks tenor preservation, vehicle avoidance, scene coherence and figurative meaning. It evaluates five text-to-image models and compares V-Score with a TIFA-like protocol. It also proposes VISTA-Guard, a prompt-rewriting agent that reduces vehicle intrusion.

Critical strengths

The conceptual distinction between visual presence and visual role is clear and useful.

The benchmark covers four languages and two annotation views.

The human study supports V-Score as a proxy for human judgment.

The paper promises to release resources that can support future work.

The writing and pictures are good.

Critical weaknesses

The FVI boundary is subjective and not fully validated.

The metric and the mitigation are evaluated in a closed loop.

The TIFA comparison is not a faithful implementation of the original metric.

Questions for authors

How do annotators agree on the tenor and vehicle labels?

Can VISTA-Guard be evaluated by human raters rather than by the same V-Score?

How does V-Score compare with the actual TIFA and DSG implementations on the same images?

Are the four language subsets comparable in source and naturalness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.