- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates Figurative Vehicle Intrusion (FVI), a failure mode in text-to-image (TTI) generation where models incorrectly render figurative expressions as literal visual objects. The authors argue that conventional image-text faithfulness metrics primarily reward the presence of prompt-mentioned concepts, potentially overlooking cases where figurative language is assigned an incorrect visual role. To study this problem, they introduce VISTA, a multilingual benchmark containing 1,930 figurative prompts across Chinese, English, Japanese, and Arabic, annotated by Figurative Form and Mapping Mechanism. They further propose V-Score, a question-answering-based metric that evaluates tenor preservation, vehicle avoidance, scene coherence, and figurative meaning. Experiments on five TTI models demonstrate that vehicle intrusion remains common, while conventional presence-oriented evaluation can fail to identify these errors. Finally, the authors introduce VISTA-Guard, a lightweight LLM-based prompting strategy that explicitly distinguishes tenor and vehicle roles before generation. The method improves V-Score by 5.2–9.9 points across the evaluated generators without modifying their underlying models.
Critical strengths
-
The paper identifies a meaningful and relatively underexplored limitation of existing TTI evaluation. The distinction between literal object presence and semantic faithfulness is important, particularly for applications involving narrative prose, metaphors, and figurative descriptions. The examples in Figures 1 and 2 effectively illustrate why rendering every mentioned concept does not necessarily produce a faithful image. The problem formulation is intuitive and provides a useful perspective beyond conventional object-presence and attribute-binding evaluation.
-
The multilingual benchmark and role-aware evaluation framework are valuable contributions. VISTA covers four languages and organizes figurative expressions using complementary linguistic and semantic taxonomies. V-Score evaluates not only vehicle intrusion but also tenor preservation, scene coherence, and figurative meaning, which helps discourage trivial solutions that simply remove figurative content. The experiments demonstrate that the identified failure occurs across multiple TTI generators. The human evaluation also provides encouraging preliminary evidence for V-Score, with a reported Spearman correlation of 0.939 between MLLM-based scores and mean human judgments.
Critical weaknesses
-
The reliability and generalizability of V-Score are not sufficiently established. Although the authors construct 1,930 benchmark prompts, the main experiments evaluate only 193 samples, while the comparison against TIFA-like evaluation uses just 40 prompts and the human validation involves only 30 images. These subsets are relatively small for supporting broad conclusions about multilingual figurative understanding and metric reliability. More importantly, V-Score relies heavily on an MLLM evaluator answering questions generated specifically around the proposed tenor–vehicle framework. The high correlation with human judgments is encouraging, but the human participants answer the same predefined questions using the same scoring criteria. This primarily validates agreement within the proposed evaluation framework rather than demonstrating that V-Score captures broader human perceptions of image faithfulness. A larger independent human preference study, ideally involving direct comparisons of generated images without exposing the predefined evaluation questions, would provide stronger evidence.
-
The comparison with existing TTI evaluation methods is relatively limited. The paper argues that conventional faithfulness metrics can reward figurative vehicle intrusion, but its primary quantitative comparison uses a custom TIFA-like protocol on only 40 prompts. This is not equivalent to evaluating the full official TIFA implementation or other recent evaluation frameworks such as DSG and Gecko. Consequently, the observed disagreement between TIFA-like scores and V-Score demonstrates that the two particular evaluation protocols measure different properties, but does not fully establish the limitations of existing TTI evaluation methods in general. More comprehensive comparisons with established metrics and human judgments are necessary to demonstrate the incremental value of V-Score. Furthermore, the authors should clarify how the proposed metric differs from existing figurative-language and visual-metaphor evaluation approaches beyond explicitly penalizing literal vehicle realization.
-
The effectiveness of VISTA-Guard is not sufficiently isolated from straightforward role-aware prompt rewriting. The proposed mitigation identifies the tenor and vehicle, instructs the generator to preserve the tenor, and explicitly discourages literal realization of the vehicle. While the resulting improvements are substantial, this intervention is closely aligned with the criteria directly rewarded by V-Score, especially the vehicle-avoidance question. The current baselines include ordinary prompting and generic anti-hallucination instructions, but do not include a simple, explicitly role-aware prompt rewriting baseline without the proposed agent or skill framework. Therefore, it is difficult to determine whether the improvements arise from the proposed skill-based mechanism or merely from providing clearer instructions about figurative interpretation. An ablation comparing direct tenor–vehicle prompt rewriting, automatic role extraction, and the complete VISTA-Guard pipeline would better establish the necessity of the proposed design.
-
The treatment of figurative meaning as a fixed visual role assignment may oversimplify legitimate artistic interpretations. The paper generally assumes that the vehicle should not appear as an independent visual entity, but figurative image generation can intentionally combine tenor and vehicle through surrealism, symbolic imagery, or visual metaphors. In such cases, literalizing part of the vehicle may be a valid artistic interpretation rather than an error. This distinction is especially important for metaphors and idioms, where the intended visual realization can be subjective and culturally dependent. Although the benchmark excludes cases requiring literal vehicle presence, the paper does not sufficiently quantify annotation ambiguity or agreement on these distinctions. Additional analysis of ambiguous cases, inter-annotator agreement on tenor–vehicle roles, and sensitivity to alternative valid visual interpretations would strengthen the conceptual validity of the benchmark.
Questions for authors
-
Can the authors expand the human evaluation beyond the current 30 images and include direct human judgments of overall semantic faithfulness without providing the predefined V-Score questions? This would help establish whether V-Score generalizes beyond its own evaluation criteria.
-
Can the authors compare V-Score against additional established TTI metrics, such as official TIFA, DSG, and Gecko, on a larger subset of VISTA? In particular, how does V-Score correlate with human judgments compared with these alternatives?
-
How does VISTA-Guard compare against a simple LLM-based prompt rewriting baseline that explicitly identifies the tenor and vehicle and instructs the generator to avoid literal vehicle rendering? Can the authors isolate the contributions of role identification, skill invocation, and guarded-context generation?
-
How were ambiguous figurative interpretations handled during benchmark construction? Did multiple annotators independently verify tenor–vehicle assignments, and what was their agreement? How would V-Score handle visually creative interpretations in which a vehicle is intentionally represented through symbolic or surreal imagery?
Thank you for the thorough rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns.
The expanded comparisons with TIFA, DSG, and Gecko, along with the independent human preference study, have substantially addressed my concerns about the validity of V-Score and its distinction from existing metrics. The additional annotation details and discussion of ambiguous figurative interpretations also help clarify the intended scope of the benchmark.
I still see two limitations: the human preference agreement remains relatively modest (59.63%), and VISTA-Guard performs nearly identically to direct role-aware prompting on VISTA (86.62 vs. 86.44). I encourage the authors to acknowledge these limitations and avoid overstating the generality of V-Score or the methodological novelty of VISTA-Guard. Reporting the pre-adjudication inter-annotator agreement for tenor–vehicle annotations would also further strengthen the benchmark's reliability.
Overall, I find that the rebuttal has meaningfully addressed my main concerns, and I appreciate the authors' careful responses. I will raise my rating from 2 to 3.