acceptodds
Under review as a conference paper at ICLR 2027

Blending Concepts: Benchmarking Visual Metaphor Generation in Text-to-Image Models

Abstract

Text-to-image (T2I) models have achieved remarkable success at faithfully rendering specified objects and attributes, yet their ability to produce visual metaphors—images that convey abstract ideas by combining elements from two distinct domains—remains largely unexamined. To bridge this gap, we introduce VMetaphor-Bench, a fine-grained diagnostic benchmark for visual metaphor generation in T2I models. It comprises 1,500 visual metaphors curated from real-world creative imagery, organized into three levels and ten categories, with each sample paired with two prompts of differing specificity. For evaluation, we develop a hybrid framework within an MLLM-as-judge paradigm, combining a multiple-choice question (MCQ) based protocol of 9,594 questions across four levels of metaphorical fidelity with a dimension-based scoring protocol along three perceptual dimensions. Extensive evaluation of 12 representative T2I models reveals that even the strongest proprietary models struggle with compositional structuring and cross-domain mapping, key aspects of metaphorical expression, highlighting visual metaphor generation as an important frontier for future T2I research.

Primary area:
datasets and benchmarks
Keywords:
Visual Metaphor, Text-to-Image Models, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FhTqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

VMetaphor-Bench is a fine-grained diagnostic benchmark comprising. Contains 1,500 visual metaphors curated from real-world creative imagery, categorized across 3 levels and 10 thematic domains.

Conventional T2I benchmarks such as GenEval and T2I-CompBench primarily assess literal object, attribute, and spatial fidelity, overlooking cross-domain conceptual blending. Existing visual-metaphor benchmarks (eg MetaCLUE) largely focus on metaphor understanding. Generation evaluations are typically small-scale and rely on generic metrics (FID/ CLIP) or humans, and, lack fine-grained diagnostics of where and why models fail at visual metaphor generation, which VMetaphor-Bench aims to provide.

Process

Rigorous evaluation and dataset generation process

Dataset Generation

  1. Image collection: ~5000 candidate images from Pinterest: general and domain-specific visual-metaphor search terms.
  2. Initial filtering: human (2 separate) annotators remove duplicates, ambiguous metaphors, and low-quality images -> ~ 1600 retained.
  3. Structured annotation: an annotation model (gemini 3.1) generates source domain, target domain, structure type, meaning, and element-level correspondences.
  4. Prompt generation (stage #2 annotation): annotation model produces conceptual and descriptive prompts from image and annotation information.
  5. Manual verification: 3 researchers independetly examine and correct imagery, annotations, and prompts -> 1500 images.
  6. Question generation: MM-LLM (gemini 3.1) constructs MCQs from structured annotations and descriptive prompts.
  7. Question verification: Human annotators validate the resulting ~10k MCQs.

Benchmark task & Eval for MLMs

Check metaphorical fidelity, and expressive quality. Use dimension-based scoring (Efficacy, Logic, Harmony) + MCQ scoring.

  1. Image generation: 12 T2I models being tested generate images from benchmark prompts.
  2. Automatic assessment: another MLLM(Qwen3.5-27B) evaluates generated images -> third one (GPT-5.4) provides cross-judge comparisons.
  3. Human validation: 5 annotators independently assess sampled outputs ->examine model coverage and generation-seed sensitivity.
Critical strengths
  1. Useful dataset generation work, involving rigorous testing and eval frameworks. Vis. metaphor generation is a useful area as well
  2. Strong overall benchmark with 1.5k images & ~10k MCQs, 10 themes.
  3. Good test model coverage (12) on the new benchmark
  4. Good human validation of the benchmark results
  5. Clear and detailed results for all dimensions (efficacy, harmony, logic) and the MCQs.
Critical weaknesses
  1. It will be useful to perform larger human validation study to check the scores (to see if the intended metaphor gets communicated). To quote the paper “models can depict the right elements but struggle to compose them into coherent metaphorical structures”.. this might imply that the generated images might really make little sense to real humans, implying the other scores might not really matter much at all
  2. The MLLM judges can have shared biases (eg: familiarity with common ad metaphors, similar pretraining datasets can infuse similar biases)
  3. Similarly in the same section, more details are required on the “Human Alignment Study” - inter-rater agreement statistics will be useful
  4. Dataset biases/imbalances: source was just one social media site (pinterest). Similarly, “replacement is far more challenging than fusion or juxtaposition”, but the dataset contains mostly fusion images (per A.2 appendix)
  5. Removing “Ambiguous samples” might also have removed the best/most challenging datapoints
  6. The “conceptual prompt” appears to cover the exact metaphor needed to be generated. So, the benchmark is not exactly testing for creative space exploration of MLLMs (as a future follow-up work, I’d love to see another level of a “core message prompt” - which would describe a vague idea to deliver through the image)
  7. Table 7(A.3) shows that the default resolution generated across the models are very different. This makes it harder to read the comparis
Questions for authors
  1. You decided to use Gemini 3.1 Pro for its “strong visual reasoning capabilities” for the benchmark dataset generation, while you decided to go with Qwen 27B, again for “strong visual reasoning capabilities” for automatic benchmark task judgement - any reason for this? Any comparisons b/w these and other MLMs on the different tasks?
  2. Re. the “overall dimension score averages the three dimensions” - is there a good reason to give equal weight to these? An image with weak metaphor might get good score just because of “harmony”
4 comments
AuthorsOpenReview ↗

Response to Reviewer FhTq

We greatly appreciate your thoughtful review and encouraging assessment of our benchmark and evaluation. Your suggestions have strengthened our analysis and presentation. Below, we provide our responses and hope to address your concerns.

W1: Could a larger human study verify whether the intended metaphors are actually communicated to viewers?

A1: Thank you for this suggestion. We conducted an annotation-blind human study using the same 180 images from six models as Appendix A.7. Three participants who had not seen the annotations viewed only the images, rated their understanding clarity, and wrote interpretations or indicated “cannot determine”.

The mean blind clarity rating was 3.08/5, and the core meaning recovery rate was 36.9%. Recovery ranged from 15.6% for Janus-Pro to 64.4% for GPT Image 1.5, showing substantial differences alongside considerable room for improvement. Across all images, mean blind clarity ratings correlated strongly with the previous reference-conditioned human Efficacy scores (Spearman’s ρ = 0.816), providing complementary evidence that these scores relate to viewers’ perceived clarity.

We also agree that correct elements and visual harmony alone do not establish successful communication. The other scores serve complementary diagnostic purposes: they help distinguish whether a failure concerns domain presence, structural composition, element mapping, conceptual coherence, or visual presentation. Reporting them separately reveals images that look appealing yet fail to convey the intended meaning, which is a finding of our benchmark.

W2: The MLLM judges can have shared biases

A2: Thank you for raising this concern. We agree that judges may share preferences for familiar metaphors. Beyond cross-judge consistency, our human-alignment studies show that both judges broadly agree with human assessments under the same evaluation protocol. While this agreement does not rule out all shared biases, it provides evidence that the scores reflect human judgments within the evaluated scope. We have explicitly acknowledged potential shared judge preferences in the Limitations section.

W3: More details are required on the Human Alignment Study - inter-rater agreement statistics will be useful

A3: Thank you for this helpful suggestion. We computed inter-rater agreement among five annotators in the original 200-image human-alignment study. Table R1 reports pairwise agreement for MCQs and ordinal Krippendorff’s α for the dimension scores. Agreement varies across evaluation levels, and the dimension scores show appreciable individual variability.

Table R1. Inter-rater agreement.

ComponentStatisticEstimate
DomainPairwise agreement (%)80.10
StructurePairwise agreement (%)69.10
MeaningPairwise agreement (%)62.15
MappingPairwise agreement (%)55.57
Metaphoric EfficacyOrdinal α0.424
Metaphor LogicOrdinal α0.441
Perceptual HarmonyOrdinal α0.333

Although individual ratings vary, the reliability of the five-rater mean is higher, with absolute-agreement ICC(A,5) values of 0.805 for Efficacy, 0.812 for Logic, and 0.740 for Harmony.

We also performed five-fold leave-one-annotator-out analysis. Each held-out annotator and both judges were compared against the same reference formed by the other four annotators, using voting for MCQs and averaging for dimension ratings.

Table R2. Agreement with the same four-annotator reference, averaged across folds.

EvaluatorOverall MCQ agreement (%) ↑Overall dimension MAE ↓
Held-out human76.280.849
Qwen3.5-27B81.230.684
GPT-5.481.500.802

Despite individual variability, both judges show closer overall agreement with the four-person reference than the average held-out annotator. This shows that the judges’ deviations from aggregated human judgments are smaller overall than those of individual annotators.

W4: Dataset biases/imbalances

A4: Thank you for highlighting these points. Pinterest provides access to diverse creative imagery, but collecting from a single platform can introduce selection biases. We acknowledge this limitation in Section 5 and do not claim that the benchmark represents all visual-metaphor distributions. Regarding structural imbalance, we did not impose structure-specific quotas; the distribution reflects the collected corpus. The predominance of fusion is also consistent with prior findings [1].

The replacement finding uses accuracy within each structure type: 19.3% for replacement, 68.7% for fusion, and 70.7% for juxtaposition (Appendix A.4). Equal weighting across structures yields ranking correlations of ρ = 0.979 for both MCQ and dimension scores relative to the original weighting (Appendix A.2). L2/L4 remain below L1/L3. These results support robustness to structure weighting.

[1] Margariti et al. A Path to Our Heart: Visual Metaphors and “White” Space in Advertising Aesthetic Pleasure. Int. J. Advertising, 2022.

AuthorsOpenReview ↗

Response to Reviewer FhTq (part 2)

W5: Removing ‘Ambiguous samples’ might also have removed the best/most challenging datapoints

A5: Thank you for pointing out this trade-off. Ambiguous imagery can indeed contain rich and challenging metaphors. Our filtering aimed to retain samples whose annotations could be reliably verified: when both reviewers had difficulty establishing the intended meaning or cross-domain correspondences, we could not confidently validate the reference annotation. Retaining such samples would also make evaluation difficult to interpret, since a low score could reflect an uncertain reference rather than a failure of the generated image. The criterion was therefore interpretability for reliable annotation and evaluation, rather than metaphor complexity.

W6: The ‘conceptual prompt’ appears to cover the exact metaphor needed to be generated

A6: Thank you for this thoughtful suggestion. We agree that our conceptual prompts specify the metaphor to be realized, so the benchmark evaluates models’ ability to visually express a specified metaphorical intent, rather than open-ended metaphor ideation. We adopted this design to provide a shared semantic target for consistent comparison and fine-grained diagnosis. Conceptual prompts explicitly state the intended meaning, whereas descriptive prompts describe its visual realization without explicitly stating that meaning.

A “core message prompt” would usefully extend this scope by allowing models to choose the concepts and their relationships to communicate a message. Evaluating this setting would require accommodating a broader range of metaphorical realizations beyond the current reference annotations. We have added this direction to Section 5, proposing more abstract or core-message prompts together with evaluation protocols that accommodate diverse interpretations and visual realizations.

W7: Table 7(A.3) shows that the default resolution generated across the models are very different

A7: Thank you for highlighting the differences in generation resolution. We used model-specific settings to evaluate each model in its standard usage configuration. Janus-Pro uses its native 384×384 generation setup, while Seedream 5.0 Lite’s API has a minimum pixel requirement that excludes 1024×1024 output. For Qwen-Image-2512, we adopted the 1328×1328 square-image setting provided in its official example.

Q1: Why use Gemini 3.1 Pro for dataset construction and Qwen 27B for evaluation? Any comparisons b/w these and other MLMs on the different tasks?

A8: Thank you for asking us to clarify the model-selection rationale. The two stages have different requirements. Dataset construction is a one-time process whose annotation quality directly affects all subsequent evaluations, so we prioritized annotation quality when selecting Gemini 3.1 Pro. In our early qualitative trials with other leading proprietary multimodal models, human inspection favored Gemini’s annotations. These trials informed our choice but were not a quantitative model comparison. The resulting annotations and prompts then underwent multiple stages of human verification and correction.

For automatic evaluation, we selected Qwen3.5-27B as a strong open-weight model to make repeated, large-scale benchmarking more affordable and accessible. This choice is supported by the comparisons in Section 4.5: Qwen and GPT-5.4 yield highly consistent model rankings, with Spearman’s ρ of 0.993 for MCQ scores and 0.951 for dimension scores. Both also show good alignment with human assessments under the same protocol. Thus, our selection combines quality-focused, human-verified dataset construction with a cost-conscious evaluator supported by cross-judge and human-alignment evidence.

Q2: Re. the ‘overall dimension score averages the three dimensions’ - is there a good reason to give equal weight to these?

A9: Thank you for raising this point. We use equal weighting to provide a transparent summary of three complementary aspects of visual metaphor quality: communicating the intended meaning, maintaining coherent cross-domain mappings, and achieving a coherent visual presentation. Without a validated basis for assigning different relative weights, we give each dimension equal weight on the shared 1–5 scale.

A high Harmony score can raise the average despite weaker metaphorical expression. We therefore report all three dimension scores separately alongside the overall score and the four-level MCQ results. The overall dimension score summarizes these complementary qualities. It is not a standalone criterion for successful metaphor communication. Indeed, our analysis explicitly highlights cases where visual harmony exceeds metaphorical efficacy, using this discrepancy to diagnose models’ limitations.

Acknowledging the responses. Thanks for details on annotation-blind human evaluation + inter-rater reliability

maintaining initial rating

AuthorsOpenReview ↗

Response to Reviewer FhTq

We appreciate your follow-up and are glad we could clarify these points. We sincerely appreciate your recognition of our contributions. Thank you again for the time and care you put into reviewing our paper.

Reviewer_W5cgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for visual metaphor synthesis comprising 1500 samples organized in a taxonomy of three levels and ten categories. They propose a multiple-choice evaluation protocol across four hierarchical levels of metaphorical fidelity: domain presence, structure type, meaning, and element mapping. The evaluation protocol is supplemented by an MLLM-as-a-judge scoring method for metaphoric efficacy, metaphor logic, and perceptual harmony. They evaluate 12 state-of-the-art image generation models, and reveal their limitations.

Critical strengths
  • The paper uses real visual metaphors found in the wild, and annotations are verified by trained individuals.
  • The proposed multiple-choice setup for evaluating visual metaphors is interesting and allows a fine-grained understanding of failures and successes.
  • The MLLM as a judge is evaluated in terms of human agreement which shows to be within a point in Likert scale.
  • A comprehensive suite of models is evaluated highlighting a substantial gap between proprietary and open-source models, as well as reveal that structure and mapping is the primary bottleneck.
Critical weaknesses
  • One concern I have is the notion of conceptual vs. descriptive prompts. From the examples provided in Figure 3, and the conceptual metaphors in Figure 4, conceptual prompts also seem descriptive to me. For example, in the metaphor in Figure 3, I think a conceptual prompt would be "the earth is a ticking bomb" - more abstract and open to interpretation. The lack of more abstract prompts makes it seem that the task evaluated is more about instruction following in "unrealistic conditions" rather than visual metaphor generation.
Questions for authors
  • What are your thoughts regarding the critical weakness mentioned above? What is the benchmark really evaluating?
  • Another concern regards the multiple-choice options. I wonder if the "plausible but incorrect distractors" and "cannot be determined" options were also human-validated. How was their "plausibility"/difficulty measured?
3 comments
AuthorsOpenReview ↗

Response to Reviewer W5cg

We sincerely thank Reviewer W5cg for the careful review and constructive suggestions. We are pleased that you recognized the value of our real-world imagery, fine-grained evaluation, and model coverage. We appreciate your thoughtful questions about the prompt design and multiple-choice options, which have helped us explain our benchmark more clearly. Below, we provide our point-by-point responses and hope to address your concerns.

W1/Q1: Conceptual prompts still seem descriptive, raising the concern that the benchmark evaluates instruction following rather than visual metaphor generation. Why not use more abstract prompts, and what is the benchmark really evaluating?

A1: Thank you for raising this concern. VMetaphor-Bench evaluates whether T2I models can express an intended metaphor by generating a visual realization from conceptual intent or following a concrete visual description.

Descriptive prompts provide concrete visual details and element-level correspondences, resembling conventional T2I scene descriptions, without explicitly stating the intended meaning. Conceptual prompts provide the source and target domains, structural cues, and intended meaning, without specifying those details or correspondences. Specifying what an image should communicate does not fully specify how to depict it in detail. Under conceptual prompts, models must determine more of the visual realization. Under descriptive prompts, they must faithfully realize the supplied description. Both conditions require the resulting image to convey the metaphor, beyond merely depicting the named objects.

We agree that “the earth is a ticking bomb” is more open-ended. While it suggests impending danger, the message could concern climate change, warfare, or other threats. Leaving that message unspecified allows outputs to address different communicative goals, complicating comparison against a common semantic target. We therefore state the intended meaning in conceptual prompts to specify the communicative goal while leaving the detailed visual realization open. Assessing open-ended metaphor ideation, where the message itself is not fixed, requires an extended evaluation protocol and is a direction for future work.

We have clarified the information provided by each prompt type and the evaluation purpose in Section 3.1, and the purpose of comparing the two prompt settings in Section 4.4. We have also added more abstract prompts and evaluation protocols accommodating diverse interpretations and visual realizations as future work in Section 5 (Limitations).

Q2: Another concern regards the multiple-choice options. I wonder if the ‘plausible but incorrect distractors’ and ‘cannot be determined’ options were also human-validated. How was their ‘plausibility’/difficulty measured?

A2: Thank you for highlighting the need to clarify our option-validation procedure. The distractors were human-reviewed using the same explicit criteria used for their generation: they should be visually or conceptually relevant, while remaining clearly distinct from the annotated answer. In particular, synonyms, hypernyms, hyponyms, and near-equivalent paraphrases were excluded to reduce ambiguity. Here, “plausibility” was assessed qualitatively against these criteria; we did not assign numerical plausibility scores or separately calibrate item difficulty. Our aim was to provide meaningful alternatives that support reliable assessment of whether a generated image expresses the intended metaphor.

The “cannot be determined” option is a standardized response added to allow the judge to indicate insufficient visual evidence. We have clarified these review criteria and the role of “cannot be determined” in Section 3.2 (MCQ-based Evaluation).

Thank you for the response. I will maintain my positive score since I think the dataset is useful for the community. However, I remain concerned about the scope of the benchmark and the extent to which it meaningfully evaluates visual metaphor generation ability, given the highly descriptive nature of the prompts.

AuthorsOpenReview ↗

Response to Reviewer W5cg

Thank you for maintaining your positive assessment and recognizing the dataset’s usefulness to the community! Our conceptual prompts seek to balance openness in visual realization with a shared metaphorical intent for consistent evaluation. Your feedback has helped us clarify this scope and identify more open-ended generation as a future direction. We appreciate your constructive suggestions.

Reviewer_s9bzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VMetaphor-Bench contains 1,500 curated visual metaphors with structured annotations and paired conceptual/descriptive prompts. It combines 9,594 multiple-choice questions on domains, structure, meaning, and element mappings with three perceptual dimensions. Tests on 12 generators find lower structure/mapping scores than domain/meaning scores. Appendix experiments examine human agreement and sensitivity to seeds. MetaCLUE and I Spy a Metaphor already studied metaphor generation, including human judgments; this paper adds a larger evaluation with metaphor-specific diagnostic questions (§§2–4).

Critical strengths
  • The four MCQ levels distinguish object recognition from errors in relationships and mappings. Paired prompt granularities test the effect of supplying visual details. Three people verify the images, annotations, and prompts, and the tables and examples make the benchmark easier to inspect (§3; Tables 2–3/8; Figs. 3–4).
  • Reliability checks extend beyond the main-text human study: a second judge, a balanced 180-image study covering six models and three structures with five evaluators, three-seed generation on 100 prompts across seven models, and structure-reweighted rankings (A.2, A.7–A.8; Tables 10–11). Together they support broad model comparisons.
Critical weaknesses
  • L4 evaluates reference mappings even for conceptual prompts that omit them (§3.1–3.2). Its 3,594 questions make up about 37.5% of the micro-averaged total. In Fig. 3, a globe-bomb receives high perceptual scores while failing the reference's grid-to-latitude/longitude correspondence. Some low L4 scores may therefore reflect alternative realizations rather than failed metaphor generation. The frequency is unknown, but the distinction matters for the claimed mapping bottleneck and the gains from descriptive prompts. Conceptual prompts already constrain structure, so this concern is specific to unspecified mapping details.
  • The efficacy protocol supplies information that an unaided viewer would lack. The dimension judge and human interface show reference annotations before assessing efficacy in image-alone terms (Figs. 11/14); MCQ options also supply semantic context. Human agreement under this shared protocol supports annotation-conditioned evaluation, but cannot establish that viewers recover the meaning without the annotation. L4 is asked separately from L1–L3, so the issue is not L4-to-L1 leakage. The paper needs to distinguish reference-conditioned fidelity from its stronger communication claim.
Questions for authors
  • Is L4 intended to measure reference fidelity or open conceptual validity? Could a small audit of existing failures separate invalid metaphors, valid alternative mappings, and judge errors?
  • Can the efficacy claim be stated in terms of reference-conditioned interpretation? If unaided interpretation is central, could a small annotation-blind check test it directly?
  • Could the related-work discussion clarify what VMetaphor-Bench adds to the already-cited MetaCLUE and I Spy a Metaphor in task definition, annotations, and evaluation? II-Bench, CII-Bench, and MetaphorStar offer additional context on image-implication and metaphor understanding; a brief comparison could distinguish these tasks from image generation. Forceville (1996) and Phillips and McQuarrie (2004) may also help situate the structural taxonomy.
3 comments
AuthorsOpenReview ↗

Response to Reviewer s9bz

We sincerely thank Reviewer s9bz for the thoughtful review and constructive feedback. We are encouraged by your recognition of our diagnostic framework, paired prompts, and reliability analyses. Your suggestions have helped us clarify the scope of our evaluation and strengthen its empirical support. Below, we provide our point-by-point responses and hope to address your concerns.

W1/Q1: Is L4 measuring reference fidelity or open conceptual validity, given that conceptual prompts omit specific mappings and L4 accounts for 37.5% of the MCQs? Could an audit distinguish invalid metaphors, valid alternative mappings, and judge errors?

A1: Thank you for raising this important point. L4 is intended to diagnose whether models establish meaningful cross-domain visual correspondences under the specified domains, structure, and intended meaning. We operationalize this assessment through annotated element mappings. These annotations do not exhaust every valid realization, particularly when conceptual prompts leave specific mapping details unspecified.

Following your suggestion, we conducted a manual author audit of 150 conceptual-prompt outputs with at least one L4 failure, covering all 12 generators. 122 outputs (81.3%) were judged to be invalid metaphor realizations. Among the 236 failed L4 questions audited, 11.0% involved valid alternative mappings, and 2.5% were potential judge errors. Thus, valid alternatives exist, but do not explain the majority of the audited failures. The audit supports the finding that constructing coherent metaphorical structures and visual mappings remains a major challenge, while distinguishing this finding from the stronger assumption that every reference mismatch constitutes an invalid metaphor.

The complementary evaluation levels also recognize successful aspects of valid alternatives:

Audited subsetImagesDomainStructureMeaningMappingMCQ Overall
All audited outputs15086.362.088.038.964.3
Outputs containing valid alternative mappings21100.090.590.550.077.5

Although L4 may not fully credit some valid alternative mappings, these outputs achieve high Domain, Structure, and Meaning scores. This illustrates how the complementary evaluation levels capture successful aspects of alternative realizations.

Regarding L4’s 37.5% share of the MCQs, we additionally recomputed the overall scores by assigning equal weight to the four evaluation levels on the full conceptual-prompt results. The ranking of all 12 models remains unchanged, supporting the robustness of the model ordering to this aggregation choice.

We have clarified the L4 definition in §3.2 to explicitly state that it evaluates whether the annotated source–target element correspondences are visually realized in the generated image.

W2/Q2: Can the efficacy claim be stated in terms of reference-conditioned interpretation? If unaided interpretation is central, could a small annotation-blind check test it directly?

A2: Thank you for raising this important point. As described in Section 3.2, our efficacy evaluator receives the structured reference annotation to establish the intended meaning as the evaluation target and assess how clearly the generated image expresses it. The original Efficacy score is thus a reference-conditioned assessment of metaphorical expression.

To directly examine unaided interpretation, we conducted an annotation-blind study using the same 180 generated images from six models as the human evaluation in Appendix A.7, with three participants who had not previously seen the reference annotations. Participants viewed only the images, rated how clearly they understood the metaphorical meaning, and provided a free-text interpretation or indicated that they could not determine the meaning. We assess core meaning recovery by comparing these interpretations with the intended meaning, allowing semantically equivalent wording. The mean blind clarity rating was 3.08/5, and the core meaning recovery rate was 36.9%.

Across all 180 images, image-level mean blind clarity ratings correlated strongly with the previous reference-conditioned human Efficacy scores (Spearman’s ρ = 0.816, 95% CI [0.757, 0.857]). This indicates that images rated as expressing the intended meaning more clearly also tend to be perceived as clearer by viewers without reference information.

Q3: More discussion of related work.

A3: Thank you for these helpful suggestions. We have revised the manuscript to clarify the relationship to prior work:

  • Section 2.3: We clarify how VMetaphor-Bench differs from MetaCLUE and I Spy a Metaphor in task definition, annotations, and evaluation.
  • Section 2.3: We discuss II-Bench, CII-Bench, and MetaphorStar to distinguish image-implication and metaphor understanding from image generation.
  • Section 3.1: We ground our structural taxonomy in Phillips and McQuarrie (2004), which builds partly on Forceville (1996).
Reviewer_s9bzOpenReview ↗

Response to the authors

Thank you for the clarifications and the revised related-work section.

The new L4 definition (p5 l.263–264) and the added related work, including the Table 1 correction, resolve Q3 and the definitional part of W1.

W1. The L4 audit (11.0% plausible alternative mappings and 2.5% judge errors among 236 failed items) and the unchanged ranking under equal weighting are useful, but they are not in the paper. Please add them to the appendix with the audit procedure and agreement, and qualify the "primary bottleneck" paragraph (p7) and the explanation of descriptive-prompt gains (p8 l.422–427), since descriptive prompts state the mappings that L4 checks.

W2. The blind test (36.9% core-meaning recovery, 15.6%–64.4% across models) supports my concern that Efficacy depends on the reference annotation and L3 on the given options. Please state this, report the blind test, and revise the sentences suggesting that the meaning comes from the image alone (p5 l.262; p7 l.344–345 and l.354–355; Fig. 11).

I keep my rating at 3.

AuthorsOpenReview ↗

Response to Reviewer s9bz

Thank you for the helpful follow-up and your continued support. We are incorporating your suggestions into the revised manuscript, including the additional appendix analyses and clarifications of the evaluation scope. We will upload the updated manuscript and summarize the changes shortly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Highlights of Contributions

Dear Reviewers and Area Chair,

We sincerely thank the reviewers and Area Chair for their time, thoughtful assessments, and constructive feedback on our submission. We appreciate the opportunity to clarify our work and strengthen the manuscript through this discussion.

We are encouraged that all three reviewers gave our submission a positive rating of 3, recognizing the value of the benchmark, its fine-grained diagnostic framework, and its empirical evaluation. During the discussion, we provided point-by-point responses, additional human evaluation and audit analyses, and corresponding manuscript revisions. We summarize these revisions and the contributions recognized by the reviewers below.

1. Summary of Revisions

  1. Addition: L4 failure audit with inter-annotator agreement, together with an analysis of ranking robustness to evaluation-level weighting. Appendix A.9

  2. Addition: Annotation-blind human evaluation of metaphor clarity and intended-meaning recovery. Appendix A.10

  3. Modification: Clearer definitions of L3 and L4, and clarification of reference-conditioned Efficacy versus unaided interpretation. Section 3.2, Figure 11

  4. Modification: More precise interpretation of structure/mapping bottlenecks and descriptive-prompt gains. Sections 4.2 and 4.4

  5. Modification: Clearer descriptions of conceptual and descriptive prompts and the purpose of comparing these settings. Sections 3.1 and 4.4

  6. Modification: Expanded comparisons with prior visual-metaphor generation and understanding work, including a correction to MetaCLUE’s evaluation entry. Section 2.3, Table 1

  7. Addition: Theoretical grounding of the structural taxonomy in Phillips and McQuarrie (2004) and Forceville (1996). Section 3.1

  8. Modification: Clarification of distractor validation and the role of the “cannot be determined” option. Section 3.2

  9. Addition: Discussion of shared judge preferences and future evaluation with more abstract or core-message prompts. Section 5

Our responses additionally report inter-rater reliability and leave-one-annotator-out analyses for the original human-alignment study, alongside clarifications of other evaluation and dataset-design choices.

Reviewer s9bz confirmed that the revised L4 definition and related-work discussion “resolve Q3 and the definitional part of W1.” The additional appendix analyses and manuscript clarifications address their follow-up requests. Reviewer FhTq acknowledged the “details on annotation-blind human evaluation + inter-rater reliability.” Reviewer W5cg reaffirmed that “the dataset is useful for the community.” All three reviewers maintained their positive ratings.

2. Highlights of Contributions

  1. A visual-metaphor generation benchmark grounded in real-world imagery. VMetaphor-Bench comprises 1,500 curated metaphors with structured annotations and paired conceptual/descriptive prompts. Reviewer W5cg highlighted the use of “real visual metaphors found in the wild” and noted that “annotations are verified by trained individuals.” Reviewer FhTq recognized the dataset construction and its “rigorous testing and eval frameworks.”

  2. A unified framework for fine-grained diagnosis. The benchmark combines 9,594 MCQs across four levels of metaphorical fidelity with three complementary quality dimensions. Reviewer s9bz noted that “the four MCQ levels distinguish object recognition from errors in relationships and mappings” and that “paired prompt granularities test the effect of supplying visual details.” Reviewer W5cg emphasized that the evaluation enables “a fine-grained understanding of failures and successes.”

  3. Broad model coverage and informative empirical findings. Evaluation of 12 generators reveals recurring challenges in structural composition and element mapping, alongside differences between visual harmony and metaphorical expression. Reviewer W5cg recognized that “a comprehensive suite of models is evaluated,” while Reviewer FhTq highlighted the “clear and detailed results for all dimensions (efficacy, harmony, logic) and the MCQs.”

  4. Reliability analyses supporting broad model comparisons. Our evaluation includes cross-judge comparisons, human validation, generation-seed analysis, and structure-reweighted rankings. Reviewer s9bz highlighted these checks and concluded that “Together they support broad model comparisons.” Reviewer FhTq likewise recognized the “good human validation of the benchmark results.” The additional audit and annotation-blind study further clarify how the benchmark’s scores should be interpreted.

We sincerely appreciate the reviewers’ positive assessments and constructive suggestions. Their feedback has helped us strengthen the empirical support, clarify the evaluation scope, and improve the precision of our conclusions.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.