- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Summary
VMetaphor-Bench is a fine-grained diagnostic benchmark comprising. Contains 1,500 visual metaphors curated from real-world creative imagery, categorized across 3 levels and 10 thematic domains.
Conventional T2I benchmarks such as GenEval and T2I-CompBench primarily assess literal object, attribute, and spatial fidelity, overlooking cross-domain conceptual blending. Existing visual-metaphor benchmarks (eg MetaCLUE) largely focus on metaphor understanding. Generation evaluations are typically small-scale and rely on generic metrics (FID/ CLIP) or humans, and, lack fine-grained diagnostics of where and why models fail at visual metaphor generation, which VMetaphor-Bench aims to provide.
Process
Rigorous evaluation and dataset generation process
Dataset Generation
- Image collection: ~5000 candidate images from Pinterest: general and domain-specific visual-metaphor search terms.
- Initial filtering: human (2 separate) annotators remove duplicates, ambiguous metaphors, and low-quality images -> ~ 1600 retained.
- Structured annotation: an annotation model (gemini 3.1) generates source domain, target domain, structure type, meaning, and element-level correspondences.
- Prompt generation (stage #2 annotation): annotation model produces conceptual and descriptive prompts from image and annotation information.
- Manual verification: 3 researchers independetly examine and correct imagery, annotations, and prompts -> 1500 images.
- Question generation: MM-LLM (gemini 3.1) constructs MCQs from structured annotations and descriptive prompts.
- Question verification: Human annotators validate the resulting ~10k MCQs.
Benchmark task & Eval for MLMs
Check metaphorical fidelity, and expressive quality. Use dimension-based scoring (Efficacy, Logic, Harmony) + MCQ scoring.
- Image generation: 12 T2I models being tested generate images from benchmark prompts.
- Automatic assessment: another MLLM(Qwen3.5-27B) evaluates generated images -> third one (GPT-5.4) provides cross-judge comparisons.
- Human validation: 5 annotators independently assess sampled outputs ->examine model coverage and generation-seed sensitivity.
Critical strengths
- Useful dataset generation work, involving rigorous testing and eval frameworks. Vis. metaphor generation is a useful area as well
- Strong overall benchmark with 1.5k images & ~10k MCQs, 10 themes.
- Good test model coverage (12) on the new benchmark
- Good human validation of the benchmark results
- Clear and detailed results for all dimensions (efficacy, harmony, logic) and the MCQs.
Critical weaknesses
- It will be useful to perform larger human validation study to check the scores (to see if the intended metaphor gets communicated). To quote the paper “models can depict the right elements but struggle to compose them into coherent metaphorical structures”.. this might imply that the generated images might really make little sense to real humans, implying the other scores might not really matter much at all
- The MLLM judges can have shared biases (eg: familiarity with common ad metaphors, similar pretraining datasets can infuse similar biases)
- Similarly in the same section, more details are required on the “Human Alignment Study” - inter-rater agreement statistics will be useful
- Dataset biases/imbalances: source was just one social media site (pinterest). Similarly, “replacement is far more challenging than fusion or juxtaposition”, but the dataset contains mostly fusion images (per A.2 appendix)
- Removing “Ambiguous samples” might also have removed the best/most challenging datapoints
- The “conceptual prompt” appears to cover the exact metaphor needed to be generated. So, the benchmark is not exactly testing for creative space exploration of MLLMs (as a future follow-up work, I’d love to see another level of a “core message prompt” - which would describe a vague idea to deliver through the image)
- Table 7(A.3) shows that the default resolution generated across the models are very different. This makes it harder to read the comparis
Questions for authors
- You decided to use Gemini 3.1 Pro for its “strong visual reasoning capabilities” for the benchmark dataset generation, while you decided to go with Qwen 27B, again for “strong visual reasoning capabilities” for automatic benchmark task judgement - any reason for this? Any comparisons b/w these and other MLMs on the different tasks?
- Re. the “overall dimension score averages the three dimensions” - is there a good reason to give equal weight to these? An image with weak metaphor might get good score just because of “harmony”
4 comments
Response to Reviewer FhTq (part 2)
W5: Removing ‘Ambiguous samples’ might also have removed the best/most challenging datapoints
A5: Thank you for pointing out this trade-off. Ambiguous imagery can indeed contain rich and challenging metaphors. Our filtering aimed to retain samples whose annotations could be reliably verified: when both reviewers had difficulty establishing the intended meaning or cross-domain correspondences, we could not confidently validate the reference annotation. Retaining such samples would also make evaluation difficult to interpret, since a low score could reflect an uncertain reference rather than a failure of the generated image. The criterion was therefore interpretability for reliable annotation and evaluation, rather than metaphor complexity.
W6: The ‘conceptual prompt’ appears to cover the exact metaphor needed to be generated
A6: Thank you for this thoughtful suggestion. We agree that our conceptual prompts specify the metaphor to be realized, so the benchmark evaluates models’ ability to visually express a specified metaphorical intent, rather than open-ended metaphor ideation. We adopted this design to provide a shared semantic target for consistent comparison and fine-grained diagnosis. Conceptual prompts explicitly state the intended meaning, whereas descriptive prompts describe its visual realization without explicitly stating that meaning.
A “core message prompt” would usefully extend this scope by allowing models to choose the concepts and their relationships to communicate a message. Evaluating this setting would require accommodating a broader range of metaphorical realizations beyond the current reference annotations. We have added this direction to Section 5, proposing more abstract or core-message prompts together with evaluation protocols that accommodate diverse interpretations and visual realizations.
W7: Table 7(A.3) shows that the default resolution generated across the models are very different
A7: Thank you for highlighting the differences in generation resolution. We used model-specific settings to evaluate each model in its standard usage configuration. Janus-Pro uses its native 384×384 generation setup, while Seedream 5.0 Lite’s API has a minimum pixel requirement that excludes 1024×1024 output. For Qwen-Image-2512, we adopted the 1328×1328 square-image setting provided in its official example.
Q1: Why use Gemini 3.1 Pro for dataset construction and Qwen 27B for evaluation? Any comparisons b/w these and other MLMs on the different tasks?
A8: Thank you for asking us to clarify the model-selection rationale. The two stages have different requirements. Dataset construction is a one-time process whose annotation quality directly affects all subsequent evaluations, so we prioritized annotation quality when selecting Gemini 3.1 Pro. In our early qualitative trials with other leading proprietary multimodal models, human inspection favored Gemini’s annotations. These trials informed our choice but were not a quantitative model comparison. The resulting annotations and prompts then underwent multiple stages of human verification and correction.
For automatic evaluation, we selected Qwen3.5-27B as a strong open-weight model to make repeated, large-scale benchmarking more affordable and accessible. This choice is supported by the comparisons in Section 4.5: Qwen and GPT-5.4 yield highly consistent model rankings, with Spearman’s ρ of 0.993 for MCQ scores and 0.951 for dimension scores. Both also show good alignment with human assessments under the same protocol. Thus, our selection combines quality-focused, human-verified dataset construction with a cost-conscious evaluator supported by cross-judge and human-alignment evidence.
Q2: Re. the ‘overall dimension score averages the three dimensions’ - is there a good reason to give equal weight to these?
A9: Thank you for raising this point. We use equal weighting to provide a transparent summary of three complementary aspects of visual metaphor quality: communicating the intended meaning, maintaining coherent cross-domain mappings, and achieving a coherent visual presentation. Without a validated basis for assigning different relative weights, we give each dimension equal weight on the shared 1–5 scale.
A high Harmony score can raise the average despite weaker metaphorical expression. We therefore report all three dimension scores separately alongside the overall score and the four-level MCQ results. The overall dimension score summarizes these complementary qualities. It is not a standalone criterion for successful metaphor communication. Indeed, our analysis explicitly highlights cases where visual harmony exceeds metaphorical efficacy, using this discrepancy to diagnose models’ limitations.
Acknowledging the responses. Thanks for details on annotation-blind human evaluation + inter-rater reliability
maintaining initial rating
Response to Reviewer FhTq
We appreciate your follow-up and are glad we could clarify these points. We sincerely appreciate your recognition of our contributions. Thank you again for the time and care you put into reviewing our paper.
Response to Reviewer FhTq
We greatly appreciate your thoughtful review and encouraging assessment of our benchmark and evaluation. Your suggestions have strengthened our analysis and presentation. Below, we provide our responses and hope to address your concerns.
A1: Thank you for this suggestion. We conducted an annotation-blind human study using the same 180 images from six models as Appendix A.7. Three participants who had not seen the annotations viewed only the images, rated their understanding clarity, and wrote interpretations or indicated “cannot determine”.
The mean blind clarity rating was 3.08/5, and the core meaning recovery rate was 36.9%. Recovery ranged from 15.6% for Janus-Pro to 64.4% for GPT Image 1.5, showing substantial differences alongside considerable room for improvement. Across all images, mean blind clarity ratings correlated strongly with the previous reference-conditioned human Efficacy scores (Spearman’s ρ = 0.816), providing complementary evidence that these scores relate to viewers’ perceived clarity.
We also agree that correct elements and visual harmony alone do not establish successful communication. The other scores serve complementary diagnostic purposes: they help distinguish whether a failure concerns domain presence, structural composition, element mapping, conceptual coherence, or visual presentation. Reporting them separately reveals images that look appealing yet fail to convey the intended meaning, which is a finding of our benchmark.
A2: Thank you for raising this concern. We agree that judges may share preferences for familiar metaphors. Beyond cross-judge consistency, our human-alignment studies show that both judges broadly agree with human assessments under the same evaluation protocol. While this agreement does not rule out all shared biases, it provides evidence that the scores reflect human judgments within the evaluated scope. We have explicitly acknowledged potential shared judge preferences in the Limitations section.
A3: Thank you for this helpful suggestion. We computed inter-rater agreement among five annotators in the original 200-image human-alignment study. Table R1 reports pairwise agreement for MCQs and ordinal Krippendorff’s α for the dimension scores. Agreement varies across evaluation levels, and the dimension scores show appreciable individual variability.
Table R1. Inter-rater agreement.
Although individual ratings vary, the reliability of the five-rater mean is higher, with absolute-agreement ICC(A,5) values of 0.805 for Efficacy, 0.812 for Logic, and 0.740 for Harmony.
We also performed five-fold leave-one-annotator-out analysis. Each held-out annotator and both judges were compared against the same reference formed by the other four annotators, using voting for MCQs and averaging for dimension ratings.
Table R2. Agreement with the same four-annotator reference, averaged across folds.
Despite individual variability, both judges show closer overall agreement with the four-person reference than the average held-out annotator. This shows that the judges’ deviations from aggregated human judgments are smaller overall than those of individual annotators.
A4: Thank you for highlighting these points. Pinterest provides access to diverse creative imagery, but collecting from a single platform can introduce selection biases. We acknowledge this limitation in Section 5 and do not claim that the benchmark represents all visual-metaphor distributions. Regarding structural imbalance, we did not impose structure-specific quotas; the distribution reflects the collected corpus. The predominance of fusion is also consistent with prior findings [1].
The replacement finding uses accuracy within each structure type: 19.3% for replacement, 68.7% for fusion, and 70.7% for juxtaposition (Appendix A.4). Equal weighting across structures yields ranking correlations of ρ = 0.979 for both MCQ and dimension scores relative to the original weighting (Appendix A.2). L2/L4 remain below L1/L3. These results support robustness to structure weighting.
[1] Margariti et al. A Path to Our Heart: Visual Metaphors and “White” Space in Advertising Aesthetic Pleasure. Int. J. Advertising, 2022.