- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Visual Simile Generation (VSG), a diagnostic task for text-to-image models prompted with similes of the form “⟨A⟩ that looks like ⟨B⟩.” The paper argues that successful generation requires preserving ⟨A⟩ as the main subject, transferring the intended resemblance attribute from ⟨B⟩, and avoiding literal transfer of unrelated features of ⟨B⟩. It formalizes these as three dimensions of canonicity and corresponding failure modes.
The main contributions are VSG-Bench, a benchmark of 549 prompts spanning four resemblance axes and 13 fine-grained relations, with each item annotated with a licensed resemblance channel and literal particulars to exclude, and CANONSCORE, a three-legged evaluation framework measuring subject anchoring, resemblance transfer, and abstraction from the comparison object.
The paper evaluates 16 recent T2I systems and finds that no model is consistently strong across all three dimensions, with a notable tradeoff between resemblance transfer and over-literalization. The validation includes multiple independent VLM judges, leave-one-judge-out robustness analysis, prompt-rewriting interventions using Gemma and ViPE, and a human study showing that CANONSCORE largely tracks the direction of human judgments across conditions.
Critical strengths
-
The paper provides a novel, clear, and useful decomposition of visual simile generation into subject preservation, resemblance transfer, and abstraction from literal comparison-object features. This is consequential because it exposes distinct failure dimensions that standard single-score T2I metrics cannot separate.
-
The paper identifies a meaningful resemblance–literalization tradeoff across models. This is more than a benchmark ranking: it reveals a systematic limitation in current T2I systems, where stronger resemblance transfer often comes with increased literalization of the comparison object.
-
The evaluation is broad and comparatively robust. The paper benchmarks 16 T2I systems over a structured benchmark spanning four resemblance axes and 13 fine-grained relations, and further uses multiple independent VLM judges, leave-one-judge-out analysis, and human validation, increasing confidence that the main trends are not artifacts of a single evaluator.
Critical weaknesses
-
The paper argues that visual similes are difficult because the intended resemblance mapping changes across object pairs. Although this explanation is intuitive and likely correct, the paper does not empirically establish that the difficulty mainly stems from inferring the intended cross-domain mapping. Comparing implicit similes (open similes) with prompts that explicitly state the intended attribute would help test this claim (closed similes).
-
The licensed channels are human-vetted, but it is unclear whether they reflect the dominant human interpretation of the original simile. An independent elicitation study, where annotators infer the intended resemblance attribute without being shown the proposed channel, would strengthen this claim.
-
The human study validates whether changes in individual CANONSCORE legs under prompt augmentation move in the same direction as human judgments. However, it does not directly validate the absolute values of the CANONSCORE legs on the base prompts. The paper nevertheless interprets these base-prompt leg values to characterize model behavior, and in some cases compares models along individual legs. It therefore remains unclear how well the absolute CS_anchor, CS_resem, and CS_bound scores correspond to human judgments. A complementary validation could ask humans to rate base-prompt outputs directly along each criterion and compare those ratings with the corresponding CANONSCORE leg.
-
The paper motivates its three-way decomposition as capturing recurring failure modes, but does not quantify how often these failures occur individually or jointly at the image level, nor whether substantial errors fall outside this taxonomy. Even if CANONSCORE is intentionally continuous, an auxiliary human annotation of a representative subset for presence/absence of each failure type would help establish the empirical coverage of the proposed decomposition and clarify how often the modes co-occur.
-
The paper does not show representative Gemma or ViPE rewritten prompts. These examples would help interpret the reported score shifts.
Questions for authors
-
The paper attributes the difficulty of visual similes to the need to infer the intended resemblance mapping from the object pair. Do you have evidence that this inference step is a substantial source of failure? Do models generate closed-simile images successfully?
-
Were the licensed channels ever validated through an independent elicitation setting in which annotators saw only the original simile and were asked to identify the intended resemblance attribute, without being shown the proposed licensed channel? If so, how often did annotators converge on the benchmark annotation?
-
The human study evaluates whether changes in CANONSCORE under prompt augmentation agree in direction with changes in human judgments. Do you have any validation of the absolute values of the individual CANONSCORE legs on the base prompts—for example, human ratings of CS_anchor, CS_resem, and CS_bound on the same outputs?
-
Do you have any human error analysis measuring how often the three proposed failure modes occur individually or jointly, and whether annotators observe substantial failures that do not fit any of the three categories?
6 comments
Official Response to Reviewer pP4B, Part 2/5
W2, Q2: Verification and agreement for the per-item annotations
Q: Does the licensed channel reflect a human reading of the simile, and how often do annotators converge on the benchmark annotation?
We audited 56 items from a relation-stratified random downsample VSG-Bench, working with three human raters independently (none of them is an author). A majority endorsed all 56 items' licensed channels and all their 168 literal particulars, with chance-corrected agreement of 0.98 and 0.89. No annotation was rejected.
How the dataset was originally vetted. At construction, every synthesized candidate outside the seed set went through two vetting passes, a human reviewer and a cross-lineage model, and both had to accept (App. A.3). Our original vetting (data curation) covered the noun pair, the relation and the licensed channel (Fig. 8, criterion 3). The vetting was strict in effect: 58 of the 110 rejected MultiMM candidates were rejected on the relation or channel. The three literal particulars were written afterwards by the annotation prompt of Fig. 9. Our follow-up audit below is their first independent human review, and a second review of the channels.
Audit samples and protocol. The 56 items chosen for the audit come from a fixed-seed random draw over the 549 items, stratified by relation; all 13 relations are represented by at least 3 items. None of the three raters is an author of this paper. Each rater saw the simile and answered two questions per item, alone and in a fixed random order:
- Licensed channel. Is the annotated trait a reasonable way in which ⟨A⟩ looks like ⟨B⟩? (Yes / No / Unsure)
- Literal particulars. For each of the three annotated features of ⟨B⟩, should a picture of the simile leave it out? Raters answered Yes only when the feature (i) is not the licensed channel, (ii) would not appear in an ordinary image of ⟨A⟩ alone, and (iii) is depictable in a single image. These are the first three conditions of the annotation instruction in Fig. 9.
| Annotation | Units | Endorsed by majority (≥ 2 of 3) | Endorsed by all 3 auditors | Rejected by majority | Raw pairwise agreement | Gwet's AC1 [95% CI] |
|---|---|---|---|---|---|---|
| Licensed channel | 56 items | 56 (100%) | 54 (96.4%) | 0 | 97.6% | 0.98 [0.94, 1.00] |
| Literal particulars | 168 features | 168 (100%) | 142 (84.5%) | 0 | 89.7% | 0.89 [0.84, 0.93] |
| Full item (channel + all 3 particulars) | 56 items | 56 (100%) | n/a | 0 | n/a | n/a |
CIs come from 2,000 bootstrap resamples over items. Agreement treats Unsure as a third category; it was only used once, on 1 of 504 particular votes, and never on a channel.
- No annotation is rejected. No channel and no particular received a majority No. Where raters split, it was always a single dissent: 2 of 56 channels and 26 of 168 particulars. The dissents are spread over 21 items and all five particular categories.
- Agreement statistic. Endorsement is near the ceiling: 98.8% of channel votes and 94.8% of particular votes are Yes. Under that skew, Fleiss' κ falls to near zero even at 98% raw agreement, a known property of κ when one category dominates (Feinstein & Cicchetti, 1990). We therefore report Gwet's AC1 (Gwet, 2008), which corrects for chance without that distortion, together with raw agreement.
- Endorsement holds across the benchmark's structure. Majority endorsement of the channel is 100% on each of the four resemblance axes. Majority endorsement of the particulars is 100% in each of the five categories; the per-vote Yes rate ranges from 89.7% (colour) to 97.3% (part/component).
- The split votes do not drive the trade-off or the CS_bound ranking. On the 56 items, we recomputed CS_bound after removing all 26 particulars that received any vote other than Yes, for all 16 systems under each judge:
| Judge | r(CS_resem, CS_bound), all particulars | r(CS_resem, CS_bound), split particulars removed | Spearman ρ of CS_bound model ranking vs. original |
|---|---|---|---|
| CLIP-FlanT5-XXL | −0.88 | −0.92 | 0.94 |
| InternVL3-14B | −0.90 | −0.91 | 0.97 |
| Qwen2.5-VL-32B | −0.89 | −0.91 | 0.98 |
The trade-off is unchanged or slightly stronger without the split particulars, and the model ranking on CS_bound is preserved.
In the revision, we will add our audit findings to Appendix A.3: the sampling procedure, the exact rater instructions, the agreement table, per-relation counts, the list of split votes and the CS_bound sensitivity check. Furthermore, we will amend Appendix A.4 with the description that the literal particulars are model-written under Fig. 9 and human-audited on this sample.
References:
Feinstein & Cicchetti (1990), J. Clin. Epidemiol. 43(6), 543–549, doi:10.1016/0895-4356(90)90158-
L. Gwet (2008), Br. J. Math. Stat. Psychol. 61(1), 29–48, doi:10.1348/000711006X126600.
Official Response to Reviewer pP4B, Part 3/5
W3, Q3: Agreement with existing human ratings (absolute-value agreement)
Q: Beyond the direction of change, what absolute-value agreement do the existing human ratings support?
Absolute values. The existing survey contains these ratings: human raters scored each base-prompt output directly on each criterion (q1 to q4). On the 65 base-prompt images:
| CS_anchor vs q1 · q2 | CS_resem vs q3 | CS_bound vs (1 − q4) | |
|---|---|---|---|
| Mean offset (metric − human) | +0.245 | +0.178 | −0.036 |
| Lin's CCC [95% CI] | 0.37 [0.23, 0.50] | 0.40 [0.10, 0.67] | 0.68 [0.54, 0.77] |
CS_bound matches the human scale directly. CS_anchor and CS_resem sit above it, by 0.245 and 0.178, so their absolute levels should not be read as human-scale values. Appendix B.3 already states how leg values are read: "a leg value is interpreted against other conditions and other systems, never as an absolute level, because the level is the part that moves with the judge." At that level, the model orderings on the 5 T2I backbone models agree with humans on base prompts: Spearman at 1.0 with 95% CI at [0.4, 1.0], 0.9 [0.3, 1.0] and 1.0 [0.6, 1.0] , across the three CanonScore legs - CS_anchor, CS_resem, and CS_bound, respectively.
In the revision, we will add these tables above to Appendix C.2, with all the details reported above as-is.
Official Response to Reviewer pP4B, Part 1/5
We appreciate Reviewer pP4B for the thoughtful review and suggestions. Please find our responses in the following multi-part thread.
W1, Q1: Closed similes that state the licensed attribute
Q: Is inferring the intended mapping a substantial source of failure? Do models generate closed-simile images more successfully?
Only partially. Stating the intended mapping in the prompt reduces over-literalization for every model, but it leaves resemblance almost unchanged and lowers anchoring.
We ran the closed-simile condition as the reviewer suggested. Every benchmark entry already stores a templated paraphrase that names the licensed channel (Sec. 3.4, field iii). For example, "the moon that looks like a boat" becomes "the moon with the outline of a boat". We generated one image per closed simile for all 14 open-weight models (549 prompts each, with each model's default settings as in the main run), then scored the images with the unchanged CanonScore pipeline against the same annotations.
| 14 of 14 up | | CS_resem | 0.682 | 0.693 | +0.011
+0.003,+0.019| 9 of 14 up | | CS_anchor | 0.490 | 0.446 | −0.044 [−0.052, −0.036] | 14 of 14 down |
Values average the three judges and the 14 models. CIs come from 5,000 bootstrap resamples over the 549 prompts. CS_resem is the mean of s_channel and s_relation (Eq. 3). s_channel uses the same wording as the closed prompt, so we also checked s_relation on its own: +0.013
+0.005,+0.021.
If it's told which attribute to transfer, models intend to import less of ⟨B⟩. CS_bound rises by 0.062, the largest change in the table, and it rises for every model (+0.022 to +0.143). CS_bound measures abstraction, the mode Sec. 5.1 identifies as the one T2I systems fail worst. So inferring the licensed attribute is one source of over-literalization. The gain is modest: it is about a quarter of the spread that separates the sixteen systems on this leg (0.254, Table 9), and smaller than the shift under Gemma-4 rewriting (+0.091, Table 13).
Knowing the mapping does not close the resemblance gap. With the channel given explicitly, CS_resem rises by 0.011 (s_relation alone: +0.013). No single model gains more than 0.042.
This is consistent with how the paper frames the task. Our conclusion (Sec. 6) describes the task's difficulty as "a measurement problem before it is a generation problem". The licensed attribute shifts with every object pair, which is why each VSG-Bench item carries an annotated channel: without it, "looks like" cannot be scored. The closed-simile run shows that once the channel is given, the generation gap remains. Models told what to transfer still do not render it on ⟨A⟩ much better. It also shows why the three legs are reported separately: a single score would hide that this condition helps one leg and costs another.
The CS_anchor drop is not only a probe-wording effect. s_prompt still asks "⟨A⟩ that looks like ⟨B⟩", which the closed prompt no longer says, and it shows the largest drop (0.759 to 0.725). The two sub-scores that do not depend on this wording fall as well: their product, s_subj-main · (1 − s_juxt), falls from 0.608 to 0.582, and for 13 of 14 models.
In the revision we will add the closed-simile condition to the appendix, with both tables, per-model values and the generation configuration.