acceptodds
Under review as a conference paper at ICLR 2027

: Evaluating Visual Simile Canonicity in Text-to-Image Models

Abstract

When a text-to-image (T2I) model is asked for "a curved moon", it will draw a crescent with ease. However, when it is prompted with a simile such as "a moon that looks like a boat", it often draws a plain boat, a moon beside a boat, a full moon, or a moon with apparent masts and sails. Such difficulty is because *"looks like"*'s implication shifts with every pair of things being compared to in the prompt, e.g, a moon that resembles a boat takes a boat's curve, while a cherry that resembles a ruby takes a ruby's color. To bridge this gap so that T2I models can better handle such prompts with implicated resemblance, we thus formalize the **Visual Simile Generation (VSG)** task: given a simile in form of " that look(s) like ", a T2I model must: 1., install as the dominant subject, 2., identify and transfer the implicated attribute from the comparison object onto the subject , and 3., render no concrete features of literally. We curate **VSG-Bench**, consisting of 549 simile prompts spanning 13 resemblance relations over four axes. At evaluation, each prompt licenses the one attribute of a generation is expected to transfer, and names the concrete features of it must leave out. We also establish the composite CanonScore metric, which scores a generation on each of the three requirements separately: for the subject, for the transferred attribute, and for staying clear of 's concrete features. We benchmark 16 recent T2I systems spanning various architectures. We find no current T2I system can satisfy all three requirements simultaneously. More signifcantly, corroborated by human raters, our full paradigm effectively exposes **the resemblance-literalization tradeoff**, that the T2I systems conveying the intended resemblance attribute best are also the ones literalizing the most of the comparison objects.

Primary area:
datasets and benchmarks
Keywords:
Text-To-Image Generation, Figurative Language Understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pP4BOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Visual Simile Generation (VSG), a diagnostic task for text-to-image models prompted with similes of the form “⟨A⟩ that looks like ⟨B⟩.” The paper argues that successful generation requires preserving ⟨A⟩ as the main subject, transferring the intended resemblance attribute from ⟨B⟩, and avoiding literal transfer of unrelated features of ⟨B⟩. It formalizes these as three dimensions of canonicity and corresponding failure modes. The main contributions are VSG-Bench, a benchmark of 549 prompts spanning four resemblance axes and 13 fine-grained relations, with each item annotated with a licensed resemblance channel and literal particulars to exclude, and CANONSCORE, a three-legged evaluation framework measuring subject anchoring, resemblance transfer, and abstraction from the comparison object.
The paper evaluates 16 recent T2I systems and finds that no model is consistently strong across all three dimensions, with a notable tradeoff between resemblance transfer and over-literalization. The validation includes multiple independent VLM judges, leave-one-judge-out robustness analysis, prompt-rewriting interventions using Gemma and ViPE, and a human study showing that CANONSCORE largely tracks the direction of human judgments across conditions.

Critical strengths
  1. The paper provides a novel, clear, and useful decomposition of visual simile generation into subject preservation, resemblance transfer, and abstraction from literal comparison-object features. This is consequential because it exposes distinct failure dimensions that standard single-score T2I metrics cannot separate.

  2. The paper identifies a meaningful resemblance–literalization tradeoff across models. This is more than a benchmark ranking: it reveals a systematic limitation in current T2I systems, where stronger resemblance transfer often comes with increased literalization of the comparison object.

  3. The evaluation is broad and comparatively robust. The paper benchmarks 16 T2I systems over a structured benchmark spanning four resemblance axes and 13 fine-grained relations, and further uses multiple independent VLM judges, leave-one-judge-out analysis, and human validation, increasing confidence that the main trends are not artifacts of a single evaluator.

Critical weaknesses
  1. The paper argues that visual similes are difficult because the intended resemblance mapping changes across object pairs. Although this explanation is intuitive and likely correct, the paper does not empirically establish that the difficulty mainly stems from inferring the intended cross-domain mapping. Comparing implicit similes (open similes) with prompts that explicitly state the intended attribute would help test this claim (closed similes).

  2. The licensed channels are human-vetted, but it is unclear whether they reflect the dominant human interpretation of the original simile. An independent elicitation study, where annotators infer the intended resemblance attribute without being shown the proposed channel, would strengthen this claim.

  3. The human study validates whether changes in individual CANONSCORE legs under prompt augmentation move in the same direction as human judgments. However, it does not directly validate the absolute values of the CANONSCORE legs on the base prompts. The paper nevertheless interprets these base-prompt leg values to characterize model behavior, and in some cases compares models along individual legs. It therefore remains unclear how well the absolute CS_anchor, CS_resem, and CS_bound scores correspond to human judgments. A complementary validation could ask humans to rate base-prompt outputs directly along each criterion and compare those ratings with the corresponding CANONSCORE leg.

  4. The paper motivates its three-way decomposition as capturing recurring failure modes, but does not quantify how often these failures occur individually or jointly at the image level, nor whether substantial errors fall outside this taxonomy. Even if CANONSCORE is intentionally continuous, an auxiliary human annotation of a representative subset for presence/absence of each failure type would help establish the empirical coverage of the proposed decomposition and clarify how often the modes co-occur.

  5. The paper does not show representative Gemma or ViPE rewritten prompts. These examples would help interpret the reported score shifts.

Questions for authors
  1. The paper attributes the difficulty of visual similes to the need to infer the intended resemblance mapping from the object pair. Do you have evidence that this inference step is a substantial source of failure? Do models generate closed-simile images successfully?

  2. Were the licensed channels ever validated through an independent elicitation setting in which annotators saw only the original simile and were asked to identify the intended resemblance attribute, without being shown the proposed licensed channel? If so, how often did annotators converge on the benchmark annotation?

  3. The human study evaluates whether changes in CANONSCORE under prompt augmentation agree in direction with changes in human judgments. Do you have any validation of the absolute values of the individual CANONSCORE legs on the base prompts—for example, human ratings of CS_anchor, CS_resem, and CS_bound on the same outputs?

  4. Do you have any human error analysis measuring how often the three proposed failure modes occur individually or jointly, and whether annotators observe substantial failures that do not fit any of the three categories?

6 comments
AuthorsOpenReview ↗

Official Response to Reviewer pP4B, Part 1/5

We appreciate Reviewer pP4B for the thoughtful review and suggestions. Please find our responses in the following multi-part thread.


W1, Q1: Closed similes that state the licensed attribute

Q: Is inferring the intended mapping a substantial source of failure? Do models generate closed-simile images more successfully?

Only partially. Stating the intended mapping in the prompt reduces over-literalization for every model, but it leaves resemblance almost unchanged and lowers anchoring.

We ran the closed-simile condition as the reviewer suggested. Every benchmark entry already stores a templated paraphrase that names the licensed channel (Sec. 3.4, field iii). For example, "the moon that looks like a boat" becomes "the moon with the outline of a boat". We generated one image per closed simile for all 14 open-weight models (549 prompts each, with each model's default settings as in the main run), then scored the images with the unchanged CanonScore pipeline against the same annotations.

LegOpen simile (paper)Closed simileChange [95% CI]Models moving in this direction
CS_bound0.4780.540+0.062
+0.052,+0.071+0.052, +0.071

| 14 of 14 up | | CS_resem | 0.682 | 0.693 | +0.011

+0.003,+0.019+0.003, +0.019

| 9 of 14 up | | CS_anchor | 0.490 | 0.446 | −0.044 [−0.052, −0.036] | 14 of 14 down |

Values average the three judges and the 14 models. CIs come from 5,000 bootstrap resamples over the 549 prompts. CS_resem is the mean of s_channel and s_relation (Eq. 3). s_channel uses the same wording as the closed prompt, so we also checked s_relation on its own: +0.013

+0.005,+0.021+0.005, +0.021

.

  1. If it's told which attribute to transfer, models intend to import less of ⟨B⟩. CS_bound rises by 0.062, the largest change in the table, and it rises for every model (+0.022 to +0.143). CS_bound measures abstraction, the mode Sec. 5.1 identifies as the one T2I systems fail worst. So inferring the licensed attribute is one source of over-literalization. The gain is modest: it is about a quarter of the spread that separates the sixteen systems on this leg (0.254, Table 9), and smaller than the shift under Gemma-4 rewriting (+0.091, Table 13).

  2. Knowing the mapping does not close the resemblance gap. With the channel given explicitly, CS_resem rises by 0.011 (s_relation alone: +0.013). No single model gains more than 0.042.

  3. This is consistent with how the paper frames the task. Our conclusion (Sec. 6) describes the task's difficulty as "a measurement problem before it is a generation problem". The licensed attribute shifts with every object pair, which is why each VSG-Bench item carries an annotated channel: without it, "looks like" cannot be scored. The closed-simile run shows that once the channel is given, the generation gap remains. Models told what to transfer still do not render it on ⟨A⟩ much better. It also shows why the three legs are reported separately: a single score would hide that this condition helps one leg and costs another.

  4. The CS_anchor drop is not only a probe-wording effect. s_prompt still asks "⟨A⟩ that looks like ⟨B⟩", which the closed prompt no longer says, and it shows the largest drop (0.759 to 0.725). The two sub-scores that do not depend on this wording fall as well: their product, s_subj-main · (1 − s_juxt), falls from 0.608 to 0.582, and for 13 of 14 models.

CS_anchor sub-score (Table 2)Open simileClosed simileChange
s_prompt ("⟨A⟩ that looks like ⟨B⟩")0.7590.725−0.034
s_subj-main (⟨A⟩ as main subject)0.7900.776−0.014
1 − s_juxt (not two separate objects)0.7760.761−0.015
s_subj-main · (1 − s_juxt)0.6080.582−0.026

In the revision we will add the closed-simile condition to the appendix, with both tables, per-model values and the generation configuration.

AuthorsOpenReview ↗

Official Response to Reviewer pP4B, Part 2/5

W2, Q2: Verification and agreement for the per-item annotations

Q: Does the licensed channel reflect a human reading of the simile, and how often do annotators converge on the benchmark annotation?

We audited 56 items from a relation-stratified random downsample VSG-Bench, working with three human raters independently (none of them is an author). A majority endorsed all 56 items' licensed channels and all their 168 literal particulars, with chance-corrected agreement of 0.98 and 0.89. No annotation was rejected.

How the dataset was originally vetted. At construction, every synthesized candidate outside the seed set went through two vetting passes, a human reviewer and a cross-lineage model, and both had to accept (App. A.3). Our original vetting (data curation) covered the noun pair, the relation and the licensed channel (Fig. 8, criterion 3). The vetting was strict in effect: 58 of the 110 rejected MultiMM candidates were rejected on the relation or channel. The three literal particulars were written afterwards by the annotation prompt of Fig. 9. Our follow-up audit below is their first independent human review, and a second review of the channels.

Audit samples and protocol. The 56 items chosen for the audit come from a fixed-seed random draw over the 549 items, stratified by relation; all 13 relations are represented by at least 3 items. None of the three raters is an author of this paper. Each rater saw the simile and answered two questions per item, alone and in a fixed random order:

  1. Licensed channel. Is the annotated trait a reasonable way in which ⟨A⟩ looks like ⟨B⟩? (Yes / No / Unsure)
  2. Literal particulars. For each of the three annotated features of ⟨B⟩, should a picture of the simile leave it out? Raters answered Yes only when the feature (i) is not the licensed channel, (ii) would not appear in an ordinary image of ⟨A⟩ alone, and (iii) is depictable in a single image. These are the first three conditions of the annotation instruction in Fig. 9.
AnnotationUnitsEndorsed by majority (≥ 2 of 3)Endorsed by all 3 auditorsRejected by majorityRaw pairwise agreementGwet's AC1 [95% CI]
Licensed channel56 items56 (100%)54 (96.4%)097.6%0.98 [0.94, 1.00]
Literal particulars168 features168 (100%)142 (84.5%)089.7%0.89 [0.84, 0.93]
Full item (channel + all 3 particulars)56 items56 (100%)n/a0n/an/a

CIs come from 2,000 bootstrap resamples over items. Agreement treats Unsure as a third category; it was only used once, on 1 of 504 particular votes, and never on a channel.

  1. No annotation is rejected. No channel and no particular received a majority No. Where raters split, it was always a single dissent: 2 of 56 channels and 26 of 168 particulars. The dissents are spread over 21 items and all five particular categories.
  2. Agreement statistic. Endorsement is near the ceiling: 98.8% of channel votes and 94.8% of particular votes are Yes. Under that skew, Fleiss' κ falls to near zero even at 98% raw agreement, a known property of κ when one category dominates (Feinstein & Cicchetti, 1990). We therefore report Gwet's AC1 (Gwet, 2008), which corrects for chance without that distortion, together with raw agreement.
  3. Endorsement holds across the benchmark's structure. Majority endorsement of the channel is 100% on each of the four resemblance axes. Majority endorsement of the particulars is 100% in each of the five categories; the per-vote Yes rate ranges from 89.7% (colour) to 97.3% (part/component).
  4. The split votes do not drive the trade-off or the CS_bound ranking. On the 56 items, we recomputed CS_bound after removing all 26 particulars that received any vote other than Yes, for all 16 systems under each judge:
Judger(CS_resem, CS_bound), all particularsr(CS_resem, CS_bound), split particulars removedSpearman ρ of CS_bound model ranking vs. original
CLIP-FlanT5-XXL−0.88−0.920.94
InternVL3-14B−0.90−0.910.97
Qwen2.5-VL-32B−0.89−0.910.98

The trade-off is unchanged or slightly stronger without the split particulars, and the model ranking on CS_bound is preserved.

In the revision, we will add our audit findings to Appendix A.3: the sampling procedure, the exact rater instructions, the agreement table, per-relation counts, the list of split votes and the CS_bound sensitivity check. Furthermore, we will amend Appendix A.4 with the description that the literal particulars are model-written under Fig. 9 and human-audited on this sample.

References: 

Feinstein & Cicchetti (1990), J. Clin. Epidemiol. 43(6), 543–549, doi:10.1016/0895-4356(90)90158-

L. Gwet (2008), Br. J. Math. Stat. Psychol. 61(1), 29–48, doi:10.1348/000711006X126600.

AuthorsOpenReview ↗

Official Response to Reviewer pP4B, Part 3/5

W3, Q3: Agreement with existing human ratings (absolute-value agreement)

Q: Beyond the direction of change, what absolute-value agreement do the existing human ratings support?

Absolute values. The existing survey contains these ratings: human raters scored each base-prompt output directly on each criterion (q1 to q4). On the 65 base-prompt images:

CS_anchor vs q1 · q2CS_resem vs q3CS_bound vs (1 − q4)
Mean offset (metric − human)+0.245+0.178−0.036
Lin's CCC [95% CI]0.37 [0.23, 0.50]0.40 [0.10, 0.67]0.68 [0.54, 0.77]

CS_bound matches the human scale directly. CS_anchor and CS_resem sit above it, by 0.245 and 0.178, so their absolute levels should not be read as human-scale values. Appendix B.3 already states how leg values are read: "a leg value is interpreted against other conditions and other systems, never as an absolute level, because the level is the part that moves with the judge." At that level, the model orderings on the 5 T2I backbone models agree with humans on base prompts: Spearman at 1.0 with 95% CI at [0.4, 1.0], 0.9 [0.3, 1.0] and 1.0 [0.6, 1.0] , across the three CanonScore legs - CS_anchor, CS_resem, and CS_bound, respectively.

In the revision, we will add these tables above to Appendix C.2, with all the details reported above as-is. 

Reviewer_5Ls2OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies image generation from prompts such as “a moon that looks like a boat.” It introduces 549 prompts with annotations specifying which resemblance attribute should transfer and which literal features should remain absent. CanonScore separately measures subject preservation, resemblance, and abstraction. Experiments on 16 generators identify a resemblance–abstraction tradeoff, supported by judge comparisons and a human study.

Critical strengths
  1. Distinguishing the intended resemblance from unwanted literal features makes an ambiguous generation task concrete. The annotations offer useful diagnostic information beyond generic prompt alignment.
  2. The empirical analysis includes meaningful robustness checks across three judges and multiple resemblance categories. The appendix also distinguishes model-level trends from image-level correlations, helping qualify the reported tradeoff.
Critical weaknesses
  1. The anchoring score does not clearly isolate subject preservation. Eq. (2) includes the whole-simile matching score, so an image that preserves the subject but lacks the intended resemblance can also lose anchoring credit. This overlaps with the paper’s separate Under Transfer failure mode and weakens the central claim of diagnosing which requirement failed.
  2. Human validation mainly compares condition changes across five models on 13 prompts; it does not establish reliable diagnosis for individual images. The scoring description also needs clarification: Figure 10 contains positively worded questions, while its caption describes inconsistent reverse coding. This may be a documentation error, but the mapping affects interpretation of the validation.
Questions for authors
  1. On the existing human-rated images, does removing the whole-simile factor improve anchoring agreement, particularly when the subject is preserved but resemblance is missing?
  2. What is the exact mapping from Figure 10 responses to the three human scores? After resolving the coding discrepancy, what image-level agreement and prompt-level uncertainty do the existing ratings support?
4 comments
AuthorsOpenReview ↗

Official Response to Reviewer 5Ls2, Part 1/3

We appreciate Reviewer 5Ls2 for the thoughtful review and suggestions. Please find our responses in the following thread. In order to provide a full context, we would like to address the main weaknesses in this order: W2b -> W2a ->W1.


W2b, Q2a: Figure 10 item mapping and the human-score calculation

Q: What is the exact mapping from Figure 10 responses to the three human scores?

Thank you for the careful reading. After checking, we found three documenting errors, and we corrected all of them here. None of these corrections changes the paper's findings: every pooled human change keeps its direction and significance, and every correlation in Table 10 remains intact. One count is lower than reported, sign agreement on CS_anchor (3 of 5, not 4 of 5), and we report it below.

The corrected mappings in Figure 10. Only q4 is reverse-coded.

Item (Figure 10)WordingPolarityHuman score
q1"This image is a good depiction of ⟨prompt⟩."positiveanchoring (factor)
q2"The main object in this image is ⟨A⟩, and ⟨B⟩ does not appear as a separate object anywhere in the image."positiveanchoring (factor)
q3"In this image, ⟨A⟩ has the ⟨channel⟩ of ⟨B⟩."positiveresemblance
q4"In this image, at least one of these ⟨literal particulars of B⟩ on or around ⟨A⟩."negativeover-literalization, i.e. reverse-coded abstraction

Each 5-point response r is rescaled to (r − 1)/4. For each rater we compute anchoring = q1 · q2, resemblance = q3 and abstraction = 1 − q4, then average over raters. Like Eq. 2, human anchoring combines whole-prompt depiction (q1) with subject preservation (q2). The survey has 93 people, with a median of 18 raters per image (range 15 to 27). "Four ratings" in Section 5.2 refers to the four items, not four raters.

The three corrections.

  1. The Figure 10 caption lists the third and fourth items as reverse-coded. Only the fourth is reverse-coded.
  2. Appendix C.2 incorrectly states q2 as reverse-coded, and the submitted pooled anchoring change (−0.070) corresponds to that coding. With q2 corrected as above, the correct pooled anchoring change should be −0.057.
  3. Sign agreement on CS_anchor is 3 of 5, not 4 of 5, which makes the total 12 of 15, not 13 of 15. This count does not depend on the q2 coding.

Recomputed survey numbers, q2 as worded.

Human scoreBaseGemma-4Change [95% CI]Paperp
Anchoring, q1 · q20.3370.280−0.05 [−0.087, −0.029]−0.0704.2e−4
Resemblance, q30.5840.469−0.115−0.1157.8e−10
Abstraction, 1 − q40.5150.700+0.186+0.1863.6e−13

The tests are two-sided Wilcoxon signed-rank on per-person means over the 93 human raters. The Anchoring CI comes from 5,000 bootstrap resamples over all 93 raters' sessions. All three pooled changes clear a Bonferroni correction for six comparisons (for anchoring, 6 × 4.2e−4 ≈ 0.0025). The human Anchoring level across both prompt conditions (base prompt vs. Gemma-rewritten) is 0.31, supporting the "near 0.60 against 0.30" claim in Section 5.2.

Agreement with humans, over generations from 5 T2I methodsPaperCorrected
Spearman, CS_anchor (product) vs human anchoring (Table 10)+0.90+0.90
Spearman, CS_anchor arithmetic mean / geometric mean / minimum+0.80 / +0.50 / +0.40+0.80 / +0.50 / +0.40
Spearman, CS_resem / CS_bound vs their human scores+1.00 / +0.80+1.00 / +0.80
Sign agreement, CS_anchor / CS_resem / CS_bound4/5, 5/5, 4/53/5, 5/5, 4/5
Model-leg cells agreeing in sign (Sec. 5.2, Fig. 11)13 of 1512 of 15

Every correlation in Table 10 is unchanged, so the product remains the strongest CS_anchor aggregation.

Reading the 3 of 5. The two disagreeing models are SD 3.5 and Show-o2-7B. Their CS_anchor changes are +0.006 and −0.014, and their human changes are −0.029 and +0.025. The Figure 11 caption already places both metric changes within 0.03 of zero, and Appendix C.2 reads such a cell as "a sign assigned to a quantity that barely moved rather than a reversal of a clear effect". The rank correlation stays at +0.90 because it compares the ordering of the five changes, and the signs differ only where both changes are near zero.

In the revision, we will:

  • correct the Figure 10 caption;
  • correct the item description and the pooled anchoring value in Appendix C.2, and name SD 3.5 and Show-o2-7B in its disagreeing-cell sentence;
  • update the CS_anchor sign-agreement column of Table 10, the sign counts in Section 5.2 and the Figure 11 caption;
  • redraw the anchoring panel of Figure 11 on the corrected human axis;
  • state the number of raters per image in Section 5.2.
AuthorsOpenReview ↗

Official Response to Reviewer 5Ls2, Part 2/3

W2a, Q2b: Agreement with existing human ratings (image-level diagnosis and prompt-level uncertainty)

Q: Beyond the direction of change, what image-level agreement do the existing human ratings support, and with what prompt-level uncertainty?

With the coding problem addressed in Figure 10, we show that each CanonScore leg detects its corresponding failure mode at AUROC 0.81 to 0.88. Also, each CanonScore leg tracks its human score image by image at Spearman 0.55 to 0.60. 

Scope of our existing human evaluations (App C.): We rated 130 images (5 T2I generators × 13 prompts, one prompt per relation, base and Gemma-4 rewritten), with a median of 18 raters per image (range 15 to 27). Human scores follow the mapping in our corrected reply on Figure 10. Metric scores average the three judges. 95% CIs come from 5,000 bootstrap resamples over the 13 prompts, keeping each prompt's models and conditions together, so they include prompt-to-prompt variation.

  1. Each CanonScore leg gives lower scores to the images that humans label as in failure modes. An image carries a human failure label for a mode when the mean of the matching item is ≤ 0.375, a mean rating of 2.5 on the 5-point scale: q2 for Wrong Subject, q3 for Under Transfer, and 1 − q4 for Over-Literalization. The Wrong Subject label uses q2 rather than q1 · q2 because Sec. 3.3 defines that failure as ⟨A⟩ losing main-object status or ⟨B⟩ appearing separately, which is what q2 asks. The 130 rated images cover all 13 relations, all 5 backbones and both prompt conditions.
Failure mode (human label)LegPositives / 130AUROC [95% CI]Balanced accuracy
Wrong Subject (q2)CS_anchor130.82 [0.67, 0.93]0.70
Under Transfer (q3)CS_resem450.81 [0.68, 0.93]0.74
Over-Literalization (1 − q4)CS_bound250.88 [0.79, 0.95]0.80

Balanced accuracy uses one threshold per leg: the median of that leg over all 8,784 base images from the sixteen systems (0.57, 0.76 and 0.48). The thresholds are not fitted to the human labels. With the human cut-off moved to 0.25 or 0.5, the three AUROCs stay between 0.78 and 0.92.

  1. Each CanonScore leg ranks the human-rated images similarly to human raters do. For each leg, we rank the 130 rated images by the metric and by the matching human score, and report the Spearman correlation between the two rankings. Each bootstrap resample draws whole prompts, so we also show the 95% CIs (marked within brackets[]) to include prompt variation.
CS_anchor vs q1 · q2CS_resem vs q3CS_bound vs 1 − q4
Spearman, all 130 human-rated images0.60 [0.39, 0.73]0.55 [0.29, 0.78]0.55 [0.29, 0.75]
Spearman, 65 base images (point estimate)0.640.340.66
Spearman, 65 Gemma-4 images (point estimate)0.540.630.37
Human split-half reliability (Spearman–Brown)0.860.920.87

All three lower bounds are at or above 0.29, so the association stays positive when the 13 prompts are resampled. CS_resem has the widest interval and is the least certain of the three. The last row correlates two random halves of the raters, corrected to the full rater count, and gives an approximate upper bound for any metric against these human scores.

The association also holds within each prompt condition, so it is not produced by the shift between base and Gemma-4-rewritten prompts. It is weaker for CS_resem on base images (0.34) and for CS_bound on Gemma-4 images (0.37). Excluding fast, zero-variance or duplicate human rater sessions changes no pooled value by more than 0.02.

In the revision, we will add these tables above to Appendix C.2, with all the details reported above as-is. 

AuthorsOpenReview ↗

Official Response to Reviewer 5Ls2, Part 3/3

W1, Q1: The whole-simile probe regarding CS_anchor

Q: Because Eq. 2 (CS_anchor) includes s_prompt, can an image with the right subject but no resemblance be diagnosed as Wrong Subject instead of Under Transfer? Does dropping s_prompt improve agreement with humans?

No. And we confirm that dropping s_prompt shifts which human score the leg agrees with; it does not raise agreement overall. 

We tested both concerns with the already-stored probe scores. We compared Eq. 2 with an ablated CS_anchor that drops s_prompt: s_subj-main · (1 − s_juxt). Metric scores average the three judges. 95% CIs come from 5,000 bootstrap resamples over the 13 survey prompts.

  1. Under Transfer is not mistaken for Wrong Subject. We took the 31 human-rated images where raters judged the subject kept and the resemblance missing (mean q2 ≥ 0.75, mean q3 ≤ 0.375). For each image we take the CanonScore leg with the lowest percentile, ranked against all base, Gemma-4 and ViPE images, since raw levels are not comparable across legs (Appendix B.3). That leg is CS_resem on 24 of the 31 images (77%), CS_bound on 6 (19%) and CS_anchor on 1 (3%). The totals are the same with and without s_prompt, and the two versions agree on 29 of the 31 individual images.
  2. Dropping s_prompt changes what CS_anchor measures. The survey lets us check this directly, because it asks about anchoring in two parts. Item q1 asks whether the image depicts the whole simile ("This image is a good depiction of ⟨prompt⟩"). Item q2 asks about the subject and separation ("The main object in this image is ⟨A⟩, and ⟨B⟩ does not appear as a separate object"). Human anchoring in the paper is q1 · q2, the counterpart of Eq. 2; q2 alone is the counterpart of the ablated version. Of the two human scores, q1 · q2 is the more reliable: its split-half reliability is 0.86, against 0.56 for q2 alone.
Spearman [95% CI], 130 imagesHuman q1 · q2 (paper)Human q2 alone
Eq. 2: s_prompt · s_subj-main · (1 − s_juxt)0.60 [0.39, 0.73]0.11 [−0.17, 0.34]
Ablated Eq. 2: s_subj-main · (1 − s_juxt)0.02 [−0.23, 0.26]0.60 [0.41, 0.72]

Each version tracks the human score built the same way, at the same level. Dropping s_prompt raises agreement with q2 alone from 0.11 to 0.60, and lowers agreement with full human anchoring from 0.60 to 0.02. The leg moves from "⟨A⟩ is depicted as the subject of the simile" to "⟨A⟩ is the main object". Eq. 2 measures the first, which is how Sec. 3.2 defines subject preservation. Point 1 shows that the diagnosis the reviewers ask about is the same under either definition.

  1. The three CanonScore legs move independently. Across all 549 prompts, Gemma-4 rewriting raises CS_anchor by 0.078 and lowers CS_resem by 0.098 (Sec. 5.2, Table 13), with near-parallel per-model lines (Fig. 4). The direction is the same for each of the three judges, with and without s_prompt. A resemblance-dominated anchoring score could not produce this.

Why Eq. 2 (CS_anchor) includes s_prompt. Sec. 3.2 defines subject preservation as “prior to the other two” criteria: the image must first be a depiction of ⟨A⟩ as the subject of the simile. Sec. 3.3 names Wrong Subject as the failure of this condition: the image “fails to establish ⟨A⟩ as the anchored, dominant subject”. In Eq. 2, s_subj-main and (1 − s_juxt) check the two causes Sec. 3.3 lists, and s_prompt checks that the image depicts the simile as a whole.

In the revision, we will add the ablation, both columns of the table above and the subset analysis of point 1, with its percentile reference and tie rules, to the appendix as a robustness check.

AuthorsOpenReview ↗

We would love to hear more from you.

Dear Reviewer 5Ls2,

We appreciate your constructive suggestions in your original review. However, we have yet to receive your further opinions since our latest update. With the discussion period approaching its end, we would be very grateful to have your feedback.

Sincerely, Authors of Submission 27033

Reviewer_fnxzOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the ability of text-to-image models to generate visual similes and the associated evaluation challenges. We construct a benchmark, VSG-Bench, and propose a metric, CanonScore, which enables fine-grained evaluation across three dimensions: subject preservation, the transfer of similar attributes, and the concretization of the comparative object. Experiments on 16 systems reveal that holistic alignment metrics may inadvertently reward "over-concretization," highlighting the limitations of existing evaluation approaches.

Critical strengths
  1. The authors propose a fine-grained evaluation framework for visual similes that distinguishes between subject preservation, insufficient attribute transfer, and over-concretization. Through sample-by-sample annotation, the framework enables actionable diagnostics, better aligning with the practical requirements of the task.
  2. Experiments involving multiple models, various metrics, and prompt rewrites demonstrate that holistic alignment scores can mask—or even reward—over-concretization. Further analyses, including grouping by relational type, assessing the robustness of the evaluation models, and conducting human evaluations, reinforce these findings.
Critical weaknesses
  1. The three dimensions of CanonScore are not entirely independent; the CSanchorCS_{anchor} metric is influenced by both subject accuracy and attribute transfer, which may lead to confusion between "Wrong Subject" and "Under Transfer" errors. The current ablation studies do not sufficiently demonstrate that the aggregation method can accurately distinguish between these distinct error types.
  2. The evaluation relies on manual annotations—such as permitted attributes and prohibited concrete features—yet there is insufficient evidence regarding independent verification or inter-annotator consistency. Existing human evaluations primarily examine trends rather than directly testing the classification accuracy for the three error types across the full dataset.
  3. The negative correlation between the transfer of similar attributes and the degree of abstraction is observed primarily at the level of model averages, rather than at the level of individual images. Consequently, the findings should be interpreted as tendencies in system behavior; it cannot be inferred that these two objectives are inherently mutually exclusive.
Questions for authors
  1. In Section 4, CSanchorCS_{anchor} incorporates a component (sprompts_{\mathrm{prompt}}) that measures the overall alignment with the simile. However, even if the subject is correctly rendered, the absence of the intended similar attribute could lower this score. Could the authors explain how this design avoids misclassifying "Under Transfer" as a "Wrong Subject" error? Furthermore, if this probe were removed—evaluating subject preservation solely based on the presence of the subject and the juxtaposition of the comparative object—would the primary diagnostic results change?
  2. Were the permission attributes and the three prohibited concrete features for each sample in VSG-Bench subjected to independent human review? Furthermore, can the authors report the category-specific consistency of CanonScore’s misclassifications on the human-annotated images, rather than just the direction of the change in average scores before and after prompt rewriting?
8 comments
AuthorsOpenReview ↗

Official Response to Reviewer fnxz, Part 1/4

We appreciate Reviewer fnxz for the thoughtful review and suggestions. Please find our responses in the following multi-part thread.


W1, Q1a, Q1b: The whole-simile probe regarding CS_anchor

Q: Because Eq. 2 (CS_anchor) includes s_prompt, can an image with the right subject but no resemblance be diagnosed as Wrong Subject instead of Under Transfer? Does dropping s_prompt improve agreement with humans?

No. And we confirm that dropping s_prompt shifts which human score the leg agrees with; it does not raise agreement overall. 

We tested both concerns with the already-stored probe scores. We compared Eq. 2 with an ablated CS_anchor that drops s_prompt: s_subj-main · (1 − s_juxt). Metric scores average the three judges. 95% CIs come from 5,000 bootstrap resamples over the 13 survey prompts.

  1. Under Transfer is not mistaken for Wrong Subject. We took the 31 human-rated images where raters judged the subject kept and the resemblance missing (mean q2 ≥ 0.75, mean q3 ≤ 0.375). For each image we take the CanonScore leg with the lowest percentile, ranked against all base, Gemma-4 and ViPE images, since raw levels are not comparable across legs (Appendix B.3). That leg is CS_resem on 24 of the 31 images (77%), CS_bound on 6 (19%) and CS_anchor on 1 (3%). The totals are the same with and without s_prompt, and the two versions agree on 29 of the 31 individual images.
  2. Dropping s_prompt changes what CS_anchor measures. The survey lets us check this directly, because it asks about anchoring in two parts. Item q1 asks whether the image depicts the whole simile ("This image is a good depiction of ⟨prompt⟩"). Item q2 asks about the subject and separation ("The main object in this image is ⟨A⟩, and ⟨B⟩ does not appear as a separate object"). Human anchoring in the paper is q1 · q2, the counterpart of Eq. 2; q2 alone is the counterpart of the ablated version. Of the two human scores, q1 · q2 is the more reliable: its split-half reliability is 0.86, against 0.56 for q2 alone.
Spearman [95% CI], 130 imagesHuman q1 · q2 (paper)Human q2 alone
Eq. 2: s_prompt · s_subj-main · (1 − s_juxt)0.60 [0.39, 0.73]0.11 [−0.17, 0.34]
Ablated Eq. 2: s_subj-main · (1 − s_juxt)0.02 [−0.23, 0.26]0.60 [0.41, 0.72]

Each version tracks the human score built the same way, at the same level. Dropping s_prompt raises agreement with q2 alone from 0.11 to 0.60, and lowers agreement with full human anchoring from 0.60 to 0.02. The leg moves from "⟨A⟩ is depicted as the subject of the simile" to "⟨A⟩ is the main object". Eq. 2 measures the first, which is how Sec. 3.2 defines subject preservation. Point 1 shows that the diagnosis the reviewers ask about is the same under either definition.

  1. The three CanonScore legs move independently. Across all 549 prompts, Gemma-4 rewriting raises CS_anchor by 0.078 and lowers CS_resem by 0.098 (Sec. 5.2, Table 13), with near-parallel per-model lines (Fig. 4). The direction is the same for each of the three judges, with and without s_prompt. A resemblance-dominated anchoring score could not produce this.

Why Eq. 2 (CS_anchor) includes s_prompt. Sec. 3.2 defines subject preservation as "prior to the other two" criteria: the image must first be a depiction of ⟨A⟩ as the subject of the simile. Sec. 3.3 names Wrong Subject as the failure of this condition: the image “fails to establish ⟨A⟩ as the anchored, dominant subject”. In Eq. 2, s_subj-main and (1 − s_juxt) check the two causes Sec. 3.3 lists, and s_prompt checks that ⟨A⟩ is anchored as the subject of the simile.

In the revision, we will add the ablation, both columns of the table above and the subset analysis of point 1, with its percentile reference and tie rules, to the appendix as a robustness check.

AuthorsOpenReview ↗

Official Response to Reviewer fnxz, Part 2/4

W2a, Q2a: Verification and agreement for the per-item annotations

Q: Were the licensed channel and the three literal particulars of each VSG-Bench item independently reviewed by humans, and how consistent are the annotators?

We audited 56 items from a relation-stratified random downsample VSG-Bench, working with three human raters independently (none of them is an author). A majority endorsed all 56 items' licensed channels and all their 168 literal particulars, with chance-corrected agreement of 0.98 and 0.89. No annotation was rejected.

How the dataset was originally vetted. At construction, every synthesized candidate outside the seed set went through two vetting passes, a human reviewer and a cross-lineage model, and both had to accept (App. A.3). Our original vetting (data curation) covered the noun pair, the relation and the licensed channel (Fig. 8, criterion 3). The vetting was strict in effect: 58 of the 110 rejected MultiMM candidates were rejected on the relation or channel. The three literal particulars were written afterwards by the annotation prompt of Fig. 9. Our follow-up audit below is their first independent human review, and a second review of the channels.

Audit samples and protocol. The 56 items chosen for the audit come from a fixed-seed random draw over the 549 items, stratified by relation; all 13 relations are represented by at least 3 items. None of the three raters is an author of this paper. Each rater saw the simile and answered two questions per item, alone and in a fixed random order:

  1. Licensed channel. Is the annotated trait a reasonable way in which ⟨A⟩ looks like ⟨B⟩? (Yes / No / Unsure)
  2. Literal particulars. For each of the three annotated features of ⟨B⟩, should a picture of the simile leave it out? Raters answered Yes only when the feature (i) is not the licensed channel, (ii) would not appear in an ordinary image of ⟨A⟩ alone, and (iii) is depictable in a single image. These are the first three conditions of the annotation instruction in Fig. 9.
AnnotationUnitsEndorsed by majority (≥ 2 of 3)Endorsed by all 3 auditorsRejected by majorityRaw pairwise agreementGwet's AC1 [95% CI]
Licensed channel56 items56 (100%)54 (96.4%)097.6%0.98 [0.94, 1.00]
Literal particulars168 features168 (100%)142 (84.5%)089.7%0.89 [0.84, 0.93]
Full item (channel + all 3 particulars)56 items56 (100%)n/a0n/an/a

CIs come from 2,000 bootstrap resamples over items. Agreement treats Unsure as a third category; it was only used once, on 1 of 504 particular votes, and never on a channel.

  1. No annotation is rejected. No channel and no particular received a majority No. Where raters split, it was always a single dissent: 2 of 56 channels and 26 of 168 particulars. The dissents are spread over 21 items and all five particular categories.
  2. Agreement statistic. Endorsement is near the ceiling: 98.8% of channel votes and 94.8% of particular votes are Yes. Under that skew, Fleiss' κ falls to near zero even at 98% raw agreement, a known property of κ when one category dominates (Feinstein & Cicchetti, 1990). We therefore report Gwet's AC1 (Gwet, 2008), which corrects for chance without that distortion, together with raw agreement.
  3. Endorsement holds across the benchmark's structure. Majority endorsement of the channel is 100% on each of the four resemblance axes. Majority endorsement of the particulars is 100% in each of the five categories; the per-vote Yes rate ranges from 89.7% (colour) to 97.3% (part/component).
  4. The split votes do not drive the trade-off or the CS_bound ranking. On the 56 items, we recomputed CS_bound after removing all 26 particulars that received any vote other than Yes, for all 16 systems under each judge:
Judger(CS_resem, CS_bound), all particularsr(CS_resem, CS_bound), split particulars removedSpearman ρ of CS_bound model ranking vs. original
CLIP-FlanT5-XXL−0.88−0.920.94
InternVL3-14B−0.90−0.910.97
Qwen2.5-VL-32B−0.89−0.910.98

The trade-off is unchanged or slightly stronger without the split particulars, and the model ranking on CS_bound is preserved.

In the revision, we will add our audit findings to Appendix A.3: the sampling procedure, the exact rater instructions, the agreement table, per-relation counts, the list of split votes and the CS_bound sensitivity check. Furthermore, we will amend Appendix A.4 with the description that the literal particulars are model-written under Fig. 9 and human-audited on this sample.

References: 

Feinstein & Cicchetti (1990), J. Clin. Epidemiol. 43(6), 543–549, doi:10.1016/0895-4356(90)90158-

L. Gwet (2008), Br. J. Math. Stat. Psychol. 61(1), 29–48, doi:10.1348/000711006X126600.

AuthorsOpenReview ↗

Official Response to Reviewer fnxz, Part 3/4

W2b, Q2b: Agreement with existing human ratings (failure-category accuracy)

Q: Beyond the direction of change, what failure-category accuracy do the existing human ratings support?

Category-specific consistency. On the 71 images with at least one human failure label, we compare the lowest human score with the CanonScore leg that has the lowest percentile, ranked against all base, Gemma-4 and ViPE images:

Human lowest ↓ / CanonScore lowest →CS_anchorCS_resemCS_bound
Wrong Subject (10 images)316
Under Transfer (38 images)1298
Over-Literalization (23 images)0122

The two assignments agree on 54 of 71 images (76%). The two categories carrying 61 of the 71 images match in 29 of 38 and 22 of 23 cases. The Wrong Subject row has the fewest images and rests on the least reliable human item (split-half reliability 0.56 for q2, against 0.87 to 0.92 for q3 and 1 − q4).

In the revision, we will add these tables above to Appendix C.2, with all the details reported above as-is. 

Comments on the paper

3 comments

Dear reviewers, the authors have posted their responses to your reviews. Pls have a look and discuss. (Thanks to Reviewer pP4B so far...)

Thanks, AC for #27033

AuthorsOpenReview ↗

Summary of Author Response, Part 1/2

Dear Reviewers and Area Chairs,

We sincerely thank all three reviewers for their constructive feedback. For convenience, we would like to summarize the additional experiments, analyses, clarifications, and revision commitments across our responses.

We appreciate that the reviewers recognized the core value of our work:

  • A clear and useful decomposition of Visual Simile Generation into subject preservation, resemblance transfer, and abstraction from literal comparison-object features (pP4B, fnxz, 5Ls2);
  • Actionable diagnostics beyond generic text--image alignment, exposing failures that holistic scores can conceal or reward (fnxz, 5Ls2);
  • A meaningful resemblance--literalization tradeoff revealing a systematic limitation of current T2I systems (pP4B); and
  • Broad evaluation and robustness checks, covering 16 T2I systems, multiple independent judges, structured resemblance categories, and human validation (pP4B, 5Ls2).

Additional Experiments and Analyses

  1. For 5Ls2 (W1, Q1) and fnxz (W1, Q1), CanonScore anchoring ablation. In an ablation, we removed the whole-simile probe and compared both formulations against the corresponding human measures. We show that removing it changes what anchoring measures rather than uniformly improving agreement. Among 31 images with a preserved subject but missing resemblance, CS_resem is the lowest-percentile dimension on 24, while CS_anchor is lowest on only one. These totals remain unchanged under the ablation, and the formulations agree on 29/31 individual diagnoses.

  2. For 5Ls2 (W2, Q2), fnxz (W2, Q2), and pP4B (W3, Q3), image-level human validation. On 130 human-rated images spanning five backbones, 13 prompts, and two prompt conditions, the three CanonScore dimensions detect their corresponding human-labelled failures with AUROC 0.81-0.88 and correlate with human scores at Spearman 0.55-0.60. We report uncertainty using bootstrap resampling over whole prompts. The lowest-percentile diagnostic agrees with the lowest human-scored failure category on 54/71 failing images (76%), with stronger agreement for Under Transfer and Over-Literalization than for Wrong Subject.

  3. For fnxz (W2, Q2) and pP4B (W2, Q2), independent annotation audit. Three non-author raters audited 56 relation-stratified items, covering all 13 relations. Every licensed channel and all 168 literal particulars received majority endorsement, with Gwet's AC1 of 0.98 and 0.89, respectively. Removing particulars with any dissenting vote preserves the resemblance-abstraction trend and largely preserves model rankings, with ranking correlations between 0.94 and 0.98 across judges.

  4. For pP4B's follow-up (W2, Q2), independent channel selection with the benchmark annotation withheld. Five additional human raters evaluated the same 56 similes used in the aforementioned 3.'s audit, choosing a best-fitting interpretation and other plausible interpretations from all 13 resemblance types presented in randomized order. The benchmark annotation matched 71.8% of first choices and was included among the acceptable interpretations in 89.3% of responses. This complements the endorsement audit by testing whether raters select the benchmark's interpretation without being told which channel it specifies.

  5. For pP4B (W1, Q1), closed-simile generation across all 14 open-weight T2I models. We generated images for all 549 prompts after explicitly stating the intended resemblance attribute. Abstraction improves for 14/14 models, with mean CS_bound increasing by +0.062. However, resemblance improves only slightly (+0.011), while anchoring decreases (-0.044). These results show that identifying the intended mapping contributes to the difficulty, but supplying that mapping does not resolve the generation gap.

  6. For pP4B (W4, Q4), failure frequency and co-occurrence. Among the 65 human-rated base-prompt images, 26 show no labelled failure, 29 show one, and 10 show multiple failures. Having the Over-Literalization failure mode alone is the most frequent combination (17 images). These results quantify the proposed failure modes and their overlap on the evaluated subset.

AuthorsOpenReview ↗

Summary of Author Response, Part 2/2

Clarifications Provided

  • Absolute-score interpretation (pP4B-W3). CS_anchor and CS_resem exceed the corresponding human means by 0.245 and 0.178 on base images; their values should therefore be interpreted comparatively rather than as calibrated human-scale scores. Base-prompt model rankings nevertheless correlate with human rankings at 1.0, 0.9, and 1.0 across the three dimensions.
  • Human-score calculation and corrections (5Ls2-W2). We clarify that only the fourth survey item as in Figure 10 is reverse-coded. We corrected the pooled human anchoring change from -0.070 to -0.057, anchoring sign agreement from 4/5 to 3/5, and total sign agreement from 13/15 to 12/15. The pooled changes retain their direction and statistical significance, and all correlations in Table 10 remain unchanged. In the revision, we will restrict the multiple-comparison statement to the three pooled changes and describe the five-model rank correlations as descriptive.
  • Scope of the tradeoff (fnxz-W3). The strong model-level correlation (-0.92) describes current systems' average behavior, rather than an inherent incompatibility between resemblance and abstraction. Image-level correlations are weaker (-0.35 to -0.41), and 6.7% of base images score at least 0.75 on both dimensions.
  • Rewriting setup and taxonomy coverage (pP4B-W4, W5). We supplied representative rewrites and clarified that Gemma rewriting is an experimental manipulation. We also explicitly acknowledge that the existing human survey did not include an open-ended item for errors outside the three criteria.

Revision Commitments

In the revised manuscript, we will:

  • Add the closed-simile experiment, anchoring ablation, and associated robustness analyses;
  • Incorporate both annotation studies, including sampling procedures, rater instructions, agreement statistics, and sensitivity checks;
  • Expand Appendix C.2 with image-level validation, prompt-level confidence intervals, absolute-score comparisons, and failure co-occurrence tables and an UpSet plot;
  • Correct Figure 10, Table 10, Figure 11, and the affected human-validation descriptions and counts;
  • Make the comparative interpretation of CanonScore and the system-level scope of the tradeoff more explicit; and
  • Include representative rewrites, subject-omission statistics, and complete rewriter instructions.

Again, we wholeheartedly appreciate the reviewers' feedback. Their suggestions have strengthened the benchmark validation and clarified the interpretation of our findings. We hope the additional evidence and corrections address the concerns raised and will be considered in the final evaluation.

Sincerely,

Authors of Submission 27033.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.