acceptodds
Under review as a conference paper at ICLR 2027

PRISM: A Category-Theoretic Framework for Measuring and Refining Multimodal Analogies

Abstract

Analogical reasoning involves identifying and preserving relational structures across domains. However, existing approaches to AI-driven multimodal analogy generation lack an interpretable measure of whether this structure is understood and maintained in the generated output. We address this gap with Pullback Refinement via Interpretable Structural Mapping (PRISM), a modality-agnostic framework for measuring and improving relational alignment in multimodal analogies, evaluated on visual metaphor generation. PRISM represents analogies as explicit relational mappings grounded in category theory and uses VLMs to instantiate these structures across modalities. Its first component, the pullback score, quantifies relational alignment from the resulting graph representation. On the AnaloBench benchmark, selecting the correct analogy purely by pullback score achieves 82.5% accuracy, demonstrating that the score captures meaningful relational information. PRISM’s second component is an iterative refinement loop that uses the pullback score as an in-context feedback signal to iteratively revise the generated image towards greater relational depth. VLM-as-a-judge and human evaluations show that PRISM consistently improves metaphor consistency and analogy appropriateness over zero-shot generation, with human participants preferring the refined output in 57.65% of pairwise comparisons. However, a qualitative analysis reveals that refinement can favour visually crowded compositions rather than genuinely deeper relational correspondences.

Primary area:
generative models
Keywords:
analogical reasoning, generative ai, vision language models, visual metaphors, analogies, ai reasoning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Ad9AOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces PRISM, a category-theoretic framework for representing, measuring, and refining multimodal analogies through explicit relational structure. PRISM extracts source and target domains into structured graphs of entities and relations, replaces entities with semantic roles, and defines a pullback score that measures cross-domain relational alignment using matched relation similarities together with a relational-chain bonus. The pullback score is validated on AnaloBench, where selecting the analogous story using the pullback score achieves 82.5% accuracy, compared with 88.5% for direct zero-shot prompting. The framework is then applied to visual metaphor generation through an iterative refinement loop. At each iteration, PRISM generates an image, extracts source and target relational graphs, computes structural feedback, and revises the generation prompt using the pullback score together with coverage, matched relations, extracted graphs, CLIPScore, the original metaphor, and the previous prompt. The paper evaluates the method on 200 metaphors using multiple image-generation baselines, GEPA-based prompt optimization, ablations, two VLM judges, and a human study with 80 participants. PRISM improves metaphor consistency and analogy appropriateness over zero-shot generation according to both VLM and human evaluation, with participants preferring the refined image in 57.65% of forced-choice comparisons. The paper also analyzes failure modes, including increasingly cluttered outputs and the use of rendered text as a shortcut for increasing the pullback score.

Critical strengths
  1. The paper provides an explicit and interpretable representation of relational structure rather than treating analogy quality as a black-box scalar judgment. PRISM extracts source and target entities and relations, replaces entities with semantic roles, and enforces globally consistent cross-domain relation mappings. This is a meaningful attempt to connect classical structural accounts of analogy with current multimodal foundation models. The fact that removing semantic-role abstraction reduces AnaloBench accuracy from 82.5% to near chance provides useful evidence that the representation is doing nontrivial work.

  2. The evaluation combines strong baseline coverage, human validation, and explicit analysis of PRISM’s failure modes. The authors use multiple image-generation baselines, two independent VLM judges, an 80-participant human study, GEPA comparisons, several ablations, and a second refinement model. They also explicitly report weaknesses of their own metric, including maximalism and the use of rendered text as a shortcut that can raise the pullback score without improving relational quality. This breadth and self-critical analysis strengthen confidence in the empirical findings and help contextualize the reported gains.

Critical weaknesses

The refinement procedure does not use the pullback score in isolation. At each iteration, the refinement model receives a rich feedback set that includes the pullback score, edge coverage, matched relation pairs, the extracted source and target graphs, and CLIPScore, together with the original metaphor and previous prompt. The extracted graphs and matched pairs already provide explicit information about which relations are preserved or missing. As a result, the current experiments do not isolate whether the pullback score itself contributes meaningfully to the observed gains, or whether the improvement is primarily driven by the richer structured feedback. The separate AnaloBench experiment shows that the pullback score contains useful analogical signal, achieving 82.5% accuracy, but this is an indirect validation of its role in visual metaphor refinement. It does not establish that the scalar score provides additional value once the refinement model already receives the extracted graphs, matched relations, coverage information, and CLIPScore. An ablation that removes only the scalar pullback score while keeping all other feedback unchanged would therefore be important for establishing the contribution of the paper's central scoring mechanism.

Questions for authors

Could the authors report an ablation in which the refinement model receives exactly the same feedback as full PRISM except for the scalar pullback score? Would performance remain comparable, or does removing the score materially reduce the gains? Given that AnaloBench validates the score only as a standalone analogy-selection signal, this ablation would help establish whether it provides additional value within the full refinement pipeline.

1 comment

Dear Reviewer Ad9A, thank you for taking the time to read our paper and to provide feedback. We are happy to hear that you saw the value of using PRISM to meaningfully "connect classical structural accounts of analogy with current multimodal foundation models".

Could the authors report an ablation in which the refinement model receives exactly the same feedback as full PRISM except for the scalar pullback score? Would performance remain comparable, or does removing the score materially reduce the gains?

As we understand it, the main concern that you address is that we do "not use the pullback score in isolation". You then suggest that we should "report an ablation in which the refinement model receives exactly the same feedback as full PRISM except for the scalar pullback score".

We have run this ablation. Without the pullback score, most of PRISM's improvement disappears: the overall judge score drops from 7.87 to 6.44, with lower scores on every axis for both judges. In this ablation, the refinement model receives the same feedback as full PRISM (the input metaphor, CLIPScore, the extracted domain graphs and the matched relation pairs), and only the pullback score is removed.

Removing all structural feedback, so that the refinement model receives only the input metaphor and CLIPScore, reaches 6.99. The graphs and matched pairs therefore help only when the pullback score is present to indicate whether, and by how much, relational correspondence is improving. We added the ablations in the paper (Table 2).

Reviewer_tBj7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PRISM, a framework for evaluating and improving visual metaphors. It uses a VLM to extract graphs representing the source and target domains, then computes a pullback score based on their relational alignment. These graphs and scores guide an iterative process that revises the image-generation prompt. Experiments show improvements over zero-shot generation in metaphor consistency and analogy appropriateness, supported by both VLM and human evaluations. Overall, I think it is an interesting paper, proposing a new method. However, it would benefit from some additional experiments and comparisons.

Critical strengths

S1- The paper is well written and easy to understand. The authors provide many qualitative examples, which are convincing in the quality of their implementation (though I think some important evaluation are missing).

S2 - The proposed approach is more interpretable compared to previous methods. The authors also do a good job at explaining it, with qualitative figures making it clear and understandable.

S3 - The empirical results are also quite convincing. Both the selected models improve metaphor and analogy.

Critical weaknesses

W1 - It is possible that the scores can reward shortcuts. Unnormalized relation matching can favor crowded images, while written labels help domain extraction without improving visual analogy. I am particularly concerned because both failure modes appear in Section 6 W2 - The score’s validity as a measure of visual relational quality is not fully convincing. Equation (5) sums matched similarities and rewards longer paths without normalizing for graph size. This can cause bias. The paper itself identifies density bias on AnaloBench and clutter and text shortcuts in image refinement. These affect the central measurement claim. I would say that direct validation against human judgments of relational structure, or controlling for edge count (also text presence and visual complexity) would mitigate this concern

W3 - Comparison with prior metaphor-generation methods is limited. The main evaluation compares PRISM against zero-shot generators and a general prompt optimizer, rather than established visual metaphor-generation. I would prefer if it directly compared against previous approaches (I Spy a Metaphor paper https://aclanthology.org/2023.findings-acl.465/). I notice that appendix G includes chain-of-thought baselines, but their relationship to that method is unclear. A comparison using the same underlying models would better establish PRISM’s added value over existing task-specific approaches, and I think that is an important experiment which should be included in the main body.

Additional Comment - This is not a weakness, but the paper has two formatting issues that I noticed. Figure 5 in page 7 extends the page border at the bottom. And one of the reference in page 15 extends the right margin (line 788-ish).

Questions for authors

Q1. How much improvement remains when only the pullback score is removed from otherwise identical refinement feedback?

Q2. Does the score predict human relational-quality judgments after controlling for extracted graph size and written text?

Q3. Would normalized matching scores or exact matching on these small graphs improve robustness to density and greedy selection?

Q4. Looking at the qualitative examples, I would be particularly interested to see PRISMs performance in visual metonymy. Primarily because metaphor involves cross-domain mappings, but metonymy involves mapping in the same domain, which is more challenging and nuanced. Could PRISM’s graph representation capture these within-domain associations, and what changes would the pullback formulation require?

4 comments

Dear Reviewer tBj7, thank you for taking the time to provide feedback for our paper. We would like to address the weaknesses you mentioned and the questions you raised one at a time.

W1: It is possible that the scores can reward shortcuts.

We discuss both shortcuts, crowded images and embedded text as limitations in Section 6. They each affect the pipeline differently.

Crowding cannot inflate the pullback score, because each domain graph is limited to 3-7 edges regardless of how many elements the image contains. A relation pair counts only if its similarity exceeds θ\theta under a consistent node mapping (see under W2). Adding elements therefore raises the score only if it replaces weaker relations with relations that correspond across domains. Crowding is a stylistic failure not a cheap way to game the score. The value of the pullback score without the opportunity to use any shortcuts is measured using AnaloBench, where the pullback score achieved an 82.5% accuracy without having any opportunity to exploit the pitfalls of crowded images or text. We believe that this shows that it captures relevant information about the relations underlying analogies.

Furthermore, we note that evaluating good visual metaphors is a subjective and difficult task because it is situated in the viewer's personal context [1,2,3,4] (also see the 'On depicted vs inferred relations.' section in our response to Reviewer 1). We tackle this by combining VLM judges with a human evaluation study. In the human study, the participants find that the refined images are better than their zero-shot counterparts along the axes of analogy appropriateness and metaphor consistency. These refinements are something that the participants actually perceive, so we assume that this evaluation goes beyond the pitfalls and shows that even despite there being some crowded visual metaphors and some text included in others, overall the metaphors are improved.

W2, Q2 & Q3: Equation (5) sums matched similarities and rewards longer paths without normalizing for graph size.

Equation (5) intentionally grows with the number of matched relations. Under Structure-Mapping Theory [5], domains sharing more relational structure form stronger analogies (the systematicity principle). The chain bonus rewards matches that form connected relational chains rather than isolated correspondences. Both components are necessary: removing the chain bonus lowers AnaloBench accuracy from 82.5% to 76%.

Graph size is nevertheless controlled by construction, capped to 7 edges (prompt A.3). 86% of refinement rounds reach exactly this cap, and on AnaloBench all four candidates share the same edge count in 189/200 tasks. Therefore, normalizing by graph size would rescale every candidate equally and leave the ranking unchanged. We will add an explicit explanation of this graph size cap to the report.

W3: Comparison with prior metaphor-generation methods is limited.

Prior visual-metaphor pipelines do not measure relational alignment between the source and target domains, which is PRISM's core contribution. They are also built on earlier image generators ([6] uses DALL-E 2; [7] uses Stable Diffusion 3.5 Medium and Janus-Pro-7B). We therefore compare against GEPA, a general-purpose iterative prompt optimiser, using both the pullback score and CLIPScore as objectives.

Nonetheless, following your suggestion we have added a comparison against 'I Spy a Metaphor' [6]. That work released the HAIVMet dataset, which pairs textual metaphors with the visual metaphors produced by its pipeline using DALL-E 2. 75 of the textual metaphors in our evaluation set are taken verbatim from HAIVMet, so we score the released HAIVMet images for these metaphors with our VLM judges:

MCAACI
HAIVMet (DALL·E 2)5.615.295.32
Zero-shot (GPT Image 1.5)6.426.358.12
PRISM7.657.448.30

PRISM scores higher than HAIVMet on all three axes. Comparing PRISM against zero-shot images from the same generator isolates the contribution of refinement: PRISM significantly improves metaphor consistency (+1.23) and analogy appropriateness (+1.09), but not conceptual integration (+0.18, p = 0.67). The gain in conceptual integration therefore comes mainly from the stronger generator, while the gains in metaphor consistency and analogy appropriateness come from PRISM's refinement.

Q1. How much improvement remains when only the pullback score is removed from otherwise identical refinement feedback?

We ran this ablation. Without the pullback score, most of PRISM's improvement disappears: the overall judge score drops from 7.87 to 6.44, with lower scores on every axis for both judges. We have also added this ablation to the paper (Table 2).

Q4. Could PRISM's graph representation capture these within-domain associations [of visual metonymy], and what changes would the pullback formulation require?

Thank you for this question; it sparked an interesting discussion!

The graph representation could capture these associations, but the scoring mechanism would need to change. Structure-Mapping Theory defines analogy as an alignment between two relational structures, and PRISM's extraction requires distinct source and target domains (Prompt A.1). Metonymy, however, operates within a single domain [8]. Thus, PRISM in its current form does not directly apply. A metonymic vehicle and its referent [8,9] could instead be represented in a single graph, linked by metonymic relations (e.g. part-whole, symbol-entity, place-institution) and relevant contextual relations. The pullback formulation would then score the validity of these within-domain associations rather than cross-domain correspondences. Only the scoring step is specific to analogy; the graph representation can be extended to metonymy.

The ethics statement does not discuss the risk that generated visual metaphors may reinforce harmful stereotypes.

We extended the ethics statement to address this risk. Cross-domain mappings can produce harmful associations, particularly when the target is a social group or culturally sensitive concept: source domains may carry stereotypical properties [11], which text-to-image models can amplify [12]. Safety filters targeting explicit or violent content are unlikely to catch such harm, as it emerges from combinations of individually benign elements [13]. PRISM offers a mitigation point unavailable in end-to-end generation: because source and target domains and their mapped relations are explicit before image generation, harmful mappings could be detected at the domain-graph level rather than in the final image. The revised statement also addresses the deliberate misuse of visual metaphors for persuasion and misinformation [10].

We also corrected the formatting issues. Thank you for pointing them out.

[1] Forceville, C. (1994). Pictorial metaphor in advertisements. Metaphor and symbol, 9(1), 1-29.
[2] Refaie, E. E. (2003). Understanding visual metaphor: The example of newspaper cartoons. Visual communication, 2(1), 75-95.
[3] El Refaie, E. (2009). Multiliteracies: How readers interpret political cartoons. Visual communication, 8(2), 181-205.
[4] Kövecses, Z. (2015). Where metaphors come from: Reconsidering context in metaphor. Oxford University Press.
[5] Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2), 155-170.
[6] Chakrabarty, T., Saakyan, A., Winn, O., Panagopoulou, A., Yang, Y., Apidianaki, M., & Muresan, S. (2023, July). I spy a metaphor: Large language models and diffusion models co-create visual metaphors. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 7370-7388).
[7] Koushik, G. A., Nazarieh, F., Birch, K., Qian, S., & Kanojia, D. (2025). The Mind's Eye: A Multi-Faceted Reward Framework for Guiding Visual Metaphor Generation. arXiv preprint arXiv:2508.18569.
[8] Kövecses, Z., & Radden, G. (1998). Metonymy: Developing a cognitive linguistic view. Cognitive Linguistics, 9(1), 37–77.
[9] Panther, K. U., & Radden, G. (Eds.). (1999). Metonymy in language and thought (Vol. 4). John Benjamins Publishing.
[10] Messaris, P. (1998). Visual persuasion: The role of images in advertising. Sage.
[11] Ana, O. S. (1999). 'Like an animal I was treated': Anti-immigrant metaphor in US public discourse. Discourse & society, 10(2), 191-224.
[12] Bianchi, F., Kalluri, P., Durmus, E., Ladhak, F., Cheng, M., Nozza, D., ... & Caliskan, A. (2023, June). Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM conference on fairness, accountability, and transparency (pp. 1493-1504).
[13] Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020). The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33, 2611-2624.

Thank you for your detailed response.

I agree that prior visual-metaphor pipelines do not measure relational alignment between the source and target domains, however I would encourage adding a small section in the appendix discussing the results you presented in response to W3.

I am partially convinced by the author's response and I have raised by score from 2 (Weak Rejection) to 3 (Weak Acceptance).

Thank you for raising the score! We've updated the the pdf and added table 3 as a comparison with HAIVmet in the main paper.

I am partially convinced...

Could you please let us know which parts remain unconvincing, as we are still in the discussion period it'd be useful to know if there is something we could do to make our submission stronger.

Best,

The Authors

Reviewer_BVdQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

PRISM extracts two domain graphs from a visual metaphor, scores semantic edge alignment with node consistency and a chain bonus, and combines structural diagnostics with CLIPScore to refine image prompts. It evaluates analogy selection, visual generation, and human judgments.

Critical strengths

Explicit correspondences and missing relations make the refinement feedback inspectable.

Critical weaknesses
  1. In the provided code, the 20-item ablation subset reproduces PRISM's 7.87 and all six component means. However, the two-judge GEPA-CLIP, GEPA-pullback, and one-shot files yield 7.85, 7.28, and 7.60, versus 6.76, 6.62, and 6.47 in the table. GEPA-CLIP contains 19 items; on the shared 19, its mean is 7.8509 versus PRISM's 7.8377, which does not reproduce the claimed clear advantage. Please identify the experimental/evaluation versions, item IDs, subset selection, and exclusions for each row, and release the corresponding files.
  2. The extraction validation does not establish visual grounding. Prompt A.3 grounds source relations in the depicted image but permits target relations that hold in the abstract concept, so matches can partly reflect commonsense completion. SCAR tests graph extraction for given domains, rather than end-to-end domain identification and visible relations in generated images; the 82.5% AnaloBench result is textual. Generated-image annotations should distinguish depicted from inferred relations and test whether score changes track human-assessed relational improvements.
  3. Equation 5 sums similarities, making the score sensitive to the number of matched edges despite its chain bonus. In Table 8, increasing Sonnet's limit from 5 to 10 rounds at ε=0.05 raises the score from 3.6607 to 4.1000, but MC rises by only 0.0266 and the AA/CI means remain unchanged. These means do not establish that a higher structural score improves visual analogy. Appendix C and Figure 7 already discuss dense graphs and shortcuts; validation controlling for graph size is still needed to justify using this score to guide and stop refinement.
  4. The ablations do not isolate the structural mechanism. Removing domain guidance also removes the intended metaphor and CLIPScore, while one-shot extraction changes graph quality. Under matched models, generation budgets, and other feedback, removing the structural score and match/unmatched-relation feedback would distinguish their contribution from generic iterative refinement.
Questions for authors

Pls refer to the weaknesses above.

7 comments

Dear Reviewer BVdQ, thank you for reviewing our paper and providing feedback. We address the weaknesses raised below.

The two-judge GEPA-CLIP, GEPA-pullback, and one-shot files yield 7.85, 7.28, and 7.60, versus 6.76, 6.62, and 6.47 in the table.

The anonymous Github files contained results from an initial ablation run with only 20 items. We subsequently reran the ablations at a larger scale, producing the results reported in the table. We have now updated the Github accordingly. Thank you for identifying this inconsistency.

Generated-image annotations should distinguish depicted from inferred relations and test whether score changes track human-assessed relational improvements.

On depicted vs inferred relations. Our extraction mechanism deliberately draws on knowledge beyond what is depicted. Viewers of visual metaphors rely on cultural and contextual knowledge for sensemaking [2,3,4]. Their existing knowledge of depicted entities also shapes interpretation [5], making metaphor interpretation subjective, as reflected in our human inter-rater agreement results.

On tracking human judgements. Human agreement is low for absolute scores (Krippendorff's alpha=0.082 for analogy appropriateness, 0.111 for metaphor consistency, and 0.233 for forced-choice judgements). This makes it difficult to establish whether score changes track human assessments. Instead, we validate the signal captured by the pullback score through the AnaloBench baseline, where it achieves 82.5% accuracy in identifying an analogous story among four multiple-choice options.

In Table 8, increasing Sonnet's limit from 5 to 10 rounds at ε=0.05 raises the score from 3.6607 to 4.1000, but MC rises by only 0.0266 and the AA/CI means remain unchanged. These means do not establish that a higher structural score improves visual analogy.

Table 8 calibrates the stopping criterion on a 25-metaphor held-out set; it does not test whether higher pullback scores yield better visual analogies. We address this question directly through our ablations (see below).

Validation controlling for graph size is still needed to justify using this score to guide and stop refinement.

The score is designed to increase with matched relations. Under structure-mapping theory [1], domains sharing more relational structure form stronger analogies (systematicity principle). The chain bonus favours connected relational chains over isolated correspondences; removing it reduces AnaloBench accuracy from 82.5% to 76%.

Graph size is controlled in three ways:

  1. Threshold. Relations are matched only when their cosine similarity exceeds θ\theta, and accepted pairs must preserve a consistent node mapping. Additional relations therefore increase the score only when they genuinely correspond across domains.
  2. Bounded size. Prompt A.3 constrains each domain graph to 3–7 edges. In practice, 86% of refinement rounds contain exactly 7 target-domain edges, while all four AnaloBench candidates share the same edge count in 189 of 200 tasks.
  3. Constant size during refinement. Graph size remains effectively fixed across rounds. Score changes driving feedback and the epsilon-based stopping criterion therefore reflect changes in matched relations rather than graph size.

Removing the structural score and match/unmatched-relation feedback would distinguish their contribution from generic iterative refinement.

We ran two additional ablations. Without the pullback score, most of PRISM's improvement disappears: the overall judge score drops from 7.87 to 6.44, with lower scores on every axis for both judges. Here, the refinement model receives the same feedback as full PRISM (input metaphor, CLIPScore, extracted domain graphs and matched relation pairs), but without the pullback score. Removing all structural feedback, leaving only the input metaphor and CLIPScore, yields 6.99. Thus, graphs and matched pairs help only when accompanied by the pullback score, which indicates whether and by how much relational correspondence improves. We added these ablations to the paper (Table 2).

[1] Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2), 155-170.
[2] Forceville, C. (1994). Pictorial metaphor in advertisements. Metaphor and symbol, 9(1), 1-29.
[3] Refaie, E. E. (2003). Understanding visual metaphor: The example of newspaper cartoons. Visual communication, 2(1), 75-95.
[4] El Refaie, E. (2009). Multiliteracies: How readers interpret political cartoons. Visual communication, 8(2), 181-205.
[5] Kövecses, Z. (2015). Where metaphors come from: Reconsidering context in metaphor. Oxford University Press.

Thank you for the clarifications and additional experiments, which largely address my concerns. Overall, I believe the paper would benefit from further refinement in quality.

Thank you for your response, we are happy to hear that our clarifications and additional experiments addressed your concerns. We are keen to continue improving the paper, and would be grateful if you could specify the aspects of quality that you think can still be improved, so that we can address them within the remaining discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.