acceptodds
Under review as a conference paper at ICLR 2027

MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learning

Abstract

Understanding metaphors in images remains difficult for current AI systems. Multimodal Large Language Models (MLLMs) answer basic Visual Question Answering (VQA) questions well but often miss the cultural, emotional, and contextual implications of visual content, which require multi-hop reasoning, cultural context, and Theory of Mind (ToM). Reinforcement learning (RL) is a natural tool for eliciting such reasoning, but the target output of the task, an interpretation, cannot be verified automatically. We propose MetaphorStar, an end-to-end visual RL framework for image implication that addresses this by decomposing the reference interpretation of each image into a set of True-False propositions, each of which yields a binary, verifiable reward. The framework includes three components: the fine-grained dataset MetaphorQA, the visual RL method MetaphorGRPO, and the benchmark MetaphorBench. Trained with rewards computed only from these propositions and a format check, the MetaphorStar family (3B/7B/32B) raises the item-level True-False accuracy of the 7B base model from 28% to 70% and transfers to Multiple-Choice and Open-Style questions that are never seen during training; among 32 open and proprietary MLLMs, including the newest GPT-5.6-Sol, Gemini-3.1-Pro, and Qwen3.8-Max, MetaphorStar-32B obtains the highest True-False and Open-Style scores, while the strongest proprietary models remain ahead on Multiple-Choice questions. We further find that, in our setting, supervised warmup on distilled rationales lowers performance and collapses policy entropy, that the LLM judge commonly used for open-style evaluation ranks verbose SFT-trained models above more accurate RL models, and that the trained models also score above their base models on general visual reasoning benchmarks. We analyze parameter scaling, data scaling, reward weighting, and different backbones including Qwen3-VL and LLaVA. All models, data, and code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Metaphor, Vision-language Reasoning, Vision Language Model, Visual Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZgGoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

MetaphorStar uses RL to train vision-language models to understand the hidden meaning of metaphorical images, such as comics and memes. Each image's human-written interpretation is turned into about ten true/false statements, which VLMs generate and the authors check by hand. Training is plain GRPO without an SFT stage, with a reward for the correct answer and the output format. Trained on 984 statements from 100 images, the 7B model goes from 28% to 70% on a 50-item true/false benchmark, where an item counts only if all its statements are correct. It also improves on multiple-choice and open-answer questions it never trained on.

Critical strengths
  1. The core idea is simple and useful. Breaking an interpretation into true/false statements gives a subjective task checkable rewards, with no judge needed during training. The same recipe could apply to humor, satire, or advertisements.
  2. The gains are large and data-efficient. With only 984 statements, the 7B model goes from 28% to 70% on true/false items and from 46% to 74% on multiple-choice questions. All five backbones improve, including LLaVA-1.5 and Qwen3-VL-8B.
  3. The evaluation is careful and honest. Always answering "true" scores 0%, test images are checked for overlap with training images, and paired significance tests are reported (Table 11, p. 17).
Critical weaknesses
  1. Two controls are missing. The paper argues that true/false statements are a better reward than multiple-choice questions (Table 1), but it never trains with multiple-choice rewards to test this. Some false statements also contradict themselves (Table 16, p. 22), so a model could spot them without the image, and no text-only baseline is reported.
  2. The "SFT Curse" claim rests on one small setup. SFT uses only 984 examples from a single teacher model, and it drops multiple-choice accuracy below the base model (46% to 28%, Table 8). This may reflect the small setup rather than SFT in general.
  3. The test set is a little small. All main results use 50 items from 44 images, so one item is worth 2 points.
Questions for authors
  1. What are the true/false scores of MetaphorStar-7B and MetaphorStar-32B on the 200 extra held-out items in Table 12?
  2. How does training on multiple-choice questions for the same images compare with training on true/false statements?
  3. Clarification: Why does the Qwen2.5-VL-32B base model score lower on MMMU (33.6) than the 7B (42.0) and 3B (42.3) models in Table 6? I ask because the original Qwen2.5-VL report shows MMMU rising with model size, and this one number accounts for over half of the 32B model's reasoning gain.
3 comments
AuthorsOpenReview ↗

Response to Reviewer ZgGo

We thank Reviewer ZgGo for noting that checkable True-False rewards need no judge, that the gains are large and data-efficient across all five backbones, and that the evaluation is careful and honest.

New results are in the General Response above: Table A (complete held-out pool, W3/Q1), Tables B and D (image controls, W1), Table C (MCQ-GRPO, W1/Q2), Table E (SFT ablations, W2), and the three-annotator label check (Comment 2/2). We answer each point below.

W3 & Q1: MetaphorStar-7B and MetaphorStar-32B on the complete held-out pool. On the complete 1,236-entry held-out pool (12,165 TFQs, Table A), MetaphorStar-7B reaches 71.9% strict item accuracy (vs. 39.4% base, +32.5) and MetaphorStar-32B reaches 76.1% (vs. 52.3% base, +23.7, p=1.2×10−56p = 1.2 \times 10^{-56}). MetaphorStar-3B rises from 22.5% to 64.0% (+41.5, p=1×10−117p = 1 \times 10^{-117}), and proposition accuracy reaches 95.0%, 96.0% and 96.8%. This is the whole pool that Table 12 samples from, so test-set size no longer limits the base-versus-trained comparison. We will add Table A to Appendix B and Section 5.1.

W1 (Part 1) & Q2: Training on multiple-choice questions vs. True-False propositions. Training with MCQ rewards on the same 100 images does not improve TFQ accuracy, with or without matched compute (Table C). On the 1,236 held-out entries, MCQ-GRPO gets 40.8% strict against 39.4% for the base model, and a compute-matched run gets 40.4%. TFQ-GRPO reaches 71.9%, +31.5 over the matched run. One MCQ per image gives one reward signal, while ~9.8 TFQs per image each check a separate claim.

W1 (Part 2): Self-contradictory false statements and text-only / shuffled-image controls. You are right that some false statements in Table 16 (e.g., entry E6) are internally inconsistent; they are still correctly labeled False, and we will mark them in Table 16. Three annotators independently relabeled all 492 MetaphorBench TFQs: mean pairwise agreement is at least 98% and no label changed (Comment 2/2). The controls you ask for are in Tables B and D:

  1. Item-strict accuracy: On all 1,236 entries, every model in Table B scores at most 10.4% without the image and at most 2.0% under shuffled images, against 64.0%–76.1% for the trained models with it.
  2. Proposition accuracy: Training also raises text-only accuracy (7B: 51.6% →\rightarrow 64.4%), but it stays far below the 96.0% reached with the image (+31.6 proposition points from the correct image).

We will add Table B to Appendix B and Section 5.1 in the revised version.

W2: Scope of the SFT warmup finding ("SFT Curse"). Table 8 does test one setup, so we varied the schedule, the learning rate and the entropy bonus (Table E). On the 46% to 28% MCQ drop, we re-ran both SFT checkpoints and it persists under a lenient parser (see our reply to c5oT, Q3). A lower learning rate does recover ground (lr 1×10−51 \times 10^{-5} reaches 51.0 on a 200-entry sample against 32.0 for the Table 8 setup), and the entropy bonus reaches 66.5 against 63.0, but none of the five variants reaches direct RL at 73.0. Policy entropy drops from 1.33 (base) and 1.23 (direct RL) to 0.30 and 0.29 after the SFT arms (Figure 6). On data size the two arms of Table 8 are matched: MetaphorQA-Lite-SFT holds one trajectory per proposition of the same 984-item MetaphorQA-Lite that MetaphorGRPO trains on, so sample size is not the difference. RL does not win by seeing more data: with 0.1k propositions, a tenth of what the SFT arm saw, direct RL already reaches 48% TFQ and 64% MCQ against SFT's 42% and 28% (Table 7), and TFQ grows monotonically to 84% at 13.6k. We distilled from one teacher, and the revised version will say so. Section 1 already scopes the term to our setting, and both warmup-then-RL runs still fall short of direct RL (63.0 and 66.5 against 73.0), so we keep it. We will add the SFT hyperparameters and Table E to Appendix D.

Q3: Qwen2.5-VL-32B base score on MMMU (33.6) in Table 6. The MMMU column is the official MMMU test split (10,500 questions) scored by the evaluation server; all three base numbers are official submissions: 42.3 (3B), 42.0 (7B), 33.6 (32B). The Qwen2.5-VL report gives MMMU val, so the two are not comparable. The 32B base falls furthest behind in one category: Humanities & Social Science 24.3% against 64.2% (7B) and 65.8% (3B), which alone accounts for 3.6 of the 8.4-point gap to 7B. The scores themselves need no correction. MMMU is the largest single contributor, as you note, and the gain is not confined to it: the other nine reasoning scores sum to +12.4 for 32B, with 8 of the 9 improving. We will label the split in Table 6.

If you have any further questions, please feel free to let us know.

Thank you for the response and the new experiments. The MCQ-reward baseline, the no-image controls, and the full held-out pool address most of my concerns. Please add them to the main paper.

One point remains. The 32B base model scores 24.3% on MMMU Humanities & Social Science, against about 65% for the 3B and 7B models. This looks like a parsing problem, not the model's real ability. Please check these outputs or report the reasoning gains without MMMU.

I keep my rating of weak acceptance.

AuthorsOpenReview ↗

Response to Reviewer ZgGo's follow-up

We thank Reviewer ZgGo for the quick follow-up and for confirming that the MCQ-reward baseline, the no-image controls and the complete held-out pool resolve those concerns.

All three are now in the main paper. The revised PDF marks every addition in blue:

  • Complete image-disjoint held-out pool, 1,236 entries and 12,165 True-False questions: Table 7 in Section 5.1.
  • MCQ reward with matched data and compute: Table 10 in Section 6.2.
  • No-image and deranged-image controls: Figure 3(b) in Section 5.2, with the full three-scale table in Appendix B.3 (Table 15); the prompt control and the image-free training run are in Appendix B.4 (Table 16).

MMMU Humanities & Social Science. We went back to the submission behind that number. The MMMU column of Table 6 is the official test split, which has no public labels: the 10,500 predictions per model are submitted to the official MMMU evaluation server on EvalAI, and the scores are the ones that server returns. The category figures we quoted in our first response are that server's own breakdown of the same submissions, not a re-scoring of our own. Here is the complete official result for Qwen2.5-VL-32B, with the 7B and 3B submissions alongside:

MMMU test categoryQuestionsQwen2.5-VL-32BQwen2.5-VL-7BQwen2.5-VL-3B
Art & Design1,16343.357.857.2
Business1,42851.149.442.2
Science2,42629.132.635.3
Health & Medicine1,75236.446.847.9
Humanities & Social Science94724.364.265.8
Tech & Engineering2,78425.929.330.9
Overall10,50033.642.042.3

Weighting the six categories by their official sizes returns 33.60, 42.02 and 42.35, which is exactly the 33.6 / 42.0 / 42.3 printed in Table 6, so the breakdown and the headline numbers come from one and the same submission.

That submission does not show a uniformly degraded 32B. It is the strongest of the three models on Business, and it trails the other two by 3 to 6 points on Science and Tech & Engineering, the two largest categories at 2,426 and 2,784 questions, together half the split. A pipeline-wide extraction failure for the 32B would not leave one category leading and the two largest ones intact. Every model in Table 6 was run through one VLMEvalKit configuration, with the same prompt, the same decoding settings and the same answer-extraction rule for base and trained checkpoints, so the comparison within the table is controlled. We report the scores as the official server returned them.

The reasoning conclusion does not depend on MMMU. Table 6 reports ten reasoning scores from nine benchmarks. Setting MMMU aside, the remaining nine scores give a mean change of +1.4 for 32B, with 8 of the 9 improving, +2.6 for 7B and +0.5 for 3B, against the +2.9 / +3.1 / +0.7 that the full set gives. The direction and the ordering across scales are unchanged. MMMU is the largest single contributor for the 32B, as you note, but it is not what carries the result.

If you have any further questions, please feel free to let us know.

Reviewer_V9dZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an end-to-end visual RL framework for image metaphor understanding. It converts subjective image interpretations into labeled True False questions to provide deterministic rewards for GRPO training. It presents MetaphorQA, MetaphorGRPO, and MetaphorBench. Experiments show improvements across TFQ, MCQ, and OSQs, and suggest that SFT warmup can hurt RL performance.

Critical strengths
  1. interesting and challenging research topic. Understanding visual metaphors and abstract image implications is an important yet underexplored challenge for MLLMs, requiring reasoning beyond literal visual perception.

  2. Interesting SFT versus RL analysis. The comparison between direct RL and SFT warmup reveals substantial performance differences and raises interesting questions about policy entropy and exploration.

Critical weaknesses
  1. Limited methodological novelty and missing RL baselines. The method primarily combines proposition-based supervision with standard GRPO. A direct MCQ-based GRPO baseline with matched training data and compute is essential to establish the advantage of TFQ-based rewards over simpler alternatives.

  2. Questionable reliability of TFQ ground truth. TFQ labels are derived from subjective image interpretations and may contain ambiguity or bias. Although the authors report manual verification, they provide insufficient details on independent human validation or annotation agreement. A single incorrect or ambiguous proposition can invalidate an entire test item under the strict accuracy metric.

  3. Insufficient evidence for improved metaphor understanding. TFQs mix literal visual facts, cultural knowledge, and abstract implications. Without separating these categories or including language-only/image-shuffling controls, it is difficult to determine whether the gains reflect deeper metaphor understanding rather than general proposition verification or linguistic shortcuts.

  4. Incomplete prompting baselines and outdated primary backbones. The few-shot CoT baselines lack essential implementation details, and few-shot evaluations on stronger proprietary models are missing. The main model family also relies on Qwen2.5-VL, despite Qwen3-VL-8B + MetaphorGRPO already matching MetaphorStar-32B on TFQ and MCQ.

Questions for authors

why did you choose to finetune on qwen2.5-vl instead of qwen3-vl? it’s a 2026 paper for 2027 conference, and qwen3-vl released in Oct 2025

2 comments
AuthorsOpenReview ↗

Response to Reviewer V9dZ

We thank Reviewer V9dZ for noting that visual metaphor understanding is important and underexplored, and that our SFT versus RL analysis raises interesting questions about entropy and exploration.

New results are in the General Response above: Table C (MCQ-GRPO baseline, W1), Table A and the three-annotator label check (W2), Tables B and D (image controls, W3), Table G (outside benchmarks, W3), and Table F (few-shot CoT, W4). We answer each point below.

W1: Methodological framing and direct MCQ-based GRPO baseline. The contribution is the reward representation, not a new RL objective: a subjective interpretation becomes binary propositions carrying reference labels, so the reward is exact-match with no LLM judge in the loop. Section 3.3 states that the GRPO objective itself is unchanged. That is what makes RL applicable to a task whose answer is an open paragraph, and Table C isolates the reward format: with the same images, base model and compute, MCQ rewards leave TFQ accuracy where the base model already was. The second contribution is the end-to-end recipe, where Table E and Figure 6 show that distilled-SFT warmup collapses policy entropy and no variant we tried reaches direct RL. On the same images and base model, MCQ-GRPO at the same 75 updates and rollouts scores 40.4% strict TFQ on the 1,236-entry pool, against 39.4% base. TFQ-GRPO reaches 71.9%, +31.5 over the matched run. MCQ rewards do not help on MCQ either: 24.0% and 40.0% against 46.0% base and 74.0% for TFQ-GRPO (Table 8).

W2: Reliability of TFQ ground truth and human annotation agreement. Three PhD-level annotators independently annotated all 492 MetaphorBench TFQs against the images (Comment 2/2).

  1. Inter-annotator agreement: Mean pairwise agreement across all 492 propositions is at least 98%. All disagreements fell within one item and were resolved 2-vs-1 in favor of the released label, so no label changed. All 50-item MetaphorBench scores rest on human-verified labels.
  2. Proposition accuracy alongside strict accuracy: On the complete 1,236-entry held-out pool (12,165 TFQs, General Response, Table A), MetaphorGRPO raises both strict item accuracy (22.5% →\rightarrow 64.0%, 39.4% →\rightarrow 71.9%, and 52.3% →\rightarrow 76.1% at 3B/7B/32B) and proposition accuracy (84.7% →\rightarrow 95.0%, 88.8% →\rightarrow 96.0%, and 92.4% →\rightarrow 96.8%). We will add the human validation results and proposition accuracies to Section 3.2 and Appendix B in the revised version.

W3: Language-only / image-shuffling controls and literal vs. implication propositions.

  1. No-image and shuffled-image controls (General Response, Table B): On all 1,236 held-out entries, every model falls to at most 10.4% item-strict accuracy without the image and at most 2.0% under shuffled images, where proposition accuracy drops to 49.7%–58.9% against a 52.9% always-True rate, so strict item gains require the correct image. On the full pool the correct image adds +31.6 proposition points for MetaphorStar-7B. On the balanced GoFigure split (Table G), where chance is 50.0, MetaphorStar-7B scores 80.8 with the image, 62.6 without it and 54.4 with a deranged image.
  2. Separating the categories: GPT-6.1-Sol labeled every TFQ of a 200-entry sample by what decides it. 92.6% turn on the image's implication, 4.1% on cultural knowledge, 2.1% on what is visibly depicted, 1.2% unlabeled. Against the prompt-matched base, MetaphorGRPO gains +6.0 proposition points on the implication statements (90.3% →\rightarrow 96.3%). On High-Level Bench-EN MCQ, which asks about the implication, it improves all five backbones by +16 to +28 (Table 5). We will add the category breakdown to Appendix B.

W4 & Q1: Finetuning on Qwen2.5-VL vs. Qwen3-VL, and few-shot CoT details.

  1. Qwen3-VL vs. Qwen2.5-VL: As you note, Qwen3-VL-8B + MetaphorGRPO matches MetaphorStar-32B on TFQ (74%, +30 over base 44%) and MCQ (78%, +16 over base 62%), and raises OSQ from 3.12 to 3.64 (Tables 5 and 11). We began the scaling study on Qwen2.5-VL, which offers a 3B/7B/32B family (Section 6.1, Figure 5). We will highlight Qwen3-VL-8B + MetaphorGRPO in the Abstract, Introduction, and Section 5.1 of the revised version.
  2. Few-shot CoT: Table 8 used n=3n = 3 fixed MetaphorQA-Lite demonstrations in the describe →\rightarrow analyze →\rightarrow answer format. We ran the same three conditions on Qwen3.8-Max, the strongest proprietary model in Table 5, and on two models outside it (Table F). Few-shot CoT costs Qwen3.8-Max 20.0 points of strict TFQ (70.0% to 50.0%), GPT-6.1-Sol 16.0 (74.0% to 58.0%) and GPT-5.2 4.0 (60.0% to 56.0%), and it does not raise GPT-6.1-Sol's MCQ either, so it does not help these stronger models. We will add Table F and the verbatim prompts to Appendix C.

If you have any further questions, please feel free to let us know.

well addressed.

Reviewer_c5oTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets the metaphor understanding abilities of the VLMs. Specifically, the paper finds that current VLMs are not good at understanding metaphors and presents the MetaphorBench benchmark to support this. To improve the performance of VLMs, the paper proposes MetaphorQA (a training set) and MetaphorGRPO to train the VLMs to understand metaphors. Overall, experiments suggest that many SoTA models suffer from this, and GRPO seems to improve the performance.

Critical strengths
  • The paper motivates the right problem and presents the benchmark to support the claims and present the GRPO-based recipe to improve performance. This gives a complete view of the problem and a solution to overcome it.
  • Extensive experiments are conducted, which consistently show that MetaphorGRPO outperforms the baselines.
  • Extensive ablations across the model family show consistent improvements.
Critical weaknesses
  • Benchmark tasks are too small. The 50-item benchmark can’t support the claims against other models. One item is 2 percentage points.
  • The training TFQs, the MCQs and the OSQ references all come from II-Bench’s human implications, written by the same annotation pipeline with the same interpretive conventions. Gains on MCQ and OSQ may reflect learning II-Bench’s style of interpretation rather than general metaphor understanding. There is no evaluation on independent sources.
  • Evaluation protocols differ between models. MetaphorStar answers with its trained model via the describe --> analyze --> answer template, while baselines use a “baseline prompt”.
Questions for authors
  • What do the base and trained models score on MetaphorBench TFQs with the image removed (text only)? How much of MetaphorGRPO’s gain remains if training also removes images?
  • Can you evaluate all MetaphorStar models (and the base models) on the 200 further held-out items in Table 12, or on the full held-out pool?
  • What are the SFT hyperparameters? Why do both SFT variants get exactly 28% MCQ: is that a parsing or format failure? Does the effect hold with shorter SFT, lower learning rates, or RL after SFT with an entropy bonus?
1 comment
AuthorsOpenReview ↗

Response to Reviewer c5oT

We thank Reviewer c5oT for noting that the problem is well motivated, that the benchmark plus GRPO recipe gives a complete view, and that the ablations across the model family consistently show gains.

New results are in the General Response above: Table A (complete held-out pool, W1/Q2), Table B (image controls, Q1), Table D (text-only training and the prompt control, Q1/W3), Table E (SFT ablations, Q3), and Table G (outside benchmarks, W2). We answer each point below.

W1 & Q2: Benchmark size (50 items) and evaluating all models on the complete held-out pool. On MetaphorBench 1 item is 2 points, so small gaps between top models are within noise (Table 11). We took the second option in your Question 2 and evaluated all three MetaphorStar models and their base models on the complete held-out pool: 1,236 entries, 12,165 TFQs (Table A). MetaphorGRPO improves all three sizes: 3B rises from 22.5% to 64.0% (+41.5, p=1×10−117p = 1 \times 10^{-117}), 7B from 39.4% to 71.9% (+32.5), and 32B from 52.3% to 76.1% (+23.7, p=1.2×10−56p = 1.2 \times 10^{-56}). We will add Table A to Appendix B and narrow the ranking wording to "highest among the Table 5 models, within sampling noise of the strongest proprietary model there on both TFQ and OSQ".

Q1: Text-only and shuffled-image scores, and training without images. Without the image, strict accuracy is at most 10.4% for every model in Table B, and at most 2.0% under shuffled images. Scored with the image, the image-free variant reaches 50.4% against 39.4% for the base model and 71.9% for MetaphorGRPO, so training without images keeps 11.0 of the 32.5 points and gives up 21.5. On the balanced GoFigure split, where chance is 50.0, removing the image at test time costs MetaphorStar-7B 18.2 points (Table G). We will add Table D to Section 6.4.

W2: Shared II-Bench source conventions. Our TFQs, MCQs, and OSQ references do all come from II-Bench. We therefore evaluated on two outside sources (Table G). GoFigure is English and has no authors in common with II-Bench; its 2,556 true/false claims are balanced, so chance is 50.0, and MetaphorStar-7B reaches 80.8% against 73.0% for the base model (+7.8). CII-Bench is Chinese, its images collected from the Chinese internet and its answers hand-written in Chinese; on all 800 of its questions MetaphorStar-7B reaches 51.4% against 42.5% (+8.9 [+5.7, +12.0], McNemar p=4.7×10−8p = 4.7 \times 10^{-8}). The other English visual-metaphor set we considered, MetaCLUE, does not release its annotations. Training is entirely on English II-Bench propositions, so gains on an unrelated English corpus and on Chinese questions are cross-source and cross-language transfer. It sits alongside the format transfer in the paper: the model is trained only on TFQs yet improves MCQ and OSQ, which it never sees (Table 5). We will add GoFigure and CII-Bench to Section 5.

W3: Evaluation prompt comparability. The MetaphorStar prompt does not lift the base model on TFQ or on MCQ, so neither comparison depends on it. On the complete 1,236-entry held-out pool (General Response, Table D) the prompt makes no detectable difference to the base model: 39.4% to 39.8% strict TFQ accuracy with the describe →\rightarrow analyze →\rightarrow answer prompt (+0.4 [-2.4, +3.3], p=0.83p = 0.83), and MetaphorGRPO is +32.1 above it. Table 8 already applies it to the base models as "+ Standard CoT", where TFQ falls (7B 28% to 26%, 32B 56% to 50%) and MCQ does not rise (46% to 44%, 62% unchanged). We will add the matched-prompt row and the verbatim prompts to Section 6.4 and Appendix C.

Q3: SFT hyperparameters, the two 28% MCQ scores, and SFT schedule / entropy ablations.

  1. SFT hyperparameters: Full fine-tune of Qwen2.5-VL-7B on the 984 MetaphorQA-Lite-SFT trajectories, vision tower frozen, lr 5×10−55 \times 10^{-5}, cosine, warmup 0.1, 5 epochs, per-device batch 2, grad accum 2, bf16, ZeRO-3, loss on prompt tokens, then MetaphorGRPO (Table 13).
  2. The two 28% MCQ scores: Format is not the whole story. We re-ran both SFT checkpoints on the same 50 questions. They commit to no option on 36% (SFT) and 18% (SFT & MetaphorGRPO) of items, the behavior described in Section 6.4 where SFT models list several readings instead of choosing one. Re-scored with the lenient parser, both still fall well below the base model. We will add this parse audit to Appendix D.
  3. Shorter SFT, lower LR, entropy bonus: On the 200-entry sample of Table E a shorter schedule is worse, a lower learning rate reaches 51.0 against 32.0, and an entropy bonus after SFT reaches 66.5 against 63.0, yet no variant overtakes direct MetaphorGRPO at 73.0. We will add Table E and the hyperparameters to Appendix D.

If you have any further questions, please feel free to let us know.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank all reviewers for their constructive feedback. Reviewers noted that:

  • visual metaphor understanding is important and underexplored, and the benchmark plus GRPO recipe gives a complete view of the problem and a solution (V9dZ, c5oT);
  • breaking interpretations into True-False statements gives checkable rewards without a judge, and the recipe could extend to humor, satire, or ads (ZgGo);
  • the gains are large and data-efficient, with all five backbones improving (ZgGo, c5oT);
  • the evaluation is careful and honest, and the SFT versus RL analysis is interesting (ZgGo, V9dZ).

Table A: Evaluation on the Complete Held-Out Pool (ZgGo W3/Q1, c5oT W1/Q2). All MetaphorStar models and their base models on the complete image-disjoint pool that Table 12 samples from: 1,236 entries, 12,165 TFQs. One TFQ per prompt, temperature 0.5:

Pair (1,236 Held-Out Entries)Strict (%)Strict Δ\DeltaProp (%)
3B →\rightarrow MetaphorStar-3B22.5 →\rightarrow 64.0+41.584.7 →\rightarrow 95.0
7B →\rightarrow MetaphorStar-7B39.4 →\rightarrow 71.9+32.588.8 →\rightarrow 96.0
32B →\rightarrow MetaphorStar-32B52.3 →\rightarrow 76.1+23.792.4 →\rightarrow 96.8

The 3B and 32B gains are significant at p≤1.2×10−56p \le 1.2 \times 10^{-56}. Throughout, pp is an exact two-sided McNemar test and brackets are image-clustered 95% CIs.

Table B: No-Image and Shuffled-Image Controls (ZgGo W1, V9dZ W3, c5oT Q1). Base and trained models on the same 1,236 entries, no-image and shuffled-image (fixed derangement, so no entry sees its own image). Strict / proposition accuracy (%):

Pair (1,236 entries)With imageNo imageShuffled image
3B →\rightarrow MetaphorStar-3B22.5 / 84.7 →\rightarrow 64.0 / 95.00.5 / 61.9 →\rightarrow 5.3 / 75.31.0 / 57.2 →\rightarrow 2.0 / 58.9
7B →\rightarrow MetaphorStar-7B39.4 / 88.8 →\rightarrow 71.9 / 96.00.0 / 51.6 →\rightarrow 0.6 / 64.40.1 / 49.7 →\rightarrow 0.6 / 53.3
32B →\rightarrow MetaphorStar-32B52.3 / 92.4 →\rightarrow 76.1 / 96.88.3 / 76.7 →\rightarrow 10.4 / 79.01.0 / 55.4 →\rightarrow 0.8 / 53.4

Without the image every pair falls to at most 10.4% item-strict accuracy, and under shuffled images to at most 2.0%, with proposition accuracy of 49.7%–58.9% against a 52.9% always-True rate. Text alone almost never passes the strict metric. Training does raise text-only proposition accuracy (7B: 51.6% →\rightarrow 64.4%), but it stays +31.6 points below the 96.0% with the image.

Table C: MCQ Rewards vs. TFQ Rewards on the Same 100 Images (ZgGo W1/Q2, V9dZ W1). MCQ-GRPO used the same 100 images, base model, and Table 13 settings. Each image gets one 6-option MCQ: the key is its own reference implication, the 5 distractors the most similar implications of other training images (TF-IDF). Because 100 MCQ prompts fill fewer rollout batches than 984 TFQ prompts, we also ran a compute-matched variant:

Model (Qwen2.5-VL-7B, 100 training images)UpdatesTFQ Strict (%), 1,236 entriesMCQ (%), 50 itemsΔ\Delta TFQ vs. Base
Qwen2.5-VL-7B (base)--39.446.0--
+ MCQ-GRPO (one MCQ per image)540.824.0+1.4 [-1.6, +4.2] (p=0.40p = 0.40)
+ MCQ-GRPO, compute-matched7540.440.0+1.0 [-2.0, +3.8] (p=0.55p = 0.55)
+ TFQ-GRPO (MetaphorStar-7B)7571.974.0+32.5

MCQ rewards help on neither format. With the same updates and rollouts they leave TFQ accuracy where the base model already was, while TFQ rewards reach 71.9%, +31.5 over compute-matched MCQ-GRPO. On MCQ both fall below the base model's 46.0% and far below TFQ-GRPO's 74.0% (Table 8); they stop committing to a single option.

Table G: Transfer to Benchmarks Outside II-Bench (c5oT W2). MetaphorStar-7B gains on both. Neither was used in training; both use the paper's parsers. GoFigure (AndreaJaunarena-Cayetano/GoFigure) has no authors in common with II-Bench: 639 synthetic English visual metaphors with paired entailing and contradicting claims, so its 2,556 items are 50/50 and chance is 50.0. CII-Bench is Chinese, with its own images and hand-written Chinese answers.

Outside benchmark (Qwen2.5-VL-7B)ItemsBaseMetaphorStar-7BΔ\Delta
GoFigure, English True-False2,55673.080.8+7.8
CII-Bench, Chinese multiple-choice80042.551.4+8.9

Both gains are significant (p=2.8×10−28p = 2.8 \times 10^{-28} and p=4.7×10−8p = 4.7 \times 10^{-8}). Training is entirely on English II-Bench True-False propositions, so these are cross-source and cross-language transfer. On GoFigure MetaphorStar-7B scores 80.8 with the image, 62.6 without it and 54.4 with a deranged image.

(Continued in Comment 2/2.)

AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

Independent Human Verification of MetaphorBench TFQ Labels (V9dZ W2, ZgGo W1). Three PhD-level annotators independently labeled all 492 MetaphorBench TFQs against the paired images. Mean pairwise agreement is at least 98%. All disagreements fell within one item and were resolved 2-vs-1 in favor of the released label, so no label changed.

Table D: Training With and Without Images (c5oT Q1/W3). Removing images from every training prompt costs 21.5 strict points. Same 984 TFQs and recipe, Qwen2.5-VL-7B on the 1,236 held-out entries:

Training setup (1,236 entries)Strict (%)
Base, baseline prompt39.4
Base, MetaphorStar prompt39.8
+ GRPO, images removed in training50.4
+ MetaphorGRPO (Table 13 recipe)71.9

The prompt alone makes no detectable difference to strict TFQ: the base model moves from 39.4% to 39.8% under the MetaphorStar prompt (+0.4 [-2.4, +3.3], p=0.83p = 0.83), and MetaphorGRPO is +32.1 above it. Removing images from the training prompts costs much more than the prompt does: the image-free variant reaches 50.4%, 21.5 points below MetaphorGRPO, so the image matters during training as well as at test time (Table B). We will add Table D to Section 6.4 and Appendix B.

Table E: SFT Schedule, Learning Rate and Entropy Bonus (c5oT Q3, ZgGo W2). No SFT variant reaches direct RL. Qwen2.5-VL-7B on a 200-entry sample of the same pool:

TrainingStrict / Prop (%)
Base, no training40.0 / 89.0
SFT, 1 epoch, lr 5×10−55 \times 10^{-5}4.5 / 66.1
SFT, 5 epochs, lr 5×10−55 \times 10^{-5} (Table 8 setup)32.0 / 88.1
SFT, 5 epochs, lr 1×10−51 \times 10^{-5}51.0 / 90.9
Table 8 SFT →\rightarrow GRPO63.0 / 94.9
Table 8 SFT →\rightarrow GRPO + entropy bonus 0.0166.5 / 95.1
MetaphorGRPO, no SFT73.0 / 95.9

A lower SFT learning rate recovers ground (lr 1×10−51 \times 10^{-5} reaches 51.0 against 32.0) and the entropy bonus is higher after SFT (66.5 against 63.0), but no variant reaches direct RL at 73.0. The 1-epoch model does not learn to stop: 99.3% of its outputs hit the token limit.

Table F: Few-Shot CoT on Strong Proprietary Models (V9dZ W4). Same MetaphorBench-50 TFQs and High-Level Bench-EN MCQs, one question per request, the paper's parsers, and n=3n = 3 fixed MetaphorQA-Lite demonstrations matching the Table 8 setting, whose images are in neither evaluation split. Each model is compared only with itself:

Model (50 items)Metric (%)Zero-shot+ CoT prompt+ Few-shot CoT (n=3n = 3)
GPT-6.1-SolTFQ strict74.062.058.0
GPT-6.1-SolMCQ82.078.080.0
GPT-5.2TFQ strict60.054.056.0
Qwen3.8-MaxTFQ strict70.050.050.0

Few-shot CoT improves none of the three: TFQ falls 20.0 points on Qwen3.8-Max, 16.0 on GPT-6.1-Sol (p=0.057p = 0.057) and 4.0 on GPT-5.2 (p=0.81p = 0.81), and MCQ falls 2.0 on GPT-6.1-Sol. Qwen3.8-Max is the strongest proprietary model in Table 5, and demonstrations cost it the most. Table 8 shows the same on TFQ for the open models.

Summary of Revisions for the Revised Version.

  • Evaluation and controls (Section 5.1, Section 6.4, Appendix B): Add the full held-out pool evaluation (Table A), the no-image and shuffled-image controls (Table B), both MCQ-GRPO runs (Table C), the image-free training and prompt controls (Table D), and the GoFigure and CII-Bench evaluations (Table G).
  • Generalization and ranking scope (Abstract, Section 1, Section 5, Section 7): Narrow transfer claims to unseen images and unseen formats within reference-based implication tasks, and state that MetaphorStar-32B obtains the highest TFQ and OSQ scores among the models in Table 5, within sampling noise of the strongest proprietary model there on both (74% vs. 70% TFQ, 3.94 vs. 3.80 OSQ, Section 5.1).
  • Human verification (Section 3.2, Appendix B): Report the three-annotator check of all 492 MetaphorBench TFQs (at least 98% mean pairwise agreement) and the TFQ category breakdown, and mark the internally inconsistent false statements in Table 16.
  • SFT warmup evidence (Section 6.4, Appendix D): Add the SFT hyperparameters, the schedule, learning-rate and entropy-bonus ablations (Table E) and the MCQ parse audit, state the single-teacher scope, and keep the "SFT Curse" term, which Section 1 already defines for our setting.
  • Reasoning and prompting details (Section 5.2, Table 6, Appendix C): State the MMMU split in Table 6, highlight Qwen3-VL-8B + MetaphorGRPO in the main text, and add the verbatim prompts and Table F to Appendix C.

If you have any further questions, please feel free to let us know.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.