- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
MetaphorStar uses RL to train vision-language models to understand the hidden meaning of metaphorical images, such as comics and memes. Each image's human-written interpretation is turned into about ten true/false statements, which VLMs generate and the authors check by hand. Training is plain GRPO without an SFT stage, with a reward for the correct answer and the output format. Trained on 984 statements from 100 images, the 7B model goes from 28% to 70% on a 50-item true/false benchmark, where an item counts only if all its statements are correct. It also improves on multiple-choice and open-answer questions it never trained on.
Critical strengths
- The core idea is simple and useful. Breaking an interpretation into true/false statements gives a subjective task checkable rewards, with no judge needed during training. The same recipe could apply to humor, satire, or advertisements.
- The gains are large and data-efficient. With only 984 statements, the 7B model goes from 28% to 70% on true/false items and from 46% to 74% on multiple-choice questions. All five backbones improve, including LLaVA-1.5 and Qwen3-VL-8B.
- The evaluation is careful and honest. Always answering "true" scores 0%, test images are checked for overlap with training images, and paired significance tests are reported (Table 11, p. 17).
Critical weaknesses
- Two controls are missing. The paper argues that true/false statements are a better reward than multiple-choice questions (Table 1), but it never trains with multiple-choice rewards to test this. Some false statements also contradict themselves (Table 16, p. 22), so a model could spot them without the image, and no text-only baseline is reported.
- The "SFT Curse" claim rests on one small setup. SFT uses only 984 examples from a single teacher model, and it drops multiple-choice accuracy below the base model (46% to 28%, Table 8). This may reflect the small setup rather than SFT in general.
- The test set is a little small. All main results use 50 items from 44 images, so one item is worth 2 points.
Questions for authors
- What are the true/false scores of MetaphorStar-7B and MetaphorStar-32B on the 200 extra held-out items in Table 12?
- How does training on multiple-choice questions for the same images compare with training on true/false statements?
- Clarification: Why does the Qwen2.5-VL-32B base model score lower on MMMU (33.6) than the 7B (42.0) and 3B (42.3) models in Table 6? I ask because the original Qwen2.5-VL report shows MMMU rising with model size, and this one number accounts for over half of the 32B model's reasoning gain.
3 comments
Thank you for the response and the new experiments. The MCQ-reward baseline, the no-image controls, and the full held-out pool address most of my concerns. Please add them to the main paper.
One point remains. The 32B base model scores 24.3% on MMMU Humanities & Social Science, against about 65% for the 3B and 7B models. This looks like a parsing problem, not the model's real ability. Please check these outputs or report the reasoning gains without MMMU.
I keep my rating of weak acceptance.
Response to Reviewer ZgGo's follow-up
We thank Reviewer ZgGo for the quick follow-up and for confirming that the MCQ-reward baseline, the no-image controls and the complete held-out pool resolve those concerns.
All three are now in the main paper. The revised PDF marks every addition in blue:
- Complete image-disjoint held-out pool, 1,236 entries and 12,165 True-False questions: Table 7 in Section 5.1.
- MCQ reward with matched data and compute: Table 10 in Section 6.2.
- No-image and deranged-image controls: Figure 3(b) in Section 5.2, with the full three-scale table in Appendix B.3 (Table 15); the prompt control and the image-free training run are in Appendix B.4 (Table 16).
MMMU Humanities & Social Science. We went back to the submission behind that number. The MMMU column of Table 6 is the official test split, which has no public labels: the 10,500 predictions per model are submitted to the official MMMU evaluation server on EvalAI, and the scores are the ones that server returns. The category figures we quoted in our first response are that server's own breakdown of the same submissions, not a re-scoring of our own. Here is the complete official result for Qwen2.5-VL-32B, with the 7B and 3B submissions alongside:
| MMMU test category | Questions | Qwen2.5-VL-32B | Qwen2.5-VL-7B | Qwen2.5-VL-3B |
|---|---|---|---|---|
| Art & Design | 1,163 | 43.3 | 57.8 | 57.2 |
| Business | 1,428 | 51.1 | 49.4 | 42.2 |
| Science | 2,426 | 29.1 | 32.6 | 35.3 |
| Health & Medicine | 1,752 | 36.4 | 46.8 | 47.9 |
| Humanities & Social Science | 947 | 24.3 | 64.2 | 65.8 |
| Tech & Engineering | 2,784 | 25.9 | 29.3 | 30.9 |
| Overall | 10,500 | 33.6 | 42.0 | 42.3 |
Weighting the six categories by their official sizes returns 33.60, 42.02 and 42.35, which is exactly the 33.6 / 42.0 / 42.3 printed in Table 6, so the breakdown and the headline numbers come from one and the same submission.
That submission does not show a uniformly degraded 32B. It is the strongest of the three models on Business, and it trails the other two by 3 to 6 points on Science and Tech & Engineering, the two largest categories at 2,426 and 2,784 questions, together half the split. A pipeline-wide extraction failure for the 32B would not leave one category leading and the two largest ones intact. Every model in Table 6 was run through one VLMEvalKit configuration, with the same prompt, the same decoding settings and the same answer-extraction rule for base and trained checkpoints, so the comparison within the table is controlled. We report the scores as the official server returned them.
The reasoning conclusion does not depend on MMMU. Table 6 reports ten reasoning scores from nine benchmarks. Setting MMMU aside, the remaining nine scores give a mean change of +1.4 for 32B, with 8 of the 9 improving, +2.6 for 7B and +0.5 for 3B, against the +2.9 / +3.1 / +0.7 that the full set gives. The direction and the ordering across scales are unchanged. MMMU is the largest single contributor for the 32B, as you note, but it is not what carries the result.
If you have any further questions, please feel free to let us know.
Response to Reviewer ZgGo
We thank Reviewer ZgGo for noting that checkable True-False rewards need no judge, that the gains are large and data-efficient across all five backbones, and that the evaluation is careful and honest.
New results are in the General Response above: Table A (complete held-out pool, W3/Q1), Tables B and D (image controls, W1), Table C (MCQ-GRPO, W1/Q2), Table E (SFT ablations, W2), and the three-annotator label check (Comment 2/2). We answer each point below.
W3 & Q1: MetaphorStar-7B and MetaphorStar-32B on the complete held-out pool. On the complete 1,236-entry held-out pool (12,165 TFQs, Table A), MetaphorStar-7B reaches 71.9% strict item accuracy (vs. 39.4% base, +32.5) and MetaphorStar-32B reaches 76.1% (vs. 52.3% base, +23.7, p=1.2×10−56). MetaphorStar-3B rises from 22.5% to 64.0% (+41.5, p=1×10−117), and proposition accuracy reaches 95.0%, 96.0% and 96.8%. This is the whole pool that Table 12 samples from, so test-set size no longer limits the base-versus-trained comparison. We will add Table A to Appendix B and Section 5.1.
W1 (Part 1) & Q2: Training on multiple-choice questions vs. True-False propositions. Training with MCQ rewards on the same 100 images does not improve TFQ accuracy, with or without matched compute (Table C). On the 1,236 held-out entries, MCQ-GRPO gets 40.8% strict against 39.4% for the base model, and a compute-matched run gets 40.4%. TFQ-GRPO reaches 71.9%, +31.5 over the matched run. One MCQ per image gives one reward signal, while ~9.8 TFQs per image each check a separate claim.
W1 (Part 2): Self-contradictory false statements and text-only / shuffled-image controls. You are right that some false statements in Table 16 (e.g., entry E6) are internally inconsistent; they are still correctly labeled False, and we will mark them in Table 16. Three annotators independently relabeled all 492 MetaphorBench TFQs: mean pairwise agreement is at least 98% and no label changed (Comment 2/2). The controls you ask for are in Tables B and D:
We will add Table B to Appendix B and Section 5.1 in the revised version.
W2: Scope of the SFT warmup finding ("SFT Curse"). Table 8 does test one setup, so we varied the schedule, the learning rate and the entropy bonus (Table E). On the 46% to 28% MCQ drop, we re-ran both SFT checkpoints and it persists under a lenient parser (see our reply to c5oT, Q3). A lower learning rate does recover ground (lr 1×10−5 reaches 51.0 on a 200-entry sample against 32.0 for the Table 8 setup), and the entropy bonus reaches 66.5 against 63.0, but none of the five variants reaches direct RL at 73.0. Policy entropy drops from 1.33 (base) and 1.23 (direct RL) to 0.30 and 0.29 after the SFT arms (Figure 6). On data size the two arms of Table 8 are matched: MetaphorQA-Lite-SFT holds one trajectory per proposition of the same 984-item MetaphorQA-Lite that MetaphorGRPO trains on, so sample size is not the difference. RL does not win by seeing more data: with 0.1k propositions, a tenth of what the SFT arm saw, direct RL already reaches 48% TFQ and 64% MCQ against SFT's 42% and 28% (Table 7), and TFQ grows monotonically to 84% at 13.6k. We distilled from one teacher, and the revised version will say so. Section 1 already scopes the term to our setting, and both warmup-then-RL runs still fall short of direct RL (63.0 and 66.5 against 73.0), so we keep it. We will add the SFT hyperparameters and Table E to Appendix D.
Q3: Qwen2.5-VL-32B base score on MMMU (33.6) in Table 6. The MMMU column is the official MMMU test split (10,500 questions) scored by the evaluation server; all three base numbers are official submissions: 42.3 (3B), 42.0 (7B), 33.6 (32B). The Qwen2.5-VL report gives MMMU val, so the two are not comparable. The 32B base falls furthest behind in one category: Humanities & Social Science 24.3% against 64.2% (7B) and 65.8% (3B), which alone accounts for 3.6 of the 8.4-point gap to 7B. The scores themselves need no correction. MMMU is the largest single contributor, as you note, and the gain is not confined to it: the other nine reasoning scores sum to +12.4 for 32B, with 8 of the 9 improving. We will label the split in Table 6.
If you have any further questions, please feel free to let us know.