- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates text-only shortcuts in multimodal reward models and proposes Visual-Conditional Margin Learning. The method improves preference prediction on several benchmarks, but the gains are not consistent across all settings.
Critical strengths
- The preliminary study provides clear motivation for the method. It shows that text-only preference signals exist in pretrained models and become stronger during BT training.
- Evidence-flip supervision is a useful design. It keeps the text unchanged and tests whether preference margins respond to changes in visual evidence.
- The experiments include meaningful analyses beyond benchmark accuracy. The shortcut-failure results and matched-exposure comparison are particularly informative.
Critical weaknesses
- The reliability of the evidence-flip samples deserves further attention. Section 4.3 assumes that the edited image reverses the preference between the two responses. However, Appendix B reports an 88% preference-flip success rate based on 100 manually inspected samples. Some edited examples may therefore provide incorrect supervision.I suggest a small controlled experiment. The authors could manually verify a subset of EF training samples and compare the original training setup with one using only verified samples. A size-matched random subset would help control for sample count. Reporting benchmark accuracy and evidence-flip margins would reveal how sensitive VCM is to imperfect edits.
- The mixed results across benchmarks need a clearer explanation. VCM performs well on VLFeedback and MLLM-Pair. However, the improvements are less consistent elsewhere. For InternVL3-8B in Table 1, Held-out accuracy drops from 82.44 to 81.10 compared with BT. VLRB-H also decreases from 87.32 to 85.45.Appendix E.3 shows similar variation across benchmark categories. A brief discussion of these differences would help clarify which types of preference tasks benefit most from VCM.
Questions for authors
Please see the weaknesses.
5 comments
Part2
2. W2: Explanation of Mixed Benchmark Results
Why do Held-out and VLRB-H sometimes decline? The key distinction is between source-matched evaluations, where text-only shortcuts remain predictive, and external benchmarks that better assess generalization. Held-out and VLRB-H share underlying data sources with training, although evaluation examples are strictly deduplicated. Standard BT can exploit transferable textual correlations on these sources, whereas VCM explicitly encourages visual evidence to influence preference margins. Reducing reliance on these shortcuts can therefore slightly decrease source-matched accuracy while improving visual-conditional generalization.
Our common-partition analysis directly supports this explanation. Using BT's text-only predictions to define fixed subsets, we compare both models on identical Held-out examples:
| Backbone | BT Text-only Success (%) | Δ Accuracy on Success | Δ Accuracy on Fail |
|---|---|---|---|
| Qwen2.5-VL-7B | 68.9 | -0.60 pp | +4.27 pp |
| Qwen3-VL-8B | 70.6 | -2.76 pp | +4.91 pp |
| InternVL3-8B | 69.9 | -2.70 pp | +1.82 pp |
VCM improves shortcut-fail accuracy across all three backbones, demonstrating better multimodal preference prediction when text-only cues are misleading. However, approximately 70% of Held-out examples are already correctly predicted without images, and the smaller decreases on this larger subset can offset the shortcut-fail gains, explaining the aggregate declines for Qwen3 and InternVL3. The VLRB-H decrease is consistent with a similar source-matched effect, although its exact cause has not been isolated.
Importantly, external benchmark performance improves consistently. Compared with BT, VCM improves VLFeedback by 2.35–16.31 pp and MLLM-Pair by 6.75–8.01 pp across all three backbones, supporting stronger generalization beyond source-specific correlations.
The source-matched versus external generalization pattern is discussed in revised §5.2, with the common-partition analysis provided in Appendix E.5.
Which tasks benefit most? Appendix E.3 reveals a clear pattern in the improvements over BT:
| Evaluation category | Qwen2.5 | Qwen3 | InternVL3 |
|---|---|---|---|
| MMRB VQA + Correctness | +7.30 | +3.45 | +2.25 |
| MMRB Preference | +4.44 | +7.34 | +10.40 |
| MMRB Safety | +9.25 | +13.19 | +2.17 |
All values are percentage-point improvements over BT.
The most consistent gains occur in visual correctness, fine-grained preference judgments, and safety assessment. In particular, MMRB Preference compares responses with similar correctness status, making it especially relevant to visual-conditional reward modeling. In contrast, general instruction-following, knowledge-intensive, and complex reasoning categories exhibit more backbone-dependent results, as detailed in Appendix E.3.
We have expanded the category-level discussion in Appendix E.3 to clarify these task-dependent benefits.
In summary, VCM consistently improves visual-conditional preference discrimination and external generalization, while some source-matched accuracy may be traded off when text-only shortcuts remain predictive.
Dear Reviewer 9zeB,
As the discussion period is drawing to a close, we would greatly appreciate it if you could take a moment to revisit our rebuttal and revised manuscript.
Following your suggestion, we conducted an additional quality-controlled EF experiment, comparing manually verified samples with a size-matched random subset, and reported the results in Appendix E.4. We also expanded the discussion of benchmark differences and task-dependent benefits.
We hope these additional results help address your concerns. We would be grateful for your feedback on whether any questions remain, and would be happy to provide further clarification.
Thank you sincerely for your constructive suggestions and consideration.
Best regards, Authors
Part 1
We thank the reviewer for the constructive feedback. We conducted additional controlled experiments and analyses to address both concerns. New and revised manuscript content is highlighted in blue for ease of review.
1. W1: Reliability of Evidence-Flip Samples
Our manual inspection of 100 EF pairs found 97% local-edit success and 88% preference-reversal success. To assess the impact of imperfect edits, we conducted the suggested controlled experiment.
Quality-controlled experiment. We manually screened and verified 400 high-quality EF pairs (250 POVID / 150 RLHF-V) and compared them with 400 randomly selected pairs of matched source composition. We trained Qwen2.5-VL-7B and InternVL3-8B with identical configurations, changing only the EF subset.
Quality filtering has backbone-dependent benefits. Qwen2.5 performs comparably with verified and random EF samples, suggesting relative robustness to edit noise. InternVL3 benefits more substantially from verified supervision, improving VLF by 2.57 pp, MMRB by 6.14 pp, and MLLM-Pair by 3.80 pp.
EF-margin analysis. We also report both-correct accuracy and mean EF margin on the 800 POVID/RLHF-V EF pool and the 200 RLAIF-V edits excluded from EF supervision.
Verified supervision improves both-correct accuracy on the 800-pair pool for both backbones, despite smaller mean EF margins. Results on held-out edits are more mixed, confirming that larger EF margins do not necessarily imply more accurate preference reversals.
Overall, these experiments demonstrate that VCM remains effective with imperfect EF supervision, while higher-quality edits can provide additional benefits, particularly for InternVL3.
The data-quality protocol is clarified in revised Appendix B, and the complete controlled experiment, including benchmark accuracy and EF-margin diagnostics, is now reported in Appendix E.4.