acceptodds
Under review as a conference paper at ICLR 2027

When Preference Margins Hide Text-Only Shortcuts: Visual-Conditional Margin Learning for Multimodal Reward Models

Abstract

Multimodal reward models (MM-RMs) should assess response quality using cross-modal information, yet often rely on text-only shortcuts, leading to underuse of visual evidence and potentially impairing reward judgments and downstream optimization. By analyzing text-only preference signals in pretrained representations and their evolution during reward-model training, we find that the standard Bradley–Terry (BT) objective widely used to train reward models from pairwise preferences, despite improving multimodal preference prediction, further amplifies pre-existing textual preference signals and leaves the contribution of visual evidence to the pairwise preference margin unsupervised. Based on these observations, we propose visual-conditional margin learning (VCM), a unified framework with two levels of visual-conditional supervision: visual-gain supervision from text-only to multimodal conditioning, and stronger evidence-flip (EF) supervision enforcing preference-margin changes under local reversals of decisive visual evidence while keeping the question and responses fixed, thereby cancelling the shared text-only component. We evaluate our method across multiple MM-RM backbones and multiple multimodal preference benchmarks, showing improved reward-model performance, reduced text-only shortcut dependence, and better downstream response selection in Best-of- evaluation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Reward Modeling, Preference Learning, Multimodal Alignment, Text-only Shortcuts, Unimodal Spurious Correlations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9zeBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates text-only shortcuts in multimodal reward models and proposes Visual-Conditional Margin Learning. The method improves preference prediction on several benchmarks, but the gains are not consistent across all settings.

Critical strengths
  1. The preliminary study provides clear motivation for the method. It shows that text-only preference signals exist in pretrained models and become stronger during BT training.
  2. Evidence-flip supervision is a useful design. It keeps the text unchanged and tests whether preference margins respond to changes in visual evidence.
  3. The experiments include meaningful analyses beyond benchmark accuracy. The shortcut-failure results and matched-exposure comparison are particularly informative.
Critical weaknesses
  1. The reliability of the evidence-flip samples deserves further attention. Section 4.3 assumes that the edited image reverses the preference between the two responses. However, Appendix B reports an 88% preference-flip success rate based on 100 manually inspected samples. Some edited examples may therefore provide incorrect supervision.I suggest a small controlled experiment. The authors could manually verify a subset of EF training samples and compare the original training setup with one using only verified samples. A size-matched random subset would help control for sample count. Reporting benchmark accuracy and evidence-flip margins would reveal how sensitive VCM is to imperfect edits.
  2. The mixed results across benchmarks need a clearer explanation. VCM performs well on VLFeedback and MLLM-Pair. However, the improvements are less consistent elsewhere. For InternVL3-8B in Table 1, Held-out accuracy drops from 82.44 to 81.10 compared with BT. VLRB-H also decreases from 87.32 to 85.45.Appendix E.3 shows similar variation across benchmark categories. A brief discussion of these differences would help clarify which types of preference tasks benefit most from VCM.
Questions for authors

Please see the weaknesses.

5 comments

Part 1

We thank the reviewer for the constructive feedback. We conducted additional controlled experiments and analyses to address both concerns. New and revised manuscript content is highlighted in blue for ease of review.

1. W1: Reliability of Evidence-Flip Samples

Our manual inspection of 100 EF pairs found 97% local-edit success and 88% preference-reversal success. To assess the impact of imperfect edits, we conducted the suggested controlled experiment.

Quality-controlled experiment. We manually screened and verified 400 high-quality EF pairs (250 POVID / 150 RLHF-V) and compared them with 400 randomly selected pairs of matched source composition. We trained Qwen2.5-VL-7B and InternVL3-8B with identical configurations, changing only the EF subset.

BackboneEF400Held-outVLFVLRB-HMMRBMLLM-Pair
Qwen2.5-VL-7BVerified80.16±0.7982.41±0.8187.8576.6365.48
Random80.46±0.5382.16±0.8188.1275.9765.00
InternVL3-8BVerified80.82±1.0778.44±0.3286.2566.8162.94
Random80.30±1.0275.87±0.5586.7860.6759.14

Quality filtering has backbone-dependent benefits. Qwen2.5 performs comparably with verified and random EF samples, suggesting relative robustness to edit noise. InternVL3 benefits more substantially from verified supervision, improving VLF by 2.57 pp, MMRB by 6.14 pp, and MLLM-Pair by 3.80 pp.

EF-margin analysis. We also report both-correct accuracy and mean EF margin on the 800 POVID/RLHF-V EF pool and the 200 RLAIF-V edits excluded from EF supervision.

Evaluation setBackboneBoth-correct (%) Verified / RandomMean EF margin Verified / Random
EF pool (800)Qwen2.527.00 / 23.252.394 / 3.260
InternVL316.62 / 15.121.888 / 2.482
Held-out EF edits (200)Qwen2.513.00 / 11.500.731 / 0.639
InternVL312.50 / 17.500.488 / 0.619

Verified supervision improves both-correct accuracy on the 800-pair pool for both backbones, despite smaller mean EF margins. Results on held-out edits are more mixed, confirming that larger EF margins do not necessarily imply more accurate preference reversals.

Overall, these experiments demonstrate that VCM remains effective with imperfect EF supervision, while higher-quality edits can provide additional benefits, particularly for InternVL3.

The data-quality protocol is clarified in revised Appendix B, and the complete controlled experiment, including benchmark accuracy and EF-margin diagnostics, is now reported in Appendix E.4.

Part2

2. W2: Explanation of Mixed Benchmark Results

Why do Held-out and VLRB-H sometimes decline? The key distinction is between source-matched evaluations, where text-only shortcuts remain predictive, and external benchmarks that better assess generalization. Held-out and VLRB-H share underlying data sources with training, although evaluation examples are strictly deduplicated. Standard BT can exploit transferable textual correlations on these sources, whereas VCM explicitly encourages visual evidence to influence preference margins. Reducing reliance on these shortcuts can therefore slightly decrease source-matched accuracy while improving visual-conditional generalization.

Our common-partition analysis directly supports this explanation. Using BT's text-only predictions to define fixed subsets, we compare both models on identical Held-out examples:

BackboneBT Text-only Success (%)Δ Accuracy on SuccessΔ Accuracy on Fail
Qwen2.5-VL-7B68.9-0.60 pp+4.27 pp
Qwen3-VL-8B70.6-2.76 pp+4.91 pp
InternVL3-8B69.9-2.70 pp+1.82 pp

VCM improves shortcut-fail accuracy across all three backbones, demonstrating better multimodal preference prediction when text-only cues are misleading. However, approximately 70% of Held-out examples are already correctly predicted without images, and the smaller decreases on this larger subset can offset the shortcut-fail gains, explaining the aggregate declines for Qwen3 and InternVL3. The VLRB-H decrease is consistent with a similar source-matched effect, although its exact cause has not been isolated.

Importantly, external benchmark performance improves consistently. Compared with BT, VCM improves VLFeedback by 2.35–16.31 pp and MLLM-Pair by 6.75–8.01 pp across all three backbones, supporting stronger generalization beyond source-specific correlations.

The source-matched versus external generalization pattern is discussed in revised §5.2, with the common-partition analysis provided in Appendix E.5.

Which tasks benefit most? Appendix E.3 reveals a clear pattern in the improvements over BT:

Evaluation categoryQwen2.5Qwen3InternVL3
MMRB VQA + Correctness+7.30+3.45+2.25
MMRB Preference+4.44+7.34+10.40
MMRB Safety+9.25+13.19+2.17

All values are percentage-point improvements over BT.

The most consistent gains occur in visual correctness, fine-grained preference judgments, and safety assessment. In particular, MMRB Preference compares responses with similar correctness status, making it especially relevant to visual-conditional reward modeling. In contrast, general instruction-following, knowledge-intensive, and complex reasoning categories exhibit more backbone-dependent results, as detailed in Appendix E.3.

We have expanded the category-level discussion in Appendix E.3 to clarify these task-dependent benefits.

In summary, VCM consistently improves visual-conditional preference discrimination and external generalization, while some source-matched accuracy may be traded off when text-only shortcuts remain predictive.

Dear Reviewer 9zeB,

As the discussion period is drawing to a close, we would greatly appreciate it if you could take a moment to revisit our rebuttal and revised manuscript.

Following your suggestion, we conducted an additional quality-controlled EF experiment, comparing manually verified samples with a size-matched random subset, and reported the results in Appendix E.4. We also expanded the discussion of benchmark differences and task-dependent benefits.

We hope these additional results help address your concerns. We would be grateful for your feedback on whether any questions remain, and would be happy to provide further clarification.

Thank you sincerely for your constructive suggestions and consideration.

Best regards, Authors

Reviewer_AVcxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies text-only shortcuts in multimodal reward models. Frozen representation probes and training-dynamics analyses show that preference signals are available without images and become stronger during standard Bradley–Terry (BT) training. The authors propose visual-conditional margin learning (VCM), which supplements BT with visual-gain supervision relative to text-only scoring and evidence-flip (EF) supervision using local image edits intended to reverse preferences. Experiments cover three main backbones, two smaller models, several preference benchmarks, and Best-of-N response selection. The gains vary across benchmarks and under matched edited-data exposure.

Critical strengths
  1. The paper is clearly organized, and the preliminary analysis gives a useful motivation for the method. The frozen probes and BT training dynamics show how good multimodal preference accuracy can coexist with strong text-only predictability.

  2. The EF objective has a clear connection to the shortcut problem. Keeping the question and both responses fixed while changing the image allows the shared text-only component to cancel from the cross-state margin difference.

  3. VCM improves VLF and MLLM-Pair over BT across all three main backbones. The Best-of-N evaluation, in which reward models select from the same candidate responses, also provides useful evidence for downstream response selection.

Critical weaknesses
  1. The benefit of the additional objectives is less consistent once exposure to EF data is matched (Table 6). For InternVL3, VCM substantially trails BT on VLRB-H and MMRB, while improving VLF. Appendix E.1 acknowledges these mixed results, but I would appreciate more discussion of the trade-offs. It remains unclear when explicit margin supervision is preferable to simply training on the edited preference pairs.

  2. The interpretation of Figure 5(b) could be clearer. A larger EF margin shows a stronger change between the two visual states, but does not by itself establish that the model ranks the responses correctly in both states. I also could not find whether the EF pairs used in this figure are independent of training. These details matter for assessing whether the intended preference reversals generalize beyond the training contrasts.

Questions for authors
  1. Which EF pairs are used for Figure 5(b), and are they independent of training? Could you also report the fraction of pairs for which both the original and edited preferences are correct under BT and VCM?

  2. In Figure 4, are the shortcut-success and shortcut-fail subsets defined by a shared text-only reference or separately by each evaluated model? If the partitions are model-specific, could you also show the comparison on a common partition to help interpret the change in SFD?

  3. Appendix B reports an 88% preference-reversal success rate in a manual inspection of 100 EF pairs. How were edits that failed to reverse the preference handled? Were any such cases retained in the EF training set?

5 comments

Part 1

We thank the reviewer for the constructive feedback. We address each concern with additional clarifications and experimental evidence. New and revised manuscript content is highlighted in blue for ease of review.

1. W1: Benefits and Trade-offs of Explicit Margin Supervision

When is explicit margin supervision preferable to simply adding EF samples?

The key distinction is what is supervised. BT treats original and edited preference pairs as independent examples, whereas VCM explicitly constrains their cross-state preference-margin difference. This encourages the reward model to adjust its preference according to visual evidence, rather than merely fitting additional preference labels. Thus, explicit margin supervision is particularly valuable for improving visual-conditional preference discrimination, as supported by the gains on external benchmarks such as VLFeedback and MLLM-Pair.

Understanding the trade-offs. Additional visual-conditional constraints can introduce optimization tension with standard BT supervision, particularly when source-specific textual correlations remain predictive. Notably, even incorporating EF samples through ordinary BT training reduces accuracy on source-matched evaluations:

BackboneHeld-out (BT → BT with EF)VLRB-H (BT → BT with EF)
Qwen2.5-VL-7B80.91 → 79.5287.32 → 82.24
Qwen3-VL-8B82.00 → 81.7189.19 → 86.92
InternVL3-8B82.44 → 81.7687.32 → 85.31

Both Held-out and VLRB-H share underlying sources with training but use disjoint evaluation examples. These results suggest a trade-off between preserving source-specific preference patterns and strengthening visual dependence.

Investigating InternVL3's optimization sensitivity. We acknowledge that InternVL3 exhibits less consistent gains than the Qwen backbones. Importantly, our original experiments used the same training hyperparameters across all backbones, without backbone-specific tuning. Different architectural properties, including the visual encoder and multimodal representations, may make InternVL3 more sensitive to this shared configuration.

Consistent with this possibility, we observed higher late-stage gradient norms and lower EF-constraint satisfaction on InternVL3 than on the Qwen models. We also identified a concrete optimization-budget discrepancy in the matched-exposure experiment (Appendix E.1): BT and SA process edited samples independently, yielding 71 optimizer steps, whereas VCM must group the original and edited visual states within each batch to compute the EF loss, resulting in 63 steps under the original configuration despite identical EF data exposure.

To investigate whether the weaker results could be recovered with a closer optimization budget, we reduced VCM's effective global batch size from 96 to 84, obtaining 72 steps.

InternVL3-8BStepsHeld-outVLFVLRB-HMMRBMLLM-Pair
BT7181.76±0.8075.89±0.8685.3174.6060.24
SA7180.08±0.4273.88±0.4874.7774.6659.35
VCM (original)6380.46±0.4977.81±0.8478.1067.0960.32
VCM (batch 84)7282.17±1.2778.42±0.8484.7872.1961.78

All methods receive matched EF data exposure, following Appendix E.1.

The adjusted VCM configuration improves all five benchmarks over the original setting, outperforming BT on three benchmarks and SA on four. In particular, the substantial recovery on VLRB-H and MMRB indicates that InternVL3's weaker original performance is not an inherent limitation of the VCM objective.

We emphasize that this is a targeted sensitivity experiment, not a full backbone-specific hyperparameter search. Since changing batch size also changes optimization dynamics, we do not attribute the recovery solely to step matching or claim to have isolated its architectural cause. Nevertheless, these findings show that VCM's performance on InternVL3 is recoverable with a modest configuration adjustment.

The revised Appendix E.1 clarifies how all baselines receive matched EF exposure while VCM additionally pairs visual states for the EF loss. We also discuss source-matched versus external evaluation behavior in revised §5.2.

Part 2

2. W2 & Q1: EF Evaluation Protocol and Both-Correct Accuracy

We apologize that the evaluation split in Figure 5(b) was not sufficiently clear in the original submission. The revised §5.4, Figure 5(b) caption, and Appendix B now clarify this protocol.

As described in Appendix B, we constructed 1,000 EF pairs: 500 from POVID, 300 from RLHF-V, and 200 from RLAIF-V. The 800 POVID/RLHF-V edits are used for EF supervision, whereas the 200 RLAIF-V edits are excluded from EF training and used in Figure 5(b). Their original preference pairs may still participate in standard BT training; only the edited visual states are held out from EF supervision.

We agree that a larger EF margin alone does not guarantee correct rankings under both visual states. We therefore additionally evaluate both-correct accuracy, requiring correct preference predictions under both the original and edited images.

BackboneTraining-source EF (BT → VCM)Held-out EF edits (BT → VCM)
Qwen2.5-VL-7B13.50 → 24.13%14.5 → 18.0%
Qwen3-VL-8B12.50 → 20.13%22.5 → 25.0%
InternVL3-8B11.88 → 14.25%18.0 → 18.5%

VCM improves both-correct accuracy across all three backbones and both evaluation sets, providing direct evidence of improved preference reversal beyond EF-margin magnitude. The smaller gains on held-out edits, particularly for InternVL3, also indicate that cross-state preference learning remains sensitive to the backbone and optimization setting, consistent with our analysis above.

The full both-correct evaluation is now included in Appendix E.5.

3. Q2: Model-Specific Shortcut Partitions

Our SFD analysis follows Li et al., The Devil Is in the Details (SA; Appendix C). We use model-specific partitions, defined by each evaluated model's text-only predictions, because shortcut reliance reflects learned model behavior rather than an intrinsic binary property of an example.

To address the comparability concern, we additionally evaluate both methods under a common partition defined by BT's text-only predictions:

DatasetBackboneΔ Shortcut-Fail Acc.SFD (BT → VCM)
Held-outQwen2.5+4.27 pp29.71 → 24.85
Qwen3+4.91 pp33.45 → 25.78
InternVL3+1.82 pp27.96 → 23.45
VLFQwen2.5+27.07 pp34.36 → 17.00
Qwen3+16.17 pp38.75 → 24.83
InternVL3+11.83 pp27.42 → 11.28

Under identical partitions, VCM improves shortcut-fail accuracy and reduces SFD in all six settings. This confirms that the improvement is not merely an artifact of model-specific partitioning: VCM improves multimodal predictions on fixed examples where BT's text-only predictions fail.

The common-partition results and evaluation details are now provided in Appendix E.5.

4. Q3: EF Data Quality and Filtering

Our construction pipeline discards invalid image generations but does not apply additional per-sample preference-reversal filtering. Manual inspection of 100 randomly sampled EF pairs found 97% local-edit success and 88% preference-reversal success.

Given the inherent noise in preference annotations and the additional verification cost, we considered this quality level a reasonable trade-off for scalable EF construction.

We also conducted a controlled comparison of 400 model-verified versus 400 size-matched random EF pairs on two backbones. The results indicate that quality filtering can improve downstream performance, with backbone-dependent effects.

The filtering procedure is now clarified in Appendix B, with the controlled experiment reported in Appendix E.4. Please also see our response to Reviewer 9zeB (W1) for a summary of the benchmark and EF-margin results.

Dear Reviewer AVcx,

As the discussion deadline approaches, we would greatly appreciate it if you could revisit our rebuttal and the revised manuscript.

We have provided additional evidence addressing your concerns, including the matched EF-exposure comparison, both-correct accuracy, and common-partition shortcut analysis, with further clarifications incorporated into the revised manuscript and appendices.

We hope these analyses help resolve the main questions raised in your review. Please let us know if any concerns remain; we would be very happy to clarify them.

Thank you very much for your time and constructive feedback.

Best regards, Authors

Reviewer_kpstOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper starts with a relevant and important problem - the method of rewarding model training (Bradley-Terry method) can reward the correct ones without considering / bypassing the actual image. The authors propose visual-conditioning margin learning which consists of two parts - visual gain supervision from text-only to multimodal conditioning and evidence-flip method. They effectively cancel the text only component and put more effort into actually analyzing the image. The first part of the paper is dedicated to establishing the issue and they established it in three separate experiments using two backbones. The proposed method makes sense and the experimental section is mostly broad and extensive. Authors evaluate Qwen2.5, Qwen3 and InternVL3.

Critical strengths

The specified problem authors point out is important and authors show it empirically in three separate experiments using two backbones in Figure2. They first show with a linear reward head the text only and multimodal rewards are very similar and then show the accuracies similarly improve while training and thirdrelative ordering of preference margins are similar in both modes. This is irrefutable empirical evidence of the issue the authors trying to solve.

Critical weaknesses

The main weakness I see is the novelty of the paper. The paper is effectively a combination of https://arxiv.org/pdf/2503.03122 and https://aclanthology.org/2025.acl-long.1462.pdf which are both cited in this paper. The problem, which is the main hook of the paper the authors trying to solve is already shown in the first paper and the editing method is already discussed in the second paper. Even the empirical results shown in the paper Figure2 and more or less shown in the Figure1 of the first paper. The related works section clearly mentions these papers but does not separate previous work with the current paper.

Questions for authors

Clearly and concisely explain the difference / improvement from the papers mentioned in the weakness section.

4 comments

We thank the reviewer for highlighting these connections. Our core novelty lies in addressing a supervision gap in standard BT reward learning: correct preference prediction does not necessarily require visual evidence to contribute to the preference margin. VCM addresses this limitation through explicit visual-conditional margin supervision, rather than merely identifying text-only shortcuts or applying counterfactual image editing.

1. Difference from The Devil Is in the Details (SA)

Analysis. While SA primarily investigates text-only shortcuts through cross-distribution generalization and post-training analyses, our work further examines their presence before reward training and amplification during BT optimization. Specifically, Fig. 2 shows that pretrained backbones already contain preference-predictive textual signals, while text-only accuracy and multimodal–text-only margin correlation progressively increase during BT training.

Method and improvement. SA mitigates shortcuts by reweighting BT examples using the Shortcut-Failure Coefficient (SFC), but its objective still supervises only the overall preference margin. In contrast, VCM changes what is supervised, rather than merely how examples are weighted. It introduces visual-gain supervision to explicitly constrain the visual contribution to preference margins, and EF supervision to further constrain their response to altered visual evidence.

2. Difference from Symmetrical Visual Contrastive Optimization (S-VCO)

Learning objective. The key distinction lies in the supervised quantity. S-VCO constrains individual response likelihoods across visual conditions to improve image–response alignment, but does not directly enforce pairwise scalar reward-margin reversal. In contrast, VCM directly supervises how the relative preference between two responses changes with visual evidence, targeting the decision mechanism of scalar reward models.

EF design and improvement. By keeping the query and both candidate responses fixed across visual states, EF enables the cross-state preference-margin difference to exactly cancel the shared text-only component (Eq. 7). Consequently, an image-independent reward model cannot satisfy a positive EF constraint. Importantly, EF is a stronger extension of VCM rather than its foundation: visual-gain supervision already provides explicit visual-conditional constraints without requiring edited images.

Manuscript revision. We have further emphasized this supervision-level distinction in the revised Introduction (§1), with related prior work discussed in §2. Revised content is highlighted in blue for ease of review.

We hope these clarifications help distinguish our contributions from prior work. We would be happy to further discuss any remaining concerns regarding the novelty of VCM and provide additional clarification if needed. We sincerely appreciate the reviewer's thoughtful feedback.

Dear Reviewer kpst,

As the discussion period approaches its end, we would sincerely appreciate it if you could revisit our rebuttal and manuscript revisions.

We have specifically addressed your main concern regarding novelty, clarifying how VCM differs from SA and S-VCO in its direct supervision of visual contributions to pairwise reward margins. We have also highlighted this distinction in the revised manuscript.

We would be grateful to know whether these clarifications address your concerns or if any important distinction remains unclear. We would greatly appreciate your reconsideration of our contribution in light of these clarifications.

Thank you very much for your thoughtful review and valuable feedback.

Best regards, Authors

My issue is mainly clarified. I'm increasing my score to 3.

Thank you very much for reconsidering our work and for your thoughtful feedback throughout the discussion. We are glad that our clarifications have addressed your main concern.

We sincerely appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.