acceptodds
Under review as a conference paper at ICLR 2027

Delta-Vision Reward: A Distribution-Aware Multimodal Reward Model

Abstract

Reward models are central to aligning multimodal large language models, yet most existing methods reduce each response to a single scalar score. This simplification ignores disagreement in evaluation and weakens supervision on ambiguous or contested samples. We present Delta-Vision Reward (DVR), a distribution-aware multimodal reward model that predicts a Gaussian reward distribution together with a structured rationale. The predicted mean captures the central preference signal, while the variance models disagreement induced by a teacher reward ensemble. DVR is trained in two stages. We first perform supervised distributional imitation, teaching the model to match teacher-derived reward statistics and explanatory reports. We then refine pairwise preference behavior by focusing only on response pairs where the student disagrees with the teacher consensus or exhibits insufficient preference margin. This targeted refinement improves learning efficiency by concentrating supervision on hard cases. Experiments on VL-Reward-Bench and MM-RLHF-Reward Bench show that DVR is a strong open-source reward model, with clear gains in preference prediction, confidence-sensitive evaluation, and distribution alignment. We further show that DVR produces actionable feedback that helps improve responses from a base multimodal model. These results suggest that reward modeling for multimodal alignment benefits from preserving uncertainty rather than collapsing it into a scalar target.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal reward model, distribution-aware reward, Gaussian reward distribution, teacher ensemble, disagreement modeling, distributional imitation, disagreement-driven preference refinement, multimodal alignment, KL divergence supervision, RLHF, structured rationale, VL-Reward-Bench, MM-RLHF-Reward Bench

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6GetOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a multimodal reward model named Delta-Vision Reward (DVR), capable of predicting the mean reward, the variance reflecting the degree of disagreement among four teacher models, and structured reasoning rationales. Its training process involves Gaussian distribution imitation learning supervised by reasoning rationales, followed by pairwise reward fine-tuning targeting response pairs with ranking errors or insufficient discriminability. The study's primary contribution lies in integrating distributional supervision, explanatory feedback, and targeted fine-tuning into multimodal reward modeling. Validation efforts encompass two reward benchmarks, component ablation studies, selective prediction, controlled visual occlusion experiments, independent review panel assessments, and human-evaluated response revisions based on equalized budgets.

Critical strengths

The visual masking experiment shows that the model responds to information needed for the task. With the question and responses held fixed, masking a relevant image region increases predicted variance by 0.93, compared with only 0.08 when masking an equally sized irrelevant region. This suggests that the model reacts to missing visual evidence, rather than simply to any change in the image.

The feedback experiment shows practical benefits under a fair comparison. All feedback conditions use the same base model, one revision, and the same token limits. Under human evaluation, full DVR feedback achieves an 81.3% win rate against the original responses, compared with 74.8% for rationale-only feedback. This supports the usefulness of the complete feedback.

Critical weaknesses
  1. The method has limited novelty. Learning reward distributions to represent uncertainty has been explored in prior work [1]. Using teacher-generated rationales as additional supervision is also established [2], as is selecting informative samples to refine reward models [3]. DVR combines these ideas through Gaussian fitting of teacher scores, rationale distillation, and pairwise training on ranking errors or low-margin pairs. Although its offline sample-selection rule differs from [3], it remains a straightforward form of hard-example selection. Moreover, the refinement stage uses reward means and margins rather than the predicted variance. The contribution therefore appears mainly to be an integration of existing ideas for multimodal reward modeling, with limited methodological innovation.

2.There are numerical inconsistencies in some of the uncertainty analysis results, which undermine confidence in the associated conclusions. In Table 26, if the accuracy rates for each interval are weighted by sample size, the calculated overall accuracy is approximately 77.25%, rather than the 72.46% reported. The same table indicates that at a variance threshold of 1.0, the coverage rate should be 59.57%, whereas Table 28 reports a value of 68.1%. Similarly, the overall accuracy derived from the grouped results in Table 30 is approximately 75.59%, not 72.46%. These discrepancies are too large to be attributed solely to rounding errors. The authors should clarify whether these tables are based on different runs, sample sets, or aggregation rules, and provide corrected, consistent results.

[1] Sheng Xu, Bo Yue, Hongyuan Zha, and Guiliang Liu. A Distributional Approach to Uncertainty-Aware Preference Alignment Using Offline Demonstrations. ICLR, 2025. [2] Cheng-Yu Hsieh et al. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Findings of ACL, 2023. [3] Zexu Sun et al. Uncertainty and Influence aware Reward Model Refinement for Reinforcement Learning from Human Feedback. ICLR, 2025.

Questions for authors

Please clarify the numerical inconsistencies in Tables 26, 28, and 30. Weighting the group accuracies by their sample counts gives approximately 77.25% in Table 26 and 75.59% in Table 30, but both tables report 72.46% overall accuracy. Table 26 also implies 59.57% coverage at a variance threshold of 1.0, whereas Table 28 reports 68.1%. Do these tables use different data, model runs, or calculation methods? Please explain the differences and correct any errors.

Does filtering by predicted variance work better than filtering by the score difference between two responses? Please compare the two methods while keeping the same percentage of samples, and report accuracy on the retained samples. This would show whether predicting variance provides an advantage over simply retaining pairs with larger score differences.

1 comment

We thank you for recognizing the visual-grounding and feedback controls. We address novelty, statistical consistency, and score-gap selection below.

1. Contribution relative to prior work and the role of variance

We agree that Gaussian modeling, rationale distillation, and hard-example selection are not individually new. Secs. 1–2 and 5 position DVR as an integrated multimodal reward model with empirical analysis, not a new distribution family or sampling principle.

[1] uses distributional Bellman updates and CVaR for preference alignment; DVR distills image-conditioned, response-level teacher-score moments. [2] establishes rationale supervision, which we use rather than claim to originate. [3] selects policy-interaction samples using ensemble uncertainty and gradient influence; DVR uses offline student–teacher ranking/margin mismatch (Sec. 3.4, Eqs. (7)–(9)). We will sharpen these distinctions and cite [2].

Teacher–teacher score dispersion supervises Stage I; student–teacher ranking/margin mismatch selects Stage II pairs. The latter does not use predicted variance, as correctly noted. We will distinguish these meanings of “disagreement,” not describe Stage II as variance-driven.

Does distributional supervision contribute beyond refinement? Sec. 4.4/Table 4 and Appendix J.5/Table 91 report VL-Reward accuracy:

BackboneMean-only SFTDistributional SFTMean-only + refinementFull DVR
Qwen-VL-8B68.5070.2071.3072.46
InternVL2-4B66.1267.5868.7169.96

Distributional supervision adds 1.70/1.46 percentage points before refinement and 1.16/1.25 under the same refinement procedure. Thus, its contribution is not subsumed by hard-pair training, although these comparisons do not isolate disagreement information from every auxiliary-supervision effect.

The empirical contribution extends beyond teacher imitation. Appendix J.3/Table 89 gives independent-panel pairwise/hard-case agreement of 74.8/61.5 (mean-only), 76.9/65.8 (distributional SFT), and 79.6/70.4 (full DVR). The visual intervention (Sec. 4.5/Table 6; J.1/Table 87) and controlled human feedback evaluation (Sec. 4.6; J.8/Table 98), recognized in the review, further support the integrated model's utility—not novelty of its components.

2. Tables 26, 28, and 30: statistical scopes must be reconcilable

Thank you for checking the arithmetic. Appendix A.2.1/Eq. (10) defines sample-averaged accuracy; C.2.1/Eq. (22) defines variance bins; C.3.1/Eq. (23) defines coverage. Under the same sample population, run aggregation, and variance statistic, the following identities must hold:

\mathrm{Coverage}(1)=\frac{n_{[0,0.5)}+n_{[0.5,1)}}{N}.$$ The printed rows imply: | Calculation | Implied by printed rows | Reported summary | |---|---:|---:| | Table 26 weighted accuracy | 77.25% | 72.46% | | Table 26 coverage at 1 | 59.57% | 68.1% in Table 28 | | Table 30 weighted accuracy | 75.59% | 72.46% | These differences cannot be attributed to rounding. Our current presentation does not sufficiently document the cross-table statistical-scope mapping. We will explicitly identify each analysis's population, run aggregation, response-to-pair variance definition, and denominator, and reconcile the corresponding counts and summaries. A different statistical scope is a valid explanation only if the resulting numbers can be reconstructed; merely labeling the discrepancy a scope difference is insufficient. These are arithmetic checks, not replacement benchmark results. Pending reconciliation, we do not rely on the disputed selection/calibration summaries or assume all associated conclusions are unaffected. The separately reported ablations, interventions, and external evaluations provide complementary evidence. ### 3. Variance versus score gaps at equal coverage We agree that Appendix C.3/Table 28 does not answer this comparison: it reports variance-threshold selection, not a matched-coverage score-gap baseline. We therefore do not claim demonstrated superiority over gap-based selection. Sec. 3.5 uses mean scores for ranking and independent-Gaussian confidence for reliability. Writing $g=|\mu_A-\mu_B|$ and $u=\sigma_A^2+\sigma_B^2$, that confidence is $\Phi(g/\sqrt{u})$. Equal gaps can have different predicted reliability, but this mathematical distinction is not an empirical comparison. The direct test is to retain exactly the same number of pairs from identical predictions using largest $g$, smallest $u$, and largest $g/\sqrt{u}$, then compare retained accuracy and risk–coverage curves. Such results are not included in the reported experiments. We will narrow the claim accordingly: DVR supplies a learned disagreement signal, while its incremental selective-prediction value over score gaps remains to be established. We appreciate these suggestions and will clarify contribution boundaries and statistical reporting.
Reviewer_jLtUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on the problem that traditional multimodal reward models use a single scalar score, which fails to preserve disagreement among different teacher evaluators. To address this issue, the authors propose DVR, which uses a Gaussian distribution to approximate the reward mean and variance of multiple teacher evaluators and distills them together with structured rationales. This enables the student model not only to predict rewards but also to estimate evaluation disagreement, supporting selective prediction and response revision.

Critical strengths
  1. The paper proposes a Distribution-Aware Multimodal Reward Model that models the reward statistics of multiple teacher evaluators using a Gaussian distribution, together with structured rationales.
  2. The experiments show improvements across several downstream tasks, including reward benchmarks, selective prediction, and response revision.
Critical weaknesses
  1. Although the paper provides evidence for the usefulness of learned disagreement through controlled visual perturbations and independent evaluation, the sources of teacher disagreement are not sufficiently discussed. Disagreement may come from uncertainty in the task itself, but it may also arise from differences in teacher capabilities or scoring scales. Moreover, the paper uses reward variance to measure teacher disagreement, but score-level disagreement does not necessarily imply preference-level disagreement. The authors should clarify these different sources of disagreement and explain why reward variance is an appropriate measure.
  2. The experiments show that distributional supervision improves downstream performance. However, it remains unclear whether these gains come from modeling teacher disagreement or simply from the additional variance prediction task. The authors should further analyze the contribution of disagreement information itself.
  3. The writing and organization could be improved:
    • Some important experimental analyses are placed in the appendix, while there is still room in the main text. I suggest moving some of these results into the main paper to make the presentation clearer.
    • The paper inconsistently uses Qwen-VL-8B and Qwen3-VL-8B (Figure 1). The model names should be consistent.
  4. The references contain several issues:
Questions for authors

Could the authors clarify the sources of teacher disagreement and provide more evidence that the improvements come from modeling disagreement rather than the additional supervision task?

1 comment

We thank the reviewer for recognizing DVR's formulation and utility. We distinguish teacher-score dispersion, preference disagreement, and human uncertainty below.

1. Sources of disagreement and scoring-scale controls

Sec. 3.2, Eq. (3), defines the target as the sample variance of four teacher scores, not an identified measure of intrinsic task ambiguity. It can reflect missing visual evidence, competing evaluation criteria, teacher capabilities, and scoring conventions. Appendix C's opening paragraph explicitly states that it does not separately identify epistemic and aleatoric uncertainty or recover the full human preference distribution.

Several existing analyses address these sources:

  • Scales: Appendix B.1.2 specifies a shared 0–10 scoring prompt. B.3.3/Table 20 reports teacher means of 5.42–5.73 and standard deviations of 1.89–2.14. B.2.3/Table 16 compares raw scores, min–max, Z-score/rescaling, and rank aggregation: VL-Reward accuracy is 72.46%, 71.89%, 71.52%, and 70.94%, respectively. These checks support the practical aggregation choice; neither a shared range nor similar moments proves identical calibration.
  • Capabilities/composition: Appendix J.6/Table 93 reports 71.71–72.12% VL-Reward accuracy after removing each teacher, versus 72.46% with all four. Table 94 gives 70.88% for two strong teachers versus 69.94% for strong + weak teachers. Teacher quality matters; we do not interpret all heterogeneity as task ambiguity.
  • Visual evidence: Sec. 4.5/Table 6 and Appendix J.1/Table 87 fix the query and responses. On 312 vision-dependent samples, equal-area relevant/irrelevant occlusion increases teacher variance by 0.85/0.07 and predicted variance by 0.93/0.08. Predicted increases on 84 language-dominant controls are only 0.05/0.03. Because teacher identities remain unchanged, these within-sample contrasts show evidence-sensitive disagreement beyond fixed teacher differences.

2. Score variance is not preference-vote disagreement

We agree with this distinction. Teachers may assign different absolute scores while unanimously preferring A to B. Appendix B.3 separately reports score correlations (Tables 17–18) and preference agreement (Table 19): Pearson correlations are 0.721–0.768, while pairwise preference agreement is 77.1–80.6%. These are different measurements, not interchangeable validations.

For teacher-specific margins dk=sk(A)−sk(B)d_k=s_k(A)-s_k(B),

Var⁡k(dk)=Var⁡k(sk(A))+Var⁡k(sk(B))−2Cov⁡k(sk(A),sk(B)).\operatorname{Var}_k(d_k)=\operatorname{Var}_k(s_k(A))+\operatorname{Var}_k(s_k(B))-2\operatorname{Cov}_k(s_k(A),s_k(B)).

Thus, per-response variances alone cannot recover margin dispersion or preference-vote disagreement. DVR preserves cardinal score dispersion as a compact supervision target. Sec. 3.5 explicitly labels the induced pairwise confidence as an independent-Gaussian approximation; it does not model the covariance above. We will consistently use “teacher-score disagreement,” rather than implying equivalence to preference reversals or human ambiguity.

3. Disagreement information versus an additional prediction task

Matched component ablations. Sec. 4.4/Table 4 and Appendix J.5/Table 91 report VL-Reward accuracy:

BackboneMean-only SFTDistributional SFTMean-only + refinementFull DVR
Qwen-VL-8B68.5070.2071.3072.46
InternVL2-4B66.1267.5868.7169.96

Distributional supervision adds 1.70/1.46 points before refinement and 1.16/1.25 under the same refinement procedure. It therefore contributes beyond refinement. Moreover, Sec. 3.3/Eq. (5) jointly trains mean and variance: the mean-error term is weighted by 1/(2σθ2)1/(2\sigma_\theta^2), rather than adding a detached prediction head.

External validation. Appendix J.3/Table 89 reports independent-panel pairwise/hard-case agreement of 74.8/61.5 for mean-only RM, 76.9/65.8 for distributional SFT, and 79.6/70.4 for full DVR. Together with the fixed-text visual interventions, this supports meaningful supervision beyond simply fitting teacher labels.

These results do not fully exclude generic auxiliary-task or loss-weighting effects; fixed/shuffled-variance controls would directly test sample-specific disagreement information. We therefore claim demonstrated utility of teacher-derived distributional supervision, not a conclusive causal decomposition of its gains. Likewise, J.5/Table 92's larger gains on high-disagreement samples concern the complete pipeline, not the variance head alone.

4. Presentation and references

We will move the scale checks and key J.1/J.3/J.5 results into the main text, standardize backbone naming across Secs. 3.3/4.1, Figure 1, and the appendix, and provide the exact checkpoint identifier. We will remove all “Corresponds to … in the table” artifacts and correct LLaVA-Critic's venue year to CVPR 2025. We appreciate these concrete suggestions and will make the scope of the disagreement signal explicit throughout.

Reviewer_NvexOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether multimodal reward models can benefit from disagreement among teacher models. The authors use 4 teacher models to score responses and train a student model to predict the mean score, score variance, and an explanatory rationale. They then apply pairwise ranking training to improve scoring performance. The main contribution is to incorporate teacher disagreement as an additional supervision signal alongside mean scores and rationales in multimodal RM. The authors evaluate scoring performance, disagreement prediction, and feedback utility using two multimodal reward benchmarks, component ablations, image perturbation experiments, and response revision experiments.

Critical strengths
  1. The paper explores the idea that score differences among teacher models may contain information beyond the mean score. By preserving and learning these differences, it offers a worthwhile direction for training RMs with multiple teachers.

  2. The ablation experiments provide some support for the method. The models use the same pairwise ranking training. Adding distributional supervision leads to improvements on both backbone models. However, further analysis is needed to determine whether these gains come specifically from teacher disagreement.

Critical weaknesses
  1. There are clear inconsistencies in the experimental records and results. The appendix states that Claude 3.7 Sonnet was evaluated in October 2024, but the model was released in 2025. The group accuracies in Table 26 do not match the reported overall accuracy, and the sample counts do not agree with the coverage rates in Table 28. These issues need to be explained before the related findings can be trusted.

  2. The role of teacher disagreement remains unclear. Distributional supervision brings small improvements, but without controls using fixed or shuffled variance values, it is hard to tell whether the gains come from disagreement information itself. The authors also use variance to select reliable predictions without comparing it with selection based on score gaps. It is therefore unclear how much additional value the variance provides.

Questions for authors
  1. Please clarify whether the inconsistencies in the experimental records and results affect the main conclusions. For example, the reported evaluation date for Claude 3.7 is earlier than its release date. The group data in Table 26 also do not match the overall accuracy or the coverage rates in Table 28. Are these reporting errors, or were different calculation methods used? If I have misunderstood these points, or if the inconsistencies can be reasonably explained without affecting the main conclusions, I would be open to revising my score.

  2. I wonder whether the gains actually come from teacher disagreement, since the current experiments do not clearly separate its effect from other changes to the training objective. Could the authors provide controls using fixed or shuffled variance values? A comparison with selection based on score gaps at the same coverage rate would also help clarify whether disagreement information adds value beyond mean scores.

1 comment

We thank the reviewer for recognizing the value of teacher disagreement and the cross-backbone ablations. We clarify the API chronology, statistical scopes, and what the existing experiments establish.

1. The October 2024 date refers to API infrastructure, not evaluation of the 2025 model

The October 15–20, 2024 date in Appendix A.4.1 refers to establishment of our internal API infrastructure, not the execution date of every model evaluation subsequently performed through it. The 3.7 Sonnet entry in A.4.1 and the API configuration in A.7.2 identify version 20250219, corresponding to the 2025 model, not a model available in October 2024.

We acknowledge that “API calls conducted” incorrectly describes the infrastructure timestamp and invites the reviewer's interpretation. We will replace that wording with “internal API infrastructure established” and clearly separate infrastructure metadata, model versions, and evaluation timestamps throughout A.4–A.5. We are not claiming to have evaluated a model before its release.

2. Statistical scopes must be explicit and numerically reconcilable

Appendix A.2.1, Eq. (10), defines sample-averaged accuracy; C.2.1, Eq. (22), defines variance bins; and C.3.1, Eq. (23), defines coverage as the retained fraction. Under a common population and aggregation, the printed Table 26 rows indeed imply 77.25% accuracy and 59.57% coverage at τ=1\tau=1, rather than 72.46% and 68.1%. We agree that rounding cannot explain this.

We agree that citing different statistical scopes is insufficient unless readers can reconstruct the summaries. The current presentation does not provide a sufficiently explicit cross-table mapping. We will specify each analysis's sample population, run aggregation, pair-level variance definition, and denominator, and reconcile the bin totals and threshold counts. Pending that reconciliation, we do not rely on the disputed Appendix C summaries to establish variance's advantage. Below we instead cite separately reported component, intervention, and external-evaluation experiments.

3. What evidence already supports distributional supervision?

Incremental benefit beyond refinement. Sec. 4.4/Table 4 and Appendix J.5/Table 91 report VL-Reward accuracy:

BackboneMean-only SFTDistributional SFTMean-only + refinementFull DVR
Qwen-VL-8B68.5070.2071.3072.46
InternVL2-4B66.1267.5868.7169.96

Distributional supervision adds 1.70/1.46 percentage points before refinement and 1.16/1.25 under the same refinement procedure. Thus, refinement alone does not explain the full improvement.

Evidence-sensitive disagreement. Sec. 4.5/Table 6 and Appendix J.1/Table 87 hold the query and both responses fixed. On 312 vision-dependent samples, equal-area relevant/irrelevant occlusion increases predicted variance by 0.93/0.08 and teacher variance by 0.85/0.07. On 84 language-dominant controls, the predicted increases are only 0.05/0.03. This supports dependence on task-relevant visual evidence rather than generic image corruption.

Transfer beyond teachers. Appendix J.3/Table 89 reports independent-panel pairwise/hard-case agreement of 74.8/61.5 for mean-only RM, 76.9/65.8 for distributional SFT, and 79.6/70.4 for full DVR.

Boundary of these findings. We agree that none of these experiments substitutes for fixed/shuffled-variance controls. Such controls are not included in the reported results. They would preserve mean targets, rationales, architecture, training budget, and refinement data while removing the sample-specific correspondence of variance labels. Accordingly, our supported claim is that joint distributional supervision improves reward modeling—not that existing ablations conclusively exclude auxiliary-task or loss-weighting explanations. Sec. 3.2/Eq. (3) also explicitly defines the target as teacher-score disagreement, not the full distribution of human preferences.

4. Variance versus score gaps at matched coverage

Sec. 3.5 uses the mean for ranking and the distribution for confidence:

P(A>B)=Φ ⁣(μA−μBσA2+σB2).P(A>B)=\Phi\!\left(\frac{\mu_A-\mu_B}{\sqrt{\sigma_A^2+\sigma_B^2}}\right).

A large score gap and low disagreement are therefore distinct signals, but this mathematical distinction does not establish empirical superiority. Appendix C.3/Table 28 reports variance-threshold selection, not an equal-coverage score-gap comparison. We agree that superiority over gap-based selection is not demonstrated. We therefore restrict the claim to variance providing an abstention signal, without asserting that it outperforms score gaps. Comparing largest ∣μA−μB∣|\mu_A-\mu_B|, smallest σA2+σB2\sigma_A^2+\sigma_B^2, and their standardized ratio at identical retained counts is the appropriate direct test.

We appreciate these suggestions and will make the chronology, statistical scopes, and boundaries of the empirical claims explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.