- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a multimodal reward model named Delta-Vision Reward (DVR), capable of predicting the mean reward, the variance reflecting the degree of disagreement among four teacher models, and structured reasoning rationales. Its training process involves Gaussian distribution imitation learning supervised by reasoning rationales, followed by pairwise reward fine-tuning targeting response pairs with ranking errors or insufficient discriminability. The study's primary contribution lies in integrating distributional supervision, explanatory feedback, and targeted fine-tuning into multimodal reward modeling. Validation efforts encompass two reward benchmarks, component ablation studies, selective prediction, controlled visual occlusion experiments, independent review panel assessments, and human-evaluated response revisions based on equalized budgets.
Critical strengths
The visual masking experiment shows that the model responds to information needed for the task. With the question and responses held fixed, masking a relevant image region increases predicted variance by 0.93, compared with only 0.08 when masking an equally sized irrelevant region. This suggests that the model reacts to missing visual evidence, rather than simply to any change in the image.
The feedback experiment shows practical benefits under a fair comparison. All feedback conditions use the same base model, one revision, and the same token limits. Under human evaluation, full DVR feedback achieves an 81.3% win rate against the original responses, compared with 74.8% for rationale-only feedback. This supports the usefulness of the complete feedback.
Critical weaknesses
- The method has limited novelty. Learning reward distributions to represent uncertainty has been explored in prior work [1]. Using teacher-generated rationales as additional supervision is also established [2], as is selecting informative samples to refine reward models [3]. DVR combines these ideas through Gaussian fitting of teacher scores, rationale distillation, and pairwise training on ranking errors or low-margin pairs. Although its offline sample-selection rule differs from [3], it remains a straightforward form of hard-example selection. Moreover, the refinement stage uses reward means and margins rather than the predicted variance. The contribution therefore appears mainly to be an integration of existing ideas for multimodal reward modeling, with limited methodological innovation.
2.There are numerical inconsistencies in some of the uncertainty analysis results, which undermine confidence in the associated conclusions. In Table 26, if the accuracy rates for each interval are weighted by sample size, the calculated overall accuracy is approximately 77.25%, rather than the 72.46% reported. The same table indicates that at a variance threshold of 1.0, the coverage rate should be 59.57%, whereas Table 28 reports a value of 68.1%. Similarly, the overall accuracy derived from the grouped results in Table 30 is approximately 75.59%, not 72.46%. These discrepancies are too large to be attributed solely to rounding errors. The authors should clarify whether these tables are based on different runs, sample sets, or aggregation rules, and provide corrected, consistent results.
[1] Sheng Xu, Bo Yue, Hongyuan Zha, and Guiliang Liu. A Distributional Approach to Uncertainty-Aware Preference Alignment Using Offline Demonstrations. ICLR, 2025. [2] Cheng-Yu Hsieh et al. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Findings of ACL, 2023. [3] Zexu Sun et al. Uncertainty and Influence aware Reward Model Refinement for Reinforcement Learning from Human Feedback. ICLR, 2025.
Questions for authors
Please clarify the numerical inconsistencies in Tables 26, 28, and 30. Weighting the group accuracies by their sample counts gives approximately 77.25% in Table 26 and 75.59% in Table 30, but both tables report 72.46% overall accuracy. Table 26 also implies 59.57% coverage at a variance threshold of 1.0, whereas Table 28 reports 68.1%. Do these tables use different data, model runs, or calculation methods? Please explain the differences and correct any errors.
Does filtering by predicted variance work better than filtering by the score difference between two responses? Please compare the two methods while keeping the same percentage of samples, and report accuracy on the retained samples. This would show whether predicting variance provides an advantage over simply retaining pairs with larger score differences.
We thank you for recognizing the visual-grounding and feedback controls. We address novelty, statistical consistency, and score-gap selection below.
1. Contribution relative to prior work and the role of variance
We agree that Gaussian modeling, rationale distillation, and hard-example selection are not individually new. Secs. 1–2 and 5 position DVR as an integrated multimodal reward model with empirical analysis, not a new distribution family or sampling principle.
[1] uses distributional Bellman updates and CVaR for preference alignment; DVR distills image-conditioned, response-level teacher-score moments. [2] establishes rationale supervision, which we use rather than claim to originate. [3] selects policy-interaction samples using ensemble uncertainty and gradient influence; DVR uses offline student–teacher ranking/margin mismatch (Sec. 3.4, Eqs. (7)–(9)). We will sharpen these distinctions and cite [2].
Teacher–teacher score dispersion supervises Stage I; student–teacher ranking/margin mismatch selects Stage II pairs. The latter does not use predicted variance, as correctly noted. We will distinguish these meanings of “disagreement,” not describe Stage II as variance-driven.
Does distributional supervision contribute beyond refinement? Sec. 4.4/Table 4 and Appendix J.5/Table 91 report VL-Reward accuracy:
Distributional supervision adds 1.70/1.46 percentage points before refinement and 1.16/1.25 under the same refinement procedure. Thus, its contribution is not subsumed by hard-pair training, although these comparisons do not isolate disagreement information from every auxiliary-supervision effect.
The empirical contribution extends beyond teacher imitation. Appendix J.3/Table 89 gives independent-panel pairwise/hard-case agreement of 74.8/61.5 (mean-only), 76.9/65.8 (distributional SFT), and 79.6/70.4 (full DVR). The visual intervention (Sec. 4.5/Table 6; J.1/Table 87) and controlled human feedback evaluation (Sec. 4.6; J.8/Table 98), recognized in the review, further support the integrated model's utility—not novelty of its components.
2. Tables 26, 28, and 30: statistical scopes must be reconcilable
Thank you for checking the arithmetic. Appendix A.2.1/Eq. (10) defines sample-averaged accuracy; C.2.1/Eq. (22) defines variance bins; C.3.1/Eq. (23) defines coverage. Under the same sample population, run aggregation, and variance statistic, the following identities must hold:
\mathrm{Coverage}(1)=\frac{n_{[0,0.5)}+n_{[0.5,1)}}{N}.$$ The printed rows imply: | Calculation | Implied by printed rows | Reported summary | |---|---:|---:| | Table 26 weighted accuracy | 77.25% | 72.46% | | Table 26 coverage at 1 | 59.57% | 68.1% in Table 28 | | Table 30 weighted accuracy | 75.59% | 72.46% | These differences cannot be attributed to rounding. Our current presentation does not sufficiently document the cross-table statistical-scope mapping. We will explicitly identify each analysis's population, run aggregation, response-to-pair variance definition, and denominator, and reconcile the corresponding counts and summaries. A different statistical scope is a valid explanation only if the resulting numbers can be reconstructed; merely labeling the discrepancy a scope difference is insufficient. These are arithmetic checks, not replacement benchmark results. Pending reconciliation, we do not rely on the disputed selection/calibration summaries or assume all associated conclusions are unaffected. The separately reported ablations, interventions, and external evaluations provide complementary evidence. ### 3. Variance versus score gaps at equal coverage We agree that Appendix C.3/Table 28 does not answer this comparison: it reports variance-threshold selection, not a matched-coverage score-gap baseline. We therefore do not claim demonstrated superiority over gap-based selection. Sec. 3.5 uses mean scores for ranking and independent-Gaussian confidence for reliability. Writing $g=|\mu_A-\mu_B|$ and $u=\sigma_A^2+\sigma_B^2$, that confidence is $\Phi(g/\sqrt{u})$. Equal gaps can have different predicted reliability, but this mathematical distinction is not an empirical comparison. The direct test is to retain exactly the same number of pairs from identical predictions using largest $g$, smallest $u$, and largest $g/\sqrt{u}$, then compare retained accuracy and risk–coverage curves. Such results are not included in the reported experiments. We will narrow the claim accordingly: DVR supplies a learned disagreement signal, while its incremental selective-prediction value over score gaps remains to be established. We appreciate these suggestions and will clarify contribution boundaries and statistical reporting.