acceptodds
Under review as a conference paper at ICLR 2027

Right Answers for the Right Reasons? Bridging Reasoning Utility and Visual Dependency in Multimodal Reinforcement Learning

Abstract

Reinforcement learning with verifiable rewards (RLVR) has advanced multimodal reasoning, but final-answer supervision cannot distinguish sound reasoning from flawed trajectories that happen to produce correct answers. Our model-based and human assessments reveal substantial perception and reasoning errors among answer-correct responses, motivating feedback that differentiates successful trajectories. We propose Centered-Adaptive Residual Visual Transport (CA-RVT), a design that evaluates reasoning traces using the policy's own answer-confidence probes. CA-RVT measures how each trace changes reference-answer confidence with and without direct image access, relative to trace-free baselines. It combines multimodal reasoning utility with a conservative cross-view residual and adapts the latter to visual dependency across training problems. By reusing the policy for confidence scoring, it requires neither process annotations nor external reward or judge models during training, keeping the post-training pipeline lightweight. With Qwen2.5-VL-7B-Instruct post-trained on only 2.1K Geometry3K problems, CA-RVT achieves competitive performance across mathematical reasoning and general multimodal benchmarks. It attains the highest four-benchmark mathematical average among the compared open-source models with complete results, and the best results on all seven general multimodal benchmarks in our comparison. On questions answered correctly by all audited models, it also achieves lower judged spurious-correctness rates. Additional experiments across training datasets and different backbones further support its applicability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal reasoning, multimodal large language models, reinforcement learning with verifiable rewards

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RTZxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies answer-correct but process-flawed multimodal reasoning trajectories and proposes Centered-Adaptive Residual Visual Transport (CA-RVT). For each generated trace, four no-gradient policy probes measure reference-answer confidence with and without the trace and with and without the image. The reward combines signed image-present utility, a conservative cross-view residual, a visual-dependency gate, and batch-centered adaptive calibration, while retaining outcome and format rewards. Starting from Qwen2.5-VL-7B and training on 2.1K Geometry3K problems, the method improves mathematical and general multimodal results over matched GRPO and reports lower model-judged spurious-correctness rates.

Critical strengths
  1. CA-RVT provides trajectory-sensitive feedback without process annotations or a separate learned reward model. The matched GRPO comparison uses the same initialization, examples, and update budget, and the reported training overhead is modest. Improvements also transfer to another training distribution and another backbone.

  2. The paper provides broad validation: component and aggregation ablations, reward sensitivity, three runs for the full method, free-form transfer, text–image information variants, reasoning-quality audits, and a fixed-pair declaration-removal test. This is substantially stronger evidence than leaderboard comparisons alone.

Critical weaknesses
  1. The policy-confidence signal remains a proxy for answer support, not a verifier of “right reasons.” A trace can increase correct-option likelihood by restating conclusions, option references, or correlated textual cues while containing incorrect perception or inference. Removing explicit answer declarations in Appendix D.5 is helpful, but intermediate calculations and option references remain. The final process-quality claims rely primarily on a Qwen2.5-VL-72B judge; the paper does not provide an independently replicated human audit of the trained models, annotator agreement, or judge-error analysis.

  2. The evaluation provenance requires clarification. The 2.1K training size matches the Geometry3K training split, while MathVerse explicitly contains 507 Geometry3K-derived problems and MathVista also includes Geometry3K as a source. Even if official splits are disjoint, the reported transfer average includes same-source evaluation and no source-excluded analysis. Moreover, GRPO and most component variants are not repeated across seeds; the 0.58-point gain attributed to centered calibration is comparable to the reported 0.5-point spread of the full method.

Questions for authors
  1. Please provide exact/image-level overlap checks between the Geometry3K training set and every evaluation split, and report MathVerse/MathVista results after excluding all Geometry3K-sourced items. MathVerse’s construction documents its Geometry3K subset here, while MathVista lists Geometry3K among its source datasets here.

  2. Can the authors provide a blinded multi-annotator human audit of the final models’ answer-correct traces, including annotator count, agreement, and uncertainty handling, and repeat matched GRPO plus the principal MM/CV/CA ablations across seeds? These results would determine whether the gains reflect process quality and whether the smaller component effects are reliable.

Reviewer_haavOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that many answer-correct rollouts from Qwen2.5-VL-7B on Geometry3K contain perception or reasoning errors: only 62% are judged "ideal" by a 72B judge and 59% by humans. It proposes CA-RVT, an auxiliary GRPO reward for correct rollouts computed with the policy's own answer-confidence probes. The reward has three parts: (1) signed multimodal utility, the change in reference-answer probability when the trace is added with the image present; (2) a cross-view residual, the minimum of the positive gains with and without the image, gated by how much the image helps without a trace; and (3) batch-centered calibration of that residual. Trained on 2.1K Geometry3K problems, the method reaches a 56.13 four-benchmark math average, versus 53.40 for GRPO and 50.60 for the base model. It also improves all seven general benchmarks slightly and lowers judged spurious-correctness rates.

Critical strengths
  1. A consistent, cheap improvement over matched GRPO. Under the same data, base model and update budget, CA-RVT improves GRPO on all four math benchmarks (+2.7 average) and on all seven general benchmarks, with only 15% extra training compute and no external reward model. The three-seed spread of 0.5 points suggests the gain over GRPO is real.
Critical weaknesses
  1. Most of the gain comes from a known reward signal, and the novel components are within noise. MM utility alone, which is essentially the probability-based self-reward of prior work, already accounts for +2.0 of the +2.7 average gain over GRPO. The cross-view residual and calibration add only +0.12 and +0.58, both within the reported 0.5-point seed spread. The cross-view term also contributes at most about 0.04 reward, so the component behind the method’s name has little influence on training. In addition, the closest alternatives (a probability-based self-reward and Vision-SR1) are not compared in the same setting.
  2. The problem may largely stem from the multiple-choice training format, and the free-form baseline is missing. All training problems are multiple-choice, and many “right answer, wrong reasoning” cases depend on the options themselves: picking the closest option, eliminating options, or guessing. Since Geometry3K answers are numeric, GRPO with the options removed and a free-form verifier is a natural, cheap baseline that the paper does not report.
Questions for authors

see above

Reviewer_JZcdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a limitation of outcome-only reinforcement learning for visual reasoning: a correct answer may come from a flawed trace. It proposes CA-RVT, which scores each generated trace by measuring its effect on the model’s confidence in the reference answer, both with and without the image. The reward combines the gain observed with the image, a conservative cross-view bonus, and a factor based on the question’s estimated visual dependence. On four mathematical reasoning benchmarks, the method improves the average accuracy of a Qwen2.5-VL-7B model trained on 2.1K geometry questions from 53.40% with GRPO to 56.13%. The paper also reports ablations, transfer experiments, and an audit of answer-correct traces.

Critical strengths
  1. The paper addresses a real weakness of answer-only rewards. Its audit finds many answer-correct traces with perception or reasoning errors, and the matched-question evaluation attempts to compare reasoning quality while holding question identity and answer correctness fixed.
  2. The matched GRPO comparison is useful. The method improves all four mathematical benchmarks, and the additional training-data and backbone experiments suggest that the result is not confined to a single run setting. The reported 15.1% training-time overhead is also reasonably modest for a reward that requires no separate model at inference.
Critical weaknesses
  1. The methodological contribution is somewhat incremental. Prior work has examined gains in reference-answer probability relative to a no-reasoning baseline, as well as confidence growth and visual dependence in multimodal reasoning. This paper combines these ideas using two probing views, a minimum-based bonus, and batch-relative calibration, but does not include a matched comparison with a closely related confidence-based reward method. In the ablation study, adding calibration to the fixed MM+CV reward raises the four-benchmark average from 55.55% to 56.13%. The paper reports repeated runs for the full method, but not for the ablations, making it difficult to assess how much this component contributes.
  2. The evidence for visual grounding is less direct. A reasoning trace may raise confidence in the correct answer without accurately reflecting the image. The fixed-pair test and removal of answer declarations provide useful checks, but the main comparisons of reasoning quality rely on an automated judge. The reported human assessment focuses on the initial error survey, rather than a matched comparison of the trained models. Testing the models with altered or conflicting visual evidence would provide stronger support for the paper’s right reasons claim.
Questions for authors
  1. Under the same backbone, data, and update budget, how does CA-RVT compare with an adapted PDCR or another confidence-growth baseline? Could the authors also report variation across runs for GRPO and the main reward ablations, particularly for the 0.58-point gain attributed to calibration?
  2. Does a high cross-view reward actually distinguish traces that state correct visual facts from traces that reach the correct answer using mistaken visual facts? An image-counterfactual analysis or a small human-audited matched comparison would help resolve this.
  3. How does the reward behave when Δmm,i\Delta_{\mathrm{mm},i} is greater than, close to, or smaller than Δtxt,i\Delta_{\mathrm{txt},i}? Have the authors analyzed how frequently these cases occur and whether this relative relationship is associated with reasoning quality or final accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.