- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies answer-correct but process-flawed multimodal reasoning trajectories and proposes Centered-Adaptive Residual Visual Transport (CA-RVT). For each generated trace, four no-gradient policy probes measure reference-answer confidence with and without the trace and with and without the image. The reward combines signed image-present utility, a conservative cross-view residual, a visual-dependency gate, and batch-centered adaptive calibration, while retaining outcome and format rewards. Starting from Qwen2.5-VL-7B and training on 2.1K Geometry3K problems, the method improves mathematical and general multimodal results over matched GRPO and reports lower model-judged spurious-correctness rates.
Critical strengths
-
CA-RVT provides trajectory-sensitive feedback without process annotations or a separate learned reward model. The matched GRPO comparison uses the same initialization, examples, and update budget, and the reported training overhead is modest. Improvements also transfer to another training distribution and another backbone.
-
The paper provides broad validation: component and aggregation ablations, reward sensitivity, three runs for the full method, free-form transfer, text–image information variants, reasoning-quality audits, and a fixed-pair declaration-removal test. This is substantially stronger evidence than leaderboard comparisons alone.
Critical weaknesses
-
The policy-confidence signal remains a proxy for answer support, not a verifier of “right reasons.” A trace can increase correct-option likelihood by restating conclusions, option references, or correlated textual cues while containing incorrect perception or inference. Removing explicit answer declarations in Appendix D.5 is helpful, but intermediate calculations and option references remain. The final process-quality claims rely primarily on a Qwen2.5-VL-72B judge; the paper does not provide an independently replicated human audit of the trained models, annotator agreement, or judge-error analysis.
-
The evaluation provenance requires clarification. The 2.1K training size matches the Geometry3K training split, while MathVerse explicitly contains 507 Geometry3K-derived problems and MathVista also includes Geometry3K as a source. Even if official splits are disjoint, the reported transfer average includes same-source evaluation and no source-excluded analysis. Moreover, GRPO and most component variants are not repeated across seeds; the 0.58-point gain attributed to centered calibration is comparable to the reported 0.5-point spread of the full method.
Questions for authors
-
Please provide exact/image-level overlap checks between the Geometry3K training set and every evaluation split, and report MathVerse/MathVista results after excluding all Geometry3K-sourced items. MathVerse’s construction documents its Geometry3K subset here, while MathVista lists Geometry3K among its source datasets here.
-
Can the authors provide a blinded multi-annotator human audit of the final models’ answer-correct traces, including annotator count, agreement, and uncertainty handling, and repeat matched GRPO plus the principal MM/CV/CA ablations across seeds? These results would determine whether the gains reflect process quality and whether the smaller component effects are reliable.