acceptodds
Under review as a conference paper at ICLR 2027

Region-Level Credit Assignment from On-Policy Rollouts in Multimodal Reinforcement Post-Training

Abstract

Multimodal reinforcement post-training can improve a response's reward without resolving the visual mistakes within it. A shared outcome advantage can reinforce incorrect regional selections alongside correct ones, while omitted objects receive no explicit target in the sampled response. This leaves a gap between optimizing response quality and learning accurate, complete visual grounding. We preent MOVE (Model’s own Visual Errors), a framework that turns the policy's own regional errors into corrective supervision. MoVE contrasts corrected and error-preserving regional conditions to refine credit within a sampled response, and distills corrected targets to supervise objects absent from it. These complementary signals address both incorrect selections and omissions, tying training feedback to specific, verifiable regional mistakes. MoVE provides a unified framework for region-aware multimodal post-training, extending GRPO, DAPO, and OPD with targeted visual correction. Inference still uses only the original image and query. Across five grounding benchmarks, MoVE improves average Micro-F1 in matched GRPO and DAPO comparisons at both 2B and 4B, including an increase from to over DAPO with Qwen3-VL-2B. Corrected-region training benefits three model families, while the extension to visual question answering improves mean accuracy by relative to the base models across four fine-grained benchmarks and two model scales.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal reinforcement learning, visual grounding, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NTiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MOVE (Model’s own Visual Errors), a framework that turns the policy’s own regional errors into corrective supervision. MoVE contrasts corrected and error-preserving regional conditions to refine credit within a sampled response, and distills corrected targets to supervise objects absent from it. MOVE is pluggable and can directly extends on GRPO, DAPO etc.. Across 2B and 4B model families, experiments demonstrates the effectiveness of the proposed approach MOVE.

Critical strengths
  1. The paper introduce a novel approach called MOVE which is a framework that turns the policy’s own regional errors into corrective supervision. MoVE contrasts corrected and error-preserving regional conditions to refine credit within a sampled response, and distills corrected targets to supervise objects absent from it.
  2. MOVE is pluggable and can directly extends on GRPO, DAPO etc..
  3. Across 2 model sizes, MOVE demonstrate performance improvement, although not consistent.
Critical weaknesses
  1. The performance gain is very marginal and not significant and not consistently improving when comparing with baseline such as GRPO etc.
  2. The tested on model families are very small, generalization to bigger model is questionable.
Questions for authors
  1. Could you demonstrate the significance of your methods?
  2. Can the improvement be still gained from a larger model family?
Reviewer_kVjsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate region-level credit assignment in RL post-training. They found that when TP, FP, and FN are aggregated into a single scalar F1 reward within a rollout, conventional outcome advantages tend to falsely encourage FP regions and offer no supervision for FN instances. To solve these questions, the paper presents MoVE, an approach that exploits the verifier's TP, FP, and FN decomposition to create corrected and error-preserving training conditions.

Critical strengths
  1. The paper cleanly separates the mixed-evidence problem into reweighting generated tokens and teaching ungenerated targets. By reusing the verifier's intermediate matching via parallel teacher-forcing passes, it adds only 2.3% training time and zero inference overhead.
  2. The authors rule out simpler heuristic alternatives (direct heuristic TP FP token reweighting, cross-entropy on corrected targets, random negatives), showing that the effectiveness of the proposed methods. Furthermore, Figure A2 proves that the contrast signal separates TP spans from FP spans within the same rollout.
Critical weaknesses
  1. While the results in Table 1 show consistent improvements, the multi seed evaluation in Table A2 reveals that the average gain narrows to +0.35%. Additionally, in the fine-grained VQA experiments, although MoVE achieves strong results on HR-Bench 4k and 8k, it underperforms the Vision-OPD on V* and ZoomBench.
  2. The framework introduces numerous hyperparameters, including residual scale, temperature, distillation weight, negative budget .... But no sensitivity analysis is provided.
Questions for authors

1, The transfer from visual grounding to QA feels somewhat ad-hoc and disconnected from the core grounding formulation. Could the authors elaborate on the design philosophy behind this adaptation, and clarify how the auxiliary regions are verified in VQA. 2, The framework involves several hyperparameters. Could the authors provide a sensitivity analysis on these key hyperparameters to demonstrate the robustness of MoVE? 3, Table 3 compares Base + MoVE only against the untrained Base model across the nine backbones. Could the authors provide matched Base + GRPO baseline results to clearly isolate MoVE's specific performance gains over standard GRPO?

Reviewer_mptwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoVE, a framework for region-level credit assignment in multimodal reinforcement post-training. It uses verified errors from on-policy grounding rollouts to refine token-level advantages and distill corrected targets for missed regions. Experiments show improvements over GRPO and DAPO across multi-object grounding and fine-grained VQA benchmarks without additional inference-time inputs.

Critical strengths
  • This paper propose an interesting design, MoVE.
  • This paper is well-written and easy to follow.
  • This experiments are conducted on four settings with different seeds.
Critical weaknesses
  • Potential target leakage in token-level evidence contrast. The positive and negative conditions are serialized bounding-box sequences that substantially overlap with the model output, and a negative condition may directly reproduce the error-containing rollout being evaluated. Consequently, the likelihood difference may primarily measure output–hint consistency or coordinate copying rather than genuine visual correctness. The paper lacks controls that remove or perturb the coordinate hints to isolate whether the resulting credit signal reflects visual grounding.
  • Potential privileged-target leakage in corrected-region self-distillation. The teacher is conditioned on B+B^+, which already contains the corrected bounding boxes and is then serialized into the target sequence z+z^+. The resulting KL objective may therefore distill coordinate copying from an answer-revealing condition rather than improved visual localization. The paper does not include controls that mask or perturb the coordinates in B+B^+ to isolate the source of the transferred knowledge.
  • Partially off-policy supervision for missed regions. Although B+B^+ is constructed from an on-policy rollout, it forcibly inserts annotated false-negative boxes that were never generated by the current policy. The distillation loss is consequently evaluated along corrected prefixes z+z^+ that may lie outside the policy’s visited sequence distribution. While this does not invalidate the on-policy RL estimator itself, it makes the auxiliary branch partially off-policy and may introduce distribution mismatch or exposure bias; the paper provides limited analysis beyond teacher-confidence filtering.
Questions for authors

Please kindly refer to the weakness section.

1 comment

Thank you for the detailed response and additional ablations. The results provide useful evidence that EC and CRSD improve performance beyond the tested alternatives. However, they do not fully rule out that the gains arise partly from coordinate-hint consistency or answer-revealing privileged supervision, rather than genuinely visual credit assignment. The clarification regarding the partially off-policy auxiliary objective is reasonable, though the associated distribution mismatch remains insufficiently analyzed. Therefore, my concerns are partially resolved, and I will maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.