acceptodds
Under review as a conference paper at ICLR 2027

Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning

Abstract

Reinforcement learning with verifiable rewards scales multimodal reasoning, but an outcome reward says how much a trajectory is worth, not how that value should be spread over the decisions that produced it. We introduce Residual Visual Credit Optimization (RVCO), which treats token credit as a conserved routing problem. A controlled visual intervention yields a per-token evidence response; robust within-trajectory coordinates remove incidental scale; and a budgeted entropic router distributes a fixed amount of sequence utility according to perceptual dependence. A residual support path guarantees positive credit at every valid position, and an analytic correction restores the prescribed credit mass exactly. The resulting field is selective, bounded, full-support, and invariant to response-local score shifts, and recovers hard token selection as a limiting case. Across four model families and seven reasoning benchmarks, RVCO improves accuracy over strong RLVR baselines while maintaining late-stage optimization stability, corruption robustness, and competitive training cost. Rewards, rollouts, and the group-relative advantage estimator are unchanged; only the geometry of token-level credit differs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal reinforcement learning, RLVR, token credit assignment, visual grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_X5wkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies token-level credit assignment in multimodal RLVR. The authors propose Residual Visual Credit Optimization (RVCO), which uses visual perturbations to identify image-dependent tokens and redistributes the sequence-level learning signal toward them under a fixed credit budget. RVCO combines entropic routing with a residual path that preserves nonzero credit for all tokens. Experiments across multiple VLMs and benchmarks show consistent, though generally modest, improvements over prior multimodal RLVR baselines.

Critical strengths
  1. I think the method is clearly formulated. RVCO cleanly separates credit budget, routing sharpness, and support preservation.

  2. The evaluation is quite broad, covering multiple VLM families, model scales, reasoning benchmarks, and robustness settings.

  3. The training overhead is modest, and I think acceptable. RVCO leaves the reward and advantage estimator unchanged and introduces only modest training overhead.

Critical weaknesses
  1. The methodological novelty is kind of limited. The core routing operator is based on an existing LML projection, while the main contribution lies in combining several relatively standard components.

  2. The gains over the strongest baselines are modest and the main evaluation relies on accuracy@8. Given that RL typically improves the pass@k performance of the model, I think acc@8 is not enough to clearly establish whether RVCO improves the underlying policy or mainly increases response diversity.

  3. The conservation ablation is not fully controlled. Replacing exact mass conservation also changes the overall scale of the token weights, so the observed gain cannot be attributed to conservation alone.

  4. Furthermore, the visual credit signal is only indirectly validated. Sensitivity to a perturbed image does not necessarily imply that a token is meaningfully visually grounded. I hope the authors could provide more detailed and direct validation of the visual credit signal.

Questions for authors
  1. Could the authors report multi-seed results with variance estimates?

  2. Does RVCO still outperform the baselines under accuracy@1 or greedy decoding?

  3. Is there a direct evaluation showing that high-credit tokens correspond to visually grounded decisions?

  4. Could the authors provide a scale-matched ablation for exact conservation? The current unit-mean variant also changes the global scale of the token weights by T/mT/m, making it unclear whether the performance difference comes from conservation itself or from a different effective update magnitude.

2 comments

We thank reviewer X5wk for recognizing RVCO's explicit formulation, broad evaluation, and modest training overhead.

W1 (Novelty beyond standard components).

RVCO's novelty lies in the constrained credit geometry, not the LML solver itself. It converts intervention-based evidence into a jointly optimized, budget-preserving, full-support token field while leaving the verifier and advantage estimator unchanged. The matched comparisons in reviewer 7qTe-W1 establish gains beyond conventional perception-aware weighting at nearly identical update magnitudes.

W2/Q1–Q2 (Accuracy@8 and statistical reliability).

RVCO improves response correctness, not merely best-of-eight success. Three independent Qwen3-VL-4B training seeds yield 44.2 ± 0.3 versus 42.8 ± 0.6 sampled pass@1, alongside 45.4 ± 0.4 versus 44.1 ± 0.5 greedy accuracy. Acc@8 improves from 61.9 ± 0.4 to 63.9 ± 0.3. The 8B comparison independently shows consistent single-response gains.

These results distinguish training-seed variability from decoding variance and demonstrate improvements beyond response diversity.

W3/Q4 (Scale-matched conservation ablation).

Conservation fixes the allocation budget; the observed improvement is not explained by gradient amplification. Scale-matched unit-mean normalization is mathematically identical to RVCO, as shown in Appendix A.6.

The controlled 8B experiment in reviewer 7qTe-W2 instead compares distinct allocation rules under identical total credit mass. RVCO achieves 66.8 Acc@8 with a 1.04 relative pre-clip norm, versus 65.5/1.02 for shuffled weights and 63.0/1.05 for entropic-only routing. Similar update scales but different performance directly support structured credit allocation.

W4/Q3 (Direct visual-credit validation).

Independent annotations establish that high-credit positions preferentially correspond to verified visual decisions. Across 200 held-out trajectories, annotated visual facts receive 1.49× response-mean credit, compared with 0.84× for nonvisual spans. Removing their supporting regions produces a 0.31-nat/token likelihood gap versus 0.04 for nonvisual reasoning.

This separates meaningful evidence dependence from generic image corruption sensitivity and provides external validation of the credit signal without introducing token-level annotations into training.

I thank the authors for their detailed rebuttal and additional experiments, which address several of my empirical concerns. However, my concern regarding the methodological novelty still remains, as the proposed approach largely builds on existing techniques. I appreciate the authors' efforts but I will maintain my original rating.

Reviewer_SjJxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RVCO for token credit assignment in multimodal reinforcement learning. Specifically, it computes token probabilities under original and masked images, then assign token weights. It gives more weight to visually dependent tokens, while keeping a small weight on the other tokens. Experiments across several model families report gains over existing RL methods, with additional tests on robustness and training stability.

Critical strengths
  1. The proposed RVCO focuses updates on visually dependent tokens while keeping the whole reasoning response involved in training. This design is reasonable.

  2. Tables 1 and 2 show gains across different model sizes and families. This makes the results more convincing.

  3. Table 4 reports better corruption robustness with only a small increase in training time. This is useful for practical training.

Critical weaknesses
  1. As shown in lines 205–221, RVCO uses probability changes after image masking to estimate token importance. This may capture sensitivity to image corruption rather than useful visual information, and the paper does not discuss this.

  2. As shown in lines 238–264, the method combines entropy-based weighting, residual mixing, and normalization. Therefore,the novelty of the method appears limited. The authors may further clarify the novelty.

  3. Moreover, in Table 1, RVCO improves over VPPO by 1.1 points on both the 8B and 32B models. So the practical value and reliability of these gains remain unclear.

Questions for authors

Please refer to the weakness.

1 comment

We thank reviewer SjJx for recognizing the clear presentation, cross-model improvements, and practical efficiency.

W1 (Visual grounding versus corruption sensitivity).

RVCO measures decision-specific visual dependence, not arbitrary corruption magnitude. Equation (2) fixes the sampled token, question, and prefix while intervening only on the image. The resulting evidence determines credit allocation; the verifier's advantage still determines the direction of reinforcement.

We additionally audit 200 held-out Geo3K/MathVerse trajectories with independent, blinded visual-fact and region annotations. Equal-area evidence/background interventions are evaluated on identical frozen responses:

Annotated spanNormalized credit ↑Evidence-mask gap ↑
Verified visual facts1.490.31
Nonvisual reasoning0.840.04

Credits are normalized by response-mean weight; gaps are in nats/token. Verified visual decisions receive 1.77× the relative credit and exhibit substantially greater evidence-specific sensitivity. The annotation-independent comparison supports meaningful visual attribution beyond generic masking sensitivity.

W2 (Limited novelty).

The contribution is joint, conserved evidence routing rather than independently smoothed token weighting. The shared-budget constraint introduces cross-token competition, while residual correction preserves full-support credit without increasing mass. The mechanism-matched comparison in 7qTe-W1 shows RVCO outperforming all four adapted perception-aware alternatives at comparable update magnitudes.

W3 (Modest gains and reliability).

The improvements extend beyond Acc@8 and a single training run. Three independent 4B seeds yield +1.4 pass@1, +1.3 greedy, and +2.0 Acc@8 over VPPO. The 8B checkpoints show the same positive direction across all three metrics. These results reinforce the broader cross-backbone and corruption-robustness findings in Tables 1–4.

Comments on the paper

1 comment

We thank reviewer 7qTe for recognizing the well-motivated formulation, extensive evaluation, and compatibility with standard RLVR.

W1/Q3 (Methodological novelty and matched baselines).

RVCO is a constrained credit-allocation formulation, not a new projection primitive. Although its entropic router connects to LML, RVCO jointly enforces evidence-dependent competition, positive support, and exact credit conservation. Unlike independent gates, the shared budget couples token allocations:

=-\frac{q_tq_s}{\tau\sum_r q_r}<0, \quad s\neq t, \qquad q_t=g_t(1-g_t).$$ We further compare perception-aware credit rules within an identical 8B policy objective, with matched mass and comparable update magnitudes. PAPO's auxiliary objective is adapted to token coefficients. | Credit rule | Acc@8 ↑ | Relative update norm | |---|---:|---:| | PAPO-derived | 65.2 | 0.97 | | PGPO-derived | 65.9 | 1.01 | | PRPO-derived | 65.6 | 0.99 | | VPPO-derived | 65.8 | 1.02 | | **RVCO** | **66.8** | **1.00** | RVCO retains a **+0.9-point advantage** over the strongest matched credit rule, without increased update magnitude. > **W2/Q1 (Conservation versus gradient scaling).** **Fixed credit mass does not imply fixed gradient norms; RVCO's gains arise from structured allocation rather than update amplification.** Appendix A.6 establishes $w_t^{\mathrm{mean}}=(T/m)w_t^{\mathrm{RVCO}}$; scale-matching these two fields makes them algebraically identical. We therefore compare genuinely different allocations under the same budget $m_i=\lceil\rho T_i\rceil$, measuring pre-clip gradient norms on matched frozen rollouts: | Allocation | Acc@8 ↑ | Pre-clip norm | |---|---:|---:| | Budget-matched uniform | 65.3 | 1.00 | | Shuffled RVCO | 65.5 | 1.02 | | Independent sigmoid | 64.5 | 1.04 | | Entropic ($\beta=0$) | 63.0 | 1.05 | | **RVCO** | **66.8** | **1.04** | **Shuffling preserves weight mass and distribution yet loses 1.3 points.** Removing residual support reduces accuracy by 3.8 points despite a slightly higher pre-clip gradient norm. These controls support the benefits of evidence-aligned routing and residual support beyond mere gradient scaling. > **W3/Q2 (Residual support and ranking invariance).** **Residual support preserves rankings but changes credit magnitudes and optimization geometry.** Equation (6) is strictly increasing in the entropic gate; hence identical frozen evidence necessarily yields identical top-$\rho$ sets, including their overlap under paired interventions. Thus, the differing overlaps in Appendix Table 8 cannot represent residual-induced rank changes under identical evidence. RVCO's benefit instead lies in guaranteed positive credit and attenuated allocation sensitivity: $$0<w_t\leq1,\qquad \|J_w\|_2\leq\frac{c(1-\beta)}{4\tau}.$$ The controlled $\beta=0$ comparison above directly supports the contribution of residual support to training performance. > **W4/Q4 (Independent seeds and pass@1).** **The improvements persist across independent training seeds and single-response evaluation.** We train 4B models with three paired independent seeds and separately evaluate sampled pass@1 and greedy accuracy. The existing 8B checkpoints provide a same-scale comparison. | Backbone / runs | Method | Pass@1 ↑ | Greedy ↑ | Acc@8 ↑ | |---|---|---:|---:|---:| | 4B / 3 | VPPO | 42.8 ± 0.6 | 44.1 ± 0.5 | 61.9 ± 0.4 | | 4B / 3 | **RVCO** | **44.2 ± 0.3** | **45.4 ± 0.4** | **63.9 ± 0.3** | | 8B / 1 | VPPO | 46.3 | 47.5 | 65.7 | | 8B / 1 | **RVCO** | **47.1** | **48.2** | **66.8** | Values for 4B are mean ± sample standard deviation across training seeds. Sampled pass@1 averages correctness over eight sampled responses per question. RVCO improves both single-response accuracy and Acc@8, with consistent gains across independent 4B runs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.