acceptodds
Under review as a conference paper at ICLR 2027

Eyes in the Update? Learning Where Rewards Land in Multimodal Reasoning

Abstract

Multimodal post-training has moved from broad instruction tuning toward reinforcement learning for image and video reasoning, where outcome rewards can improve final-answer accuracy. Yet these rewards are assigned to complete generated trajectories, while visual evidence is used only at sparse and uneven positions inside the response. A growing line of work has therefore begun to make multimodal RL more token-sensitive through grounding signals or external token-level weighting. We ask a more specific question: can the allocation of the same trajectory-level reward be inferred from the model's own multimodal computation, rather than from an auxiliary annotator, perception module, or process label? We propose Gaze-RL, a representation-driven policy optimization method that redistributes completion-level advantage through a latent reward landing field. The field is computed from internal cross-modal geometry and local predictive uncertainty, assigning larger update weight to positions that are both visually anchored and decision-bearing while remaining mean-one, so the total credit of each rollout is preserved and only its positionwise allocation changes. Gaze-RL requires no process supervision, auxiliary perception module, or architectural modification. Trained on a mixed image-video RL dataset, Gaze-RL improves Qwen2.5-VL and InternVL3.5 across broad image and video benchmarks. It also yields stronger evidence sensitivity and shorter, more concentrated reasoning dynamics. These results suggest that multimodal RL benefits from learning how existing rewards should land inside the reasoning trajectory.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Reinforcement Learning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_BLRFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that GRPO gives every token in a response the same credit, even though only a few tokens actually use visual evidence. It proposes GAZE-RL, which redistributes each response's advantage across tokens. Tokens whose hidden states are similar to the image features, and that are uncertain decision points, receive more credit, while the total credit per response stays the same. Experiments on Qwen2.5-VL and InternVL3.5 report gains over GRPO, DAPO, VPPO, and Video-R1 on image and video benchmarks, with additional analyses of where the credit lands.

Critical strengths
  1. Insightful analyses. Three analyses stand out: layer selection validated against a behavioral visual-dependency score with a shuffled-visual control (Table 12); fixed-checkpoint masking with attention and shuffled-landing controls (Table 17); and a 400-rollout audit of where negative-advantage mass lands (Table 16). They give real evidence about what the landing field selects.
  2. Practical. The method needs no external annotator or perception module, adds small overhead, and the code is released.
Critical weaknesses

1. The core method is the same as an uncited prior method. PEPO (arXiv 2603.22847, March 2026) already reweights token advantages by combining the similarity between response and image hidden states with token entropy. It uses the same normalization and the same blending with uniform credit, and GAZE-RL's main equations follow the same structure. The only differences are that entropy is additionally gated by a visual support term and only selected layers are used instead of all layers. PEPO is neither cited nor compared. This is consequential because the paper's main claimed contribution already exists, so its novelty narrows to the visual support gate.

2. The visual support gate is not shown to matter, and the headline evidence is selectively averaged. In the paper's own ablation, removing the visual-support gate (plain entropy weighting) costs only about 0.3 points on images and 0.9 on video, in a single run and within the range of the hyperparameter sweeps. The three-seed significance test averages only the four image benchmarks with the strongest gains. It also leaves out NExT-QA, where all RL methods fall below the base model. On InternVL3.5-8B Video-MME, GAZE-RL is 7–9 points below every RL baseline. This is consequential because, without evidence that the gate helps, the paper shows no contribution beyond PEPO.

Questions for authors
  1. How does GAZE-RL differ from PEPO, whose weighting matches yours almost exactly? Please provide a head-to-head comparison under identical training.
  2. Can you report multi-seed results for plain entropy weighting vs. the full method, and recompute the significance test over all benchmarks (including NExT-QA and the general image set)? Is the InternVL3.5-8B Video-MME entry (67.0 vs. 73.9–75.6) correct?
Reviewer_ca6uOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

It proposes a token-level weighting method for GRPO that redistributes advantage across response tokens while keeping the average weight at 1. The weights combine visual similarity between hidden states and predictive entropy, with additional support from earlier visually grounded tokens. The method is trained for 200 updates on ViRL39K and Video-R1 and evaluated on three VLM backbones across 14 image and 6 video benchmarks. The appendix includes Gemma-3-4B experiments, three-seed comparisons, and additional diagnostics.

Critical strengths
  1. Broad evaluation with useful diagnostics. The experiments cover three backbones and 20 benchmarks. The appendix validates visual anchoring against a counterfactual dependency measure, compares token rankings at fixed checkpoints, and reports three-seed results with confidence intervals. These analyses strengthen the empirical evidence.

  2. Low computational overhead. The method requires no additional models or annotations and achieves similar training time to GRPO with only 0.4 GB extra peak memory, making it practical to adopt.

Critical weaknesses
  1. The improvements are not consistent, and the closest baseline is incompletely evaluated. On Video-MME, InternVL3.5-8B scores 67.0, below GRPO (73.9), DAPO (75.6), and Video-R1 (74.7). On NExT-QA, both Qwen models underperform their base checkpoints (71.7 vs. 72.5 and 67.2 vs. 67.7). These failures are not discussed. Most gains over DAPO are below 1 point, while VPPO, the closest related baseline, is missing from the video evaluation and three-seed comparisons.

  2. The benefit of the visual-support coefficient ζtζ_t is unclear. This component introduces O(T2)O(T^2) computation and three additional hyperparameters without sensitivity analysis. Yet removing it and using plain entropy changes the image average only from about 60.8 to 60.5 and the video average from 54.9 to 54. Without repeated runs, it is difficult to establish whether this complex component meaningfully improves performance.

Questions for authors

Clarifications That Could Change My Assessment:

  1. Why does InternVL3.5-8B score only 67.0 on Video-MME when all baselines exceed 73.9? GRPO also improves 9.4 points over the base model in just 200 updates. Were frame sampling and prompts identical across methods?

  2. Could the authors provide three-seed results for the "entropy w/o visual support" variant and VPPO? This would help determine whether the contribution of ζtζ_t exceeds seed variation.

  3. The token-weighting mechanism closely resembles PEPO (arXiv:2603.22847), including layer-averaged similarity, normalized similarity and entropy, tanh-based weighting, and mixing with uniform weights. The main differences appear to be ζtζ_t, layer selection, and a fixed rather than scheduled γγ. Could the authors clarify the distinction? This arXiv work did not affect my score.

Suggestions:

  1. Since token weights are computed from the policy being trained, the increase in top-20% weight mass from 20% to about 44% (Fig. 2) may reflect representation drift rather than better credit assignment. Computing weights with a frozen reference model could help distinguish these effects.

  2. Table 5 has missing entries for training steps and maximum new tokens. The image-editing experiment in Fig. 6 lacks quantitative evaluation.

1 comment

Thank you for your detailed and thoughtful responses. After carefully reading the responses, I believe my questions have been adequately addressed. I have no further concerns and will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.