- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VAttnPO refines GRPO's shared sequence advantage using token-level visual attention. Within each update, the frozen old policy teacher-forces the correct answer to construct a visual-patch reference and extracts attention from sampled responses. Jensen–Shannon similarity to the reference is centered within each response, weighted by relative position, and balanced into a zero-mean correction to the sequence advantage. The authors report that the six-image-task average on Qwen3-VL-8B improves from 70.97 with GRPO to 73.50. Behavioral diagnostics and component controls examine the visual reference and credit construction, with further evaluations on document, reliability, video, and cross-backbone tasks.
Critical strengths
-
The method provides a precisely constrained way to use visual correspondence for token credit. Centering and balancing preserve the trajectory-mean advantage while distributing credit according to the answer-conditioned reference. The wrong-answer and shuffled-reference controls in Table 4 support the importance of the reference's content and spatial alignment.
-
The main experiments demonstrate broad task-level improvements under shared training settings. All six image tasks improve over GRPO, with a 2.53-point average gain and a 2.23-point margin over OPD, the strongest compared baseline by average score (Table 1). Shared training data and optimization budgets support this empirical comparison.
Critical weaknesses
-
The empirical benefit over existing visual-credit methods remains unclear. The baselines cover generic RLVR and distillation but omit VPPO [1] and AT-RL [2], which also use token-level visual signals to refine multimodal policy updates. Neither is discussed in the related work. This leaves VAttnPO's additional empirical benefit over this directly relevant class of methods untested.
-
The rationale for monotonically increasing position weights remains incomplete. Figure 1(c) reports a non-monotonic correct–incorrect alignment gap, largest at the terminal token, while Section 4.4 uses an increasing sigmoid. The no-position and position-shuffled controls support the usefulness of position modulation, but do not explain why later positions should systematically receive larger weights. It remains unclear how the diagnostic supports this particular position prior beyond the terminal-token effect.
[1] Spotlight on Token Perception for Multimodal Reinforcement Learning.
[2] Credit Where It's Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning.
Questions for authors
-
How does VAttnPO's answer-conditioned, zero-mean credit correction differ from VPPO and AT-RL? Is a comparison with either method available under the same training data and rollout budget?
-
Why use monotonically increasing position weights when the diagnostic alignment gap is non-monotonic? After excluding the terminal token, do the existing analyses still support stronger modulation of later positions?
I would consider raising my rating if these concerns are satisfactorily addressed.
W1 & Q1: Comparisons with existing visual-credit methods.
We agree that VPPO and AT-RL are previous studies directly comparable to this one. We have also included a comparison with VGPO. VPPO uses token visual dependency for trajectory-level advantage shaping and token-level gradient filtering. AT-RL applies cluster-based soft weighting based on cross-modal connectivity. VGPO combines visual attention compensation with dual-grained advantage re-weighting to counter temporal visual forgetting. In contrast, VAttnPO uses agreement with an answer-conditioned visual reference and relative-position weighting to construct a zero-mean additive correction.
We have supplemented our evaluation with VPPO, VGPO, and AT-RL. All four methods, including VAttnPO, are implemented within the GRPO framework using the same backbone Qwen3−VL−8B−Instruct, training data, and rollout budget.
VAttnPO achieves an AVE of 73.50, exceeding VPPO, VGPO, and AT-RL by 1.73, 1.48, and 1.27 points, respectively, with improvements on all six image tasks. Across the complete evaluation, VAttnPO scores higher than VPPO on all 19 individual metrics. At the reported precision, it exceeds VGPO on 18 metrics and matches it on Video-MME. It also exceeds AT-RL on 16 metrics, matches it on LongVideoBench and Video-MME, and scores 0.03 points lower on DocVQA. These comparisons further support the effectiveness of our visual attention-guided credit assignment.
W2 & Q2: Rationale for relative-position weighting.
The non-monotonic alignment gap in Figure 1c is consistent with our motivation for relative-position weighting. Early CoT tokens may attend to different image regions, and fluctuations in their agreement with the answer-conditioned visual reference are therefore expected. As reasoning progresses, we observe that correct responses tend to become more aligned with this reference. This motivates weaker guidance early and stronger guidance in the latter half of the response.
To examine whether the gains persist when visual credit is confined to reasoning tokens, we additionally evaluate an Only CoT variant. It retains relative-position weighting but applies visual credit correction only within CoT.
Only CoT improves over GRPO by 2.04 points, showing that position-weighted visual credit remains effective when tokens outside CoT receive no visual credit correction. Extending credit assignment to the full completion provides a further 0.49-point improvement. These results establish that the gains persist within the reasoning process itself and are not confined to contributions from the terminal token.