- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AT-RL, a lightweight, plug-and-play reinforcement learning algorithm that addresses variation in token-level dependence on visual information during multimodal reasoning. It uses METIS to group tokens by the similarity of their visual attention distributions, then redistributes the response-level advantage to individual tokens according to cluster connectivity strength. Experiments on the Qwen2.5-VL series (3B–32B), with comparisons against several classic reinforcement learning algorithms, demonstrate clear advantages for AT-RL. Combining it with algorithms such as GSPO and SAPO yields further gains.
Critical strengths
- The paper demonstrates the method's effectiveness with several competitive reinforcement learning algorithms (e.g., improving SAPO from 49.94 to 53.25). These results also establish its plug-and-play nature: the gains from identifying anchor tokens and redistributing credit can complement those of other reinforcement learning algorithms.
- Variation in how different tokens depend on visual information is a real and important issue in multimodal reasoning. The authors present a clear motivation that connects to their method design.
Critical weaknesses
- The models used are too outdated to establish whether the conclusions hold for contemporary frontier models. The algorithm is primarily evaluated on Qwen2.5-VL. Although Appendix Table 13 includes Qwen3-VL results, this model still lags behind more recent model series, and the table covers only three benchmarks, omitting GeoQA and WeMath. I am concerned about the method's transferability to the latest models (e.g., Qwen3.5).
- Several unclear descriptions make the paper difficult to follow. For example:
- The method section contains omissions in notation and explanation, including the undefined in Section 3.1 and the unspecified computation of in Eq. (14).
- The main text claims to use six mathematical benchmarks but lists only five: MathVerse, MathVista, MathVision, GeoQA, and InterGPS.
- AT-RL uses METIS in terms of efficiency and edge-cut minimization, but the empirical justification for this specific choice remains limited.
Questions for authors
- Could the authors report the main results on the latest models, such as Qwen3.5-9B with GRPO + AT-RL? This would help strengthen the evidence for the method's effectiveness.
- Could the non-uniform reweighting of token-level advantages in AT-RL destabilize RL training? Please provide training diagnostics such as policy entropy, KL divergence, reward evolution, and gradient norms.
- The discussion states, “Credit Goes to Anchors — Whether Right or Wrong.” However, errors may arise from formatting, language, or mathematical reasoning, as well as from visual input. What motivates this design?
7 comments
Response to Reviewer MrMj (Part 2 of 2)
Q3: About “Whether Right or Wrong.”
The phrase refers to applying the same allocation rule to positive and negative advantages. The task outcome, processed by the base RL algorithm, determines the advantage and thus the learning direction. Visual connectivity determines , which allocates that signal across tokens. In , positive advantages reinforce the corresponding generation patterns, while negative advantages suppress them.
The design uses visual-grounding information to improve multimodal credit allocation while retaining soft-weighted updates for other tokens, including language, formatting, and mathematical reasoning. Anchors are defined by cross-modal connectivity and can carry reasoning content tied to visual conditions. In Table 4, soft weighting achieves 45.27, compared with 42.58 when updating only the top 15% of tokens, supporting the combination of stronger grounding and continued contributions from other tokens.
Section 6 and Appendix Discussion 4 state these roles explicitly: the outcome determines the update direction, connectivity determines allocation, and soft weighting retains other tokens.
Revisions. Section 6: p. 10, lines 521–528; Appendix A, Discussion 4: p. 16, lines 810–817; advantage weighting: p. 7, lines 334–344; weighting-control interpretation: p. 25, lines 1313–1319.
Q4: About Notation and the Benchmark Inventory.
We apologize for the notation omissions and the typographical errors in the benchmark count and name; these are now corrected. Section 3.1 defines the keys indexed by in the attention denominator and the calibration parameters . We have also completed the definitions of connectivity , initial importance , centroid-attenuated importance , propagated importance , and the final multiplier . Appendices B.2–B.3 specify initialization, centroid computation, attenuation, and neighborhood updates step by step. Appendix B.7 gives the final normalization, and Algorithm 1 connects the complete pipeline. Sections 4.2–4.3 retain a concise explanation and point to these computational details.
The benchmark inventory now correctly states five image-based mathematical-reasoning benchmarks: MathVerse, MathVista, MathVision, GeoQA, and WeMath. InterGPS was a naming error in the evaluation list and has been corrected to WeMath, matching the result tables. Geometry-3K, used for training, is identified separately in the training-data paragraph.
Revisions. Section 3.1 notation: p. 4, lines 184–190; main-text algorithm explanation: p. 6, lines 310–314, and p. 7, lines 324–341; detailed computations and notation: p. 17, lines 864–917; normalization and complete pseudocode: p. 18, lines 918–965; benchmark inventory: p. 7, lines 374–377.
Q5: About the Choice of METIS.
Fit to the graph representation. AT-RL constructs a sparse, undirected, weighted token graph whose edges measure similarity between calibrated visual-attention footprints. METIS (Karypis and Kumar (1998)) minimizes cross-group edge weight under balance constraints, grouping tokens with similar visual attention. Its multilevel coarsening and refinement suit the graph-partitioning step performed once per rollout. We have clarified this correspondence in Section 4.2 and Appendix L.3.
Grouping ablation. In Table 6, Raw Attention Weighting obtains 42.35 and 63.30 on MathVerse and MathVista. METIS Only raises these scores to 43.18 and 64.30, gains of 0.83 and 1.00 percentage points. The complete AT-RL pipeline reaches 44.60 and 65.60. This ablation supports adding graph grouping to raw attention weighting and then refining the grouped signal into the final weights. Appendix L.3 now connects the ablation to the METIS objective and its role in AT-RL; Appendix F specifies the timing scope for the complete processing pipeline.
Revisions. Section 4.2: p. 6, lines 305–309; Appendix L.3: p. 25, lines 1323–1331; Table 6: p. 9, lines 469–477.
Dear Reviewer MrMj,
Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.
As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!
Best regards, The Authors
Response to Reviewer MrMj (Part 1 of 2)
We thank the reviewer for the careful evaluation and constructive feedback. We address each point below; key revisions are marked in blue.
Q1: About Evaluation on New Base Models.
We have added experiments on Qwen3.5-9B, evaluating GRPO and SAPO with and without AT-RL on five benchmarks: MMMU, MMMU-Pro, MathVision, MathVista, and WeMath. All methods use the same 10K training subset of ViRL-39K. Each paired comparison retains the corresponding base algorithm's training setting, and AT-RL uses the shared parameters in Appendix B. The results are included in Appendix M, Table 16:
AT-RL improves all five benchmark scores with both base algorithms. The average gains are +1.90 percentage points over GRPO and +2.20 over SAPO. The caption explicitly defines each gain relative to its matched RL baseline.
We retain the same attention-extraction principle: use the final four full-attention layers and average all query heads. For Qwen3.5-9B, these are layers 20, 24, 28, and 32, indexed from 1; Appendix E specifies the configuration. Together with the InternVL2.5-8B and Qwen3-VL-8B experiments in Table 13, these results support transfer of AT-RL across the evaluated model families and generations.
Revisions. Main-text summary in Section 5.5: p. 10, lines 488–495; new experiments and Table 16: p. 26, lines 1352–1373; extraction configuration: p. 20, lines 1050–1064; main-text experiment overview: p. 7, lines 361–363.
Q2: About Optimization Controls and Training Stability.
Optimization mechanism. AT-RL reweights the advantage inside the base RL objective while retaining the base algorithm's own ratio control, optimizer, and regularization settings: clipping for GRPO and soft gating for SAPO (Gao et al. (2025)). We compute the weights without gradient tracking. Nonnegative weights preserve the advantage sign, and T1∑tωt=1 fixes the average token multiplier. The actual parameter update depends jointly on the weights and the corresponding token gradients; Eq. (25) in the revised Appendix L.2 makes this relationship explicit.
Repeated runs. Table 14 reports three independent seeds under the aligned Qwen2.5-VL-7B-Instruct setting. The standard deviations of average benchmark accuracy are 0.16, 0.13, and 0.15 for GRPO + AT-RL, DAPO + AT-RL, and SAPO + AT-RL, respectively, with performance gains maintained across the repeated experiments. These results provide empirical evidence for the repeatability of final performance.
The new Appendix L.4 consolidates these optimization controls and repeated-run results, linking them to the normalization derivation in Appendix B and the no-gradient computation path in Appendix F. This clarifies how the nonuniform weights enter the training objective.
Revisions. Weight–update relationship: p. 25, lines 1307–1319; optimization controls and repeated runs: p. 25, lines 1334–1345; Table 14: p. 23, lines 1207–1216; normalization: p. 18, lines 920–927; no-gradient extraction: p. 21, lines 1080–1085.