acceptodds
Under review as a conference paper at ICLR 2027

Credit Where It’s Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet how visual evidence is integrated during reasoning remains poorly understood. We investigate multimodal RLVR through the lens of cross-modal attention connectivity and find that only a small fraction of tokens—approximately 15%—exhibit strong visual-textual coupling. These high-connectivity tokens act as anchors that ground reasoning in the image, while most other tokens follow linguistic patterns. During RLVR training, credit assignment naturally concentrates on these anchors, progressively sharpening their visual grounding. Based on this insight, we propose Anchor-Token Reinforcement Learning (AT-RL), a lightweight framework that selectively reinforces high-connectivity tokens through graph-based clustering of attention topology. Evaluated on the Qwen2.5-VL series (3B–32B), AT-RL introduces only 1.2% computational overhead while enabling the 32B model to surpass the 72B-Instruct baseline on MathVista, achieving a score of 80.2, with consistent improvements across STEM, video, and general tasks. In contrast, training exclusively on low-connectivity tokens causes severe performance degradation, highlighting the importance of precise credit assignment to visually grounded tokens. These results suggest that reasoning quality depends more on the fidelity of cross-modal anchoring than on token quantity alone, providing an operational account of how visual evidence is selected and reinforced during multimodal RLVR.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MrMjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AT-RL, a lightweight, plug-and-play reinforcement learning algorithm that addresses variation in token-level dependence on visual information during multimodal reasoning. It uses METIS to group tokens by the similarity of their visual attention distributions, then redistributes the response-level advantage to individual tokens according to cluster connectivity strength. Experiments on the Qwen2.5-VL series (3B–32B), with comparisons against several classic reinforcement learning algorithms, demonstrate clear advantages for AT-RL. Combining it with algorithms such as GSPO and SAPO yields further gains.

Critical strengths
  1. The paper demonstrates the method's effectiveness with several competitive reinforcement learning algorithms (e.g., improving SAPO from 49.94 to 53.25). These results also establish its plug-and-play nature: the gains from identifying anchor tokens and redistributing credit can complement those of other reinforcement learning algorithms.
  2. Variation in how different tokens depend on visual information is a real and important issue in multimodal reasoning. The authors present a clear motivation that connects to their method design.
Critical weaknesses
  1. The models used are too outdated to establish whether the conclusions hold for contemporary frontier models. The algorithm is primarily evaluated on Qwen2.5-VL. Although Appendix Table 13 includes Qwen3-VL results, this model still lags behind more recent model series, and the table covers only three benchmarks, omitting GeoQA and WeMath. I am concerned about the method's transferability to the latest models (e.g., Qwen3.5).
  2. Several unclear descriptions make the paper difficult to follow. For example:
    • The method section contains omissions in notation and explanation, including the undefined Θ\Theta in Section 3.1 and the unspecified computation of ϕt\phi_t in Eq. (14).
    • The main text claims to use six mathematical benchmarks but lists only five: MathVerse, MathVista, MathVision, GeoQA, and InterGPS.
    • AT-RL uses METIS in terms of efficiency and edge-cut minimization, but the empirical justification for this specific choice remains limited.
Questions for authors
  1. Could the authors report the main results on the latest models, such as Qwen3.5-9B with GRPO + AT-RL? This would help strengthen the evidence for the method's effectiveness.
  2. Could the non-uniform reweighting of token-level advantages in AT-RL destabilize RL training? Please provide training diagnostics such as policy entropy, KL divergence, reward evolution, and gradient norms.
  3. The discussion states, “Credit Goes to Anchors — Whether Right or Wrong.” However, errors may arise from formatting, language, or mathematical reasoning, as well as from visual input. What motivates this design?
7 comments
AuthorsOpenReview ↗

Response to Reviewer MrMj (Part 1 of 2)

We thank the reviewer for the careful evaluation and constructive feedback. We address each point below; key revisions are marked in blue.

Q1: About Evaluation on New Base Models.

We have added experiments on Qwen3.5-9B, evaluating GRPO and SAPO with and without AT-RL on five benchmarks: MMMU, MMMU-Pro, MathVision, MathVista, and WeMath. All methods use the same 10K training subset of ViRL-39K. Each paired comparison retains the corresponding base algorithm's training setting, and AT-RL uses the shared parameters in Appendix B. The results are included in Appendix M, Table 16:

MethodMMMUMMMU-ProMathVisionMathVistaWeMathAvg.Δ vs. matched RL baseline
GRPO78.870.480.087.176.078.46—
GRPO + AT-RL80.371.482.689.378.280.36+1.90
SAPO79.471.081.588.677.379.56—
SAPO + AT-RL81.272.284.391.279.981.76+2.20

AT-RL improves all five benchmark scores with both base algorithms. The average gains are +1.90 percentage points over GRPO and +2.20 over SAPO. The caption explicitly defines each gain relative to its matched RL baseline.

We retain the same attention-extraction principle: use the final four full-attention layers and average all query heads. For Qwen3.5-9B, these are layers 20, 24, 28, and 32, indexed from 1; Appendix E specifies the configuration. Together with the InternVL2.5-8B and Qwen3-VL-8B experiments in Table 13, these results support transfer of AT-RL across the evaluated model families and generations.

Revisions. Main-text summary in Section 5.5: p. 10, lines 488–495; new experiments and Table 16: p. 26, lines 1352–1373; extraction configuration: p. 20, lines 1050–1064; main-text experiment overview: p. 7, lines 361–363.

Q2: About Optimization Controls and Training Stability.

Optimization mechanism. AT-RL reweights the advantage inside the base RL objective while retaining the base algorithm's own ratio control, optimizer, and regularization settings: clipping for GRPO and soft gating for SAPO (Gao et al. (2025)). We compute the weights without gradient tracking. Nonnegative weights preserve the advantage sign, and 1T∑tωt=1\frac1T\sum_t\omega_t=1 fixes the average token multiplier. The actual parameter update depends jointly on the weights and the corresponding token gradients; Eq. (25) in the revised Appendix L.2 makes this relationship explicit.

Repeated runs. Table 14 reports three independent seeds under the aligned Qwen2.5-VL-7B-Instruct setting. The standard deviations of average benchmark accuracy are 0.16, 0.13, and 0.15 for GRPO + AT-RL, DAPO + AT-RL, and SAPO + AT-RL, respectively, with performance gains maintained across the repeated experiments. These results provide empirical evidence for the repeatability of final performance.

The new Appendix L.4 consolidates these optimization controls and repeated-run results, linking them to the normalization derivation in Appendix B and the no-gradient computation path in Appendix F. This clarifies how the nonuniform weights enter the training objective.

Revisions. Weight–update relationship: p. 25, lines 1307–1319; optimization controls and repeated runs: p. 25, lines 1334–1345; Table 14: p. 23, lines 1207–1216; normalization: p. 18, lines 920–927; no-gradient extraction: p. 21, lines 1080–1085.

AuthorsOpenReview ↗

Response to Reviewer MrMj (Part 2 of 2)

Q3: About “Whether Right or Wrong.”

The phrase refers to applying the same allocation rule to positive and negative advantages. The task outcome, processed by the base RL algorithm, determines the advantage and thus the learning direction. Visual connectivity determines ωt\omega_t, which allocates that signal across tokens. In AAT(i,t)=ωtA(i)A_{\mathrm{AT}}^{(i,t)}=\omega_tA^{(i)}, positive advantages reinforce the corresponding generation patterns, while negative advantages suppress them.

The design uses visual-grounding information to improve multimodal credit allocation while retaining soft-weighted updates for other tokens, including language, formatting, and mathematical reasoning. Anchors are defined by cross-modal connectivity and can carry reasoning content tied to visual conditions. In Table 4, soft weighting achieves 45.27, compared with 42.58 when updating only the top 15% of tokens, supporting the combination of stronger grounding and continued contributions from other tokens.

Section 6 and Appendix Discussion 4 state these roles explicitly: the outcome determines the update direction, connectivity determines allocation, and soft weighting retains other tokens.

Revisions. Section 6: p. 10, lines 521–528; Appendix A, Discussion 4: p. 16, lines 810–817; advantage weighting: p. 7, lines 334–344; weighting-control interpretation: p. 25, lines 1313–1319.

Q4: About Notation and the Benchmark Inventory.

We apologize for the notation omissions and the typographical errors in the benchmark count and name; these are now corrected. Section 3.1 defines the keys indexed by kk in the attention denominator and the calibration parameters Θ\Theta. We have also completed the definitions of connectivity CtC_t, initial importance ϕt(0)\phi_t^{(0)}, centroid-attenuated importance ϕt(1)\phi_t^{(1)}, propagated importance ϕt(2)\phi_t^{(2)}, and the final multiplier ct(i)=ωtc_t^{(i)}=\omega_t. Appendices B.2–B.3 specify initialization, centroid computation, attenuation, and neighborhood updates step by step. Appendix B.7 gives the final normalization, and Algorithm 1 connects the complete pipeline. Sections 4.2–4.3 retain a concise explanation and point to these computational details.

The benchmark inventory now correctly states five image-based mathematical-reasoning benchmarks: MathVerse, MathVista, MathVision, GeoQA, and WeMath. InterGPS was a naming error in the evaluation list and has been corrected to WeMath, matching the result tables. Geometry-3K, used for training, is identified separately in the training-data paragraph.

Revisions. Section 3.1 notation: p. 4, lines 184–190; main-text algorithm explanation: p. 6, lines 310–314, and p. 7, lines 324–341; detailed computations and notation: p. 17, lines 864–917; normalization and complete pseudocode: p. 18, lines 918–965; benchmark inventory: p. 7, lines 374–377.

Q5: About the Choice of METIS.

Fit to the graph representation. AT-RL constructs a sparse, undirected, weighted token graph whose edges measure similarity between calibrated visual-attention footprints. METIS (Karypis and Kumar (1998)) minimizes cross-group edge weight under balance constraints, grouping tokens with similar visual attention. Its multilevel coarsening and refinement suit the graph-partitioning step performed once per rollout. We have clarified this correspondence in Section 4.2 and Appendix L.3.

Grouping ablation. In Table 6, Raw Attention Weighting obtains 42.35 and 63.30 on MathVerse and MathVista. METIS Only raises these scores to 43.18 and 64.30, gains of 0.83 and 1.00 percentage points. The complete AT-RL pipeline reaches 44.60 and 65.60. This ablation supports adding graph grouping to raw attention weighting and then refining the grouped signal into the final weights. Appendix L.3 now connects the ablation to the METIS objective and its role in AT-RL; Appendix F specifies the timing scope for the complete processing pipeline.

Revisions. Section 4.2: p. 6, lines 305–309; Appendix L.3: p. 25, lines 1323–1331; Table 6: p. 9, lines 469–477.

Dear Reviewer MrMj,

Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.

As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!

Best regards, The Authors

Reviewer_GZz5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes multimodal RLVR through cross-modal attention and reports that only about 15% of generated tokens are strongly connected to image patches ("perceptual anchors") . AT-RL builds a token graph from the bias-corrected attention of the last four layers, partitions it with METIS, and rescales each token's sequence-level advantage by its cluster's share of total connectivity, keeping the mean weight at 1 . It plugs into GRPO, DAPO, GSPO and SAPO without changing inference. On Qwen2.5-VL-7B, SAPO + AT-RL averages 53.25 on five math benchmarks (zero-shot 45.01), and a 32B model reaches 80.2 on MathVista.

Critical strengths

1,A simple plug-in with consistent gains: +3.21 to +3.59 average points across four RL algorithms on 7B, small seed variance, and transfer to two other architectures.

2,Evidence that the selected anchors are visually grounded: their top-attended patches fall in annotated key regions 73.8% of the time versus 17.5% for random tokens, and masking those patches costs 10.2 points versus 4.2 for random patches.

Critical weaknesses

1,The central claim about RLVR dynamics is inconsistent and unsupported. The abstract states that "during RLVR training, credit assignment naturally concentrates on these anchors", whereas the introduction states that standard RLVR "often inherits the base model topology without significant refinement" and that only AT-RL sharpens anchor connectivity, citing an analysis of connectivity evolution. No such analysis is shown for either method. If standard RLVR already concentrates credit on anchors, explicit reweighting needs another justification; if it does not, this is exactly the evidence the method needs.

2, It is only partly shown that credit goes to the right tokens. The grounding analyses validate the selection signal, but not that the weights ω up-weight the tokens that determine correctness. The gain also hinges on a hand-designed bias-correction curve whose parameters are not justified: without it, AT-RL is worse than GRPO (34.20 vs 35.10). Random and reverse weighting, both with mean weight 1, fall below the untrained model (38.10 and 36.80 vs 39.62), which is hard to interpret.

3, Inconsistent and incomplete reporting. Identical configurations have different numbers across tables (e.g., GRPO on MathVerse: 41.81 in Table 1 vs 41.48 in Table 4); the headline 32B result comes without a training configuration; and the 15% anchor statistic rests on 540 tokens from 100 samples.

Questions for authors

A , Please reconcile the abstract and the introduction, and show how anchor connectivity evolves during training under GRPO and under AT-RL.

B , Can you show directly that ω concentrates credit on correctness-determining tokens (e.g., via token-level interventions), and report 7B results without bias correction?

5 comments
AuthorsOpenReview ↗

Response to Reviewer GZz5

We thank the reviewer for the constructive feedback. We address each point below; key manuscript revisions are highlighted in blue.

Q1: About Training Configuration and Data Consistency.

Configurations. Tables 1 and 4 use Qwen2.5-VL-7B-Instruct with different training data: the 10K subset of ViRL-39K and GeoQA-8K, respectively. Their MathVerse scores of 41.81 and 41.48 thus reflect different settings. Section 5.1 and the captions now specify training datasets and full model variants. Table 10 uses Qwen2.5-VL-32B-Instruct with SAPO + AT-RL on the same 10K ViRL-39K subset. Appendices B and G specify shared parameters and their selection protocol.

Expanded statistics. We extended the connectivity analysis to the full 10K ViRL-39K training subset, obtaining a high-connectivity token fraction of 14.3%, supporting the approximately 15% minority-anchor pattern. Appendix A reports this measurement separately from the propagation-source quota q=0.15q=0.15.

Revisions. Setup: p. 7, lines 364–368; captions: p. 8, lines 378–381, p. 9, lines 459–461, and p. 22, lines 1134–1137; statistics: p. 14, lines 713–717.

Q2: About Credit Assignment and Bias Correction.

Visual relevance and training outcomes. In Appendix J, anchors' top-attended patches overlap annotated key regions 73.8% of the time versus 17.5% for random tokens. Deleting corresponding image regions reduces accuracy by 10.2 points versus 4.2 for random regions. Table 4 varies the token-credit allocation rule: uniform GRPO scores 41.48, top-15% hard truncation 42.58, and AT-RL soft weighting 45.27. These results support the signal's visual relevance and the training benefit of structured soft allocation, respectively. The main text consistently describes connectivity as an attention-based grounding proxy.

Mean-one weighting controls. Normalization fixes the average multiplier; the update also depends on which token gradients receive each weight. For the unclipped policy-gradient contribution at the rollout policy, with gt=∇θlog⁡πθ(t∣context)g_t=\nabla_\theta\log\pi_\theta(t\mid\mathrm{context}),

=\frac{A}{T}\sum_{t=1}^{T}(\omega_t-1)g_t.$$ Random weighting removes the relationship to visual grounding; reverse weighting favors lower-connectivity tokens. Their scores of **38.10 and 36.80**, below zero-shot's **39.62**, show that the same average multiplier can yield different, including harmful, update directions. Table 4 and Appendix L.2 therefore identify structured allocation as the source of AT-RL's benefit. **Calibration and new 7B ablation.** Calibration addresses positional preferences and padding-related bias documented by [Zhao et al. (2025a)](https://arxiv.org/abs/2508.17807) before computing connectivity and footprint similarity. Appendix B.1 combines an exponential position trend, smooth cosine variation, and positive baseline, with shared coefficients $\lambda_{\exp}=0.15$, $\gamma=4$, and $\lambda_{\cos}=0.05$. Appendix L.1 now explains this design and cites the supporting study. The new Table 15 uses Qwen2.5-VL-7B-Instruct trained on GeoQA-8K and evaluated on MathVerse under Table 4's setting: | Method | MathVerse Acc@1 (%) | | --- | ---: | | GRPO, uniform weighting | 41.48 | | AT-RL without bias correction | 42.98 | | AT-RL with bias correction | 45.27 | Uncorrected AT-RL improves over GRPO by **1.50** points; calibration adds **2.29** points. Together with the original 3B ablation, this supports correcting the attention signal used to construct token weights. Section 5.5 summarizes this additional result. **Revisions.** Proxy formulation: p. 5, lines 223–226; main-text ablation summary: p. 10, lines 497–502; calibration rationale: p. 24, lines 1278–1288; Table 15: p. 25, lines 1296–1302; weighting analysis and Eq. (25): p. 25, lines 1307–1319. Existing grounding evidence is in Tables 11–12 and Sections J.1–J.2, pp. 22–23. ### **Q3: About Narrative Clarity and Credit Allocation.** The abstract, introduction, and method now describe the same implemented mechanism: GRPO assigns a sequence-level advantage to generated tokens; AT-RL allocates it through structured visual-connectivity weights, $A_{\mathrm{AT}}^{(i,t)}=\omega_tA^{(i)}$. The abstract states that AT-RL concentrates reinforcement signals on perceptual anchors through connectivity-based advantage reweighting. The introduction explains this mechanism and links it to Table 4's weighting controls and Appendix J's grounding analyses. “Concentrated credit” thus refers consistently to the computable allocation mechanism evaluated by these experiments. **Revisions.** Abstract: p. 1, lines 21–23; introduction: p. 2, lines 84–87; definitions: p. 7, lines 324–341.

Thank you for the detailed response and the revisions, which address my concerns about reporting consistency and calibration. My remaining concern is this: the revision states that Table 1 was trained on a 10K subset of ViRL-39K, whereas Section 5.1 of the submitted version stated that training used Geometry-3K supplemented with GeoQA, yielding approximately 3,000 samples, with ViRL cited only as established practice. Since the main results themselves are unchanged, could you confirm that the setup description in the submitted version did not match the experiments it reported, and explain how this arose?

Thank you for carefully reviewing our responses and the revised manuscript. We appreciate your recognition of the clarifications on reporting consistency and calibration, and your thoughtful follow-up gives us the opportunity to clarify this remaining discrepancy.

To confirm, the experiments reported in Table 1 used the 10K subset of ViRL-39K. During manuscript reorganization and template conversion for resubmission, the setup paragraph was not updated consistently with the experiments reported in Table 1. As a result, the description in Section 5.1 of the submitted version did not accurately reflect this training setting. We sincerely apologize for overlooking this discrepancy during our final checks.

We have corrected the experimental setup description and the Table 1 caption in the revised manuscript to explicitly identify the training data and subset size. The numerical results in Table 1 remain unchanged.

Reviewer_po35OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AT-RL studies token-level credit assignment in multimodal RLVR by using cross-modal attention as a grounding proxy. It applies positional correction to token-to-image attention, builds a graph from attention-footprint similarity, partitions tokens via METIS, and converts sequence-level advantages into softly weighted token advantages. The method plugs into several policy-optimization algorithms and is tested across multimodal reasoning tasks and multiple backbones.

Critical strengths
  1. The weighting works as a drop-in on top of several policy-optimization objectives without changing the reward. The reported gains span model sizes and include InternVL2.5 and Qwen2.5-VL backbones, suggesting the approach is not locked to one architecture.

  2. The attention-based pipeline — positional correction, graph construction, METIS partitioning, soft weighting — is a concrete, end-to-end mechanism that gives a tangible alternative to uniform token treatment. The region-deletion analysis (Appendix J.7) at least confirms that the selected tokens overlap with image-relevant regions.

Critical weaknesses
  1. The experimental protocol has reproducibility gaps that are hard to overlook. Tables 1, 4, 6, 9, and 14 are not clearly mapped to their data, checkpoints, and hyperparameters; the gain summaries in Appendix J have internal inconsistencies; and the benchmark inventory (InterGPS vs. WeMath, Expanded Training Data composition) is unclear. The validation data used for grid search aren't specified — it matters whether MathVerse or MathVista testmini scores were used for selection vs. only for sensitivity analysis. The implementation for the reported results is also not provided, and the claimed 1.2% overhead needs a breakdown (especially whether it includes attention materialization and which attention backend was used).

  2. Prior work already explores concentrated updates on minority tokens, so the novelty here lies specifically in the attention correction, graph refinement, and soft weighting. The paper includes a VPPO comparison, but it needs to more clearly explain what distinguishes AT-RL's mechanism from entropy-based token selection and PAPO. The causal-credit language also oversells the evidence: the image-region deletion test shows region importance, not causal responsibility of the associated generated tokens. Appendix J.7 is careful about this distinction, but the main text is not. The claims should match what attention-based weighting actually demonstrates.

Questions for authors
  1. Can you provide full pseudocode connecting centroid attenuation and neighborhood propagation to the final normalized token weights, along with the actual layer/head configuration used for each backbone?

  2. Please map Tables 1, 4, 6, 9, and 14 to their data, checkpoints, and hyperparameters. The Appendix J gain summaries need reconciling, and the benchmark inventory (InterGPS vs. WeMath) needs clarifying.

  3. What controls were used for the VPPO comparison? Under comparable training conditions, what concretely distinguishes AT-RL from entropy-based minority-token selection and PAPO?

  4. Beyond the image-region deletion test, what evidence supports assigning causal responsibility to the selected tokens? If the claim is really about an attention-based proxy, the main text should say so as clearly as Appendix J.7 does.

  5. Can you provide the implementation used for the reported results and a matched comparison of total training time and memory, including attention materialization and the attention backend? What exactly does the 1.2% figure include?

5 comments
AuthorsOpenReview ↗

Response to Reviewer po35 (Part 1 of 2)

We thank the reviewer for the constructive feedback. We address each point below; key manuscript revisions are highlighted in blue.

Q1: About Algorithm Formulation and Pseudocode.

Sections 4.2–4.3 now connect refinement to the final token weights. Appendix B provides the complete computations, and Algorithm 1 connects calibration, graph grouping, centroid attenuation, propagation, and advantage weighting.

Initialization and attenuation. We initialize ϕt(0)=Ct/max⁡uCu\phi_t^{(0)}=C_t/\max_u C_u, placing importance on a [0,1][0,1] scale within each response. Each group centroid is computed in calibrated attention space. A token's importance is multiplied by 0.60.6 if its cosine similarity to its centroid is below 0.750.75, yielding ϕt(1)\phi_t^{(1)}. Equations (13)–(15) specify these steps, including zero-connectivity and zero-norm handling.

Propagation. The top 15% of tokens ranked by ϕ(1)\phi^{(1)} are sources. Each source selects up to four adjacent nodes with the highest similarities in the thresholded graph. For a neighbor satisfying 0.5sut+0.5ϕu(1)>0.650.5s_{ut}+0.5\phi_u^{(1)}>0.65, the candidate score is 0.5ϕu(1)+0.5sutϕt(1)0.5\phi_u^{(1)}+0.5s_{ut}\phi_t^{(1)}. Its final score ϕu(2)\phi_u^{(2)} is the maximum of its original score and all eligible candidates. All candidates use fixed pre-propagation scores, giving one order-independent step (Eqs. (16)–(18)).

Final weights. Propagated scores determine cluster mass and token advantages:

\omega_t=\frac{T W(G_k)}{|G_k|},\qquad A_{\mathrm{AT}}^{(i,t)}=\omega_t A^{(i)},\quad t\in G_k.$$ Cluster-size normalization ensures $\frac{1}{T}\sum_t\omega_t=1$. Equations (9)–(10) define the weights in the main text, and Eq. (19) derives the normalization identity. **Revisions.** Main-text weights: p. 7, lines 324–341; initialization, attenuation, and propagation: p. 17, lines 864–902; normalization and Algorithm 1: p. 18, lines 918–965. ### **Q2: About Backbone Layer and Head Configurations.** Across all six backbones, we select the final four language-decoder layers providing full attention maps, extract generated-text-to-visual-token attention, and average all query heads. $H$ denotes the backbone-specific query-head count; indices below start at 1. | Backbone | Decoder layers | Extracted layers | Query heads used | KV heads | | --- | ---: | :---: | ---: | ---: | | Qwen2.5-VL-3B-Instruct | 36 | 33–36 | 16 | 2 | | Qwen2.5-VL-7B-Instruct | 28 | 25–28 | 28 | 4 | | Qwen2.5-VL-32B-Instruct | 64 | 61–64 | 40 | 8 | | InternVL2.5-8B | 32 | 29–32 | 32 | 8 | | Qwen3-VL-8B | 36 | 33–36 | 32 | 8 | | Qwen3.5-9B | 32 | 20, 24, 28, 32 | 16 | 4 | For the first five backbones, these are the final four decoder layers. Qwen3.5-9B has a hybrid architecture; its listed layers and head counts refer to full-attention modules. Aggregation uses query heads, producing $\bar{A}\in\mathbb{R}^{T\times P}$ for $T$ generated tokens and $P$ visual tokens. Our added Qwen3.5-9B experiments yield average gains of **+1.90 and +2.20** points over matched GRPO and SAPO baselines across five benchmarks (Table 16). **Revisions.** Extraction rule: p. 5, lines 235–245; Appendix E configurations: p. 20, lines 1050–1064; Appendix M results: p. 26, lines 1352–1373. ### **Q3: About Computational Overhead and Efficiency.** **The reported 1.2% covers the complete AT-RL pipeline:** attention extraction and materialization, bias correction, graph construction, METIS partitioning, attenuation, propagation, and normalization. It is the share of reported per-iteration runtime attributable to these operations. Policy optimization retains FlashAttention-2. An auxiliary path without gradient tracking extracts generated-text-to-visual-token attention from the four selected full-attention layers and averages layers and query heads into a $T\times P$ matrix, retaining the original normalization over all permitted keys. The resulting weights enter advantage weighting; extraction and graph processing are outside the policy-backward graph. Post-processing runs once per rollout. METIS uses the thresholded sparse graph; propagation visits at most four neighbors per source among the top 15% of tokens. These operations account for the small measured runtime share. **Revisions.** Timing scope in Section 5.3: p. 9, lines 453–455; full profiling and computation path in Appendix F: p. 20, lines 1071–1079, and p. 21, lines 1080–1090.
AuthorsOpenReview ↗

Response to Reviewer po35 (Part 2 of 2)

Q4: About Training Data Across Tables.

Section 5.1 and the captions now specify the settings:

Training dataExperiments
ViRL-39K, 10K subsetTables 1, 9, 10, 14
GeoQA-8KTable 2 (both 3B and 7B), Tables 4 and 15
Geometry-3KTables 5–8
Video-R1-data, 10K subsetTable 3

Tables 1, 4, 6, and 14 use Qwen2.5-VL-7B-Instruct. Tables 4 and 6 use GeoQA-8K and Geometry-3K, respectively. Table 9 uses Qwen2.5-VL-3B-Instruct with GRPO + AT-RL, comparing β=0.02\beta=0.02 and β=0\beta=0. Table 10 uses Qwen2.5-VL-32B-Instruct with SAPO + AT-RL. Appendices B and G specify shared parameters and parameter selection; each ablation varies the stated component.

In Table 2, “Expanded Training Data” denotes GeoQA-8K (R1-V release) for both Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct. We apologize for the reporting typo in Table 2; it is now corrected. The 7B GRPO MathVerse score is 41.48, matching Table 4, with Avg. 46.70 and gains of +1.86 and +1.69 over zero-shot. We also apologize for the benchmark-list typos, now corrected to MathVerse, MathVista, MathVision, GeoQA, and WeMath. Table 3 separates Video-R1-data training from VSI-Bench, VideoMMMU, and MMVU evaluation.

Revisions. Setup: p. 7, lines 364–377; Tables 1–2: p. 8, lines 378–407; Tables 3–6 captions: p. 9, lines 432–434, 459–461, and 469–471; Table 9: p. 19, lines 999–1002; Table 10: p. 22, lines 1134–1137; Table 14: p. 23, lines 1207–1208.

Q5: About Gain Summaries in Appendix J.

We apologize for the arithmetic errors in the gain summary, now corrected. J.4 uses Table 14's displayed Avg. values and matched baselines: 50.5 − 47.3 = +3.2 for GRPO, 51.6 − 48.0 = +3.6 for DAPO, and 53.3 − 49.9 = +3.4 points for SAPO. J.5 references this common summary.

Revisions. p. 23, lines 1218–1221 and 1232–1233.

Q6: About Baseline Comparisons and Methodological Differences.

VPPO. We use the original paper's parameter settings (Huang et al., 2025), including token-level gradient filtering and trajectory-level advantage shaping, with training data and initial checkpoints aligned to our experiments. Section 5.1 and Appendix K now specify these controls.

FT-RL. We apologize for the imprecise description and omitted citation, now corrected. FT-RL is the high-entropy forking-token baseline following Wang et al. (2025b). It concentrates updates on high-entropy tokens and is now defined and cited in Related Work, Section 5.1, Table 1, and Appendix K. Table 1 includes this baseline: 44.85 average versus 50.52 for GRPO + AT-RL. AT-RL instead derives visual connectivity from calibrated cross-modal attention and applies graph grouping, attenuation, and propagation to produce normalized soft advantage weights.

PAPO. Wang et al. (2025c) add an Implicit Perception Loss encouraging divergence between outputs conditioned on original and masked images, with Double Entropy regularization. AT-RL retains the task reward and allocates the existing outcome-based advantage across tokens. The comparison now distinguishes the auxiliary perception objective from advantage allocation.

Revisions. Related Work: p. 4, lines 162–169; setup: p. 7, lines 369–373; Table 1 caption: p. 8, lines 378–381; Appendix K: p. 24, lines 1251–1271.

Q7: About Validation Data for Hyperparameter Selection.

We use the Geometry-3K test split (Geo3K-test) as held-out validation for hyperparameter selection, then fix the configuration across evaluation benchmarks. Geometry-3K is distinct from the GeoQA evaluation benchmark. MathVerse and MathVista testmini serve evaluation and the sensitivity analyses in Tables 7–8; selection uses Geo3K-test. Both tables use Qwen2.5-VL-7B-Instruct with SAPO + AT-RL trained on Geometry-3K. Their captions now specify these settings and the analysis purpose.

Revisions. Appendix G: p. 21, lines 1094–1100; Tables 7–8: p. 16, lines 819–821 and 832–835.

Q8: About the Interpretation of Attention-Based Credit Assignment.

The introduction and method now consistently define connectivity as an attention-based grounding proxy. Region deletion and annotated-region alignment assess visual relevance; controlled weighting comparisons assess the optimization benefit. Figure 2 and Section 4.3 describe normalized grounding-based allocation. The revised “Whether Right or Wrong” explanation makes the roles explicit: the outcome-based advantage determines update direction, while connectivity-derived soft weights allocate it across tokens and retain other tokens' contributions.

Revisions. p. 2, lines 93–95; p. 5, lines 223–226; p. 6, lines 287–288; p. 7, lines 348–352; p. 16, lines 810–817.

Dear Reviewer po35,

Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.

As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear ACs and Reviewers,

We sincerely thank all three reviewers for their constructive feedback and for recognizing AT-RL's plug-and-play compatibility, consistent performance gains, and motivation for visually grounded credit assignment.

We have focused the revision on clarifying the experimental comparisons, making the method explicit, and strengthening the supporting evidence. The revisions summarized below have already been incorporated into the manuscript, with the key changes highlighted in blue for easy reference. We group shared concerns to provide an overview of how they have been addressed.

ReviewerConcernResponse and manuscript revision
po35, GZz5Experimental settings and consistency across tables.We have clarified the configuration behind each comparison. The revised setup and captions identify the training data and initial models, distinguish experiments conducted under different settings, and reconcile values and gain summaries for comparable results. We apologize for the typographical and gain-calculation errors, which have now been corrected.
po35, MrMjIncomplete algorithm and protocol descriptions.We have completed the algorithmic specification and documented the experimental protocols. The manuscript now connects all refinement steps to the final token weights through explicit equations and pseudocode, and specifies backbone configurations, parameter selection, and the scope of computational profiling.
po35, GZz5, MrMjInterpretation and justification of attention-based credit assignment.We have unified the central claim and its supporting evidence throughout the paper. The revised text explains how outcome-based advantages determine update direction and grounding weights allocate the signal while retaining other tokens' contributions. It links visual-grounding analyses and controlled weighting comparisons to their respective roles in supporting the method.
po35Distinction from existing token-selection and perception-based methods.We have clarified the contribution relative to VPPO, FT-RL, and PAPO. The revised baseline descriptions and comparisons specify the comparison settings and explain how AT-RL's calibrated attention graph and soft advantage allocation differ from the selection signals and objectives used by these methods.
GZz5Evidence for calibration and the representativeness of anchor statistics.We have added the requested 7B bias-correction ablation and expanded the connectivity analysis to the full training subset. The added results demonstrate the benefit of calibration and support the minority-anchor pattern at a substantially larger analysis scale. Both results and their interpretation are now included in the manuscript.
MrMjTransferability to newer model generations.We have added Qwen3.5-9B experiments across five benchmarks and two RL algorithms. AT-RL improves performance in both paired comparisons across all five benchmarks, extending the evidence beyond the original model generation. The results and training configuration are included in the revised paper.

The revised manuscript now presents consistent experimental comparisons, an explicit computational pipeline, and additional evidence for calibration and model transfer. Detailed settings, numerical results, and page and line references are provided in our individual responses. We thank the ACs and reviewers for their time and welcome further discussion.

Best regards,
The Authors

AuthorsOpenReview ↗

General Response Before the Discussion Closes

Dear Area Chair and Reviewers,

Thank you for the time you have spent reviewing our paper and discussing the revisions with us. As the discussion period draws to a close, we would welcome your thoughts on our revised manuscript and follow-up responses while there is still time for us to reply.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.