acceptodds
Under review as a conference paper at ICLR 2027

Reinforced Attention Learning

Abstract

Post-training via Reinforcement Learning (RL) has catalyzed significant reasoning gains in Large Language Models (LLMs) by scaling test-time computation. While recent efforts attempt to extend this paradigm to Multimodal Large Language Models (MLLMs) by incentivizing verbose rationales, we demonstrate that such gains are marginal for perception; indeed, excessive verbalization can inadvertently degrade performance. We propose Reinforced Attention Learning (RAL), a policy-gradient framework that treats internal attention distributions as an optimization target. Unlike conventional RL that focuses on high-reward token sequences, RAL explicitly explores and rewards optimal information allocation patterns during the generation process. By shifting optimization from "what to generate" to "where to attend", RAL enhances the model's ability to isolate salient signals within complex multimodal contexts. Extensive experiments across diverse image and video benchmarks show that RAL consistently outperforms standard GRPO and other baselines. We further extend this approach to On-Policy Attention Distillation, proving that transferring latent attention behaviors to student models yields superior cross-modal alignment compared to standard knowledge distillation. Our results establish attention policies as a principled, generic alternative for multimodal post-training. We will release our code and data upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
post training, vision language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kLQyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper starts from the observation that the usual way of extending RL post-training to multimodal models, rewarding long verbalised rationales, buys little on perception-heavy VQA and can make things worse. The authors respond by moving the optimisation target off the output token distribution and onto the model's internal attention distribution.

Concretely, for each generated token t they define an "attention policy" p_t over the preceding positions, using the attention weights of the final transformer layer averaged across heads and renormalised. The auxiliary objective is the advantage-weighted Jensen-Shannon divergence between the current attention policy and the old one.

Evaluation covers 8 image benchmarks and 7 video benchmarks against the base model, GRPO, Video-R1-7B and on-policy knowledge distillation.

Critical strengths

The RAL-zero result is the part I would keep. Removing the think blocks from both SFT and RL is a reasonably clean intervention, and the outcome is not marginal: 76.2 against 73.7 on NExTQA, 65.1 against 61.6 on VideoMME over the base model. Most of this field treats verbalised reasoning as the mechanism through which RL improves MLLMs. This says otherwise, with numbers. I would have preferred the paper to be about this.

Second, the empirical sweep is broad and the gains over GRPO are consistent: all eight image benchmarks in Table 2, six of seven video benchmarks in Table 1, a positive distillation result on seven of eight image benchmarks. If the effect is real and the attribution holds, a drop-in auxiliary loss that needs only last-layer attention weights and an advantage estimate you already have, and that doesn't ask you to save anything extra at rollout time, would be a genuinely useful thing to have in the toolkit. That "if" is carrying a lot of weight in my assessment.

Critical weaknesses

Most of the headline gains are on image benchmarks, and look at what GRPO alone does to the base model there. V* goes 70.7 to 68.6. MME drops from 2309.3 to 2258.7, which is not a rounding artefact on a benchmark with that range. I'm reading those two off Table 2 and rounding, so check me, but the sign isn't in doubt. Six of eight, all downward. That is not a weak perceptual policy. That is what forgetting looks like after fine-tuning on a different domain. RAL exceeds the base model on all eight. So the most parsimonious reading of Table 2 is that the attention term slows drift away from the SFT model, while the paper reads the same table as improved perception. An image-domain RL baseline, or a direct forgetting diagnostic, would tell them apart. As far as I can tell neither is there.

Baseline tuning is a related problem. A rollout batch of 8 with group size 8 is small, the GRPO hyperparameters are not swept, one epoch on 51.2k is short. GRPO degrading the base model on six of eight image benchmarks tells me at least as much about the baseline as it does about GRPO.

In Table 2 the RAL-zero row and the +Attention Distillation row share four identical entries: 72.3, 38.4, 43.4, 78.4 on V*, MMMU-Pro, MuirBench and CVBench. Four matches between two different training conditions is more coincidence than I expect, and I would like to know whether a row was populated correctly.

What I would have wanted: an attention-level or grounding metric, IoU against annotated evidence regions would do, because right now the only evidence that attention moved toward the right content is three heatmaps.

Questions for authors
  1. How is p_old obtained, and what are the value and gradient of the attention loss at the start of an update? If it is registered in the training forward pass, is the term not identically zero, and if it is frozen from an earlier step, can you show JSD as a function of training step? Please also give the alpha actually used rather than the sweep range, and say which split, if any, was used to pick it.

  2. A GRPO run with the thinking blocks removed, matched to RAL-zero on data and compute. Without it I cannot tell whether RAL-zero's gains come from the attention term or from dropping the CoT. This is the one I most want answered.

  3. Is the image-benchmark improvement in Table 2 a change in perception or a reduction in forgetting? A forgetting diagnostic, or an RL run that includes image-domain data, would settle it; the video-only training data makes the image gains unreadable as they stand.

Reviewer_wPNzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reinforced Attention Learning (RAL), an auxiliary objective for GRPO-style post-training of multimodal LLMs that advantage-weights the Jensen-Shannon divergence between current and behavior-policy attention distributions. It also introduces on-policy attention distillation from a larger teacher. Experiments on image and video QA benchmarks compare RAL and attention distillation against GRPO and token-level distillation.

Critical strengths
  • The paper presents an interesting and well-motivated approach to multimodal RL. By treating attention distributions as an additional optimization target and introducing an advantage-weighted divergence loss, RAL provides a new perspective on improving multimodal perception beyond conventional token-level optimization.
  • The paper provides extensive experimental evaluations across various image and video QA benchmarks, including long-context video understanding tasks. The results demonstrate consistent improvements over the GRPO baseline on most benchmarks, supporting the effectiveness of the proposed method.
Critical weaknesses
  • The contribution is somewhat overstated. Although RAL is presented as a new paradigm for multimodal RL, its implementation still relies on conventional token-level RL with an additional attention regularization loss. The paper should more accurately position its contribution as an auxiliary optimization strategy rather than a fundamentally different training paradigm.
  • The key design choices lack sufficient experimental justification. The method uses attention from the final Transformer layer and averages it across attention heads, but the paper does not investigate alternative layer selections or head aggregation strategies. Without these ablations, it is unclear how sensitive the method is to these design choices.
  • The generalizability of the method remains insufficiently validated. Although RAL is claimed to be compatible with different model architectures and RL algorithms, the experiments are limited to Qwen2.5-VL-7B with GRPO. Evaluations on other backbone models and RL algorithms, such as PPO or DAPO, would provide stronger evidence for its general applicability.
  • The analysis of important hyperparameters is insufficient. Although the paper lists the search ranges for the attention loss weights λ_attn and γ_attn, it does not report how different values affect performance or clearly explain how the final values were selected. Additional sensitivity experiments are needed to assess the robustness of the proposed method.
Questions for authors
  • Since RAL is implemented as an auxiliary attention loss added to standard RL, could the authors further clarify what distinguishes it from conventional regularization methods and why it should be considered a new training paradigm?
Reviewer_FLbxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that optimizing token distributions during RL post-training provides limited supervision for perception. It instead treats the last-layer, head-averaged attention distribution as a policy. RAL adds an advantage-weighted JSD term to GRPO: positive-advantage attention patterns are preserved, while negative-advantage patterns are pushed away. The same idea is also used for on-policy attention distillation. Experiments with Qwen-2.5-VL-7B on Video-R1 show that RAL generally outperforms GRPO and that attention distillation improves over standard on-policy distillation.

Critical strengths
  • RAL leaves the rollout and reward unchanged and only adds an attention loss, making it relatively easy to integrate into existing RL pipelines.
  • RAL outperforms GRPO on 14 of 15 benchmarks and mitigates GRPO's degradation from the base model on several image tasks.
Critical weaknesses
  • In Eq. 3, the gradient is zero when pθ=poldp_\theta=p_{old}. The loss therefore cannot directly reinforce high-reward attention after rollout as an ordinary policy-gradient objective would. It can only become active after GRPO has already changed the parameters, making it look more like a regularizer on GRPO updates than an independent attention-policy objective.
  • The method averages all heads and all preceding positions in the final layer, then applies the same sequence-level advantage everywhere. It neither distinguishes visual from textual tokens nor provides fine-grained credit assignment, so the reported gains cannot yet be attributed to better visual grounding.
  • Important controls are missing. PAPO is closely related in both goal and setting but is not compared. RAL-zero also lacks a GRPO-zero baseline, making it unclear whether its gains come from the attention loss or from removing CoT and changing the training format.
Questions for authors
  1. How is $p_{old}$ updated, and how many optimization passes are performed per rollout batch? Please explain how the attention loss becomes effective when its initial gradient is zero.
  2. Could the authors compare visual-token-only and all-token objectives, as well as different layer/head aggregation choices, and use a quantitative grounding metric to verify that RAL changes how the model uses visual evidence?
  3. Please add controlled comparisons with PAPO and GRPO-zero and report the selected $\lambda_{attn}$. Also, are the four identical entries for Attention Distillation and RAL-zero in Table 2 a reporting error?

If the rebuttal shows that Eq. 3 provides a meaningful gradient in practice and adds grounding analyses together with controlled PAPO and GRPO-zero comparisons, I would consider raising my score from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.