- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper starts from the observation that the usual way of extending RL post-training to multimodal models, rewarding long verbalised rationales, buys little on perception-heavy VQA and can make things worse. The authors respond by moving the optimisation target off the output token distribution and onto the model's internal attention distribution.
Concretely, for each generated token t they define an "attention policy" p_t over the preceding positions, using the attention weights of the final transformer layer averaged across heads and renormalised. The auxiliary objective is the advantage-weighted Jensen-Shannon divergence between the current attention policy and the old one.
Evaluation covers 8 image benchmarks and 7 video benchmarks against the base model, GRPO, Video-R1-7B and on-policy knowledge distillation.
Critical strengths
The RAL-zero result is the part I would keep. Removing the think blocks from both SFT and RL is a reasonably clean intervention, and the outcome is not marginal: 76.2 against 73.7 on NExTQA, 65.1 against 61.6 on VideoMME over the base model. Most of this field treats verbalised reasoning as the mechanism through which RL improves MLLMs. This says otherwise, with numbers. I would have preferred the paper to be about this.
Second, the empirical sweep is broad and the gains over GRPO are consistent: all eight image benchmarks in Table 2, six of seven video benchmarks in Table 1, a positive distillation result on seven of eight image benchmarks. If the effect is real and the attribution holds, a drop-in auxiliary loss that needs only last-layer attention weights and an advantage estimate you already have, and that doesn't ask you to save anything extra at rollout time, would be a genuinely useful thing to have in the toolkit. That "if" is carrying a lot of weight in my assessment.
Critical weaknesses
Most of the headline gains are on image benchmarks, and look at what GRPO alone does to the base model there. V* goes 70.7 to 68.6. MME drops from 2309.3 to 2258.7, which is not a rounding artefact on a benchmark with that range. I'm reading those two off Table 2 and rounding, so check me, but the sign isn't in doubt. Six of eight, all downward. That is not a weak perceptual policy. That is what forgetting looks like after fine-tuning on a different domain. RAL exceeds the base model on all eight. So the most parsimonious reading of Table 2 is that the attention term slows drift away from the SFT model, while the paper reads the same table as improved perception. An image-domain RL baseline, or a direct forgetting diagnostic, would tell them apart. As far as I can tell neither is there.
Baseline tuning is a related problem. A rollout batch of 8 with group size 8 is small, the GRPO hyperparameters are not swept, one epoch on 51.2k is short. GRPO degrading the base model on six of eight image benchmarks tells me at least as much about the baseline as it does about GRPO.
In Table 2 the RAL-zero row and the +Attention Distillation row share four identical entries: 72.3, 38.4, 43.4, 78.4 on V*, MMMU-Pro, MuirBench and CVBench. Four matches between two different training conditions is more coincidence than I expect, and I would like to know whether a row was populated correctly.
What I would have wanted: an attention-level or grounding metric, IoU against annotated evidence regions would do, because right now the only evidence that attention moved toward the right content is three heatmaps.
Questions for authors
-
How is p_old obtained, and what are the value and gradient of the attention loss at the start of an update? If it is registered in the training forward pass, is the term not identically zero, and if it is frozen from an earlier step, can you show JSD as a function of training step? Please also give the alpha actually used rather than the sweep range, and say which split, if any, was used to pick it.
-
A GRPO run with the thinking blocks removed, matched to RAL-zero on data and compute. Without it I cannot tell whether RAL-zero's gains come from the attention term or from dropping the CoT. This is the one I most want answered.
-
Is the image-benchmark improvement in Table 2 a change in perception or a reduction in forgetting? A forgetting diagnostic, or an RL run that includes image-domain data, would settle it; the video-only training data makes the image gains unreadable as they stand.