acceptodds
Withdrawn from ICLR 2027

Learning When to Look: Attention-Proposed Temporal Rewards for Grounded Video Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video reasoning is a core capability of multimodal large language models (MLLMs), crucial for downstream tasks such as video question answering and captioning. Recent work has advanced this capability through reinforcement learning (RL), following its success in the language domain. However, video reasoning poses distinct challenges. A model must first locate the segments relevant to the question along the temporal axis, then reason from the visual evidence they provide. Existing rewards supervise only the final answer, leaving the first of these unconstrained. Recent approaches annotate temporal spans or invoke external perception tools at inference, incurring annotation or computational cost, and even an explicit interval reward leaves the fraction of correct answers unsupported by the model's own attention unchanged. We propose VideoAPR, a reinforcement learning framework that grounds video reasoning in the model's own attention. We introduce AP-GRPO, which derives a temporal reward from where the policy already attends, requiring no temporal annotation during training. Within each rollout, position-debiased attention proposes a candidate temporal segment, and the response the policy has already produced is rescored with the visual context restricted to that segment and to an equal-length control window elsewhere in the clip, with the margin between the two forming the reward. Verification is performed only during training, so inference remains a single forward pass at no additional cost. Extensive experiments demonstrate that VideoAPR achieves state-of-the-art performance across diverse video reasoning benchmarks, outperforming recent methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Reasoning, Reinforcement Learning, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RnGyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether reinforcement-learning-trained video models answer from the moments they attend to. Using a calibrated attention-extraction procedure over prefill attention with position debiasing, the authors show that the fraction of correct predictions unsupported by the model's own attention drops from 0.64 in the base model to about 0.47 after outcome-reward RL, and that an explicit interval IoU reward leaves this fraction unchanged.

Critical strengths
  • The core idea is elegant: verifying the attended interval by restricted-context rescoring against an equal-length control window removes the need for temporal annotation, and the equal-length and correctness-gating design choices are each justified.
  • The diagnostic analysis is a genuine contribution: showing that an interval IoU reward raises grounded and ungrounded predictions in the same proportion is a clear negative result about what interval supervision constrains.
Critical weaknesses
  • The state-of-the-art claim is thin: the average margin over VideoRFT is 0.8 points from single training runs with no variance or significance estimates, and VideoAPR trails VideoRFT on two of the six benchmarks.
  • The finding that RL leaves correct predictions ungrounded is established only on Charades-STA temporal grounding, yet the conclusion and the motivation generalize it to video reasoning broadly without testing the extraction or the finding on any other domain.
Questions for authors
  • Does the Video-R1-260k RL data mixture contain Charades-STA or other temporal-grounding items with human interval annotations?
  • Can you report a second training seed or a variance estimate for the main table?
  • Could you validate the attention extraction against annotated spans on one non-Charades benchmark, and report an ungrounded rate on one of the six evaluation benchmarks using an independent grounding measure?
Reviewer_yTTgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose VideoAPR, extracts video segments from attention and uses the response likelihood difference between these segments and distant control segments as an RL reward for grounded video reasoning.

Critical strengths

Segment extraction and contrastive scoring only during training, with no additional inference steps, makes the method straightforward to use in existing video models. The ablation provides meaningful evidence that this reward improves downstream performance. Table 3 controls initialization, data, and the number of RL steps, and the contrastive variant improves over both vanilla GRPO and the focus-only variant across all six benchmarks. The MVBench improvement over the controlled GRPO baseline is substantial, from 61.9 to 68.8.

Critical weaknesses
  1. Table 1 measures grounding through agreement between attention derived and predicted intervals, but agreement does not establish that the answer depends on this evidence.
  2. Equation 6 checks correctness only on the full video, so the likelihood margin can be positive even when the model answers incorrectly from the attended window alone. So the claim that correctness gating prevents rewarding responses that fail under cropping does not follow from the objective.
  3. APPO (Henghui Du et al., arXiv:2602.23823) also uses attention to guide RL for video reasoning through frame selection and perception-token reweighting, but is neither discussed nor evaluated. A comparison with matched initialization, training data, and budget would clarify the benefit of VideoAPR’s restricted context likelihood reward over this related approach.
  4. In Table 1, VideoAPR’s grounded and ungrounded components sum to 0.567 rather than the reported accuracy of 0.607, contradicting the partition in Equation 4. The reported ungrounded rate is also inconsistent with 0.149 / 0.607, so the row needs correction.
Questions for authors
  • Does the model still answer correctly using only the attended window? How does accuracy change with matched random windows or after removing the selected evidence?

  • How does performance change when attention heads are selected without temporal annotations?

  • How does VideoAPR compare with APPO under matched initialization, training data, and training budget?

Reviewer_gcFZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • VideoAPR adds an attention-proposed temporal reward to GRPO for video reasoning. It extracts an attended window and scores each generated response under that window versus an equal-length distant control window.
  • The likelihood margin supplements answer and format rewards for correct responses. Verification occurs only during training, leaving inference unchanged.
  • Evaluation uses Qwen2.5-VL-7B across six benchmarks, with reward ablations and tests at two frame budgets.
Critical strengths
  • Targeted reward design: Contrasting attended and control windows provides a plausible way to encourage evidence-sensitive responses without additional inference procedures.
  • Useful controlled ablations: Under the same initialization, data, and training steps, the contrastive reward improves over both ordinary GRPO and attended-window scoring alone. This supports its effectiveness beyond the base training recipe.
Critical weaknesses
  • The attention-specific grounding claim is insufficiently validated. Missing random- or fixed-window controls leave unclear whether gains require attention-based proposals or arise from generic crop-contrast training. The grounding metric also relies on the attention extraction mechanism used by the method, limiting independent evidence of more faithful reasoning.
  • Generality and robustness remain uncertain. Training is demonstrated only on Qwen2.5-VL-7B, with no validation on newer backbones such as Qwen3-VL. Multi-seed uncertainty and training-cost comparisons are also absent, making the modest average advantage over existing RL methods difficult to assess.
Questions for authors
  • Clarification: Were the main comparisons repeated across training seeds, and what is the variance of the reported gains?
  • Clarification: How much additional training compute does verification require, and does the advantage persist against GRPO with a matched compute budget?
  • Assessment-relevant experiments: Does attention-based window selection outperform random or fixed windows under the identical reward? Independent evidence-retention/deletion tests would also help establish improved grounding.
  • Assessment-relevant experiment: Does the reward retain its benefit on a newer backbone such as Qwen3-VL? This would help determine whether the contribution generalizes beyond the evaluated model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.