acceptodds
Under review as a conference paper at ICLR 2027

Learning to Look Beyond a Single Frame for Video Understanding

Abstract

Large multimodal models (LMMs) have achieved incredible success in video understanding. However, they exhibit a fundamental limitation: single-frame bias. Instead of exploiting the full temporal sequence, they base their understanding on static cues from a few frames, typically the first or the last frame. This shortcut undermines the model’s ability to develop a holistic understanding of video content, often resulting in systematic errors. To mitigate this problem, we propose MF-GRPO, a novel framework that employs reinforcement learning (RL) to encourage LMMs to engage with the full temporal context. At its core is the multi-frame focus advantage enhancement, which is triggered using a contrastive objective. Specifically, the model receives the advantage enhancement when its understanding of the entire video sequence surpasses its understanding of a partially masked version of the same sequence. This enhancement mechanism explicitly encourages the model to integrate information across the entire temporal span, beyond the excessive reliance on superficial cues of a single frame. Furthermore, this RL training is designed to be progressive. We start with shorter video sequences and gradually extend their length, enabling the model to develop the multi-frame focus ability smoothly from easy to hard. Quantitative experiments show MF-GRPO effectively improves the model performance on various video understanding benchmarks. Critically, qualitative visualizations confirm that our model learns to allocate attention more broadly across frames, successfully mitigating temporal blindness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eMvROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a method to tackle the single-frame bias of large video language models. They do so by re-weighting the GRPO reward function, encouraging the model to achieve better predictions on longer videos compared to shorter reference videos. This is done in a progressive, curriculum-learning manner, where the number of frames of both the longer and shorter videos are gradually increased. In doing so, their proposed method outperforms the base model, as well as the base model post-trained with the standard GRPO objective over the six benchmarks evaluated. Furthermore, the authors perform some ablations and visualizations to gain more insights into the workings of the approach.

Critical strengths
  • The problem is important, and the approach is clearly motivated.
  • The method is clearly described and makes sense.
  • The paper is well-written.
Critical weaknesses

The main weakness of the work is that the contributions of the proposed method can not be isolated, as an experiment comparing it to training the baseline model with curriculum learning only is missing. This is necessary to disentangle the effects of the progressive training from the proposed multi-frame focus and validate the key claims of the paper.

Another important weakness is the fact that the vast majority of the analysis is done with a single model, Qwen2.5, which is quite outdated at this point. The appendix A.1 introduces another small model in InternVL3-2B, but also this is a relatively smaller and older model, and the A.1 table is incomplete as it does not include the baseline model performance. Another issue is that the claim in A.2, "Our method improves the performance from 47.33% to 49.22%, showing that mitigating single-frame bias also benefits long-video understanding," does not specify relative to which baseline this increase in performance is. In general, Appendix A contains a start towards important additions to the paper, such as performance on other models and a wider variety of standard tasks, but is completely unintegrated and unconvincing in its current state.

Questions for authors
  • How does the method perform compared to a baseline trained only with curriculum learning? How does it perform to an SFT baseline?
  • Does the method work on newer models?
  • While the attention visualizations are informative, a quantitative analysis of attention distributions would strengthen the paper.
  • Does the method work if the contrast video is always one frame, as the title implies, instead of N/2?
  • Why is fps=2 specified if the number of frames sampled is fixed?
Reviewer_7R5fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MF-GRPO to mitigate single-frame bias in video understanding. The method compares model performance on full videos and partially masked videos, amplifying positive advantages when the full-video input leads to better understanding. It also adopts a progressive training strategy that gradually increases video length. Experiments show improvements across several video understanding benchmarks, while attention visualizations suggest broader attention allocation across frames.

Critical strengths
  • The paper identifies single-frame bias as a limitation of video understanding and presents a straightforward contrastive mechanism to encourage the use of temporal information. The overall approach is easy to understand.

  • The method is evaluated on multiple video understanding benchmarks, with additional attention visualizations providing qualitative evidence that it encourages broader attention across frames.

Critical weaknesses
  • Contrastive training on original and perturbed inputs has been explored in prior work, including T-GRPO in Video-R1 [1] and PAPO [2]. Replacing frame shuffling with frame masking and injecting the contrastive signal through advantage scaling appears to be an incremental modification, whose distinct contribution is not sufficiently established.

[1] Feng et al. Video-R1: Reinforcing Video Reasoning in MLLMs. 2025.

[2] Wang et al. Perception-Aware Policy Optimization for Multimodal Reasoning.

  • The paper amplifies positive advantages by 20% when the contrastive condition is satisfied, but does not compare this with uniformly amplifying all positive advantages or randomly triggering the same enhancement at a matched frequency. Thus, it remains unclear whether the gains result from better temporal learning or simply from changing the weighting of positive updates.

  • Compared with Video-R1, the method changes both the contrastive input construction and the way the training signal is applied. The current experiments do not isolate the contribution of either change, nor directly validate the claim that additional rewards may be canceled out by group normalization.

  • The average improvement over Vanilla GRPO is around 2 percentage points across six benchmarks, with only a 0.6-point gain on MVBench. On AoTBench, MF-GRPO mainly recovers the performance lost during GRPO training rather than substantially exceeding the CoT-SFT baseline. Although results are averaged over three seeds, standard deviations or confidence intervals are not reported, making the reliability of smaller gains difficult to assess. Evaluation on stronger video models is also limited.

Questions for authors
  • How does MF-GRPO compare with uniformly amplifying positive advantages and randomly triggering the enhancement at the same frequency, under identical training conditions?

  • Can the authors separately ablate the two main differences from Video-R1: frame masking versus frame shuffling, and advantage scaling versus additional reward? Can they also provide direct empirical evidence for the claim that group normalization diminishes the effect of additional rewards?

  • What are the standard deviations or confidence intervals across the three random seeds, particularly for benchmarks with small improvements? Does MF-GRPO consistently outperform Vanilla GRPO across runs?

Reviewer_notnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MF-GRPO, a RL method intended to reduce LMM's reliance on individual frames. It gives successful answers extra training weight when full-video responses outperform reduced-frame responses, and gradually increases the number of input frames during training. Experiments on six video benchmarks, supplemented by tests on another model and long videos, show consistent improvements over standard GRPO.

Critical strengths
  1. Simple and practical training design. The method uses existing video question–answer data without requiring frame-level annotations, external expert models, or additional inference computation. This makes it straightforward to incorporate into existing training pipelines without changing deployment.

  2. Consistent benefits across settings. The method improves all six main benchmarks over standard GRPO. Improvements on more than one base model family and a long-video benchmark somehow suggest its usefulness in different evaluation settings.

Critical weaknesses
  1. The technical contribution over closely related methods is limited. Existing works already contrast original and perturbed videos (e.g., order) and explicitly discusses direct advantage adjustment. The paper does not provide a matched comparison showing that its frame-removal strategy, multiplicative adjustment, and curriculum improve upon that approach. Consequently, outperforming vanilla GRPO alone does not adequately establish the proposed method’s contribution. The major difference from existing works is the frame removal method for video perturbation, which is relatevely weak.

  2. Missing controlled comparisons with the closest baselines. The main matched baseline is vanilla GRPO, whereas Video-R1/T-GRPO and ArrowRL already use contrastive video reinforcement learning to improve temporal understanding. Video-R1 appears in the results table only as a reference, not under matched training conditions, and ArrowRL is absent. Without comparisons using the same backbone, training data, and budget, the reported gains do not establish whether MF-GRPO improves upon existing approaches rather than merely outperforming a generic RL baseline.

  3. The training signal does not reliably identify multi-frame reasoning. Comparing eight sampled full-video answers with one greedy reduced-video answer mixes the effect of additional frames with differences in decoding. The enhancement can therefore trigger without the model actually using additional visual information. Benchmark gains and attention visualizations do not resolve this ambiguity, leaving the paper’s central claim (that it reduces single-frame reliance) insufficiently supported.

Questions for authors

Please refer to weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.