acceptodds
Under review as a conference paper at ICLR 2027

FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit

Abstract

Reinforcement learning with verifiable rewards, particularly Group Relative Policy Optimization (GRPO), has improved video reasoning in multimodal large language models. However, standard video GRPO varies only linguistic trajectories while conditioning every rollout in a group on the same sampled frames, which can produce homogeneous rewards and weaken group-relative learning when the shared visual evidence is insufficient or distracting. We propose FrameGRPO, a simple and effective reinforcement-learning framework that requires neither auxiliary networks nor changes to the model architecture. Specifically, FrameGRPO combines two lightweight components: (1) Multi-Frame Rollouts, which use only uniform temporal subsampling of a single maximal-frame decode to construct aligned views that diversify visual evidence and reasoning trajectories; and (2) Gated Temporal Efficiency Credit, which selectively reinforces a correct lower-ratio trajectory when all rollouts in a fixed full-ratio reference subset fail, without imposing an unconditional preference for fewer frames. Extensive experiments across diverse video understanding and reasoning benchmarks demonstrate robust overall improvements over standard GRPO, highlighting temporal-view diversification as an effective mechanism for exploration and credit assignment in video MLLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Reasoning, Reinforcement Learning, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_fhZWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FrameGRPO varies frame-sampling density across rollouts for the same question. Two of eight rollouts use the full view; the others use uniformly subsampled views. Correct sparse-view answers receive extra credit only when both full-view answers fail. All answers are then optimized under the full-view input. Experiments on two backbones report generally improved performance over GRPO without an additional inference-time module.

Critical strengths
  1. Low integration cost. The method reuses one decoded frame sequence and requires no architecture changes. Table 9 reports lower peak GPU memory and training time comparable to GRPO on Qwen2.5-VL, with some overhead on Qwen2-VL. No extra module is needed at inference.
  2. Controlled comparisons and repeated runs. The evaluation includes same-data GRPO baselines, two backbones, and three seeds on Qwen2.5-VL (Tables 1 and 5). Repeated improvements on several benchmarks strengthen the empirical case.
  3. Evidence across data budgets. Table 6 compares GRPO and FrameGRPO with 2K, 4K, and 8K training examples. Improvements on several tasks at each scale suggest that the benefit is not confined to one training-set size, although it is not universal across benchmarks.
Critical weaknesses
  1. Cross-view optimization needs justification. Answers are sampled under a sparse view, but both terms of the likelihood ratio in Eq. 5 condition on the full view. This is not an importance ratio relative to the actual sampling distribution. Table 3 supports the choice empirically, but the implications of applying sparse-view successes and failures to the full-view policy remain unclear.
  2. Reward diversity does not necessarily imply useful exploration. A lower all-correct rate can arise because sparse views lose necessary evidence. The gating ablations are informative, but aggregate curves do not quantify how often sparse views reveal additional successes rather than merely introduce failures. This distinction matters to the proposed mechanism.
Questions for authors
  1. How should Eq. 5 be interpreted under the change in visual conditioning? How large is the old-policy log-likelihood difference between sampled and full views, especially at low sampling ratios?
  2. Can existing logs report gate frequency and correctness by sampling ratio, including groups where only sparse-view answers succeed versus only full-view answers succeed?
  3. What frame counts, sampling rules, pixel budgets, and decoding settings are used at evaluation? Are they matched between GRPO and FrameGRPO, and how do they relate to the 32-frame training view?
5 comments
AuthorsOpenReview ↗

Response to Reviewer fhZW (Part 1/2)

We sincerely thank the reviewer for recognizing FrameGRPO’s low integration cost, absence of additional inference modules, controlled comparisons and repeated improvements, and effectiveness across different training-data budgets. We appreciate the constructive questions and provide additional measurements and clarifications below.


W1 & Q1: Cross-view optimization and likelihood differences


Thank you for highlighting this distinction. Eq. 5 is an off-policy trajectory-transfer surrogate: the behavior policy generates trajectories under sampled temporal views, while the target policy learns from them under the full view. The likelihood ratio measures policy change relative to the old full-view policy, rather than correcting for the sampling probability under each generating view. Group-relative advantages determine whether a sampled trajectory is reinforced or suppressed under the full context, while TEC selectively adds credit to successful sparse-view trajectories. Recent studies establish an off-policy interpretation of group-relative REINFORCE [1] and demonstrate the effectiveness of off-policy learning in language reasoning [2] and image reasoning [3].

To quantify the difference between the generating and optimization views, we evaluate the same trajectory under both views using the same old policy: Di=log⁡πold(oi∣Vρi,qi)−log⁡πold(oi∣V1,qi).D_i = \log \pi_{\mathrm{old}}(o_i \mid V_{\rho_i}, q_i) - \log \pi_{\mathrm{old}}(o_i \mid V_1, q_i). Here, the first term uses the sampled view that generated the trajectory, and the second uses the full view. A positive gap indicates that the trajectory is more likely under its generating view. The mean gaps by sampling ratio are:

Sampling ratioMean Dᵢ ↓
(75%, 100%]0.81
(50%, 75%]1.72
(25%, 50%]3.37
(0%, 25%]5.85

We also compare frame sampling with noise perturbation following NoisyRollout [3], and frame shuffling, as used in Video-R1 [4]:

Exploration strategyMean Dᵢ ↓
Noisy frames7.24
Shuffled frames4.37
Frame sampling2.94

Frame sampling yields the lowest average gap, supporting its suitability for transferring exploratory trajectories back to the original view. These measurements complement Table 5’s empirical evidence for full-view optimization.

References

[1] Yao et al. Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends. ICLR 2026.

[2] Yan et al. Learning to Reason under Off-Policy Guidance. NeurIPS 2025.

[3] Liu et al. NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation. NeurIPS 2025.

[4] Feng et al. Video-R1: Reinforcing Video Reasoning in MLLMs. NeurIPS 2025.

AuthorsOpenReview ↗

Response to Reviewer fhZW (Part 2/2)


W2 & Q2: Useful exploration and outcomes by sampling ratio


We agree that increased reward diversity alone does not establish useful exploration. Noise perturbation degrades visual appearance, while frame shuffling disrupts temporal order; both can make reasoning more difficult. In contrast, changing the sampling ratio affects reasoning non-monotonically: fewer frames can remove distractions, while more frames can recover missing evidence.

Beyond the lower all-correct rate, the all-incorrect group rate decreases from 11.8% with GRPO to 5.6% with FrameGRPO, consistent with Figure 1. Thus, greater reward diversity coincides with fewer all-incorrect groups. By exposing complementary temporal evidence, frame sampling helps uncover successful answers in cases where rollouts sharing a fixed view all fail, alleviating the loss of learning signals from uniformly incorrect groups.

We further analyzed accuracy by sampling ratio and the complementary success patterns. Accuracy is measured over individual rollouts within each ratio bin:

Sampling ratioAccuracy
(75%, 100%]77.1%
(50%, 75%]75.8%
(25%, 50%]73.9%
(0%, 25%]69.9%

The following frequencies are measured over all FrameGRPO rollout groups:

Group-level eventFrequency
Gate active: all full-view reference rollouts fail15.5%
Only sparse-view answers succeed10.0%
Only full-view answers succeed0.9%

“Only sparse-view answers succeed” means that at least one sparse-view answer is correct while all full-view reference answers are incorrect; “only full-view answers succeed” denotes the converse.

Although accuracy decreases on average at lower sampling ratios, 10.0% of groups contain successful sparse-view trajectories absent from the full-view reference outcomes. This directly demonstrates complementary discoveries beyond merely introducing additional failures. TEC selectively reinforces these verified successes, turning the additional exploration into useful learning signals.


Q3: Evaluation settings and their relationship to training


GRPO and FrameGRPO use identical evaluation settings for each backbone across all seven benchmarks, including frame sampling, pixel budgets, prompts, and decoding parameters. We follow Video-R1 [1] and use the decoding configuration from the official Qwen2.5-VL demo.

SettingEvaluation configuration
Frame countTarget 2 FPS; 4–64 frames, subject to available source frames, rounded down to an even count.
SamplingUniformly spaced indices across the entire video, rounded to the nearest integer; temporal order preserved.
Pixel budget100,352–200,704 pixels per frame; aspect ratio approximately preserved, with dimensions aligned to multiples of 28.
DecodingTemperature 0.01; top-p 0.001; maximum 1,024 generated tokens; one response per example.
Context limit16,384 tokens.

For longer videos, the 64-frame cap lowers the effective sampling rate, while the selected frames still span the entire video.

The 32-frame setting refers to the full-ratio view during training. At evaluation, both methods use the same protocol above, with up to 64 frames. Each example uses one video view and one generated response. Multi-Frame Rollouts and TEC operate only during training, so FrameGRPO receives the same evaluation input and decoding budget as GRPO, with no additional inference overhead.

Reference

[1] Feng et al. Video-R1: Reinforcing Video Reasoning in MLLMs. NeurIPS 2025.

Dear Reviewer fhZW,

With the rebuttal period ending in less than 24 hours, we would appreciate your feedback on our response. We have added experiments and analyses on cross-view likelihood gaps by sampling ratio, gate frequency, and complementary sparse-view successes, and clarified the off-policy interpretation of Eq. 5 and the matched evaluation settings.

We hope these additional results and clarifications address your concerns. Please let us know if any questions remain; we would be happy to discuss them further. Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_cbphOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FrameGRPO to improve reinforcement learning for video reasoning. The motivation is that standard GRPO uses the same video frames for all rollouts, which limits exploration and leads to similar rewards, so this paper proposes Multi-Frame Rollouts to generate responses using different numbers of video frames, and Gated Temporal Efficiency Credit to give additional rewards to successful responses from fewer frames when the full-frame rollouts fail. This method does not require changes to the model architecture, thus no additional inference overhead, and the improvements were shown on 7 video understanding benchmarks across 2 model backbones.

Critical strengths
  1. The motivation for improving GRPO by using different video frames to encourage more diverse rollouts is intuitive and convincing.
  2. The proposed method is simple and does not need any changes to the model architecture.
  3. I appreciate the experiments on 7 video understanding benchmarks across 2 model backbones to show the improvements.
  4. I also appreciate the ablation studies. E.g., Table 2 shows the effectiveness of Multi-Frame Rollouts and Temporal Efficiency Credit, while Figure 1 shows that the method improves reward diversity during training.
  5. Good inference efficiency (no additional inference overhead).
Critical weaknesses
  1. The method builds on existing GRPO by using different frame sampling ratios and modifying the reward, which are relatively simple techniques. Thus, the technical contribution is weak.
  2. Several improvements are small, e.g., +0.5% on TempCompass and +0.1% on NExT-GQA for Qwen2.5-VL-7B. It even degrades performance by 0.2% on VSI-Bench for Qwen2-VL-7B.
  3. The experiments only cover 7B models. Would this method still work on much larger models, e.g., 100B+, which already have high accuracy?
  4. The training dataset is small, as it is filtered from 116K samples to only 8K examples based on question difficulty.
Questions for authors
  1. Would this method work with 100B+ models, e.g., Qwen3.5-122B-A10B (weakness 3)?
  2. How would the method perform if trained on the full dataset without difficulty-based filtering (weakness 4)?
4 comments
AuthorsOpenReview ↗

Response to Reviewer cbph (Part 1/2)

We sincerely thank the reviewer for recognizing our convincing motivation, simple design without architectural changes, broad evaluation across seven benchmarks and two backbones, thorough ablations, and absence of additional inference overhead. We appreciate the constructive feedback and provide clarifications and additional experiments below.


W1: Technical contribution


We appreciate that the simplicity and practicality of FrameGRPO were recognized across the reviews. Our contribution addresses an important optimization bottleneck in video reasoning: low reward variance weakens the learning signals obtained from sampled rollouts. In video GRPO, conditioning every rollout on the same frames can produce correlated successes or failures, limiting the benefit of exploration through stochastic decoding alone.

FrameGRPO turns the frame sampling ratio into an exploration variable within each group. Because relevant evidence and distractions are unevenly distributed over time, different ratios can reveal complementary reasoning trajectories and restore informative reward contrasts. Gated Temporal Efficiency Credit then selectively reinforces successful lower-ratio trajectories when the full-view reference fails, and full-view optimization transfers these discoveries into the policy used at inference.

The technical value lies in identifying and addressing this video-specific source of advantage collapse through coordinated exploration, credit assignment, and trajectory transfer. Our lightweight implementation makes this mechanism practical to adopt without architectural changes or additional inference overhead.


W2: Magnitude and consistency of improvements


The empirical value of FrameGRPO is supported by the consistency of its improvements across diverse evaluation conditions. FrameGRPO outperforms GRPO in the vast majority of evaluated settings, spanning seven benchmarks, two backbones, component ablations, and three random seeds. Together, these experiments show that the benefits extend beyond an individual benchmark or training run.

The other three reviewers also recognized the empirical effectiveness, with Reviewer fhZW noting that “Repeated improvements on several benchmarks strengthen the empirical case.” In the revised paper, Appendix A reports robustness across three random seeds in Table 7 and training-set sizes of 2K, 4K, 8K, and 16K in Table 9, with improvements across nearly all settings. These results support robustness to training randomness and effectiveness across different data budgets. The additional experiments on a larger model and data without difficulty filtering, discussed below, further extend the empirical evidence.

AuthorsOpenReview ↗

Response to Reviewer cbph (Part 2/2)


W3 & Q1: Applicability to larger, higher-accuracy models


Training a 100B+ model is beyond our available computational resources. To examine whether FrameGRPO remains effective beyond 7B models, we conducted additional experiments with Qwen2.5-VL-32B-Instruct, a substantially larger backbone with strong starting accuracy.

MethodVideo-MMETempCompassVideoEval-ProMMVUVSI-BenchNExT-GQAVideo-MMMU
Qwen2.5-VL-32B-Instruct63.275.041.169.939.172.758.8
+ GRPO63.074.843.070.138.873.556.8
+ FrameGRPO64.375.444.272.339.374.060.2

GRPO provides limited overall improvement over the original model and reduces accuracy on several benchmarks. A plausible explanation is insufficient reward contrast among the generated responses, which weakens the group-relative learning signal. Other work [1] likewise identifies homogeneous rewards and advantage collapse as causes of training stagnation.

For models with higher initial accuracy, FrameGRPO can mitigate weak reward contrasts by diversifying temporal evidence and selectively reinforcing complementary successes. It improves over both GRPO and the original model on all seven benchmarks, extending our empirical validation to larger models with higher initial accuracy.

Reference

[1] He et al. Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation. ICML 2026.


W4 & Q2: Larger training sets without difficulty filtering


Due to computational and time constraints, we could not complete training on the full 116K dataset. To test dependence on difficulty filtering and effectiveness with more training data, we conducted additional experiments on randomly selected 8K and 16K datasets without difficulty-based filtering.

MethodData sizeVideo-MMETempCompassVideoEval-ProMMVUVSI-BenchNExT-GQAVideo-MMMU
GRPO8K60.173.641.965.932.279.450.8
FrameGRPO8K62.274.543.466.935.979.552.2
GRPO16K59.374.741.566.233.279.250.9
FrameGRPO16K62.775.142.467.037.679.754.0

FrameGRPO improves over GRPO on all seven benchmarks at both data sizes, demonstrating that its effectiveness does not depend on difficulty filtering. Moreover, the benefits of FrameGRPO over GRPO become more pronounced as the training data size increases.

This widening advantage supports FrameGRPO’s ability to benefit from a larger, unfiltered training set. It is also consistent with the training dynamics in Figure 1: temporal exploration maintains informative reward contrasts into later training stages, helping sustain useful learning signals as training progresses.

Dear Reviewer cbph,

With the rebuttal period ending in less than 24 hours, we would appreciate your feedback on our response. We have validated our method’s effectiveness on a larger model with higher initial accuracy and larger training datasets without difficulty filtering, and clarified our technical contributions and the consistency of the empirical improvements.

We hope these additional results and clarifications address your concerns. Please let us know if any questions remain; we would be happy to discuss them further. Thank you for your time and consideration.

Best regards,
The Authors

I appreciate the clarifications on the 32B model and larger unfilter datasets, so the weakness 3 and 4 are addressed. However, in weakness 1, while authors clarified the motivation and design of FrameGRPO, the technic is still a relatively simple extension of GRPO. Similarly for weakness 2, I acknowledge that the improvements are consistent, yet still small or even degrade Qwen2-VL-7B on VSI-Bench. Thus I kept my weak reject rating.

Reviewer_mSuuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FrameGRPO addresses advantage collapse in video GRPO, where all rollouts in a group see the same frames. Each group of 8 rollouts keeps 2 on the full 32-frame view and samples the others from uniformly subsampled views. A gated credit rewards a correct lower-ratio rollout only when both full-ratio rollouts fail, and policy optimization is conditioned on the full-ratio input. It is evaluated on seven benchmarks with two backbones, with ablations and three seeds on Qwen2.5.

Critical strengths
  1. A simple training recipe with measurable gains. It needs no new network or inference module. On Qwen2.5 it improves all seven benchmarks over GRPO by 0.1 to 3.8 points (Table 1), and the trend repeats over three seeds (Table 5).

  2. The gate has direct ablation support. On VSI-Bench, GRPO reaches 34.1, multi-frame rollouts alone 34.7, and the full method 37.9 (Table 2). Fixed credit and ungated credit reach 34.7 and 34.9 (Table 7).

Critical weaknesses
  1. The update does not match the sampling distribution. Responses are sampled from pi_old(.|V_rho, q), but the ratio in Eq. 5 uses pi_old(o|V_1, q) as its denominator. This is not the behavior probability, so the usual on-policy PPO/GRPO reading does not apply directly. It is closer to a heuristic that transfers sparse-view trajectories to the full view, and the cross-condition bias is not explained. Two full-view failures also do not show that a sparse-view success came from better visual evidence rather than generation randomness.

  2. The mechanism behind the gains is not isolated, and some claims are too strong. The current ablations cannot separate better visual exploration from a regularization effect of seeing fewer frames. "All evaluated settings" in the introduction conflicts with VSI-Bench on Qwen2 (27.6 to 27.4). Qwen2 GPU-hours also rise from 136.1 to 158.8 (about 16.7%), so training is not cheaper across the board.

Questions for authors
  1. Can you justify the cross-condition bias in Eq. 5, and test behavior-probability correction, weighted imitation or another trajectory-transfer objective?
  2. How does FrameGRPO compare with fixed low-frame GRPO matched on average visual tokens, and with ordinary frame dropout?
  3. Training data is filtered to 8K examples using 8 samples from the base model. Does the method work as well on data without this difficulty filtering?
  4. Clarification, not a basis for my rating: STRIVE (arXiv 2604.01824) also jointly groups video variants and text outputs to address low reward variance, yet the paper cites it only under retrieval. How do you see the relationship?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mSuu (Part 1/2)

We sincerely thank the reviewer for recognizing FrameGRPO’s simple training recipe, measurable improvements, and thorough ablation studies validating the effectiveness of our method. We appreciate the constructive questions and provide additional experiments and clarifications below.


W1 & Q1: Cross-condition optimization and behavior-probability correction


Thank you for highlighting this distinction. Eq. 5 is an off-policy trajectory-transfer surrogate: the behavior policy generates trajectories under sampled temporal views, while the target policy learns from these trajectories under the full view. This separates exploration from policy optimization, allowing reasoning paths discovered under complementary temporal evidence to improve the full-view policy. The denominator measures policy change relative to the old full-view policy, rather than correcting for the sampling probability under each generating view. Recent studies establish an off-policy interpretation of group-relative REINFORCE [1] and demonstrate the effectiveness of off-policy learning in language reasoning [2] and image reasoning [3], showing that trajectories collected outside the target policy’s sampling distribution can provide useful learning signals.

To directly test sensitivity to the denominator, we added Behavior-corrected FrameGRPO, which replaces the probability ratio in Eq. 5 with: riBC(θ)=πθ(oi∣V1,qi)πold(oi∣Vρi,qi).r_i^{\mathrm{BC}}(\theta) = \frac{\pi_{\theta}(o_i \mid V_1, q_i)}{\pi_{\mathrm{old}}(o_i \mid V_{\rho_i}, q_i)}. Here, the denominator evaluates the old-policy probability under the view that generated each trajectory, while the numerator remains conditioned on the full view. All other settings are unchanged, isolating the effect of the denominator choice.

MethodVideo-MMETempCompassVideoEval-ProMMVUVSI-BenchNExT-GQAVideo-MMMU
Behavior-corrected FrameGRPO60.973.741.966.935.779.152.2
FrameGRPO61.374.942.466.737.979.654.2

The original objective performs better on most benchmarks, supporting its effectiveness as a trajectory-transfer surrogate.

References

[1] Yao et al. Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends. ICLR 2026.

[2] Yan et al. Learning to Reason under Off-Policy Guidance. NeurIPS 2025.

[3] Liu et al. NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation. NeurIPS 2025.


W2 & Q2: Temporal exploration versus fewer-frame regularization


To examine whether the gains can be explained simply by using fewer frames, we added GRPO matched to FrameGRPO’s average visual-token budget and GRPO with frame dropout.

MethodVideo-MMETempCompassVideoEval-ProMMVUVSI-BenchNExT-GQAVideo-MMMU
GRPO (matched visual tokens)60.474.141.066.233.279.450.8
GRPO (frame dropout)60.472.741.265.833.579.351.6
FrameGRPO61.374.942.466.737.979.654.2

FrameGRPO consistently outperforms both controls. The visual-token-matched GRPO baseline gives every rollout the same reduced temporal evidence, so an insufficient or distracting view can still produce homogeneous group outcomes. Frame dropout randomly removes frames, potentially creating uneven temporal gaps that make motion and event progression harder to follow. In contrast, FrameGRPO’s uniform subsampling provides more regular coverage of the video at each sampling ratio, varying evidence density while retaining a structured temporal view.

We also thank the reviewer for pointing out the imprecise statements. We have revised them accordingly.

AuthorsOpenReview ↗

Response to Reviewer mSuu (Part 2/2)


Q3: Training without difficulty filtering


We additionally trained GRPO and FrameGRPO on 8K randomly selected examples without difficulty filtering.

MethodVideo-MMETempCompassVideoEval-ProMMVUVSI-BenchNExT-GQAVideo-MMMU
GRPO60.173.641.965.932.279.450.8
FrameGRPO62.274.543.466.935.979.552.2

FrameGRPO improves over GRPO on all seven benchmarks without difficulty filtering, demonstrating that its effectiveness does not depend on the filtering procedure. This is consistent with our motivation: varying temporal evidence can recover informative reward contrasts when a fixed view produces homogeneous outcomes.


Q4: Relationship to STRIVE


Thank you for highlighting STRIVE [1]. Low reward variance is a well-recognized challenge in video reinforcement learning that has motivated substantial effort to mitigate it. Below, we clarify the distinctions between FrameGRPO and STRIVE.

1. Exploration strategy. STRIVE’s primary importance-aware variant construction computes question–frame relevance using model-derived embedding similarity or gradient attribution, then performs relevance-guided sampling. This introduces additional model computation and relevance-scoring overhead, while making exploration depend on the scoring function’s preferences. FrameGRPO directly varies the frame sampling ratio without relevance scoring. The resulting changes in evidence sufficiency and distraction naturally vary reasoning difficulty while preserving temporal order.

2. Use of exploratory successes. STRIVE jointly normalizes rewards across visual variants and textual outputs. FrameGRPO uses Gated Temporal Efficiency Credit to make more effective use of exploratory trajectories, selectively reinforcing a correct lower-ratio trajectory when all rollouts in the fixed full-ratio reference subset fail. This directs additional credit toward complementary successes that were not discovered by the sampled full-view rollouts.

3. Policy optimization. STRIVE applies the GRPO objective over its variant-conditioned groups. FrameGRPO conditions policy optimization on the full video view, allowing successful reasoning trajectories discovered under sparse views to contribute directly to policy learning under the complete sampled video context. Our optimization-input ablation supports this transfer design.

Reference

[1] Bahrami et al. STRIVE: Structured Spatiotemporal Exploration for Reinforcement Learning in Video Question Answering. arXiv 2026.

Dear Reviewer mSuu,

With the rebuttal period ending in less than 24 hours, we would appreciate your feedback on our response. We have added experiments on behavior-probability correction, visual-token-matched GRPO, frame dropout, and training without difficulty filtering, and clarified our contributions.

We hope these additional results and clarifications address your concerns. Please let us know if any questions remain; we would be happy to discuss them further. Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_5f79OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper points out that in GRPO training for video MLLMs, all rollouts observe the same frames, which makes the rewards within a group homogeneous, a problem known as advantage collapse. To address this, the authors first propose Multi-Frame Rollouts, which randomly vary the frame sampling ratio for each rollout while performing the policy update on the full-frame input. They also propose Gated Temporal Efficiency Credit, which gives an additional reward to rollouts that answer correctly with fewer frames, but only when all full-frame rollouts fail. Experiments are conducted on two 7B backbones and seven video benchmarks.

Critical strengths

S1. The method is simple and practical. It only changes how rollouts are constructed and how advantages are computed, without any auxiliary network or extra inference cost, so it can be plugged directly into existing GRPO pipelines. Although the gains are modest, the paper shows that they are not due to chance. The motivation behind the proposed method is also clear.

S2. The ablation studies on design choices are thorough. They cover a wide range of factors, including the size of the reference subset, the credit design, training data size, random seeds, and training cost. In particular, Table 3 shows that rollouts obtained from reduced inputs improve performance only when they are optimized on the full input, and that optimizing them on the reduced inputs actually performs worse than GRPO.

Critical weaknesses

W1. For image reasoning RL, NoisyRollout (NeurIPS 2025) mixes rollouts from original and degraded inputs within a single group and conditions the policy update only on the original input. This is very similar to the proposed method, yet the paper does not cite it. Even the group construction, the optimization on the original input, and the observation that optimizing on degraded inputs performs worse are the same as in FrameGRPO’s Multi-Frame Rollouts. The temporal subsampling here is therefore closer to a natural adaptation of this recipe to video.

Questions for authors

Q1. Could you discuss how FrameGRPO differs from NoisyRollout?

Q2. Could you compare FrameGRPO against a control in which the frames are replaced with random noise or shuffled frames under the same rollout budget?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5f79

We sincerely thank the reviewer for recognizing FrameGRPO’s clear motivation, simple and practical design, compatibility with existing GRPO pipelines without auxiliary networks or additional inference cost, and thorough ablation studies. We appreciate the constructive feedback and provide clarifications and additional experiments below.


W1 & Q1: Differences from NoisyRollout and our contributions


Thank you for pointing us to NoisyRollout, an elegant and effective approach to improving visual reasoning through noise-driven exploration. We have added a citation to NoisyRollout in the revision. Below, we discuss the distinctions and our contributions.

1. Exploring temporal evidence. Adding noise is an effective exploration strategy, but it generally increases perceptual difficulty by degrading visual evidence. FrameGRPO instead varies which temporal evidence the model observes. Since relevant events and distractors occur at different times, changing the sampling ratio affects reasoning non-monotonically: fewer frames can remove distractions, while more frames can recover missing evidence. Frame sampling therefore explores complementary evidence configurations with different reasoning opportunities, providing a natural exploration mechanism for video reasoning while preserving the visual fidelity and chronological order of retained frames.

2. Smaller cross-view likelihood gaps support trajectory transfer. To assess how well exploratory trajectories remain supported under the original input, we measure the following log-likelihood gap during training: Di=log⁡πold(oi∣Vpert,qi)−log⁡πold(oi∣Vorig,qi).D_i = \log \pi_{\mathrm{old}}(o_i \mid V_{\mathrm{pert}}, q_i) - \log \pi_{\mathrm{old}}(o_i \mid V_{\mathrm{orig}}, q_i). Here, VpertV_{\mathrm{pert}} is the perturbed video view that generated trajectory oio_i, and VorigV_{\mathrm{orig}} is the original, unperturbed video view. Both likelihoods evaluate the same trajectory under the same old policy. A positive gap indicates that the trajectory is more likely under its generating view; a smaller positive average indicates less likelihood mismatch when transferring these trajectories to optimization on the original input.

Exploration strategyMean Dᵢ ↓
Random noise7.24
Shuffled frames4.37
Frame sampling2.94

Frame sampling yields the lowest average gap, supporting its suitability for transferring exploratory trajectories back to the original view.

3. Selective credit assignment and sustained exploration without annealing. Beyond constructing exploratory views, FrameGRPO introduces Gated Temporal Efficiency Credit (TEC). It selectively reinforces correct lower-ratio trajectories only when all rollouts in the fixed full-ratio reference subset fail. This identifies useful discoveries missed by the full-view rollouts, without unconditionally rewarding fewer frames.

NoisyRollout notes that certain noise characteristics may hinder convergence, and its ablation shows that removing noise annealing reduces accuracy. With smaller cross-view distribution shifts, FrameGRPO requires no annealing and continues to provide informative reward contrasts even late in training.


Q2: Controls with random noise and shuffled frames


We conducted the requested controls by replacing frame sampling with random noise or frame shuffling. To isolate the exploration mechanism, we removed TEC from all variants and kept the rollout budget and all other training settings identical. The noise control follows NoisyRollout’s noise injection mechanism, with the noise step randomly sampled between 0 and 500. “No Expl.” denotes the baseline without additional visual exploration.

ExplorationVideo-MMETempCompassVideoEval-ProMMVUVSI-BenchNExT-GQAVideo-MMMU
No Expl.60.074.441.065.634.179.552.0
Random noise60.073.139.566.433.279.151.4
Shuffled frames60.274.640.464.831.978.849.8
Multi-Frame Rollouts60.774.441.866.334.779.852.2

The results favor Multi-Frame Rollouts over both alternative exploration strategies. Random noise introduces a larger cross-view distribution shift, which may weaken the transfer of exploratory trajectories to the original input. Frame shuffling disrupts chronological relationships essential to temporal reasoning. Multi-Frame Rollouts preserve visual fidelity and temporal order while exploring complementary combinations of relevant evidence and distractions. This allows exploration to uncover useful reasoning trajectories with less disruption to the underlying video structure, making frame sampling particularly suited to video reasoning RL.

Thank you for the detailed response and additional experiments. I have carefully read the rebuttal, and it addresses most of my concerns. I continue to consider this work a valuable contribution and remain positive about the paper overall.

We sincerely thank you for the time and care you have devoted to reviewing our paper, response, and additional experiments. Your constructive feedback has helped us clarify our contributions and strengthen the empirical evidence. We are pleased that our rebuttal has addressed most of your concerns, and we greatly appreciate your continued positive assessment of the paper and its value.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.