acceptodds
Under review as a conference paper at ICLR 2027

Learning to Reason Efficiently in Videos: Reinforcement Learning with Short- and Long-Term Rewards

Abstract

Recent advancements in multimodal reasoning models have shown promising results, but these models often inherit a language-centric reasoning bias and fall short in true multimodal understanding. In long video understanding, a core challenge in multimodal reasoning, such models tend to generate lengthy and reflective chain-of-thought (CoT) outputs that do not meaningfully contribute to video comprehension. Moreover, the excessive token generation during reasoning leads to inefficiencies in further training. To address these issues, we propose a simple post-training framework for building efficient video reasoning models. Specifically, to mitigate the text-biased reasoning patterns inherited from language models, we first distill an Instruction multimodal model using the video reasoning trajectories of a native multimodal reasoning model, allowing it to learn dynamic video reasoning. During the reinforcement learning stage, to improve data efficiency, we leverage the instruction-tuned model to filter long-form video question-answering data, retaining only a compact set of high-quality training samples. For reward design, we introduce a hybrid mechanism that incorporates both short-term and long-term rewards, optimized via group relative policy optimization (GRPO). Extensive experiments on diverse video benchmarks show that our framework transforms Qwen3-VL-Instruct into a more accurate and efficient video reasoning model, substantially outperforming the open-source Qwen3-VL-Thinking baseline. Consistent gains on Qwen2.5-VL and Qwen3.5 further demonstrate its generalizability. These results highlight the importance of dynamic reasoning and efficient reward design for advancing multimodal video reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video-LLMs, video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DH9kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The method trains video reasoning with final-answer correctness plus an intermediate-answer signal. It combines Seed1.8 distillation with GRPO on filtered long-video questions; the reward favors correct recovery while penalizing changes between intermediate and final answers. Results beat outcome-only GRPO across several benchmarks and backbones, but the reward's efficiency rationale is not fully validated.

Critical strengths
  1. Against outcome-only GRPO with the same teacher SFT and filtered RL data, the full method improves all five general-understanding benchmarks. Gains also appear without teacher supervision.
  2. Accuracy improves across several Qwen backbones, especially Qwen3.5. The recipe uses a relatively small RL set and verifies intermediate answers without a separate process reward model.
Critical weaknesses
  1. The reward can favor a worse intermediate state. Because R=3rl−rs−1R = 3r_l - r_s - 1, an incorrect intermediate answer earns more than a correct one when final correctness is fixed. Applying the same trajectory advantage to all tokens may therefore reward delaying answerability. Report training-time early/final accuracy and transition rates, and ablate the reward ordering. Appendix C.3 does not remove this incentive: a correct response before the checkpoint can receive less reward than a longer response that first fails and then recovers.
  2. The accuracy–efficiency advantage over direct answering remains unclear. The method achieves 69.9 on Video-MME versus 69.5 for Instruct, but underperforms on the long-video subset, MLVU, and VSI-Bench. Moreover, Figure 1(b) lacks response-length comparisons with Instruct, SFT-only, and matched GRPO baselines, making it difficult to attribute efficiency gains to the proposed reward. Reporting accuracy, response length, and latency for these baselines would strengthen the claim.
Questions for authors
  1. How do intermediate-answer accuracy and incorrect-to-correct versus correct-to-correct transition rates change during training? Does final accuracy improve without reducing early answerability?
  2. What happens if trajectories correct both early and finally receive at least as much reward as trajectories that recover from an early error, with all else fixed?
  3. Could the authors report response length and inference latency alongside accuracy for direct-answer Instruct, SFT-only, outcome-only GRPO, and the full method? A direct-answer baseline fine-tuned on the same QA data would further clarify the benefit of explicit reasoning.
Reviewer_Ewr9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a two-stage post-training framework for building efficient video reasoning models from instruction-tuned MLLMs. The first stage distills dynamic-length reasoning trajectories from a native multimodal thinking model into Qwen3-VL-8B-Instruct via SFT on 22k video QA samples. The RL stage first filters easy samples via multiple pre-sampling, then trains with GRPO using a hybrid reward that probes an intermediate answer at an early reasoning checkpoint and combines it with final-answer correctness. Experiments on nine video benchmarks show the model outperforms Qwen3-VL-Thinking on most benchmarks with substantially more concise, difficulty-adaptive reasoning, and gains generalize to Qwen2.5-VL, Qwen3.5, and Qwen2-VL.

Critical strengths
  1. The efficiency focus is well-motivated and backed by evidence: the model produces concise, difficulty-adaptive reasoning and achieves gains with only 0.5k RL samples and 10 hours of training.
  2. The empirical evaluation is thorough, including multi-backbone generalization, an RL-only ablation without teacher distillation, frame-budget robustness, and an internally fair comparison against a GRPO baseline under the same setup.
Critical weaknesses
  1. Limited Novelty: distillation SFT and pass-rate-based data filtering are standard practice, and the core contribution — probing correctness at one fixed token position and combining two binary signals with hand-tuned coefficients — is a coarse heuristic that is not compared against existing process-reward methods (e.g., VisualPRM, step-wise GRPO).
  2. The reward design may incentivize degenerate behavior: trajectories that are initially wrong but finally correct receive a higher reward than consistently correct ones, effectively rewarding "wrong-then-recover" patterns, yet no evidence is provided that this reflects genuine reflection rather than learned detours.
  3. Small gains over the GRPO baseline on several benchmarks (e.g., +0.3 on VSI-Bench, +0.6 on Tomato) are reported without variance or significance analysis, and the claim of "substantially outperforming" Qwen3-VL-Thinking is overstated given clear losses on VSI-Bench and MMVU under different token budgets.
Questions for authors

See the weaknesses

Reviewer_cUvWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a reinforcement learning framework to improve video reasoning while reducing unnecessary reasoning tokens. The motivation is that existing multimodal reasoning models often generate overly long reasoning chains, so this paper first uses a stronger multimodal reasoning model to generate reasoning trajectories for SFT, then filters video QA data and introduces short-term and long-term rewards for GRPO training. Improvements were shown on many video understanding and reasoning benchmarks across 3 model backbones.

Critical strengths
  1. The motivation for improving video reasoning efficiency is intuitive and convincing. I appreciate the observation that stronger reasoning models do not necessarily perform better on video understanding tasks, as shown in Figure 1.
  2. The proposed short-term and long-term reward design is simple and intuitive. It encourages the model to correct mistakes while penalizing reasoning that flips correct into incorrect answers.
  3. I appreciate the experiments on video understanding and reasoning benchmarks to show the improvements. The generalization experiments across multiple QWEN models are also helpful.
  4. The ablation studies are useful. E.g., Table 5 shows the effectiveness of the proposed reward design and compares different short-term reward positions.
Critical weaknesses
  1. The method mainly combines existing techniques, e.g., CoT distillation (Video-R1: Reinforcing Video Reasoning in MLLMs, NeurIPS 2025), data filtering, and GRPO, with a relatively simple reward function. Thus, the technical contribution is weak.
  2. The improvements over the standard GRPO are sometimes small. E.g., Table 4 shows improvements of only +0.3% on VSI-Bench, +0.6% on MMVU, +0.3% on LongVideo-Reason, and +0.6% on Tomato. The method also performs much worse than Qwen3-VL-Thinking on MMVU (71.0% vs. 78.1%) and VSI-Bench (54.4% vs. 59.4%).
  3. The RL stage only uses 500 filtered video QA samples. Would it work with a much more diverse and larger RL dataset?
  4. The experiments only cover models up to 9B parameters. Would it work on much larger models, e.g., 100B+, which may already have stronger video reasoning capabilities?
Questions for authors
  1. Would this method work with a much larger RL dataset, e.g., 10K or 100K samples (weakness 3)?
  2. Would this method work on much larger models, e.g., Qwen3.5-122B-A10B (weakness 4)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.