acceptodds
Under review as a conference paper at ICLR 2027

Frame Differential On-Policy Self-Distillation for Video Reasoning

Abstract

Reinforcement learning (RL) has substantially improved the reasoning ability of multimodal language models through verifiable rewards and increasingly fine-grained visual or temporal credit assignment. In video reasoning, however, current RL methods typically train with a fixed sparse frame budget: increasing the number of frames makes autoregressive rollouts expensive, while too few frames may miss temporally localized events and fine-grained visual details. We present **Frame Differential On-Policy Self-Distillation (FD-OPSD)**, which transfers the useful evidence of dense frame observations to a sparse frame policy during RL training. FD-OPSD compares the policy's token level preferences for the same sampled response under sparse and dense views, and distills the resulting frame differential signal without an external teacher or dense autoregressive rollout. The method preserves sparse-frame rollouts and leaves inference unchanged. Across Qwen2.5-VL-7B and Qwen3-VL-4B on six video reasoning benchmarks, FD-OPSD yields higher overall average performance than the strongest corresponding GRPO, T-GRPO, or Video-KTR baselines across the 16, 32, and 64 frame evaluation settings. These results show that dense visual evidence can be transferred selectively during training through token level self-distillation while retaining sparse frame rollouts and unchanged inference.

Primary area:
reinforcement learning
Keywords:
Video Reasoning, Reinforcement Learning, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wsXwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FD-OPSD, which augments GRPO with self-distillation from a denser video view. Responses are generated using eight frames and scored by the same policy using 32 frames. Changes in token likelihoods, together with teacher confidence and trajectory reward, determine selective distillation weights. Experiments on two Qwen backbones and six benchmarks show modest average improvements over the strongest RL baselines.

Critical strengths
  1. Practical use of dense observations. Dense view scoring provides an auxiliary signal without a separate teacher or dense autoregressive rollouts, and introduces no additional inference cost.

  2. Reasonably broad empirical evaluation. Experiments cover two backbones, six benchmarks, and multiple frame budgets, with component ablations and training-time comparisons.

Critical weaknesses
  1. Limited and uncertain benefit of OPSD. In my own experiments, OPSD has not yielded useful improvements, which makes me skeptical of its practical value here.

  2. The reported average gains over the strongest RL baseline are only 0.8–1.3 points. Given the finite evaluation sets and the variability I have observed in such evaluations, these gains are not convincing.

  3. Baseline performance appears too low. Qwen models often perform better with direct answer prompting than with CoT. I would expect Qwen3-VL-4B to score above 60 on VideoMME with 32 frames, whereas the paper reports only 53.6. The evaluation appears to use CoT, raising concerns that the prompting setup understates baseline performance and inflates the apparent improvement from post-training.

Questions for authors
  1. What motivates the use of OPSD?

  2. Is CoT required during evaluation? Please specify the prompts, decoding settings, and answer extraction, and compare direct answer and CoT evaluation for both the instruction checkpoints and FD-OPSD under identical frame and pixel budgets.

Reviewer_qJUKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FD-OPSD, a self-distillation method for video reasoning that aims to produce responses from fewer frames approaching the quality of those generated with more frames. It adds a self-distillation loss to RL training: the student generates responses using 8 frames, while the same teacher mode scores the responses using 32 frames. The method improves average performance across two models and six benchmarks, outperforming the GRPO baseline and several related methods. The authors also provide extensive ablation studies and mechanism analyses to support the results.

Critical strengths
  1. The self-distillation setup is practical: it reuses the same model as the teacher, requires no separately trained teacher model.
  2. The method improves training efficiency compared with direct RL with more frames. Combining sparse-frame rollouts with dense-frame scoring avoids expensive generation.
  3. The paper is easy to follow, and the experiments are reasonably extensive. The method is clearly presented, with evaluations across multiple models, benchmarks, and frame budgets, along with several ablation studies and mechanism analyses.
Critical weaknesses
  1. The advantage of using more frames often comes from capturing evidence that sparse inputs miss. But distillation cannot give the student access to information absent from its input. In such cases, I do not think the proposed method can address the lack of visual evidence.
  2. The experimental setup does not align well with the motivation. The student is trained with 8 frames, but the main evaluations use only 16, 32, and 64 frames. Without 8-frame results, there is a concern that performance under the sparse-frame budget remains limited.
  3. Although the results show consistent improvements in average performance across models and frame budgets. The ablation results also suggest that adding an OPD loss to RL provides limited benefits. These results provide only limited support for the method's effectiveness.
  4. Increasing the distillation weight from 0.01 to 0.1 causes the average score to drop from 58.2 to 33.2. Such a sharp drop is difficult to be explained by an imbalance in loss weights and raises concerns about a potential conflict between the OPD optimization direction and the RL objective.
  5. The authors suggest that the method primarily improves video-semantic tokens, but the underlying mechanism is not sufficiently explained. The analysis is more descriptive than a causal validation of this claim.
  6. The related work lacks sufficient discussion of and comparison with other methods that use OPD for video-language reasoning, raising concerns about the novelty of the proposed method.
  7. Figure 1 lacks polish. Some text and background regions are blurry, giving it the appearance of an unrefined draft rather than a publication-ready figure.
Questions for authors
  1. Have the authors considered cases where the teacher has access to additional evidence unavailable to the student, and analyzed whether the method is effective in such cases?
  2. Why does the evaluation use different frame budgets from training? Can the authors report results using the same 8-frame budget as the student?
  3. A 32-frame input is still not particularly dense compared with an 8-frame input. Given that 64-frame evaluation results are already reported, have the authors tried using a 64-frame teacher?
  4. Overall, the improvements across benchmarks are limited. Can the authors provide stronger evidence of the method's effectiveness?
  5. Can the authors provide analysis about the sharp drop at a distillation weight of 0.1?
  6. The paper lacks discussion of OPD methods for video-language reasoning. Can the authors clarify how their method differs from similar approaches?
Reviewer_87K9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FD-OPSD, a video-RL method that transfers dense-frame evidence into a policy trained with sparse-frame rollouts. The policy generates responses from a sparse view, then the same model scores each fixed response under a denser view, and the token-level log-likelihood difference between the two views is distilled back through a confidence-aware, reward-gated fidelity term added to the GRPO loss. No external teacher, no dense rollout, inference unchanged. Across Qwen2.5-VL-7B and Qwen3-VL-4B on six video benchmarks at 16/32/64 frames, it beats the strongest matched GRPO/T-GRPO/Video-KTR baseline by 0.8–1.3 points on average while costing 40–47% less training time than dense-rollout or multi-pass alternatives. The signal concentrates on video-semantic tokens.

Critical strengths
  1. The core trick is clean and cheap: score the same response under sparse and dense views, take the token-level logprob difference, distill that. No dense rollout, no extra teacher, inference unchanged. That's a genuinely economical way to get dense-frame signal.
  2. The efficiency is concrete: 11.8h vs 19.7h (dense GRPO), 22.3h (T-GRPO), 19.2h (Video-KTR) — 40–47% cheaper while scoring higher. A real accuracy-cost win.
  3. They retrained the baselines under a matched protocol (same data, rollout count, response budget) instead of citing published numbers — that's what makes the comparison believable.
  4. The "dense isn't always better" honesty — 8% of cases where sparse beats dense-16 — directly justifies the confidence gating, so CASD isn't an arbitrary trick.
  5. The mechanism analysis carries confidence intervals (2.03×/2.30× semantic-to-stopword, 95% CI lower bound above 1) and holds across training, so the signal isn't noise.
Critical weaknesses
  1. The gains are small and there's no variance. The overall average beats the strongest baseline by only 0.8–1.3 points, and Table 1 reports no multiple seeds, no error bars. RL run-to-run variance is usually bigger than 1 point, so I can't tell whether "consistently outperforms the strongest baseline" is real or noise.
  2. Only Qwen-VL is tested (7B and 4B). One architecture family, so the generality claim isn't established — the authors acknowledge this in the limitations, but it's still a gap.
  3. No reproducibility statement, and no mention of releasing code, checkpoints, or seeds. For an RL method, that matters.
  4. The hand-designed R_think length reward (penalize <50 or >512 tokens) isn't ablated against the fidelity term, so I can't tell whether the two objectives fight each other.
Questions for authors
  • Can you report multiple-seed results or error bars for Table 1? A ~1-point gain needs a variance estimate to judge significance.
  • Is there any interaction between the fidelity objective and the R_think length reward — does the dense teacher's token preference conflict with the length shaping?
  • Have you checked the method on a non-Qwen architecture? Even a single result would strengthen the generality claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.