acceptodds
Withdrawn from ICLR 2027

Less from More: Reinforcing Sparse Video Reasoning from Dense References

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video-language models commonly assume that more temporal observations lead to more reliable reasoning. We question this assumption and argue that the key challenge is not merely processing more video frames efficiently, but learning to reason reliably under limited temporal evidence. We propose SAVER, a dense-to-sparse post-training framework that uses dense video views as training-time references for sparse-frame inference. During reinforcement post-training, paired dense and sparse views are optimized with grounding rewards and a reliability-gated reference reward, encouraging sparse-view predictions to preserve task-relevant temporal evidence. Notably, SAVER is trained only on 1,250 randomly sampled temporal grounding examples, without using any video question answering annotations. Across three temporal grounding benchmarks and six video question-answering benchmarks, SAVER consistently improves performance across frame budgets. In particular, SAVER can match or surpass dense-frame Qwen3.5 baselines while using substantially fewer frames. These results show that temporal grounding can serve as an effective evidence-localization proxy for learning sparse video reasoning that transfers to broader video understanding tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Language Models, Efficient Video Understanding, Temporal Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DmN1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SAVER, a RL framework for improving video understanding under sparse-frame inputs. The same policy processes paired dense and sparse video views, with grounding rewards supervising both branches and a gated reference reward encouraging agreement with the dense prediction when it achieves a higher grounding reward. Trained on 1,250 temporal-grounding examples without videoQA annotations during post-training, SAVER improves averaged performance across three grounding and six videoQA benchmarks. Experiments include multiple model sizes, frame rates, reward ablations, and comparisons with alternative training and selection methods.

Critical strengths
  1. Practical setting. SAVER improves performance at matched frame budgets without an additional selection module at inference time.

  2. Interesting transfer. Post-training on only 1,250 grounding examples improves several QA benchmarks without QA supervision during post-training. Ablations support contributions from joint-view training and the reference reward.

Critical weaknesses
  1. Limited evidence for the added value of dense references. Joint-view training already achieves 27.4 mIoU versus SAVER’s 30.0. Ground-truth references also perform similarly (29.5 mIoU; 50.1 QA accuracy). Variability estimates and training-cost comparisons are needed to establish the benefit of dense predictions.

  2. Unexplained transfer gains. At 0.1 fps, SAVER-4B gains 20.3 points on VideoMMMU but only about 1.0 point on average across the other five QA benchmarks. Some settings regress, and improved use of temporal evidence is not directly established.

  3. Unclear qualitative figures. Figure 2,3’s small thumbnails obscure the relevant action, and the displayed timestamps do not directly reflect the labeled frame rates. The actual model inputs and illustrative frames should be distinguished.

Questions for authors
  1. How are training steps, rollout counts, and compute matched across Tables 3–4?

  2. Are dense-reference gains over ground-truth references consistent across seeds, and what is their additional training cost?

Reviewer_fpkAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAVER, a reinforcement post-training framework for video understanding under sparse temporal observations. A shared policy processes paired dense and sparse views of the same video. Both views receive temporal-grounding supervision, while sparse predictions receive an additional interval-agreement reward when their paired dense prediction achieves a higher grounding reward. Using 1250 temporal-grounding examples and no video-QA post-training annotations, SAVER improves the efficiency performance across grounding and Video-QA benchmarks.

Critical strengths
  1. Sound and well-motivated methodological framework. The overall training framework is conceptually sound. Pairing two views of the same video and optimizing a shared policy provides a coherent way to transfer guidance across observation budgets.

  2. Promising empirical results with limited supervision. Using only 1,250 grounding examples and no video-QA post-training annotations, SAVER improves macro-averaged performance across multiple model sizes and frame budgets.

  3. Detailed experimental reporting and useful practical efficiency measurements. The appendix provides detailed results across frame rates and benchmarks, analyses of training configurations, and practical profiling of visual-token counts, peak GPU memory, GPU inference latency, and end-to-end latency. This information is valuable for practitioners assessing the accuracy–cost trade-off and choosing an appropriate input budget for deployment.

Critical weaknesses
  1. The gated reward can favor an inferior prediction, and its optimization consequences remain unresolved. Consider a video of duration T=100T=100, with ground-truth and dense intervals both equal to [40,60][40,60], and unit reward coefficients. Excluding the common format reward, a perfect sparse prediction receives reward 11, whereas [40,59][40,59] receives 0.94+0.95=1.890.94+0.95=1.89. Although this case is rare, I still question the poor guidance under this reward design. The authors acknowledge the non-monotonicity of the reward, but do not quantify how often this conflict occurs or explain its effect under their actual optimization procedure. Positive aggregate results alone do not resolve this objective-level concern.

  2. The proposed evidence-preservation mechanism is insufficiently supported. Improved localization and QA accuracy do not establish whether SAVER improves the use of retained visual evidence, strengthens temporal priors, or elicits capabilities already present in the base model. In Figure 3, none of the displayed sparse timestamps fall inside the target interval; better localization could consequently reflect inference from surrounding states or duration priors. The generic total-variation risk bound provides intuition, but does not tightly connect the gated interval reward to reduced distributional discrepancy or explain grounding-to-QA transfer.

  3. The distinctive contribution of dense references is not established robustly. Joint dense–sparse training already accounts for much of the grounding improvement. Moreover, replacing dense predictions with ground-truth references yields similar results: 29.5 versus 30.0 mIoU and 50.1 versus 50.8 QA accuracy, as shown in Table 4. Is a dense reward necessary?

  4. SAVER is introduced without an expansion of the acronym or an explanation of the method name.

Questions for authors
  1. Does the actual optimization mitigate the illustrated reward–quality conflict? Please provide more detailed theoretical analysis to support the effectiveness of this reward design, e.g., the derivation of the form of the gradients.

  2. Do the improvements depend on task-relevant sparse visual evidence? Could the authors evaluate performance separately when target evidence is directly visible, inferable from surrounding states, or unavailable? Controlled changes to the sampling phase at a fixed frame budget would be particularly informative.

  3. Is the additional benefit of dense-reference guidance reproducible and specific to the reference? Please report variability across repeated runs for the joint-view, ground-truth-reference, and dense-reference configurations, including their per-benchmark QA results. A controlled, reproducible advantage from dense references would strengthen the methodological contribution.

Reviewer_X9qCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SAVER, a dense-to-sparse reinforcement post-training framework for video understanding. Dense and sparse views of the same video are processed by a shared policy and optimized using temporal grounding rewards. An additional interval-agreement reward is activated when the sampled dense prediction has higher grounding quality than the paired sparse prediction. Training uses 1,250 temporal grounding examples without video-QA annotations. Evaluation covers three grounding and six video-QA benchmarks across three model sizes and multiple frame rates. The results show improved macro-averaged performance under matched input budgets, although transfer varies across individual tasks and the specific advantage of dense references over alternative supervision remains modest.

Critical strengths
  1. The same-backbone, same-frame-budget comparisons are a strong aspect of the evaluation. At 0.1 fps, SAVER-2B improves average grounding mIoU from 18.4 to 30.0 and average video-QA accuracy from 45.9 to 50.8. The latter nearly matches the dense 2.0-fps baseline’s 51.0 while reducing average QA frames from 139 to 38. Achieving this with grounding-only post-training provides meaningful evidence of cross-task transfer.

  2. The ablations examine sparse-only, dense-only, joint-view training, reference rewards, and reliability gating. Joint training reaches 27.4 grounding mIoU and 48.7 average QA accuracy, while the full method reaches 30.0 and 50.8. Comparisons with sparse SFT, Time-R1 RL, and ground-truth reference supervision help contextualize the proposed recipe. Practical profiling also complements frame-count reporting, while the paper correctly distinguishes base-model input-budget savings from a SAVER-specific speedup.

Critical weaknesses
  1. Replacing dense references with ground-truth intervals yields 29.5 grounding mIoU and 50.1 QA accuracy, compared with 30.0 and 50.8 for dense references. Without repeated runs or uncertainty estimates, these differences do not firmly establish a meaningful advantage of the proposed reference source. Cross-task gains are also uneven: at 0.1 fps, SAVER-4B gains 20.3 points on VideoMMMU, whereas the other five QA benchmarks improve by only 1.0 point on average. SAVER-2B loses 2.3 points on VideoMME at the same frame rate. These results support improved aggregate performance, but a broad account of improved temporal-evidence reasoning requires more targeted validation.

  2. The reference bonus disappears when a sparse prediction catches up with or surpasses its paired dense prediction. Consequently, a slightly inferior sparse prediction that receives a large agreement bonus can obtain a higher total reward than a more accurate prediction receiving no bonus. The paper appropriately acknowledges the lack of monotonicity, but does not quantify how often such reversals occur or their effect on learning. In addition, the total-variation risk inequality in Equation 11 is an intuition rather than a guarantee for the proposed interval-level reward: optimizing sampled interval agreement is not shown to control distributional total variation.

Questions for authors
  1. Are the improvements over ground-truth reference supervision reproducible across training seeds and sampled 1,250-example subsets under matched rollout and optimization budgets?

  2. How frequently does the gated reference reward rank a worse-grounded sparse prediction above a better one? Does a reward modification that avoids these reversals preserve the reported gains?

  3. What explains the unusually low NExT-GQA baseline for Qwen3.5-4B and the particularly large VideoMMMU gains? Please report grounding parse-failure rates and evidence-sensitive analyses that separate format recovery or textual reasoning changes from improved use of temporal observations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.