- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Video-SR1, a self-rewarding RL framework for video reasoning on Qwen3.5-9B. GSRPO has the policy label each sentence of its own reasoning trace as correct or incorrect, matches sentences across rollouts by BGE cosine similarity, and computes a group-relative advantage per sentence. A temporal-sensitivity reward adds the KL between predictions under the original and a frame-shuffled video. Before RL, the policy is warm-started by SFT on rationales and sentence labels generated by Gemini-3.1-Pro. Experiments on six video benchmarks, with and without ground-truth answer rewards, report gains of 1.9-4.5 points over the strongest baseline.
Critical strengths
- Sentence-level credit assignment with semantic alignment across rollouts is a reasonable idea.
Critical weaknesses
1. Even taken at face value, the headline comparison is confounded by Gemini distillation. VIDEO-SR1 receives step-by-step rationales and sentence assessments from the powerful Gemini-3.1-Pro teacher, whereas VL-Calibration and UMPIRE receive standard QA fine-tuning (Sec. 3.1 and App. A). The paper reports neither the Gemini-trained checkpoint’s accuracy before RL nor competing RL objectives initialized from that same checkpoint. The “Base model” row does not identify this missing SFT control. The entire reported margin could be teacher distillation rather than GSRPO.
2. Experimental credibility is insufficiently substantiated.
-
(a) The experimental scale requires clarification against the disclosed compute budget. Appendix A reports two H100 GPUs and global batch size 64. The described protocol suggests five training-data settings—four benchmark-specific in-domain settings and one pooled setting for the two transfer benchmarks. With three methods without answer rewards, six with answer rewards, and four additional ablation configurations, this suggests 5 × (3 + 6 + 4) = 65 RL runs. If RL uses the reported hardware and batch size, and these runs execute sequentially, even one minute per step requires approximately 226 days for 5,000 steps per run. In my experience, a video-RL step at BS64 takes far longer than one minute. Yet Qwen3.5-9B’s March 2 release preceded the September 25 submission deadline by approximately 207 days, excluding data generation, SFT, and evaluation. The paper does not specify the actual RL resources, parallel execution, checkpoint sharing, or measured runtime needed to reconcile this estimate.
-
(b) Extensive data reuse makes the self-reward stability claim insufficiently supported. Appendix A reports 500 VLM4D warm-start examples, while Sec. 3.1 allocates approximately half the in-domain examples to warm-starting and half to RL. If these 500 examples represent distinct source questions, the corresponding VLM4D-only RL split should contain approximately 500 prompts. Under a batch of 64 prompts per step, 5,000 steps imply 5,000 × 64 / 500 = 640 dataset passes. Repeatedly optimizing model-generated rewards without ground-truth answer correction requires direct evidence against overfitting and reward drift. Figure 5b’s “nearly constant” ratio is aggregated across benchmarks and does not establish VLM4D-specific reward reliability throughout this prolonged training.
Questions for authors
-
What are the pre-RL SFT results, and how do competing objectives perform from the identical Gemini-trained checkpoint?
-
Please provide the actual RL hardware, batch definitions, run count, parallelism, GPU-hours, and representative timestamped training logs.
-
How many unique VLM4D RL prompts are used, and what independent-training and sentence-correctness evidence supports stable self-rewarding throughout 5,000 steps?
2 comments
Q2 and W2a: Clarification on compute and training configurations
There are two major assumptions underlying the proposed calculation that require clarification: 1) RL runs were not performed sequentially and 2) the batch size for RL was 16, not 64, which was reported as the group size G = 16 in the RL configuration section of Appendix A..
The reporting of 2 NVIDIA H100s in the appendix refers to the hardware requirements per training job, not the total number of GPUs used for all training runs. Training was conducted on a shared cluster containing 112 H100 GPUs, with many two-GPU jobs running concurrently (usually 16-24 jobs concurrently, depending on availability). With a batch size of 16, each step requires an average of 1 to 2 minutes. Thus, across the 65 total RL runs, the GPU hours was (2 GPUs x 1.5 x 5000 x 65)/60 = 16,250 hours. With an average of 20 jobs running concurrently, this amounts to only 12 days to perform all training runs for all the main results across all data settings and about 5 days for all the ablations. We expanded the RL configuration section and added an RL compute section in Appendix A in the updated manuscript to include these details.
Q3 and W2b: Clarification on VLM4D data reuse and reward stability
There are 500 unique prompts in the VLM4D RL training split, but again the batch size for RL was 16 (not 64) and includes the 16 different GRPO-style rollouts/responses for a randomly selected example from the RL data split. Thus, 5,000 RL steps is only 10 dataset passes (not 640) for VLM4D. Similar to above, we expanded Appendix A to include these details.
This clarification should directly address the reviewer’s objection regarding the possible VLM4D data reuse and associated reward stability concerns. Nonetheless, to directly inspect the reward stability of Video-SR1 on VLM4D, we include below the ratio of true to perceived accuracy during RL for this dataset.
| RL step | With ground-truth rewards | Without ground-truth rewards |
|---|---|---|
| 1000 | 1.03 | 0.93 |
| 2000 | 1.05 | 0.92 |
| 3000 | 0.98 | 0.93 |
| 4000 | 0.98 | 0.91 |
| 5000 | 0.96 | 0.90 |
We thank the reviewer for their detailed feedback. Below, we include pre-RL SFT results and outline planned/ongoing RL runs that were requested by the reviewer to directly investigate the confounding effects of the Gemini-based distillation during SFT. We also include critical clarifications regarding the proposed calculations for compute and VLM4D data reuse. Notably, the RL batch size was 16 (not 64), training runs were performed concurrently (not sequentially) over a couple weeks, and the VLM4D RL runs included only 10 data passes (not 640).
Q1 and W1: Reporting pre-RL SFT results, along with new RL runs, to evaluate confounding effects of teacher distillation from Gemini
We agree that the potential confounding effects of distillation through the Gemini-based synthetic data should be directly investigated. Below, we discuss 1) the pre-RL SFT results that drove the original justification for using standard QA SFT for the baselines, 2) the existing results that shed light on this concern, and 3) the new RL runs we are actively performing to directly evaluate this possible confounding effect.
Original justification based on pre-RL SFT results: We chose to use standard QA fine-tuning to initialize the baseline methods since we observed mixed results in the test accuracy across benchmarks after the standard warm-start versus the Gemini-based synthetic data that contains deliberately corrupted reasoning traces and corresponding self-reward reasoning. As shown in the table below (which is added to the paper in Appendix F), the Gemini synthetic warm-start achieves improved performance on only 3 of the 6 benchmarks and shows worse performance on 2 benchmarks (VideoEspresso and MVBench). Importantly, the VideoEspresso dataset already contains strong chain-of-thought reasoning traces, which is included in the standard QA fine-tuning. A second reason for using the standard QA fine-tuning for baselines is that the synthetic data includes non-standard features (e.g., corrupted traces and sentence-level self-reward rationales) that are not explicitly leveraged by the baseline methods during RL and may interact in unexpected ways with these methods. Nonetheless, we agree, despite the lack of improved pre-RL test accuracy, the Gemini synthetic warm-start may provide a stronger initialization for RL and, as described below, we are actively performing new RL runs to directly evaluate this.
Existing results: In addition to the results above, there is relevant controlled evidence already in the ablation results in Appendix E. Within each training-data setting, the full model and both reward ablations start from exactly the same Gemini synthetic warm-start. The ablations therefore retain the teacher-derived initialization while removing the specified RL reward component. Without answer rewards, removing sentence-level self-rewards reduces accuracy by 3.3–10.4 percentage points across the six benchmarks, while removing temporal-sensitivity rewards reduces accuracy by 1.2–4.4 points. With answer rewards, the corresponding reductions are 3.5–10.1 and 1.3–4.0 points. These differences arise under a fixed teacher-trained initialization and therefore cannot be explained by differences in teacher distillation between these configurations. These ablations directly support the contribution of the RL reward components conditional on the supervised warm start. We agree that they do not substitute for comparing competing RL objectives from the same checkpoint, which we do in the new RL runs we describe below.
New RL runs (in progress): We are currently performing the requested RL runs, in the settings with and without the ground-truth rewards, and will make a follow-up post here as soon as the results are available. For the sake of time, we are focusing on the strongest baselines (VL-Calibration and UMPIRE) and the benchmarks that seem to benefit most from the Gemini-based synthetic warm-start (VLM4D, NExT-QA, and STAR). We will also perform this analysis for the other baselines and benchmarks, but may run out of time before the end of the discussion period. When available, all results will be added to the manuscript.