acceptodds
Under review as a conference paper at ICLR 2027

Video-SR1: Reinforcement Learning with Self-Rewarding Video-Language Reasoning

Abstract

Vision-language models increasingly support video, but reinforcement learning (RL) for video reasoning still relies largely on sparse final-answer rewards. Such supervision is poorly suited for long video-reasoning traces, where correct observations and localized perceptual or temporal errors may coexist within the same response. We introduce Video-SR1, a self-rewarding framework for dense RL post-training of video-language models. Its core algorithm, Group and Sentence Relative Policy Optimization (GSRPO), uses the policy to apply self-rewards across each sentence in its own reasoning trace, aligns semantically corresponding sentences across responses, and performs group-relative credit assignment at the sentence level. Further, GSRPO complements the self-rewards with a counterfactual temporal-sensitivity reward that encourages reasoning to respond to changes in video frame order. Across six diverse video-reasoning benchmarks, Video-SR1 significantly improves accuracy, sample efficiency, and cross-domain transfer relative to strong baselines in settings with and without the sparse ground-truth answer reward included during RL. Error analyses and ablations reveal two complementary factors underlying Video-SR1's gains: the temporal-sensitivity reward promotes reasoning that responds to the video's spatiotemporal structure, while local self-rewards ensure the temporally-sensitive reasoning remains accurate and grounded in the visual evidence. Together, our results suggest that a central limitation of RL for video reasoning is not only reward quality, but reward resolution: assigning credit at the level of intermediate claims can make each rollout more informative and provide a stronger training signal for grounded multimodal reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language model, video reasoning, reinforcement learning, self-rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HNnpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Video-SR1, a self-rewarding RL framework for video reasoning on Qwen3.5-9B. GSRPO has the policy label each sentence of its own reasoning trace as correct or incorrect, matches sentences across rollouts by BGE cosine similarity, and computes a group-relative advantage per sentence. A temporal-sensitivity reward adds the KL between predictions under the original and a frame-shuffled video. Before RL, the policy is warm-started by SFT on rationales and sentence labels generated by Gemini-3.1-Pro. Experiments on six video benchmarks, with and without ground-truth answer rewards, report gains of 1.9-4.5 points over the strongest baseline.

Critical strengths
  1. Sentence-level credit assignment with semantic alignment across rollouts is a reasonable idea.
Critical weaknesses

1. Even taken at face value, the headline comparison is confounded by Gemini distillation. VIDEO-SR1 receives step-by-step rationales and sentence assessments from the powerful Gemini-3.1-Pro teacher, whereas VL-Calibration and UMPIRE receive standard QA fine-tuning (Sec. 3.1 and App. A). The paper reports neither the Gemini-trained checkpoint’s accuracy before RL nor competing RL objectives initialized from that same checkpoint. The “Base model” row does not identify this missing SFT control. The entire reported margin could be teacher distillation rather than GSRPO.

2. Experimental credibility is insufficiently substantiated.

  • (a) The experimental scale requires clarification against the disclosed compute budget. Appendix A reports two H100 GPUs and global batch size 64. The described protocol suggests five training-data settings—four benchmark-specific in-domain settings and one pooled setting for the two transfer benchmarks. With three methods without answer rewards, six with answer rewards, and four additional ablation configurations, this suggests 5 × (3 + 6 + 4) = 65 RL runs. If RL uses the reported hardware and batch size, and these runs execute sequentially, even one minute per step requires approximately 226 days for 5,000 steps per run. In my experience, a video-RL step at BS64 takes far longer than one minute. Yet Qwen3.5-9B’s March 2 release preceded the September 25 submission deadline by approximately 207 days, excluding data generation, SFT, and evaluation. The paper does not specify the actual RL resources, parallel execution, checkpoint sharing, or measured runtime needed to reconcile this estimate.

  • (b) Extensive data reuse makes the self-reward stability claim insufficiently supported. Appendix A reports 500 VLM4D warm-start examples, while Sec. 3.1 allocates approximately half the in-domain examples to warm-starting and half to RL. If these 500 examples represent distinct source questions, the corresponding VLM4D-only RL split should contain approximately 500 prompts. Under a batch of 64 prompts per step, 5,000 steps imply 5,000 × 64 / 500 = 640 dataset passes. Repeatedly optimizing model-generated rewards without ground-truth answer correction requires direct evidence against overfitting and reward drift. Figure 5b’s “nearly constant” ratio is aggregated across benchmarks and does not establish VLM4D-specific reward reliability throughout this prolonged training.

Questions for authors
  1. What are the pre-RL SFT results, and how do competing objectives perform from the identical Gemini-trained checkpoint?

  2. Please provide the actual RL hardware, batch definitions, run count, parallelism, GPU-hours, and representative timestamped training logs.

  3. How many unique VLM4D RL prompts are used, and what independent-training and sentence-correctness evidence supports stable self-rewarding throughout 5,000 steps?

2 comments

We thank the reviewer for their detailed feedback. Below, we include pre-RL SFT results and outline planned/ongoing RL runs that were requested by the reviewer to directly investigate the confounding effects of the Gemini-based distillation during SFT. We also include critical clarifications regarding the proposed calculations for compute and VLM4D data reuse. Notably, the RL batch size was 16 (not 64), training runs were performed concurrently (not sequentially) over a couple weeks, and the VLM4D RL runs included only 10 data passes (not 640).

Q1 and W1: Reporting pre-RL SFT results, along with new RL runs, to evaluate confounding effects of teacher distillation from Gemini

We agree that the potential confounding effects of distillation through the Gemini-based synthetic data should be directly investigated. Below, we discuss 1) the pre-RL SFT results that drove the original justification for using standard QA SFT for the baselines, 2) the existing results that shed light on this concern, and 3) the new RL runs we are actively performing to directly evaluate this possible confounding effect.

Original justification based on pre-RL SFT results: We chose to use standard QA fine-tuning to initialize the baseline methods since we observed mixed results in the test accuracy across benchmarks after the standard warm-start versus the Gemini-based synthetic data that contains deliberately corrupted reasoning traces and corresponding self-reward reasoning. As shown in the table below (which is added to the paper in Appendix F), the Gemini synthetic warm-start achieves improved performance on only 3 of the 6 benchmarks and shows worse performance on 2 benchmarks (VideoEspresso and MVBench). Importantly, the VideoEspresso dataset already contains strong chain-of-thought reasoning traces, which is included in the standard QA fine-tuning. A second reason for using the standard QA fine-tuning for baselines is that the synthetic data includes non-standard features (e.g., corrupted traces and sentence-level self-reward rationales) that are not explicitly leveraged by the baseline methods during RL and may interact in unexpected ways with these methods. Nonetheless, we agree, despite the lack of improved pre-RL test accuracy, the Gemini synthetic warm-start may provide a stronger initialization for RL and, as described below, we are actively performing new RL runs to directly evaluate this.

ModelVLM4DVideoEspressoNExT-QASTARVideo-MME-ShortMVBench
SFT with standard QA fine-tuning58.644.885.376.183.276.5
SFT with Gemini synthetic data59.343.785.477.283.276.4

 

Existing results: In addition to the results above, there is relevant controlled evidence already in the ablation results in Appendix E. Within each training-data setting, the full model and both reward ablations start from exactly the same Gemini synthetic warm-start. The ablations therefore retain the teacher-derived initialization while removing the specified RL reward component. Without answer rewards, removing sentence-level self-rewards reduces accuracy by 3.3–10.4 percentage points across the six benchmarks, while removing temporal-sensitivity rewards reduces accuracy by 1.2–4.4 points. With answer rewards, the corresponding reductions are 3.5–10.1 and 1.3–4.0 points. These differences arise under a fixed teacher-trained initialization and therefore cannot be explained by differences in teacher distillation between these configurations. These ablations directly support the contribution of the RL reward components conditional on the supervised warm start. We agree that they do not substitute for comparing competing RL objectives from the same checkpoint, which we do in the new RL runs we describe below.

 

New RL runs (in progress): We are currently performing the requested RL runs, in the settings with and without the ground-truth rewards, and will make a follow-up post here as soon as the results are available. For the sake of time, we are focusing on the strongest baselines (VL-Calibration and UMPIRE) and the benchmarks that seem to benefit most from the Gemini-based synthetic warm-start (VLM4D, NExT-QA, and STAR). We will also perform this analysis for the other baselines and benchmarks, but may run out of time before the end of the discussion period. When available, all results will be added to the manuscript.

Q2 and W2a: Clarification on compute and training configurations

There are two major assumptions underlying the proposed calculation that require clarification: 1) RL runs were not performed sequentially and 2) the batch size for RL was 16, not 64, which was reported as the group size G = 16 in the RL configuration section of Appendix A..

The reporting of 2 NVIDIA H100s in the appendix refers to the hardware requirements per training job, not the total number of GPUs used for all training runs. Training was conducted on a shared cluster containing 112 H100 GPUs, with many two-GPU jobs running concurrently (usually 16-24 jobs concurrently, depending on availability). With a batch size of 16, each step requires an average of 1 to 2 minutes. Thus, across the 65 total RL runs, the GPU hours was (2 GPUs x 1.5 x 5000 x 65)/60 = 16,250 hours. With an average of 20 jobs running concurrently, this amounts to only 12 days to perform all training runs for all the main results across all data settings and about 5 days for all the ablations. We expanded the RL configuration section and added an RL compute section in Appendix A in the updated manuscript to include these details.

 

Q3 and W2b: Clarification on VLM4D data reuse and reward stability

There are 500 unique prompts in the VLM4D RL training split, but again the batch size for RL was 16 (not 64) and includes the 16 different GRPO-style rollouts/responses for a randomly selected example from the RL data split. Thus, 5,000 RL steps is only 10 dataset passes (not 640) for VLM4D. Similar to above, we expanded Appendix A to include these details.

This clarification should directly address the reviewer’s objection regarding the possible VLM4D data reuse and associated reward stability concerns. Nonetheless, to directly inspect the reward stability of Video-SR1 on VLM4D, we include below the ratio of true to perceived accuracy during RL for this dataset.

RL stepWith ground-truth rewardsWithout ground-truth rewards
10001.030.93
20001.050.92
30000.980.93
40000.980.91
50000.960.90
Reviewer_3SDxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the limitation of GRPO for video reasoning: applying a single reward uniformly across an entire response fails to identify incorrect intermediate statements. It proposes Group and Sentence Relative Policy Optimization (GSRPO), which replaces response-level rewards with sentence-level rewards. The model assesses each sentence itself and applies this self-reward signal for localized credit assignment. The authors also introduce a temporal-sensitivity reward based on frame-order perturbations and optional sparse answer reward. Across six benchmarks, the full method improves performance both with and without ground-truth answer rewards.

Critical strengths
  1. The motivation is sound and valuable. Typical GRPO may omit localized errors in a rollout. Finer-grained rewards target a real and important problem.
  2. The overall design aligns well with the motivation. The method uses sentence-level self-rewards to address localized errors, and its components are designed to complement one another in serving this goal.
  3. The experiments are extensive and reasonably comprehensive, evaluating the effectiveness of the components across multiple settings. Diagnostic analyses also help explain the sources of improvement, providing substantial empirical support.
Critical weaknesses
  1. GSRPO depends on reliable sentence matching across rollouts. However, grouping based on sentence embeddings matching does not establish that the matched reasoning steps are comparable, calling into question the validity of the resulting reward comparisons and advantage estimates.
  2. The warm start uses deliberately injected localized corruptions, but whether these match the blind spots in actual model responses remains insufficiently validated.
  3. The method samples at most 15 frames per video and does not evaluates on benchmarks featuring longer videos, raising concerns about its applicability to longer-video scenarios.
  4. The necessity of sentence-level updates has not been independently established. The reported results concern RL performance after the warm start, without a direct comparison against updates using response-level rewards under otherwise identical conditions.
Questions for authors
  1. Could the authors provide a detailed analysis or illustrative examples to justify the sentence segmentation strategy?
  2. Could the authors report and analyze results without the warm start to clarify its necessity?
  3. Could the authors evaluate the method on benchmarks featuring longer videos?
4 comments

We thank the reviewer for recognizing the value of GSRPO's motivation, the complementary design of its components, and the extensive experiments and diagnostic analyses. We address the concerns and questions below.

W1 and Q1: Sentence alignment and segmentation

We agree that embedding similarity alone does not guarantee that matched sentences represent comparable reasoning steps. In GSRPO, semantic alignment provides an approximate semantic correspondence between reasoning steps across rollouts that may differ in length, ordering, and structure. To directly assess the reviewer’s concern, we conduct an additional manual analysis of sentence alignment quality.

Additional alignment analysis: To directly assess whether embedding-based matching produces meaningful comparison groups, we manually audited 200 randomly sampled matched sentence pairs. We classified each pair into three categories: (1) semantically comparable, where both sentences address the same underlying observation or inference, including contradictory claims; (2) related but not directly comparable, where sentences share contextual or semantic overlap but address different observations or reasoning steps; and (3) mismatched, where sentences concern unrelated observations or reasoning steps.

Table: Manual assessment of sentence alignment quality for 200 randomly sampled matched sentence pairs.

Alignment categoryCountPercentage
Semantically comparable16482.0%
Related but not directly comparable3015.0%
Mismatched63.0%

The results show that 82% of matched pairs are appropriately comparable, while 15% are related but not directly comparable and only 3% are mismatched. These results suggest that the semantic alignment procedure generally identifies meaningful comparison groups, rather than simply matching sentences with superficial linguistic similarity. Importantly, the first category includes contradictory observations, which are especially useful for distinguishing correct and incorrect intermediate reasoning. We acknowledge that the mismatched 3% introduce potential noise into the group-relative advantage estimates, but the predominance of semantically comparable matches supports the use of sentence-level alignment. We add the analysis and illustrative examples to Appendix G in the updated manuscript.

W2: Synthetic corruptions versus natural reasoning errors (analysis in progress)

The synthetic warm-start includes temporal reversals, entity swaps, incorrect counts, hallucinated objects/actions, incorrect causal relations, and unsupported conclusions (Section 2.2). These overlap with the broad perceptual and reasoning failures identified in Figure 6, where Video-SR1 reduces error rates. However, we agree this does not directly measure how closely the synthetic errors match natural model mistakes. We are actively performing a finer-grained audit of model-generated rollouts, using the same taxonomy as the synthetic data, to quantify this overlap and identify underrepresented errors. We will post the results here when available and add them to the manuscript.

W3 and Q3: Longer-video evaluation (analysis in progress)

Yes, we are currently running new evaluations on Video-MME-Long (with video lengths ranging from 30 to 60 minutes) to determine how well Video-SR1 transfers to longer videos. We will post the results here when available and add them to the manuscript.

W4: Necessity of sentence-level updates (analysis in progress)

We agree that directly comparing sentence-level versus response-level updates while holding the reward signals fixed would provide a useful additional evaluation. Our existing ablations in Appendix E already establish the importance of sentence-level self-rewards: removing them reduces accuracy by 3.3-10.4 percentage points without ground-truth answer rewards and 3.5-10.1 points with answer rewards, with all configurations initialized from the same Gemini synthetic warm-start. To further investigate the benefits of the sentence-level granularity, we will evaluate an additional ablation that aggregates the same sentence-level self-rewards into a single response-level reward, while keeping the warm-start, temporal-sensitivity reward, and other training configurations unchanged. When the results are available, we will make a follow-up post and update the manuscript.

Q2: Necessity of the synthetic warm-start (analysis in progress)

Yes, we are actively performing new RL runs starting from standard QA SFT without the Gemini synthetic self-assessment warm-start, keeping RL data, training budget, and reward components fixed. This will test whether the warm-start is necessary for reliable sentence-level self-rewarding, rather than merely improving initial answer accuracy. We will report these results when available.

We have completed the comparison between synthetic versus real/natural error types (W2), the long-video evaluation (W3 & Q3), and the ablations for the sentence-level self-reward (W4) and synthetic warm-start (Q2). In the updated manuscript, we add the additional error audit in Appendix I, we add the long-video evaluation in Appendix J, and we add the new ablations to the existing ablations in Appendix E.

Results for W2: Comparing synthetic corruptions versus natural reasoning errors

We randomly sampled 300 model-generated rollouts (50 per benchmark setting) containing at least one sentence flagged as incorrect by the self-rewards. We manually categorized the first flagged sentence in each rollout using the synthetic-data error categories, and compared the distribution with our existing audit of 1,000 synthetic examples (Section 2.2). The distribution of the error types across the synthetic versus real data are shown in the table below.

The six targeted corruption types cover approximately 92% of human-confirmed errors. Their frequencies differ: action errors are underrepresented in the synthetic data (20% vs. 29%), while entity and causal errors are overrepresented. These results suggest that the synthetic data achieves broad coverage of the observed error types, but not an exact distributional match.

Error typeNatural reasoning errors in real rolloutsSynthetic reasoning errors in generated data
Temporal order19%15%
Entity identity6%16%
Count7%11%
Action29%20%
Causal relation6%19%
Unsupported visual detail21%16%
Other8%3%
Falsely identified as an error4%—

 

Results for W3 and Q3: Longer-video evaluation

The table below shows the results for Video-MME-Long (with subtitles) for the settings without and with the ground-truth rewards. All methods perform significantly worse on Video-MME-Long relative to Video-MME-Short. However, Video-SR1 achieves the highest accuracy in settings without and with the ground-truth rewards, exceeding the strongest evaluated baseline by 0.6 percentage points in each. Although the margin between Video-SR1 and baselines is smaller compared to the short video subset, these results demonstrate that the proposed method can extend to datasets with longer videos.

ModelWithout ground-truth rewardsWith ground-truth rewards
GRPO—74.2
Vision-SR1—74.6
Video-R1—74.4
UMPIRE74.174.9
VL-Calibration73.675.1
Video-SR174.775.7

 

Results for W4 and Q2: Ablations for sentence-level updates (W4) and synthetic warm-start (Q2)

We add two ablations: 1) we average the same sentence-level self-rewards into a single scalar per response and use this scalar for sparse response-level updates, 2) we replace the Gemini synthetic warm-start with standard QA SFT.

Ablating sentence-level rewards (addressing W4): Compared with averaging the same self-rewards at response level, sentence-level updates improve all six benchmarks by 2.0-5.2 points without ground-truth rewards and 1.9-4.9 points with ground-truth rewards. This directly supports the benefit of localized credit assignment: the same underlying sentence assessments are less effective when collapsed into a uniform response-level reward.

Ablating the synthetic warm-start (addressing Q2): Replacing the synthetic warm-start with standard QA SFT lowers accuracy on every benchmark: 1.6–4.0 points without the ground-truth rewards and 1.6–4.3 points with the ground-truth rewards. Standard QA initialization also required extensive prompt tuning to produce the structured self-reward outputs. These results strongly support the warm-start's benefit for self-rewarding RL, without establishing it is strictly necessary.

Without ground-truth answer rewards

ConfigurationVLM4DVideoEspressoNExT-QASTARVideo-MME-ShortMVBench
No sentence-level self-rewards56.743.184.674.681.174.8
No temporal-sensitivity reward62.845.786.279.482.477.5
Sparse self-rewards (new)61.945.585.578.683.177.2
No synthetic warm-start (new)63.146.485.779.084.477.9
Video-SR167.149.587.980.686.880.4

With ground-truth answer rewards

ConfigurationVLM4DVideoEspressoNExT-QASTARVideo-MME-ShortMVBench
No sentence-level self-rewards60.745.385.776.483.576.9
No temporal-sensitivity reward66.849.487.980.384.278.9
Sparse self-rewards (new)65.948.687.379.584.078.7
No synthetic warm-start (new)66.549.087.679.384.479.2
Video-SR170.852.189.282.287.281.5

Before the discussion period ends, we wanted to follow up to check if you’ve had a chance to review our response. We have added several new analyses and experiments (outlined above and added to the manuscript) that we believe directly address your questions and concerns. Please let us know if there is anything further we can do.

Thank you for the response and clarifications. I maintain my positive assessment of the paper and my recommendation for acceptance.

Reviewer_xELxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

VIDEO-SR1 introduces a self-rewarding RL framework for video-language reasoning. Its core contribution is Group and Sentence Relative Policy Optimization (GSRPO). GSRPO replaces sparse outcome supervision with dense, critic-free credit assignment. It aligns semantically similar sentences across rollouts and combines policy-generated sentence-level self-rewards with a counterfactual frame-shuffling temporal reward. Preceded by a synthetic warm-start, VIDEO-SR1 is validated across six diverse video benchmarks. It significantly improves accuracy, sample efficiency, and cross-domain transfer over strong baselines, both with and without ground-truth answer rewards.

Critical strengths
  1. Video reasoning suffers from localized perceptual and temporal errors that sparse outcome rewards cannot disentangle. GSRPO cleverly resolves this without a memory-heavy value critic. By semantically aligning sentences across rollouts for local normalization, it achieves fine-grained credit assignment. This directly addresses a core bottleneck in multimodal RL and strongly supports acceptance.
  2. The empirical design is exceptionally thorough. Evaluating both with and without ground-truth rewards proves the intrinsic value of the self-derived signals. Furthermore, the diagnostic analyses—tracking calibration stability, "right answer, wrong reasoning" rates, and frame-shuffle sensitivity—convincingly substantiate why the method works. The clear synergy between local self-rewards and temporal sensitivity solidifies the paper's empirical contribution.
Critical weaknesses
  1. VIDEO-SR1’s warm-start uses 138,000 synthetic rationales and evaluations generated by a superior teacher model (Gemini-3.1-Pro). However, baselines only receive standard QA fine-tuning on source data. This creates a severe confounding factor. It is unclear how much of the reported gain stems from the GSRPO algorithm versus distilling reasoning from a stronger proprietary foundation model.
Questions for authors
  1. To isolate GSRPO's algorithmic contribution from teacher distillation, how do baselines (like GRPO and Video-R1) perform when initialized from the exact same Gemini-3.1-Pro warm-start checkpoint?
4 comments

We thank the reviewer for recognizing the value of our proposed approach to sentence-level credit assignment and the thoroughness of our diagnostic analyses.

Matched-checkpoint comparisons: We are currently performing the requested RL training runs, with all methods initialized from the same Video-SR1 SFT checkpoint, to evaluate the possible confounding effects of the Gemini synthetic warm-start. When the results are available, we will make a follow-up post and update the manuscript. In the meantime, we include below some existing results that shed light on this question.

VideoEspresso results: Importantly, VideoEspresso already includes teacher distillation, as the standard training split contains strong chain-of-thought reasoning traces before the final answer. This data is included in the standard QA fine-tuning. We continue to see Video-SR1 out-perform baselines in this setting, which uses strong pre-RL teacher distillation for all methods. However, the VideoEspresso data still does not incorporate the self-reward reflections that are included in the Gemini synthetic data. Therefore, this setting does not perfectly control for the possible confounding effects and, as a result, the ongoing RL runs described above are still needed.

MethodWithout ground-truth answer rewardsWith ground-truth answer rewards
GRPO—46.5
Vision-SR1—45.8
Video-R1—47.6
UMPIRE45.948.8
VL-Calibration46.448.6
Video-SR149.552.1

Pre-RL SFT results after standard QA warm-start versus Gemini synthetic warm-start: We initially evaluated the possible gains from the Gemini synthetic warm-start by testing the different pre-RL SFT checkpoints on each benchmark. As shown in the table below (which is added to the paper in Appendix F), the Gemini synthetic warm-start achieves improved performance on only 3 of the 6 benchmarks and shows worse performance on 2 benchmarks, especially VideoEspresso. These mixed results led to our decision of using the standard QA fine-tuning for the baselines. However, we agree that, despite the lack of improved pre-RL test accuracy, the Gemini synthetic warm-start may still provide a stronger initialization for RL and, as described above, we are actively performing new RL runs to directly evaluate this.

ModelVLM4DVideoEspressoNExT-QASTARVideo-MME-ShortMVBench
SFT with standard QA fine-tuning58.644.885.376.183.276.5
SFT with Gemini synthetic data59.343.785.477.283.276.4

Ablation results: In addition to the results above, there is relevant controlled evidence already in the ablation results in Appendix E. Within each training-data setting, the full model and both reward ablations start from exactly the same Gemini synthetic warm-start. The ablations therefore retain the teacher-derived initialization while removing the specified RL reward component. Without answer rewards, removing sentence-level self-rewards reduces accuracy by 3.3–10.4 percentage points across the six benchmarks, while removing temporal-sensitivity rewards reduces accuracy by 1.2–4.4 points. With answer rewards, the corresponding reductions are 3.5–10.1 and 1.3–4.0 points. These differences arise under a fixed teacher-trained initialization and therefore cannot be explained by differences in teacher distillation between these configurations. These ablations directly support the contribution of the RL reward components conditional on the SFT warm-start. However, again we agree that they do not substitute for comparing competing RL objectives from the same checkpoint.

Without ground-truth answer rewards

ConfigurationVLM4DVideoEspressoNExT-QASTARVideo-MME-ShortMVBench
Without sentence-level self-rewards56.743.184.674.681.174.8
Without temporal-sensitivity reward62.845.786.279.482.477.5
Video-SR167.149.587.980.686.880.4

With ground-truth answer rewards

ConfigurationVLM4DVideoEspressoNExT-QASTARVideo-MME-ShortMVBench
Without sentence-level self-rewards60.745.385.776.483.576.9
Without temporal-sensitivity reward66.849.487.980.384.278.9
Video-SR170.852.189.282.287.281.5

Results for the requested RL runs that start from the same Gemini synthetic warm-start checkpoint for all methods

Below, we provide the results for the RL runs requested by the reviewer. The figures with the RL curves are added in Appendix H of the updated manuscript. We are also performing checkpoint-controlled RL runs that exclude the ground-truth reward, but these are still in progress. We will make a follow-up post as soon as they are available. We add the new results in the appendix for now, but are happy to replace the main results with these results if preferred.

Overall, we see that the Gemini synthetic warm-start significantly improves the post-RL performance of VL-Calibration, Vision-SR1, and Video-R1, shows mixed results with GRPO, and decreases the performance of UMPIRE. Notably, for all methods, we don’t see significant improvements with the Gemini warm-start with VideoEspresso, which already contains strong chain-of-thought reasoning inside the original QA fine-tuning. Excluding VideoEspresso, we observe an average delta of +.56, +1.4, and +.42 for VL-Calibration, Vision-SR1, and Video-R1, respectively, with the Gemini warm-start, while observing an average delta of -.8 for UMPIRE.

Video-SR1 outperforms the strongest baseline, VL-Calibration, on every benchmark, by 4.1, 3.7, 2.0, 2.3, 2.5, and 2.6 percentage points, respectively. These results should directly address the reviewer's concerns regarding the confounding effects of the Gemini synthetic warm-start, as these performance gains by Video-SR1 cannot be attributed to unequal teacher-derived initialization.

ModelVLM4DVideoEspressoNExT-QASTARVideo-MME-ShortMVBench
Base model59.343.785.477.283.276.4
GRPO60.844.486.077.383.377.3
Vision-SR163.045.786.678.883.977.7
Video-R163.244.086.278.784.077.8
UMPIRE63.846.686.378.883.977.6
VL-Calibration66.748.487.279.984.778.9
Video-SR170.852.189.282.287.281.5

We have now also completed the updated checkpoint-controlled RL runs for the setting without ground-truth rewards (where all methods start from the same Gemini synthetic warm-start). Similar to the results described above, we see that the Gemini synthetic warm-start substantially improves the post-RL performance of VL-Calibration, with consistent increases in accuracy across all benchmarks (increase of +1.1 absolute points on average) relative to the warm-start based on standard QA fine-tuning. UMPIRE again shows mixed results, with post-RL accuracy increasing on 3 benchmarks (NExT-QA, Video-MME-Short, MVBench) and decreasing on the other 3 benchmarks (VLM4D, VideoEspresso, STAR). Video-SR1 consistently improves performance, relative to the strongest baselines for each benchmark, across all settings (i.e., without vs with ground-truth rewards and Gemini synthetic vs standard QA SFT warm-start). Overall, these results confirm that Video-SR1’s performance gains are not the result of unequal teacher distillation during SFT. We add these results to Appendix H, but again we are happy to replace the main results with these if preferred.

ModelVLM4DVideoEspressoNExT-QASTARVideo-MME-ShortMVBench
Base model59.343.785.477.283.276.4
UMPIRE60.945.185.677.983.477.8
VL-Calibration62.946.986.178.584.978.5
Video-SR167.149.587.980.686.880.4

We just wanted to follow up to see if you had a chance to review our response. We made a significant effort to re-run all baselines starting from the same Gemini synthetic warm-start. We believe this fully addresses the issue you describe in the weaknesses and questions section of your review. Please let us know if there is anything further we can do before the discussion period ends.

Comments on the paper

6 comments
  1. BS vs G: Our response above uses a response-count convention for batch size. It does not conflate batch size with group size. The apparent disagreement stems from two different conventions for defining "batch size" (BS). We refer to batch size as the total number of generated responses used for policy optimization, i.e., batch size is (number of examples) x G, where G denotes the number of responses sampled per prompt. This convention is also used in existing GRPO literature [1-3]. As described in Appendix A (and in our response above), each batch includes the 16 different GRPO-style rollouts for a randomly selected example from the RL data split. Therefore, for our setting, BS=G=16. We acknowledge that defining BS as the number of distinct prompts is equally valid. Therefore, to avoid further ambiguity, we will update the terminology in our appendix to follow the reviewer's convention, explicitly distinguishing prompt batch size, group size G, and the total number of generated responses.

 

  1. Quality of the synthetic data: The purpose of the Gemini synthetic warm-start was to improve self-assessment capabilities that are leveraged during RL, not to directly improve task accuracy before RL. Therefore, pre-RL test accuracy on these two cross-domain transfer benchmarks does not directly measure this self-assessment capability or establish the quality of the synthetic annotations. More direct evidence about annotation quality comes from the manual review reported in Appendix C: 1,000 randomly sampled synthetic examples, with 96% audit accuracy, checking sentence labels against the video and the validity of the injected errors. This audit supports the quality of the generated supervision. Ultimately, the rationale for the generation strategy is to supply the positive and negative sentence-level supervision needed for self-assessment. Therefore, its downstream benefit must be evaluated through controlled RL experiments, rather than inferred from the pre-RL SFT answer-accuracy.

 

  1. Requested RL runs (in progress): As described in our previous response, we agree similar pre-RL SFT accuracy does not establish equivalent initialization for RL and are actively performing the requested RL runs. We will make a follow-up post as soon as those results are available.

 

References

  1. Hou, Z., et al. (2026). Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning. arXiv:2607.07508.
  2. Wang, H., et al. (2025). Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding. NeurIPS 2025.
  3. Li, Z., et al. (2026). When Do Larger Batches Help Scale LLM Reinforcement Learning? arXiv:2608.29296.

Results for the requested RL runs that start from the same Gemini synthetic warm-start checkpoint for all methods

Below, we provide the results for the RL runs requested by the reviewer. The figures with the RL curves are added in Appendix H of the updated manuscript. The RL runs that exclude the ground-truth reward are still in progress and we will make a follow-up post as soon as they are available. We add the new results in the appendix for now, but are happy to replace the main results with these results if preferred.

Overall, we see that the Gemini synthetic warm-start significantly improves the post-RL performance of VL-Calibration, Vision-SR1, and Video-R1, shows mixed results with GRPO, and decreases the performance of UMPIRE. Notably, for all methods, we don’t see significant improvements with the Gemini warm-start with VideoEspresso, which already contains strong chain-of-thought reasoning inside the original QA fine-tuning. Excluding VideoEspresso, we observe an average delta of +.56, +1.4, and +.42 for VL-Calibration, Vision-SR1, and Video-R1, respectively, with the Gemini warm-start, while observing an average delta of -.8 for UMPIRE.

ModelVLM4DVideoEspressoNExT-QASTARVideo-MME-ShortMVBench
Base model59.343.785.477.283.276.4
GRPO60.844.486.077.383.377.3
Vision-SR163.045.786.678.883.977.7
Video-R163.244.086.278.784.077.8
UMPIRE63.846.686.378.883.977.6
VL-Calibration66.748.487.279.984.778.9
Video-SR170.852.189.282.287.281.5

 

Video-SR1 outperforms the strongest baseline, VL-Calibration, on every benchmark, by 4.1, 3.7, 2.0, 2.3, 2.5, and 2.6 percentage points, respectively. These margins cannot be attributed to unequal teacher-derived initialization. In addition, with consistently improved post-RL performance for 3 of the 5 baselines, this supports downstream utility of the Gemini synthetic warm-start beyond the pre-RL answer accuracy. However, we still see that its effectiveness depends on the RL objective.

For UMPIRE, a possible explanation for the decline is a change in how its uncertainty score relates to correctness. UMPIRE is an uncertainty estimator that combines response-embedding diversity and model probabilities, and explicitly excludes consistently wrong responses from its target error model. Gemini SFT could make the policy more confident or consistent on some incorrect answers even while improving average accuracy. Such errors could then receive overly favorable intrinsic rewards, weakening the learning signal. However, this is a proposed explanation, not an established cause.

We have now also completed the updated checkpoint-controlled RL runs for the setting without ground-truth rewards (where all methods start from the same Gemini synthetic warm-start). Similar to the results described above, we see that the Gemini synthetic warm-start substantially improves the post-RL performance of VL-Calibration, with consistent increases in accuracy across all benchmarks (increase of +1.1 absolute points on average) relative to the warm-start based on standard QA fine-tuning. UMPIRE again shows mixed results, with post-RL accuracy increasing on 3 benchmarks (NExT-QA, Video-MME-Short, MVBench) and decreasing on the other 3 benchmarks (VLM4D, VideoEspresso, STAR). Video-SR1 consistently improves performance, relative to the strongest baselines for each benchmark, across all settings (i.e., without vs with ground-truth rewards and Gemini synthetic vs standard QA SFT warm-start). Overall, these results confirm that Video-SR1’s performance gains are not the result of unequal teacher distillation during SFT. We add these results to Appendix H, but again we are happy to replace the main results with these if preferred.

ModelVLM4DVideoEspressoNExT-QASTARVideo-MME-ShortMVBench
Base model59.343.785.477.283.276.4
UMPIRE60.945.185.677.983.477.8
VL-Calibration62.946.986.178.584.978.5
Video-SR167.149.587.980.686.880.4

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.