acceptodds
Under review as a conference paper at ICLR 2027

Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning

Abstract

Recent advances in multimodal reasoning, particularly Thinking with Images, have achieved remarkable progress by empowering Multimodal Large Language Models (MLLMs) to actively inspect and reason over visual context. Extending this visual reasoning paradigm to dynamic videos requires models to actively localize and synthesize visual evidence across the temporal dimension. However, enabling MLLMs to think with videos faces two fundamental challenges: (i) from the data perspective, existing training data lacks tasks that guide models to actively seek and verify evidence across temporal intervals; and (ii) from the training supervision perspective, current methods lack verified temporal process supervision, leading to unfaithful reasoning and superficial shortcuts. To address these challenges, we propose Video-Thinker to unify data construction and training supervision, enabling intrinsic temporal reasoning without relying on external tools. From the data perspective, to formally capture both temporal and semantic dependencies across events, we construct a video reasoning graph to model their temporal ordering and semantic relations. Leveraging this graph, we synthesize reasoning tasks with guaranteed evidence necessity, ensuring that correct answers strictly require observing and integrating all designated temporal clues. From the training supervision perspective, following a cold-start Supervised Fine-Tuning (SFT) phase, we introduce an evidence-aware Reinforcement Learning (RL) method that provides verified temporal process supervision to ensure faithful grounding throughout reasoning. Extensive experiments across six challenging video reasoning benchmarks demonstrate that Video-Thinker achieves an average improvement of +6.86% over the base model with remarkable data efficiency, requiring only 20K training instances.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video reasoning, Multimodal large language model, Thinking with videos

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZHYfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Video-Thinker post-trains Qwen3-VL-8B-Instruct for temporal video reasoning on 20K synthesized instances. Each video becomes a graph whose temporal edges follow Allen's interval relations (Eq 1) and whose entailment edges come from one Gemini-3.1-Flash call per video. Each question uses a pairwise non-entailing evidence set A, with answer y = R_Q(A) and distractors d_i = R_Q(A ∖ {s_i}) (Eq 2), so each segment should be necessary. After an SFT cold start on <time>/<caption> trajectories, GRPO adds a multi-span tIoU evidence score to the advantage, A_i = z(a_i) + λ(e_i − ē), with λ ≤ 1/√(G−1) (Eq 6-8). At 16 frames the model averages 50.15 on six benchmarks against 43.29 for its base (Table 1); Table 2 ablates the training stages and Table 3 probes grounding and captioning.

Critical strengths
  1. The data construction attacks shortcut answering at its source. Slot-omission distractors d_i = R_Q(A ∖ {s_i}) over a pairwise non-entailing evidence set (Eq 2) make every selected segment necessary by construction, given correct entailment labels, and blind-attack filtering drops items answerable without the video. Temporal edges come deterministically from Allen's 13 interval relations (Eq 1), so that part of the graph involves no LLM. I found no prior video-reasoning training set built on a formal necessity construction, and the pipeline is specified well enough to reproduce (Alg 2; App G.2).

  2. The process term is measured in isolation on the same backbone. The evidence score enters the advantage as a mean-centered term, bounded by λ ≤ 1/√(G−1) so that correct trajectories keep a positive advantage (Eq 7-8; App I). Against outcome-only RL on the identical backbone, it lifts the six-benchmark average from 47.91 to 50.15, a +2.24 gain (Table 2). That is the paper's most direct causal evidence, and I credit the authors for also reporting VideoTT, where the process term does not help.

  3. The headline gain is measured on benchmarks outside the training corpora. None of the six Table 1 benchmarks is among the four training corpora of Table 4, and the base and Video-Thinker share the backbone and the 16-frame protocol of Sec 4.1 (L371-373). The +6.86 average gain (50.15 vs 43.29, Table 1) is therefore a held-out, same-backbone result, and the backbone issue in W1 does not affect it.

Critical weaknesses
  1. The "state-of-the-art" claim (Sec 4.2) cannot be separated from an 8B-versus-7B backbone gap. Every post-trained baseline except VideoPro-GRPO is 7B, and the Qwen3-VL-8B-Instruct base, before any reasoning post-training, already averages 43.29, above 14 of the 17 post-trained baselines (Table 1). With no run of the recipe on a 7B backbone and no baseline re-run on the 8B one, only the same-backbone gain can be credited to the method. Fix: train the identical SFT and process-RL recipe on a 7B backbone such as Qwen2.5-VL-7B, or re-run Video-R1 or VideoChat-R1 on Qwen3-VL-8B; both fit the paper's own setup.

  2. Two of the three sources behind the grounding table also supply training videos, and no held-out split is stated. Table 3 evaluates on 1K samples from ActivityNet, YouCook2, and TutorialVQA, yet ActivityNet supplies 17,103 of the 22,948 training-source videos and YouCook2 is also a training source (Table 4). I searched for "held-out", "disjoint", "split", "de-dup", and "exclude" and found nothing that applies to Table 3, so its gains may reflect train/test overlap rather than the intrinsic grounding Sec 4.3 claims to isolate. Fix: report the de-duplication counts and the TutorialVQA-only scores, or re-evaluate on disjoint videos; neither needs training.

  3. The "guaranteed evidence necessity" rests on one unvalidated LLM pass. Entailment edges for all 22,948 source videos (Table 4) come from one Gemini-3.1-Flash call per video (Sec 3.1; App G.2), and I found no accuracy check of these labels. By the paper's own argument (Sec 3.2), a missed edge leaves a superfluous segment in A; by Eq 3, a spurious one adds wrong intervals to the certified spans that Eq 6 scores against. Neither Sec 3.2 safeguard, the d_i ≠ y check or blind-attack filtering, audits these labels. Fix: report human-audited edge precision and recall on sampled videos; this needs annotation, not training.

Questions for authors
  1. Are the Table 3 ActivityNet and YouCook2 test videos disjoint from the MDVC, VideoXum, and YouCook2 training videos? Please give the de-duplication procedure and held-out counts; this answer is the most likely to change my score.
  2. Table 2 varies only training stages; Sec 4.2 credits the necessity construction only across models (L415-416, L427-428). What does SFT + outcome RL (47.91) reach with 20K flat-caption items (L74) of the same segments?
  3. Table 2 has no variant that adds the same evidence score e to the reward instead of the advantage. How does it compare, given that Sec 3.3 (L283) says reward-level grounding alters the optimal policy?
  4. At 16 frames the base scores 54.89 on VideoMME (Table 1), while public aggregators list roughly 71% for this backbone. What do the base and Video-Thinker score with a larger frame budget, and does the ranking hold?
  5. VRBench contributes 3.64 of the +6.86 average gain (21.84/6), yet the base (59.40) trails 7B Video-R1 (69.25). Which App D phase does App F's top-1 accuracy measure, and how often does the base fail the <answer> parse?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ZHYf (Part 1/3)

We thank Reviewer ZHYf for the rigorous, detail-oriented review and constructive suggestions. Below we address W1–W2 in Part 1, W3 and Q1–Q2 in Part 2, and Q3–Q5 in Part 3.


W1. The "state-of-the-art" claim (Sec 4.2) cannot be separated from an 8B-versus-7B backbone gap. Every post-trained baseline except VideoPro-GRPO is 7B, and the Qwen3-VL-8B-Instruct base, before any reasoning post-training, already averages 43.29, above 14 of the 17 post-trained baselines (Table 1). With no run of the recipe on a 7B backbone and no baseline re-run on the 8B one, only the same-backbone gain can be credited to the method. Fix: train the identical SFT and process-RL recipe on a 7B backbone such as Qwen2.5-VL-7B, or re-run Video-R1 or VideoChat-R1 on Qwen3-VL-8B; both fit the paper's own setup.

We agree that cross-model comparisons in Table 1 reflect end-to-end system performance across pre-packaged 7B and 8B checkpoints, and we have refined Section 4.2 to credit our method specifically with its controlled, same-backbone gains on Qwen3-VL-8B-Instruct. Starting from an 8B model does not automatically yield post-training gains: on Qwen3-VL-8B-Instruct, single-stage 10K + SFT at 41.18% and 10K + RL at 40.81% in Table 2 both drop below the untaught 8B base at 43.29%, as does the external 8B baseline VideoPro-GRPO at 33.46% in Table 1. On the identical Qwen3-VL-8B-Instruct backbone, our two-stage 20K recipe delivers a +6.86 pp six-benchmark average gain over the base model from 43.29% to 50.15%, improving every benchmark by +2.48 pp to +21.84 pp, alongside a +20.29 pp grounding mIoU gain from 32.45% to 52.74% in Table 3 and +7.38 pp to +24.41 pp gains over same-backbone Qwen3-VL-8B-Instruct tool-invocation pipelines in Appendix J. Under matched 20K data on the same Qwen3-VL-8B-Instruct backbone, we also compare directly against outcome-only GRPO at 47.91% six-benchmark average and 45.28% on Video-Holmes, as well as scalar temporal-IoU reward shaping at 45.84% on Video-Holmes in Table 10, where our decoupled advantage injection reaches 47.03%, a +1.75 pp gain over Outcome RL that is 3.1×3.1\times larger than scalar reward shaping.


W2. Two of the three sources behind the grounding table also supply training videos, and no held-out split is stated. Table 3 evaluates on 1K samples from ActivityNet, YouCook2, and TutorialVQA, yet ActivityNet supplies 17,103 of the 22,948 training-source videos and YouCook2 is also a training source (Table 4). I searched for "held-out", "disjoint", "split", "de-dup", and "exclude" and found nothing that applies to Table 3, so its gains may reflect train/test overlap rather than the intrinsic grounding Sec 4.3 claims to isolate. Fix: report the de-duplication counts and the TutorialVQA-only scores, or re-evaluate on disjoint videos; neither needs training.

Following your suggestion, we audited canonical video IDs across all training and evaluation sources, with the full breakdown in Q1 below, and re-evaluated Table 3 on both the strictly disjoint subset of 830 samples across 685 videos with zero overlap against our 20K post-training set and the out-of-source TutorialVQA-only subset of 76 samples with zero overlap, as reported in Section 4.3, Appendix F, and Table 6:

Evaluation SubsetModelmIoU[email protected][email protected]Grounding Avg.METEORROUGE-LBLEU@1Captioning Avg.
Original Full Set (N=1,000N = 1{,}000)Qwen3-VL-8B-Instruct32.4545.8229.1435.8015.1816.7512.3614.76
Original Full Set (N=1,000N = 1{,}000)Video-Thinker (Ours)52.7484.6258.3565.2417.8522.9417.9119.57
Strictly Disjoint (N=830N = 830, 685 videos)Qwen3-VL-8B-Instruct32.5545.6629.2835.8315.2316.6812.4314.78
Strictly Disjoint (N=830N = 830, 685 videos)Video-Thinker (Ours)52.2383.9857.7164.6417.7122.7617.7919.42

Excluding all 170 overlapping samples across 143 videos changes Video-Thinker's Grounding Average by only -0.60 pp from 65.24% to 64.64% and Captioning Average by -0.15 pp from 19.57% to 19.42%, preserving a +28.81 pp grounding gain and +4.64 pp captioning gain over the base model on strictly unseen videos; on out-of-source TutorialVQA with zero overlap, Video-Thinker similarly improves Grounding Average by +29.36 pp from 35.17% to 64.53% and Captioning Average by +4.79 pp from 14.52% to 19.31%.

AuthorsOpenReview ↗

Response to Reviewer ZHYf (Part 2/3)

W3. The "guaranteed evidence necessity" rests on one unvalidated LLM pass. Entailment edges for all 22,948 source videos (Table 4) come from one Gemini-3.1-Flash call per video (Sec 3.1; App G.2), and I found no accuracy check of these labels. By the paper's own argument (Sec 3.2), a missed edge leaves a superfluous segment in A; by Eq 3, a spurious one adds wrong intervals to the certified spans that Eq 6 scores against. Neither Sec 3.2 safeguard, the d_i ≠ y check or blind-attack filtering, audits these labels. Fix: report human-audited edge precision and recall on sampled videos; this needs annotation, not training.

Two annotators audited Gemini-3.1-Flash entailment labels across 100 stratified videos matching Table 4's proportions of 40 MDVC, 35 VideoXum, 17 VidSTG, and 8 YouCook2 videos, covering 486 segments and 1,423 undirected or 2,846 directed pairs. Applying the atomic checklist in Appendix G.2 with κ=0.91\kappa = 0.91 identified 199 directed edges and 2,647 non-entailing pairs in Table 9:

ScopeVideos (Segs)PairsTPFPFNTNPrecisionRecallF1Acc.
Stratified 4-Source100 (486)2,84617214272,63392.47%86.43%89.35%98.56%

Because our checklist enforces single-step verification without transitive closure on predicted edges, the 14 false positives out of 2,846 pairs, or 0.49% of pairs with FPR=0.53%\text{FPR} = 0.53\%, keep direct expansion of the certified interval union in Eq. 3 minimal. If any of the 27 false negatives, or 0.95% of pairs, admits two related segments into the candidate set, the generator abstains or fails the slot-omission check in Eq. 2 when omitting one slot leaves the answer recoverable, as supported by Table 5 where masking any slot's interval union reduces accuracy from 58.22% to 31.23% across visible slots and 29.55% on clean slots.


Q1. Are the Table 3 ActivityNet and YouCook2 test videos disjoint from the MDVC, VideoXum, and YouCook2 training videos? Please give the de-duplication procedure and held-out counts; this answer is the most likely to change my score.

They are 83.00% disjoint at the sample level, covering 830 of 1,000 samples across 685 of 828 videos. Normalizing all IDs to 11-character YouTube IDs by stripping v_ so ActivityNet and YouCook2 share one ID space with 0 overlap, alongside VidOR IDs, shows that Table 4's 22,948 entries comprise 9,123 MDVC and 7,980 VideoXum entries sharing 6,214 ActivityNet IDs, or 10,889 unique ActivityNet videos; with 4,055 VidSTG and 1,790 YouCook2 videos from 1,333 train and 457 val clips, the pool has 16,734 unique videos, and our 20K post-training set covers 11,438 unique videos, with 10K SFT from 6,842 and 10K RL from 6,915, sharing 2,319. Matching Table 3's 828 videos and 1,000 samples against all 11,438 post-training videos yields:

SourceSamplesVideos∩\cap SFT∩\cap RL∩\cap (SFT ∪\cup RL)Disjoint Videos (Samples)
ActivityNet (val)6625687275119 (137)449 (525, 79.31%)
YouCook2 (val)262184151424 (33)160 (229, 87.40%)
TutorialVQA7676000 (0)76 (76, 100.00%)
Total (Table 3)1,0008288789143 (170)685 (830, 83.00%)

Because VideoXum uses a custom split and cross-source ID filtering was not enforced before 20K sampling, 119 ActivityNet videos with 137 samples and 24 YouCook2 videos with 33 samples overlapped with post-training. As shown in W2, re-evaluating on the 685 disjoint videos with 830 samples and on the 76 TutorialVQA videos yields virtually identical gains of +28.81 pp and +29.36 pp in Grounding Avg.


Q2. Table 2 varies only training stages; Sec 4.2 credits the necessity construction only across models (L415-416, L427-428). What does SFT + outcome RL (47.91) reach with 20K flat-caption items (L74) of the same segments?

Re-synthesizing 20K flat-caption QA pairs and CoT traces across 22,948 source entries and retraining SFT+GRPO was not feasible within the rebuttal window, but two comparisons support our graph-based construction without process supervision. On Qwen3-VL-8B-Instruct in Table 2, + SFT drops below Base from 43.29% to 41.18%, whereas + SFT + Outcome-based RL uses zero process supervision with λ=0\lambda = 0 and binary answer correctness only, yet lifts the six-benchmark average to 47.91%, gaining +6.73 pp over + SFT, +18.94 pp on VRBench, and +4.89 pp on Video-Holmes; this aligns with our pairwise non-entailment and slot-omission distractors in Eq. 2 and is supported by Table 5, where masking any slot's certified interval union reduces accuracy from 58.22% to 31.23%. In Table 1, outcome GRPO models trained on flat corpora at matched or 4×4\times to 8×8\times larger scales, including VR-Thinker on 20K at 32.36%, VerlPO on 117K at 41.28%, and Video-R1 on 165K at 42.56%, also trail our 20K + SFT + Outcome-based RL at 47.91%.

AuthorsOpenReview ↗

Response to Reviewer ZHYf (Part 3/3)

Q3. Table 2 has no variant that adds the same evidence score e to the reward instead of the advantage. How does it compare, given that Sec 3.3 (L283) says reward-level grounding alters the optimal policy?

From the identical 10K SFT checkpoint, 10K RL dataset, and evidence score e(i)e^{(i)} in Eq. 6 with λ=0.25\lambda = 0.25, we trained a controlled Evidence-in-Reward RL baseline that adds e(i)e^{(i)} directly into the scalar reward before GRPO group standardization, and compared it against our Evidence-in-Advantage RL in Eq. 7 on Video-Holmes in Table 10:

ConfigurationAdvantageHolmesΔ\Delta Out.Δ\Delta Ours
Qwen3-VL-8B-Instruct—40.39%-4.89 pp+6.64 pp
+ SFT—38.74%-6.54 pp+8.29 pp
+ SFT + Outcome RLz(a(i))z(a^{(i)})45.28%—+1.75 pp
+ SFT + Evidence-in-Rewardz(a(i)+λe(i))z(a^{(i)} + \lambda e^{(i)})45.84%+0.56 pp+1.19 pp
+ SFT + Evidence-in-Advantagez(a(i))+λ(e(i)−eˉ)z(a^{(i)}) + \lambda(e^{(i)} - \bar{e})47.03%+1.75 pp—

Reward injection gains +0.56 pp over Outcome RL from 45.28% to 45.84%, compared to +1.75 pp to 47.03% for Eq. 7, a 3.1×3.1\times larger process improvement. Adding the evidence score into the scalar reward couples outcome normalization with evidence variance in mixed groups and still rewards high-IoU hallucinations when all responses in a group are wrong, whereas Eq. 7 normalizes answer correctness independently, preserves positive advantages for all correct rollouts when λ≤1/G−1\lambda \le 1/\sqrt{G-1}, and mutes the process term when all rollouts are wrong.


Q4. At 16 frames the base scores 54.89 on VideoMME (Table 1), while public aggregators list roughly 71% for this backbone. What do the base and Video-Thinker score with a larger frame budget, and does the ranking hold?

Public leaderboards report ~71% for Qwen3-VL-8B-Instruct on VideoMME under dense sampling of 256 to 768 frames with direct option prediction, whereas Table 1 uses a 16-frame uniform budget across videos up to 1 hour. Scaling across 16, 32, 64, and 128 uniform frames on VideoMME and Video-Holmes in Table 8 shows that both models rise substantially while Video-Thinker preserves its lead at every budget:

BenchmarkModel16f32f64f128fMeanΔ16→Peak\Delta_{16 \to \text{Peak}}
VideoMME (N=2,700N=2{,}700)Qwen3-VL-8B-Instruct54.89%60.15%63.48%65.44%60.99%+10.55 pp
VideoMME (N=2,700N=2{,}700)Video-Thinker57.37%63.07%66.22%68.07%63.68%+10.70 pp
VideoMME (N=2,700N=2{,}700)Δ\Delta (Ours - Base)+2.48 pp+2.92 pp+2.74 pp+2.63 pp+2.69 pp—
Video-Holmes (N=1,837N=1{,}837)Qwen3-VL-8B-Instruct40.39%43.17%44.26%43.98%42.95%+3.87 pp
Video-Holmes (N=1,837N=1{,}837)Video-Thinker47.03%49.59%51.44%51.71%49.94%+4.68 pp
Video-Holmes (N=1,837N=1{,}837)Δ\Delta (Ours - Base)+6.64 pp+6.42 pp+7.18 pp+7.73 pp+6.99 pp—

On VideoMME, scaling from 16 to 128 frames lifts Base by +10.55 pp to 65.44% and Video-Thinker by +10.70 pp to 68.07%, maintaining a +2.48 to +2.92 pp lead at every budget alongside a +6.42 to +7.73 pp lead on Video-Holmes.


Q5. VRBench contributes 3.64 of the +6.86 average gain (21.84/6), yet the base (59.40) trails 7B Video-R1 (69.25). Which App D phase does App F's top-1 accuracy measure, and how often does the base fail the <answer> parse?

In Tables 1 and 2, VRBench evaluates the official outcome-level multiple-choice QA phase MCQ-O of the 1,010-video, 9,468-question release cited in Appendix D, namely arXiv:2506.10857, without ground-truth step annotations at test time. Auditing all 9,468 raw outputs shows that the Qwen3-VL-8B-Instruct base model fails the <answer> parse on only 2.82% of questions, or 267 out of 9,468, corresponding to a 97.18% valid parse rate, compared with 0.86% or 81 out of 9,468 for + SFT and 0.68% or 64 out of 9,468 for Video-Thinker. Even when restricting evaluation strictly to validly parsed responses, conditional accuracy rises from 61.12% on Base and 61.71% on + SFT to 81.79% on Video-Thinker, a +20.67 pp gain over Base and +20.08 pp over + SFT. Thus, the +21.84 pp strict-accuracy gain on VRBench from 59.40% to 81.24% cannot be explained by <answer> parse failures and reflects the +20.08 pp post-SFT RL improvement; in addition, excluding VRBench entirely, Video-Thinker still improves over Base on all five remaining benchmarks by +3.86 pp on average from 40.07% to 43.93% and beats Video-R1 by +6.71 pp from 37.22% to 43.93%.

Reviewer_ESZKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Video-Thinker combines graph-based training data construction with evidence-aware reinforcement learning for video reasoning. The graph represents temporal relations and semantic entailment between events. The model first learns structured reasoning with temporal intervals and captions through SFT, then uses GRPO with a centered temporal IoU evidence term in the advantage. The paper uses Qwen3-VL-8B-Instruct with 10K SFT instances and 10K RL instances. Across six benchmarks, it reports 50.15% average accuracy, compared with 43.29% for the base model.

Critical strengths
  1. The method addresses a useful limitation of answer-only supervision. Its temporal IoU score checks coverage of the designated evidence slots, and the centered evidence term gives feedback beyond final answer correctness (Section 3.3, Eqs. 6-7). This gives a concrete way to encourage temporal grounding during training, which is relevant when a correct answer can come from a shortcut.

  2. The reported gains have support from both reasoning and grounding evaluations. The average accuracy improves by 6.86 percentage points over the base model, and by 2.24 points over SFT plus outcome-based RL (Tables 1-2). Table 3 also reports grounding mIoU increasing from 32.45% to 52.74%. The separate grounding evaluation makes the result more informative than an answer-accuracy comparison alone, although its train/test separation needs clarification.

Critical weaknesses

W1. The evidence-necessity guarantee is stronger than the validation provided. Section 3.2 combines pairwise non-entailment, omission-based distractors, and blind-attack filtering. These are useful safeguards, but pairwise non-entailment alone does not show that every selected event is needed for a particular question. A different distractor also does not establish that partial visual evidence leaves the answer indistinguishable. The supplementary code itself describes the antichain condition as necessary but not sufficient; its generation checks do not independently verify question-specific semantic necessity. I did not find reported answer accuracy after removing each designated evidence slot. The evaluation also uses at most 16 uniformly sampled frames (Section 4.1), so predicting intervals does not establish that the model observed the relevant event in those frames. This matters because faithful use of all required evidence is a central claim. A small evidence-removal check, with the actually visible frames recorded, would help; otherwise, the guarantee should be stated more narrowly.

W2. The evidence advantage is not shown to preserve the accuracy-optimal policy. Section 3.3 uses the zero sum of the centered evidence scores to argue that task-accuracy optimality is preserved. Centering does not make the evidence contribution to the policy gradient vanish, because different trajectories have different score-function gradients. Appendix I derives a positive-advantage bound for correct responses in mixed correct/incorrect groups. This does not establish policy invariance. In the all-correct case, which the paper explicitly treats as an evidence tie-breaker, a correct response with below-average evidence can have negative advantage. Table 2 supports a benefit from the full evidence-aware training, but it does not compare advantage injection with adding the same evidence term to the reward, or vary the evidence weight. Thus, the empirical gain does not yet support the claimed special policy-preservation property. The authors should qualify that claim and clarify what advantage injection changes relative to an equivalent reward term.

Questions for authors
  1. Does each generated question actually require every designated evidence slot? Section 3.2 combines pairwise non-entailment, omission-based distractors, and blind-attack filtering. The supplementary code describes the antichain condition as necessary but not sufficient. What additional check establishes that the remaining visual evidence cannot support the correct answer when a slot is hidden? Please specify the blind-attack model, number of attempts, acceptance threshold, and observed accuracy. A small comparison of full evidence, no video, and removal of each slot would help validate the guarantee. Please explain how removal handles alternative intervals that also satisfy the same slot. This would affect whether I consider evidence necessity established or a useful construction heuristic.

  2. What property does the evidence advantage preserve? Section 3.3 claims preservation of task-accuracy optimality, while Appendix I derives a positive-advantage bound for correct responses in mixed correct/incorrect groups. Please state the assumptions needed to connect these claims, including the intended behavior when every response is correct. If policy invariance is not established, would you limit the claim to the sign bound? Also, how does Eq. 7 differ from mean-centering the corresponding scalar score with the group statistics held fixed? A mathematical explanation, or a comparison using the same evidence signal and explicitly matched normalization, would help assess the need for advantage injection. This answer would affect my assessment of the claimed theoretical contribution.

  3. Are the generated temporal clues supported by the frames available to the model? Section 4.1 uses up to 16 uniformly sampled frames at evaluation. In Figure 5, Video-Thinker's first predicted interval is 00:03-00:04, while its reference frame is labeled 14.667 s, outside that interval. Figure 8 marks some intervals as having no matching frame supplied. Are these limitations of the illustration, or do they reflect missing evidence in the actual input? Please provide the sampled-frame timestamps for these cases and explain how caption content is checked separately from temporal overlap. This would clarify how strongly the examples support faithful visual grounding; the illustration mismatches alone do not establish that the predictions are wrong.

  4. Is the grounding evaluation independent of the post-training videos? Table 4 lists ActivityNet as the base corpus for MDVC and VideoXum, and YouCook2 is also a training source. Table 3 evaluates samples from ActivityNet, YouCook2, and TutorialVQA. Please report the original dataset splits used for SFT, RL, and this evaluation, together with counts of shared original video IDs across sources and between post-training and evaluation. Those intersections cannot be checked from the supplied files. If any evaluation videos were used for post-training, results on the disjoint subset would help interpret the grounding gains. Shared source datasets alone do not establish leakage; the split and intersection counts would directly affect the weight I give to Table 3.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ESZK (Part 1/4)

We thank Reviewer ESZK for the thorough review and for recognizing that our multi-span formulation handles alternative entailing occurrences and all-wrong rollout groups thoughtfully with clear ablations. Below we address W1 in Part 1, W2 in Part 2, Q1–Q2 in Part 3, and Q3–Q4 in Part 4.


W1. The evidence-necessity guarantee is stronger than the validation provided. Section 3.2 combines pairwise non-entailment, omission-based distractors, and blind-attack filtering. These are useful safeguards, but pairwise non-entailment alone does not show that every selected event is needed for a particular question. A different distractor also does not establish that partial visual evidence leaves the answer indistinguishable. The supplementary code itself describes the antichain condition as necessary but not sufficient; its generation checks do not independently verify question-specific semantic necessity. I did not find reported answer accuracy after removing each designated evidence slot. The evaluation also uses at most 16 uniformly sampled frames (Section 4.1), so predicting intervals does not establish that the model observed the relevant event in those frames. This matters because faithful use of all required evidence is a central claim. A small evidence-removal check, with the actually visible frames recorded, would help; otherwise, the guarantee should be stated more narrowly.

We agree that pairwise non-entailment is a structural necessary condition on the video reasoning graph, and we have refined Section 3.2 and Appendix C to state clearly that question-level evidence necessity relies on combining it with reflexive equivalent-span closure in Eq. 3, counterfactual slot-omission distractors in Eq. 2 supplemented with non-entailing video events to form 4 options when k<3k < 3, programmatic surface-shortcut checks, and blind-attack filtering as detailed in Q1.

Following your suggestion, we audited 500 synthesized multi-slot questions with k∈{2,3}k \in \{2, 3\} and 1,184 designated evidence slots under the 16-frame uniform sampling protocol, recording the exact sampled-frame timestamps against each slot's certified interval union in Eq. 3. Across all 1,184 slots, 16 uniform frames achieve a 94.26% slot-level frame hit rate covering 1,116 of 1,184 slots, with all required slots simultaneously visible in N=438N = 438 of 500 questions, or 87.60%, spanning S=1,012S = 1{,}012 visible slots and Sclean=846S_{\text{clean}} = 846 non-overlapping slots that share zero sampled frames with any other slot. Evaluating Qwen3-VL-8B-Instruct on the N=438N = 438 all-slots-visible subset in Appendix C and Table 5 yields:

Evaluation Condition (Qwen3-VL-8B-Instruct)Evaluation GranularityAccuracyQuestion-Level Δ\Delta vs. Full
Full Video (16 Uniformly Sampled Frames)Question-level (N=438N = 438 questions)58.22% (255 / 438)—
Matched Frame Removal (Non-Evidence Frames)Question-level (N=438N = 438 questions)57.31% (251 / 438)-0.91 pp
Leave-One-Slot-Out (All Visible Slots, masking I(sj)\mathcal{I}(s_j))Slot-level (S=1,012S = 1{,}012 slot trials)31.23% (316 / 1,012)—
Leave-One-Slot-Out (Clean Omission, zero frame overlap)Slot-level (Sclean=846S_{\text{clean}} = 846 slot trials)29.55% (250 / 846)—
No Video (Independent Text-Only Blind Evaluation)Question-level (N=438N = 438 questions)24.66% (108 / 438)-33.56 pp

At the question level across the 438 questions, removing an equal number of non-evidence frames leaves accuracy nearly unchanged at 57.31% versus 58.22%, whereas an independent post-filter text-only blind evaluation drops by 33.56 pp to the random-chance floor of 24.66%. At the slot level, masking the frames within any single slot's certified interval union reduces accuracy to 31.23% across all 1,012 visible slots and 29.55% on the 846 clean non-overlapping slots, showing that masking any designated evidence slot substantially degrades accuracy toward the blind floor.

AuthorsOpenReview ↗

Response to Reviewer ESZK (Part 2/4)

W2. The evidence advantage is not shown to preserve the accuracy-optimal policy. Section 3.3 uses the zero sum of the centered evidence scores to argue that task-accuracy optimality is preserved. Centering does not make the evidence contribution to the policy gradient vanish, because different trajectories have different score-function gradients. Appendix I derives a positive-advantage bound for correct responses in mixed correct/incorrect groups. This does not establish policy invariance. In the all-correct case, which the paper explicitly treats as an evidence tie-breaker, a correct response with below-average evidence can have negative advantage. Table 2 supports a benefit from the full evidence-aware training, but it does not compare advantage injection with adding the same evidence term to the reward, or vary the evidence weight. Thus, the empirical gain does not yet support the claimed special policy-preservation property. The authors should qualify that claim and clarify what advantage injection changes relative to an equivalent reward term.

We agree that zero-sum mean centering does not imply global policy-gradient invariance because score-function gradients vary across sampled rollouts. We have updated Section 3.3 and Appendix I to qualify this statement and restrict our formal claim to the exact group-wise guarantees of Eq. 7: sign preservation in mixed-correctness groups under λ≤1/G−1\lambda \le 1/\sqrt{G-1}, within-group preference refinement among sampled correct rollouts in all-correct groups, and zeroing with λ=0\lambda = 0 in all-incorrect groups, as detailed in Q2 below.

To directly test our advantage injection against adding the same evidence term into the scalar reward, we started from the identical 10K SFT checkpoint, 10K RL dataset, and multi-span evidence score e(i)e^{(i)} in Eq. 6 with λ=0.25\lambda = 0.25, and trained a controlled Evidence-in-Reward RL baseline that adds e(i)e^{(i)} directly into the scalar reward before GRPO group standardization on Video-Holmes, as shown in Section 4.2, Appendix I, and Table 10:

ConfigurationAdvantage FormulationVideo-Holmes (N=1,837N=1{,}837)Δ\Delta vs. Outcome RLΔ\Delta (Ours vs. Row)
Qwen3-VL-8B-Instruct (Base)—40.39%-4.89 pp+6.64 pp
+ SFT—38.74%-6.54 pp+8.29 pp
+ SFT + Outcome-based RLz(a(i))z(a^{(i)})45.28%—+1.75 pp
+ SFT + Evidence-in-Reward RLz(a(i)+λe(i))z(a^{(i)} + \lambda e^{(i)})45.84%+0.56 pp+1.19 pp
+ SFT + Evidence-in-Advantage RL (Ours)z(a(i))+λ(e(i)−eˉ)z(a^{(i)}) + \lambda(e^{(i)} - \bar{e})47.03%+1.75 pp—

Scalar reward injection improves over Outcome-based RL by only +0.56 pp from 45.28% to 45.84%, whereas our decoupled advantage injection achieves a +1.75 pp gain to 47.03%, a 3.1×3.1\times larger process improvement. Adding the evidence score into the scalar reward couples the outcome normalization scale with evidence variance in mixed groups and still reinforces high-IoU hallucinations when all responses in a group are wrong, whereas Eq. 7 normalizes answer correctness independently and mutes the process term on all-incorrect groups.

AuthorsOpenReview ↗

Response to Reviewer ESZK (Part 3/4)

Q1. Does each generated question actually require every designated evidence slot? Section 3.2 combines pairwise non-entailment, omission-based distractors, and blind-attack filtering. The supplementary code describes the antichain condition as necessary but not sufficient. What additional check establishes that the remaining visual evidence cannot support the correct answer when a slot is hidden? Please specify the blind-attack model, number of attempts, acceptance threshold, and observed accuracy. A small comparison of full evidence, no video, and removal of each slot would help validate the guarantee. Please explain how removal handles alternative intervals that also satisfy the same slot. This would affect whether I consider evidence necessity established or a useful construction heuristic.

Beyond pairwise non-entailment on the graph, question-level necessity is supported by requiring the generator to abstain if no conclusion depends on all selected slots, deriving counterfactual orderings from timestamps for sequential signatures, and mapping each slot sis_i to a distinct affirmative omission distractor in Eq. 2, supplemented with non-entailing video events to reach 4 options when k<3k < 3. Every candidate must also pass programmatic surface-shortcut checks against polarity asymmetry, length spread above 0.6 or gold length exceeding 1.15×1.15\times the longest distractor, lexical superset or centroid bias, and stem-to-caption word overlap of 60% or higher. Surviving items are filtered with Qwen3-VL-8B-Instruct in text-only mode without video frames or captions across 8 option-permuted trials at temperature 0.7, retaining only items with blind accuracy at or below the 25% random-chance threshold of at most 2 out of 8 correct trials, and an independent post-filter blind evaluation on N=438N = 438 questions yields 24.66%. Finally, as shown in the W1 table above, when removing slot sjs_j in our leave-one-slot-out check, we mask all sampled frames within the full reflexive equivalence union I(sj)=T(sj)∪⋃u⇒sjT(u)\mathcal{I}(s_j) = \mathcal{T}(s_j) \cup \bigcup_{u \Rightarrow s_j} \mathcal{T}(u) in Eq. 3 rather than only T(sj)\mathcal{T}(s_j), removing all alternative entailing occurrences and reducing accuracy from 58.22% on Full Video and 57.31% under Non-Evidence Removal to 31.23% across all 1,012 visible slots and 29.55% across the 846 non-overlapping slots.


Q2. What property does the evidence advantage preserve? Section 3.3 claims preservation of task-accuracy optimality, while Appendix I derives a positive-advantage bound for correct responses in mixed correct/incorrect groups. Please state the assumptions needed to connect these claims, including the intended behavior when every response is correct. If policy invariance is not established, would you limit the claim to the sign bound? Also, how does Eq. 7 differ from mean-centering the corresponding scalar score with the group statistics held fixed? A mathematical explanation, or a comparison using the same evidence signal and explicitly matched normalization, would help assess the need for advantage injection. This answer would affect my assessment of the claimed theoretical contribution.

Yes, we limit our formal claim in Section 3.3 and Appendix I to the sign-preservation bound in mixed groups and within-group preference refinement in all-correct groups, rather than global policy-gradient invariance. On format-compliant rollouts after SFT, Eq. 7 guarantees three group-level behaviors: in mixed groups, λ≤1/G−1\lambda \le 1/\sqrt{G-1} ensures Ai>0A_i > 0 for every correct rollout and Ai<0A_i < 0 for every wrong rollout when λ<1/G−1\lambda < 1/\sqrt{G-1}; in all-correct groups where z(a(i))=0z(a^{(i)}) = 0, Ai=λ(e(i)−eˉ)A_i = \lambda(e^{(i)} - \bar{e}) differentiates sampled correct rollouts by grounding quality without assigning positive credit to any wrong answer, though it does not imply global policy invariance; and in all-incorrect groups, λ=0\lambda = 0 yields Ai=0A_i = 0 to avoid reinforcing ungrounded guesses. Regarding your mathematical question: if the outcome group statistics μa\mu_a and σa\sigma_a are held fixed and we set α=λ(σa+δ)\alpha = \lambda(\sigma_a + \delta), mean-centering the scalar score a(i)+αe(i)a^{(i)} + \alpha e^{(i)} and dividing by σa+δ\sigma_a + \delta is algebraically identical to Eq. 7, since (a(i)+αe(i))−(μa+αeˉ)σa+δ=z(a(i))+λ(e(i)−eˉ)\frac{(a^{(i)} + \alpha e^{(i)}) - (\mu_a + \alpha \bar{e})}{\sigma_a + \delta} = z(a^{(i)}) + \lambda(e^{(i)} - \bar{e}). In contrast, standard GRPO reward shaping with r(i)=a(i)+λe(i)r^{(i)} = a^{(i)} + \lambda e^{(i)} normalizes by the combined standard deviation σr\sigma_r of r(i)r^{(i)}, which couples the mixed-group outcome scale to the variance and covariance of e(i)e^{(i)} and scales all-wrong hallucinations to unit variance, consistent with Eq. 7's 3.1×3.1\times larger gain of +1.75 pp versus +0.56 pp in the W2 table above.

Reviewer_9M7HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Video-Thinker, which unifies data construction and training supervision, to enable intrinsic temporal reasoning without relying on external tools. The approach has two main components: (1) Graph-based training-data construction: Annotated video events are organized using temporal relations and caption-level entailment. The graph supports the generation of questions intended to require multiple nonredundant evidence segments, together with corresponding temporal annotations. (2) Evidence-aware reinforcement learning: Following supervised fine-tuning on 10K reasoning trajectories, the model undergoes RL on 10K instances. A temporal grounding score measures coverage of required evidence intervals and contributes a mean-centered term to the GRPO advantage. The model shows promising performance on six benchmarks.

Critical strengths
  • The proposed approach is well-motivated, and data construction and supervision are coherently designed around temporal evidence.
  • The experiments are comprehensive. Clear gains are obtained against existing approachs. Ablations provide clear evidence that temporal supervision improves reasoning.
Critical weaknesses
  • All models are evaluated using a prompt requiring <time> and <caption> outputs—the precise format explicitly taught to Video-Thinker during training. This introduces a potential format-adaptation confound for competing models.
  • The uniform 16-frame budget provides a controlled sparse-input comparison, but does not establish superiority under each model’s intended operating conditions. Differences in backbones and training data further complicate the comparisons.
  • The paper does not ablate the graph-based synthesis against flat-caption synthesis under matched conditions, despite data construction being a central contribution. Nor does it compare the proposed advantage adjustment with a straightforward temporal-IoU reward baseline.
Questions for authors
  • How is blind-attack filtering implemented?
  • Since adding predicted intervals cannot decrease the maximum-over-predictions score in Eq. (6), what prevents a model from outputting many candidate spans?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9M7H (Part 1/3)

We thank Reviewer 9M7H for the constructive feedback and for recognizing that our data construction and training supervision are coherently designed around temporal evidence with clear empirical gains. Below we address W1–W2 in Part 1, W3 in Part 2, and Q1–Q2 in Part 3.


W1. All models are evaluated using a prompt requiring <time> and <caption> outputs—the precise format explicitly taught to Video-Thinker during training. This introduces a potential format-adaptation confound for competing models.

To separate temporal reasoning gains from <time> and <caption> format adaptation, we ran a controlled prompt ablation across 2,837 questions on Video-Holmes (N=1,837N=1{,}837) and SciVideoBench (N=1,000N=1{,}000), evaluating both Qwen3-VL-8B-Instruct and Video-Thinker under the default Structured prompt and a Tag-Free prompt that replaces <time> and <caption> with natural-language visual descriptions under identical 16-frame inputs and decoding settings (Section 4.1, Appendix G.1, Table 7):

ModelPrompt ProtocolVideo-Holmes (N=1,837N=1{,}837)SciVideoBench (N=1,000N=1{,}000)MeanΔ\Delta vs. Struct. BaseΔ\Delta vs. Tag-Free Base
Qwen3-VL-8B-Instruct (Base)Structured (<time>/<caption>)40.39%28.70%34.55%—+0.07 pp
Qwen3-VL-8B-Instruct (Base)Tag-Free (Natural Language)41.05%27.90%34.48%-0.07 pp—
Video-Thinker (Ours)Tag-Free (Natural Language)44.09%31.20%37.65%+3.10 pp+3.17 pp
Video-Thinker (Ours)Structured (<time>/<caption>)47.03%31.40%39.22%+4.67 pp+4.74 pp

Removing the tag requirements changes the untaught Qwen3-VL-8B-Instruct base model by only +0.66 pp on Video-Holmes and -0.80 pp on SciVideoBench (34.55% vs. 34.48% mean), showing that the structured prompt does not penalize untaught models. Comparing each model under its native format (Structured for Video-Thinker versus Tag-Free for Base) preserves the full +4.74 pp mean gain (+5.98 pp on Video-Holmes, +3.50 pp on SciVideoBench), and Video-Thinker still gains +3.17 pp even when stripped of its learned tags. In addition, the strongest baselines in Table 1 (VideoChat-R1.5, Video-MTR, Video-R2, Video-R1) are also post-trained with explicit temporal grounding supervision, and all variants in Table 2 (+ SFT at 41.18%, + SFT + Outcome-based RL at 47.91%, and Video-Thinker at 50.15%) share the identical 10K <time>/<caption> SFT cold-start data.


W2. The uniform 16-frame budget provides a controlled sparse-input comparison, but does not establish superiority under each model’s intended operating conditions. Differences in backbones and training data further complicate the comparisons.

Because Video-Thinker is trained at FPS=1\text{FPS}=1 up to 384 frames, we scaled the test-time budget across F∈{16,32,64,128}F \in \{16, 32, 64, 128\} uniform frames on Video-Holmes (N=1,837N=1{,}837) and VideoMME (N=2,700N=2{,}700; Section 4.1, Appendix G.1, Table 8):

BenchmarkModel16 Frames32 Frames64 Frames128 FramesMean (16–128f)Δ16→Peak\Delta_{16 \to \text{Peak}}
Video-Holmes (N=1,837N=1{,}837)Qwen3-VL-8B-Instruct (Base)40.39%43.17%44.26%43.98%42.95%+3.87 pp
Video-Holmes (N=1,837N=1{,}837)Video-Thinker (Ours)47.03%49.59%51.44%51.71%49.94%+4.68 pp
Video-Holmes (N=1,837N=1{,}837)Δ\Delta (Ours vs. Base)+6.64 pp+6.42 pp+7.18 pp+7.73 pp+6.99 pp—
VideoMME (N=2,700N=2{,}700)Qwen3-VL-8B-Instruct (Base)54.89%60.15%63.48%65.44%60.99%+10.55 pp
VideoMME (N=2,700N=2{,}700)Video-Thinker (Ours)57.37%63.07%66.22%68.07%63.68%+10.70 pp
VideoMME (N=2,700N=2{,}700)Δ\Delta (Ours vs. Base)+2.48 pp+2.92 pp+2.74 pp+2.63 pp+2.69 pp—
Two-Benchmark MeanΔ\Delta (Ours vs. Base)+4.56 pp+4.67 pp+4.96 pp+5.18 pp+4.84 pp—

Across all four budgets, Video-Thinker outperforms the base model on both benchmarks, with the two-benchmark mean gain widening monotonically from +4.56 pp (16f) to +5.18 pp (128f), reaching 51.71% (+7.73 pp) on Video-Holmes and 68.07% (+2.63 pp) on VideoMME at 128 frames. While external models in Table 1 differ in backbones and training corpora, our method attribution rests on controlled comparisons on the identical Qwen3-VL-8B-Instruct backbone: Video-Thinker gains +6.86 pp average accuracy over Base in Table 1 and +20.29 pp grounding mIoU in Table 3 (while also outperforming the external 8B baseline VideoPro-GRPO at 33.46% by +16.69 pp), and our two-stage 20K recipe (10K SFT + 10K RL) and RL ablations in Table 2 and W3 below are evaluated on the exact same 8B backbone (4×4\times–15×15\times smaller than VideoChat-R1.5 at 80K, VerlPO at 117K, Video-R1 at 165K, and VideoRFT at 310K).

AuthorsOpenReview ↗

Response to Reviewer 9M7H (Part 2/3)

W3. The paper does not ablate the graph-based synthesis against flat-caption synthesis under matched conditions, despite data construction being a central contribution. Nor does it compare the proposed advantage adjustment with a straightforward temporal-IoU reward baseline.

Re-synthesizing 20K flat-caption QA pairs and CoT trajectories across all 22,948 source entries via Gemini-3.1-Flash and retraining two-stage SFT and GRPO from scratch was not feasible within the rebuttal window, but two comparisons in Tables 1, 2, and 5 provide supporting evidence for our graph-based data construction without process supervision. First, in Table 2, + SFT + Outcome-based RL uses zero process supervision with λ=0\lambda = 0, rewarding only binary answer correctness, yet improves the six-benchmark average from 41.18% on + SFT and 43.29% on Base to 47.91%, including +18.94 pp on VRBench and +4.89 pp on Video-Holmes. Because Outcome-based RL supervises only the final option letter, this gain aligns with our pairwise non-entailment constraint and slot-omission distractors in Eq. 2, which we also evaluate directly in Appendix C and Table 5, where masking any single slot's certified interval union in Eq. 3 reduces Qwen3-VL-8B-Instruct accuracy from 58.22% on full videos to 31.23% across all visible slots and 29.55% on non-overlapping slots. Second, in Table 1, VR-Thinker is also trained on 20K instances with outcome-based GRPO without graph-based necessity constraints and reaches 32.36% average accuracy compared with 47.91% for our 20K + SFT + Outcome-based RL, and GRPO baselines trained on 4×4\times to 8×8\times larger flat corpora such as VerlPO at 117K and 41.28% and Video-R1 at 165K and 42.56% also trail our 20K + SFT + Outcome-based RL baseline.

To compare our advantage injection against a scalar temporal-IoU reward baseline, we started from the identical 10K SFT checkpoint, 10K RL dataset, and multi-span evidence score e(i)e^{(i)} in Eq. 6 with λ=0.25\lambda = 0.25, and trained a controlled Evidence-in-Reward RL baseline that adds e(i)e^{(i)} directly into the scalar reward before GRPO group standardization on Video-Holmes, as detailed in Section 4.2, Appendix I, and Table 10:

ConfigurationAdvantage FormulationVideo-Holmes (N=1,837N=1{,}837)Δ\Delta vs. Outcome RLΔ\Delta (Ours vs. Row)
Qwen3-VL-8B-Instruct (Base)—40.39%-4.89 pp+6.64 pp
+ SFT—38.74%-6.54 pp+8.29 pp
+ SFT + Outcome-based RLz(a(i))z(a^{(i)})45.28%—+1.75 pp
+ SFT + Evidence-in-Reward RLz(a(i)+λe(i))z(a^{(i)} + \lambda e^{(i)})45.84%+0.56 pp+1.19 pp
+ SFT + Evidence-in-Advantage RL (Ours)z(a(i))+λ(e(i)−eˉ)z(a^{(i)}) + \lambda(e^{(i)} - \bar{e})47.03%+1.75 pp—

Scalar reward injection improves over Outcome-based RL by only +0.56 pp from 45.28% to 45.84%, whereas our decoupled advantage injection achieves a +1.75 pp gain to 47.03%, a 3.1×3.1\times larger process improvement. Adding the evidence score directly into the scalar reward couples outcome normalization with evidence variance in mixed-correctness groups and still rewards high-IoU hallucinations when all responses in a group are wrong, whereas Eq. 7 normalizes answer correctness independently and mutes the process term on all-incorrect groups.

AuthorsOpenReview ↗

Response to Reviewer 9M7H (Part 3/3)

Q1. How is blind-attack filtering implemented?

During candidate generation with 4 options per question, where non-entailing video events are supplemented when k<3k < 3, we first apply programmatic surface-shortcut checks to reject questions with text-only cues, including cases where the correct answer is the only affirmative or negated option, exceeds the longest distractor by more than 15% or yields a relative length spread above 0.6, forms a lexical superset or overlap centroid of the distractors, or has a question stem sharing at least 60% of its content words with any evidence caption, and we deterministically shuffle the option order. Surviving candidates are then evaluated with the pre-trained base model Qwen3-VL-8B-Instruct in a strictly video-blind, text-only setting across 8 trials with distinct option permutations at temperature 0.7, receiving only the question and options without any video frames or captions. We retain a candidate only if its blind accuracy does not exceed the 25% random-chance threshold of at most 2 out of 8 correct trials, and discard it otherwise.


Q2. Since adding predicted intervals cannot decrease the maximum-over-predictions score in Eq. (6), what prevents a model from outputting many candidate spans?

While Eq. 6 takes the maximum over predicted spans, three factors jointly help discourage interval spamming. First, our format check requires every <time>start-end</time> tag to be paired with a visual description in <caption>...</caption>, and GRPO anchors the policy to the 10K SFT cold-start reference via the KL penalty in Eq. 5 with β=0.04\beta = 0.04, so emitting speculative <time>/<caption> blocks beyond the concise evidence slots incurs a direct token-level KL cost. Second, under Eq. 7, the process score is subordinate to answer correctness with λ=0.25\lambda = 0.25 and is muted to zero on all-incorrect groups; flooding the chain of thought with irrelevant segments pollutes the reasoning context and degrades final answer accuracy, incurring a negative outcome advantage in mixed-correctness groups that outweighs any marginal gain in the evidence score. Third, in Table 3, Video-Thinker improves both grounding mIoU across predicted spans by +20.29 pp from 32.45% to 52.74% and captioning BLEU@1 by +5.55 pp from 12.36% to 17.91%, consistent with improved grounding quality rather than interval spamming.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank Reviewer 9M7H, Reviewer ESZK, and Reviewer ZHYf for their thoughtful reviews and constructive suggestions. We appreciate your recognition of our graph-based task synthesis and evidence-aware RL framework, our multi-span scoring rule that handles alternative entailing occurrences and all-wrong rollout groups, and the data efficiency of our 20K two-stage training recipe.

Below we summarize the new experiments, audits, and clarifications added during the rebuttal, with all revisions highlighted in purple in the updated PDF:

  1. Prompt-format and frame-budget inference ablations in Section 4.1, Appendix G.1, and Tables 7–8: Across 2,837 questions on Video-Holmes and SciVideoBench, replacing <time> and <caption> instructions with a Tag-Free natural-language prompt leaves the untaught Qwen3-VL-8B-Instruct base model virtually unchanged at 34.55% versus 34.48% mean accuracy, showing that the structured prompt does not handicap untaught models. Meanwhile, Video-Thinker gains +4.74 pp under each model's native format and +3.17 pp even when stripped of <time> and <caption> tags at test time. Scaling test-time inputs across 16, 32, 64, and 128 uniform frames on Video-Holmes and VideoMME further shows that our mean advantage over the base model widens monotonically from +4.56 pp at 16 frames to +5.18 pp at 128 frames, reaching 51.71% on Video-Holmes and 68.07% on VideoMME.
  2. Decoupled advantage injection versus scalar temporal-IoU reward baseline in Section 4.2, Appendix I, and Table 10: Under identical 10K SFT initialization and 10K RL data on Video-Holmes, adding the evidence score directly into the scalar reward before GRPO standardization improves over Outcome-based RL by only +0.56 pp, whereas our decoupled advantage injection in Eq. 7 achieves +1.75 pp, a 3.1×3.1\times larger process gain. We also update Section 3.3 and Appendix I to state the exact group-wise guarantees of Eq. 7, namely sign preservation in mixed groups, preference refinement among correct rollouts in all-correct groups, and zeroing in all-incorrect groups, along with its algebraic relation to fixed-statistics mean centering.
  3. Evidence-necessity and human entailment-edge audits in Sections 3.1–3.2, Appendices C & G.2, and Tables 5 & 9: Auditing 500 synthesized multi-slot questions under 16 uniform frames shows a 94.26% slot-level frame hit rate and 87.60% all-slots-visible questions covering N=438N = 438 questions. On Qwen3-VL-8B-Instruct, at the question level across those 438 questions, removing matched non-evidence frames leaves accuracy nearly unchanged at 57.31% versus 58.22%, whereas an independent post-filter text-only blind evaluation drops by 33.56 pp to 24.66%. At the slot level, masking any single slot's certified interval union in Eq. 3 reduces accuracy to 31.23% across all 1,012 visible slots and 29.55% on the 846 non-overlapping slots. In addition, two-annotator human auditing with Cohen's κ=0.91\kappa = 0.91 across 100 stratified source videos and 2,846 directed candidate pairs verifies 92.47% precision, 86.43% recall, 89.35% F1, and 98.56% pairwise accuracy for Gemini-3.1-Flash entailment labelling.
  4. Canonical video-ID de-duplication for Table 3, VRBench parse-compliance audit, and qualitative case clarifications in Section 4.3, Appendices D, F & H, and Table 6: Normalizing video IDs across Table 4, where the 22,948 source dataset entries correspond to 16,734 cross-source unique videos and 11,438 post-training videos, and Table 3 with 1,000 samples across 828 unique videos shows that 83.00% of Table 3 samples, namely 830 samples across 685 videos including all 76 out-of-source TutorialVQA videos, have zero overlap with our 20K post-training set. Re-evaluating on the 830-sample strictly disjoint subset and the 76-sample TutorialVQA subset preserves the full gains of +28.81 pp and +29.36 pp in Grounding Average and +4.64 pp and +4.79 pp in Captioning Average over the base model. We also report the VRBench MCQ-O <answer> parse-failure audit across all 9,468 questions, showing a 2.82% parse failure rate on Base and a +20.67 pp gain in conditional accuracy from 61.12% to 81.79%, alongside the exact sampled-frame timestamps for Figures 5 and 8.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.