- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Video-Thinker post-trains Qwen3-VL-8B-Instruct for temporal video reasoning on 20K synthesized instances. Each video becomes a graph whose temporal edges follow Allen's interval relations (Eq 1) and whose entailment edges come from one Gemini-3.1-Flash call per video. Each question uses a pairwise non-entailing evidence set A, with answer y = R_Q(A) and distractors d_i = R_Q(A ∖ {s_i}) (Eq 2), so each segment should be necessary. After an SFT cold start on <time>/<caption> trajectories, GRPO adds a multi-span tIoU evidence score to the advantage, A_i = z(a_i) + λ(e_i − ē), with λ ≤ 1/√(G−1) (Eq 6-8). At 16 frames the model averages 50.15 on six benchmarks against 43.29 for its base (Table 1); Table 2 ablates the training stages and Table 3 probes grounding and captioning.
Critical strengths
-
The data construction attacks shortcut answering at its source. Slot-omission distractors d_i = R_Q(A ∖ {s_i}) over a pairwise non-entailing evidence set (Eq 2) make every selected segment necessary by construction, given correct entailment labels, and blind-attack filtering drops items answerable without the video. Temporal edges come deterministically from Allen's 13 interval relations (Eq 1), so that part of the graph involves no LLM. I found no prior video-reasoning training set built on a formal necessity construction, and the pipeline is specified well enough to reproduce (Alg 2; App G.2).
-
The process term is measured in isolation on the same backbone. The evidence score enters the advantage as a mean-centered term, bounded by λ ≤ 1/√(G−1) so that correct trajectories keep a positive advantage (Eq 7-8; App I). Against outcome-only RL on the identical backbone, it lifts the six-benchmark average from 47.91 to 50.15, a +2.24 gain (Table 2). That is the paper's most direct causal evidence, and I credit the authors for also reporting VideoTT, where the process term does not help.
-
The headline gain is measured on benchmarks outside the training corpora. None of the six Table 1 benchmarks is among the four training corpora of Table 4, and the base and Video-Thinker share the backbone and the 16-frame protocol of Sec 4.1 (L371-373). The +6.86 average gain (50.15 vs 43.29, Table 1) is therefore a held-out, same-backbone result, and the backbone issue in W1 does not affect it.
Critical weaknesses
-
The "state-of-the-art" claim (Sec 4.2) cannot be separated from an 8B-versus-7B backbone gap. Every post-trained baseline except VideoPro-GRPO is 7B, and the Qwen3-VL-8B-Instruct base, before any reasoning post-training, already averages 43.29, above 14 of the 17 post-trained baselines (Table 1). With no run of the recipe on a 7B backbone and no baseline re-run on the 8B one, only the same-backbone gain can be credited to the method. Fix: train the identical SFT and process-RL recipe on a 7B backbone such as Qwen2.5-VL-7B, or re-run Video-R1 or VideoChat-R1 on Qwen3-VL-8B; both fit the paper's own setup.
-
Two of the three sources behind the grounding table also supply training videos, and no held-out split is stated. Table 3 evaluates on 1K samples from ActivityNet, YouCook2, and TutorialVQA, yet ActivityNet supplies 17,103 of the 22,948 training-source videos and YouCook2 is also a training source (Table 4). I searched for "held-out", "disjoint", "split", "de-dup", and "exclude" and found nothing that applies to Table 3, so its gains may reflect train/test overlap rather than the intrinsic grounding Sec 4.3 claims to isolate. Fix: report the de-duplication counts and the TutorialVQA-only scores, or re-evaluate on disjoint videos; neither needs training.
-
The "guaranteed evidence necessity" rests on one unvalidated LLM pass. Entailment edges for all 22,948 source videos (Table 4) come from one Gemini-3.1-Flash call per video (Sec 3.1; App G.2), and I found no accuracy check of these labels. By the paper's own argument (Sec 3.2), a missed edge leaves a superfluous segment in A; by Eq 3, a spurious one adds wrong intervals to the certified spans that Eq 6 scores against. Neither Sec 3.2 safeguard, the d_i ≠ y check or blind-attack filtering, audits these labels. Fix: report human-audited edge precision and recall on sampled videos; this needs annotation, not training.
Questions for authors
- Are the Table 3 ActivityNet and YouCook2 test videos disjoint from the MDVC, VideoXum, and YouCook2 training videos? Please give the de-duplication procedure and held-out counts; this answer is the most likely to change my score.
- Table 2 varies only training stages; Sec 4.2 credits the necessity construction only across models (L415-416, L427-428). What does SFT + outcome RL (47.91) reach with 20K flat-caption items (L74) of the same segments?
- Table 2 has no variant that adds the same evidence score e to the reward instead of the advantage. How does it compare, given that Sec 3.3 (L283) says reward-level grounding alters the optimal policy?
- At 16 frames the base scores 54.89 on VideoMME (Table 1), while public aggregators list roughly 71% for this backbone. What do the base and Video-Thinker score with a larger frame budget, and does the ranking hold?
- VRBench contributes 3.64 of the +6.86 average gain (21.84/6), yet the base (59.40) trails 7B Video-R1 (69.25). Which App D phase does App F's top-1 accuracy measure, and how often does the base fail the <answer> parse?
3 comments
Response to Reviewer ZHYf (Part 2/3)
W3. The "guaranteed evidence necessity" rests on one unvalidated LLM pass. Entailment edges for all 22,948 source videos (Table 4) come from one Gemini-3.1-Flash call per video (Sec 3.1; App G.2), and I found no accuracy check of these labels. By the paper's own argument (Sec 3.2), a missed edge leaves a superfluous segment in A; by Eq 3, a spurious one adds wrong intervals to the certified spans that Eq 6 scores against. Neither Sec 3.2 safeguard, the d_i ≠ y check or blind-attack filtering, audits these labels. Fix: report human-audited edge precision and recall on sampled videos; this needs annotation, not training.
Two annotators audited Gemini-3.1-Flash entailment labels across 100 stratified videos matching Table 4's proportions of 40 MDVC, 35 VideoXum, 17 VidSTG, and 8 YouCook2 videos, covering 486 segments and 1,423 undirected or 2,846 directed pairs. Applying the atomic checklist in Appendix G.2 with identified 199 directed edges and 2,647 non-entailing pairs in Table 9:
| Scope | Videos (Segs) | Pairs | TP | FP | FN | TN | Precision | Recall | F1 | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| Stratified 4-Source | 100 (486) | 2,846 | 172 | 14 | 27 | 2,633 | 92.47% | 86.43% | 89.35% | 98.56% |
Because our checklist enforces single-step verification without transitive closure on predicted edges, the 14 false positives out of 2,846 pairs, or 0.49% of pairs with , keep direct expansion of the certified interval union in Eq. 3 minimal. If any of the 27 false negatives, or 0.95% of pairs, admits two related segments into the candidate set, the generator abstains or fails the slot-omission check in Eq. 2 when omitting one slot leaves the answer recoverable, as supported by Table 5 where masking any slot's interval union reduces accuracy from 58.22% to 31.23% across visible slots and 29.55% on clean slots.
Q1. Are the Table 3 ActivityNet and YouCook2 test videos disjoint from the MDVC, VideoXum, and YouCook2 training videos? Please give the de-duplication procedure and held-out counts; this answer is the most likely to change my score.
They are 83.00% disjoint at the sample level, covering 830 of 1,000 samples across 685 of 828 videos. Normalizing all IDs to 11-character YouTube IDs by stripping v_ so ActivityNet and YouCook2 share one ID space with 0 overlap, alongside VidOR IDs, shows that Table 4's 22,948 entries comprise 9,123 MDVC and 7,980 VideoXum entries sharing 6,214 ActivityNet IDs, or 10,889 unique ActivityNet videos; with 4,055 VidSTG and 1,790 YouCook2 videos from 1,333 train and 457 val clips, the pool has 16,734 unique videos, and our 20K post-training set covers 11,438 unique videos, with 10K SFT from 6,842 and 10K RL from 6,915, sharing 2,319. Matching Table 3's 828 videos and 1,000 samples against all 11,438 post-training videos yields:
| Source | Samples | Videos | SFT | RL | (SFT RL) | Disjoint Videos (Samples) |
|---|---|---|---|---|---|---|
| ActivityNet (val) | 662 | 568 | 72 | 75 | 119 (137) | 449 (525, 79.31%) |
| YouCook2 (val) | 262 | 184 | 15 | 14 | 24 (33) | 160 (229, 87.40%) |
| TutorialVQA | 76 | 76 | 0 | 0 | 0 (0) | 76 (76, 100.00%) |
| Total (Table 3) | 1,000 | 828 | 87 | 89 | 143 (170) | 685 (830, 83.00%) |
Because VideoXum uses a custom split and cross-source ID filtering was not enforced before 20K sampling, 119 ActivityNet videos with 137 samples and 24 YouCook2 videos with 33 samples overlapped with post-training. As shown in W2, re-evaluating on the 685 disjoint videos with 830 samples and on the 76 TutorialVQA videos yields virtually identical gains of +28.81 pp and +29.36 pp in Grounding Avg.
Q2. Table 2 varies only training stages; Sec 4.2 credits the necessity construction only across models (L415-416, L427-428). What does SFT + outcome RL (47.91) reach with 20K flat-caption items (L74) of the same segments?
Re-synthesizing 20K flat-caption QA pairs and CoT traces across 22,948 source entries and retraining SFT+GRPO was not feasible within the rebuttal window, but two comparisons support our graph-based construction without process supervision. On Qwen3-VL-8B-Instruct in Table 2, + SFT drops below Base from 43.29% to 41.18%, whereas + SFT + Outcome-based RL uses zero process supervision with and binary answer correctness only, yet lifts the six-benchmark average to 47.91%, gaining +6.73 pp over + SFT, +18.94 pp on VRBench, and +4.89 pp on Video-Holmes; this aligns with our pairwise non-entailment and slot-omission distractors in Eq. 2 and is supported by Table 5, where masking any slot's certified interval union reduces accuracy from 58.22% to 31.23%. In Table 1, outcome GRPO models trained on flat corpora at matched or to larger scales, including VR-Thinker on 20K at 32.36%, VerlPO on 117K at 41.28%, and Video-R1 on 165K at 42.56%, also trail our 20K + SFT + Outcome-based RL at 47.91%.
Response to Reviewer ZHYf (Part 3/3)
Q3. Table 2 has no variant that adds the same evidence score e to the reward instead of the advantage. How does it compare, given that Sec 3.3 (L283) says reward-level grounding alters the optimal policy?
From the identical 10K SFT checkpoint, 10K RL dataset, and evidence score in Eq. 6 with , we trained a controlled Evidence-in-Reward RL baseline that adds directly into the scalar reward before GRPO group standardization, and compared it against our Evidence-in-Advantage RL in Eq. 7 on Video-Holmes in Table 10:
| Configuration | Advantage | Holmes | Out. | Ours |
|---|---|---|---|---|
| Qwen3-VL-8B-Instruct | — | 40.39% | -4.89 pp | +6.64 pp |
| + SFT | — | 38.74% | -6.54 pp | +8.29 pp |
| + SFT + Outcome RL | 45.28% | — | +1.75 pp | |
| + SFT + Evidence-in-Reward | 45.84% | +0.56 pp | +1.19 pp | |
| + SFT + Evidence-in-Advantage | 47.03% | +1.75 pp | — |
Reward injection gains +0.56 pp over Outcome RL from 45.28% to 45.84%, compared to +1.75 pp to 47.03% for Eq. 7, a larger process improvement. Adding the evidence score into the scalar reward couples outcome normalization with evidence variance in mixed groups and still rewards high-IoU hallucinations when all responses in a group are wrong, whereas Eq. 7 normalizes answer correctness independently, preserves positive advantages for all correct rollouts when , and mutes the process term when all rollouts are wrong.
Q4. At 16 frames the base scores 54.89 on VideoMME (Table 1), while public aggregators list roughly 71% for this backbone. What do the base and Video-Thinker score with a larger frame budget, and does the ranking hold?
Public leaderboards report ~71% for Qwen3-VL-8B-Instruct on VideoMME under dense sampling of 256 to 768 frames with direct option prediction, whereas Table 1 uses a 16-frame uniform budget across videos up to 1 hour. Scaling across 16, 32, 64, and 128 uniform frames on VideoMME and Video-Holmes in Table 8 shows that both models rise substantially while Video-Thinker preserves its lead at every budget:
| Benchmark | Model | 16f | 32f | 64f | 128f | Mean | |
|---|---|---|---|---|---|---|---|
| VideoMME () | Qwen3-VL-8B-Instruct | 54.89% | 60.15% | 63.48% | 65.44% | 60.99% | +10.55 pp |
| VideoMME () | Video-Thinker | 57.37% | 63.07% | 66.22% | 68.07% | 63.68% | +10.70 pp |
| VideoMME () | (Ours - Base) | +2.48 pp | +2.92 pp | +2.74 pp | +2.63 pp | +2.69 pp | — |
| Video-Holmes () | Qwen3-VL-8B-Instruct | 40.39% | 43.17% | 44.26% | 43.98% | 42.95% | +3.87 pp |
| Video-Holmes () | Video-Thinker | 47.03% | 49.59% | 51.44% | 51.71% | 49.94% | +4.68 pp |
| Video-Holmes () | (Ours - Base) | +6.64 pp | +6.42 pp | +7.18 pp | +7.73 pp | +6.99 pp | — |
On VideoMME, scaling from 16 to 128 frames lifts Base by +10.55 pp to 65.44% and Video-Thinker by +10.70 pp to 68.07%, maintaining a +2.48 to +2.92 pp lead at every budget alongside a +6.42 to +7.73 pp lead on Video-Holmes.
Q5. VRBench contributes 3.64 of the +6.86 average gain (21.84/6), yet the base (59.40) trails 7B Video-R1 (69.25). Which App D phase does App F's top-1 accuracy measure, and how often does the base fail the <answer> parse?
In Tables 1 and 2, VRBench evaluates the official outcome-level multiple-choice QA phase MCQ-O of the 1,010-video, 9,468-question release cited in Appendix D, namely arXiv:2506.10857, without ground-truth step annotations at test time. Auditing all 9,468 raw outputs shows that the Qwen3-VL-8B-Instruct base model fails the <answer> parse on only 2.82% of questions, or 267 out of 9,468, corresponding to a 97.18% valid parse rate, compared with 0.86% or 81 out of 9,468 for + SFT and 0.68% or 64 out of 9,468 for Video-Thinker. Even when restricting evaluation strictly to validly parsed responses, conditional accuracy rises from 61.12% on Base and 61.71% on + SFT to 81.79% on Video-Thinker, a +20.67 pp gain over Base and +20.08 pp over + SFT. Thus, the +21.84 pp strict-accuracy gain on VRBench from 59.40% to 81.24% cannot be explained by <answer> parse failures and reflects the +20.08 pp post-SFT RL improvement; in addition, excluding VRBench entirely, Video-Thinker still improves over Base on all five remaining benchmarks by +3.86 pp on average from 40.07% to 43.93% and beats Video-R1 by +6.71 pp from 37.22% to 43.93%.
Response to Reviewer ZHYf (Part 1/3)
We thank Reviewer ZHYf for the rigorous, detail-oriented review and constructive suggestions. Below we address W1–W2 in Part 1, W3 and Q1–Q2 in Part 2, and Q3–Q5 in Part 3.
We agree that cross-model comparisons in Table 1 reflect end-to-end system performance across pre-packaged 7B and 8B checkpoints, and we have refined Section 4.2 to credit our method specifically with its controlled, same-backbone gains on Qwen3-VL-8B-Instruct. Starting from an 8B model does not automatically yield post-training gains: on Qwen3-VL-8B-Instruct, single-stage 10K + SFT at 41.18% and 10K + RL at 40.81% in Table 2 both drop below the untaught 8B base at 43.29%, as does the external 8B baseline VideoPro-GRPO at 33.46% in Table 1. On the identical Qwen3-VL-8B-Instruct backbone, our two-stage 20K recipe delivers a +6.86 pp six-benchmark average gain over the base model from 43.29% to 50.15%, improving every benchmark by +2.48 pp to +21.84 pp, alongside a +20.29 pp grounding mIoU gain from 32.45% to 52.74% in Table 3 and +7.38 pp to +24.41 pp gains over same-backbone Qwen3-VL-8B-Instruct tool-invocation pipelines in Appendix J. Under matched 20K data on the same Qwen3-VL-8B-Instruct backbone, we also compare directly against outcome-only GRPO at 47.91% six-benchmark average and 45.28% on Video-Holmes, as well as scalar temporal-IoU reward shaping at 45.84% on Video-Holmes in Table 10, where our decoupled advantage injection reaches 47.03%, a +1.75 pp gain over Outcome RL that is 3.1× larger than scalar reward shaping.
Following your suggestion, we audited canonical video IDs across all training and evaluation sources, with the full breakdown in Q1 below, and re-evaluated Table 3 on both the strictly disjoint subset of 830 samples across 685 videos with zero overlap against our 20K post-training set and the out-of-source TutorialVQA-only subset of 76 samples with zero overlap, as reported in Section 4.3, Appendix F, and Table 6:
Excluding all 170 overlapping samples across 143 videos changes Video-Thinker's Grounding Average by only -0.60 pp from 65.24% to 64.64% and Captioning Average by -0.15 pp from 19.57% to 19.42%, preserving a +28.81 pp grounding gain and +4.64 pp captioning gain over the base model on strictly unseen videos; on out-of-source TutorialVQA with zero overlap, Video-Thinker similarly improves Grounding Average by +29.36 pp from 35.17% to 64.53% and Captioning Average by +4.79 pp from 14.52% to 19.31%.