- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper adapts a pretrained VLM to a video QA benchmark at test time without labels. For each of 32 unlabeled test questions, it samples K=4 subsets of F=4 frames, generates N=8 reasoning rollouts per subset, and rewards each rollout by the frequency of its extracted answer among all 32 rollouts minus an entropy penalty. The model is updated with GRPO for 5 epochs. The same reward updates per-frame sampling weights through a multiplicative-weights bandit initialized from CLIP scores, and the averaged distribution is used to select 32 frames at inference. The adapted InternVL3-8B reaches a 55.13 average over VideoMMMU, MMVU, SciVideoBench, Video-MME and LongVideoBench (base 51.37), above SFT+RL video reasoning models such as Video-R1 (51.94) and Video-R2 (53.92); the adapted Qwen2.5-VL-7B reaches 51.84 (base 48.82). Ablations report transfer across datasets, gains from as few as one adaptation sample, and higher accuracy with majority-vote rewards than with ground-truth rewards.
Critical strengths
A label-free adaptation signal that uses redundancy specific to video. Pooling answers across different frame subsets of the same video gives a consensus estimate that a single-image method such as TTRV cannot form. With InternVL3-8B, where the base and the adapted model both use 32 frames, the adapted model improves on all five benchmarks (average 51.37 → 55.13). Table 4 shows that adapting on one benchmark improves others: adapting on MMVU raises VideoMMMU from 49.33 to 53.66, and adapting on VideoMMMU raises SciVideoBench from 26.50 to 30.20. If these gains survive the controls requested below, a cheap label-free procedure that transfers across benchmarks would be useful in practice.
Critical weaknesses
- The experiments do not show that the gain comes from better video reasoning, and two results point to a change in output behavior. (a) Adapting Qwen2.5-VL-7B on a single unlabeled sample (V=1, Table 5) raises VideoMMMU-Perception from 58.33 to 67.00, more than three quarters of the gain obtained with 32 samples. A single question is unlikely to teach transferable video reasoning; it can, however, teach the model to finish within the length limit and to emit an answer that the extractor h parses. (b) Training with ground-truth answers gives 60.96 on MMVU, below the 65.11 obtained with majority-vote pseudo-labels. If the reward improved the model by teaching correct answers, true labels should not do worse. The paper reads both results as evidence of generalization but does not test the simpler explanation. The following are needed: the rate of unparseable or length-truncated outputs and the distribution over answer options, before and after adaptation; and a control that adapts on mismatched video–question pairs or on blank frames. If the gains persist under that control, they do not come from video understanding, and the title and main claim would need to change.
- The headline comparison is not like-for-like, and the protocol behind Table 1 is unclear. (a) The claim that TTA-Vid outperforms models trained on large-scale data holds only for InternVL3-8B. Video-R1, VideoChat-R1 and Video-RFT build on Qwen2.5-VL-7B; with that backbone TTA-Vid averages 51.84, below all five reasoning models that report an average (51.94–53.92), although it uses 32 frames against their 128. (b) Section 4.3 states that TTA-Vid "consistently improves the base backbones", but in the Qwen2.5-VL-7B rows LongVideoBench falls from 65.10 to 57.81. The base row's Video-MME (56.00) and LongVideoBench (65.10) values appear swapped: the Qwen2.5-VL technical report (Table 8) gives 65.1 on Video-MME without subtitles and 56.0 on LongVideoBench val. With the two values swapped back, Video-MME falls from 65.10 to 61.14. That base also uses up to 768 frames against TTA-Vid's 32. (c) It is unclear which adaptation set produced each number in Table 1. The InternVL3-8B VideoMMMU entry (53.66) equals the MMVU-adapted entry of Table 4, while the VideoMMMU-adapted (in-domain) entry is 51.48; Table 2 reports the same 53.66 as an in-domain result averaged across runs; the SciVideoBench entry (29.80) matches no entry of Table 4. If each column reports the best source dataset, the selection is made on the test set. (d) Sections 3.3 and 4.2 state that the adapted model is evaluated on the full test set, which includes the 32 adaptation questions, while the Table 3 caption says evaluation is on unseen batches. On MMVU these questions are 5% of the 625-question set, against a reported gain of 4.8 points.
Questions for authors
Clarifications that may change my rating:
- Which adaptation set, and how many runs, produced each number in Table 1?
- What are the accuracies with the 32 adaptation questions excluded from the test sets?
- Are the Video-MME and LongVideoBench values of the Qwen2.5-VL-7B base row swapped? What does that base score with 32 frames?
- For each benchmark, what fraction of outputs fails answer extraction or hits the length limit, before and after adaptation? How does the distribution of predicted options change?
- What accuracy do you obtain when adaptation uses mismatched video–question pairs, or the same questions with blank frames?
- Why does the ground-truth reward underperform the majority-vote reward? Were the hyperparameters and number of steps identical?
- In Section 4.4, self-consistency (62.72 on MMVU) is compared with 65.60, which is the InternVL3-8B result of Table 1, while the preceding experiment in the same paragraph uses Qwen2.5-VL-7B. Which backbone and how many samples does the self-consistency baseline use? The same section argues that adaptation is cheaper because it runs once per batch; how many GPU-hours does one adaptation run (5 epochs of 1,024 rollouts of up to 1,024 tokens) take?
Presentation fixes: Section 4.4 attributes +4.33 (VideoMMMU) and +4.8 (MMVU) to frame sampling, but in Table 2 these are the total gains over the base model; the batched reward gives +3.0 and +4.0, and generalized frame sampling adds +1.33 and +0.8. Equation references are unresolved ("Equation ??"); the Table 1 caption mentions "two instructional video question-answering tasks" for five benchmarks; the example "Perception (72.67 → 74.67)" in Section 4.4 does not appear in any table.
Suggestion that does not affect my rating: the frequency reward requires a discrete answer extractor, and all five benchmarks are evaluated with closed-form answers. Given the title "video reasoning", either evaluate one open-ended task or state the closed-form scope in the title and abstract.