- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TORM, a framework for improving spatial-temporal reasoning in video-language models through continuous latent reasoning. During training, the method generates question-relevant thought videos using selected keyframes and teacher-generated CoT plans. The generated videos are encoded into visual features to supervise a small number of latent tokens. A second training stage removes the latent alignment loss and optimizes only the final answers. At inference time, the model performs reasoning through a bounded latent rollout without generating additional videos or using external tools. Experiments on multiple video understanding benchmarks demonstrate improvements over standard supervised fine-tuning.
Critical strengths
-
The paper explores an interesting direction of internalizing visual reasoning to reduce inference-time overhead. Using generated videos as training-time supervision while avoiding video generation during inference is a reasonable and potentially useful design.
-
The method is simple and easy to follow. The experiments cover multiple video understanding benchmarks and include ablations on latent token size and the two-stage training strategy.
-
The proposed method achieves noticeable improvements over the SFT baseline on several benchmarks, suggesting that the additional latent supervision can be beneficial.
Critical weaknesses
-
The necessity of generated thought videos is not sufficiently demonstrated. The central design choice is to generate thought videos and use their visual features to supervise latent tokens. However, the paper does not compare this approach with simpler alternatives, such as directly using features from the original videos or question-relevant keyframes. Since the original videos already contain the visual evidence needed to answer the questions, these comparisons are important for establishing the contribution of thought-video generation.
-
The reliability of thought-video supervision is unclear. Thought videos are generated from a selected keyframe and a teacher-generated CoT plan, rather than directly extracted from the original videos. The generated videos may therefore contain incorrect actions or temporal events. The paper does not sufficiently evaluate whether these videos are consistent with the original videos or analyze how generation errors affect training.
-
Limited analysis of what the latent tokens actually learn. Although TORM improves performance on video reasoning benchmarks, it is unclear what information the latent tokens learn from the generated videos. For example, do they capture motion and event order, or mainly general visual information about the scene? More experiments on motion direction and event order would help clarify this.
-
Further analysis of experimental robustness would be helpful. The model is trained using approximately 5K generated thought videos and achieves improvements across several benchmarks. However, the paper lacks analysis of how performance changes with different amounts of training data or across multiple random seeds. In addition, the latency comparison would be more convincing if all methods were evaluated under comparable hardware and end-to-end inference settings.
Questions for authors
-
Why is it necessary to generate thought videos rather than directly supervise the latent tokens using features from the original videos or selected keyframes? Could the authors provide controlled comparisons with these alternatives?
-
How do the authors ensure that the generated thought videos are consistent with the original videos? Have they evaluated the impact of incorrect or hallucinated temporal events on training?
-
What information do the latent tokens actually learn? Can the authors provide experiments showing whether they capture motion direction and event order rather than only general scene information?
-
How does performance change with the number of generated thought videos? Were the latency results obtained under comparable hardware and inference settings?
2 comments
Thank you for the detailed responses and additional experiments, which have addressed some of my concerns. However, the benefits of generated thought videos over simpler alternatives remain modest, and the evidence that latent tokens capture meaningful temporal reasoning is still insufficient, so I will maintain my original rating.
We thank the reviewer for the constructive feedback. We address the necessity and reliability of thought-video supervision, the role of latent trajectories, and experimental robustness.
Q1. Why Generated Thought Videos?
We compare STORM with two matched-budget alternatives: original-frame supervision (uniformly sampled video frames as latent targets) and keyframe supervision (features of the selected question-relevant keyframe). All use the same training data and optimization settings.
Keyframes outperform uniform frames by 5.5/4.8/0.9 points, showing the value of question-relevant visual selection. STORM adds 2.3/0.9/1.9 points over keyframes, suggesting that question-conditioned temporal transitions provide useful supervision beyond static representations. Average gains are 5.4 points over original frames and 1.7 over keyframes.
Q2. Thought-Video Reliability
We evaluated 100 randomly sampled thought videos post hoc, using three independent human annotators and majority voting. Pass rates were 93% for keyframe relevance, 89% for CoT/source consistency, 86% for temporal coherence, and 82% for overall usability; inter-annotator agreement was substantial (Fleiss' κ = 0.74).
We also sampled 100 thought videos and directly compared them with their source videos:
This second evaluation measures source fidelity, rather than only internal temporal plausibility. Most samples preserve question-relevant actions, motion, and event order. Imperfect generation remains a source of supervision noise; we will report the evaluation protocols and representative failure cases in the revision.
Q3. What Do the Latent States Learn?
To test their causal role, we conducted a trajectory-swap intervention on 100 matched MVBench pairs. Each target retains its video, question, answer options, and pre-latent KV cache; we replace only the eight latent-feedback vectors with those from another example in the same task category but with a different correct answer.
Swapping reduces accuracy from 66% to 31%, lowers mean gold-answer log probability by 1.60, and changes predictions in 69% of cases. Temporal-order and motion/action predictions flip in 70% and 65% of cases, respectively. Since the visual input and question stay fixed, this intervention demonstrates that latent trajectories causally affect predictions involving temporal and motion information.
On VideoEspresso, STORM's gains over matched SFT + textual CoT are concentrated in temporally demanding categories:
Thus, both interventions and task-level gains support the contribution of latent trajectories to temporal reasoning.
Q4. Robustness and Efficiency
STORM was evaluated on six benchmarks and at multiple model scales. We agree that data-scaling and multi-seed studies would further assess stability and will note these as future evaluations.
For controlled latency, we use one NVIDIA H200, BF16, batch size 1, identical 32-frame inputs, and the same 20 MMVU examples. Timings include visual encoding, prefill, reasoning, and answer decoding.
STORM is 2.34× faster than textual CoT and adds only 0.08 s/item over direct SFT. Its eight-step latent rollout takes 0.12 s/item (~26% of total latency), yielding bounded reasoning overhead without external video generation.
We thank the reviewer for suggestions that have strengthened our evaluation.