acceptodds
Under review as a conference paper at ICLR 2027

STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models

Abstract

Many video reasoning tasks require tracking motion, temporal order, and evolving visual states across frames. Existing methods built on large vision-language models (LVLMs) often address this challenge by externalizing reasoning through textual chain-of-thought (CoT), keyframe selection, repeated frame reinsertion, or external tool use. While effective, such pipelines increase inference-time latency and engineering complexity, and they force temporal-visual evidence to be serialized into text or repeatedly re-encoded from frames. Inspired by the intuition that visual reasoning can occur implicitly before verbalization, we propose STORM (Spatial-Temporal reasoning via internalized Modeling), a two-stage framework that teaches LVLMs to reason through bounded continuous latent trajectories instead of explicit textual CoT. In Stage I, STORM aligns latent tokens with thought-video representations derived from generated videos, grounding the latent states in dynamic visual evidence. In Stage II, the model is further trained with answer-only supervision, encouraging the reasoning process to be internalized without step-by-step annotations. Generated thought videos are used only during training; at inference, STORM performs a bounded latent rollout without regenerating videos, reinserting frames, or invoking external visual tools. Experiments on VideoMME, MVBench, TempCompass, and MMVU show that STORM improves video reasoning accuracy while substantially reducing inference overhead compared with tool- or video-generation-based reasoning pipelines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodality, Video Reasoning, Latent space

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GWXkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TORM, a framework for improving spatial-temporal reasoning in video-language models through continuous latent reasoning. During training, the method generates question-relevant thought videos using selected keyframes and teacher-generated CoT plans. The generated videos are encoded into visual features to supervise a small number of latent tokens. A second training stage removes the latent alignment loss and optimizes only the final answers. At inference time, the model performs reasoning through a bounded latent rollout without generating additional videos or using external tools. Experiments on multiple video understanding benchmarks demonstrate improvements over standard supervised fine-tuning.

Critical strengths
  1. The paper explores an interesting direction of internalizing visual reasoning to reduce inference-time overhead. Using generated videos as training-time supervision while avoiding video generation during inference is a reasonable and potentially useful design.

  2. The method is simple and easy to follow. The experiments cover multiple video understanding benchmarks and include ablations on latent token size and the two-stage training strategy.

  3. The proposed method achieves noticeable improvements over the SFT baseline on several benchmarks, suggesting that the additional latent supervision can be beneficial.

Critical weaknesses
  1. The necessity of generated thought videos is not sufficiently demonstrated. The central design choice is to generate thought videos and use their visual features to supervise latent tokens. However, the paper does not compare this approach with simpler alternatives, such as directly using features from the original videos or question-relevant keyframes. Since the original videos already contain the visual evidence needed to answer the questions, these comparisons are important for establishing the contribution of thought-video generation.

  2. The reliability of thought-video supervision is unclear. Thought videos are generated from a selected keyframe and a teacher-generated CoT plan, rather than directly extracted from the original videos. The generated videos may therefore contain incorrect actions or temporal events. The paper does not sufficiently evaluate whether these videos are consistent with the original videos or analyze how generation errors affect training.

  3. Limited analysis of what the latent tokens actually learn. Although TORM improves performance on video reasoning benchmarks, it is unclear what information the latent tokens learn from the generated videos. For example, do they capture motion and event order, or mainly general visual information about the scene? More experiments on motion direction and event order would help clarify this.

  4. Further analysis of experimental robustness would be helpful. The model is trained using approximately 5K generated thought videos and achieves improvements across several benchmarks. However, the paper lacks analysis of how performance changes with different amounts of training data or across multiple random seeds. In addition, the latency comparison would be more convincing if all methods were evaluated under comparable hardware and end-to-end inference settings.

Questions for authors
  1. Why is it necessary to generate thought videos rather than directly supervise the latent tokens using features from the original videos or selected keyframes? Could the authors provide controlled comparisons with these alternatives?

  2. How do the authors ensure that the generated thought videos are consistent with the original videos? Have they evaluated the impact of incorrect or hallucinated temporal events on training?

  3. What information do the latent tokens actually learn? Can the authors provide experiments showing whether they capture motion direction and event order rather than only general scene information?

  4. How does performance change with the number of generated thought videos? Were the latency results obtained under comparable hardware and inference settings?

2 comments

We thank the reviewer for the constructive feedback. We address the necessity and reliability of thought-video supervision, the role of latent trajectories, and experimental robustness.

Q1. Why Generated Thought Videos?

We compare STORM with two matched-budget alternatives: original-frame supervision (uniformly sampled video frames as latent targets) and keyframe supervision (features of the selected question-relevant keyframe). All use the same training data and optimization settings.

SupervisionVideoMMEMVBenchTempCompass
Uniform original frames53.255.471.5
Selected keyframe58.760.272.4
STORM thought video61.061.174.3

Keyframes outperform uniform frames by 5.5/4.8/0.9 points, showing the value of question-relevant visual selection. STORM adds 2.3/0.9/1.9 points over keyframes, suggesting that question-conditioned temporal transitions provide useful supervision beyond static representations. Average gains are 5.4 points over original frames and 1.7 over keyframes.

Q2. Thought-Video Reliability

We evaluated 100 randomly sampled thought videos post hoc, using three independent human annotators and majority voting. Pass rates were 93% for keyframe relevance, 89% for CoT/source consistency, 86% for temporal coherence, and 82% for overall usability; inter-annotator agreement was substantial (Fleiss' κ = 0.74).

We also sampled 100 thought videos and directly compared them with their source videos:

Source-video fidelity criterionPass rate
Action consistency95%
Event-order and motion consistency96%
No answer-relevant hallucinated events98%

This second evaluation measures source fidelity, rather than only internal temporal plausibility. Most samples preserve question-relevant actions, motion, and event order. Imperfect generation remains a source of supervision noise; we will report the evaluation protocols and representative failure cases in the revision.

Q3. What Do the Latent States Learn?

To test their causal role, we conducted a trajectory-swap intervention on 100 matched MVBench pairs. Each target retains its video, question, answer options, and pre-latent KV cache; we replace only the eight latent-feedback vectors with those from another example in the same task category but with a different correct answer.

MVBench categoryPairsPrediction flips
Temporal order2070%
State change2080%
Motion/action2065%
Future-action prediction2075%
Causal/event reasoning2055%
Overall10069%

Swapping reduces accuracy from 66% to 31%, lowers mean gold-answer log probability by 1.60, and changes predictions in 69% of cases. Temporal-order and motion/action predictions flip in 70% and 65% of cases, respectively. Since the visual input and question stay fixed, this intervention demonstrates that latent trajectories causally affect predictions involving temporal and motion information.

On VideoEspresso, STORM's gains over matched SFT + textual CoT are concentrated in temporally demanding categories:

TaskGain
Event Process+14.9 pp
Interaction Dynamics+12.9 pp
Causal Inference+11.3 pp
Emotional Recognition+1.5 pp

Thus, both interventions and task-level gains support the contribution of latent trajectories to temporal reasoning.

Q4. Robustness and Efficiency

STORM was evaluated on six benchmarks and at multiple model scales. We agree that data-scaling and multi-seed studies would further assess stability and will note these as future evaluations.

For controlled latency, we use one NVIDIA H200, BF16, batch size 1, identical 32-frame inputs, and the same 20 MMVU examples. Timings include visual encoding, prefill, reasoning, and answer decoding.

MethodSeconds/item
Original Qwen2.5-VL-7B, direct0.40
Qwen2.5-VL-7B-SFT, direct0.39
Qwen2.5-VL-7B-SFT + textual CoT1.10
STORM, 8 latent steps0.47

STORM is 2.34× faster than textual CoT and adds only 0.08 s/item over direct SFT. Its eight-step latent rollout takes 0.12 s/item (~26% of total latency), yielding bounded reasoning overhead without external video generation.

We thank the reviewer for suggestions that have strengthened our evaluation.

Thank you for the detailed responses and additional experiments, which have addressed some of my concerns. However, the benefits of generated thought videos over simpler alternatives remain modest, and the evidence that latent tokens capture meaningful temporal reasoning is still insufficient, so I will maintain my original rating.

Reviewer_vfkzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STORM trains a VLM to answer through a short continuous latent rollout, using generated thought video features as intermediate supervision. Its two stage training first aligns latent states with these features and then optimizes answer prediction alone, removing video generation and external tools from inference.

Critical strengths

Using generated videos only during training, and the eight slot latent budget bounds the additional decoding steps. The inference procedure avoids the orchestration required by external visual tools. And improves over the same data SFT baseline across all six evaluated benchmarks. Table 6 also shows that the complete training procedure outperforms either stage alone.

Critical weaknesses
  1. The paper does not compare generated thought video supervision with direct supervision from source video or keyframe features. Without these controls, it is unclear whether generating new videos contributes anything beyond ordinary visual feature distillation.
  2. The thought videos are generated from a single keyframe and a teacher reasoning content, but their motion and event order are not validated against the source videos. Errors in the teacher reasoning or generated motion could therefore train the latent states to encode actions or event orders that never occurred.
Questions for authors
  1. With training data and optimization steps held fixed, how does thought video supervision compare with supervision from source video or static keyframe features?
  2. Are the actions and event order in the generated thought videos consistent with those in the source videos?
3 comments

We thank the reviewer for recognizing the benefits of STORM's bounded latent reasoning and two-stage training framework. We address the concerns regarding the necessity and reliability of generated thought-video supervision below.

Q1. Generated Thought Videos vs. Original Video and Keyframe Supervision

We agree that comparing generated thought-video supervision against directly extracted visual features is important for isolating the contribution of the proposed supervision strategy.

To address this, we conduct two additional controlled experiments: (1) Original-video supervision, where frames are uniformly sampled from the original video and their visual features are used as latent supervision targets; and (2) Keyframe supervision, where features extracted from the selected question-relevant keyframe serve as supervision targets. Both use the same settings as STORM with matched training data and optimization budgets.

Supervision sourceVideoMMEMVBenchTempCompass
Uniformly sampled original frames53.255.471.5
Selected keyframe58.760.272.4
Generated thought video (STORM)61.061.174.3

The results demonstrate that STORM consistently outperforms both alternative supervision strategies across all three benchmarks.

First, selected-keyframe supervision improves over uniformly sampled original frames by 5.5, 4.8, and 0.9 percentage points on VideoMME, MVBench, and TempCompass, respectively. This suggests that selecting question-relevant visual evidence provides more informative supervision than uniformly sampling frames, where relevant information may be diluted by less informative content.

Second, STORM further improves over keyframe supervision by 2.3, 0.9, and 1.9 points, respectively. While selected keyframes provide focused visual evidence, they lack the temporal transitions represented by generated thought videos. These results suggest that question-conditioned visual trajectories provide additional supervisory information beyond static keyframe representations.

Overall, STORM achieves average improvements of 5.4 points over original-frame supervision and 1.7 points over keyframe supervision, supporting the effectiveness of generated thought-video supervision.

Q2. Reliability and Temporal Consistency of Generated Thought Videos

We agree that the reliability of generated thought videos is important, since inaccurate synthetic events may introduce noise into latent supervision.

To assess supervision quality, we previously conducted a post-hoc human evaluation of 100 randomly sampled thought videos, with three independent annotators and majority-vote aggregation.

Evaluation criterionPass rate
Keyframe relevance93%
CoT/source consistency89%
Temporal coherence86%
Overall usability82%

The evaluation achieved substantial inter-annotator agreement (Fleiss' κ = 0.74).

While these results support the overall quality of generated supervision, we distinguish temporal coherence within a generated video from factual consistency with the original source video.

To directly assess source-video fidelity, we additionally sampled 100 thought videos and compared them against their corresponding original videos, evaluating action consistency, event ordering, motion consistency, and answer-relevant hallucinations.

CriterionPass rate
Action consistency95%
Event-order and motion consistency96%
Absence of answer-relevant hallucinated events98%

The results indicate that generated thought videos largely preserve the question-relevant spatiotemporal information in the original videos. Specifically, 95% of evaluated samples maintain consistent actions, 96% preserve event ordering and motion, and 98% contain no answer-relevant hallucinated events.

Unlike the previous temporal-coherence evaluation, which assesses the internal plausibility of generated videos, this evaluation directly measures consistency with the source videos. These findings suggest that thought-video generation generally provides reliable temporal supervision, with relatively few incorrect or hallucinated events that could mislead latent-state learning.

We clarify that no additional rejection filtering was performed during the original training-data construction. Both evaluations were conducted post hoc to assess supervision quality. We will include the evaluation protocols and representative failure cases in the revised manuscript to clarify the reliability and limitations of generated supervision.

We thank the reviewer again for these constructive suggestions, which helped us strengthen the experimental validation of generated thought-video supervision.

Thank you for adding the source-frame and keyframe baselines and for comparing the generated thought videos directly with their source videos. These experiments address part of my initial concerns. My main question remains whether generating a thought video adds value beyond selecting relevant source evidence or directly distilling the teacher’s reasoning. I am keeping my score at 2 for the following reasons.

  1. The uniformly sampled source frames may miss question-relevant events, while a single selected keyframe cannot capture temporal transitions. The generated-video condition also incorporates the teacher’s reasoning plan.
  2. The reported 96% consistency rate is difficult to interpret without knowing how many clips contained events whose order could be assessed and what level of motion agreement counted as a pass. The 98% hallucination result also leaves out an important distinction. A generated clip could show the first two actions in the correct order, introduce no new events, and still omit the third action that determines the answer. This leaves unclear how such omissions were scored.
  3. In Figure 8, reversing the latent slots leaves same-video retrieval scores essentially unchanged. Shared scenes, people, and objects may already be enough to retrieve a clip from the same video. Whether the trained model relies on that evidence to answer questions remains open.

We thank the reviewer for the follow-up. We clarify the supervision controls, source-video fidelity, and temporal reasoning.

Q1. Contribution Beyond Evidence Selection and Teacher Guidance

Our keyframe-supervision baseline uses multiple question-relevant frames selected from the original video, rather than a single frame. Their features serve as latent targets. Single-keyframe conditioning applies only to Wan generation. We apologize for the ambiguity.

To test teacher reasoning separately, we evaluate teacher-guided source-frame supervision. The same teacher CoT used for STORM generation selects relevant original-video frames, without generating new content. Their features supervise the student with matched training data and budget.

Supervision sourceVideoMMEMVBenchTempCompass
Uniform original frames53.255.471.5
Multiple relevant keyframes58.760.272.4
Teacher-guided source frames55.757.673.0
Thought video (STORM)61.061.174.3

STORM exceeds teacher-guided frames by 5.3/3.5/1.3 points and multiple-keyframe supervision by 2.3/0.9/1.9 points on VideoMME/MVBench/TempCompass. Teacher-guided frame selection is lower than the keyframe baseline on VideoMME and MVBench, though higher on TempCompass. Thus, using teacher CoT to select existing source frames does not reproduce STORM's gains.

These results support the complete teacher-conditioned thought-video supervision pipeline.

Q2. Temporal Consistency and Event Omissions

We compare 100 randomly sampled thought videos with their source videos. All rates use the same 100-video sample.

CriterionPassed/totalPass rate
Action consistency95/10095%
Event-order and motion consistency96/10096%
Absence of answer-relevant hallucinations98/10098%
Answer-critical event coverage99/10099%

Temporal consistency assesses question-relevant event order, motion, and state transitions, not pixel-level matching. The 96% rate is measured per clip.

Because absence of hallucinations does not imply complete evidence coverage, we assess answer-critical event coverage separately: a thought video fails if it omits any source-video event necessary to answer the question, even if remaining events are coherent and hallucination-free. 99/100 preserve all answer-critical events. This separates omissions from hallucinations.

Q3. Does Temporal Information Affect Answer Prediction?

Figure 8's same-video retrieval assesses visual grounding, since shared scene semantics may suffice even when temporal order changes. We do not treat retrieval alone as proof of temporal reasoning.

To test the causal role of temporal information, we conduct a controlled intervention on 100 temporal-order and motion-direction examples. We reverse the order of sampled frames from the same source video, preserving visual content while altering temporal relationships. We extract latent trajectories from original and reversed inputs, then swap trajectories while holding the original video, question, answer choices, and pre-latent KV cache fixed. An order-preserving control resamples nearby frames without changing their temporal order.

We measure answer-change rate (relative to native predictions) and counterfactual adoption (selection of the reversed-order answer).

Latent trajectoryAnswer changeCounterfactual adoption
Native0%0%
Order-preserving control3%2%
Temporal reversal80%68%

Temporal reversal changes 80% of predictions versus 3% for the order-preserving control. 68% adopt the counterfactual answer versus 2% for the control, and 85% (68/80) of changed predictions adopt the expected counterfactual rather than an unrelated choice. With the original inference context fixed, these results provide causal evidence that temporally altered latent trajectories affect answer prediction, beyond generic frame-resampling perturbations.

Together, these experiments distinguish visual grounding from the causal contribution of temporal information to answer generation. We thank the reviewer for the suggestions.

Reviewer_Laa8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STORM trains video-language models to reason through a short sequence of continuous latent states. It uses selected keyframes and teacher-generated reasoning plans to generate “thought videos”, then pools their visual features into targets for the student model’s latent states. The first training stage combines latent-state alignment with answer supervision. The second stage removes the alignment loss and continues training with answer supervision alone. At inference, the model performs up to eight latent reasoning steps before generating an answer, without generating videos or invoking external visual tools. The authors evaluate STORM on six video benchmarks using Qwen2.5-VL-7B, with additional experiments on Qwen2.5-VL-3B.

Critical strengths
  1. This paper keeps the inference pipeline simple. Its use of generated videos as training supervision allows it to benefit from intermediate visual evidence without requiring a video generator or external visual tools at test time.

  2. STORM improves performance on all six benchmarks, covering both general video understanding and more demanding video reasoning. The ablations further support the two-stage training design, with the full method outperforming either stage alone.

Critical weaknesses
  1. The comparison with standard SFT is insufficient. While the paper states that the 7B SFT baseline used the same training data, it doesn't explain whether its total training steps match those of the two-stage training STORM. Furthermore, Table 3 only compares the original Qwen2.5-VL-3B-Instruct with STORM, lacking a 3B SFT baseline trained on the same data, making it difficult to distinguish whether the performance improvement comes from ordinary fine-tuning or the design of the proposed method.

  2. The evaluation of inference efficiency is incomplete. Table 4 provides latency comparisons with LongVT and Video-Thinker, but lacks conditional speed tests with the original model's direct answer, the SFT model's direct answer, and text CoT inference. STORM performs up to 8 hidden state inference steps before generating the answer, the paper doesn't separately report the time spent on this part and its proportion of the overall inference time. Therefore, the additional overhead of this inference and the trade-off between accuracy improvement and inference latency are currently unclear.

Questions for authors
  1. Please clarify whether STORM and the SFT baseline were compared under matched training settings. A fair comparison should use the same training data, effective batch size, and total number of training steps, counting both stages of STORM.
  2. Please add a 3B SFT baseline to Table 3 using the same training data and a matched training budget. This would help distinguish the gains from STORM from those of ordinary fine-tuning.
  3. For the efficiency evaluation, please compare STORM with three baselines: direct answering with the original model, direct answering with the SFT model, and textual CoT. Measure inference latency on the same hardware with the same batch size and video inputs. Use the same timing boundaries for all methods and specify whether video preprocessing, visual encoding, and answer generation are included. Besides, report the latent rollout time and its share of total inference time, including results for a full eight-step rollout.
  4. Regarding data quality, Appendix C notes that errors in keyframe selection or video generation may affect performance. Please describe any manual checks or filtering of the selected keyframes and generated thought videos, and explain how inaccurate samples were handled.
3 comments

We thank the reviewer for recognizing the effectiveness and inference efficiency of STORM. We address the concerns regarding training fairness, the 3B baseline, inference latency, and data quality below.

Q1. Matched Training Budget

We clarify that both STORM and the SFT baseline use the same training dataset, with answer-only supervision for SFT. STORM is trained for one epoch in Stage I and one epoch in Stage II, totaling two epochs.

ModelVideoMMEMVBenchTempCompass
SFT (1 epoch)55.460.569.9
SFT (2 epochs)56.759.671.4
STORM (Stage I + II)61.061.174.3

The effective batch size is 1, and the total number of optimizer steps is 5,000 for both two-epoch SFT and STORM, counting both training stages. These results demonstrate that STORM's improvements are not simply attributable to a longer training duration, supporting the benefit of the proposed latent supervision and two-stage training procedure.

Q2. Additional 3B SFT Baseline

We agree that comparing STORM against an SFT baseline at the 3B scale would better isolate the contribution of the proposed method.

ModelVideoMMETempCompassMMVU
Qwen2.5-VL-3B-Instruct50.955.042.0
Qwen2.5-VL-3B-SFT (matched budget)53.460.749.5
Qwen2.5-VL-3B + STORM55.467.558.5

STORM improves over the original 3B model by 4.5, 12.5, and 16.5 percentage points on the three benchmarks, respectively.

STORM improves over the trained 3B model with matched budget by 2.0, 6.8, and 9.0 percentage points on the three benchmarks, respectively.

Q3. Inference Efficiency

We agree that a controlled comparison with direct answering and textual CoT provides a more informative evaluation of inference efficiency.

We have conducted additional latency measurements using one NVIDIA H200 GPU, BF16 precision, batch size 1, and identical 32-frame video inputs. All methods are evaluated on the same 20 MMVU examples. The measurements include visual encoding, prefill, reasoning, and answer decoding, excluding model loading and disk I/O.

MethodInference-time reasoningLatency (s/item)
Original Qwen2.5-VL-7BDirect answer0.40
Qwen2.5-VL-7B-SFTDirect answer0.39
Qwen2.5-VL-7B-SFT + CoT120 text tokens (avg.)1.10
STORM8 latent states0.47

STORM achieves approximately 2.34× lower latency than textual CoT, while adding only 0.08 s/item compared with direct SFT inference. This corresponds to approximately 20.5% additional end-to-end latency.

We additionally profile the eight-step latent rollout, which takes 0.12 seconds and accounts for 26% of STORM's overall inference time.

We will clarify the measurement protocol and distinguish these controlled measurements from the cross-system runtime comparisons with LongVT and Video-Thinker in Table 4.

Q4. Thought-Video Quality and Filtering

We have previously conducted a post-hoc human evaluation of 100 randomly sampled thought videos, assessed by three independent annotators with majority voting.

Evaluation criterionPass rate
Keyframe relevance93%
CoT/source consistency89%
Temporal coherence86%
Overall usability82%

The evaluation achieves substantial inter-annotator agreement (Fleiss' κ = 0.74), indicating that most generated thought videos provide usable supervision.

We clarify that no additional rejection filtering was applied during the original data construction process. The evaluation was performed post hoc to assess supervision quality. We will include the annotation criteria and representative failure cases in the revised manuscript and discuss inaccurate generated supervision as a limitation.

We thank the reviewer again for these suggestions, which have helped us clarify the training comparisons and strengthen the experimental evaluation.

Thank you for the additional experiments and clarifications. The matched-budget SFT comparisons and controlled latency measurements substantially address my concerns about training fairness and inference overhead.

I have one remaining concern about the trajectory-swap experiment reported in your response to Reviewer GWXk. The accuracy drop from 66% to 31% shows that the latent states affect predictions. However, the donor samples also differ in visual content and correct answers, so the drop could reflect answer-related information or a mismatch between the substituted states and the target input. Could you clarify how the current evidence distinguishes these explanations and supports the claim that the latent states encode motion and event order?

I will maintain my rating of 2 for now. If this concern is satisfactorily addressed, I will raise my rating to 3 (Weak Acceptance).

We thank the reviewer for the constructive follow-up and for acknowledging that our additional SFT and latency experiments address the earlier concerns.

We agree that the original cross-video trajectory-swap experiment establishes the causal influence of latent states but does not fully distinguish temporal information from answer-related features or cross-video representation mismatch. We therefore conduct an additional controlled temporal intervention.

Controlled Temporal Intervention

We evaluate 100 temporal-order and motion-direction examples. We construct counterfactual inputs by reversing the sampled frame order within the same source video, preserving visual content while altering temporal relationships. We extract latent trajectories from the original and reversed videos, then swap them while keeping the original video, question, answer choices, and pre-latent KV cache fixed.

We also introduce an order-preserving control, which resamples nearby frames without changing temporal order, to isolate temporal sensitivity from general representation perturbations.

We measure (1) Answer-change rate, the fraction of predictions differing from the native prediction, and (2) Counterfactual-answer adoption rate, the fraction shifting specifically to the answer corresponding to reversed temporal order.

Latent trajectoryAnswer-change rateCounterfactual-answer adoption
Native0%0%
Order-preserving control3%2%
Temporal reversal80%68%

The results show a clear distinction between temporal and non-temporal perturbations. The order-preserving control changes only 3% of predictions, with 2% adopting the counterfactual answer. In contrast, temporal reversal changes 80% of predictions, with 68% adopting the counterfactual answer. Among changed predictions, 85% (68/80) shift toward the expected counterfactual answer rather than unrelated choices.

Importantly, both interventions preserve the original inference context, while temporal reversal specifically alters the frame ordering used to construct the substituted latent trajectory. The substantially higher counterfactual-answer adoption rate (68% vs. 2%) indicates that the effect is driven by temporal information rather than arbitrary representation disruption.

These results provide stronger causal evidence that STORM's latent trajectories encode temporally sensitive, answer-relevant information used during prediction.

We thank the reviewer again for the suggestions, which have strengthened our analysis of STORM's latent reasoning mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.