acceptodds
Under review as a conference paper at ICLR 2027

Encoded but Unread: Locating and Recovering Within-Event Motion in Video Language Models

Abstract

Video large language models (Video-LLMs) answer questions about a video from sparsely sampled frames that a vision encoder turns into visual tokens for a language model to read. Questions about motion, such as how long an expression is held or whether a movement pauses, can only be answered by comparing what the model sees at different moments. Whether current Video-LLMs make this comparison, where the information is lost when they do not, and what recovery costs remain unclear. We introduce WarpPairs, a diagnostic benchmark of time-warp minimal pairs: each item re-indexes the frames of a real face, generated or action clip, so that positive and negative differ only in temporal structure and the label follows exactly from the edit. Six Video-LLMs perform at or near chance on all five edit types. The loss has two regimes. Below a threshold set by the temporal receptive field of a visual token, the sampler's frame stride times the encoder's temporal merge, a local scramble is attenuated for a linear probe on the tokens and invisible to the model, and the threshold moves with the architecture and with the sampling stride as predicted. Above the threshold the motion is recoverable from the tokens but absent from the answer: a linear classifier on difference statistics of the visual tokens reaches 72–90% balanced accuracy out of fold on three video sources, whereas the language model remains at chance. Supplying the missing comparison as temporal-difference (TD) tokens, computed from the model's own visual tokens, recovers the readout at a cost that depends on where the tokens enter and on the base model: at the decoder input they cut the fine-tuning steps to a fixed criterion up to six-fold at the same final accuracy, whereas the same signal added residually to the visual tokens is left at low weight by the model and saves nothing. The recovered readout transfers to unseen actors but only partly to other edits and sources. On controlled edits, these results separate a temporal-resolution scale set by sampling and encoding from a readout failure in the language model, and show that supplying the comparison saves most where frames-only fine-tuning stays longest in a constant-answer phase.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video language models, temporal reasoning, diagnostic benchmark, minimal pairs, motion understanding, linear probing, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6BEoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the reason of why VLM can recognize an action yet miss details such as a brief pause or a change in speed. It builds WARP PAIRS by changing the timing and order of video frames while largely preserving appearance. Small edits are hard to detect after sampling and encoding. For larger edits, a trained classifier can often generate the answer from the model’s visual representations, even when the model is close to the random-guessing. The paper then adds TD tokens, which summarize changes between consecutive visual representations. These reduce the training steps needed to learn selected edits by up to sixfold, but bring limited gains in final accuracy and transfer to natural motion.

Critical strengths

The controlled edits make the failures easier to interpret. By largely keeping appearance fixed and changing temporal properties, the benchmark helps separate sensitivity to motion details from recognition of the action or the scene.

Critical weaknesses

The experiments show when small temporal edits become detectable, but do not clearly separate information missed during frame sampling from information weakened during encoding. Using more frames changes the available evidence, and comparing models that merge different numbers of frames also changes the encoder. This limits how precisely the paper locates the source of the failure.

TD adds both temporal difference statistics and a trainable module with extra tokens. Yet statistics without TD already perform similarly in the probe tests for peak hold and freeze (Table 19). Adding the TD statistics back into visual tokens does not help, but that branch stays weakly weighted. The results therefore do not fully establish whether the speedup comes from differencing, the added module, or where the information enters the model.

Questions for authors

Can you test the effect of temporal merging while keeping the sampled frames fixed? This would help distinguish losses caused by sampling from those caused by encoding several frames together.

Would the same TD module and token budget still speed up training with statistics that do not use temporal differences? Also, does adding TD statistics into visual tokens remain ineffective when the fusion strength is increased?

Reviewer_2owgOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies where fine-grained within-event motion information is lost in Video-LLMs. The authors introduce WARPPAIRS, a controlled diagnostic benchmark in which real video clips are temporally re-indexed to create minimal pairs differing only in temporal structure. The benchmark contains five edit types, including peak holds, freezes, speed changes, onset-offset asymmetry, and local frame scrambling.

The paper separates temporal failures into two regimes. First, for changes below a temporal-resolution scale determined by the sampling stride and encoder temporal merging, the temporal signal is strongly attenuated in the visual representation. Second, above this scale, temporal information can often be recovered from the visual tokens using a linear probe, while the language model itself remains near chance, suggesting an encoded-but-unread failure.

The authors further introduce temporal-difference (TD) tokens computed from differences between the model's own visual tokens. When inserted at the decoder input, these tokens can substantially reduce the number of fine-tuning steps required to recover the temporal task, particularly for Qwen2.5-VL. The paper evaluates six Video-LLMs in zero-shot settings, performs probing and fine-tuning experiments across several video sources, varies the sampling rate and architecture, and includes multiple controls, seeds, transfer experiments, and alternative motion-enhancement baselines.

Critical strengths
  1. Strong controlled diagnostic benchmark.

WARPPAIRS is a well-designed diagnostic benchmark. By constructing temporally modified versions from the same underlying real clip, the paper controls identity, appearance, background, and much of the semantic content while changing the temporal structure. This makes the benchmark substantially more informative for locating temporal failures than conventional video QA benchmarks where temporal reasoning can be confounded with semantic or visual cues.

  1. The paper goes beyond measuring failure and attempts to localize it.

The encoded-versus-decoded analysis is the strongest contribution of the paper. The authors distinguish failure caused by temporal sampling/encoding from failure of the language-model readout. The receptive-field intervention is particularly convincing because the predicted transition changes when the sampling stride is changed and differs across architectures. The probing controls, including shuffled labels, single-slot features, no-differencing features, and strictly linear differences, further strengthen this analysis.

  1. Extensive and unusually careful validation.

The paper evaluates multiple Video-LLMs and multiple video domains, includes actor/source-grouped cross-validation, several random seeds, alternative motion-enhancement approaches, different frame budgets, learning-rate/rank sweeps, held-out edit rules, and transfer to other domains. The authors also report negative results rather than hiding them, for example the inconsistent TD-token behavior on InternVL3.5 and the limited transfer to natural motion benchmarks. This makes the empirical conclusions considerably more credible.

Critical weaknesses
  1. The strongest benchmark tasks may partially measure engineered temporal artifacts rather than general motion understanding.

PEAK HOLD and FREEZE are explicitly repeated-frame detection problems, and the authors acknowledge that one of the TD statistics directly captures positions that remain unchanged. The strong TD-token result is therefore partly demonstrated on tasks for which the supplied representation contains a highly direct cue to the answer. This concern is reinforced by the much weaker transfer to near-still variants and linear deceleration, where exact repetition is removed. I would like stronger evidence that the encoded-but-unread conclusion and the TD-token advantage extend to less artifact-driven within-event motion.

  1. The headline recovery-cost result is substantially model-dependent.

The strongest result is on Qwen2.5-VL, where TD tokens reduce the steps to criterion by roughly 2x to 6x depending on the seed. On Qwen3-VL the advantage is smaller, while on InternVL3.5 the behavior is inconsistent across seeds and one TD-token run never reaches the criterion. Therefore, the claim that explicitly supplying the temporal comparison provides a substantially cheaper recovery route seems robust for Qwen2.5-VL but much less general across architectures. The association with the constant-answer phase is also correlational rather than causal, as the authors acknowledge.

  1. There is no separate validation split for the recovery experiments.

The fixed 400-item subset used to measure steps to criterion is drawn from the faces test set, and the paper explicitly states that there is no separate validation split. This same subset is repeatedly evaluated across checkpoints and is also used in learning-rate/rank and other recovery analyses. The final 808-item test additionally contains these items. Even if this does not directly train the model, using test data throughout the recovery analysis makes the reported time-to-accuracy comparison less clean than using a dedicated validation set followed by a fully untouched test set.

Minor issues:

  • In Section 4.1, the sentence "Two architecture parameters fix how many native frames one visual token summarises" is imprecise. The temporal merge factor is architectural, but the native-frame stride depends on the sampling configuration, clip duration, and frame budget rather than only on the architecture.
  • The term "temporal receptive field" may sound stronger than what is actually established. The authors correctly clarify later that R = ms predicts attenuation rather than an information-theoretic inability to represent changes. I suggest making this qualification more explicit when the term is first introduced.
  • The central methodological novelty should be framed primarily as a diagnostic framework and benchmark rather than the TD-token mechanism itself. Adjacent-frame difference representations and motion pathways are already present in related methods such as GMoT and COMET, and the paper itself appropriately describes TD tokens as a simple diagnostic instance rather than a new architectural design.
Questions for authors
  1. Could the authors report the steps-to-criterion experiment using a validation set that is completely disjoint from the final test set? This would clarify whether the reported recovery-speed differences remain unchanged without repeatedly evaluating checkpoints on test examples.

  2. How much of the TD-token advantage remains if statistics that directly identify repeated or nearly unchanged frames are removed? In particular, an ablation of the below-threshold/unchanged-position statistic on PEAK HOLD and FREEZE would help establish whether TD tokens facilitate general temporal comparison or mainly expose the specific construction of these tasks.

  3. Can the authors clarify the intended scope of the temporal receptive-field claim? Appendix F shows that the proposed scale does not predict where increasing the frame budget helps on FAVOR-Bench. Is the claim intended specifically as a property of controlled local temporal perturbations such as WARPPAIRS, rather than as a general predictor of temporal resolution on natural video?

  4. The TD-token recovery advantage is strong on Qwen2.5-VL, smaller on Qwen3-VL, and inconsistent on InternVL3.5. What property of the architecture or optimization do the authors believe determines whether TD tokens provide the

Reviewer_LFyWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces WarpPairs, a benchmark of five temporal edits created by re-indexing video frames. Across six Video-LLMs, it reports near-chance answers despite temporal information being recoverable through supervised probes. It also studies a sampling-related detectability threshold and introduces temporal-difference (TD) tokens, which reduce fine-tuning steps by two to six times on Qwen2.5-VL, with similar final accuracy but limited transfer.

Critical strengths

The controlled edits, actor/source-grouped probe evaluation, and shuffled-label controls provide credible evidence that some temporal information remains accessible despite poor model answers

Critical weaknesses

The main recovery results have limited significance beyond detecting artificial edits. Peak hold and freeze rely on exact frame repetition, and the TD features explicitly measure unchanged positions. A simple pixel-difference classifier already achieves 96% and 78% on these tasks (Table 22). TD accuracy drops to 57% and 59% on near-still variants (Table 11), and the method provides no improvement on FAVOR-Bench (Table 14). Although acknowledged, these limitations substantially weaken the case that the method recovers useful motion understanding.

The experiments do not isolate the claimed effect of token placement. Decoder-input TD adds a learned convolutional and attention-pooling pathway with 16 extra tokens, whereas fused TD uses a residual contribution initialized at only one tenth of the visual-token scale (Appendix B). Capacity, representation, and signal strength therefore change alongside placement. Without matched controls, the results show that two implementations train differently, rather than establishing why placement determines recovery cost. Reusing a test subset for checkpoint evaluation and hyperparameter sweeps, without a separate validation split, further weakens confidence in the reported efficiency gains.

Questions for authors

Clarification: Which design choices and hyperparameters were selected using the 400-item test subset? Were the reported speedups confirmed on data untouched by those choices?

Evidence supporting the current recovery claim: Does the speedup persist after removing the feature that directly measures unchanged positions?

Evidence supporting the current placement claim: Does decoder-input TD still outperform an equally sized auxiliary-token pathway using non-differenced features, and does fused TD remain ineffective when its capacity and signal scale are matched?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.