- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Problem: Uniform frame sampling in video MLLMs cannot determine the rate of change which needs the time spent between two frames. Prior work either uses absolute time tokens or embeds timestamps in text, or uses timestamps to define RoPE embeddings - which only affect QK computation.
Approach: This paper proposes TensorClock which embeds time information (absolute time stamp and elapsed time from previous frame) at three places: (i) RoPE, (ii) into Gated DeltaNet, and (iii) value projection. The model is trained on counterfactual data in which the pixels are fixed while the timing changes.
Outcome: In uniform sampling, TensorClock retains the base model abilities but performs substantially better than a control baseline when different frames may have very different elapsed time between them. The experiments are conducted on Qwen3.6-35B-A3B.
Critical strengths
- Making video LLMs time-aware (literal timestamp as well as time elapsed between frames) is an interesting problem to tackle.
- The proposed method on splitting the injection of literal timestamp (into RoPE) and elapsed time (into recurrent Gated Delta Net) and both (into value projection) is a sensible choice and shown to work on the synthetically constructed videos.
- The results (Table 2) clearly show that using this technique the model is able to pick out actual provided time informatin.
Critical weaknesses
- Unclear practical relevance. While it is interesting to study and instill time information in a video LLM, this problem is substantially different from detecting speediness of a video [1, 2]. Here, one needs to provide actual physical timestamps for every frame. Thus, it is not obvious what sort of practical scenarios can this be helpful for. I would urge the authors to actually think about possible practical applications that demand the time difference between frames to be known. Plus, it would be good to clarify that this method cannot detect speediness of a video - i.e., if I have a video of a car passing by and suddenly the video is shown in slow motion - how can this method help in such a case to say determine the speed of the car?
- Limited and unclear evaluation. The method is shown only on one model Qwen3.6-35B-A3B which makes it limited. Also, the evaluation setup is not clearly explained - captions of the Tables do not even specify what dataset was used, how it was constructed, etc. Plus, details on how it was (post?) trained are missing. Also, does this time-training hurt performance on other benchmarks? This is not even considered.
- Missing baselines. At least one closed model should be evaluated on this task (since usually closed models are substantially better than open models on video tasks). Furthermore, what if you embed literal timestamps onto the frames directly [3]? This simple baseline is not considered.
- Difficult read. Much of the paper seemed really hard to read with unnecessarily complex terminology used and often missing a natural flow.
[1] SpeedNet: Learning the Speediness in Videos", Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. CVPR 2020 [2] Seeing Fast and Slow: Learning the Flow of Time in Videos, Yen-Siang Wu, Rundong Luo, Jingsen Zhu, Tao Tu, Ali Farhadi, Matthew Wallingford, Yu-Chiang Frank Wang, Steve Marschner, Wei-Chiu Ma. ECCV 2026. [3] Number it: Temporal Grounding Videos like Flipping Manga. Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, Xu Yang. CVPR 2025.
Questions for authors
- This method assumes you have actual physical timestamp to be provided. In case of a slowed down video, the physical time still runs at its speed - how would you provide it in such a case?
4 comments
2. Slow motion: capture time versus playback time
The appropriate timeline depends on the question being asked. Locating an event within an edited video calls for playback time, whereas reasoning about the original motion rate requires capture time.
For example, an event lasting one second during capture may span five seconds after slow-motion editing. If the original capture timestamps or a known playback-to-capture mapping are available, we can assign each selected frame its corresponding capture time before passing it to the model. These timestamps are metadata associated with the observations and remain fixed regardless of how quickly a viewer watches the video.
Tensor Clock uses this supplied timeline to condition the model’s computation. This makes it complementary to SpeedNet and Seeing Fast and Slow, which study how to infer temporal scale from visual or multimodal cues. When the capture timestamps and playback-to-capture mapping are both unavailable, Tensor Clock alone cannot recover the original time scale from the edited playback timestamps.
For the reviewer’s car example, reasoning about the original motion rate therefore requires capture timestamps or a known mapping from playback to capture time. Estimating speed in physical units, such as metres per second, additionally requires spatial scale or geometric calibration. Our evaluated rate-comparison task measures whether the model can compare relative motion rates under supplied timing. Calibrated vehicle-speed estimation would require these additional inputs and a separate evaluation.
3. Training configuration and broader video performance.
Both matched models start from Qwen3.6-35B-A3B, with the vision encoder and visual merger frozen. They receive the same training examples, timestamp text, LoRA targets, optimization schedule, and training seeds.
Training data. The training mixture contains 6,740 examples from 4,660 videos, covering five complementary task families:
-
Rate comparison: 1,620 examples from rendered scenes. Counterfactual variants retain the same scene frames while changing their timestamps and the corresponding relative-rate answers.
-
Event timing: 1,920 examples from executable traces, including wall contacts, blinking events, and state transitions. The generating programs provide the event-time annotations.
-
Collision timing: 1,200 examples derived from CLEVRER, with collision annotations mapped to timestamps using the source frame rate.
-
Temporal accumulation: 1,500 examples in which observed events contribute to quantities that decay over elapsed time, providing supervision for time-dependent accumulation.
-
Retention: 500 examples included to support retention of the backbone’s existing capabilities during temporal post-training.
Table 1. Configuration of the submitted matched comparison.
| Configuration | Setting |
|---|---|
| Training data | 6,740 examples from 4,660 videos |
| LoRA | Rank 16; scaling factor 32; dropout 0.05 |
| Adapter targets | Attention and Gated DeltaNet projections |
| Optimizer | AdamW; weight decay 0.1 |
| Learning-rate schedule | 1e-5; cosine decay; 3% warmup |
| Effective batch size | 32 |
| Training duration | 2 epochs; 422 optimizer steps |
| Maximum sequence length | 3,072 tokens |
| Precision | BF16 |
| Trainable parameters | Text control: 16.25M; Tensor Clock: 35.91M |
Tensor Clock adds 19.66M time-conditioned value parameters to the shared 16.25M LoRA parameters. Its recurrent-decay interpolation increases linearly from the native to the time-conditioned setting over the first 211 steps. Thus, the comparison matches training data and optimization budget while explicitly accounting for the additional parameters.
To assess broader video performance, we evaluate the same checkpoints and input interface on TOC-Bench, covering 2,323 questions from 1,951 source videos.
Table 2. TOC-Bench results for the submitted training configuration.
Trained-model entries are arithmetic means over two runs. Changes are computed relative to the base model before rounding; pp denotes percentage points.
| Model | Accuracy (%) | Change from base (pp) | Abstention (%) |
|---|---|---|---|
| Base model | 38.74 | — | 0.00 |
| Text-only control | 37.93 | -0.82 | 0.17 |
| Tensor Clock | 37.73 | -1.01 | 1.05 |
Tensor Clock’s average TOC-Bench accuracy is 1.01 percentage points below the base model, with an abstention rate of 1.05%. Together with the temporal-task gains above, these results quantify the observed trade-off on an independent video understanding benchmark.
4. Proprietary-model and visual-timestamp baselines.
We organize the comparison around two questions: what can a strong proprietary model achieve without task-specific training, and how does visually attaching timestamps compare with conditioning internal computation on time?
Proprietary reference. We use GPT-5.6-sol as the proprietary reference. It receives the same explicitly supplied frames, their timestamps, pair grouping, questions, and answer options as all other models. Decoding settings are fixed across conditions, and every response is scored with the same parser used for all models. Responses that cannot be parsed are counted as incorrect.
Visual-timestamp baseline. Following Number it [3], we print each frame’s timestamp in a fixed border. We use timestamps rather than frame indices, because indices alone do not specify irregular elapsed intervals. We evaluate both visual-only and text-plus-visual timing, on the proprietary model and on the Qwen3.6-35B-A3B base model. Before evaluation, we verify that the timestamps remain precise and readable after each model’s image preprocessing. If an overlay changes image geometry or token budgets, the corresponding controls are rerun under matched conditions.
Table 1. Baselines under true timing.
Rate comparison uses the original coupled schedules. Per-question accuracy covers 1,620 questions. All-timings accuracy is the percentage of 180 eligible scenes for which every timing and answer ordering is correct. Event timing (LFT-irregular) and collision timing (CLEVRER-derived) report per-video F1 with a 0.1 s tolerance. Reference rows are trained with the submitted recipe (paper Table 1).
| Model | Task-specific training | Time representation | Per-question acc. (%) | All-timings acc. (%) | Event F1 | Collision F1 |
|---|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | — | Text | 52.96 | 2.22 | 0.384 | 0.355 |
| Qwen3.6-35B-A3B | — | Visual | 54.18 | 4.43 | 0.434 | 0.378 |
| Qwen3.6-35B-A3B | — | Text + visual | 53.24 | 3.43 | 0.443 | 0.323 |
| GPT-5.6-sol | — | Text | 74.32 | 10.87 | 0.512 | 0.442 |
| GPT-5.6-sol | — | Visual | 76.77 | 12.87 | 0.524 | 0.431 |
| GPT-5.6-sol | — | Text + visual | 75.54 | 12.21 | 0.501 | 0.403 |
| Tensor Clock (reference) | ✓ | Text + internal timing | 99.20 | 95.83 | 0.563 | 0.474 |
Thank you for the detailed feedback. We will make the timestamp assumption, intended applications, and evaluation protocol more explicit.
1.Practical relevance and the source of timestamps
A direct application is video understanding after non-uniform frame sampling under a fixed input budget. A sampler may retain more observations in segments with substantial changes and fewer elsewhere. The selected frames retain their timestamps in the source video, but their positions in the input sequence no longer directly indicate elapsed time. For example, consecutive input frames may be separated by 0.1 s or 1 s, affecting the interpretation of motion speed, event duration, and intervals between events. These timestamps can be retained automatically by the sampling pipeline, without additional manual annotation. For reasoning about physical motion, we further require a known correspondence between the source-video timeline and acquisition time.
The analysis in Appendix C supports this motivation. Our motion-direction probes show that the frozen visual encoder’s ability to distinguish motion varies with within-pair displacement. A small-scale analysis across seven natural-video benchmarks further shows substantial variation in within-pair displacement under a common 64-frame sampling budget. Four benchmarks have median displacements below the range that was more readily distinguishable in the probe experiment. These observations motivate studying how observation intervals can be selected according to motion characteristics while preserving the associated timing information.
Tensor Clock addresses the subsequent question of how available timing information can enter model computation once the observations have been selected. Our counterfactual tasks hold the decoded images fixed while changing the temporal schedules and corresponding answers, allowing us to examine whether predictions respond to timing information. Temporal grounding experiments on Charades-STA and QVHighlights provide complementary evaluations on natural video. The sampling analysis motivates the setting, the controlled experiments test the use of timing information, and the grounding experiments assess transfer to natural-video tasks.
Related settings include dropped-frame video, event-triggered observations, and robot camera logs that retain acquisition timestamps. For example, a robot camera can record the acquisition time of each image, preserving the actual observation intervals even when the observation frequency varies. In streaming systems with bounded attention and recurrent memory, elapsed time can also condition state updates across observations. The controlled experiments under windowed attention in Appendix I motivate future work on extending Tensor Clock to timestamped embodied perception and streaming video understanding.