acceptodds
Under review as a conference paper at ICLR 2027

TENSOR CLOCK: PUTTING ELAPSED TIME INSIDE VIDEO LANGUAGE MODELS

Abstract

Vision language models are now widely applied to video understanding. Questions about how fast an object moves or when an event happens, however, expose a gap: when frames are sampled non-uniformly, as a fixed frame budget requires, frame order no longer determines elapsed time, so identical frames can imply different speeds and the same motion can occur at different moments. Timestamps written into the prompt do not close this gap; a model trained with true frame-pair timestamps in its prompt still cannot tell apart rates that differ only in timing. This paper introduces Tensor Clock, an interface that lets elapsed time control the internal computation of a video language model rather than only appear in its input. Built on Qwen3.6-35B-A3B, it derives from the source timestamps of each frame pair the within-pair interval, the observation time, and the time elapsed since the previous block. These quantities set seconds-based rotary positions, drive elapsed-time decay in a measured subset of Gated DeltaNet recurrent heads, and condition the value features. Counterfactual training data, in which pixels are fixed and only the timing changes, force the answer to depend on time. Against a control trained with identical data, text timestamps, and schedule, Tensor Clock raises counterfactual rate accuracy from 0% to 95.8% and improves event-localization F1 by 0.07 and 0.03 on two event-timing tasks. Intervening on a trained model shows how the two carriers divide the work: rate comparison depends entirely on Tensor Clock, whereas event timing is carried by the text timestamps and further sharpened by Tensor Clock.

Primary area:
foundation or frontier models, including LLMs
Keywords:
video language models, physical time, temporal grounding, nonuniform sampling, adaptive frame sampling, motion rate reasoning, event localization, counterfactual evaluation, timestamps, positional encoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R6yvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Problem: Uniform frame sampling in video MLLMs cannot determine the rate of change which needs the time spent between two frames. Prior work either uses absolute time tokens or embeds timestamps in text, or uses timestamps to define RoPE embeddings - which only affect QK computation.

Approach: This paper proposes TensorClock which embeds time information (absolute time stamp and elapsed time from previous frame) at three places: (i) RoPE, (ii) into Gated DeltaNet, and (iii) value projection. The model is trained on counterfactual data in which the pixels are fixed while the timing changes.

Outcome: In uniform sampling, TensorClock retains the base model abilities but performs substantially better than a control baseline when different frames may have very different elapsed time between them. The experiments are conducted on Qwen3.6-35B-A3B.

Critical strengths
  • Making video LLMs time-aware (literal timestamp as well as time elapsed between frames) is an interesting problem to tackle.
  • The proposed method on splitting the injection of literal timestamp (into RoPE) and elapsed time (into recurrent Gated Delta Net) and both (into value projection) is a sensible choice and shown to work on the synthetically constructed videos.
  • The results (Table 2) clearly show that using this technique the model is able to pick out actual provided time informatin.
Critical weaknesses
  • Unclear practical relevance. While it is interesting to study and instill time information in a video LLM, this problem is substantially different from detecting speediness of a video [1, 2]. Here, one needs to provide actual physical timestamps for every frame. Thus, it is not obvious what sort of practical scenarios can this be helpful for. I would urge the authors to actually think about possible practical applications that demand the time difference between frames to be known. Plus, it would be good to clarify that this method cannot detect speediness of a video - i.e., if I have a video of a car passing by and suddenly the video is shown in slow motion - how can this method help in such a case to say determine the speed of the car?
  • Limited and unclear evaluation. The method is shown only on one model Qwen3.6-35B-A3B which makes it limited. Also, the evaluation setup is not clearly explained - captions of the Tables do not even specify what dataset was used, how it was constructed, etc. Plus, details on how it was (post?) trained are missing. Also, does this time-training hurt performance on other benchmarks? This is not even considered.
  • Missing baselines. At least one closed model should be evaluated on this task (since usually closed models are substantially better than open models on video tasks). Furthermore, what if you embed literal timestamps onto the frames directly [3]? This simple baseline is not considered.
  • Difficult read. Much of the paper seemed really hard to read with unnecessarily complex terminology used and often missing a natural flow.

[1] SpeedNet: Learning the Speediness in Videos", Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. CVPR 2020 [2] Seeing Fast and Slow: Learning the Flow of Time in Videos, Yen-Siang Wu, Rundong Luo, Jingsen Zhu, Tao Tu, Ali Farhadi, Matthew Wallingford, Yu-Chiang Frank Wang, Steve Marschner, Wei-Chiu Ma. ECCV 2026. [3] Number it: Temporal Grounding Videos like Flipping Manga. Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, Xu Yang. CVPR 2025.

Questions for authors
  • This method assumes you have actual physical timestamp to be provided. In case of a slowed down video, the physical time still runs at its speed - how would you provide it in such a case?
4 comments

Thank you for the detailed feedback. We will make the timestamp assumption, intended applications, and evaluation protocol more explicit.

1.Practical relevance and the source of timestamps

A direct application is video understanding after non-uniform frame sampling under a fixed input budget. A sampler may retain more observations in segments with substantial changes and fewer elsewhere. The selected frames retain their timestamps in the source video, but their positions in the input sequence no longer directly indicate elapsed time. For example, consecutive input frames may be separated by 0.1 s or 1 s, affecting the interpretation of motion speed, event duration, and intervals between events. These timestamps can be retained automatically by the sampling pipeline, without additional manual annotation. For reasoning about physical motion, we further require a known correspondence between the source-video timeline and acquisition time.

The analysis in Appendix C supports this motivation. Our motion-direction probes show that the frozen visual encoder’s ability to distinguish motion varies with within-pair displacement. A small-scale analysis across seven natural-video benchmarks further shows substantial variation in within-pair displacement under a common 64-frame sampling budget. Four benchmarks have median displacements below the range that was more readily distinguishable in the probe experiment. These observations motivate studying how observation intervals can be selected according to motion characteristics while preserving the associated timing information.

Tensor Clock addresses the subsequent question of how available timing information can enter model computation once the observations have been selected. Our counterfactual tasks hold the decoded images fixed while changing the temporal schedules and corresponding answers, allowing us to examine whether predictions respond to timing information. Temporal grounding experiments on Charades-STA and QVHighlights provide complementary evaluations on natural video. The sampling analysis motivates the setting, the controlled experiments test the use of timing information, and the grounding experiments assess transfer to natural-video tasks.

Related settings include dropped-frame video, event-triggered observations, and robot camera logs that retain acquisition timestamps. For example, a robot camera can record the acquisition time of each image, preserving the actual observation intervals even when the observation frequency varies. In streaming systems with bounded attention and recurrent memory, elapsed time can also condition state updates across observations. The controlled experiments under windowed attention in Appendix I motivate future work on extending Tensor Clock to timestamped embodied perception and streaming video understanding.

2. Slow motion: capture time versus playback time

The appropriate timeline depends on the question being asked. Locating an event within an edited video calls for playback time, whereas reasoning about the original motion rate requires capture time.

For example, an event lasting one second during capture may span five seconds after slow-motion editing. If the original capture timestamps or a known playback-to-capture mapping are available, we can assign each selected frame its corresponding capture time before passing it to the model. These timestamps are metadata associated with the observations and remain fixed regardless of how quickly a viewer watches the video.

Tensor Clock uses this supplied timeline to condition the model’s computation. This makes it complementary to SpeedNet and Seeing Fast and Slow, which study how to infer temporal scale from visual or multimodal cues. When the capture timestamps and playback-to-capture mapping are both unavailable, Tensor Clock alone cannot recover the original time scale from the edited playback timestamps.

For the reviewer’s car example, reasoning about the original motion rate therefore requires capture timestamps or a known mapping from playback to capture time. Estimating speed in physical units, such as metres per second, additionally requires spatial scale or geometric calibration. Our evaluated rate-comparison task measures whether the model can compare relative motion rates under supplied timing. Calibrated vehicle-speed estimation would require these additional inputs and a separate evaluation.

3. Training configuration and broader video performance.

Both matched models start from Qwen3.6-35B-A3B, with the vision encoder and visual merger frozen. They receive the same training examples, timestamp text, LoRA targets, optimization schedule, and training seeds.

Training data. The training mixture contains 6,740 examples from 4,660 videos, covering five complementary task families:

  • Rate comparison: 1,620 examples from rendered scenes. Counterfactual variants retain the same scene frames while changing their timestamps and the corresponding relative-rate answers.

  • Event timing: 1,920 examples from executable traces, including wall contacts, blinking events, and state transitions. The generating programs provide the event-time annotations.

  • Collision timing: 1,200 examples derived from CLEVRER, with collision annotations mapped to timestamps using the source frame rate.

  • Temporal accumulation: 1,500 examples in which observed events contribute to quantities that decay over elapsed time, providing supervision for time-dependent accumulation.

  • Retention: 500 examples included to support retention of the backbone’s existing capabilities during temporal post-training.

Table 1. Configuration of the submitted matched comparison.

ConfigurationSetting
Training data6,740 examples from 4,660 videos
LoRARank 16; scaling factor 32; dropout 0.05
Adapter targetsAttention and Gated DeltaNet projections
OptimizerAdamW; weight decay 0.1
Learning-rate schedule1e-5; cosine decay; 3% warmup
Effective batch size32
Training duration2 epochs; 422 optimizer steps
Maximum sequence length3,072 tokens
PrecisionBF16
Trainable parametersText control: 16.25M; Tensor Clock: 35.91M

Tensor Clock adds 19.66M time-conditioned value parameters to the shared 16.25M LoRA parameters. Its recurrent-decay interpolation increases linearly from the native to the time-conditioned setting over the first 211 steps. Thus, the comparison matches training data and optimization budget while explicitly accounting for the additional parameters.

To assess broader video performance, we evaluate the same checkpoints and input interface on TOC-Bench, covering 2,323 questions from 1,951 source videos.

Table 2. TOC-Bench results for the submitted training configuration.

Trained-model entries are arithmetic means over two runs. Changes are computed relative to the base model before rounding; pp denotes percentage points.

ModelAccuracy (%)Change from base (pp)Abstention (%)
Base model38.74—0.00
Text-only control37.93-0.820.17
Tensor Clock37.73-1.011.05

Tensor Clock’s average TOC-Bench accuracy is 1.01 percentage points below the base model, with an abstention rate of 1.05%. Together with the temporal-task gains above, these results quantify the observed trade-off on an independent video understanding benchmark.

4. Proprietary-model and visual-timestamp baselines.

We organize the comparison around two questions: what can a strong proprietary model achieve without task-specific training, and how does visually attaching timestamps compare with conditioning internal computation on time?

Proprietary reference. We use GPT-5.6-sol as the proprietary reference. It receives the same explicitly supplied frames, their timestamps, pair grouping, questions, and answer options as all other models. Decoding settings are fixed across conditions, and every response is scored with the same parser used for all models. Responses that cannot be parsed are counted as incorrect.

Visual-timestamp baseline. Following Number it [3], we print each frame’s timestamp in a fixed border. We use timestamps rather than frame indices, because indices alone do not specify irregular elapsed intervals. We evaluate both visual-only and text-plus-visual timing, on the proprietary model and on the Qwen3.6-35B-A3B base model. Before evaluation, we verify that the timestamps remain precise and readable after each model’s image preprocessing. If an overlay changes image geometry or token budgets, the corresponding controls are rerun under matched conditions.

Table 1. Baselines under true timing.

Rate comparison uses the original coupled schedules. Per-question accuracy covers 1,620 questions. All-timings accuracy is the percentage of 180 eligible scenes for which every timing and answer ordering is correct. Event timing (LFT-irregular) and collision timing (CLEVRER-derived) report per-video F1 with a 0.1 s tolerance. Reference rows are trained with the submitted recipe (paper Table 1).

ModelTask-specific trainingTime representationPer-question acc. (%)All-timings acc. (%)Event F1Collision F1
Qwen3.6-35B-A3B—Text52.962.220.3840.355
Qwen3.6-35B-A3B—Visual54.184.430.4340.378
Qwen3.6-35B-A3B—Text + visual53.243.430.4430.323
GPT-5.6-sol—Text74.3210.870.5120.442
GPT-5.6-sol—Visual76.7712.870.5240.431
GPT-5.6-sol—Text + visual75.5412.210.5010.403
Tensor Clock (reference)✓Text + internal timing99.2095.830.5630.474
Reviewer_dKk1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Tensor Clock, a mechanism that incorporates physical elapsed time into the internal computation of a video-language model, Qwen3.6-35B-A3B, through seconds-based rotary positions, elapsed-time-dependent Gated DeltaNet recurrence, and time-conditioned value features. Using counterfactual training data that holds visual content fixed while changing timestamps, the method achieves 95.8% all-timings rate-comparison accuracy, compared with 0% for a matched text-only control, and improves event-localization F1 by 0.071 and collision-timing F1 by 0.034. Ablations suggest that recurrent decay contributes most to the gains.

Critical strengths

The primary critical strength is the novel, architecturally grounded mechanism that integrates elapsed time directly into the recurrent decay gates and rotary position embeddings of a hybrid VLM, successfully addressing a fundamental blind spot in how non-uniformly sampled videos are processed. This is strongly supported by mechanistic intervention experiments proving that internal tensor timing, rather than textual prompt timestamps, is what genuinely enables rate reasoning.

Critical weaknesses

The most consequential weakness is the failure to establish that Tensor Clock learns the intended local-speed reasoning independently of correlated timing cues. The near-zero performance under isolated within-pair interval changes and the sensitivity to irrelevant block spacing directly challenge the interpretation of the headline result. Resolving this issue is necessary before accepting the main methodological claim.

Questions for authors

Why does rate-comparison performance collapse when the within-pair interval is varied independently while block timing is held fixed?

Reviewer_cYhEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Tensor Clock, a temporal conditioning mechanism for video understanding, particularly under irregular frame sampling. Instead of relying solely on textual timestamps, the method injects physical-time information directly into the model through attention positional encoding, Gated DeltaNet state decay, and value representations. The authors demonstrate that this approach substantially improves temporal reasoning, especially rate estimation under irregular observations, while largely preserving general video understanding performance.

Critical strengths
  1. This paper investigates an underexplored problem in video temporal reasoning. Specifically, it focuses on motion and rate reasoning under irregularly sampled video observations, highlighting the importance of physical time rather than relying solely on frame ordering.

  2. This paper presents a reasonable approach to incorporating physical-time information into the model's internal computation. By conditioning attention positions, recurrent state decay, and value representations on observation timestamps and elapsed time, the proposed method provides a systematic way to make the hybrid backbone time-aware.

Critical weaknesses
  1. Although Tensor Clock achieves substantial improvements on the counterfactual rate comparison task, the results appear to be highly dependent on the specific temporal patterns used during training. Moreover, the comparison is potentially misleading, as it presents pretrained models without task-specific fine-tuning alongside Tensor Clock, which is specifically trained on the counterfactual tasks. Although the text-only control receives matched training, the headline improvement from 0% to 95.8% is measured using a strict all-timings metric rather than standard per-question. This presentation may overstate the generality of the reported performance gains.

  2. The paper makes a rather strong claim that rate reasoning under irregular observations becomes possible only when elapsed time is incorporated into the model's internal computation. However, the results only demonstrate the effectiveness of Tensor Clock over the evaluated text-only control under a specific synthetic counterfactual setting. The failure of this control does not necessarily imply that textual timestamps or alternative temporal representations are insufficient for rate reasoning. Therefore, the evidence does not fully support the claimed necessity of the proposed mechanism, and the conclusions should be stated more cautiously.

  3. The paper would benefit from clearer writing and more intuitive explanations of the proposed method and experimental setup. Several figures and tables are difficult to interpret without repeatedly referring to the main text and appendix.

Questions for authors
  1. Could the authors provide qualitative case studies to illustrate the practical benefits of Tensor Clock? In particular, it would be helpful to show examples where the original backbone or the text-only control fails due to irregular temporal observations, while Tensor Clock succeeds by correctly leveraging physical-time information.

  2. Could the authors provide a more detailed mechanistic analysis of how Tensor Clock enables temporal reasoning? While the paper includes component ablations, these experiments mainly identify which components contribute to performance, rather than explaining how physical-time information is represented and used within the model. For example, analyzing whether the modified GDN states encode elapsed time or motion rate would provide stronger evidence for the proposed mechanism.

  3. Tensor Clock achieves nearly 96% accuracy under the original coupled timing setting, but its performance drops to nearly 0% when within-pair intervals are varied independently of block timing in Appendix A. This raises concerns about whether the model learns generalizable motion-rate reasoning or relies on correlations in the training distribution. could the authors provide two additional experiments: (1) varying within-pair intervals while keeping visual content and block completion times fixed, and reporting per-question accuracy; and (2) retraining Tensor Clock with independently sampled within-pair intervals and inter-block spacing, followed by evaluation on unseen timing combinations? These experiments would help distinguish limitations of the proposed mechanism from those caused by the training data distribution.

2 comments

1.Qualitative case studies

We fixed the case-selection rule before inspecting any prediction. Within a scene, the decoded pixels must be bit-identical across all timings, and at least two timings must have different correct answers. Tensor Clock must be correct in both training runs under every timing. On the conflict items, the text-only control and the untrained backbone must all be wrong. 78 of 180 candidate scenes qualify; 32 of them have three timings. Among the scenes that do not qualify, 59 are excluded because the untrained backbone answers some conflict item correctly, and 7 because Tensor Clock itself errs. We will show one of these 7 in the appendix.

Table A. A representative scene. The within-pair displacement is 31.31 px in the first half and 32.60 px in the second. A reader that ignores time therefore answers "the same" under every timing. All systems receive identical text timestamps, written as <t_a–t_b seconds> on each block.

True timingWithin-pair intervalsSpeed ratioCorrect answerDisplacement-only readingTensor ClockText-only controlUntrained backbone
equal1.5 s, 1.5 s0.96the samethe same ✓the same / the samethe same / the samethe same
short then long1.0 s, 2.0 s1.92first halfthe same ✗first half / first halfthe same / the samethe same
long then short2.0 s, 1.0 s0.48second halfthe same ✗second half / second halfthe same / the samethe same

The text-only control does not simply default to "the same": across all items it produces the three labels in near-uniform proportion, yet matches the displacement-only reading on 90% of rows, i.e., it reads displacement and ignores the timestamps.

2.Mechanistic analysis

We analyse the mechanism through one quantity that is defined by the data. Each rate item carries the answer of a displacement-only reader: a reader that compares within-pair displacements and assumes every pair spans the same duration. This reader scores 0.556 per question, and it scores 0 by construction on the conflict items, where its answer differs from the correct one. The analysis asks: does the model depart from this reading, and when it departs, does it move toward the correct answer? We apply this question to three settings. In all three, the decoded frames are bit-identical and only the timestamps differ.

Table B. Per-question accuracy, training-style prompt, two training runs. "Timing neutralised" hides the text timestamps and replaces the tensor timing with a uniform schedule over the same span.

SettingWhat changesTensor Clock, true timingTensor Clock, timing neutralisedText-only control, true timing
A. Coupledwithin-pair interval and block timing together0.995 / 0.9910.556 / 0.5560.603 / 0.586
B. Within-pair onlywithin-pair interval; all block completion times fixed0.552 / 0.5560.556 / 0.5560.552 / 0.560
C. Inter-block onlywaiting between blocks, irrelevant to the answer0.705 / 0.7130.991 / 0.9950.927 / 0.924

The three settings separate which timing quantity the mechanism uses:

  • Setting A. The model departs from the displacement-only reading (agreement falls from 0.99 to 0.56), and the departure is toward the correct answer.
  • Setting B. The model stays on the displacement-only reading for 99.7% of rows.
  • Setting C. Changing only the waiting time between blocks moves the model's answers, even though the answers should not change.

So the trained model reads block-level timing (observation time and elapsed time between blocks), which is exactly what Tensor Clock routes into the rotary positions, the recurrent decay, and the value features (Section 3.2). Together with the carrier interventions in Table 3 of the paper, this localises the timing information the model uses to the tensor path. A representational probe of the recurrent state would complement this behavioural analysis; we will include it in the final version.

3.The two requested experiments

(1) Varying within-pair intervals with pixels and block completion times fixed.

This is Setting B in Table B, reported per question as requested. Tensor Clock scores 0.552 / 0.556, the same as with timing neutralised. It answers 0 of 288 conflict items, and 1 and 0 of 72 scenes are correct under every timing (Figure 4a). In this setting, all tensor timing inputs (observation times and elapsed times between blocks) are identical across timing variants by construction. The changed within-pair interval reaches the model only through the prompt text.

(2) Retraining with independently sampled intervals.

We retrain Tensor Clock and the text-only control with the same recipe on data in which the within-pair interval and the inter-block spacing are sampled independently (correlation of their logarithms −0.04) from continuous distributions. We then evaluate on held-out sets that each move one quantity outside the training support.

Table C. All-timings scene accuracy after decoupled retraining, two runs.

Held-out setTensor ClockText-only control
In-distribution test0.655 / 0.6070.869 / 0.821
Within-pair interval, interpolated0.619 / 0.5800.905 / 0.905
Within-pair interval, extrapolated0.433 / 0.3970.717 / 0.583
Speed ratio, interpolated0.554 / 0.5250.821 / 0.893
Inter-block gaps, extrapolated0.027 / 0.0410.288 / 0.411

Table D. Setting B conflict items after decoupled retraining.

ArmTrue timingTiming neutralised
Tensor Clock, run 10.1830.000
Tensor Clock, run 20.2210.000
Text-only control, run 10.3400.007
Text-only control, run 20.3610.010

After decoupled retraining, Tensor Clock begins to use the within-pair interval. Its agreement with the displacement-only reading falls from 0.997 to 0.91, and conflict accuracy rises from 0 to 0.20. On the same data, the matched text-only control reaches 0.35 and is higher on every held-out set. Both arms receive the within-pair interval through the same prompt text, so Tensor Clock's additional tensor pathway does not help with this quantity. This matches the design: Tensor Clock routes the observation time and the elapsed time between blocks into the tensor operations, while the within-pair interval reaches the model only through the text. We will state this explicitly in Sections 3.2 and 4.3 and in the limitations, and revise the claim to: Tensor Clock makes the model use physical time carried by block-level timing; per-pair local speed estimation additionally requires routing the within-pair interval into the tensor operations, which is the direct next step for the mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.