acceptodds
Under review as a conference paper at ICLR 2027

The TIME Machine: On The Power of Motion for Efficient Perception

Abstract

Video representation learning has seen tremendous progress in recent years. This has been driven by many factors, including the scale of training and the success of self-supervised models trained on next-frame prediction. While these factors have pushed the boundaries of what video models can do, they also introduce their own set of limitations. First, scaling video models can reach prohibitive costs, with recent models needing hundreds of years of video to be trained. Second, learning to predict the next frame (or embedding of the frame) naturally focuses on spatial information and as a result, video models still struggle with temporal understanding. In this paper we propose a novel approach that uses motion as a modality to alleviate both of these core issues. Given the motion in a video in the form of point tracks, we mask some of the tracks and use a masked autoencoder to reconstruct the missing tracks. This allows us to learn a representation in a self-supervised manner which we call TIME (Temporally-Informed Motion Embedding), and is focused on capturing temporal information. Also, as motion is inherently appearance-invariant, TIME needs far fewer examples to generalize well. As a result, without bells and whistles, on temporal tasks TIME performs on par with state-of-the-art models, using up to 4 orders of magnitude less training data. For general tasks, TIME can be used in combination with existing representations, and we observe that it leads to a significant improvement for V-JEPA 2, RVM and VideoMAE on standard benchmarks such as SSV2, EgoExo4D and Diving48. These results point to a new promising video paradigm for both more temporally-aware as well as more scalable models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
computer vision, self-supervised learning, point tracking, machine learning, efficiency, foundation models, motion understanding, temporal understanding, sim2real

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_v3LxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIME, a self-supervised video representation learning method based on motion trajectories. The model learns temporal representations by reconstructing masked point tracks and is trained entirely on synthetic data. The authors evaluate TIME on several benchmarks, both independently and in combination with existing video encoders, to assess its temporal understanding and generalization capabilities.

Critical strengths
  1. The paper explores learning motion representations using only point trajectories, without relying on visual appearance. Training on synthetic data provides an interesting approach to learning transferable temporal features.

  2. The experimental results show that TIME performs competitively on several temporal understanding tasks. Combining TIME with existing video encoders also improves performance on multiple benchmarks, suggesting that motion features provide complementary information.

Critical weaknesses
  1. The distinction between zero-shot transfer and supervised downstream evaluation is not sufficiently clear. The performance gains are also uneven across different tasks and video encoders.

  2. The efficiency claims do not fully account for point-track extraction, and the contribution of temporal information to the fusion gains could be better isolated. The dependence on tracking quality also raises questions about robustness in real-world scenarios.

Questions for authors
  1. TIME relies on an external point tracker at inference time. Could the authors report the computational cost of track extraction and clarify how this affects the claimed efficiency advantage over video-based models?

  2. Table 1 shows that adding TIME to V-JEPA 2 provides only a 0.22-point improvement on SSv2 and a slight decrease on Diving48. Could the authors explain why the benefits vary so much across datasets and models?

  3. Since the fusion method simply concatenates features, could the authors provide a feature-dimension-matched comparison or a control using temporally shuffled tracks to verify that the gains come from motion information rather than increased feature dimensionality?

  4. TIME is trained on clean synthetic trajectories but uses estimated trajectories on real videos. How sensitive is the model to tracking errors, occlusion failures, and different point densities?

  5. The architecture builds on masked autoencoding and factorized spatiotemporal attention used in previous work. Could the authors further clarify the main methodological novelty of TIME compared with existing motion-based representation learning methods?

2 comments

Dear reviewer v3Lx,

Thank you for your questions and suggestions.

Evaluation described as “zero-shot”

Thanks for pointing this out. We will clarify this by rewording the evaluation description as “evaluation from frozen features” throughout the paper.

Uneven model performance across datasets

Thanks for pointing out this interesting observation. There are two factors that may drive the uneven performance:

  • Temporal information is more relevant in some tasks (i.e., EgoExo4D) than others (i.e., Diving48), and therefore TIME has a stronger effect in the more temporal tasks. We measure the relevance of temporal information in a task using a proxy metric: we shuffle frames in the input video and measure the impact on accuracy. Accuracy performance drops as follows: Ego-Exo4D -45.05%, SSv2: -28.68% and Diving48: -19.24%. This confirms that Ego-Exo4D contains more temporal information while SSv2 and Diving 48 contain less. This also shows a direct correlation between the amount of temporal information in the task (EgoExo4D being the most temporal task) and the gains obtained by using TIME in that task.
  • SSv2 is included in the training set of VJEPA2 and RVM (See Appendix E), hence leaving less room for improvement in the performance.

We will include these observations in the revised paper.

Computational efficiency

Thanks for pointing this out. The focus of the proposed model is on data efficiency during learning. This is in line with the focus and analysis of the related literature (e.g., VJEPA, RVM, etc).

The computational cost of a single backward pass across models is comparable as they are in the same order of magnitude (in GFLOPs: VJEPA2: 5.6K, RVM: 2.2K, TIME: 6.6K). Hence, computational efficiency during training is driven by the amount of training data (in years of video: VJEPA2: 100, RVM: 28.5, TIME: 0.017). We will clarify and emphasize this in the revised paper.

The computational cost during inference is an interesting direction but beyond the scope of this work. TIME inference cost is comparable to that of V-JEPA2, and point tracking has higher cost (in GFLOPs: TIME: 3.3K, VJEPA2: 2.8K, CoTracker3: 6.6K). Future research could experiment with lightweight point tracking and selection of fewer points to track.

Feature-dimension-matched and temporally shuffled track comparison

Thank you for this interesting suggestion. We test the fusion method by swapping TIME features with a set of features of the same dimensionality, which are computed by passing video features through a fixed random projection followed by a ReLU. We also compare to TIME features from shuffled frames. The results are in the table below. Random video features degrade or plateau performance. Alternatively, shuffling frames in TIME suppresses performance gains for image models and reduces or completely eliminates them in video models.

SSv2 Performance
BackboneFoundation model aloneAdding random featuresAdding TIME features from shuffled framesAdding TIME features
SSv2 Acc. (%)ΔSSv2 Acc. (%)ΔSSv2 Acc. (%)Δ
CLIP (4f)18.5619.040.4828.6410.0831.5613.00
DINOv3 (4f)21.9521.960.0131.069.1134.6012.65
VideoMAEv233.0531.47-1.5836.113.0638.745.69
RVM40.0837.44-2.6439.97-0.1141.921.84
V‑JEPA249.7846.43-3.3548.06-1.7250.000.22
TIME alone---13.85-19.42-

Sensitivity to tracking quality

We perform an ablation study to test how sensitive TIME is to various tracking errors and point densities during inference. We evaluate on SSv2 using a linear probe trained on frozen features. Results can be seen in the table below. TIME features remain highly resilient. Fusing VideoMAE with TIME improves by 5.69%, while noise only decreases small levels.

Tracking error corruption methodSSv2 Acc. (%)Δ
Standalone foundation model
VideoMAEv233.05-5.69
Baseline
VideoMAEv2 + TIME38.74-
Occlusion errors
All points set as visible38.57-0.17
10% occlusion values flipped38.53-0.21
25% occlusion values flipped38.45-0.29
50% occlusion values flipped38.42-0.32
Point density
16×1638.5-0.24
8×838.11-0.63
Adding Gaussian noise to points
σ = 1 px38.14-0.60
σ = 2 px37.93-0.81
σ = 4 px37.42-1.32
σ = 8 px36.74-2.00
Random mixup of trajectory paths midway
10% of points37.77-0.97
25% of points37.71-1.03
50% of points37.47-1.27

Novelty clarification

  • As far as we are aware, TIME is the first model to be trained only on point-tracks. This allows the model to be very data efficient, achieving similar performance on temporal tasks as methods trained on orders of magnitude more data. Training on point tracks also provides a very different supervisory signal compared to video models, leading to a complementary representation which explains why adding TIME to other video models improves performance.
  • TIME is trained purely on synthetic data. Our analysis demonstrates that frozen features obtained by learning a motion representation from simulated point trajectories directly transfer to real-world downstream tasks. This brings huge flexibility for generation of training data without the need of any real-world data.
  • The most relevant parallel work that uses estimated point tracks as a supervisory signal (i.e., TrackMAE (Vandeghen et al., 2026)) requires training the entire RGB video model from scratch. Instead, TIME can be pre-trained quickly on a comparatively tiny simulation dataset and combined with several RGB models out of the box.
Reviewer_Vq2bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper proposes TIME, a motion representation model pretrained by reconstructing masked point trajectories from synthetic simulations. TIME transfers to real-world videos and complements existing image and video representations through feature concatenation.
Critical strengths
  1. Learning transferable motion representations from simple synthetic simulations is an interesting and promising direction.

  2. TIME substantially improves several frozen visual representations on Ego-Exo4D. The appendix also provides useful motion-only comparisons and component ablations.

Critical weaknesses
  1. The reported reduction in training video hours does not directly establish computational savings. Reporting end-to-end inference costs, including point tracking, would provide a clearer picture of the method’s efficiency.

  2. The CLEVRER evaluation is insufficiently specified. Neither the main text nor the PDF appendix clearly explains how collision times and counts are converted into labels, how the reported scores are computed, or how videos without collisions are handled.

  3. The temporal evaluation has limited coverage. Directional classification and synthetic collision tasks provide useful controlled tests, but offer limited evidence of temporal localization in real-world videos. Complementary evaluations on tasks such as video temporal grounding using datasets like Charades-STA, with the same grounding model evaluated with and without TIME features, would strengthen the paper.

  4. The contribution of masked trajectory pretraining could be more clearly isolated. Comparisons using the same trajectory inputs and architecture, with either no pretraining or an alternative pretraining objective under a matched data budget, would help distinguish the benefits of pretraining from those of explicit motion inputs and clarify the advantages of masked trajectory reconstruction.

  5. The gains are not consistent across datasets. Adding TIME to V-JEPA 2 gives a large improvement on Ego-Exo4D, but little benefit on SSv2 and Diving48. Claims of broad complementarity should reflect this variation.

Questions for authors
  • Could you compare RGB fusion with TIME, raw-track features, and tracker features under the same probe setup? Please also clarify the comparison with TrackMAE.
2 comments

Dear reviewer Vq2b,

Thank you for your careful review and suggestions.

Computational efficiency

Thanks for pointing this out. The focus of the proposed model is on data efficiency during learning. This is in line with the focus and analysis of the related literature (e.g., VJEPA, RVM, etc).

The computational cost of a single backward pass across models is comparable as they are in the same order of magnitude (in GFLOPs: VJEPA2: 5.6K, RVM: 2.2K, TIME: 6.6K). Hence, computational efficiency during training is driven by the amount of training data (in years of video: VJEPA2: 100, RVM: 28.5, TIME: 0.017). We will clarify and emphasize this in the revised paper.

The computational cost during inference is an interesting direction but beyond the scope of this work. TIME inference cost is comparable to that of V-JEPA2, and point tracking has higher cost (in GFLOPs: TIME: 3.3K, V-JEPA2: 2.8K, CoTracker3: 6.6K). Future research could experiment with lightweight point tracking and selection of fewer points to track.

CLEVRER

Thank you for pointing out this interesting observation. We will include the following details of CLEVRER temporal evaluation setup in the revised version:

  • We obtain ground truth labels for each task using CLEVRER annotation files. Each file contains collision events with frame numbers (i.e., 0–127) and object IDs involved.
  • We train separate probes for each task on the training set and evaluate on the test set over 5 trials.
    • “T1: Does a collision happen?”. A binary classifier predicts whether a collision happens after the video ends.
    • “T2: When does the first collision happen?”. A multiclass classifier predicts the first collision frame quarter. A total of 5 classes to account for no collision in a video.
    • “T3: Number of collisions”. A multiclass classifier predicts the total number of collisions in the video.
  • We include all videos including those that do not contain collisions (i.e., assign a “zero collisions” label).

Temporal evaluation

Thank you for this interesting suggestion. Charades-STA contains a new domain where videos are longer and untrimmed (videos are around 30 seconds long). Several research questions emerge when tracking over such long periods, including large background changes, objects appearing and re-appearing, which can affect the performance of the uniform sampling of the point-tracker in our work and potentially require a memory bank. Hence, this is a very interesting direction for future work.

Contribution of masked trajectory

Thanks for pointing this out. Appendix A.1 includes comparison to no pretraining (“Features from raw trajectories”). Our motivation to choose masked trajectories was to be able to make direct comparisons to the VideoMAE architecture and its properties. It will be interesting to explore different learning objectives in future work.

Uneven model performance across datasets

Thanks for pointing out this interesting observation. There are two factors that may drive the uneven performance:

  • Temporal information is more relevant in some tasks (i.e., Ego-Exo4D) than others (i.e., Diving48), and therefore TIME has a stronger effect in the more temporal tasks. We measure the relevance of temporal information in a task using a proxy metric: we shuffle frames in the input video and measure the impact on accuracy. Accuracy performance drops as follows: Ego-Exo4D -45.05%, SSv2: -28.68% and Diving48: -19.24%. This confirms that Ego-Exo4D contains more temporal information while SSv2 and Diving 48 contain less. This also shows a direct correlation between the amount of temporal information in the task (Ego-Exo4D being the most temporal task) and the gains obtained by using TIME in that task.
  • SSv2 is included in the training set of V-JEPA2 and RVM (See Appendix E), hence leaving less room for improvement in the performance.

We will include these observations in the revised paper.

RGB fusion with other motion architectures

We compare the fusion of multiple motion features with existing models. We test on SSv2 via a standard linear probe on frozen features. We observe that other features can hurt performance when fused with a video model, and only TIME is able to improve performance when fused with other models.

SSv2 Performance (Acc. (%))
FeaturesOnly Motion Featuresw/ VideoMAEv2w/ V-JEPA2w/ RVM
Only Video Model Features–33.0549.7840.08
Raw Trajectory Features8.6335.0948.7239.92
TAPIR backbone4.7730.6945.7936.56
CoTracker3 backbone3.8831.7447.2637.76
TIME19.4238.7450.0041.92

TrackMAE

TrackMAE (Vandeghen et al., 2026) requires pre-training the entire video model on real-world data from scratch. In addition, it relies on estimated sparse trajectories as a secondary supervisory signal. In contrast, TIME is trained purely on ground-truth trajectories from simulations without RGB signal. Furthermore, TIME uses a novel tokenization procedure to learn local and global motion representations.

We conduct a small experiment on the complementarity between TIME and TrackMAE. We use TrackMAE features (from the checkpoint in Hugging Face) and a standard linear probe. We observe that frozen TIME features lead to a significant complementary performance gain.

SSv2Ego-Exo4DDiving48
ModelAcc. (%)ΔAcc. (%)ΔAcc. (%)Δ
TrackMAE (Vandeghen et al., 2026)35.25-25.93-12.94-
TrackMAE + TIME37.632.3836.2810.3513.850.91
Reviewer_e2HqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TIME, a motion representation learned through masked reconstruction of point trajectories. The model is pretrained on synthetic Kubric trajectories and applied to real videos using tracks extracted by CoTracker3. TIME features are evaluated independently and combined with features extracted by pretrained image or video encoders for video understanding.

Critical strengths
  1. Learning transferable motion representations from synthetic trajectories is an interesting direction.
  2. Combining TIME with pretrained image and video features improves action recognition performance, particularly on Ego-Exo4D.
  3. The paper includes useful comparisons with raw trajectories and tracker features, as well as ablations on token components, training data, and probing methods.
Critical weaknesses
  1. The evaluation is incorrectly described as zero-shot. Appendix C.1 states that linear probes are trained using labeled training data from each target dataset. This supports transfer with a frozen encoder, but not zero-shot classification. The distinction should be corrected throughout the paper.
  2. The efficiency claims need clearer accounting. Table 2 compares pretraining video hours, which does not directly establish computational efficiency. Appendix G.2 discusses tracker training data, but the exact CoTracker3 checkpoint and its training history remain unclear. End-to-end inference cost, including tracking and feature extraction, is also needed to support the practical efficiency claim.
  3. The benefit of TIME varies across tasks. In Table 1, adding TIME to V-JEPA 2 gives a large improvement on Ego-Exo4D, but little change on SSv2 and Diving48. Could the authors explain this difference? Has V-JEPA 2 already captured most of the motion information provided by TIME on these datasets, or is the gain limited by the fusion method?
Questions for authors
  1. Have the authors evaluated TIME on tasks beyond classification, such as video object segmentation, to assess whether the learned motion representations generalize to other video understanding tasks?
2 comments

Dear reviewer e2Hq,

Thank you for your careful review and suggestions.

Evaluation described as “zero-shot”

Thanks for pointing this out. We will clarify this by rewording the evaluation description as “evaluation from frozen features” throughout the paper.

Computational efficiency

Thanks for pointing this out. The focus of the proposed model is on data efficiency during learning. This is in line with the focus and analysis of the related literature (e.g., VJEPA, RVM, etc).

The computational cost of a single backward pass across models is comparable as they are in the same order of magnitude (in GFLOPs: VJEPA2: 5.6K, RVM: 2.2K, TIME: 6.6K). Hence, computational efficiency during training is driven by the amount of training data (in years of video: VJEPA2: 100+, RVM: 28.5, TIME: 0.017). We will clarify and emphasize this in the revised paper.

The computational cost during inference is an interesting direction but beyond the scope of this work. TIME inference cost is comparable to that of V-JEPA2, and point tracking has higher cost (in GFLOPs: TIME: 3.3K, VJEPA2: 2.8K, CoTracker3: 6.6K). Future research could experiment with lightweight point tracking and selection of fewer points to track.

CoTracker checkpoint

We use the official checkpoint for CoTracker3 from Hugging Face Facebook Research repository as is. We will include a link to this checkpoint in the revised paper.

Uneven model performance across datasets

Thanks for pointing out this interesting observation. There are two factors that may drive the uneven performance:

  • Temporal information is more relevant in some tasks (i.e., EgoExo4D) than others (i.e., Diving48), and therefore TIME has a stronger effect in the more temporal tasks. We measure the relevance of temporal information in a task using a proxy metric: we shuffle frames in the input video and measure the impact on accuracy. Accuracy performance drops as follows: Ego-Exo4D -45.05%, SSv2: -28.68% and Diving48: -19.24%. This confirms that Ego-Exo4D contains more temporal information while SSv2 and Diving48 contain less. This also shows a direct correlation between the amount of temporal information in the task (EgoExo4D being the most temporal task) and the gains obtained by using TIME in that task.
  • SSv2 is included in the training set of VJEPA2, VideoMAEv2 and RVM (See Appendix E), hence leaving less room for improvement in the performance.

We will include these observations in the revised paper.

Additional evaluation tasks

We appreciate the suggestion. We explore TIME transfer to video segmentation and depth estimation tasks using the Kubric MOVi-E test dataset featuring camera motion and colliding household objects. We train linear probes from frozen model features using 10 splits.

  • Segmentation task. We predict labels for 1,024 uniform grid points in the clip via linear classifier. We report performance for moving and static objects separately.
  • Depth estimation task. We predict depth for 1,024 uniform grid points in the clip via linear regression on log depth. We observe that TIME features improve segmentation and depth estimation performance across all models.
MOVi-E Segmentation Performance
Frozen Model FeaturesMoving IoU ↑Moving AP ↑Object IoU ↑Object AP ↑
TIME63.085.545.174.0
DINOv357.535.098.187.0
DINOv3+TIME89.967.898.287.3
VideoMAEv262.285.078.494.5
VideoMAEv2+TIME68.989.879.995.5
V-JEPA266.388.081.396.0
V-JEPA2+TIME71.591.382.596.6
RVM68.490.086.297.9
RVM+TIME72.692.286.698.0
MOVi-E Depth Estimation Performance
Frozen Model FeaturesAbsolute Relative Error ↓Threshold Accuracy δ<1.25 ↑SI-RMSE ↓
TIME0.3010.5130.316
DINOv30.1390.8250.141
DINOv3+TIME0.1330.8420.136
VideoMAEv20.2110.6480.211
VideoMAEv2+TIME0.1990.6710.201
V-JEPA20.2010.6820.212
V-JEPA2+TIME0.1820.7180.188
RVM0.1400.8270.134
RVM+TIME0.1370.8340.132

Comments on the paper

1 comment

We would like to thank all reviewers and the area chair for their time and effort evaluating our work. We are encouraged by the positive scores. We have addressed all comments by the reviewers, including:

  • Evaluation described as “zero-shot”
    • Replaced the term “zero-shot” with “evaluation from frozen features”.
  • Computational efficiency
    • Reframed the computational efficiency as “data efficiency during learning”.
  • Uneven model performance
    • Provided two factors that explain uneven performance among datasets.
  • Additional evaluation tasks
    • Showed that TIME transfers to video segmentation and depth estimation tasks.
  • Experiment details
    • Extended CLEVRER experiment details.
    • Included CoTracker3 checkpoint details.
  • Additional ablation studies
    • Extended comparison against other motion architectures with an RGB fusion experiment.
    • Included feature-dimension-matched and temporally shuffled track comparisons.
    • Tested TIME’s sensitivity to tracking quality, occlusion, and point density.
  • Additional clarifications
    • Clarified the main methodological novelty of TIME.

Given there have been no follow-up questions, we are hopeful that there are no remaining comments or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.