- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TIME, a self-supervised video representation learning method based on motion trajectories. The model learns temporal representations by reconstructing masked point tracks and is trained entirely on synthetic data. The authors evaluate TIME on several benchmarks, both independently and in combination with existing video encoders, to assess its temporal understanding and generalization capabilities.
Critical strengths
-
The paper explores learning motion representations using only point trajectories, without relying on visual appearance. Training on synthetic data provides an interesting approach to learning transferable temporal features.
-
The experimental results show that TIME performs competitively on several temporal understanding tasks. Combining TIME with existing video encoders also improves performance on multiple benchmarks, suggesting that motion features provide complementary information.
Critical weaknesses
-
The distinction between zero-shot transfer and supervised downstream evaluation is not sufficiently clear. The performance gains are also uneven across different tasks and video encoders.
-
The efficiency claims do not fully account for point-track extraction, and the contribution of temporal information to the fusion gains could be better isolated. The dependence on tracking quality also raises questions about robustness in real-world scenarios.
Questions for authors
-
TIME relies on an external point tracker at inference time. Could the authors report the computational cost of track extraction and clarify how this affects the claimed efficiency advantage over video-based models?
-
Table 1 shows that adding TIME to V-JEPA 2 provides only a 0.22-point improvement on SSv2 and a slight decrease on Diving48. Could the authors explain why the benefits vary so much across datasets and models?
-
Since the fusion method simply concatenates features, could the authors provide a feature-dimension-matched comparison or a control using temporally shuffled tracks to verify that the gains come from motion information rather than increased feature dimensionality?
-
TIME is trained on clean synthetic trajectories but uses estimated trajectories on real videos. How sensitive is the model to tracking errors, occlusion failures, and different point densities?
-
The architecture builds on masked autoencoding and factorized spatiotemporal attention used in previous work. Could the authors further clarify the main methodological novelty of TIME compared with existing motion-based representation learning methods?
2 comments
Sensitivity to tracking quality
We perform an ablation study to test how sensitive TIME is to various tracking errors and point densities during inference. We evaluate on SSv2 using a linear probe trained on frozen features. Results can be seen in the table below. TIME features remain highly resilient. Fusing VideoMAE with TIME improves by 5.69%, while noise only decreases small levels.
| Tracking error corruption method | SSv2 Acc. (%) | Δ |
|---|---|---|
| Standalone foundation model | ||
| VideoMAEv2 | 33.05 | -5.69 |
| Baseline | ||
| VideoMAEv2 + TIME | 38.74 | - |
| Occlusion errors | ||
| All points set as visible | 38.57 | -0.17 |
| 10% occlusion values flipped | 38.53 | -0.21 |
| 25% occlusion values flipped | 38.45 | -0.29 |
| 50% occlusion values flipped | 38.42 | -0.32 |
| Point density | ||
| 16×16 | 38.5 | -0.24 |
| 8×8 | 38.11 | -0.63 |
| Adding Gaussian noise to points | ||
| σ = 1 px | 38.14 | -0.60 |
| σ = 2 px | 37.93 | -0.81 |
| σ = 4 px | 37.42 | -1.32 |
| σ = 8 px | 36.74 | -2.00 |
| Random mixup of trajectory paths midway | ||
| 10% of points | 37.77 | -0.97 |
| 25% of points | 37.71 | -1.03 |
| 50% of points | 37.47 | -1.27 |
Novelty clarification
- As far as we are aware, TIME is the first model to be trained only on point-tracks. This allows the model to be very data efficient, achieving similar performance on temporal tasks as methods trained on orders of magnitude more data. Training on point tracks also provides a very different supervisory signal compared to video models, leading to a complementary representation which explains why adding TIME to other video models improves performance.
- TIME is trained purely on synthetic data. Our analysis demonstrates that frozen features obtained by learning a motion representation from simulated point trajectories directly transfer to real-world downstream tasks. This brings huge flexibility for generation of training data without the need of any real-world data.
- The most relevant parallel work that uses estimated point tracks as a supervisory signal (i.e., TrackMAE (Vandeghen et al., 2026)) requires training the entire RGB video model from scratch. Instead, TIME can be pre-trained quickly on a comparatively tiny simulation dataset and combined with several RGB models out of the box.
Dear reviewer v3Lx,
Thank you for your questions and suggestions.
Evaluation described as “zero-shot”
Thanks for pointing this out. We will clarify this by rewording the evaluation description as “evaluation from frozen features” throughout the paper.
Uneven model performance across datasets
Thanks for pointing out this interesting observation. There are two factors that may drive the uneven performance:
We will include these observations in the revised paper.
Computational efficiency
Thanks for pointing this out. The focus of the proposed model is on data efficiency during learning. This is in line with the focus and analysis of the related literature (e.g., VJEPA, RVM, etc).
The computational cost of a single backward pass across models is comparable as they are in the same order of magnitude (in GFLOPs: VJEPA2: 5.6K, RVM: 2.2K, TIME: 6.6K). Hence, computational efficiency during training is driven by the amount of training data (in years of video: VJEPA2: 100, RVM: 28.5, TIME: 0.017). We will clarify and emphasize this in the revised paper.
The computational cost during inference is an interesting direction but beyond the scope of this work. TIME inference cost is comparable to that of V-JEPA2, and point tracking has higher cost (in GFLOPs: TIME: 3.3K, VJEPA2: 2.8K, CoTracker3: 6.6K). Future research could experiment with lightweight point tracking and selection of fewer points to track.
Feature-dimension-matched and temporally shuffled track comparison
Thank you for this interesting suggestion. We test the fusion method by swapping TIME features with a set of features of the same dimensionality, which are computed by passing video features through a fixed random projection followed by a ReLU. We also compare to TIME features from shuffled frames. The results are in the table below. Random video features degrade or plateau performance. Alternatively, shuffling frames in TIME suppresses performance gains for image models and reduces or completely eliminates them in video models.