- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the separation of camera and object motion in video representations. The authors propose a Structured Motion Model (SMM) that learns primary and residual motion tokens from frozen image features through two prediction stages. Experiments on ProbeMotion show improvements over frozen backbone descriptors on most tasks and competitive performance with larger supervised geometry models on several motion and action probes.
Critical strengths
- This paper aims to learn how to separate camera and object motion, which is a challenging yet important problem in self-supervised video representation learning.
- The experiments are relatively solid, with strong geometry baselines, component ablations, and qualitative analyses.
Critical weaknesses
- My main concern is the quantitative evidence for separating camera and object motion in real-world videos. Although the Kubric results are encouraging and Fig. 3 provides qualitative support, quantitatively evaluating both components in real videos where camera and object motion occur together would strengthen the evidence.
- Tab. 2 reports 2.3% accuracy for AVG-pool on SSv2-110k, while Tab. 3 reports 13.3% with seven frames. Although SMM still performs better with matched temporal context in Tab. 3, the reason for this baseline difference is unclear. The authors should specify the temporal context used for each method in Tab. 2 and explain this difference.
- The self-supervised comparison mainly relies on frozen DINOv2 model and DeltaTok. Comparing with a representative self-supervised video encoder under the same probing protocol would help clarify how SMM compares with established video representations.
- The related work could better cover relevant studies on video representation learning, including self-supervised video encoders [1,2] and training image encoders on videos[3,4,5,6].
[1] Jang H, et al. Visual representation learning with stochastic frame prediction. In ICML, 2024. [2] Liu Y, et al. When the future becomes the past: Taming temporal correspondence for self-supervised video representation learning. In CVPR, 2025. [3] Jabri A, Owens A, Efros A. Space-time correspondence as a contrastive random walk. In NeurIPS, 2020. [4] Shrivastava A, Owens A. Self-supervised any-point tracking by contrastive random walks. In ECCV, 2024. [5] Liu Y, et al. From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning. In CVPR, 2026. [6] Bagad P N, Zisserman A. Chirality in action: Time-aware video representation learning by latent straightening. In NeurIPS, 2025.
Questions for authors
Since the motion tokens are updated recurrently, I wonder whether errors can accumulate over longer sequences. Have the authors examined error propagation and its effect on motion probing performance over longer observed sequences?
5 comments
Response 2/2
Video encoders We now additionally run baselines on seven video models, TimeSformer [1], ViViT [2], MotionFormer [3], Chrono [4], VideoMAEv2 [5], V-JEPA 2.1 [6], and RVM [7] testing four temporal descriptors (global average, last, difference and concatenation), plus a CLS descriptor for models providing one. When picking the setup for each model that achieves the best average rank across benchmarks, SMM outperforms or matches the best model on 6 out of 7 benchmarks.
| Method | Kubric Cam ↓ | Kubric Obj ↓ | DL3DV ↓ | DAVIS2017 ↓ | YouTubeVOS ↓ | CameraBench ↑ | SSv2-110k ↑ |
|---|---|---|---|---|---|---|---|
| TimeSformer Base — last | 1.08 ± 0.00 | 0.83 ± 0.00 | 1.08 ± 0.00 | 0.88 ± 0.02 | 0.94 ± 0.01 | 19.7 ± 0.9 | 15.7 ± 0.2 |
| ViViT Base — concat | 0.92 ± 0.00 | 0.86 ± 0.00 | 1.08 ± 0.00 | 0.94 ± 0.07 | 0.98 ± 0.00 | 18.4 ± 0.3 | 13.7 ± 0.2 |
| MotionFormer Base — last | 1.15 ± 0.00 | 0.88 ± 0.00 | 1.08 ± 0.00 | 0.98 ± 0.03 | 0.95 ± 0.00 | 17.4 ± 0.1 | 13.4 ± 0.2 |
| Chrono ViT-B — concat | 0.89 ± 0.00 | 0.64 ± 0.00 | 1.06 ± 0.00 | 1.01 ± 0.00 | 0.93 ± 0.00 | 12.9 ± 0.7 | 3.2 ± 0.2 |
| VideoMAEv2 Base (paired) — concat | 0.57 ± 0.00 | 0.69 ± 0.00 | 1.01 ± 0.00 | 0.97 ± 0.03 | 0.94 ± 0.00 | 20.8 ± 1.8 | 15.6 ± 0.1 |
| V-JEPA 2.1 — diff | 0.30 ± 0.00 | 0.39 ± 0.00 | 0.77 ± 0.00 | 0.70 ± 0.01 | 0.82 ± 0.00 | 34.7 ± 0.8 | 17.8 ± 0.2 |
| RVM Base — concat | 0.24 ± 0.00 | 0.21 ± 0.00 | 0.71 ± 0.00 | 0.93 ± 0.02 | 0.85 ± 0.00 | 52.6 ± 2.2 | 22.7 ± 0.2 |
| SMM (ours) | 0.16 ± 0.01 | 0.19 ± 0.01 | 0.59 ± 0.00 | 0.87 ± 0.05 | 0.71 ± 0.02 | 52.7 ± 1.2 | 23.1 ± 1.3 |
[1] Bertasius et al. “Is Space-Time Attention All You Need for Video Understanding?”. ICML. 2021.
[2] Arnab et al. “ViViT: A video vision transformer”. ICCV. 2021.
[3] Patrick et al. “Keeping your eye on the ball: Trajectory attention in video transformers”. NeurIPS. 2021.
[4] Kim et al. “Exploring temporally-aware features for point tracking”. CVPR. 2025.
[5] Wang et al. “VideoMAE V2: Scaling video masked autoencoders with dual masking”. CVPR. 2023.
[6] Mur-Labadia et al. “V-JEPA 2.1: Unlocking dense features in video self-supervised learning”. ECCV. 2026.
[7] Zoran et al. "Recurrent video masked autoencoders". CVPR. 2026.
Additional related work While our work builds on pretrained image representations rather than directly addressing general video representation learning or tracking, we will broaden our related work discussion to include self-supervised video representation learning (Jang et al., 2024; Liu et al., 2025), learning temporal correspondences and tracking from videos (Jabri et al., 2020; Shrivastava & Owens, 2024), and adapting pretrained image representations using videos (Bagad & Zisserman, 2025; Liu et al., 2026).
Error propagation Across the tested sequence lengths (up to seven frames), probing performance remains stable or improves with additional context (see Table 13 and 14), suggesting that recurrent updates do not introduce a measurable degradation in downstream performance. A possible explanation is that SMM updates its recurrent motion tokens using the currently observed feature pairs rather than using an autoregressive rollout. When qualitatively testing extrapolation (see Figure 4), which is however not the focus of this work, we did see errors when applying the same token repeatedly (ll.405f.).
Thank you for the detailed response. The additional experiments and clarifications address my main concerns. I prefer to raise my score and would like to read the revised manuscript first.
Response 1/2
We thank the reviewer for the positive assessment and particularly for pointing out relevant related work. We will address the individual points below. In particular, we evaluate simultaneous camera and object motion on real-world nuScenes, confirming the token specialisation observed on Kubric. We also compare SMM against seven video encoders, which SMM outperforms overall. We will include those additions in our paper.
Real-world videos We have now added an evaluation based on nuScenes [1], an autonomous driving dataset with annotated camera and object positions. We filter the dataset to retain scenes with independent object motion resulting in 438 clips from which we construct training, validation, and test splits of 273, 90, and 75 clips each using disjoint driving logs. We train linear regression probes on frozen representations following Appendix A.5 and predict planar camera motion, forward and lateral translation as well as yaw rotation, and 3D object translation. The results follow the same pattern as presented in the paper (Table 5), p is substantially better on camera motion while r performs better on object translation. Concatenating p and r performs similarly suggesting that task-relevant information is primarily encoded in one of the two tokens (see ll.321ff.).
To contextualise the performance we have also evaluated the baselines shown in Table 2 as well as VGGT as a representative of the strongly supervised geometric models on nuScenes. The same trend as seen across ProbeMotion holds for nuScenes, SMM outperforms all baselines and is competitive with VGGT’s representations.
[1] Caesar et al. "nuScenes: A multimodal dataset for autonomous driving". CVPR. 2020.
Reported baseline performance For Table 2, we use our standard DINOv2 baseline based on a single frame pair (T=2), while the 13.3% in Table 3 corresponds to the multi-frame variant (m.f.), which averages frame-pair descriptors over time (ll. 246–247), using T=7 frames. Table 8 reports both single-pair and multi-frame variants for all DINOv2 descriptors across benchmarks. We will make this distinction more explicit.