acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised Learning of Structured Motion from Videos

Abstract

Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of motion: camera and object. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object movement from camera-induced variation. We study whether such structured motion representations can be recovered from frozen features of a pretrained image vision transformer. We propose the Structured Motion Model (SMM), which explicitly separates the dominant source of temporal change from residual motion through feature prediction, rather than representing video change with a single entangled latent or with unstructured, spatially dense transition tokens. Training combines self-supervised learning on real video with weak supervision of scene motion on synthetic Kubric data. We evaluate SMM on ProbeMotion, a new evaluation suite spanning synthetic and real videos with camera, object, and combined motion. SMM outperforms backbone baselines using global CLS or average-pooled features, and compares favorably to strongly supervised representations such as VGGT on several probes, despite using substantially weaker supervision. These results suggest that pretrained image models can be readily repurposed into structured motion representations, providing a useful inductive bias for learning and analyzing latent video motion.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation learning, motion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4aUaOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the separation of camera and object motion in video representations. The authors propose a Structured Motion Model (SMM) that learns primary and residual motion tokens from frozen image features through two prediction stages. Experiments on ProbeMotion show improvements over frozen backbone descriptors on most tasks and competitive performance with larger supervised geometry models on several motion and action probes.

Critical strengths
  1. This paper aims to learn how to separate camera and object motion, which is a challenging yet important problem in self-supervised video representation learning.
  2. The experiments are relatively solid, with strong geometry baselines, component ablations, and qualitative analyses.
Critical weaknesses
  1. My main concern is the quantitative evidence for separating camera and object motion in real-world videos. Although the Kubric results are encouraging and Fig. 3 provides qualitative support, quantitatively evaluating both components in real videos where camera and object motion occur together would strengthen the evidence.
  2. Tab. 2 reports 2.3% accuracy for AVG-pool on SSv2-110k, while Tab. 3 reports 13.3% with seven frames. Although SMM still performs better with matched temporal context in Tab. 3, the reason for this baseline difference is unclear. The authors should specify the temporal context used for each method in Tab. 2 and explain this difference.
  3. The self-supervised comparison mainly relies on frozen DINOv2 model and DeltaTok. Comparing with a representative self-supervised video encoder under the same probing protocol would help clarify how SMM compares with established video representations.
  4. The related work could better cover relevant studies on video representation learning, including self-supervised video encoders [1,2] and training image encoders on videos[3,4,5,6].

[1] Jang H, et al. Visual representation learning with stochastic frame prediction. In ICML, 2024. [2] Liu Y, et al. When the future becomes the past: Taming temporal correspondence for self-supervised video representation learning. In CVPR, 2025. [3] Jabri A, Owens A, Efros A. Space-time correspondence as a contrastive random walk. In NeurIPS, 2020. [4] Shrivastava A, Owens A. Self-supervised any-point tracking by contrastive random walks. In ECCV, 2024. [5] Liu Y, et al. From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning. In CVPR, 2026. [6] Bagad P N, Zisserman A. Chirality in action: Time-aware video representation learning by latent straightening. In NeurIPS, 2025.

Questions for authors

Since the motion tokens are updated recurrently, I wonder whether errors can accumulate over longer sequences. Have the authors examined error propagation and its effect on motion probing performance over longer observed sequences?

5 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the positive assessment and particularly for pointing out relevant related work. We will address the individual points below. In particular, we evaluate simultaneous camera and object motion on real-world nuScenes, confirming the token specialisation observed on Kubric. We also compare SMM against seven video encoders, which SMM outperforms overall. We will include those additions in our paper.

Real-world videos We have now added an evaluation based on nuScenes [1], an autonomous driving dataset with annotated camera and object positions. We filter the dataset to retain scenes with independent object motion resulting in 438 clips from which we construct training, validation, and test splits of 273, 90, and 75 clips each using disjoint driving logs. We train linear regression probes on frozen representations following Appendix A.5 and predict planar camera motion, forward and lateral translation as well as yaw rotation, and 3D object translation. The results follow the same pattern as presented in the paper (Table 5), p is substantially better on camera motion while r performs better on object translation. Concatenating p and r performs similarly suggesting that task-relevant information is primarily encoded in one of the two tokens (see ll.321ff.).

TokenCamera MSE ↓Object MSE ↓
Primary p0.36 ± 0.030.76 ± 0.02
Residual r0.78 ± 0.050.70 ± 0.03
Combined [p,r]0.42 ± 0.030.70 ± 0.02

To contextualise the performance we have also evaluated the baselines shown in Table 2 as well as VGGT as a representative of the strongly supervised geometric models on nuScenes. The same trend as seen across ProbeMotion holds for nuScenes, SMM outperforms all baselines and is competitive with VGGT’s representations.

MethodCamera 3-DoF MSE ↓Object 3-DoF MSE ↓
strongly supervised
VGGT avg. feat.0.30 ± 0.000.79 ± 0.00
VGGT cam. token0.28 ± 0.000.81 ± 0.00
baselines
DeltaTok0.70 ± 0.000.73 ± 0.00
DINOv2 AVG1.10 ± 0.010.85 ± 0.00
DINOv2 CLS1.09 ± 0.000.76 ± 0.00
SMM0.36 ± 0.030.70 ± 0.03

[1] Caesar et al. "nuScenes: A multimodal dataset for autonomous driving". CVPR. 2020.

Reported baseline performance For Table 2, we use our standard DINOv2 baseline based on a single frame pair (T=2), while the 13.3% in Table 3 corresponds to the multi-frame variant (m.f.), which averages frame-pair descriptors over time (ll. 246–247), using T=7 frames. Table 8 reports both single-pair and multi-frame variants for all DINOv2 descriptors across benchmarks. We will make this distinction more explicit.

AuthorsOpenReview ↗

Response 2/2

Video encoders We now additionally run baselines on seven video models, TimeSformer [1], ViViT [2], MotionFormer [3], Chrono [4], VideoMAEv2 [5], V-JEPA 2.1 [6], and RVM [7] testing four temporal descriptors (global average, last, difference and concatenation), plus a CLS descriptor for models providing one. When picking the setup for each model that achieves the best average rank across benchmarks, SMM outperforms or matches the best model on 6 out of 7 benchmarks.

MethodKubric Cam ↓Kubric Obj ↓DL3DV ↓DAVIS2017 ↓YouTubeVOS ↓CameraBench ↑SSv2-110k ↑
TimeSformer Base — last1.08 ± 0.000.83 ± 0.001.08 ± 0.000.88 ± 0.020.94 ± 0.0119.7 ± 0.915.7 ± 0.2
ViViT Base — concat0.92 ± 0.000.86 ± 0.001.08 ± 0.000.94 ± 0.070.98 ± 0.0018.4 ± 0.313.7 ± 0.2
MotionFormer Base — last1.15 ± 0.000.88 ± 0.001.08 ± 0.000.98 ± 0.030.95 ± 0.0017.4 ± 0.113.4 ± 0.2
Chrono ViT-B — concat0.89 ± 0.000.64 ± 0.001.06 ± 0.001.01 ± 0.000.93 ± 0.0012.9 ± 0.73.2 ± 0.2
VideoMAEv2 Base (paired) — concat0.57 ± 0.000.69 ± 0.001.01 ± 0.000.97 ± 0.030.94 ± 0.0020.8 ± 1.815.6 ± 0.1
V-JEPA 2.1 — diff0.30 ± 0.000.39 ± 0.000.77 ± 0.000.70 ± 0.010.82 ± 0.0034.7 ± 0.817.8 ± 0.2
RVM Base — concat0.24 ± 0.000.21 ± 0.000.71 ± 0.000.93 ± 0.020.85 ± 0.0052.6 ± 2.222.7 ± 0.2
SMM (ours)0.16 ± 0.010.19 ± 0.010.59 ± 0.000.87 ± 0.050.71 ± 0.0252.7 ± 1.223.1 ± 1.3

[1] Bertasius et al. “Is Space-Time Attention All You Need for Video Understanding?”. ICML. 2021.

[2] Arnab et al. “ViViT: A video vision transformer”. ICCV. 2021.

[3] Patrick et al. “Keeping your eye on the ball: Trajectory attention in video transformers”. NeurIPS. 2021.

[4] Kim et al. “Exploring temporally-aware features for point tracking”. CVPR. 2025.

[5] Wang et al. “VideoMAE V2: Scaling video masked autoencoders with dual masking”. CVPR. 2023.

[6] Mur-Labadia et al. “V-JEPA 2.1: Unlocking dense features in video self-supervised learning”. ECCV. 2026.

[7] Zoran et al. "Recurrent video masked autoencoders". CVPR. 2026.

Additional related work While our work builds on pretrained image representations rather than directly addressing general video representation learning or tracking, we will broaden our related work discussion to include self-supervised video representation learning (Jang et al., 2024; Liu et al., 2025), learning temporal correspondences and tracking from videos (Jabri et al., 2020; Shrivastava & Owens, 2024), and adapting pretrained image representations using videos (Bagad & Zisserman, 2025; Liu et al., 2026).

Error propagation Across the tested sequence lengths (up to seven frames), probing performance remains stable or improves with additional context (see Table 13 and 14), suggesting that recurrent updates do not introduce a measurable degradation in downstream performance. A possible explanation is that SMM updates its recurrent motion tokens using the currently observed feature pairs rather than using an autoregressive rollout. When qualitatively testing extrapolation (see Figure 4), which is however not the focus of this work, we did see errors when applying the same token repeatedly (ll.405f.).

Thank you for the detailed response. The additional experiments and clarifications address my main concerns. I prefer to raise my score and would like to read the revised manuscript first.

Reviewer_sf4GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article investigates under weak supervision, whether frozen image ViT features (DINOv2) can be converted into structured motion representations, thereby distinguishing between camera motion and object motion. The proposed SMM predicts the features of the next frame in two sequential stages: first, the main token p compensates for the primary changes, and then using the residual token r compensates for the remaining motion. The training data combines unannotated real videos and synthetic Kubric videos. synthetic Kubric videos provides only two binary labels, 'static scene' and 'static camera,' to fix the roles of the two tokens. This work also propose ProbeMotion, seven linear probing tasks on synthetic and real videos, used to test the encoding capability of frozen representations for camera motion, object motion, and action semantics.

Critical strengths

S1. The method is simple and low-cost. The backbone network remains frozen, and the lightweight module can be trained on 4 H100 GPUs in about 11 hours, and it can be used for different image backbone networks.

S2. ProbeMotion is reusable. It can be applied to any frozen representation and simultaneously covers camera motion, object motion, and action semantics.

S3. The ablation experiments are comprehensive, and the parameter count of the baseline models is matched, thus ruling out "model capacity" as an explanation for the source of token division and compositionality.

S4. Compositionality has quantitative support. In counterfactual retrieval, SMM achieves 58.5% at the object level, while the unstructured baseline model just reaches 16.7%.

Critical weaknesses

W1. Lack of demonstration of decomposition capabilities on real-world videos. Evaluation on real-world videos is limited to scenarios with a static camera. On the DL3DV dataset, there is virtually no functional distinction between p and r (0.59 vs. 0.62). Because each token in the compositional retrieval task is derived from videos containing only a single type of motion, the model’s ability to disentangle multiple concurrent motions havn't been tested.

W2. Performance improvements are not attributable to the proposed architecture. Multi-layer features play a significant role: using only the last layer increases the camera error on Kubric from 0.16 to 0.50. Comparisons with DeltaTok are hampered by differences in backbone network, resolution, and training data, while the resolution-matched variant only covers a portion of the baseline. Single-token contrast models using the same features are only reported on Kubric, and their performance is comparable to SMM.

W3. The combined contrast alters two factors simultaneously. The unstructured baseline is single-stage, thus removing both sequential structure and weak supervision assigned by stage. The ablation experiment, which removed individual weak labels, preserved the two-stage structure, but only evaluated it using the probe task, not the combined retrieval task.

W4. The novelty is overstated. The paper calls this decomposition “underexplored in representation learning,” but compensating for the dominant motion first and then modeling the residual is a well-established idea. The related work section does not discuss motion decomposition at all.

W5. Training only uses single-object scenes. Object motion separation along the depth direction is poor.

Questions for authors

Refer to weakness part.

3 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the positive assessment and for suggesting further, controlled comparisons. We will address the individual points below. In particular, we evaluate simultaneous camera and object motion on real-world nuScenes, confirming the token specialisation observed on Kubric. We also show that SMM outperforms all ablations on exact 3D retrieval under joint motion, and strengthen the DeltaTok comparison by retraining it on our data, including a variant using multi-layer DINOv2 features. We will include those additions in our paper.

Real-world dataset with concurrent camera/object motion (W1) We have now added an evaluation based on nuScenes [1], an autonomous driving dataset with annotated camera and object positions. We filter the dataset to retain scenes with independent object motion resulting in 438 clips from which we construct linear-probe training, validation, and test splits of 273, 90, and 75 clips each using disjoint driving logs. We follow our standard evaluation protocol (see Appendix A.5) and predict planar camera motion, forward and lateral translation as well as yaw rotation, and 3D object translation. The results follow the same pattern as presented in the paper (Table 5), primary motion token p is substantially better on camera motion while the residual token r performs better on object translation. Concatenating p and r performs similarly suggesting that task-relevant information is primarily encoded in one of the two tokens (see ll.321ff.).

TokenCamera MSE ↓Object MSE ↓
Primary p0.36 ± 0.030.76 ± 0.02
Residual r0.78 ± 0.050.70 ± 0.03
Combined [p,r]0.42 ± 0.030.70 ± 0.02

To contextualise the performance we have also evaluated the baselines shown in Table 2 as well as VGGT as a representative of the strongly supervised geometric models on nuScenes. The same trend as seen across ProbeMotion holds for nuScenes, SMM outperforms all baselines and is competitive with VGGT’s representations.

MethodCamera 3-DoF MSE ↓Object 3-DoF MSE ↓
strongly supervised
VGGT avg. feat.0.30 ± 0.000.79 ± 0.00
VGGT cam. token0.28 ± 0.000.81 ± 0.00
baselines
DeltaTok0.70 ± 0.000.73 ± 0.00
DINOv2 AVG1.10 ± 0.010.85 ± 0.00
DINOv2 CLS1.09 ± 0.000.76 ± 0.00
SMM0.36 ± 0.030.70 ± 0.03

[1] Caesar et al. "nuScenes: A multimodal dataset for autonomous driving". CVPR. 2020.

Additional retrieval task results (W1/W3) We additionally evaluate counterfactual retrieval with motion tokens extracted from videos containing simultaneous camera and object motion, including the requested ablations. SMM achieves the highest exact 3D retrieval accuracy, outperforming the next-best ablation by 12 percentage points, especially due to stronger object-motion retrieval.

Model3D Exact ↑3D Camera ↑3D Object ↑Planar Exact ↑
SMM27.0686.0531.1347.22
No static-scene supervision15.0179.2018.1421.97
No static-camera supervision3.7290.484.112.68
Joint extraction6.0696.146.235.03
Unstructured two-token3.9397.754.022.59
Chance2.7816.6716.676.25

Impact of multi-layer features (W2) While SMM’s performance deteriorates when using only final-layer features (Table 16b), access to intermediate features alone does not reproduce its performance. For example, multi-layer DINOv2 AVG pooled feature differences achieve 0.82/0.60 MSE on Kubric camera/object motion, compared to SMM’s 0.16/0.19 (Table 9).

DeltaTok comparison (W2) As DeltaTok uses a higher resolution, this generally disadvantages our method. We now additionally train DeltaTok as well as a DeltaTok variant with multi-layer DINOv2 features on our data for 200k iterations, batch size 128, and input resolution 224x224, matching SMM’s setup. SMM matches or outperforms the performance on 5 out of 7 benchmarks.

MethodKubric cam. ↓Kubric obj. ↓DL3DV ↓Static DAVIS ↓Static YTVOS ↓CameraBench ↑SSv2-110k ↑
DeltaTok + DINOv30.290.310.620.760.7447.813.9
DeltaTok + DINOv2 multi-layer0.230.190.600.950.7756.39.3
SMM (ours)0.160.190.590.870.7152.723.1
AuthorsOpenReview ↗

Response 2/2

Single-token model comparison (W2) We now also provide single-token ablation results on all ProbeMotion datasets as well as an additional single-factor retrieval evaluation on our counterfactual dataset. SMM and the single-token variant perform similarly on ProbeMotion, following the trend presented in the paper (Table 16a, ll.1177f.).

MethodKubric cam. ↓Kubric obj. ↓DL3DV ↓Static DAVIS ↓Static YTVOS ↓CameraBench ↑SSv2-110k ↑
SMM0.160.190.590.870.7152.723.1
Single-token0.190.160.560.920.7356.315.3

The main advantage of SMM is structuring, as seen in the retrieval evaluation. As the single token variant does not support separately extracting and recombining camera and object motion, we compare these methods on two subsets featuring only camera or object motion. Here, we see that while both models are able to retrieve the variant with the correct camera motion in most cases, the single-token model’s performance deteriorates drastically when looking at object motion retrieval. The structured SMM is therefore better able to capture more local object motion.

3-dofplanar
MotionCamera ↑Object ↑Camera ↑Object ↑
SMM97.065.698.691.7
Single-token98.726.999.863.4

Discussion of previous motion compensation works (W4) While we agree that coarse-to-fine motion compensation is well established, our work focuses on learning a compact primary/residual motion representation through feature prediction with weak scene-level supervision. We will discuss WildRayZer [1], which uses residuals from static-scene rendering to identify dynamic regions for novel-view synthesis, and RoMo [2], which uses motion segmentation to improve structure-from-motion. We will also include Midway [3], which, while it does not explicitly compensate for dominant motion, is a latent action model that uses iterative refinement. In contrast, SMM learns explicitly structured motion representations. On ProbeMotion, SMM outperforms Midway on 6 out of 7 benchmarks.

ModelKubric Cam ↓Kubric Obj ↓DL3DV Cam ↓DAVIS Obj ↓YTVOS Obj ↓CameraBench ↑SSv2-110k ↑
Midway0.40 ± 0.000.50 ± 0.000.89 ± 0.001.00 ± 0.070.99 ± 0.0155.2 ± 0.711.4 ± 0.3
SMM (ours)0.16 ± 0.000.19 ± 0.000.59 ± 0.000.87 ± 0.100.71 ± 0.0052.7 ± 1.223.1 ± 1.3

[1] Chen et al. “WildRayZer: Self-supervised Large View Synthesis in Dynamic Environments”. CVPR. 2026.

[2] Goli et al. “RoMo: Robust motion segmentation improves structure from motion”. ICCV. 2025.

[3] Hoang & Ren. “Midway network: Learning representations for recognition and motion from latent dynamics”. ICLR. 2026.

Training data (W5) We intentionally keep the training data simple, only providing supervision on synthetic single-object scenes, and training on unlabelled real videos (SSv2, DL3DV). We then test our model on more challenging real cases (including CameraBench, SSv2, and nuScenes (see above)) to test generalisation to more complex, real videos.

Object motion separation (W5) We investigated object motion separation along the depth direction and measured the effect of object motion along each axis on frozen DINOv2 features. At matched world-unit displacement, depth motion produces approximately four times smaller feature-space changes than horizontal or vertical motion.

Object-motion axisFeature MSE
Horizontal0.684 ± 0.276
Vertical0.767 ± 0.260
Depth0.177 ± 0.102

This suggests that the reduced sensitivity of the frozen DINOv2 backbone to depth motion is a contributing factor. As an exploratory experiment, we increased the object motion magnitude in depth direction when generating the counterfactual dataset. As expected, the impact of object depth motion on the feature map increases.

Object-depth displacementFeature MSE
1.5 (original)0.177 ± 0.102
2.00.251 ± 0.143
2.50.328 ± 0.179

Increasing depth displacement also improves object-motion retrieval, particularly for depth-moving objects, where accuracy increases by 11.7 percentage points. Overall object retrieval improves from 58.5% to 62.7%, while exact retrieval remains stable. This indicates that reduced backbone sensitivity contributes to weaker depth-motion retrieval, and that partially compensating for the measured feature-space imbalance substantially improves object recovery.

Object-depth displacementExact (%)Object (%)Depth-object (%)
1.5 (original)53.0858.5162.44
2.054.1461.6471.69
2.553.7362.6574.10

Thanks for your response and most of my concerns have been resolved. I will keep my original rate.

Reviewer_Bk2MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors of this paper study whether pretrained visual features can be leveraged to model scene motion in videos via latent motion-based feature prediction. The proposed Structured Motion Model (SMM) is a recurrent architecture that models temporal dynamics as two recurrent latent motion tokens, intuitively responsible for camera motion and object motion. SMM is compared against frame-level baselines and DeltaTok on a variety of camera / object motion benchmarks denoted as the ProbeMotion benchmark suite. Ablations also show how performance varies with more temporal context, across backbones, and when reading out from each latent motion type. They additionally study variations to the architecture like causal prediction as well as the impact of real data.

Critical strengths
  1. Motivation and writing in the introduction is clear and easy to follow.
  2. The ablations on how performance increases with additional context (Table 3) and analysis of motion token specialization (Table 5) are interesting. The visualizations of motion extrapolation and latent motion swapping are also useful additions.
Critical weaknesses
  1. Numerous latent action models use multiple latent tokens such as LAPA [1], Midway Network [2], and Moto [3]. At least they should be cited and compared against.
  2. It seems unfair to compare to works like DINOv2 which were only trained with single images. Also, it is a bit deceptive that this method emphasizes image features when the readout is on transition-based motion tokens.

[1] S. Ye et al. "Latent Action Pretraining from Videos." ICLR 2025. [2] C. Hoang, M. Ren. "Midway Network: Learning Representations for Recognition and Motion from Latent Dynamics." ICLR 2026. [3] Y. Chen et al. "Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos." ICCV 2025.

Questions for authors
  1. Is ProbeMotion a new benchmark or an aggregation of existing datasets? How were these dataets selected?
  2. How were the data mixtures determined for training?
  3. Were the baselines also trained on the same data? Otherwise, it is hard to disentangle the effects of data from the methodology.
6 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for their suggestions, especially for mentioning relevant latent action models to compare against. We will respond to the individual points below. We have added comparisons against additional latent action models, video encoders, and DeltaTok retrained on our data, which SMM outperforms overall. We also clarify the construction of ProbeMotion and provide an ablation of the training data mixture. We will incorporate these additions into the paper.

Latent action models We have included a comparison against the suggested models, which use multiple tokens. In contrast, SMM explicitly structures motion into primary and residual components. We tested two setups for each of them, last pair uses the tokens extracted between the final two frames of an evaluation sequence, multi-frame averages the representations across the sequence. For models with quantization, we probe the continuous pre-quantization representation. SMM outperforms all the new baselines on 5 out of 7 benchmarks, using only two explicitly structured motion tokens compared to their 4-10 tokens.

ModelKubric Cam ↓Kubric Obj ↓DL3DV Cam ↓DAVIS Obj ↓YTVOS Obj ↓CameraBench ↑SSv2 ↑Avg. Rank ↓
LAPA (last pair)0.54 ± 0.000.49 ± 0.000.82 ± 0.000.91 ± 0.010.81 ± 0.0038.1 ± 2.05.7 ± 0.13.71
LAPA (multi-frame)0.50 ± 0.000.47 ± 0.000.91 ± 0.001.00 ± 0.010.89 ± 0.0044.0 ± 3.510.3 ± 0.23.93
Midway (last pair)0.43 ± 0.000.50 ± 0.000.79 ± 0.000.96 ± 0.090.93 ± 0.0150.3 ± 1.46.1 ± 0.43.71
Midway (multi-frame)0.40 ± 0.000.50 ± 0.000.89 ± 0.001.00 ± 0.070.99 ± 0.0155.2 ± 0.711.4 ± 0.33.86
Moto (last pair)1.11 ± 0.000.69 ± 0.001.08 ± 0.000.78 ± 0.000.91 ± 0.0016.1 ± 0.63.3 ± 0.15.36
Moto (multi-frame)1.10 ± 0.000.69 ± 0.001.09 ± 0.000.96 ± 0.040.94 ± 0.0015.8 ± 1.13.4 ± 0.06.14
SMM (ours)0.16 ± 0.000.19 ± 0.000.59 ± 0.000.87 ± 0.100.71 ± 0.0052.7 ± 1.223.1 ± 1.31.29

As Midway also exhibits an iterative structure, we additionally probe the motion latents after its first (early) and last (final) refinement stages to investigate emergent structuring, but find no evidence of SMM-like specialisation. The fully refined (final) representation, used in the table above, outperforms the early representation across Kubric camera and object motion.

RepresentationTemporal variantKubric Cam ↓Kubric Obj ↓
EarlyLast pair0.460.55
FinalLast pair0.430.50
EarlyMulti-frame0.440.53
FinalMulti-frame0.400.50

Data mixture The dataset mixture was chosen for its simplicity, 50% of samples come from the synthetic dataset which can provide weak supervision, the rest is equally divided between the remaining two datasets. The new ablation shows a trade-off between structuring and generalisation to real videos. The setup with more synthetic samples obtains stronger structuring on Kubric but performs worse on DAVIS, CameraBench, and SSv2-110k while the ablation using fewer synthetic samples weakens token specialisation. For this balanced mixture, the primary token outperforms the residual token on both Kubric camera and object motion, indicating weaker specialisation. Overall, we consider our default 50/25/25 mixture a good compromise between motion-token specialisation and generalisation to real videos.

Mixture (Kubric/SSv2/DL3DV)Kubric cam. ↓Kubric obj. ↓DL3DV ↓DAVIS ↓YTVOS ↓CameraBench ↑SSv2-110k ↑
Default (50/25/25)0.160.190.590.870.7152.723.1
More synthetic (75/12.5/12.5)0.140.130.590.940.7150.419.7
Balanced (33.3/33.4/33.3)0.200.270.580.870.7056.024.1
Default (50/25/25)More synthetic (75/12.5/12.5)Balanced (33.3/33.4/33.3)
TokenCamera ↓Object ↓Camera ↓Object ↓Camera ↓Object ↓
Primary (p)0.160.210.140.240.200.18
Residual (r)0.280.190.350.130.260.27
AuthorsOpenReview ↗

Response 2/2

Using image models We compare against DINOv2 to show that SMM’s performance cannot be matched by naively using the backbone’s features directly (ll.259ff.). We now additionally run baselines on seven video models, TimeSformer [1], ViViT [2], MotionFormer [3], Chrono [4], VideoMAEv2 [5], V-JEPA 2.1 [6], and RVM [7] testing four temporal descriptors (global average, last, difference and concatenation), plus a CLS descriptor for models providing one. We keep the video encoders frozen and train linear probes on each descriptor, following the same evaluation protocol as for DINOv2 (Appendix A.5). When picking the setup for each model that achieves the best average rank across benchmarks, SMM outperforms or matches all other models on 6 out of 7 benchmarks.

MethodKubric Cam ↓Kubric Obj ↓DL3DV ↓DAVIS2017 ↓YouTubeVOS ↓CameraBench ↑SSv2-110k ↑
TimeSformer Base — last1.08 ± 0.000.83 ± 0.001.08 ± 0.000.88 ± 0.020.94 ± 0.0119.7 ± 0.915.7 ± 0.2
ViViT Base — concat0.92 ± 0.000.86 ± 0.001.08 ± 0.000.94 ± 0.070.98 ± 0.0018.4 ± 0.313.7 ± 0.2
MotionFormer Base — last1.15 ± 0.000.88 ± 0.001.08 ± 0.000.98 ± 0.030.95 ± 0.0017.4 ± 0.113.4 ± 0.2
Chrono ViT-B — concat0.89 ± 0.000.64 ± 0.001.06 ± 0.001.01 ± 0.000.93 ± 0.0012.9 ± 0.73.2 ± 0.2
VideoMAEv2 Base (paired) — concat0.57 ± 0.000.69 ± 0.001.01 ± 0.000.97 ± 0.030.94 ± 0.0020.8 ± 1.815.6 ± 0.1
V-JEPA 2.1 — diff0.30 ± 0.000.39 ± 0.000.77 ± 0.000.70 ± 0.010.82 ± 0.0034.7 ± 0.817.8 ± 0.2
RVM Base — concat0.24 ± 0.000.21 ± 0.000.71 ± 0.000.93 ± 0.020.85 ± 0.0052.6 ± 2.222.7 ± 0.2
SMM (ours)0.16 ± 0.010.19 ± 0.010.59 ± 0.000.87 ± 0.050.71 ± 0.0252.7 ± 1.223.1 ± 1.3

[1] Bertasius et al. “Is Space-Time Attention All You Need for Video Understanding?”. ICML. 2021.

[2] Arnab et al. “ViViT: A video vision transformer”. ICCV. 2021.

[3] Patrick et al. “Keeping your eye on the ball: Trajectory attention in video transformers”. NeurIPS. 2021.

[4] Kim et al. “Exploring temporally-aware features for point tracking”. CVPR. 2025.

[5] Wang et al. “VideoMAE V2: Scaling video masked autoencoders with dual masking”. CVPR. 2023.

[6] Mur-Labadia et al. “V-JEPA 2.1: Unlocking dense features in video self-supervised learning”. ECCV. 2026.

[7] Zoran et al. "Recurrent video masked autoencoders". CVPR. 2026.

ProbeMotion ProbeMotion is a curated evaluation suite comprising seven linear-probing tasks across synthetic and real videos. It combines newly generated Kubric sequences with existing real-world datasets, for which we construct task-specific evaluations, including filtering and new splits where needed. These datasets were selected to cover complementary motion regimes and evaluation settings, as described in Section 3.1 (ll. 203–207). We select Kubric as a controlled, synthetic dataset with simultaneous, 3D annotated camera and object motion. DL3DV is chosen as a real-world dataset with moving cameras and mostly static scenes. Our approximately static-camera subsets of DAVIS/YouTubeVOS provide real-world object motion and CameraBench is a diverse real-world camera motion benchmark. Something-Something v2 provides object-centric action semantics. Further details on Kubric sequence generation are provided in Appendix A.1, while Appendices A.2 and A.3 describe dataset filtering, evaluation splits, and sampling.

Baseline training The baselines in Table 2 are not trained on the same data but use publicly available checkpoints. All our ablations (see Table 16) use the same data as SMM, unless noted otherwise. We now additionally train DeltaTok on our data for 200k iterations, batch size 128, and input resolution 224x224, matching our SMM’s setup. We find that SMM outperforms it on 6 out of 7 benchmarks.

MethodKubric cam. ↓Kubric obj. ↓DL3DV ↓Static DAVIS ↓Static YTVOS ↓CameraBench ↑SSv2-110k ↑
DeltaTok (retrained)0.290.310.620.760.7447.8 ± 1.513.9
SMM (ours)0.160.190.590.870.7152.7 ± 1.223.1

Thank you again for your constructive feedback. We have addressed your concerns through the following additional experiments and clarifications:

  • Latent action models We have included the suggested latent action models in related work and compare against them on ProbeMotion, where SMM outperforms them on 5 out of 7 benchmarks using fewer tokens.
  • Using image models We have clarified the purpose of the DINOv2 baseline and extended our comparison to representations from seven video models, which SMM matches or outperforms on 6 out of 7 ProbeMotion tasks.
  • ProbeMotion We have clarified that ProbeMotion is a curated evaluation suite combining newly generated Kubric sequences with existing real-world datasets, including task-specific filtering and evaluation setups where needed.
  • Data mixture We have included a new ablation of different training data mixtures, showing that our simple default mixture provides a compromise between motion-token specialization and real-world generalization.
  • Baseline training We have clarified the baseline training setups and included a new DeltaTok baseline trained on our data, which SMM outperforms on 6 out of 7 ProbeMotion tasks.

We hope these additions address your concerns, please let us know if any questions remain.

Comments on the paper

1 comment

We thank the reviewers and AC for their time reviewing our work and for their constructive suggestions. Our work proposes a Structured Motion Model (SMM), which learns representations of primary and residual motion from frozen image features. Following the reviewers' suggestions, we have extended our comparisons against latent action and video representation models and added a new real-world motion estimation benchmark.

We have added comparisons against additional latent action models (LAPA, Midway, Moto) and retrained DeltaTok on our data for a more controlled comparison. SMM outperforms all newly added latent action models on five out of seven benchmarks and the retrained DeltaTok on six out of seven, further strengthening our results.

We now also evaluate seven video encoders as additional baselines. SMM matches or outperforms the strongest descriptors from these models, including RVM and V-JEPA 2.1, on six out of seven benchmarks.

We have included nuScenes to evaluate simultaneous camera and object motion on real-world videos. The results confirm the intended primary/residual token specialization, and SMM outperforms all evaluated self-supervised baselines, including RVM and V-JEPA 2.1.

In addition, we have conducted further ablations and analyses addressing the reviewers' questions and expanded the related work. We believe these additions have further strengthened our work and have uploaded the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.