- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces TemDy, a self-supervised learning method for capturing how visual state evolves over time, given some contextual trajectory. The authors propose modeling temporal dynamics formulated as continuous angular velocity with time-aware predictors. Their architecture also uses CLS and DELTA register tokens to separate semantic and temporal dynamics information. Models are trained with this method and baselines on robot demonstrations, videos, and still images, and evaluated on reconstruction / prediction on AgiBot, EPIC-KITCHENS-100, and Something-Something V2 as well as physical reasoning on IntPhys 2 and Physion.
Critical strengths
- The results on physical reasoning look strong compared to the baselines overall.
Critical weaknesses
- The problem and method is poorly motivated and hard to follow in the introduction. For example, how is general intelligence connected to temporal state transitions? Is it common for observed frames to be provided at varying intervals in robot manipulation? How are rotations connected to temporal progression?
- It is very unclear how this work is positioned against existing work, what has been done, and what is nearly proposed here. Latent dynamics methods like DynaMo [1] / LAPO [2] and world model works like Time-Aware World Model [3] seem very related, but not discussed.
- The figures are confusing and have poor clarity. For instance, Figure 2 top left text is covered by the illustration.
[1] Z. J. Cui, H. Pan, A. Iyer, S. Haldar, L. Pinto. "DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control." NeurIPS 2024. [2] D. Schmidt, M. Jiang. "Learning to Act without Actions." ICLR 2024. [3] A. N. Nhu, S. Son, M. Lin. "Time-Aware World Model for Adaptive Prediction and Control." ICML 2025.
Questions for authors
- What do the contours in Figure 2 mean?
- Are all ViT-B training checkpoints trained on the same data in Table 2?
- Is the NMSE metric formulated in feature space or pixel space? I believe it is the former, then it seems like results can be skewed by attributes / relationships of the feature learning to the target feature space.
7 comments
Response to Reviewer mGGM (2/2)
Q2: Consistency of training data in Table 2
Yes. All models in Table 2 use the same data listed in Table 1 for training in this work and are compared at the same number of training steps. Models in the upper section are trained from scratch starting from the same random ViT-B weights. Models in the lower section resume from their respective pretrained checkpoints and continue training on the same dataset.
Q3: NMSE and potential bias toward the teacher's target space
NMSE is defined in feature space. All methods use a common DINOv3 representation space as the target, with the same readout architecture and fitting budget. The upper section of Table 2 shows that, when training starts from the same random initialization, TemDy outperforms the baselines in its group on future-state reconstruction across all three datasets. This indicates that the gains cannot be attributed solely to DINOv3 pretrained initialization and supports the retention of more dynamic information useful for state reconstruction.
Table 3 provides further validation on public physical-reasoning benchmarks: TemDy achieves the highest mean score across the seven IntPhys 2 and Physion tasks in both the from-scratch and pretrained-initialization groups. Physion evaluates contact prediction using ground-truth contact labels, while IntPhys 2 evaluates physical-plausibility recognition through classification accuracy on pairs of physically possible and impossible videos. These results further support benefits on downstream physical tasks rather than only feature fitting in the DINOv3 target space.
Thank you for the response. My remaining concerns are below.
W1: Motivation, observation intervals, and the role of rotation
The motivation in the rebuttal differs from the paper's introduction, which motivates varying observation intervals in a single sentence and does not discuss delays or real-time execution. Beyond adding this to the introduction, my concern is that the evaluation does not test this motivation. No experiment involves control, latency, or action execution. It is unclear to me how do the experiments support the claim that continuous-time modeling matters for the robot settings used as motivation?
On rotation, whether rotational trajectories actually describe state evolution in feature space does not have some support / citation or whether if this is just an intuition
W2: Positioning relative to related dynamics methods
I appreciate that you plan to add these works to the related work. However, I am still not convinced by the positioning, and it does not make sense to me why they are not compared against. None of the baselines are existing dynamics representation learning methods, so it is hard to tell whether TemDy improves over prior work in this area or only over methods not designed to model dynamics.
- DynaMo learns latent dynamics for visual representations in robot manipulation. You use robot manipulation as motivation, both in the introduction and in the rebuttal, and robot demonstrations are about a third of your training data. It does not make sense to then treat DynaMo as too different in goal to compare against.
- LAPO: the DELTA token seems very similar to a latent action. Both are compact tokens that encode state transitions and condition a forward predictor. What distinguishes DELTA from a latent action conceptually?
- Time-Aware World Model conditions dynamics prediction on the time step, which seems closely related to your goal of continuous-time prediction. It is not clear to me how TemDy differs beyond training only a vision encoder rather than a world model. It is an immediate related work that should be discussed, and ideally compared against.
W3 & Q1: Figure 2
Thank you, this is resolved. I think I just had a faulty PDF that hit the top right of Figure 2.
Q2: Consistency of training data in Table 2
Thank you, this answers my question.
Q3: NMSE and the comparison to baselines
Thank you for clarifying that NMSE is computed against shared DINOv3 ViT-L targets. This is only stated in the appendix and should be in the main text. My concern about the evaluation remains. I think this makes NMSE a faulty metric for temporal understanding. It measures how well each representation predicts DINOv3 ViT-L features through a small readout, so a method can score well simply by being more compatible with that feature space. I also notice that pretrained DINOv3 scores worse. However, it has no temporal modeling, and its features appear to be averaged before the readout, so even its Current NMSE is poor.
Response to Reviewer mGGM (1/2)
W1: Motivation, observation intervals, and the role of rotation
General intelligence in a dynamic physical world requires understanding how states change over time and using history to predict the future in support of planning and decision-making. This motivation is consistent with prior work emphasizing causal world models, intuitive physics, and predictive planning【1】【2】.
A robot arm continues moving while inference and communication delays can misalign observation updates with action execution, a real-time execution challenge also highlighted by RTC【3】. When frames are missing or observations are irregularly spaced, continuous temporal understanding can help estimate states during observation gaps and at the actual execution time, reducing reliance on stale observations.
Rotation takes place in the normalized feature space and does not require physical objects to undergo rotational motion. The observation history determines the rotation plane, and the time integral of angular velocity determines progress along the trajectory, enabling continuous-time queries while preserving feature norms.
【1】Brenden M. Lake et al. Building Machines That Learn and Think Like People. Behavioral and Brain Sciences, 2017.
【2】Meta AI. Yann LeCun on a vision to make AI systems learn and reason like animals and humans. 2022.
【3】Kevin Black, Manuel Y. Galliker, and Sergey Levine. Real-Time Execution of Action Chunking Flow Policies. 2025.
W2: Positioning relative to related dynamics methods
This is a vision-encoder study: our central question is how to incorporate temporal modeling into visual representations through self-supervised learning, providing dynamic information for downstream physical understanding and prediction. The cited works are related but differ in their goals and formulations. DynaMo【4】 also learns visual representations, focusing on latent inverse and forward dynamics for downstream imitation learning; LAPO【5】 focuses on recovering latent actions and learning policies; and Time-Aware World Model【6】 focuses on prediction and control across different time steps.
TemDy uses history-conditioned rotational trajectories and shared angular-velocity integration to turn continuous-time state reconstruction and prediction into supervision for training a visual encoder. We will add these references to the related-work discussion and clarify the connections and differences between continuous-time visual representation learning, latent-action learning, and world models for control.
【4】Zichen Jeff Cui et al. DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control. 2024.
【5】Dominik Schmidt and Minqi Jiang. Learning to Act without Actions. ICLR, 2024.
【6】Anh N. Nhu, Sanghyun Son, and Ming Lin. Time-Aware World Model for Adaptive Prediction and Control. ICML, 2025.
W3 & Q1: Meaning and clarity of Figure 2
The contours in Figure 2 conceptually illustrate the distribution of states in the latent feature space; they do not represent model-estimated probability densities or predictive uncertainty. The trajectory depicts history-conditioned latent-state evolution. We will further proofread the captions and annotations and review the layout to clarify these meanings and improve readability. We also inspected all figures in the submitted PDF, including Figure 2, and did not find the reported text occlusion.