acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised Continuous Dynamics Learning

Abstract

Temporal modeling is essential for general intelligence to understand change and anticipate how the world evolves. Consecutive frames in a video often share similar semantics while carrying different state information. The dynamic state-transition information between these frames is important for describing their evolution along a continuous time axis. However, conventional self-supervised methods primarily learn associations between frames, lacking explicit awareness of continuous state evolution. To address this issue, we introduce TemDy, a self-supervised continuous dynamics learning framework comprising Temporal Variation Encoding and Continuous Dynamics Modeling. Temporal Variation Encoding uses 3D rotary position encoding to incorporate spatial and temporal information. A DELTA register learns to encode state transitions through supervision from deterministic state reconstruction and uncertain future prediction. Continuous Dynamics Modeling represents normalized visual states through history-conditioned rotations, separating the direction of change from its temporal progression. Integrating continuous angular velocity from an observation-derived anchor then models temporal dynamics. Evaluations of temporal localization, frame retrieval, and state reconstruction demonstrate that TemDy has improved temporal dynamics modeling capabilities compared with diverse self-supervised baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Self-Supervised Learning, Continuous Dynamics Modeling, Temporal Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_mGGMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces TemDy, a self-supervised learning method for capturing how visual state evolves over time, given some contextual trajectory. The authors propose modeling temporal dynamics formulated as continuous angular velocity with time-aware predictors. Their architecture also uses CLS and DELTA register tokens to separate semantic and temporal dynamics information. Models are trained with this method and baselines on robot demonstrations, videos, and still images, and evaluated on reconstruction / prediction on AgiBot, EPIC-KITCHENS-100, and Something-Something V2 as well as physical reasoning on IntPhys 2 and Physion.

Critical strengths
  1. The results on physical reasoning look strong compared to the baselines overall.
Critical weaknesses
  1. The problem and method is poorly motivated and hard to follow in the introduction. For example, how is general intelligence connected to temporal state transitions? Is it common for observed frames to be provided at varying intervals in robot manipulation? How are rotations connected to temporal progression?
  2. It is very unclear how this work is positioned against existing work, what has been done, and what is nearly proposed here. Latent dynamics methods like DynaMo [1] / LAPO [2] and world model works like Time-Aware World Model [3] seem very related, but not discussed.
  3. The figures are confusing and have poor clarity. For instance, Figure 2 top left text is covered by the illustration.

[1] Z. J. Cui, H. Pan, A. Iyer, S. Haldar, L. Pinto. "DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control." NeurIPS 2024. [2] D. Schmidt, M. Jiang. "Learning to Act without Actions." ICLR 2024. [3] A. N. Nhu, S. Son, M. Lin. "Time-Aware World Model for Adaptive Prediction and Control." ICML 2025.

Questions for authors
  1. What do the contours in Figure 2 mean?
  2. Are all ViT-B training checkpoints trained on the same data in Table 2?
  3. Is the NMSE metric formulated in feature space or pixel space? I believe it is the former, then it seems like results can be skewed by attributes / relationships of the feature learning to the target feature space.
7 comments
AuthorsOpenReview ↗

Response to Reviewer mGGM (1/2)

W1: Motivation, observation intervals, and the role of rotation

General intelligence in a dynamic physical world requires understanding how states change over time and using history to predict the future in support of planning and decision-making. This motivation is consistent with prior work emphasizing causal world models, intuitive physics, and predictive planning【1】【2】.

A robot arm continues moving while inference and communication delays can misalign observation updates with action execution, a real-time execution challenge also highlighted by RTC【3】. When frames are missing or observations are irregularly spaced, continuous temporal understanding can help estimate states during observation gaps and at the actual execution time, reducing reliance on stale observations.

Rotation takes place in the normalized feature space and does not require physical objects to undergo rotational motion. The observation history determines the rotation plane, and the time integral of angular velocity determines progress along the trajectory, enabling continuous-time queries while preserving feature norms.

【1】Brenden M. Lake et al. Building Machines That Learn and Think Like People. Behavioral and Brain Sciences, 2017.

【2】Meta AI. Yann LeCun on a vision to make AI systems learn and reason like animals and humans. 2022.

【3】Kevin Black, Manuel Y. Galliker, and Sergey Levine. Real-Time Execution of Action Chunking Flow Policies. 2025.

W2: Positioning relative to related dynamics methods

This is a vision-encoder study: our central question is how to incorporate temporal modeling into visual representations through self-supervised learning, providing dynamic information for downstream physical understanding and prediction. The cited works are related but differ in their goals and formulations. DynaMo【4】 also learns visual representations, focusing on latent inverse and forward dynamics for downstream imitation learning; LAPO【5】 focuses on recovering latent actions and learning policies; and Time-Aware World Model【6】 focuses on prediction and control across different time steps.

TemDy uses history-conditioned rotational trajectories and shared angular-velocity integration to turn continuous-time state reconstruction and prediction into supervision for training a visual encoder. We will add these references to the related-work discussion and clarify the connections and differences between continuous-time visual representation learning, latent-action learning, and world models for control.

【4】Zichen Jeff Cui et al. DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control. 2024.

【5】Dominik Schmidt and Minqi Jiang. Learning to Act without Actions. ICLR, 2024.

【6】Anh N. Nhu, Sanghyun Son, and Ming Lin. Time-Aware World Model for Adaptive Prediction and Control. ICML, 2025.

W3 & Q1: Meaning and clarity of Figure 2

The contours in Figure 2 conceptually illustrate the distribution of states in the latent feature space; they do not represent model-estimated probability densities or predictive uncertainty. The trajectory depicts history-conditioned latent-state evolution. We will further proofread the captions and annotations and review the layout to clarify these meanings and improve readability. We also inspected all figures in the submitted PDF, including Figure 2, and did not find the reported text occlusion.

AuthorsOpenReview ↗

Response to Reviewer mGGM (2/2)

Q2: Consistency of training data in Table 2

Yes. All models in Table 2 use the same data listed in Table 1 for training in this work and are compared at the same number of training steps. Models in the upper section are trained from scratch starting from the same random ViT-B weights. Models in the lower section resume from their respective pretrained checkpoints and continue training on the same dataset.

Q3: NMSE and potential bias toward the teacher's target space

NMSE is defined in feature space. All methods use a common DINOv3 representation space as the target, with the same readout architecture and fitting budget. The upper section of Table 2 shows that, when training starts from the same random initialization, TemDy outperforms the baselines in its group on future-state reconstruction across all three datasets. This indicates that the gains cannot be attributed solely to DINOv3 pretrained initialization and supports the retention of more dynamic information useful for state reconstruction.

Table 3 provides further validation on public physical-reasoning benchmarks: TemDy achieves the highest mean score across the seven IntPhys 2 and Physion tasks in both the from-scratch and pretrained-initialization groups. Physion evaluates contact prediction using ground-truth contact labels, while IntPhys 2 evaluates physical-plausibility recognition through classification accuracy on pairs of physically possible and impossible videos. These results further support benefits on downstream physical tasks rather than only feature fitting in the DINOv3 target space.

Thank you for the response. My remaining concerns are below.

W1: Motivation, observation intervals, and the role of rotation

The motivation in the rebuttal differs from the paper's introduction, which motivates varying observation intervals in a single sentence and does not discuss delays or real-time execution. Beyond adding this to the introduction, my concern is that the evaluation does not test this motivation. No experiment involves control, latency, or action execution. It is unclear to me how do the experiments support the claim that continuous-time modeling matters for the robot settings used as motivation?

On rotation, whether rotational trajectories actually describe state evolution in feature space does not have some support / citation or whether if this is just an intuition

W2: Positioning relative to related dynamics methods

I appreciate that you plan to add these works to the related work. However, I am still not convinced by the positioning, and it does not make sense to me why they are not compared against. None of the baselines are existing dynamics representation learning methods, so it is hard to tell whether TemDy improves over prior work in this area or only over methods not designed to model dynamics.

  • DynaMo learns latent dynamics for visual representations in robot manipulation. You use robot manipulation as motivation, both in the introduction and in the rebuttal, and robot demonstrations are about a third of your training data. It does not make sense to then treat DynaMo as too different in goal to compare against.
  • LAPO: the DELTA token seems very similar to a latent action. Both are compact tokens that encode state transitions and condition a forward predictor. What distinguishes DELTA from a latent action conceptually?
  • Time-Aware World Model conditions dynamics prediction on the time step, which seems closely related to your goal of continuous-time prediction. It is not clear to me how TemDy differs beyond training only a vision encoder rather than a world model. It is an immediate related work that should be discussed, and ideally compared against.

W3 & Q1: Figure 2

Thank you, this is resolved. I think I just had a faulty PDF that hit the top right of Figure 2.

Q2: Consistency of training data in Table 2

Thank you, this answers my question.

Q3: NMSE and the comparison to baselines

Thank you for clarifying that NMSE is computed against shared DINOv3 ViT-L targets. This is only stated in the appendix and should be in the main text. My concern about the evaluation remains. I think this makes NMSE a faulty metric for temporal understanding. It measures how well each representation predicts DINOv3 ViT-L features through a small readout, so a method can score well simply by being more compatible with that feature space. I also notice that pretrained DINOv3 scores worse. However, it has no temporal modeling, and its features appear to be averaged before the readout, so even its Current NMSE is poor.

Reviewer_uuxhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TemDy, a self-supervised video representation learner intended to model visual state changes as a continuous-time process rather than a set of frame associations. A temporal encoder uses timestamp-aware 3D rotary embeddings and a DELTA register, trained with both reconstruction of observed states and prediction of future teacher features. A second module builds a history-conditioned anchor and tangent direction on the feature sphere, then integrates a scalar angular-velocity function to obtain features at arbitrary queried times. The authors conduct experiments on several datasets to evaluate the effectiveness of their method.

Critical strengths
  1. The paper explicitly models continuous state evolution rather than relying only on discrete frame relationships. The rotation-based formulation provides a reasonable way to incorporate temporal changes and actual time intervals.

  2. The method achieves improvements in feature reconstruction across several datasets. The comparisons with existing baselines and the ablation studies provide useful evidence for the proposed temporal modeling approach.

  3. The paper includes additional analyses of DELTA conditioning and observation timestamps. These experiments help clarify how the model uses temporal information and support some of the design choices.

Critical weaknesses

The results are more convincing for feature reconstruction than for temporal discrimination, where the improvements are less consistent. Although the ablation studies are useful, the contribution of the rotation-based modeling to the overall gains and its generalization to more complex temporal dynamics could be better established.

Questions for authors
  1. Table 2 compares complete training recipes, while Table 4 studies different temporal parameterizations with a frozen encoder. How much of the overall improvement comes specifically from the rotation-based predictor rather than the temporal encoding and supervision?

  2. The held-out-time experiments in Table 4 are conducted on a relatively small evaluation set with limited query offsets. How well does the model generalize to more irregular sampling intervals or longer prediction horizons?

  3. The authors acknowledge that the fixed rotation plane limits trajectory complexity. Could they clarify how this assumption affects the modeling of more complex motions, particularly when the latent trajectory cannot be represented within a single rotation plane?

  4. The theoretical results in Section 3.2 seem to follow from standard properties of rotations. Could the authors clarify the main novelty of their formulation compared with existing continuous-time latent dynamics models?

  5. Tables 2 and 3 include several strong baselines, but omit other established video encoders such as VideoMAE V2 and InternVideo2. Could the authors explain this choice and discuss whether these models could be compared under the same evaluation protocol?

2 comments
AuthorsOpenReview ↗

Response to Reviewer uuxh (1/2)

Q1 & W1: Contribution of rotation and the difference between reconstruction and temporal discrimination

Thank you for recognizing the feature-reconstruction results. To distinguish the contributions of the history representation and the predictor, we added a factorial comparison of 1536D four-frame temporal-convolution features versus CLS+DELTA, and MLP versus rotation integration, matching observation access, target feature space, and training budget.

History representationPredictorHistory NMSE ↓Future NMSE ↓
Four-frame temporal convolutionDirect MLP0.23180.2694
Four-frame temporal convolutionRotation integration0.21370.2680
CLS+DELTAStandardized MLP0.20610.2355
CLS+DELTARotation integration0.18640.2092

With the MLP predictor fixed, CLS+DELTA outperforms four-frame temporal convolution; with CLS+DELTA fixed, rotation integration further reduces error. These results support complementary contributions from the history representation and the rotational parameterization.

For AgiBot, reconstruction NMSE more directly measures how accurately target features and the details of their temporal changes are recovered, whereas temporal discrimination requires distinguishing visually similar states at different times. Compared with the other two datasets, AgiBot contains slower motion and smaller frame-to-frame differences, making states at different times harder to distinguish.

Q2: Irregular sampling and generalization across prediction horizons

Table 4 evaluates query times without direct supervision, while Table 17 tests the use of observation intervals through timestamp interventions. We further added experiments with irregular observations, missing frames, and different prediction horizons.

First, on held-out SSv2 test videos, we fixed the checkpoints, observation intervals, and evaluation targets, and compared regular four-frame observations, irregular four-frame observations, and 25%/50% frame removal. For all three methods, missing slots are filled with the preceding valid observation and its actual timestamp. All input frames are excluded from the evaluation targets.

Observation settingDirect MLP MSE ↓FM MSE ↓TemDy MSE ↓
Regular, four frames0.0579780.0591520.054047
Irregular, four frames0.0590540.0601990.054926
25% missing, three frames0.0713860.0725450.066576
50% missing, two frames0.0883480.0895710.084374

TemDy outperforms both matched baselines in all four conditions. Irregular sampling has a relatively small effect, whereas missing frames still substantially increase error. This experiment uses the spatial-transport-plus-rotation extension and evaluates reconstruction within the observation interval.

For longer-horizon prediction, we break down the factorial comparison of 1536D history representations and predictors by prediction horizon. Short, medium, and long horizons correspond to 0–4, 4–8, and 8–16 seconds after the last observation, respectively.

History representationPredictorShort-horizon NMSE ↓Medium-horizon NMSE ↓Long-horizon NMSE ↓
Four-frame temporal convolutionDirect MLP0.23970.27180.2967
Four-frame temporal convolutionRotation integration0.23380.27160.2986
CLS+DELTAStandardized MLP0.19870.23790.2699
CLS+DELTARotation integration0.16890.21130.2474

Error increases with prediction horizon, but CLS+DELTA combined with rotation integration maintains the lowest error in all three intervals.

AuthorsOpenReview ↗

Response to Reviewer uuxh (2/2)

Q3: The fixed-plane constraint and complex motion

To quantify how much of the target-feature trajectory lies outside the predicted rotation plane, we compute the orthogonal projection error from the targets to the plane in the normalized coordinates used for NMSE, and report its fraction of the total prediction error. The horizon groups are the same as in Q2.

History representationPredictorShort-horizon out-of-plane error fractionMedium-horizon out-of-plane error fractionLong-horizon out-of-plane error fraction
CLS+DELTARotation integration0.93%2.71%3.39%

The out-of-plane error fraction increases with horizon but remains low within the evaluated range. Our rotation plane is determined by the observation history, so different histories can define different planes; for a given history, all query times share the same plane.

We further compare rotation integration with a standardized MLP that does not impose a fixed-plane constraint, using the same CLS+DELTA representation:

History representationPredictorHistory NMSE ↓Future NMSE ↓
CLS+DELTAStandardized MLP0.20610.2355
CLS+DELTARotation integration0.18640.2092

Rotation integration achieves lower error for both history reconstruction and future prediction. Together with the out-of-plane error analysis, this supports the fixed-plane constraint as an effective inductive bias within the evaluated range.

Q4: Novelty relative to continuous-time latent dynamics

Our contribution is to combine history-conditioned rotational trajectories with self-supervised visual representation learning. The model constructs an anchor and a tangent direction from the observation history, organizes states at different query times through a shared scalar angular-velocity function, and learns visual representations jointly through history reconstruction and future teacher-feature prediction.

Compared with general high-dimensional vector-field predictors, TemDy reduces continuous evolution to scalar angular-velocity integration within a history-conditioned plane, while ensuring norm preservation and temporal continuity by construction. Specifically, Appendix C establishes norm preservation and temporal continuity; Proposition 2 establishes interval-composition consistency for a fixed history and plane; Proposition 3 shows that any two unit states can be connected by rotation in a suitable plane; and Proposition 1 provides a local second-order approximation error bound when the initial state and tangent direction are matched. Appendix D.1 further discusses the relationship to general continuous-dynamics parameterizations.

Table 4 also supports this design: average NMSE is reduced by 8.7% and 9.1% relative to direct feature prediction and constant angular velocity, respectively.

Relative to LTN【1】, TemDy constructs trajectories from observation histories and organizes state evolution through shared angular-velocity integration over actual time. We will clarify this positioning further in the related-work discussion.

【1】Di Yang et al. Self-Supervised Video Representation Learning via Latent Time Navigation. AAAI, 2023. Paper.

Q5: VideoMAE V2 and InternVideo2 baselines

Thank you for the suggestion. The original submission primarily selected DINOv3 and V-JEPA2.1 as representatives of image and video self-supervised learning, respectively, and did not cover VideoMAE V2 or InternVideo2. To broaden the comparison, we added VideoMAE V2 Base and InternVideo2-dist-B/14, both at ViT-B scale, and report their IntPhys 2 accuracy results:

EncoderContinuity ↑Solidity ↑Immutability ↑Permanence ↑Four-task mean ↑
Public DINOv353.3347.9550.8350.0050.53
VideoMAE V2 Base50.0049.3246.6753.3349.83
InternVideo2-dist-B/1452.5044.5256.6751.6751.34
TemDy (pretrained)55.8352.7453.3354.7954.17
Reviewer_p4BKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TemDy, a self-supervised framework for learning continuous-time visual dynamics from video. The method introduces a DELTA register to encode temporal variations and represents feature evolution as a rotation trajectory on the unit sphere. Specifically, the predicted trajectory is parameterized by an anchor state, a tangent direction, and a time-varying angular velocity, allowing the model to predict visual features at arbitrary query timestamps. The training objective combines historical state reconstruction with future feature prediction.

The paper evaluates TemDy on multiple video datasets and downstream tasks, including feature reconstruction, temporal retrieval, physical plausibility assessment, and contact prediction. The reported results show improvements over several baselines on a number of feature reconstruction metrics, with additional experiments investigating the effects of different prediction heads and pretrained initializations.

Critical strengths
  • The motivation is clear. The problem of learning temporally consistent representations and continuous-time visual dynamics is important, and predicting visual states at arbitrary timestamps is a useful goal for temporal representation learning.

  • The rotational formulation is easy to interpret. Variable angular velocity gives the model more flexibility than a constant-speed trajectory.

  • The results suggest that the temporal encoding and prediction head can help on some downstream tasks.

Critical weaknesses
  • The local approximation and interval-composition results follow from familiar smooth-curve and rotation properties, but do not establish an advantage over other continuous-time predictors.

  • Each predicted trajectory stays in the fixed plane defined by its anchor and tangent. Variable angular velocity cannot change this plane, and the paper does not show when this restriction helps or how much it hurts on complex or long-horizon motion.

  • The Table 4 compares prediction heads, but stronger controls are needed to separate the rotational structure from the DELTA register, training objectives, capacity, and optimization, especially given the weak Flow Matching result.

  • The downstream evidence is mixed, i.e., IntPhys 2 solidity and immutability do not improve over Image SSL, and Physion Collide AUC is 77.72 versus 83.09 for V-JEPA2.1.

  • The unseen-query experiments are useful, but systematic tests with irregular observations, missing frames, and longer extrapolation are needed, together with variability and compute costs for matched baselines.

Questions for authors

No additional questions, I hope the rebuttal can address the weaknesses above, especially the fixed-plane assumption, the source of the gains, and the practical benefits of continuous-time prediction.

3 comments
AuthorsOpenReview ↗

Response to Reviewer p4BK (1/3)

W1: Theoretical properties and advantages for continuous modeling

Thank you for the comment. Compared with general high-dimensional vector-field predictors, TemDy reduces continuous state evolution to scalar angular-velocity integration within a history-conditioned plane, while ensuring norm preservation and temporal continuity by construction. This design reduces the degrees of freedom that must be explicitly modeled while retaining the ability to represent local dynamics.

Table 4 also supports the practical benefit of this design: using the same frozen encoder and training budget, TemDy reduces average NMSE by 8.7% and 9.1% relative to direct feature prediction and constant angular velocity, respectively.

W2: Fixed-plane limitations for complex and long-horizon dynamics

Thank you for raising the scope of the geometric constraint. Prediction horizon is the time difference between the query and the last observed frame; short, medium, and long horizons are 0–4 s, 4–8 s, and 8–16 s, respectively. First, to quantify how much of the target-feature trajectory lies outside the predicted rotation plane, we compute the orthogonal projection error from the targets to the predicted plane in the normalized coordinates used for NMSE, and report its fraction of total prediction error:

Prediction horizonOut-of-plane fraction of total error (%)
Short horizon (0–4 s)0.93
Medium horizon (4–8 s)2.71
Long horizon (8–16 s)3.39

The out-of-plane error fraction increases with horizon but remains below 3.4% in all three evaluated groups.

We further added a factorial comparison of 1536D four-frame temporal-convolution features versus CLS+DELTA, and MLP versus rotation integration, evaluated across prediction horizons:

History representationPredictorHistory NMSE ↓Future NMSE ↓Short horizon ↓Medium horizon ↓Long horizon ↓
Four-frame temporal convolutionDirect MLP0.23180.26940.23970.27180.2967
Four-frame temporal convolutionRotation integration0.21370.26800.23380.27160.2986
CLS+DELTAStandardized MLP0.20610.23550.19870.23790.2699
CLS+DELTARotation integration0.18640.20920.16890.21130.2474

CLS+DELTA combined with rotation integration achieves the lowest error at every evaluated horizon. Together with the out-of-plane analysis, these results support the fixed-plane constraint as an effective inductive bias within the evaluated range.

AuthorsOpenReview ↗

Response to Reviewer p4BK (2/3)

W3: Separating the contributions of rotation, DELTA, supervision, and optimization

Thank you for the suggestion. Table 4 compares temporal parameterizations using the same frozen encoder and training budget; Table 16 fixes the anchor and replaces DELTA to test dependence on historical dynamic information. We further added Image SSL CLS and JEPA SSL four-frame temporal-convolution controls. All three groups start from the same random ViT-B initialization, use the same training data and budget, and produce 1536D representations.

All metrics are reported on a percentage scale: accuracy for IntPhys 2 and AUC for Physion.

RepresentationContinuity ↑Solidity ↑Immutability ↑Support ↑Drop ↑Link ↑Collide ↑Avg. ↑
Image SSL CLS52.5048.6351.6768.0263.6962.4266.6759.09
JEPA SSL four-frame features + temporal convolution56.6746.5745.8378.2364.9661.1477.1461.51
TemDy CLS+DELTA60.0050.6852.5076.0666.5764.3877.0763.89

We report accuracy on IntPhys 2 and AUC on Physion. With all representations standardized to 1536D, TemDy achieves a seven-task mean score of 63.89, exceeding the two controls by 4.81 and 2.39 percentage points, respectively. These results support that its gains cannot be explained solely by input dimensionality.

For FM, the current prediction heads have similar parameter counts but solve different modeling tasks: FM learns a high-dimensional conditional velocity field, whereas TemDy learns scalar angular velocity within a history-conditioned plane. Limited head capacity may make these tasks differ in fitting difficulty. This is a possible explanation of the current results, not a judgment about the overall capability of FM.

W4: Mixed performance across physical tasks

These subtasks test different physical abilities. Solidity and Immutability concern non-interpenetration and property persistence, respectively; continuous-time parameterization alone does not ensure that these constraints are correctly encoded. Continuity more directly concerns temporal continuity of states and aligns with TemDy's objective of learning state changes over time. Physion Collide requires a joint representation of object relationships and motion. V-JEPA2.1's large-scale video pretraining may provide a more favorable basis for extracting such information, whereas the pretrained TemDy variant is fine-tuned from DINOv3.

Overall, TemDy ranks first in both the from-scratch and pretrained groups on the seven-task mean physical-reasoning score in Table 3. The respective means are 63.89% and 68.52%, compared with 62.32% and 64.70% for the best baseline in each group. This supports its overall performance across the evaluated physical tasks, although it does not perform best on every subtask.

AuthorsOpenReview ↗

Response to Reviewer p4BK (3/3)

W5: Irregular observations, missing frames, extrapolation, and computational cost

Thank you for the suggestion. Table 17 already tests the use of observation intervals through timestamp interventions, and Tables 10 and 11 report the computational costs of the integration head and encoder. We further added the following experiments.

Irregular observations and missing frames. On SSv2, we fixed the checkpoints of all three models, the observation intervals, and the evaluation targets, and compared regular four-frame observations, irregular four-frame observations, and 25%/50% frame removal. Missing slots are consistently filled with the preceding valid observation and its actual timestamp; all input frames are excluded from the evaluation targets.

Observation settingDirect MLP MSE ↓FM MSE ↓TemDy MSE ↓
Regular, four frames0.0579780.0591520.054047
Irregular, four frames0.0590540.0601990.054926
25% missing, three frames0.0713860.0725450.066576
50% missing, two frames0.0883480.0895710.084374

TemDy achieves the lowest error in all four conditions. Irregular sampling has a relatively small effect, whereas missing frames still substantially increase reconstruction error. Results for long-horizon extrapolation are reported in W2.

Stability across random seeds. On SSv2, we repeated the predictor and loss-ablation experiments with three random seeds, using the same data split and training budget, and report means and standard deviations.

Predictor / training objectiveHistory NMSE ↓Future NMSE ↓
Direct MLP0.7206±0.00320.8003±0.0014
FM0.6312±0.03330.4720±0.0280
Rotation integration without L_unc0.6681±0.00940.4233±0.0041
Rotation integration without L_det0.5965±0.00640.6062±0.0133
Rotation integration (full objective)0.5847±0.02620.4029±0.0027

The full objective achieves the lowest mean error for both history reconstruction and future prediction, supporting the complementary roles of the two types of supervision.

Computational cost. We measured the full prediction pipeline, including the encoder and history projection, on the same H800 with batch size 8, four 224×224 input frames, and eight query times. The encoder uses BF16 and the predictors use FP32; FM uses 16 steps and rotation integration uses 64 nodes. Latency is the median of 90 measurements.

MethodFull-pipeline latency (ms)↓Peak memory (MiB)↓FLOPs (G/batch)↓
Direct MLP10.351598.951147.852
FM11.819599.941149.671
Rotation integration11.728599.131149.938

Rotation integration has a similar full-pipeline cost to FM and adds approximately 1.38 ms of latency over Direct MLP, with small differences in memory and operation count.

Reviewer_Fsu1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TemDy, a self-supervised framework for learning continuous-time visual representations from timestamped video observations. A causal vision transformer uses spatiotemporal rotary position encodings and a dedicated DELTA register to aggregate temporal information alongside a CLS representation. A prediction module constructs a normalized state anchor, an orthogonal tangent direction, and a time-dependent scalar angular velocity. Integrating this velocity generates predicted features along a history-conditioned rotational trajectory. Training combines reconstruction of observed historical and endpoint states with prediction of unobserved future teacher features, supplemented by image self-supervision and representation regularization. The central contribution is a structured temporal parameterization that separates the trajectory plane from progression within that plane. The paper provides local approximation and fixed-history interval-composition results to motivate this design. Validation includes feature reconstruction, frame retrieval, and temporal localization on AgiBot, EPIC-KITCHENS-100, and Something-Something V2, together with physical-reasoning evaluations on IntPhys 2 and Physion. Ablations examine temporal parameterization, reconstruction objectives, DELTA conditioning, and observation timestamps. The reported results show improvements in several reconstruction settings and selected transfer tasks, while retrieval and temporal-discrimination improvements vary across datasets and model variants.

Critical strengths
  1. A coherent and interpretable temporal model. TemDy provides a concrete mechanism for predicting normalized visual states at continuous query times. Separating the trajectory plane from angular progression yields an interpretable, norm-preserving parameterization requiring only scalar quadrature. It introduces a clear temporal inductive bias whose assumptions and computational behavior can be examined directly.
  2. Useful controlled evidence supporting the proposed mechanism. The paper includes parameterization comparisons, direct-angle prediction, fixed-anchor DELTA interventions, and timestamp perturbations. Combined with reconstruction improvements across multiple datasets, it provides evidence that the model uses sequence-specific temporal information.
Critical weaknesses
  1. The benefit and scope of the central geometric restriction remain insufficiently established. All predictions for a history lie on one great circle, while Proposition 1 provides only a local approximation result. The ablations are encouraging, but do not characterize target trajectories’ deviation from this restriction or its effect across prediction horizons. A matched, more flexible continuous-time predictor would help establish the trade-off. Positioning against time-aware latent methods such as LTN also needs strengthening.
  2. The broader empirical gains are not yet sufficiently isolated or statistically supported. Appendix B.8 gives TemDy 1536-dimensional CLS+DELTA features, Direct Prediction 768-dimensional CLS features, and pretrained image encoders temporally averaged features. These choices affect the information available to downstream readouts. Although the parameterization ablation is more controlled, these issues weaken the broader evidence for reliably improved temporal understanding.
Questions for authors
  1. Clarification. Table 4 reports integration-model unseen-time NMSEs of 0.4907 and 0.4414, whereas Table 12 reports 0.460708 and 0.421449 under the Table 4 protocol. What differences in initialization, configuration, checkpoints, targets, or normalization account for these values? Please make the correspondence between the experiments explicit.
  2. Clarification. Which comparisons match observation access, temporal aggregation, readout input dimensionality, and readout parameter count? In particular, how should the Physion comparison between CLS+DELTA and CLS-only representations be interpreted? Are results available with comparable temporal information and readout capacity?
  3. Additional validation. How much of the target-feature trajectory lies outside the predicted rotation plane, and how does this deviation relate to reconstruction error across prediction horizons? Quantifying this on the existing evaluation data would help determine when the proposed restriction is an appropriate inductive bias.
  4. Additional validation. How stable are the principal parameterization and objective-ablation gains across random seeds or appropriate resampling units? Additionally, how do the authors interpret the lower AgiBot retrieval accuracy of TemDy than Image SSL in Table 7, despite its improved reconstruction error?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Fsu1 (1/2)

Q1: Numerical discrepancy between Tables 4 and 12

Thank you for identifying this inconsistency. The integration models in both tables use the same frozen DINOv3 ViT-B features and predictor-training protocol. The discrepancy comes from the evaluation metric: Table 4 reports raw NMSE, whereas Table 12 reports NMSE after removing the mean prediction bias. The latter subtracts the squared normalized mean-bias term from raw NMSE, so the values in the two tables are not directly comparable. We will explicitly define both metrics and their computational relationship in the appendix.

Q2 & W2: Observation access, readout capacity, and fair comparison

Thank you for the suggestion. Table 4 compares temporal parameterizations using the same frozen encoder and training budget; Table 16 fixes the anchor and replaces DELTA to test dependence on historical dynamic information. We further added Image SSL CLS and JEPA SSL four-frame temporal-convolution controls. All three groups start from the same random ViT-B initialization, use the same training data and budget, and produce 1536D representations.

All metrics are reported on a percentage scale: accuracy for IntPhys 2 and AUC for Physion.

RepresentationContinuity ↑Solidity ↑Immutability ↑Support ↑Drop ↑Link ↑Collide ↑Avg. ↑
Image SSL CLS52.5048.6351.6768.0263.6962.4266.6759.09
JEPA SSL four-frame features + temporal convolution56.6746.5745.8378.2364.9661.1477.1461.51
TemDy CLS+DELTA60.0050.6852.5076.0666.5764.3877.0763.89

We report accuracy on IntPhys 2 and AUC on Physion. With all representations standardized to 1536D, TemDy achieves a seven-task mean score of 63.89, exceeding the two controls by 4.81 and 2.39 percentage points, respectively. These results support that its gains cannot be explained solely by input dimensionality.

AuthorsOpenReview ↗

Response to Reviewer Fsu1 (2/2)

Q3 & W1: The single-plane constraint, out-of-plane deviation, and positioning

Thank you for raising the scope of the geometric constraint. Prediction horizon is the time difference between the query and the last observed frame; short, medium, and long horizons are 0–4 s, 4–8 s, and 8–16 s, respectively. First, to quantify how much of the target-feature trajectory lies outside the predicted rotation plane, we compute the orthogonal projection error from the targets to the predicted plane in the normalized coordinates used for NMSE, and report its fraction of total prediction error:

Prediction horizonOut-of-plane fraction of total error (%)
Short horizon0.93
Medium horizon2.71
Long horizon3.39

The out-of-plane error fraction increases with horizon but remains below 3.4% in all three evaluated groups.

We further added a factorial comparison of 1536D four-frame temporal-convolution features versus CLS+DELTA, and MLP versus rotation integration, evaluated across prediction horizons:

History representationPredictorHistory NMSE ↓Future NMSE ↓Short horizon ↓Medium horizon ↓Long horizon ↓
Four-frame temporal convolutionDirect MLP0.23180.26940.23970.27180.2967
Four-frame temporal convolutionRotation integration0.21370.26800.23380.27160.2986
CLS+DELTAStandardized MLP0.20610.23550.19870.23790.2699
CLS+DELTARotation integration0.18640.20920.16890.21130.2474

CLS+DELTA combined with rotation integration achieves the lowest error at every evaluated horizon. Together with the out-of-plane analysis, these results support the fixed-plane constraint as an effective inductive bias within the evaluated range; its applicability still depends on the target trajectory and prediction horizon.

Relative to LTN【1】, TemDy constructs trajectories from observation histories and organizes state evolution through shared angular-velocity integration over actual time. We will further clarify this positioning in the related-work discussion.

【1】Di Yang et al. Self-Supervised Video Representation Learning via Latent Time Navigation. AAAI, 2023. Paper.

Q4: Statistical stability and divergent AgiBot metrics

Thank you for the suggestion. We added multi-seed predictor and training-objective experiments on SSv2 and report the mean and standard deviation of NMSE.

Predictor / training objectiveHistory NMSE ↓Future NMSE ↓
Direct MLP0.7206±0.00320.8003±0.0014
FM0.6312±0.03330.4720±0.0280
Rotation integration without L_unc0.6681±0.00940.4233±0.0041
Rotation integration without L_det0.5965±0.00640.6062±0.0133
Rotation integration (full objective)0.5847±0.02620.4029±0.0027

The full objective achieves the lowest mean error for both history reconstruction and future prediction, supporting the complementary roles of the two types of supervision.

For AgiBot, reconstruction NMSE measures feature-recovery error, whereas Top-1 measures the exact ranking of candidate states. Compared with the other two datasets, slower motion and smaller frame-to-frame differences in AgiBot may make the top-ranked candidate more sensitive to small differences, explaining why lower reconstruction error is not accompanied by improved Top-1 accuracy.

The rebuttal addresses several of my concerns. The authors clarify the discrepancy between Tables 4 and 12, and provide additional matched dimensionality comparisons, representation predictor ablations, and out-of-plane error analyses. These results strengthen the evidence for the proposed rotational parameterization and help distinguish its contribution from that of the learned representations. The additional multi-seed experiments also provide useful evidence regarding stability.

Some concerns remain. The authors should clarify the bias-correction procedure, the comparability of observation access and prediction-head capacity, and the interpretation of out-of-plane error percentages. The statistical evidence is still somewhat limited, and the lower retrieval accuracy on AgiBot is not fully explained. Nevertheless, the additional experiments have substantially alleviated my main concerns. I therefore raise my rating to 3.

Thank you for reassessing our work and acknowledging the additional evidence. We will incorporate the supplementary experiments into the revised manuscript and further clarify the evaluation protocols, findings, and limitations to address the remaining concerns.

Comments on the paper

1 comment
Area_Chair_NZv5OpenReview ↗

Reminder: Author–Reviewer Discussion Deadline Approaching

Dear Reviewers and Authors,

Thank you to the reviewers for your time, effort, and dedication in reviewing this paper. This AC also appreciates the authors' detailed rebuttal, which addresses the reviewers' comments and includes additional clarifications and experimental results.

To the reviewers: As the author–reviewer discussion deadline approaches, please carefully review the authors' responses and assess whether they adequately address your concerns, if you haven't already. If any concerns have not been fully addressed, please let the authors know what further clarification or evidence would be helpful.

Thank you again for your valuable contributions and constructive discussion.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.