acceptodds
Under review as a conference paper at ICLR 2027

Learning Gravity from Video: Structured Latent States for Prediction and Intervention

Abstract

A physical system evolves according to its current state and governing dynamics. This suggests organizing video world models around visually inferred states and reusable evolution rules. Yet predictive learning alone does not ensure that learned representations support sustained physical evolution and meaningful intervention. We introduce GravJEPA, a self-supervised world model built on LeWorldModel that separates visual state estimation from dynamical evolution. The model infers structured latent components from video histories and propagates them with a time-homogeneous Koopman-form transition. This formulation supports direct queries at future times after observations stop. To align the inferred states with this dynamical structure, we introduce regularization objectives that encourage translation equivariance and photometric invariance in position-associated representations. The visual states are learned without ground-truth physical-state or gravity supervision. On controlled gravity videos, GravJEPA achieves lower trajectory prediction errors than an autoregressive baseline trained with the same equivariance and invariance regularization. It also outperforms this baseline in long-horizon prediction and out-of-distribution tests with unseen velocity and gravity magnitudes. Beyond prediction, matched interventions on inferred velocity- and gravity-associated components produce corresponding changes in future motion through the same transition. Together, these results establish a shared latent-state interface for forecasting and intervention, taking a step toward physics-engine-like world models learned from video.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Models, Self-Supervised Learning, Representation Learning, Joint-Embedding Predictive Architectures

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qwxBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

GravJEPA, built on LeWorldModel, learns from video a latent state you can predict with and intervene on: a ViT encoder gives position-related coordinates, an estimator infers latent velocity and gravity, and a fixed constant-acceleration transition carries the state to any future time, with an equivariance regularizer tying coordinates to frame translations. On single-object videos it beats an autoregressive baseline (ADE 1.81 vs 2.45 px), and swapping the velocity or gravity component shifts predictions the way physics says it should.

Critical strengths
  1. The intervention experiments are the best part. Replacing only the inferred velocity or gravity moves the predicted trajectory in physically sensible ways through the same fixed transition (gain around 1.1 for velocity, with shuffled-donor controls), so these components actually do something in the model rather than just being readable after the fact.
  2. The evaluation is careful where it counts: paired OOD construction with visibility checks, correlation audits of the sampler, frozen ridge readouts, seed variance on the headline table, and query-cost accounting that excludes initialization honestly.
Critical weaknesses
  1. The gap between what is shown and what the title promises is large. One object on a gray background; no contact, no occlusion, no texture, no camera motion, no actions. "Learning gravity from video" here means reading a 2D acceleration off a seven-frame observation history, with the constant-acceleration law already built in. Whether any of this survives non-uniform backgrounds or natural video is simply not tested. I would take this as a controlled case study, and the title should say so.
  2. Gravity, the headline quantity, is also the weakest result. Probe R2≈0.524R^2 \approx 0.524 against ≈0.99\approx 0.99 for position and velocity; scaling interventions have ErespE_{\mathrm{resp}} up to 0.419 and gains of 0.89–0.94; latent arithmetic on gg goes wrong up to ~0.78. With seven frames, velocity and gravity are identified through the same quadratic form, and the paper never really explains why gg comes out so much worse (noise in second differences of pp is the obvious suspect). That question sits right on the central claim.
Questions for authors
  1. Why is gravity so much less decodable than velocity while gravity interventions still behave? Is gg entangled with vv, and does the common-p diagnostic tell us anything about coordinate alignment?
  2. With a non-uniform background or several objects, what does p(Tδx)≈p(x)+Qδp(T_{\delta x}) \approx p(x) + Q\delta actually enforce? Does it tie the method to single-object scenes?
  3. Could a learned residual transition (drag, contact) sit on top of the prescribed generator, or does the scheme need AA's exact nilpotent form?
4 comments

We thank the reviewer for recognizing the rigor of our experiments and our state intervention protocol. Before our point-by-point responses, we briefly restate the core problem and contributions of this paper. Our experiments show that, although the mainstream video JEPA models evaluated can encode decodable position and other dynamics-related information, their latent evolution generally deviates from basic dynamical laws, and their representations do not yet constitute physically consistent states. The key issue is that the feature-prediction objective constrains the match between predictions and target representations but does not explicitly specify the dynamical relationships among state components, leaving latent evolution underconstrained in its physical structure. Consequently, accurate predictions at observed timesteps are insufficient to guarantee consistency in long-horizon extrapolation and state interventions.

To address this problem, we use constant-acceleration gravitational motion as a controlled setting and explicitly embed a structured dynamical system into the forward prediction of a video JEPA. Together with EIR, data augmentation, and history fitting, this organizes state components with distinct dynamical roles, so that position-related information evolves according to explicit dynamical relationships. The resulting state transitions possess time homogeneity and temporal composition consistency by construction; we further evaluate the accuracy and usability of the visually learned states through prediction, extrapolation, and state interventions. The core contribution of this paper is to transform explicit dynamics from an external evaluation criterion into an internal evolution structure that visual representation learning must satisfy.

To our knowledge, at the time of submission, this was the first work to explicitly embed a structured dynamical system within the JEPA paradigm. Following our submission, LeHami [1] and Hierarchical World Models with Implicit Dynamics [2], both coauthored by Yann LeCun, explored Hamiltonian latent evolution and the separation of state-space reachability from low-level action execution, respectively. These independent developments resonate, from different perspectives, with our investigation of structured state evolution.

[1] Romina Yalovetzky, Randall Balestriero, and Yann LeCun. LeHami: Symplectic Latent Dynamics for World Models. arXiv:2610.15267, 2026. https://arxiv.org/abs/2610.15267

[2] Gaoyue Zhou, Yvonne Wu, Zichen Jeff Cui, Nicolas Ballas, Mido Assran, Lerrel Pinto, and Yann LeCun. Hierarchical World Models with Implicit Dynamics. arXiv:2610.42146, 2026. https://arxiv.org/abs/2610.42146

1 Gravity recovery, functional use, and coordinate alignment

Question (paraphrased). Why is gravity less decodable than velocity despite useful interventions, and what does common-pp reveal about coordinate alignment and state entanglement?

Gravity recovery is sensitive to position errors amplified by second-order differencing, while gravity contributes less than velocity to displacement over the current dataset's prediction horizon. High position-probe R2R^2 therefore need not imply accurate gravity estimation, and accurate velocity can support good short-horizon prediction despite gravity errors.

The current ID prediction horizon spans 0.055–0.330 s; the corresponding displacement bounds are shown below (pixels).

Future timeVelocity ∥tv∥2\lVert tv\rVert _2Gravity ∥12t2g∥2\lVert \frac{1}{2}t^2g\rVert _2
0.055 s (first query)2.76–4.030.15–0.19
0.330 s (last query)16.58–24.195.23–6.97

Velocity accounts for the larger displacement throughout this horizon, but gravity is still used: zeroing gg increases ID FDE from 4.23 to 7.46 pixels. Imperfect gravity estimates can still support effective interventions, while the marked drop in gravity decodability under common-pp indicates incomplete alignment with position coordinates, but does not by itself establish entanglement between gravity and velocity.

2 EIR with non-uniform backgrounds and multiple objects

Question (paraphrased). What does EIR enforce with non-uniform backgrounds or multiple objects, and does it tie the method to single-object scenes?

EIR's geometric constraints remain valid with non-uniform backgrounds and multiple objects, and can be imposed on each object's state. A shared translation of all frames shifts each object's position equally, irrespective of background texture or object count, while leaving its velocity and acceleration unchanged. Thus, the same equivariance and invariance relations remain valid; applying them at the object level requires consistent object identity, which EIR alone does not establish.

Furthermore, dense features from self-supervised models such as DINOv3 could support object grouping and temporal association, providing a possible route to applying EIR to individual objects while preserving their original image coordinates. We have not yet evaluated this extension in complex-background or multi-object settings.

3 Residual dynamics and extension to additional physical phenomena

Question (paraphrased). Could a learned residual transition model drag or contact on top of the prescribed generator, or does the method require its exact nilpotent form?

Extending GravJEPA to more complex dynamics is feasible within its state-based formulation, either by extending the dynamics operator or by learning interventions on the existing state. Exact nilpotency enables our current quadratic closed-form propagation; it is not a prerequisite for this broader state-based formulation.

1. Extend the dynamics operator. For s=[p;v;g]s=[p;v;g], we could learn a structured residual to the constant-acceleration generator A0A_0:

\Delta A_\theta=\begin{bmatrix} 0&0&0\\\\-K_\theta&-D_\theta&0\\\\0&0&0 \end{bmatrix}.$$ Here $-D_\theta v$ represents linear damping and $-K_\theta p$ a restoring force. A constant generator still permits matrix-exponential propagation, but generally loses the exact quadratic truncation. Nonlinear effects may require richer Koopman observables or numerical integration. **2. Learn state interventions.** We could retain the existing core and learn how external effects update its state: collisions produce instantaneous velocity changes, $v^+=v^-+\Delta v_\theta$, whereas damping or friction supplies a state-dependent acceleration, $\dot v=g+a_\theta(s)$. Such forces continuously modify velocity rather than gravity itself. These routes overlap: linear damping can be represented either by a matrix term or by an acceleration intervention. Both routes require data containing the additional physical phenomena. The velocity- and gravity-related information already learned provides a starting point for continued training on such data, with the potential to extend physical coverage while retaining existing capabilities. ## 4 Scope and title We agree with the reviewer's suggestion to identify this work explicitly as a controlled study, and will revise the title to *Learning Gravity from Video: A Controlled Study of Structured Latent States for Prediction and Intervention*. We will also clarify in the abstract that the main study concerns a single object under constant acceleration, with a uniform background, a fixed camera, and no contact or occlusion. Here, learning concerns the inference and organization of dynamical states from video without physical-state labels; the constant-acceleration evolution law is prescribed.
Reviewer_qwxBOpenReview ↗

Thanks the authors for response

The gravity-recovery explanation and learnable-transition control make the contribution clearer, but the method is still evaluated only in a narrow controlled setting, so I will maintain my weak accept rating.

Thank you for taking the time to consider our response. We agree that evaluating trajectory prediction on real-world videos would strengthen the current study. In future work, we also hope to extend the approach to a broader range of physical dynamics.

Reviewer_uiDiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GravJEPA, a JEPA-based latent dynamics learning approach which employs a fixed constant-acceleration flow in Koopman form on a part of the latent state to encode physical dynamics. The remaining part of the latent state is intended to model visual appearance in the video. The imposed dynamics model can only represent second-order dynamics under acceleration. It cannot represent collisions or friction. Loss functions and data augmentations are proposed to impose translational equivariance and photometric invariance. The model is demonstrated on 64x64 pixels synthetic 2D videos of an object moving under gravity with different shape and color. Experimental results demonstrate improvements over a baseline (LeWM) and ablations in most evaluated cases. Adding the photometric invariance loss seems to harm position prediction.

Critical strengths
  • The proposal to impose a fixed constant-acceleration-flow seems novel in the realm of JEPA-based world models. However, it is rather restrictive in the dynamics it can model. Also the equivariance loss term appears new.
  • The experimental results demonstrate improvements. Yet the experiments use a specific toy-like 2D evaluation setting.
Critical weaknesses
  • Disentangling physical from visual features in the latent state is not new, but the paper does not discuss and differentiate from related works (e.g. [*1,*2])
  • The paper does not demonstrate that the proposed approach can also generalize to complex realistic videos.
  • The proposed approach is restricted to simple dynamics under a constant acceleration in a history of states. The model does not seem to be able to represent collisions and friction. The paper should discuss this limitation. How can the approach be generalized to more complex realistic video?
  • The abstract/intro should state that the proposed method has only been demonstrated on simple low resolution synthetic video of a moving shape in 2D.
  • Is translation equivariance useful in realistic video? What about interactions with other objects and the static background?
  • Fig. 1 is not referenced from the text.
  • l. 117, experimental evaluation is a necessary part for empirical validation, but not a "contribution".
  • The paper lacks a proper motivation of the choice of acronym GravJEPA. It seems the model is restricted to constant accelerations which is reflected in the choice of training/evaluation data with constant gravity vectors.
  • How is d chosen in the experiments?
  • How can the approach evaluate prediction accuracy in the latent embedding ? Does this require a learned map from physical latent states to position?
  • l. 216, how is the orthonormality constraint on Q enforced during training?
  • Important details how L_eq and L_photo are established/implemented are only provided in the appendix. However, this should be moved to the main paper, since this is one of the main claimed contributions of the paper.
  • l. 232, what are "future features used in the dynamical core's prediction objective" ?
  • Sec. 4, please explain which colors/shapes are used for the videos. Visualize the example videos and predictions for the videos.
  • Sec 4.1, why does the training distribution have "irregular observation timing" ?
  • How is the latent state mapped to position to compute the position-readout R^2 ? If it is a learned mapping, how is it trained?
  • The experiment section is not self-contained in the main paper and forward references key results and properties of the evaluation protocols in the appendix which are important to understand the main results/conclusions. This violates the page limit. Examples: l. 311 OOD ranges, evaluation protocol for Sec 4.4, l. 443 points to quantitative results, l. 451 smaller-scale configuration not explained.
  • The measure \epsilon_resp in eq. 11 is not used in the main paper.
  • Why are the trajectories in Figs. 2 and 3 smooth. Shouldnt they be time-discrete? How many model steps / which prediction horizon has been used to generate them ?
  • Sec 4.6, ADE and FDE measures are not defined.
  • Table 4b indicates that adding the photometric invariance loss harms the position prediction. This needs to be discussed.

[*1] Shen et al. PHANTOM: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamics. CVPR 2026 [*2] Guen and Thome. Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction. CVPR 2020 [*3] Zhu et al. Learning to disentangle latent physical factors for video prediction. GCPR 2019

Questions for authors

See "Weaknesses". Especially, the generalizability of the method to realistic videos, relation to related works that disentangle physical from visual latent dimensions, and issues in the experimental evaluation need to be addressed.

3 comments

1 Response to Reviewer: Scope, Related Work, and Evaluation

We thank the reviewer for recognizing the novelty of incorporating a fixed constant-acceleration flow into a JEPA-based world model and the improvements demonstrated in our experiments. Before our point-by-point responses, we briefly restate the core problem and contributions of this paper. Our experiments show that, although the mainstream video JEPA models evaluated can encode decodable position and other dynamics-related information, their latent evolution generally deviates from basic dynamical laws, and their representations do not yet constitute physically consistent states. The key issue is that the feature-prediction objective constrains the match between predictions and target representations but does not explicitly specify the dynamical relationships among state components, leaving latent evolution underconstrained in its physical structure. Consequently, accurate predictions at observed timesteps are insufficient to guarantee consistency in long-horizon extrapolation and state interventions.

To address this problem, we use constant-acceleration gravitational motion as a controlled setting and explicitly embed a structured dynamical system into the forward prediction of a video JEPA. Together with EIR, data augmentation, and history fitting, this organizes state components with distinct dynamical roles, so that position-related information evolves according to explicit dynamical relationships. The resulting state transitions possess time homogeneity and temporal composition consistency by construction; we further evaluate the accuracy and usability of the visually learned states through prediction, extrapolation, and state interventions. The core contribution of this paper is to transform explicit dynamics from an external evaluation criterion into an internal evolution structure that visual representation learning must satisfy.

To our knowledge, at the time of submission, this was the first work to explicitly embed a structured dynamical system within the JEPA paradigm. Following our submission, LeHami [1] and Hierarchical World Models with Implicit Dynamics [2], both coauthored by Yann LeCun, explored Hamiltonian latent evolution and the separation of state-space reachability from low-level action execution, respectively. These independent developments resonate, from different perspectives, with our investigation of structured state evolution.

[1] Romina Yalovetzky, Randall Balestriero, and Yann LeCun. LeHami: Symplectic Latent Dynamics for World Models. arXiv:2610.15267, 2026. https://arxiv.org/abs/2610.15267

[2] Gaoyue Zhou, Yvonne Wu, Zichen Jeff Cui, Nicolas Ballas, Mido Assran, Lerrel Pinto, and Yann LeCun. Hierarchical World Models with Implicit Dynamics. arXiv:2610.42146, 2026. https://arxiv.org/abs/2610.42146

1. Representation quality and related work

We agree that separating physical and visual information is not new. PhyDNet (Le Guen and Thome, CVPR 2020) separates PDE-constrained dynamics from complementary information for video prediction. PHANTOM (Shen et al., CVPR 2026) jointly models latent physical dynamics and future video generation. We will discuss these precedents and Zhu et al. (GCPR 2019), distinguishing our contribution from disentanglement in general.

Our central question is what makes a good motion-state representation, and how to design a model and learning objectives that encourage such a representation to emerge from video. Beyond decodability and low prediction error, we seek distinct dynamical roles, responses to physical changes, continued prediction and meaningful intervention after observations stop, and robustness to appearance changes that preserve dynamics.

Our JEPA combines history-based visual state inference, an explicit time-homogeneous transition, EIR, and history fitting. The transition specifies state relationships and evolution; EIR constrains responses to spatial and photometric changes; history fitting requires states to explain observed motion. The contribution is this formulation for learning useful motion states and evaluating these properties.

The t-SNE visualizations in Fig. 3(a), particularly for the position- and velocity-related components, qualitatively suggest a more organized representation space under controlled physical variations (Sec. 4.5, lines 416–422). While these visualizations do not establish the intrinsic geometry of the learned states, they complement our quantitative evaluations of state decodability, extrapolation, and intervention.

We will list specific empirical findings as contributions. Following Sec. 4.4 (lines 398–403), we describe the observed separation as partial functional specialization, rather than complete disentanglement.

2. Research scope, complex dynamics, and real-video evidence

This is a controlled study of learning visual states under a prescribed dynamical structure. The constant-acceleration law is given; the encoder and estimator learn states from video without physical-state labels. The law alone does not guarantee that visually inferred states are accurate or that their edits produce the intended changes in motion. Prediction, extrapolation, and intervention test these empirical properties. The submitted paper specifies gravity-driven motion without contact in Sec. 3 (lines 159–162) and the 64×6464\times64 single-object setting with a uniform background in Sec. 4, “Datasets” (lines 252–261). We will make this scope equally explicit in the abstract and introduction and add “A Controlled Study” to the title. The name GravJEPA reflects this gravity-based setting; the gravity vector is constant within a trajectory but varies across trajectories.

The current transition does not model collisions or general friction. Possible extensions include damping terms in the generator or event-dependent velocity updates for contact. For example, linear drag could enter through v˙=g−Dv\dot v=g-Dv, while a collision could update v+=v−+Δvv^+=v^-+\Delta v. These require suitable data and additional mechanisms; they are future directions, not demonstrated capabilities.

Additional PISA experiments provide initial evidence of real-trajectory transfer through object localization and canonical rendering. We selected 17 pre-contact clips with small objects, reliable SAM masks, and at least five input pixels of historical displacement. SAM-derived centers are rendered as canonical circles. Frozen models and their original synthetic-data readouts use seven observed frames to predict six future positions.

MethodADE ↓\downarrowFDE ↓\downarrow
Constant velocity2.3684.167
AR + EIR17.74333.680
GravJEPA0.8071.330
Constant acceleration (reference)0.7501.545

Errors against SAM-derived centers, in 64×6464\times64 image pixels. Bold excludes the constant-acceleration reference.

GravJEPA outperforms constant velocity and AR + EIR on this selected subset. The comparison with constant acceleration is inconclusive: paired confidence intervals for the error differences include zero. This experiment tests trajectory transfer after localization, not direct generalization to raw real-world RGB inputs or contact dynamics. Results and screening records: https://anonymous.4open.science/r/pisa-trajectory-results-FDCB/.

3. Translation equivariance in complex scenes

EIR constrains a shared change of image coordinates. As specified in Sec. 3.3 (lines 211–214), translation acts on the entire image, including its background, with shared parameters across frames. Applying the same translation to every frame shifts each object's image position while preserving temporal displacements and relative positions, subject to visibility and boundary effects. Background texture alone does not invalidate this relation. Translating one object relative to a fixed obstacle is a different operation: it may change interactions and is not the augmentation we use.

An object-level extension would require consistent object identities and coordinate-preserving masks or features; EIR alone does not provide object discovery or tracking. Camera motion additionally requires a suitable reference frame or motion compensation, and general perspective motion may require depth and camera geometry. These extensions are not validated by the present single-object experiments.

4. Photometric invariance: benefit and limitation

Photometric robustness and forecasting accuracy are distinct evaluation criteria. Sec. 4.6 (lines 470–475) discusses the observed trade-off: photometric consistency reduces current-position color drift from 0.44 to 0.24 pixels, but does not consistently improve future drift or FDE; accordingly, the paper interprets it as improving current position-representation robustness, without claiming a stable forecasting gain. Table 4a evaluates prediction and state readout, whereas Table 4b evaluates controlled recoloring, as specified in lines 451–452 and the table caption (lines 460–464).

The smaller ablation configuration is specified in Appendix G (p. 24, lines 1252–1255): 1,152 training trajectories (4,608 clips), 4,000 updates, batch size 64, and a two-cycle cosine schedule, distinct from the main configuration in Appendix A.5. We will summarize this distinction in the main text and clarify checkpoint-selection and aggregation details.

5. Implementation, training targets, and position evaluation

We clarify the implementation and evaluation details below, with references to the corresponding sections of the original submission.

Choice of dd. We use the first quarter of the embedding, d=D/4=4d=D/4=4 with D=16D=16, as specified in Sec. 3.1 (lines 165–168) and Appendix A.1 (lines 655–662); this is a design choice, not a claim of optimal dimensionality.

Orthonormality of QQ. QQ is the reduced-QR orthonormal factor of a learned matrix, as stated in Appendix A.3 (lines 725–729), enforcing Q⊤Q=I2Q^\top Q=I_2 up to numerical precision.

EIR losses. Sec. 3.3, Eqs. (7)–(8), defines the two required relations, and Appendix A.3, Eqs. (17)–(18) (p. 14, lines 725–748), gives their exact losses and augmentations: shared whole-image translations for Leq\mathcal L_{\mathrm{eq}} and shared photometric transforms for Lphoto\mathcal L_{\mathrm{photo}}, both acting only on pp. We agree that the essential loss formulas should also appear in the main text.

“Future features” at line 232. Z^core+\hat Z^+_{\mathrm{core}} contains the model-predicted core embeddings z^core(τ)=[p^(τ),R(p^(τ))]\hat z_{\mathrm{core}}(\tau)=[\hat p(\tau),R(\hat p(\tau))], before the memory residual, whereas Z+Z^+ contains the encoded actual future frames used as targets (Sec. 3.4, Eq. (9), and Appendix A.1, Eqs. (12)–(13)); future frames are not inputs to state inference (lines 243–244).

Position readout and R2R^2. As specified in Sec. 4, “Evaluation Protocol” (lines 281–288), and Appendix C.1 (lines 922–939, Eq. (24)), physical labels fit standardized linear ridge readouts after world-model training, on 576 calibration trajectories; the readouts are then frozen for test evaluation, and the labels do not train the world model. Table 1(b)'s caption (lines 279–280) identifies full-embedding versus component probes, and Appendix C.2 (lines 950–952) distinguishes observed-position R2R^2 from future-prediction accuracy.

ADE and FDE. ADE averages Euclidean position error over trajectories and query times, while FDE averages the final-query error. These definitions appear in the main-text evaluation protocol (lines 284–286), with formulas and pixel conversion in Appendix C.2, Eq. (25).

6. Dataset, timing, OOD ranges, and visualizations

Shapes and colors. Appendix B.3 (p. 17, lines 896–910) specifies circles, squares, equilateral triangles, the three sizes, and all eight RGB colors; we will add visual examples to make these settings easier to inspect.

Irregular observations. Irregular observation times test state inference with known variable temporal spacing, rather than assuming that frame index uniquely determines elapsed time; the uniform/irregular training mixture is stated in Sec. 4 (lines 258–261) and specified in Appendix B.4 (lines 914–917).

Horizons and smooth curves. The default is seven observed frames and six future queries at 55 ms intervals, with Href=0.33H_{\mathrm{ref}}=0.33 s (Sec. 4, lines 258–261); Sec. 4.3 (lines 318–320) defines the 2Href2H_{\mathrm{ref}} and 4Href4H_{\mathrm{ref}} evaluations. The continuous-time transition in Sec. 3.2, Eqs. (4)–(5), is compatible with discrete observations. We will specify the plotted query counts, physical horizons, and line-rendering procedure in the captions of Figs. 2 and 3 rather than leaving these to be inferred from visual smoothness.

OOD protocol. Appendix D (p. 18, lines 963–971) gives the exact ID/OOD ranges, paired construction, and common 536-history subset, excluding future visibility from filtering; we will bring the key ranges and filtering rule into the main text for easier assessment.

7. Presentation and main-paper completeness

We will add the missing in-text reference to Fig. 1, explain the GravJEPA name, make the controlled scope explicit in the abstract and introduction, state concrete findings in the contribution list, add visual examples, and clarify trajectory sampling and horizons in the captions. The dimensions, QR construction, readout protocol, ADE/FDE definitions, and data-generation details are summarized above with references to the original submission. We will improve their visibility and bring the essential loss formulas and protocol summaries into the main text. We will also include the quantitative intervention results alongside the response-error metric. These revisions will keep the central claims assessable from the main paper within the applicable page limit, with additional implementation details and analyses in the appendix.

Reviewer_z9zWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GravJEPA, a JEPA world model built on LeWorldModel, for videos of a single object moving with constant acceleration. The encoder gives an embedding where the first part is a position-like p. A small estimator infers velocity and gravity components from the history of p, and then a fixed constant-acceleration (Koopman form) transition rolls the state forward to any query time. They also add EIR, which is a translation equivariance + photometric invariance loss on p using known image transforms. On their synthetic 64x64 dataset it beats a LeWM-style AR baseline (with and without EIR) on ADE/FDE, magnitude OOD and long horizon, and they also show latent interventions on v and g.

Critical strengths

Experiments are careful, main table has 3 seeds with std. The intervention test (swap in v or g from a donor history and check if the predicted motion changes correctly) is a nice protocol, and it works well for velocity.

Critical weaknesses

The transition is basically hard-coded with the same physics that generates the data (constant acceleration, no collisions), so it is not very surprising that it does better than AR on OOD and long horizon. I’m not sure how much is really “learned” here, and the title feels a bit overclaimed. The setting is also very toy (64x64, one object, plain background), and the only baseline is their own AR model.

Also the gravity part is the weakest. g is not decoded that well (R² around 0.5), and editing g directly doesn’t change the prediction much. Most of the gain over AR seems to come from EIR, not from the structured predictor.

Questions for authors

What happens if the transition is learned instead of fixed? Any results on data that doesn’t follow the prescribed dynamics, like collisions or bouncing, or on a more realistic benchmark? EIR shifts the whole frame. How would this work on real video with background or camera motion?

2 comments

We thank the reviewer for recognizing the rigor of our experiments and our state intervention protocol. Before our point-by-point responses, we briefly restate the core problem and contributions of this paper. Our experiments show that, although the mainstream video JEPA models evaluated can encode decodable position and other dynamics-related information, their latent evolution generally deviates from basic dynamical laws, and their representations do not yet constitute physically consistent states. The key issue is that the feature-prediction objective constrains the match between predictions and target representations but does not explicitly specify the dynamical relationships among state components, leaving latent evolution underconstrained in its physical structure. Consequently, accurate predictions at observed timesteps are insufficient to guarantee consistency in long-horizon extrapolation and state interventions.

To address this problem, we use constant-acceleration gravitational motion as a controlled setting and explicitly embed a structured dynamical system into the forward prediction of a video JEPA. Together with EIR, data augmentation, and history fitting, this organizes state components with distinct dynamical roles, so that position-related information evolves according to explicit dynamical relationships. The resulting state transitions possess time homogeneity and temporal composition consistency by construction; we further evaluate the accuracy and usability of the visually learned states through prediction, extrapolation, and state interventions. The core contribution of this paper is to transform explicit dynamics from an external evaluation criterion into an internal evolution structure that visual representation learning must satisfy.

To our knowledge, at the time of submission, this was the first work to explicitly embed a structured dynamical system within the JEPA paradigm. Following our submission, LeHami [1] and Hierarchical World Models with Implicit Dynamics [2], both coauthored by Yann LeCun, explored Hamiltonian latent evolution and the separation of state-space reachability from low-level action execution, respectively. These independent developments resonate, from different perspectives, with our investigation of structured state evolution.

[1] Romina Yalovetzky, Randall Balestriero, and Yann LeCun. LeHami: Symplectic Latent Dynamics for World Models. arXiv:2610.15267, 2026. https://arxiv.org/abs/2610.15267

[2] Gaoyue Zhou, Yvonne Wu, Zichen Jeff Cui, Nicolas Ballas, Mido Assran, Lerrel Pinto, and Yann LeCun. Hierarchical World Models with Implicit Dynamics. arXiv:2610.42146, 2026. https://arxiv.org/abs/2610.42146

1 Learned versus fixed transition

Question (paraphrased). What happens if the transition is learned instead of fixed?

Learning an unrestricted transition leaves joint state and dynamics estimation underconstrained. Changes in the inferred state can be compensated by changes in the learned transition, so accurate predictions over the training horizon need not identify physically meaningful states or ensure reliable extrapolation. Fixing the transition restricts this freedom by requiring inferred states to support prescribed evolution relationships.

We compare a fixed kernel with a learnable kernel in an auxiliary synthetic-video experiment, changing only whether the kernel is learnable. We train for six-step prediction (H6) and evaluate both within this horizon and at twelve steps (H12).

MethodH6H12
Fixed kernel0.28214.4454
Learnable kernel0.256613.3187

Position MSE on the ID split (three-seed means; lower is better).

The learnable kernel slightly improves prediction within the training horizon but substantially worsens temporal extrapolation, supporting the use of a fixed kernel to restrict transition freedom.

2 Contributions beyond EIR

Concern (paraphrased). Most of the gain over AR seems to come from EIR rather than the structured predictor.

EIR cannot explain the full improvement: with the same EIR objectives, GravJEPA further improves long-horizon and OOD prediction, and better extrapolates trajectories beyond the camera view. EIR promotes spatially meaningful coordinates, while the structured predictor constrains their temporal evolution. AR + EIR and GravJEPA use the same encoder architecture and both achieve position-probe R2≈0.997R^2\approx0.997, yet their forecasting errors differ:

SettingAR + EIRGravJEPAReduction
HrefH_{\mathrm{ref}}5.404.2322%
2Href2H_{\mathrm{ref}}36.9117.9251%
4Href4H_{\mathrm{ref}}119.3176.4436%
Velocity OOD7.184.3140%
Gravity OOD5.784.4124%

FDE in pixels (lower is better); Href=0.33H_{\mathrm{ref}}=0.33 s. Long-horizon evaluation retains all trajectories, including those leaving the camera view.

In the out-of-view examples, GravJEPA continues the trajectory beyond the image boundary, whereas AR + EIR tends to keep predictions within or return them to the visible region. These results show that strong position decodability from EIR alone does not ensure accurate temporal evolution, and that structured prediction provides additional benefits for extrapolation and generalization.

3 Collisions, bouncing, and more realistic benchmarks

Any results on data that doesn't follow the prescribed dynamics, like collisions or bouncing, or on a more realistic benchmark?

Combined with SAM-based object localization, our frozen model can predict pre-contact falling trajectories from real PISA videos without retraining, outperforming constant velocity and AR + EIR. The comparison with the constant-acceleration reference is inconclusive.

MethodADE ↓\downarrowFDE ↓\downarrow
Constant velocity2.3684.167
AR + EIR17.74333.680
GravJEPA (ours)0.8071.330
Constant acceleration (reference)0.7501.545

Mean errors against SAM-derived centers in 64×6464\times64 image coordinates (pixels); bold marks the best results excluding the constant-acceleration reference.

We selected a subset of 17 PISA videos (small objects, reliable masks, at least five input pixels of historical displacement, and no contact or bouncing during evaluation). SAM-derived centers are rendered as canonical circles; seven input frames predict six future positions. Results, visualizations, and screening records are available at:

https://anonymous.4open.science/r/pisa-trajectory-results-FDCB/

Collisions and bouncing remain future work, requiring event-dependent velocity updates beyond the current constant-acceleration transition.

4 EIR with complex backgrounds and camera motion

EIR shifts the whole frame. How would this work on real video with background or camera motion?

Complex backgrounds do not invalidate EIR's translation relation; the key is to bind the constraint to the target object and reduce background interference. Dense features from self-supervised visual models such as DINOv3 can be combined with temporal motion information to construct object masks, distinguish moving targets from static backgrounds, and apply EIR to masked images while preserving their original image coordinates.

Camera motion also induces image motion in static backgrounds, requiring object motion to be separated from camera motion. One possible approach is to estimate camera motion from background feature correspondences across frames and express object positions in a common reference frame; significant parallax and complex 3D motion additionally require depth, camera poses, or appropriate geometric assumptions.

5 Scope and title

Our controlled constant-acceleration setting is designed to test whether visually learned states support prediction, extrapolation, and intervention under a prescribed dynamical structure. We will make this scope explicit by adding “A Controlled Study” to the title and clarifying the single-object, uniform-background, fixed-camera setting without contact or occlusion in the abstract. The additional PISA experiments provide initial evidence of real-video trajectory transfer through object localization and canonical rendering.

Comments on the paper

1 comment

Dear Reviewer,

As the author–reviewer discussion deadline approaches, we would appreciate it if you could review our responses and let us know if you have any remaining questions or concerns. We would be happy to provide further clarification before the discussion period ends.

Thank you again for your time and thoughtful feedback.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.