- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
GravJEPA, built on LeWorldModel, learns from video a latent state you can predict with and intervene on: a ViT encoder gives position-related coordinates, an estimator infers latent velocity and gravity, and a fixed constant-acceleration transition carries the state to any future time, with an equivariance regularizer tying coordinates to frame translations. On single-object videos it beats an autoregressive baseline (ADE 1.81 vs 2.45 px), and swapping the velocity or gravity component shifts predictions the way physics says it should.
Critical strengths
- The intervention experiments are the best part. Replacing only the inferred velocity or gravity moves the predicted trajectory in physically sensible ways through the same fixed transition (gain around 1.1 for velocity, with shuffled-donor controls), so these components actually do something in the model rather than just being readable after the fact.
- The evaluation is careful where it counts: paired OOD construction with visibility checks, correlation audits of the sampler, frozen ridge readouts, seed variance on the headline table, and query-cost accounting that excludes initialization honestly.
Critical weaknesses
- The gap between what is shown and what the title promises is large. One object on a gray background; no contact, no occlusion, no texture, no camera motion, no actions. "Learning gravity from video" here means reading a 2D acceleration off a seven-frame observation history, with the constant-acceleration law already built in. Whether any of this survives non-uniform backgrounds or natural video is simply not tested. I would take this as a controlled case study, and the title should say so.
- Gravity, the headline quantity, is also the weakest result. Probe against for position and velocity; scaling interventions have up to 0.419 and gains of 0.89–0.94; latent arithmetic on goes wrong up to ~0.78. With seven frames, velocity and gravity are identified through the same quadratic form, and the paper never really explains why comes out so much worse (noise in second differences of is the obvious suspect). That question sits right on the central claim.
Questions for authors
- Why is gravity so much less decodable than velocity while gravity interventions still behave? Is entangled with , and does the common-p diagnostic tell us anything about coordinate alignment?
- With a non-uniform background or several objects, what does actually enforce? Does it tie the method to single-object scenes?
- Could a learned residual transition (drag, contact) sit on top of the prescribed generator, or does the scheme need 's exact nilpotent form?
4 comments
2 EIR with non-uniform backgrounds and multiple objects
Question (paraphrased). What does EIR enforce with non-uniform backgrounds or multiple objects, and does it tie the method to single-object scenes?
EIR's geometric constraints remain valid with non-uniform backgrounds and multiple objects, and can be imposed on each object's state. A shared translation of all frames shifts each object's position equally, irrespective of background texture or object count, while leaving its velocity and acceleration unchanged. Thus, the same equivariance and invariance relations remain valid; applying them at the object level requires consistent object identity, which EIR alone does not establish.
Furthermore, dense features from self-supervised models such as DINOv3 could support object grouping and temporal association, providing a possible route to applying EIR to individual objects while preserving their original image coordinates. We have not yet evaluated this extension in complex-background or multi-object settings.
3 Residual dynamics and extension to additional physical phenomena
Question (paraphrased). Could a learned residual transition model drag or contact on top of the prescribed generator, or does the method require its exact nilpotent form?
Extending GravJEPA to more complex dynamics is feasible within its state-based formulation, either by extending the dynamics operator or by learning interventions on the existing state. Exact nilpotency enables our current quadratic closed-form propagation; it is not a prerequisite for this broader state-based formulation.
1. Extend the dynamics operator. For , we could learn a structured residual to the constant-acceleration generator :
\Delta A_\theta=\begin{bmatrix} 0&0&0\\\\-K_\theta&-D_\theta&0\\\\0&0&0 \end{bmatrix}.$$ Here $-D_\theta v$ represents linear damping and $-K_\theta p$ a restoring force. A constant generator still permits matrix-exponential propagation, but generally loses the exact quadratic truncation. Nonlinear effects may require richer Koopman observables or numerical integration. **2. Learn state interventions.** We could retain the existing core and learn how external effects update its state: collisions produce instantaneous velocity changes, $v^+=v^-+\Delta v_\theta$, whereas damping or friction supplies a state-dependent acceleration, $\dot v=g+a_\theta(s)$. Such forces continuously modify velocity rather than gravity itself. These routes overlap: linear damping can be represented either by a matrix term or by an acceleration intervention. Both routes require data containing the additional physical phenomena. The velocity- and gravity-related information already learned provides a starting point for continued training on such data, with the potential to extend physical coverage while retaining existing capabilities. ## 4 Scope and title We agree with the reviewer's suggestion to identify this work explicitly as a controlled study, and will revise the title to *Learning Gravity from Video: A Controlled Study of Structured Latent States for Prediction and Intervention*. We will also clarify in the abstract that the main study concerns a single object under constant acceleration, with a uniform background, a fixed camera, and no contact or occlusion. Here, learning concerns the inference and organization of dynamical states from video without physical-state labels; the constant-acceleration evolution law is prescribed.Thanks the authors for response
The gravity-recovery explanation and learnable-transition control make the contribution clearer, but the method is still evaluated only in a narrow controlled setting, so I will maintain my weak accept rating.
Thank you for taking the time to consider our response. We agree that evaluating trajectory prediction on real-world videos would strengthen the current study. In future work, we also hope to extend the approach to a broader range of physical dynamics.
We thank the reviewer for recognizing the rigor of our experiments and our state intervention protocol. Before our point-by-point responses, we briefly restate the core problem and contributions of this paper. Our experiments show that, although the mainstream video JEPA models evaluated can encode decodable position and other dynamics-related information, their latent evolution generally deviates from basic dynamical laws, and their representations do not yet constitute physically consistent states. The key issue is that the feature-prediction objective constrains the match between predictions and target representations but does not explicitly specify the dynamical relationships among state components, leaving latent evolution underconstrained in its physical structure. Consequently, accurate predictions at observed timesteps are insufficient to guarantee consistency in long-horizon extrapolation and state interventions.
To address this problem, we use constant-acceleration gravitational motion as a controlled setting and explicitly embed a structured dynamical system into the forward prediction of a video JEPA. Together with EIR, data augmentation, and history fitting, this organizes state components with distinct dynamical roles, so that position-related information evolves according to explicit dynamical relationships. The resulting state transitions possess time homogeneity and temporal composition consistency by construction; we further evaluate the accuracy and usability of the visually learned states through prediction, extrapolation, and state interventions. The core contribution of this paper is to transform explicit dynamics from an external evaluation criterion into an internal evolution structure that visual representation learning must satisfy.
To our knowledge, at the time of submission, this was the first work to explicitly embed a structured dynamical system within the JEPA paradigm. Following our submission, LeHami [1] and Hierarchical World Models with Implicit Dynamics [2], both coauthored by Yann LeCun, explored Hamiltonian latent evolution and the separation of state-space reachability from low-level action execution, respectively. These independent developments resonate, from different perspectives, with our investigation of structured state evolution.
[1] Romina Yalovetzky, Randall Balestriero, and Yann LeCun. LeHami: Symplectic Latent Dynamics for World Models. arXiv:2610.15267, 2026. https://arxiv.org/abs/2610.15267
[2] Gaoyue Zhou, Yvonne Wu, Zichen Jeff Cui, Nicolas Ballas, Mido Assran, Lerrel Pinto, and Yann LeCun. Hierarchical World Models with Implicit Dynamics. arXiv:2610.42146, 2026. https://arxiv.org/abs/2610.42146
1 Gravity recovery, functional use, and coordinate alignment
Question (paraphrased). Why is gravity less decodable than velocity despite useful interventions, and what does common-p reveal about coordinate alignment and state entanglement?
Gravity recovery is sensitive to position errors amplified by second-order differencing, while gravity contributes less than velocity to displacement over the current dataset's prediction horizon. High position-probe R2 therefore need not imply accurate gravity estimation, and accurate velocity can support good short-horizon prediction despite gravity errors.
The current ID prediction horizon spans 0.055–0.330 s; the corresponding displacement bounds are shown below (pixels).
Velocity accounts for the larger displacement throughout this horizon, but gravity is still used: zeroing g increases ID FDE from 4.23 to 7.46 pixels. Imperfect gravity estimates can still support effective interventions, while the marked drop in gravity decodability under common-p indicates incomplete alignment with position coordinates, but does not by itself establish entanglement between gravity and velocity.