acceptodds
Under review as a conference paper at ICLR 2027

Hamiltonian JEPA: Symplectic Latent Dynamics for World Models

45 views

Abstract

Joint-embedding predictive architectures (JEPAs) learn world models directly in latent space, but unconstrained autoregressive predictors may contract the latent distribution over long rollouts. We introduce Hamiltonian JEPA (LeHami), which replaces the standard predictor with an action-conditioned Hamiltonian flow over a structured latent state integrated using an exactly symplectic leapfrog step. This construction preserves phase-space volume at every rollout step and rules out uniform contraction, asymptotically stable fixed points, and merging of distinct latent states, independently of the learned parameters or action sequence. We obtain the momentum-like component from temporal derivatives of the learned position-like representation, providing an explicit second-order state for prediction and control. On conservative and actuated visual pendulum environments, LeHami substantially improves long-horizon velocity prediction over an unconstrained JEPA world model and empirically maintains constant latent volume where the baseline contracts. The learned flow also supports accurate integration at unseen sub-frame step sizes and numerical time reversal. These results show how geometric structure can provide inference-time guarantees that are complementary to representation regularization in latent world models. Code for LeHami is available at https://anonymous.4open.science/r/hamiltonian-jepa-7F08.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, joint-embedding predictive architecture, JEPA, Hamiltonian dynamics, physics-informed learning, self-supervised representation learning, and symplectic integration.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m2u2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LeHami (Hamiltonian JEPA), an action-conditioned latent world model framework designed to prevent distribution contraction and instability over long autoregressive rollouts. LeHami replaces the standard unconstrained predictor with a learned separable Hamiltonian flow over a canonical latent split (q,pq, p), integrated via an exactly symplectic leapfrog step. Momentum (pp) is derived via a zero-parameter Savitzky-Golay (SG) stencil on the encoder's position trajectory. The authors prove that this symplectic construction provably preserves phase-space volume and eliminates collapse modes (such as uniform contraction or asymptotic stability) at every rollout step. Experiments on conservative and actuated pendulum environments demonstrate that LeHami maintains constant latent volume (where baselines contract), improves long-horizon velocity prediction, and supports time-reversal and continuous-time re-stepping.

Critical strengths
  1. Well-motivated geometric framework for world models.

The paper introduces a principled Hamiltonian formulation for JEPA world models. The use of symplectic integration and structured latent dynamics is theoretically elegant and directly addresses the problem of long-horizon latent-space contraction.

  1. Strong theoretical and empirical consistency.

Theoretical guarantees regarding volume preservation, avoidance of contraction, and reversibility are clearly derived and empirically validated. The architectural choices are well aligned with the underlying mathematical framework.

Critical weaknesses
  1. The guarantee concerns ambient Lebesgue measure, not the data. • Pendulum data lives on a roughly 2-D manifold inside a 64-D latent space. Volume preservation in R⁶⁴ allows contraction along the data manifold, compensated off-manifold. • Fig. 4 (right) and Fig. 13 show log|det J| = 0 for LeHami, which holds for any symplectic net, even untrained. • LeWM's det < 1 is also not evidence of collapse, because off-manifold directions are never trained. The needed diagnostics are the Jacobian restricted to the data tangent space, rollout effective rank, or pairwise distances between rollouts from distinct anchors, to test the "merging" claim.

    1. It doesn't prevent collapse, and Table 8 suggests near-collapse. • A constant encoder with ∇V(c,u)=0 gives a neutrally stable fixed point, which Prop. 1(ii) does not exclude. • With the default q-only SIGReg, only 15% of latent variance is within-episode, against 42.8% for the baseline. The caption says near zero means "an episode identifier rather than a state". Joint SIGReg gives 0.01%. • std(p)/std(q) is about 0.10, so the principled regularization of (q,p) fails, and p is effectively unregularized.

    2. No ablation isolates symplecticity. • Missing controls: (a) LeWM plus the same SG momentum channel and 4-frame anchor; (b) a plain MLP predictor on (q,p,u) with matched parameters; (c) LeWM at d=32. • The velocity gain at K=0 comes from giving LeHami's encoder a 4-frame stencil while the baseline is deliberately per-frame. • The FLOP savings come from MLP vs. 6-layer transformer, not from Hamiltonian structure. • Capacity is also unmatched: p is a function of q, so LeHami has 32 free dimensions against 64, which plausibly explains its 6–16× worse angle error in Table 1.

    3. Planning is negative, and the framing is generous. • Table 2: LeHami succeeds 0.48/0.53/0.30 on the three clauses (position/velocity/both), against 0.80/0.80/0.64 for the best LeWM variant. • The tied kernel is a strawman, since no one would use it for velocity. Untied is the real baseline, and it wins at 13.65 vs. 10.98 TFLOP. • Matching budgets by varying R per model confounds the comparison, so compute-matched runs at equal R are needed. • With std(p) about a tenth of std(q), the latent goal cost barely weights velocity. Block normalization is untested and may explain the velocity gap.

    4. Long-horizon claims lack chance baselines. Wrapped angle MAE of 0.6–1.0 rad and ω MAE above 2 rad/s at K>20 are likely near trivial predictors for both models. Add a predict-mean or persistence baseline. • A conservative model can keep the right energy yet drift in phase, so "stays on its level set" (by construction, Figs. 10–11) says little about accuracy. • Attributing LeWM's errors to "compounding contraction" is not tested.

    5. Time reversal and re-stepping are properties of the integrator. Strang splitting satisfies Ψ₋ₕ = Ψₕ⁻¹ exactly for any H, so the roughly 10⁻⁵% round-trip error (Fig. 5 right) is floating-point precision. The meaningful test is backward-rollout accuracy against held-out earlier frames. Sub-stepping consistency shows convergence of the integrator, not correct physics at unseen frame rates. The mixed-rate experiment is described but not run.

    6. Scope. • There is one 1-DoF pendulum whose Hamiltonian is exactly separable, which perfectly matches the prior. The unit-mass assumption p = q̇ fails for M(q), as the authors admit. • No dissipative, noisy or contact environment is tested, even though real systems have friction and stabilizing controllers. Ruling out stable fixed points can be a bug for world models. • The title says "world models", but the evidence is one toy system with 210 training episodes.

    7. Statistics. There are three seeds, but Table 4 (the "one-eighth compute" claim) is a single seed with non-monotone LeWM numbers (0.590 → 0.871 → 0.487), so it is fragile. It also uses a k=20 probe where the rest use k=25. Fig. 13 is one seed.

Questions for authors

Can the authors demonstrate LeHami on substantially more complex control and visual world-model benchmarks beyond pendulum systems?

How sensitive is the framework to the quality of the estimated momentum variable obtained via the fixed Savitzky-Golay derivative estimator?

Can the authors provide a quantitative analysis relating latent-volume preservation to rollout prediction accuracy across multiple environments?

What are the results with a non-symplectic MLP on (q,p,u), and with LeWM plus the SG channel?

What is the contraction on the data tangent space, not ambient?

Why does q-only SIGReg leave 85% of variance between-episode?

LeHami argues that a contractive deterministic rollout is a collapse mode, while LIFS-JEPA [1], enforces contraction for stability. [1]Belloulata, 'Contractive MASO-Generalized Predictors for Stable Latent-Space Learning in JEPA, in Transactions on Machine Learning Research (TMLR), 2026, https://openreview.net/forum?id=k2Z2gPOtlq.

The reconciliation is that the right predictor class depends on the dynamics. For stochastic or dissipative processes such as OU, the conditional-mean map should contract. For deterministic conservative systems it should not. Stating what is contracted, state volume versus error dynamics, would pre-empt the objection.

Could the authors discuss the expected advantages and limitations of the two approaches with respect to rollout stability, latent-space structure, and long-horizon prediction accuracy?

6 comments
AuthorsOpenReview ↗

Response to m2u2 (1/3): collapse on the data, Table 8, the ablation

3.1 The guarantee is about ambient measure, not the data.

Tables A–D are in our second general comment (Table A = revised Table 1; Tables B, B' = revised Table 2). Changes are in blue in the revised PDF; "revised" refers to it, plain numbers to the submission.

You are right: Prop. 1 holds for any leapfrog step of any H, trained or random, and says nothing about the roughly 2-D surface the data occupy; LeWM's det J < 1 is not by itself evidence of collapse, and the revised paper drops that panel.

What we measure instead. Each measure compares the rollouts at step K with the truth, the encoder's latents of the true frames at the same step, as a ratio (1 is right, below 1 the rollout contracts):

  • Distance between rollouts started from different frames, over the distance between their true latents.
  • Tangent area. The latents of real frames lie on a thin surface with as many dimensions as the physical state. An anchor and its nearest true neighbours form a small patch of it. We measure how much the rollout stretches or squeezes this patch after K steps, over the same for the true latents, per direction.
  • Spread: the root-mean-square distance between all rollout states at step K, over the same for the true latents.
  • Effective rank: after subtracting the mean of the rollout states at step K, with σi\sigma_i their singular values, (∑iσi)2/∑iσi2(\sum_i\sigma_i)^2/\sum_i\sigma_i^2 counts how many directions they effectively span; we divide by the same count for the true latents.

Results (Table A). Both free predictors contract on every environment and measure. LeHami's rollouts never merge: their distance and spread stay at or above 1; where its tangent area or rank falls below 1 (two-body, cart pole), it squeezes some directions and stretches others. On plain LeWM you are right (Table D): its rollouts keep their distances and rank; its tangent area falls below 1 only on the conservative pendulum (0.78) and the cart pole (0.76). Trained or not: with random weights LeHami keeps the whole latent's volume exactly, yet its tangent area ranges from 0.32 to 5.09: the guarantee alone says nothing about the data. Revised Fig. 4 (right) shows it building up with K: the distance between rollouts falls steadily for the two unstructured predictors and stays at 1 for LeHami.

3.2 Near-collapse in Table 8.

Both points are right.

(a) A latent that does not move. Prop. 1 rules out contraction of the whole latent, not a latent that stands still (a constant encoder with ∇V=0\nabla V = 0). Revised Section 3 now says so.

(b) What the 85% is. Each actuated episode has one of nine (mass, length) settings. 85% of the variance of q is its average over each episode, which encodes the setting; only 15% is the motion within the episode. That motion lies almost on a straight line in the latent (one direction holds 99.2% of it). A swing fits on such a line, but 30% of test episodes go over the top and turn full circles, where the angle wraps around, which a line cannot represent. This, not a loss of volume, is LeHami's failure mode on the actuated set.

(c) Why std(p)/std(q) ≈ 0.10. p is a time derivative, so it has no episode average and only sees the motion within episodes: std(p)/std(q)=ρwwq=0.267×0.151=0.104\mathrm{std}(\mathbf{p})/\mathrm{std}(\mathbf{q}) = \rho_w\sqrt{w_q} = 0.267\times\sqrt{0.151} = 0.104 (Table 8's 0.10, now revised Table 10), with wqw_q the within-episode share of the variance of q and ρw\rho_w the same ratio within episodes. So p is not collapsed: it looks small only next to the episode averages that q also carries, and within episodes its ratio to q (0.27) matches the data's own (0.24). The real cost is in the loss: the squared error weights p by ρw2≈0.07\rho_w^2 \approx 0.07 of q. Two fixes would fit well, but we had no time to run them: scaling q and p by their within-episode spread, and keeping the episode constants out of the flow, V(q,u,c)V(\mathbf{q}, \mathbf{u}, c).

3.3 No ablation isolates symplecticity.

We ran the first two controls, on five environments with three seeds. All three models share the paper's per-frame ConvNet, the 32-dimensional q, the state [q, SG(q)], the losses and the 8-step rollout; only the predictor differs: (i) LeWM's transformer, i.e. LeWM with the SG momentum channel; (ii) a parameter-matched MLP; (iii) LeHami's Hamiltonian step. Results: Tables A, B, B' (second general comment) and revised Figs. 4, 17, 18.

Capacity: the controls learn the same 32 dimensions as LeHami, so latent capacity is matched and we did not run LeWM at d = 32. Stencil and predictor: at K = 0 the probe reads the encoded state, so we agree the K = 0 velocity gain comes from the stencil. The rollout gain does not: with the same stencil, the free predictors contract on all five environments (Table A), and on the conservative pendulum their velocity error at K = 10 is 2.5 to 3 times LeHami's (Table B').

AuthorsOpenReview ↗

Response to m2u2 (2/3): planning, chance baselines, backward rollout

3.4 Planning results are negative.

Tables A–D: second general comment. Changes: in blue in the revised PDF.

We agree. Planning is no longer a main result: revised Section 4.5 is one paragraph presenting it as an application with its limits, and the table moves to the appendix (revised Table 4), where the untied LeWM, which captures both the sign and the size of the velocity, is the strongest baseline. LeHami plans at a lower cost per step; it does not reach goals better. On equal R: at the same budget both were run at R = 9 (4 solves) on the position clause (0.48 against 0.80) and on the sign of the velocity (0.80 against 0.96 for untied LeWM; App. Table 10, now Table 12), so its lower success there is not due to R. The goal cost under-weights p for the reason given above for Table 8; normalising each block would be a good fit, but we did not have time to try it. Future work can use integration at a coarser step and planning backward from the goal.

3.5 Long-horizon claims lack a chance baseline.

We added both baselines you propose, repeat the start (persistence) and predict the mean, on all five environments: Tables B and B', and the chance line in revised Figs. 4 and 18. Partly right: on the pendulums at K = 70 the velocity errors are close to chance for every model except LeWM W=3; LeHami is below it, narrowly (2.19 against 2.50; 2.28 against 2.54). In position LeHami beats chance at every step shown; at K = 70 on the conservative pendulum the free predictors are worse than chance and the paper's LeWM is at chance (1.24, Table D). Energy and phase: we agree that energy conservation says little about whether a rollout is in step with the truth; our errors score the decoded angle and velocity at each step, so a rollout that runs ahead or behind is penalised. Compounding contraction is now tested on the data, at every step (Table A, revised Figs. 4 and 17).

3.6 Reversal and re-stepping are integrator properties.

We agree the round trip is an integrator identity, now presented as an implementation check. We ran the test you propose, and the revised paper now includes it (Fig. 5, right, and App. F.6): from a mid-episode frame of a held-out episode, roll LeHami back K steps (with the earlier torques when actuated) and score against the true earlier frames. Conservative pendulum: backward matches forward in angle and velocity, and both beat both baselines up to K = 20 (angle at K = 20: 0.51 backward, 0.53 forward, against 1.05 and 1.27). Actuated pendulum, angle [rad], MLP probe, mean ± std over three seeds (the baselines use no model):

angle θ [rad], Kforwardbackwardrepeat the start (backward)predict the mean (backward)
10.077 ± 0.0090.074 ± 0.0060.2421.047
50.343 ± 0.0300.332 ± 0.0311.1201.041
100.531 ± 0.0360.506 ± 0.0211.4871.025
200.716 ± 0.0500.684 ± 0.0961.0381.004

The angle is recovered as well backward as forward; the velocity beats chance up to 10 steps. A free predictor has no backward step. We did not run the mixed-rate experiment. It matters for planning and control: a model that integrates at any step size can be trained on data recorded at several rates (Doshi et al., 2024) and can plan on non-uniform time grids, fine near the present and coarse further ahead (Tan et al., 2016).

AuthorsOpenReview ↗

Response to m2u2 (3/3): scope, statistics, LIFS-JEPA, citations

3.7 Scope and assumptions.

Tables A–D: second general comment. Changes: in blue in the revised PDF.

We accept each point and narrow the claims. Separable, constant mass: a constant mass is absorbed by the latent coordinates; it may differ between episodes, but not within one (reply to 4fNR). A configuration-dependent M(q) is not: we added the cart pole to test LeHami where this assumption fails. Its rollouts still do not merge, but it is less exact (Table A). One system: we now have five. Dissipation, noise, contact: not covered; for friction a volume-preserving step is the wrong prior, and the extension is a conformal or port-Hamiltonian step (Limitations). Stable fixed points: a stable fixed point is a state that nearby states converge to, like a pendulum with friction coming to rest at the bottom. Ruling them out is a defect for dissipative or stabilised systems, and correct for conservative ones, where nearby states never converge; it is what stops distinct states merging. Title: we keep "world models", and the revised abstract and introduction state the scope.

What we mean by a world model: a model that predicts how a system evolves, including under the agent's actions, far beyond its training horizon (Ha & Schmidhuber, 2018; LeCun, 2022). Every use needs long rollouts that stay close to the true states. We test that four ways: prediction to 70 steps against chance, four measures of collapse (Table A), running backward to the true past, and keeping the system's properties (the mass, R² ≥ 0.98 after 70 steps). LeHami supports actions by construction: they enter the potential, so every step stays symplectic under any action sequence; the last two tests are on the actuated pendulum, under the recorded torques. Planning is one application: it has limits, but shows that a symplectic Hamiltonian predictor can be used to plan actions.

3.8 Statistics.

Table 4 at three seeds (conservative pendulum, rollout step K = 20, 25-NN probe; mean ± std over three seeds); it replaces App. Table 4 (now revised Table 5):

model0.75 PFLOP1.5 PFLOP3.0 PFLOP6.0 PFLOP
MAE θ [rad]LeHami0.304 ± 0.0310.213 ± 0.0080.247 ± 0.0130.207 ± 0.041
LeWM0.642 ± 0.0480.710 ± 0.1400.462 ± 0.0250.332 ± 0.020
MAE ω [rad/s]LeHami1.045 ± 0.0930.939 ± 0.0620.916 ± 0.1280.854 ± 0.075
LeWM1.604 ± 0.0231.694 ± 0.4081.345 ± 0.3151.178 ± 0.132

LeHami has the lower error at every budget. The non-monotone LeWM number was one seed (new seeds: 0.632, 0.625 at 1.5 PFLOP). "One-eighth compute" holds as "on par" (0.304 rad at 0.75 PFLOP against 0.332 at 6.0). The 20 in Table 4 is the rollout step, not the probe's neighbours; the revised caption says so. All new results are at three seeds; Fig. 13 (one seed) is replaced by Table A.

3.9 Relation to LIFS-JEPA (Belloulata, 2026).

Thank you for bringing this work to our attention. LIFS-JEPA (Belloulata & Belalia, 2026) is a JEPA for static images. Its predictor refines one prediction by iterating a contractive map towards a fixed point: the iterations are refinement steps, and converging is the goal. In a world model each iteration is a step in time, and a contracting predictor makes rollouts from different states converge: the collapse we measure. So the right predictor class depends on what the iterations mean; revised Section 2 now says so.

3.10 Citation concern.

We thank the reviewer for bringing these works to our attention. We now cite SympNets (Jin et al., 2020), the general class of symplectic networks; our leapfrog step is one of them, chosen because it preserves volume, which we use against rollout collapse, while SympNets target learning Hamiltonian systems. LNN (Cranmer et al., 2020) is cited in revised Section 2 and App. C.3, where we explain choosing the Hamiltonian over the Lagrangian form. For HamJEPA, see our reply to the public comment.

Reviewer_4fNROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces LeHami, a World Model whose latent is embedded into a separable Hamiltonian of form addressed in Eq (3). L200-201. The idea is to use stencil to approximate costate pp, combining with state qq encoded from the image/video input, construct a latent Hamiltonian system, and integrate it with leadfrog scheme, so that conservative part can be properly learned and the control signal injects into potential which makes the latent model aware of the environmental factor. Authors acompared with LEWM to show it is a good model that can predict goal-reaching actuated pendulum system better than prior works.

Critical strengths

An end-to-end approach to realize a latent structural Hamiltonian dynamical model which encodes the symplectic dynamics part to make the long horizon prediction accurate at its best extent.

Critical weaknesses

Why symplecticity in latent space matters? This is my biggest concern after reading. I understand authors stated in L145-152 and emphasize this is a volume preservation based inductive bias in latent dynamics rather than focusing on physically informed dynamics, but it does not explain why it is relevant in constructing world models.

There are some additional specific concerns regarding necessity of symplectic structural dynamics. First of all, table 1 indicates in a primal task (i.e. predicting θ\theta) LeHami is worse than LeWM and it only when predicting ω\omega it shows some advantage. One could also use SG stencil and let world model learn an augmented state (q,p)(q,p), which might also work in predicting angular momenta ω\omega from pixel input. There is no such experiment. Secondly, if there exists an actuated component, then the dynamics is not reversible due to external forces. The role of input source of control uu might break the statement of reversibility of dynamics, as the potential is a function regarding uu, which is implicity time-dependent. Lastly, the scope of experiment is still a toy pendulum problem where only a piecewise constant actuator is considered. No additional cases or real-world examples is given so I cannot see the advantage of introduce symplectic integration in latent Hamiltonian dynamics.

Questions for authors
  • What is authors' opinion regarding Toth et al., 2019, this 2021 NeurIPS paper and this recent LDR arxiv paper? Can LeHami extend to some of the experimental data provided in these papers? If not why?

  • Could authors compare LeHami with an unconstrained predictor (e.g. Mamba, GRU) using exactly the same encoder, augmented state zt=(qt,SG(qt))z_t=(q_t,\mathrm{SG}(q_t)), prediction targets, regularization, and training horizon? This would isolate the benefit of the Hamiltonian constraint from the benefit of supplying an explicit augmented state.

  • Author mentioned mass in latent dynamics can be identity (L877-897). How does this choice relate with video generated via different mass of balls with same articulated torque?

9 comments
AuthorsOpenReview ↗

Response to 4fNR (1/2): why symplecticity, the ablation, reversibility

2.1 Why symplecticity in a latent space matters.

Tables A–D are in our second general comment (Table A = revised Table 1; Tables B, B' = revised Table 2). Changes are in blue in the revised PDF; "revised" refers to it, plain numbers to the submission.

A world model is trained on short clips (8 steps here) and used far past that horizon, feeding its predictions back as inputs, under actions not in the training data. There, nothing in training constrains what the predictor does.

Where the rollout error comes from. Let zKz_K be the encoded frame K steps after the start and z~K=PredK(z0)\tilde z_K=\mathrm{Pred}^K(z_0) the rollout. With one-step errors εj=Pred(zj)−zj+1\varepsilon_j=\mathrm{Pred}(z_j)-z_{j+1} and Jacobians Ji=∂ Pred/∂zJ_i=\partial\,\mathrm{Pred}/\partial z at ziz_i, to first order

z~K−zK≈∑j=0K−1JK−1⋯Jj+1 εj.\tilde z_K-z_K\approx\sum_{j=0}^{K-1}J_{K-1}\cdots J_{j+1}\,\varepsilon_j .

Each error is carried by all later Jacobians. A contracting product shrinks the errors, but also the motion: two rollouts, z~Ka−z~Kb≈JK−1⋯J0 (z0a−z0b)\tilde z^a_K-\tilde z^b_K\approx J_{K-1}\cdots J_0\,(z^a_0-z^b_0), can become indistinguishable, and e.g., give different action sequences the same planning cost.

Why free predictors can contract. One step multiplies the volume of a small cloud of latent states by ∣det⁡Ji∣|\det J_i|. Nothing in an MLP or a transformer constrains it, and beyond the 8 trained steps nothing in the loss does: 0.99 per step is invisible in training (0.998≈0.920.99^8\approx0.92) and is already 0.6 after 50 steps (0.9950≈0.60.99^{50}\approx0.6), and keeps falling. Encoder regularizers (SIGReg, Balestriero & LeCun, 2025; VICReg, Bardes et al., 2021) act on the encoder, not on the predictor.

What symplecticity gives. det⁡Ji=1\det J_i=1 at every step, for any step size, action and learned H (Prop. 1): no uniform contraction, no attractor, no merging. Without actions, the learned energy is kept up to a small error that does not grow (Hairer et al., 2003). The step can also be split into substeps and has an exact inverse. It does not bound the error, and it holds in the whole latent, not on the data (m2u2's point), so we measure collapse on the data. This argument is now in revised Section 3.

Evidence. With the same encoder, state and training, both free predictors contract the rollout on all five environments; LeHami's rollouts do not merge, and it is the most accurate of the three in position at the long step on four (Tables A, B).

2.2 LeHami outperforms in velocity only and no ablation to isolate Hamiltonian predictor from the augmented encoded state.

In rollout, Fig. 4 shows LeHami predicting angle and velocity better than LeWM. Table 1 is at the anchor, with no rollout, and shows a trade-off: the latent is split between q and p.

We ran the experiment you ask for: LeWM's transformer and a parameter-matched MLP on the same stencil and state [q, SG(q)], on five environments, three seeds. At K = 0 all three read the same state. In rollout they contract while LeHami does not (Table A), and on the conservative pendulum at K = 10 their velocity error is 1.42 (MLP) and 1.16 (transformer) against 0.47 rad/s (Table B'). So the rollout gain comes from the Hamiltonian step, not from the stencil. On two-body and the cart pole the free predictors are as accurate or better in velocity, but they still contract. The revised Fig. 4 shows this comparison against chance.

2.3 Reversibility under actuation.

We agree that time-reversal symmetry fails under actuation. What holds is invertibility.

The physical system under a known control history is invertible. For H=K(p)+V(q,u(t))H=K(\mathbf{p})+V(\mathbf{q},u(t)) the flow is symplectic and a diffeomorphism; its inverse is the same equation run backward with the same u(t)u(t). Energy is not conserved, but non-conservation is not non-invertibility: dissipation breaks invertibility in practice, and a known external force is not dissipation.

The latent step is its own inverse under reversed actions. One step is a leapfrog ΨΔt(⋅;u)=AΔt/2(u)∘BΔt∘AΔt/2(u)\Psi_{\Delta t}(\cdot;u)=A_{\Delta t/2}(u)\circ B_{\Delta t}\circ A_{\Delta t/2}(u) with u fixed, so for every learned H

(ΨΔt(⋅;uK−1)∘⋯∘ΨΔt(⋅;u0))−1=Ψ−Δt(⋅;u0)∘⋯∘Ψ−Δt(⋅;uK−1).\big(\Psi_{\Delta t}(\cdot;u_{K-1})\circ\cdots\circ\Psi_{\Delta t}(\cdot;u_0)\big)^{-1}=\Psi_{-\Delta t}(\cdot;u_0)\circ\cdots\circ\Psi_{-\Delta t}(\cdot;u_{K-1}).

This is what Fig. 1(iii) means by "reversing actions". The round trip of Fig. 5 checks it in floating point and is now presented as an implementation check.

Does the backward rollout land on the true past? That depends on the learned H, so we tested it (table in our reply to m2u2): on the actuated pendulum, run backward with the recorded torques, LeHami recovers earlier angles as well as it predicts later ones, far better than chance, showed in the revised Fig. 5. A free predictor has no backward step. "Time-reversibility" is now "invertibility given the action history" (revised Introduction, Fig. 1 and Section 4.4).

AuthorsOpenReview ↗

Response to 4fNR (2/2): scope, prior datasets, matched predictor, unit mass

2.4 Scope.

Tables A–D: second general comment. Changes: in blue in the revised PDF.

The submission had two pendulums (conservative, three rod lengths; actuated, three lengths × three masses) and two encoders (a ConvNet, frozen DINOv2). Each torque is held for 3 frames (0.3 s; App. D.3.2). This is a choice of the data, not a requirement: the model only needs the action constant within a step, so a continuous actuator is held at one value per step, e.g. its mid-step value, which keeps the step symplectic, invertible and second order. We added the mass-spring and the two-body of the DeepMind suite and a cart pole (Towers et al., 2024) with two degrees of freedom whose mass depends on the pole angle; four other models, three seeds. Not covered: real video, contact and friction. A volume-preserving step is the right prior for conservative dynamics and actuation through the potential, and the wrong one for friction (App. C.4 gives the dissipative extension).

2.5 Opinion regarding prior work and their datasets.

HGN (Toth et al.) learns from video as a generative model: it decodes every step to a frame and scores pixels. LeHami has no decoder, puts the action in the potential, and is scored by the physical state read from the latent and by whether the rollout contracts. SyMetric (Higgins et al.) measures whether a model learned from pixels recovers the true Hamiltonian dynamics on the DeepMind suite. LDR (Li et al., 2026) is a video diffusion world model that integrates lower-order kinematics explicitly; it shares our view that an explicit integrator helps a rollout extrapolate, but has no symplectic structure and generates pixels.

Their data. LeHami only needs video, and actions if any. The suite's conservative environments are good targets: we ran the mass-spring and the two-body; its pendulum was covered by our submission, which adds actions. LDR uses PhyWorld's tasks: collision and bouncing involve contact, which a smooth Hamiltonian step does not represent; uniform motion, parabola and looming are within scope, but we did not run them due to time limitations.

2.6 Unconstrained predictor with matched everything.

We ran it: same encoder, state [q, SG(q)], targets, regularizer and horizon, with three predictors: LeHami's Hamiltonian flow, a parameter-matched MLP and LeWM's transformer. Both unstructured predictors contract the rollout on all five environments; LeHami's rollouts do not merge, and it is the most accurate of the three in position at the long step on four (Tables A, B). We chose these over a GRU or Mamba, which add memory of past states: [q, SG(q)] already carries position and velocity, so a memoryless MLP isolates the Hamiltonian step, and the transformer is the baseline's own predictor.

2.7 Unit mass and varying ball mass.

Unit mass is a choice of coordinates. Let q be the physical coordinate (the angle for the pendulum), with mass m and momentum p=m dqdt\mathbf{p} = m\,\frac{d\mathbf{q}}{dt}. LeHami's momentum is the time derivative of its latent coordinate, which amounts to unit mass. Nothing forces that coordinate to be q: the encoder can learn a stretched one, q′=c q\mathbf{q}' = c\,\mathbf{q}, whose momentum p′=dq′dt\mathbf{p}' = \frac{d\mathbf{q}'}{dt} describes the same motion with unit mass. Matching the kinetic energies, 12(dq′dt)2=12 m(dqdt)2\frac{1}{2}\left(\frac{d\mathbf{q}'}{dt}\right)^2 = \frac{1}{2}\,m\left(\frac{d\mathbf{q}}{dt}\right)^2, gives c=mc = \sqrt{m} (for the pendulum, c=m Lc = \sqrt{m}\,L). The stretch keeps Hamilton's equations, so LeHami's step stays valid in q′\mathbf{q}', and it can differ between episodes: the encoder sees the mass in the frame, so one model learns from episodes with different masses, as long as each is constant within its episode. Unit mass rules out a mass that depends on the configuration, M(q)M(\mathbf{q}): the cart pole is such a system, and there the current implementation of LeHami is less exact (Table A).

Different masses, same torque. Without actuation the pendulum's motion does not depend on the mass, so only the actuated set varies it (three masses, three lengths). The bob is drawn with a constant density, so a heavier bob is larger and its mass is visible in one frame. The same torque must move a heavy bob less, so V(q′,u)V(\mathbf{q}', u) must depend on the mass; it can, because the latent carries it. R² of the bob mass, read from the rollout by a linear probe fitted on the training episodes, on held-out test episodes (mean of three seeds; largest std 0.023):

R² of the massK = 0K = 10K = 30K = 70
LeHami0.9990.9990.9990.984
LeWM W=10.9990.9930.9880.979
LeWM W=30.9980.9950.9910.982
MLP, matched0.9990.9940.9930.993
Transformer (LeWM+SG)0.9990.9960.9950.995

Every model keeps the mass; unit mass does not stop LeHami from carrying it (also App. Table 11, now Table 13).

I appreciate authors detailed response and dedication to a revision. Overall it's a positive direction to help readers understand the idea on why preserving volume is relevant and keeping symplectic integration can achieve that as symplectic structure is more restrictive than volume preservation. The response 2.1, 2.3, 2.4, 2.5 and 2.7 resolved the concern. I have some further up comments regarding 2.2 and 2.6:

2.2 On two-body and the cart pole the free predictors are as accurate or better in velocity, but they still contract. The revised Fig. 4 shows this comparison against chance.

I appreciate the faithful report to the failure cases, but I prefer author explicitly state on why cartpole task cannot being integrated via leapfrog but only RK4, which is inconsistent with your choice of numerical integrator. Plus, if these two complex system result is not promising, does that imply LeHami has its limitation that for a realistic complex or articulated system, it is much more harder to maintain volume conservation property. The intuitive reason might be the following: since one inevitably infer velocity or moments from data and needs to encode the video observation, modelling and measurement error cannot be solely absorbed via a latent conservative Hamiltonian system.

2.6: We chose these over a GRU or Mamba, which add memory of past states.

Maybe I was not stating it clearly. In order to predict, sometimes non-stationary process or past memory as additional actuation signal might help (maybe not in simplest single pendulum cases). I hope to see authors rule out such possibility, but it is fine to state MLP and transformer results. Another comment (take this as a grain of salt) is to also defense on the choice of SG stencil as there are multiple ways to predict velocity, restricting on SG setup can isolate the effectiveness of LeHami model choice but for solving a general task why limit the choice of different models.

I do not have time to fully revise the revision manuscript as the revision has changed a lot, but I still need to state some apparent formatting suggestions:

  • Notion k,Kk,K are abused, sometimes they are kinectic energy sometimes they are rollout steps and sometimes kk and KK seems mixed up. Please consider properly renaming and unify notations.
  • L1325 is confusing.
  • Table 5 is not within margin.

My biggest concern is still that from the ideal experiments, how can one state LeHami is a contribution to develop more accurate world model, where it shall learn to predict the consequences of actions in the environment.

Reviewer_VHitOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce a variant of JEPA where the latent space transformation is given by a fixed symplectic integration of an input-dependent learnable hamiltonian. The postion-momentum encoder along with the parameters of the kinetic (input independent) and potential (input dependent) terms of the hamiltonian are learned to minimize a JEPA-style predictive loss of future encoded frames. They argue that the physical bias of the symplectic update provides benefits of: preventing representation collapse; flexible step-size rollout; and guaranteed time-reversibility. They provide a formally proved proposition on these claims and validate them empirically. They demonstrate slightly improved performance over a comparable model, LeWM (a JEPA baseline with identical structure except for the transition function) according to some performance metrics, but slightly worse performance in other settings. They demonstrate improved computational efficiency and recovery of ground truth physical variables in some settings.

Critical strengths
  • The authors develop some unique tricks for integrating symplectic integration and a learned hamiltonian into the JEPA framework (including the Savitzky-Golay stencil).
  • They perform a thorough empirical analysis of their model and the purported analytic properties on a limited dataset, including critical design ablations (e.g. where to place the SigReg regularizer).
Critical weaknesses

The novelty and significance of the work are its critical weaknesses.

With respect to novelty, while implementing a Hamiltonian-based integration scheme into JEPA is undeniably novel, the general idea of hamiltonian constrained representation learning is not new, yet the paper does not address some key related work. My suggestion to the authors is that the related work should be made significantly more prominent in the main text. Specifically, that of Greydanus et al., 2019 "Hamiltonian Neural Networks", and Toth et al 2019, "Hamiltonian Generative Networks". These works are so highly related that a single sentence in the text to discount them is insufficient in my opinion. Even if the authors do not believe they are related, they should at least be acknowledged by name more explicitly in order to avoid confusing readers who may have heard of the prior work. Furthermore, I believe some key citations are missing:

In terms of significance and impact, the empirical validation of the model is limited, and I encourage the authors to test their model on a broader range of datasets and compared to a broader range of models (perhaps, for example, comparing on the DeepMind Hamiltonian Dynamics Benchmark Suite: https://github.com/google-deepmind/dm_hamiltonian_dynamics_suite). Currently, the model is only compared against a single JEPA baseline (LeWM) and only on a single physical system (pendulum). While I greatly appreciate the physical inductive prior being introduced, I believe that a larger experimental suite is necessary to validate the benefits of the model. Furthermore, the performance of the model relative to the baseline is not dramatically improved. For example, one of the main results figures, Figure 4, demonstrates what looks like marginally improved performance for a system which the model should be optimally biased to represent: a conservative pendulum. Compared with prior work (e.g. Hamiltonian Generative Networks) which was able to forecast similar systems for very long horizon, this does not inspire much confidence; but likely there are significant differences in the dataset and evaluation methodology that I did not look into. Similarly, Tables 1 and 2 both seemed to show mixed performance compared to the baseline LeWM, perhaps gaining a bit of computational efficiency, but otherwise sometimes underperforming. The main proposed (and measured) advantage of this model (that it 'avoids representation collapse') therefore does not seem significantly consequential or beneficial to the performance, and does not alone justify the method. I encourage the authors to provide additional experimental results to validate the significant advantage of the method more generally.

Questions for authors

In the text you write: '(iii) Geometric prior, not physical modeling. Symplecticity acts purely as an inductive bias on the latent space rather than a physical simulator (Greydanus et al., 2019; Chen et al., 2019; Toth et al., 2019)' and again in the appendix: 'We are not modeling the physics of the observed system... This is the point of departure from Hamiltonian neural networks (Greydanus et al., 2019)...'. My main question is how this work truly differs from this prior work, and the work listed above? Is it only that you are claiming not to care if the hamiltonian recovers the ground truth hamiltonian? Since prior work also seems to operate in a latent space, is this difference just a matter of interpretation? I dont understand what you mean by 'a physical simulator', is it just that you are no longer using a decoder in the JEPA framework? I believe that the best interpretation of your work is the adaptation of these earlier ideas to the JEPA framework; please correct me if I am wrong. If this is true (with undoubtedly some additional tricks you found to make it work) it would be helpful to situate the paper better in the literature by explicitly presenting as such.

In the text you say: "LeWM decodes θ more accurately, though the gap narrows at higher probe capacities. This is the expected consequence of a per-frame encoder as it contains only position, so the anchor latent encodes the angle well but carries no velocity." I do not understand this argument. My understanding was LeWM uses a three-frame encoder, and could thus learn to encode both position and velocity into its latent state (e.g. by splitting the single latent into two subspaces). Can you explain further?

4 comments
AuthorsOpenReview ↗

Response to VHit (1/2): novelty, breadth, size of the gain

1.1 Novelty relative to Hamiltonian learning.

Tables A–D are in our second general comment (Table A = revised Table 1; Tables B, B' = revised Table 2). Changes are in blue in the revised PDF; "revised" refers to it, plain numbers to the submission.

We agree. HNN (Greydanus et al., 2019) learns a Hamiltonian from positions and momenta, measured or encoded from pixels. HGN (Toth et al., 2019) learns from pixels: it infers a latent phase space, unrolls a learned Hamiltonian and decodes every step back to a frame.

JEPA world models learn from video without a decoder; the loss lives in the latent. They plan robot manipulation and navigation (V-JEPA 2, Assran et al., 2025; DINO-WM, Zhou et al., 2025; LeWM, Maes et al., 2026a). Their open problem is long rollouts (Maes et al., 2026a; Zhang et al., 2026): each prediction is fed back as the next input, so a small contraction per step compounds until rollouts from different frames become indistinguishable. LeHami targets this failure with an action-conditioned Hamiltonian, which also gives an exact inverse (backward reachable sets in safe control; Mitchell et al., 2005) and a continuous-time step (data at several rates; Doshi et al., 2024).

Botev et al. (2021) found on the DeepMind suite that physics priors do not reliably predict better than generic models, and that a continuous, time-reversible step is what helps. We agree, and do not claim LeHami predicts better in general. Our claim is narrower: in a JEPA world model nothing stops the rollout from contracting, and a volume-preserving step removes that failure. Our leapfrog step is of the kind they found helpful.

Related work is now revised Section 2, with Botev et al. (2021), SyMetric (Higgins et al., 2021), LNN (Cranmer et al., 2020), SympNets (Jin et al., 2020) and HamJEPA (Alvarez, 2026), and a comparison table in revised App. F.1.

1.2 Breadth of validation.

The submission has two physically different pendulums: a conservative one, and an actuated one where the torque injects energy. It also has a second encoder, frozen DINOv2, where LeHami beats DINO-WM (App. E.2, Fig. 12). As you suggested, we added the mass-spring and the two-body of the DeepMind suite and a cart pole (Towers et al., 2024), two predictors on LeHami's state (a parameter-matched MLP, LeWM's transformer) next to LeWM on its own state, three seeds and two baselines:

  • Given the same state, both free predictors contract the rollout on all five environments; LeHami's rollouts never merge, though past the trained horizon they drift (values above 1), and on two-body and the cart pole it squeezes some directions of the data while stretching others.
  • LeHami is the most accurate of the three in position at the long step on four of five environments and beats chance everywhere; in velocity, on three of five (the pendulums and the mass-spring).
  • LeWM with a three-frame state is more accurate on four of five environments (Table D): it learns all 64 latent dimensions, while LeHami learns 32 and computes p with a fixed stencil, and it has more parameters. Nothing in its predictor stops contraction, though: on the cart pole its rollouts start to contract (tangent area 0.76). We see the two as complementary: a richer encoder, with p learned from several frames, combined with a Hamiltonian step is the next direction (revised Limitations).

1.3 Size of the gain.

(a) In context, the gain in Fig. 4 is not marginal. With the largest training budget, LeHami's angle at K = 20 is on average 7° closer than LeWM's (0.21 against 0.33 rad), and its velocity error is 27% lower (0.85 against 1.18 rad/s; Table 4 at three seeds). At K = 70 its angle is about 26° closer (0.78 against 1.24 rad) and its velocity error 20% lower (2.19 against 2.73). LeWM is more accurate in angle only at short steps (3° at K = 10); on the actuated pendulum the gain is 4° at K = 70. The revised Fig. 4 compares LeHami with the two same-state predictors and chance; LeWM moves to the appendix. On the three new environments, LeHami is the most accurate of the three same-state predictors in position at the long step on the mass-spring and the cart pole, with 2 to 3 times lower error on the mass-spring; on two-body the free predictors are more accurate (Table B).

You are right that Tables 1 and 2 are mixed. Table 1 scores the encoded state at the anchor, with no rollout, so it does not test the predictor; we moved it to the appendix (revised Table 9). Table 2 is planning, where LeHami does not reach goals better; it is now one application with its limits (revised Table 4).

(b) HGN scores rollouts by pixel error and energy conservation; it does not read physical quantities from the rollout, nor compare a Hamiltonian predictor with a free one on the same state. Our ablation does: with the encoder and state fixed, replacing a free predictor by a Hamiltonian step stops long rollouts from contracting (Table A).

AuthorsOpenReview ↗

Response to VHit (2/2): purpose, prior work, velocity in LeWM

1.4 The purpose of LeHami.

Tables A–D: second general comment. Changes: in blue in the revised PDF.

The problem is rollout collapse. Every use of a world model (predicting physical quantities, planning, training a policy on imagined rollouts) needs long rollouts that stay close to the true states. A world model is trained on short clips (8 steps here) and rolled out far beyond them. An MLP or a transformer imposes no structure on what a step does to volume, and beyond the training horizon nothing in the loss does: a contraction of 0.99 per step leaves 0.92 after 8 steps, which training barely sees, and 0.6 after 50, and it keeps falling. Encoder regularizers (SIGReg, Balestriero & LeCun, 2025; VICReg, Bardes et al., 2021) act on the encoder, not on the predictor.

We now measure collapse on the data, with four ratios of the rollout to the truth (reply to m2u2). Given the same state, an MLP and a transformer contract the rollout on all five environments and measures, more at every longer step (Table A); LeHami's rollouts never merge.

Why it matters. When rollouts from different starts move closer together than the true states, the probe gives them similar readings and the error rises toward chance: on the conservative pendulum at K = 70 the free predictors are worse than chance (1.31 rad against 1.26), LeHami is not (0.78). For planning, merged rollouts give different action sequences the same cost.

1.5 Geometric prior and physical modeling and how LeHami differentiates.

Your reading is right: LeHami brings Hamiltonian learning into JEPA world models, to address the collapse of long rollouts. We now position the paper this way (general comment 1, point 1): the revised introduction says so, and the related work is now revised Section 2, which compares LeHami with HNN, SRNN, HGN, the Hamiltonian world models and the structured JEPA predictors. We also dropped "not a physical simulator", which was unclear: it meant that nothing in training ties the latent or the learned Hamiltonian to the physical ones. The differences:

  1. Coordinates. HNN (Greydanus et al., 2019) and SRNN (Chen et al., 2019) are given the physical (q, p); HGN infers them through a decoder that re-renders every frame. LeHami has no decoder: the encoder may choose any representation in which the flow predicts well, and the physics is read by frozen probes.
  2. The problem to solve. These works score rollouts in a fixed space (states or pixels), where a contracting rollout shows up directly as error. A JEPA rollout lives in a learned latent with no decoder; we ask whether the structure removes its collapse. Moreover, HNN, SRNN and HGN are unactuated. LeHami puts the action in the potential, V(q, u), so every step stays symplectic under any action sequence.

Do we care whether the learned H is the true one? Yes, when one exists: over three seeds, the learned energy correlates with the physical one at 0.93 on the conservative pendulum, 0.96 on the mass-spring and 0.84 on the cart pole (Fig. 11 shows the level sets). The guarantee holds for any learned H, which matters under actuation, where nothing conserved is left to recover. Remark (iii), revised Section 3, is rewritten in these terms.

1.6 Why a three-frame LeWM encoder cannot carry velocity.

You are right, a JEPA latent can carry velocity in two ways. Through the encoder: with a temporal ConvNet over three frames, a tied kernel gives the frames before and after the centre mirror-image weights, so it sees how far the pendulum moved but not which way; an untied kernel can subtract them and see the direction. Through the predictor: LeWM's transformer can infer velocity from three consecutive embeddings. So whether LeWM's state carries velocity depends on design: an untied kernel captures its sign and size, a tied one only its size, and a per-frame encoder neither (velocity then exists only inside the predictor). LeHami gets velocity by construction, p=SG(q)\mathbf{p} = \mathrm{SG}(\mathbf{q}), from a per-frame encoder; its gain is that, given that state, its rollouts do not contract, while unstructured predictors do.

I want to thank the authors for their thorough reply. I appreciate that they have clarified the relationship of their work to prior work (both here and in the updated manuscript). I additionally appreciate their clarifications regarding the magnitude of their performance improvements; and the new experiments. In sum, they have addressed the majority of my concerns. I think the work is a well-motivated and well-presented contribution to the community, and therefore have raised my score.

We thank the reviewer for the careful reading and for raising the score. The comparison with prior work and the clarified size of the improvements have made the paper clearer, and we are glad the new experiments addressed most of the concerns.

Comments on the paper

6 comments
~Robert_Jenkinson_Alvarez1OpenReview ↗

Prior Hamiltonian in JEPAs and novelty overlap: Beyond Isotropy in JEPAs: Hamiltonian Geometry and Symplectic Prediction

Hamiltonian prediction within JEPA was already introduced in Beyond Isotropy in JEPAs: Hamiltonian Geometry and Symplectic Prediction (HamJEPA) (https://arxiv.org/abs/2605.20107v1), which was made public in May 2026. The overlap extends beyond using the same numerical integrator:

  1. Jointly learned Hamiltonian architecture. Both train a visual encoder and a separable Hamiltonian end to end via symplectic leapfrog prediction in latent phase space. The HamJEPA trainable residual is what enables the Hamiltonian to be learned despite the quadratic components being fixed. At the equation level, choosing K(p)=12∣∣p∣∣2K(p)=\frac12||p||^2, S=IS=I and a potential that is independent of action recovers the form of the HamJEPA predictor (§§4.1–4.2).

  2. Canonical coordinates and exact latent kinematics. HamJEPA uses the canonical symplectic matrix J and has q˙=p\dot q=p exactly in its continuous latent dynamics. Canonicality is defined by the symplectic structure. SG temporal momentum estimation is novel in respect to Hamjepa. With the learned kinetic term in the submission we have q˙=S−1∇K(S−1p)\dot q=S^{-1}\nabla K(S^{-1}p), so consistency with derivative defined momentum is something that has to be learned rather than held exactly. That introduces an estimation step on top of the established canonical structure. Neither of the constructions on their own is sufficient to guarantee recovery of physical canonical coordinates.

  3. Multi-step prediction and supervision. HamJEPA defines Φ(K)=ΨhK\Phi^{(K)}=\Psi_h^K for general K (Eq. 28), training via composition with endpoint supervision between augmented image views and multi-step rollout. The reported K=2 doesn't in any way constrain the architecture or the proofs. Submission 50639 adds autoregressive video prediction under actions with targets at successive observed times which alters the training objective and supervision, whilst still using the shared multi-step Hamiltonian construction.

  4. Volume preservation and collapse guarantees. HamJEPA proves symplecticity, unit Jacobian determinant, invertibility, reciprocal singular value pairing and the preservation of positive phase space volume over arbitrary finite integration depth (Theorem A.4, Corollary A.5, Lemma A.7, Proposition A.16). This rules out uniform contraction in all ambient directions and finite horizon merging. Conditioning the potential V(q,u)V(q,u) on specified actions extends these guarantees via compositions of symplectic steps.

  5. Continuous time dynamics and reversibility. HamJEPA defines the continuous Hamiltonian vector field (Eq 25), proves Ψh−1=Ψ−h\Psi_h^{-1}=\Psi_{-h} and uses the negative step prediction in its bidirectional objective (Eq 32). These are all very standard properties which were already in the trained JEPA predictor; their structural validity doesn't require recovering the physical Hamiltonian. Submission 50639 adds temporal re-stepping and reversal tests, energy conservation and physical energy alignment diagnostics, physical state evaluation, rollout volume metrics and planning.

  6. Encoder regularisation versus predictor geometry. HamJEPA has already drawn the distinction between marginal regularisation and predictive coupling (Prop 3.9) and combines encoder side anti-collapse regularisation with predictor side geometric preservation (§§4.3–4.4). SIGReg alters the regularisation objective but keeps the same division of labour.

The novelty appears to be in the controlled video formulation, the SG estimator, the learned kinetic term and the empirical results. All of that should be distinguished from the previously established Hamiltonian JEPA architecture and its associated geometric guarantees when it comes to assessing the novelty of the submission.

AuthorsOpenReview ↗

General comment 1 of 2: summary of the revision

We thank the reviewers and the public commenter. We believe we have addressed the reviewers' points; each reply says how. Changes are in blue in the revised PDF; the result tables (Tables A–D) are in the next general comment.

Focus of the revision. The revision explains why the unconstrained predictors of JEPA world models impose no structure on the rollout, so beyond their training horizon nothing stops them from contracting it, and shows that this happens on five environments with two predictors, an MLP and LeWM's transformer. To our knowledge, this is one of the first works to show this collapse in JEPA world models trained on video and to propose a predictor that removes it: a symplectic Hamiltonian flow, which preserves latent volume. We also compare physical predictions across predictors given the same encoded state. The comparison with a more expressive encoder (LeWM on its own three-frame state) and planning are secondary: a richer encoder improves accuracy, is complementary and sets the ground for follow-up work; we report both with their limits (Appendix F, Section 4.5).

Three points came up in several reviews:

1. What is new relative to Hamiltonian learning (responding to VHit, m2u2 and the public comment). LeHami brings Hamiltonian learning (HNN, Greydanus et al., 2019; SRNN, Chen et al., 2019; HGN, Toth et al., 2019) into JEPA world models learned from video under actions. The properties of the leapfrog step, volume preservation and an exact inverse, are classical (Arnold, 1989; Hairer et al., 2003), and we do not claim them as new. We now cite SympNets (Jin et al., 2020), the general class our step belongs to, and HamJEPA, a leapfrog predictor in a JEPA for classifying static images. Related work is now revised Section 2.

2. Extensive evaluation with more environments (responding to VHit, 4fNR and m2u2). We added three environments learned from pixels: the mass-spring and the two-body of the DeepMind Hamiltonian Dynamics Suite, and a cart pole (Towers et al., 2024) whose mass depends on the pole angle. LeHami is now tested on five environments against four other models, with three seeds and two baselines: repeat the start and predict the mean.

3. An ablation that isolates the Hamiltonian step (responding to 4fNR and m2u2). Same encoder, latent [q, SG(q)] (SG: a Savitzky–Golay time derivative), losses, regularizer and 8-step training; only the predictor differs: LeHami's Hamiltonian flow, a parameter-matched MLP, and LeWM's transformer. Results (general comment 2):

  • Both free predictors contract the rollout on every environment and measure (Table A).
  • LeHami's rollouts never merge: their distance between rollouts stays at or above 1. It is not exact: it drifts past the 8 trained steps (values above 1), and on two-body and the cart pole it squeezes some directions while stretching others, keeping the latent's volume.
  • Accuracy at long horizon. LeHami is the most accurate of the three in position at the long step on four of five environments and in velocity on three (the pendulums and the mass-spring), and beats both baselines everywhere, in velocity only narrowly on the pendulums and the cart pole (Tables B, B').
  • LeWM with its own state (Table D): with a three-frame ConvNet it is more accurate on four of five environments, with all 64 latent dimensions learned and more parameters.

More new results:

  • Backward rollout (responding to m2u2 and 4fNR): LeHami recovers earlier angles as well as it predicts later ones, under actuation too (20 steps: 0.68 rad backward, 0.72 forward, chance 1.00).
  • Table 4 at three seeds (responding to m2u2): LeHami has the lower error at every budget; at 0.75 PFLOP (0.30 rad) it matches LeWM at 6.0 PFLOP (0.33 rad).
  • The mass stays in the rollout (responding to 4fNR): a linear probe reads the bob's mass from 70-step rollouts of test episodes with R² ≥ 0.98.

Main changes to the paper:

  • The title now uses the method's name, LeHami; the subtitle is unchanged.
  • Related work is now Section 2, with a comparison table in App. F.1.
  • Fig. 4 is redrawn: LeHami against the two same-state predictors, with chance; its right panel (log|det J|) becomes the distance between rollouts. Tables A, B, B' become Tables 1 and 2; App. F.3–F.5 (Figs. 17, 18, Table D) give all measures and errors against the step, with LeWM.
  • "Invertibility given the action history" replaces "time-reversibility" (Fig. 5).
  • Limitations: a separable H with a mass constant per episode; no dissipation or contact.
  • Planning is one application, with its limits: lower cost per step, not better goal-reaching (App. D.4.2, Table 4).
  • New results are in App. F, to move to their sections for the camera-ready. Other appendix changes: App. C.3–C.4 (LNN, unit mass), E.1 (Table 4, now Table 5) and E.3 (submission Table 1, now Table 9).

Code and results: https://anonymous.4open.science/r/hamiltonian-jepa-7F08/ (rebuttal_results/).

AuthorsOpenReview ↗

General comment 2 of 2: result tables A–D

Table A. Collapse on the data. Rollout over truth (the encoder's latents of the true frames) at the longest step: distance between rollouts / tangent area / spread / rank, defined in our reply to m2u2. 1 shows no collapse, below 1 the rollout contracts, above 1 it drifts. Largest std 0.17.

Cons. pendulum, K = 30Act. pendulum, K = 30Mass-spring, K = 50Two-body, K = 70Cart pole, K = 70
LeHami1.03/1.35/1.04/1.251.00/1.04/1.01/1.211.10/1.22/1.15/1.571.50/0.94/1.59/0.831.33/0.58/1.42/0.94
MLP, matched0.81/0.49/0.78/0.560.91/0.68/0.90/0.840.73/0.17/0.70/0.370.85/0.75/0.85/0.940.72/0.22/0.71/0.56
Transformer (LeWM+SG)0.86/0.62/0.83/0.700.92/0.81/0.90/0.900.83/0.37/0.80/0.590.84/0.76/0.84/0.950.63/0.20/0.64/0.61

Table B. Position MAE, read from the rollout by a 25-nearest-neighbour probe, mean ± std over three seeds. Angle [rad] for the pendulums and the cart pole, q for the mass-spring, body position (canvas units) for two-body. Bold: best of the three same-state predictors. The baselines (last two columns) use no model.

environment, stepLeHamiMLP, matchedTransformer (LeWM's)Repeat the startPredict the mean
Cons. pend. θ, K = 100.177 ± 0.0020.416 ± 0.0210.279 ± 0.0521.9811.299
Cons. pend. θ, K = 700.779 ± 0.0341.310 ± 0.0151.313 ± 0.0461.2591.324
Act. pend. θ, K = 100.319 ± 0.0370.259 ± 0.0120.200 ± 0.0181.6881.053
Act. pend. θ, K = 700.908 ± 0.0671.073 ± 0.0271.076 ± 0.0151.1861.168
Mass-spring q, K = 100.013 ± 0.0010.030 ± 0.0010.025 ± 0.0010.3270.355
Mass-spring q, K = 500.074 ± 0.0160.223 ± 0.0050.164 ± 0.0180.4220.336
Two-body pos., K = 100.027 ± 0.0060.024 ± 0.0090.027 ± 0.0010.8281.456
Two-body pos., K = 700.421 ± 0.0470.148 ± 0.0730.183 ± 0.0372.5021.456
Cart pole φ, K = 100.109 ± 0.0050.072 ± 0.0140.058 ± 0.0120.5111.224
Cart pole φ, K = 701.041 ± 0.0981.190 ± 0.0931.104 ± 0.0812.2761.285

Table B'. Velocity MAE, same layout. On the cart pole the probe reads φ̇ poorly even before any rollout.

environment, stepLeHamiMLP, matchedTransformer (LeWM's)Repeat the startPredict the mean
Cons. pend. ω, K = 100.473 ± 0.0031.418 ± 0.0381.160 ± 0.0553.6022.386
Cons. pend. ω, K = 702.185 ± 0.2272.672 ± 0.0852.566 ± 0.1442.7412.499
Act. pend. ω, K = 100.913 ± 0.0490.895 ± 0.0450.773 ± 0.0243.1202.065
Act. pend. ω, K = 702.280 ± 0.2062.721 ± 0.0332.801 ± 0.0863.2222.541
Mass-spring p, K = 100.038 ± 0.0030.144 ± 0.0060.103 ± 0.0110.3240.367
Mass-spring p, K = 500.086 ± 0.0090.254 ± 0.0040.201 ± 0.0190.4360.347
Two-body vel., K = 100.007 ± 0.0010.007 ± 0.0030.008 ± 0.0000.2730.423
Two-body vel., K = 700.112 ± 0.0130.056 ± 0.0340.062 ± 0.0190.6880.423
Cart pole φ̇, K = 102.015 ± 0.0541.370 ± 0.1041.272 ± 0.3951.9872.544
Cart pole φ̇, K = 702.314 ± 0.1922.140 ± 0.1012.248 ± 0.2993.7782.409

Table D. LeHami against LeWM with its own state. LeWM W=1 is the paper's baseline (one frame); LeWM W=3 learns its state from three frames. Collapse as in Table A; MAE at K = 70 (K = 50 for the mass-spring), mean over three seeds. –: not run.

Cons. pendulumAct. pendulumMass-springTwo-bodyCart pole
Collapse
LeHami1.03/1.35/1.04/1.251.00/1.04/1.01/1.211.10/1.22/1.15/1.571.50/0.94/1.59/0.831.33/0.58/1.42/0.94
LeWM W=11.01/0.78/1.01/0.981.00/1.06/1.00/1.00–––
LeWM W=30.99/1.00/0.99/0.990.99/1.00/0.99/0.990.99/1.00/0.98/1.010.98/0.95/0.98/1.000.93/0.76/0.92/0.97
Position MAE
LeHami0.7790.9080.0740.4211.041
LeWM W=11.2420.984–––
LeWM W=30.5020.6440.0160.1650.203
Velocity MAE
LeHami2.1852.2800.0860.1122.314
LeWM W=12.7322.259–––
LeWM W=31.1341.8200.0150.0520.652

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.