acceptodds
Under review as a conference paper at ICLR 2027

Hamiltonian JEPA: Action-Conditioned World Models with an Inherited Isotropic Control State

20 views

Abstract

Planning from pixels needs more than a latent space that is stable and predictable. The state the planner scores must also be organized by how actions move the system. Joint-embedding predictive architectures (JEPAs) avoid pixel reconstruc- tion by predicting future representations, but existing action-conditioned JEPAs ask one embedding to serve both perception and control. We introduce H-JEPA, which separates the two. A wide perceptual code is regularized toward a well- scaled isotropic geometry with a Bures-Wasserstein prior, and a fixed orthonor- mal slice of that code is the control state, which inherits the code’s covariance without any objective of its own. The state evolves under phase-conditioned dis- sipative port-Hamiltonian dynamics whose input port has orthonormal columns. Port-inverse consistency (PIC) reads the executed action back through the trans- pose of that port. We show that this readout is exactly the rollout error projected onto the port directions, so PIC is a parameter-free reweighting of prediction error and not an auxiliary action decoder. Untying the readout from the port breaks this identity and loses half of the gain. H-JEPA matches or exceeds reconstruction-free baselines, including the action-decoding Delta-JEPA, on four pixel-based control benchmarks after at most 10 training epochs, and its largest gain is on OGB-Cube (91.9 against 79.3 percent). Ablations on PushT and OGB-Cube separate the con- tributions of the structured predictor, PIC, the prediction horizon, the state rank, and the anti-collapse prior.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, vla, robotics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2NvaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces H-JEPA, a world model architecture for action planning. It separates visual representations from the control state used for planning by regularizing the visual representations and applying a fixed orthonormal projection to obtain the control state. The evolution of this state is predicted using port-Hamiltonian dynamics. The authors report evaluation results on four visual control tasks, Two-Room, Reacher, PushT, and OGB-Cube.

Critical strengths
  1. The authors explore a promising idea by using a Hamiltonian network, which remains relatively uncommon in robotic world models, as the predictor in a JEPA world model.
  2. Through mathematical derivation, the authors show that PIC reweights prediction errors along action-related directions.
  3. The ablation study is fairly extensive and examines several important choices, including the predictor, PIC, the representation prior, and the prediction horizon.
Critical weaknesses
  1. The benefit of separating perception and control is not sufficiently isolated. On PushT, the selected configuration uses r=Dr=D, so s=U⊤hs=U^\top h preserves all information and Euclidean distances. Although the authors acknowledge this, the result does not establish the benefit of using distinct perceptual and control features. Table 4 tests regularizer placement, while the rank sweep also changes predictor capacity and PIC’s relative weighting. A comparison with U=IU=I at full dimension would help clarify what the projection contributes. Furthermore, Table 1 reports collapse with learnable UU without clarifying whether orthonormality was maintained or providing quantitative analysis. A learnable orthonormal projection is therefore an important missing control. These omissions leave a central architectural contribution insufficiently distinguished from regularization and capacity choices, which materially limits my confidence in the paper’s explanation of its gains.

  2. The experiments do not establish the specific contribution of the Hamiltonian component. Table 4 shows that the complete port-Hamiltonian predictor outperforms an MLP, but this comparison changes several structural properties together. It does not determine whether the conservative term J∇sHJ\nabla_s H contributes beyond the damping structure, constrained action-input map, and PIC objective. An ablation removing J∇sHJ\nabla_s H while retaining the other components, with matched training and evaluation settings, would directly address this question. The existing evidence supports the structured predictor as a package, but does not adequately substantiate the particular value of its Hamiltonian construction. Given the prominence of this construction in the paper’s framing, this unresolved attribution substantially weakens the methodological contribution.

Questions for authors
  1. In lines 243, do the claims about energy conservation and dissipation assume that vv is fixed? Since HH also depends on vv, its derivative includes ∇vH⊤v˙\nabla_v H^\top\dot v when vv changes. How do these claims apply to the actual rollout?

  2. Lines 312 say that rotating the port away from prediction errors would make prediction worse. Since the port and passive dynamics are learned together, could the passive dynamics compensate for such a rotation? If not, what prevents this?

  3. Equation 6 is linear in the action at a fixed phase, and the action map has unit singular values. How does the model handle an action that has little effect because an object is blocked, or whose effect changes with contact? Have the authors tested different actions from the same initial state?

5 comments
AuthorsOpenReview ↗

Perception and Control

We sincerely thank the reviewer for the careful and thorough assessment of our work. We appreciate the recognition of our core idea, mathematical analysis, and extensive ablations. More importantly, the reviewer raises several important questions about the contributions of individual components and the assumptions behind our formulation. We have taken these concerns seriously and conducted additional controlled experiments to directly address them. We believe the new results provide stronger evidence for our design choices and clarify the contributions of our work. We address each concern below.

  1. Does separating perception and control help beyond regularization or capacity? What do identity and learnable projections change?

On PushT, we deliberately kept the perceptual width at 192 to match LeWM, whose dimension ablation found performance saturates around this size (Appendix G). This is a strong practical baseline, not a claim that the true control state has 192 dimensions. We agree that at full rank UU is a rotation, not compression. The separation is in purpose: we regularize the perceptual code hh, while prediction and planning use ss. The identity control is accordingly close on PushT (94.5% vs. 93.5%); we do not attribute the gains to rotation alone. On OGB-Cube, where ss is smaller, the full model improves over a single 32-dimensional embedding (93.9 ± 1.3% vs. 90.4 ± 0.5%). A learnable orthonormal UU also performs well (92.7%, one seed), while the existing prior-placement ablation favors regularizing hh over ss (93.3% vs. 86.3%; Table 4). These results support separating the roles of perception and control. We will clarify that the OGB-Cube identity control also reduces perceptual width.

TaskConfigurationSuccess (%)
OGB-CubeFull (fixed orthonormal UU)93.9 ± 1.3
OGB-CubeSingle embedding (U=IU = I)90.4 ± 0.5
OGB-CubeLearnable orthonormal UU92.7
OGB-CubeLearnable unconstrained UU*87.0
PushTFull (fixed orthonormal UU)94.5
PushTSingle embedding (U=IU = I)93.5
AuthorsOpenReview ↗

The Contribution of the Hamiltonian J Term

  1. Does the conservative term J∇sHJ\nabla_s H contribute beyond damping, the action port and PIC?

This is a fair distinction: the original MLP comparison tested the full predictor, not JJ alone. We therefore removed only JJ while keeping the learned energy, damping, orthonormal port, and PIC unchanged. Performance falls on OGB-Cube (93.9 ± 1.3% to 91.0 ± 2.5%), PushT (94.5% to 93.2%), and Reacher (88.4 ± 1.1% to 86.8%). The improvement is modest but consistent across tasks, the single-seed comparisons are not significance claims. The full design is also considerably simpler than the transformer baseline: on the same PushT representation and horizon, it reaches 92% with 0.86M dynamics parameters, versus 84% with 3.3M (Table 3). Appendix E shows that the learned balance of passive and action-driven motion changes across tasks, Appendix F finds that a learned history mixer largely ignored motion information, motivating our parameter-free phase. Together these experiments show both the value of the structured predictor and the additional contribution of JJ.

TaskConfigurationSuccess (%)
OGB-CubeFull (with JJ)93.9 ± 1.3
OGB-CubeWithout JJ (−R∇H-R\nabla H)91.0 ± 2.5
PushTFull (with JJ)94.5
PushTWithout JJ (−R∇H-R\nabla H)93.2
ReacherFull (with JJ)88.4 ± 1.1
ReacherWithout JJ (−R∇H-R\nabla H)86.8
AuthorsOpenReview ↗

Energy Conservation Under Changing Phase

  1. In lines 243, do the claims about energy conservation and dissipation assume that vv is fixed? Since HH also depends on vv, its derivative includes ∇vH⊤v˙\nabla_v H^\top \dot v when vv changes. How do these claims apply to the actual rollout?

We thank the reviewer for pointing out this distinction. Yes, the energy argument assumes that vv is fixed, as specified in Section 3.4 when defining the state gradient. For the action-free, continuous-time dynamics at a fixed phase velocity vv,

∇sH⊤(J−R)∇sH=−∇sH⊤R∇sH≤0.\nabla_s H^\top (J - R) \nabla_s H = -\nabla_s H^\top R \nabla_s H \le 0.

Thus, JJ is energy-neutral and R⪰0R \succeq 0 makes the passive dynamics dissipative relative to the current energy landscape H(⋅,vt)H(\cdot, v_t).

During an actual rollout, vt=st−st−1v_t = s_t - s_{t-1} is updated at each step. The full energy derivative then includes

dHdt=∇sH⊤s˙+∇vH⊤v˙,\frac{dH}{dt} = \nabla_s H^\top \dot s + \nabla_v H^\top \dot v,

where the second term has no guaranteed sign. The gradient in our update is the partial derivative with respect to ss, treating vv as fixed, it does not imply that the energy decreases over the full rollout. As already noted after Eq. (6) and in Section 5 (Limitations), explicit Euler integration also does not preserve continuous-time energy properties.

The port-Hamiltonian structure is therefore a useful bias for each transition: without actions, it separates an energy-neutral component from a dissipative one for the current H(⋅,vt)H(\cdot, v_t). It is not a conservation law for the entire rollout, and neither planning nor training assumes one. We will clarify “for fixed vv” at line 243 and explicitly mention the changing-phase term alongside the existing Euler qualification.

Reviewer_MursOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Hamiltonian-JEPA, a Jepa-based wm that combines a projected control state, Hamiltonian dynamics, and port-inverse consistency (PIC). The design is well motivated, and the reported planning results are promising. Novelty is somewhat limited, but this is not critical for paper.

Critical strengths
  1. The paper focuses on a interesting problem, a current Jepa-based WMs can learn stable representations while still failing to capture action consequences well enough for planning. Proposed mechanism for fixing action-related errors looks reasonable.
  2. The reported results are competitive across control benchmarks, with substantial gains on OGB-Cube. The ablations investigate several relevant choices.
Critical weaknesses
  1. Novelty looks limited in current presentation form and require more concrete grounding to related literature. Action-sensitive latent learning is explored in Delta-JEPA [1], Hamiltonian JEPA prediction in HamJEPA [2], and Bures isotropy regularization in BIA [3]. More precise positioning would help establish the distinctive value of the proposed port–PIC coupling. Partially this works mentioned in paper, but I think side-by-side comparison with Jepa based methods and explanation of connection of ideas could make positioning of this paper stronger.

[1] - Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding.

[2] - Beyond Isotropy in JEPAs: Hamiltonian Geometry and Symplectic Prediction.

[3] - Bures-Isotropy Alignment: Manifold Learning of Generalized Category Discovery.

Questions for authors

I think paper will become stronger with carefull comparison and attribution to related literature

3 comments
AuthorsOpenReview ↗

Comparison with Related Works

I think paper will become stronger with carefull comparison and attribution to related literature

We thank the reviewer for his review and for suggesting a clearer comparison with prior work. Our design starts from a simple idea: perception needs a rich description of the scene, while planning needs a state that responds to actions. We regularize a high-dimensional visual code (h)(h) and obtain the planner’s state through a fixed orthonormal projection of that code (h)(h). This differs from LeWM [4], which plans in the full code, Sub-JEPA [5], which splits into different orthogonal representations in the same latent only for regularization, and BIA [3], which uses Bures geometry for category discovery rather than control.

Our dynamics make that state useful for planning. Unlike HamJEPA [2], which uses conservative Hamiltonian prediction between image views, we predict state transitions under explicit actions. Our dynamics are built around a learned scalar Hamiltonian, whose gradient is computed through automatic differentiation. Combined with learned damping and an orthonormal action-input map, this gives us a structured, action-conditioned model for multi-step planning, rather than an unconstrained network predicting the next state directly. The model is Markovian in the augmented state (state and estimated velocity), not in a single image code as in LeWM, Sub-JEPA, and Delta-JEPA, which use a shared representation for perception and control. We do not claim exact energy conservation in discrete rollouts.

Most importantly, the same action port is used in both directions. Our proposed Port-Inverse Consistency (PIC) reads the action contribution from an observed transition after accounting for predicted internal motion, thereby emphasizing prediction errors along action-input directions. Unlike Delta-JEPA [1], which learns a separate action decoder, or Physically Grounded JEPA [6], which uses auxiliary inverse and physical-state supervision, our approach requires neither an additional decoder nor physical-state labels.

In summary, no compared method combines a dedicated planning state, action-conditioned port-Hamiltonian rollouts, and a mathematically tied, parameter-free inverse objective. These choices matter in practice: on PushT, success falls from 93.3% to 75.0% without PIC, 83.7% with an untied readout, and 85.0% with an MLP predictor (Table 4). At matched representation and horizon, our predictor achieves 92% versus a transformer’s 84%, with 0.86M versus 3.3M dynamics parameters (Table 3). We will make the distinctions and attribution clearer in the revision.

AuthorsOpenReview ↗

Comparison Table

WorkSetting / useFormulation / supervisionDifference from our work
Our workAction-conditioned pixel world model, goal-image planning.h=E(x)h=E(x), s=U⊤hs=U^\top h, BW prior on (μh,Σh)(\mu_h,\Sigma_h). s+=s+(J−R)∇sH+Gas^+=s+(J-R)\nabla_s H+Ga, G⊤G=IG^\top G=I. PIC: a^=G⊤(Δsobs−Δspass)\hat a=G^\top(\Delta s_{\text{obs}}-\Delta s_{\text{pass}}).Our contribution: a projected planner state plus dissipative, controlled port dynamics and a tied action readout; PIC exactly reweights error along port directions.
Delta-JEPA [1]Action-conditioned pixel world model, visual planning.z^+=P(z,a)\hat z^+=P(z,a), a^=D(z+−z)\hat a=D(z^+-z). L=Lpred+λLactionL=L_{\text{pred}}+\lambda L_{\text{action}} (learned LDAD).Action recovery is prior art, LDAD is an independent learned decoder, unlike PIC =G⊤=G^\top tied to the forward action port.
HamJEPA [2]Self-supervised image-view prediction, CIFAR/ImageNet feature probes.E(v)=(q,p)E(v)=(q,p), H(q,p)=12∥p∥2+V(q)H(q,p)=\tfrac{1}{2}\lVert p\rVert^2+V(q). (q+,p+)=leapfrogH(q,p)(q^+,p^+)=\mathrm{leapfrog}_H(q,p), non-isotropic anti-collapse.No actions or control planning. Conservative symplectic view-to-view flow, versus action-forced, damped Euler rollouts for planning.
BIA [3]Generalized category discovery / clustering.ΣB=ZZ⊤\Sigma_B=ZZ^\top (class-token Gram), min⁡dB2(ΣB,I)≃max⁡∥Z∥∗\min d_B^2(\Sigma_B,I)\simeq\max\lVert Z\rVert_{\ast} (fixed row norms).Shares Bures-based isotropy, but regularizes a sample Gram for clustering, ours matches code mean/covariance via Gaussian Bures-Wasserstein for control.
LeWorldModel [4]End-to-end action-conditioned pixel world model, planning.z^+=P(z,a)\hat z^+=P(z,a), L=Lpred+λSIGReg(z)L=L_{\text{pred}}+\lambda\mathrm{SIGReg}(z).Single embedding zz serves perception and planning, SIGReg, generic predictor, no tied action readout.
Sub-JEPA [5]Action-conditioned pixel world model, planning.z^+=P(z,a)\hat z^+=P(z,a), L=Lpred+λK∑kSIGReg(Pkz)L=L_{\text{pred}}+\frac{\lambda}{K}\sum_k\mathrm{SIGReg}(P_k z), PkP_k fixed.Random subspaces are only for regularization, planner still uses zz. Our fixed slice s=U⊤hs=U^\top h is itself predicted and planned in, adds port-PIC.
Physically Grounded JEPA [6]Action-conditioned robotic planning, uses physical-state labels during training.z^+=F(z,a)\hat z^+=F(z,a), a^=H(z,z+)\hat a=H(z,z^+),x^phys=G(z−,z)\hat x_{\text{phys}}=G(z^-,z). L=Lpred+αLstate+βLIDML=L_{\text{pred}}+\alpha L_{\text{state}}+\beta L_{\text{IDM}}.Separate learned inverse and state-alignment heads, with physical-state supervision, ours uses only image–action trajectories and the tied port readout.
References

[1] Zhang et al., Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding. https://arxiv.org/abs/2606.31232

[2] Alvarez, Beyond Isotropy in JEPAs: Hamiltonian Geometry and Symplectic Prediction. https://arxiv.org/abs/2605.20107

[3] Tang et al., Bures-Isotropy Alignment: Manifold Learning of Generalized Category Discovery (ICLR 2026). https://openreview.net/forum?id=nfVKTJ1MJ3

[4] Maes et al., LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. https://arxiv.org/abs/2603.19312

[5] Zhao et al., Sub-JEPA: Subspace Gaussian Regularization for Stable End-to-End World Models. https://arxiv.org/abs/2605.09241

[6] Liu et al., Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning. https://arxiv.org/abs/2609.03565

Thank you for clarifications, I will maintain my positive score and increase confidence.

Reviewer_y7bgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents Hamiltonian JEPA, which attempts to separate out perception and control, which are normally entangled into a single representation. The encoder maps each image to a 192-dimensional embedding, and a Bures-Wasserstein penalty pushes these embeddings to be evenly spread in all directions with unit variance. The control state is obtained by multiplying the embedding by a fixed random orthonormal matrix. Because this projection is fixed and linear, the control state automatically takes on the embedding’s well-spread covariance, so it needs no loss of its own.

Critical strengths

The experiments are well designed and results seem promising, especially in table 3. The method also seems to have efficiency gains in addition to performance (needing 10 epochs vs 50). The paper contains broad ablations, and the overall idea, of separating perception and control, is interesting and not widely studied in the literature.

Critical weaknesses

The paper writing could use work. Notably, the entire main body of the paper contains zero figures or diagrams. A visual aid, if properly done in a quality way, would greatly have enhanced readability and understanding. I was able to piece together the main ideas after several readings, but I would recommend a large revision of the paper, with an aim of improving readability. The appendix included several diagrams, but it is very important for the main body to be readable.

Most experiments seem to be single seed with no error bars. This makes any conclusions difficult to draw confidently. Four benchmark tasks is also on the lower end, and the authors would do well to add at least 2 more.

We should also note: H-JEPA is already widely used to refer to Hierarchical JEPA (https://arxiv.org/abs/2610.06805), and the authors should consider another name.

Questions for authors
  1. Can you incorporate figure 1 into the main body, and clarify excessive jargon in the writing?
  2. Can you expand the experiments to include more tasks?
  3. Can you rerun all experiments with proper random seeds and error bars, to ensure the results are significant?
  4. Can you rename the method, to avoid overlap with existing usages of H-JEPA?
4 comments
  1. Can you incorporate figure 1 into the main body, and clarify excessive jargon in the writing?

We agree with the reviewer and will move Figure 1 into the main paper and substantially reduce jargon.

The central idea is simple: a rich visual representation should capture the scene, while a dedicated control state should capture what matters for planning. We then learn how this state evolves under actions, using the same action mapping to identify and emphasize prediction errors relevant to control.

The technical details arise from making this simple idea work reliably. For example, Bures-Wasserstein regularization prevents representation dimensions from collapsing by maintaining a useful gradient even as their variance approaches zero. The orthonormal action port enables an exact inverse readout without an additional network, directly connecting action prediction and representation learning.

These design choices required careful mathematical justification and ablations, which made the presentation denser than intended. We will address this by introducing the intuition before the equations, explaining each component in plain language, moving secondary derivations to the appendix, and using Figure 1 to guide readers through the complete architecture.

AuthorsOpenReview ↗

Additional Tasks

  1. Can you expand the experiments to include more tasks?

We thank the reviewer for this suggestion. We have expanded our evaluation with two additional DeepMind Control Suite tasks, Cartpole and Acrobot, using 500 evaluation episodes for each result. With a control-state rank of 16, our method outperforms LeWM on both tasks. These results extend our evaluation from four to six control tasks, and we will include them in the revised paper.

Additional control benchmarks (success rate, %)

TaskLeWMOurs (rank 16)Gain (pp)
DMC Cartpole48.462.0+13.6
DMC Acrobot40.055.0+15.0
AuthorsOpenReview ↗

Additional Seeds

  1. Can you rerun all experiments with proper random seeds and error bars, to ensure the results are significant?

We thank the reviewer for highlighting the importance of evaluating the reliability of our results across multiple training seeds. We agree that this is particularly important for understanding the contribution of each component. In response, we extended our ablations on PushT and OGB-Cube to multiple independent training seeds, evaluated on the same test episodes for fair comparison. The results, reported as mean ± standard deviation below, consistently support our main findings: the structured predictor improves performance, PIC provides a substantial gain, and tying the inverse readout to the action port is important. We believe these additional experiments strengthen the evidence for our design choices.

PushT

r=64r = 64 · epoch 10 · 300 evaluation episodes per seed

ConfigurationSuccess rate (%), mean ± SD
HH + PIC, prior on hh (full)90.8 ± 2.3
MLP + PIC85.8 ± 0.7
HH, no PIC76.0 ± 1.0
HH + PIC, untied readout QQ83.8 ± 3.5
HH + PIC, prior on ss86.3 ± 0.0

OGB-Cube

r=32r = 32 · epoch 10 · 300 evaluation episodes per seed

ConfigurationSuccess rate (%), mean ± SD
MLP + PIC84.0 ± 1.5
HH, no PIC89.0 ± 0.0
AuthorsOpenReview ↗

Method Name

  1. Can you rename the method, to avoid overlap with existing usages of H-JEPA?

We thank the reviewer for this suggestion. We agree that the current name may cause confusion with existing JEPA methods. We will rename our method HPI-JEPA (Hamiltonian Port-Inverse JEPA), which better reflects its distinctive combination of action-conditioned Hamiltonian dynamics and port-inverse consistency.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Summary of revisions and response to all reviewers

We thank all three reviewers for their time and for feedback that has made the paper clearer and stronger.

The idea, in one paragraph. A world model has two jobs. It must describe what the camera sees, and it must give the planner a state in which the effect of an action is easy to read. Current action-conditioned JEPAs ask one embedding to do both. HPI-JEPA separates them. A wide perceptual code describes the scene and is kept from collapsing by a Bures-Wasserstein prior. The planner works in a fixed projection of that code, which inherits its well-spread geometry for free. A port-Hamiltonian predictor moves this state forward and keeps the motion the system makes on its own apart from the push of the action. Port-inverse consistency then runs the action port backwards, and we prove that this makes the model most accurate exactly in the directions where actions act, without adding a single parameter.

Where the reviewers agreed. All three reviewers found the idea promising and rated soundness as having only minor or fixable concerns. Reviewers Murs and y7bg highlighted the results, including the large gain on OGB-Cube (91.9 against 79.3 percent for the strongest baseline), the well-designed experiments, and the efficiency of reaching these results in 10 training epochs where Delta-JEPA trains for 50. All three reviewers noted the breadth of the ablations, and Reviewer 2Nva recognized our derivation showing that PIC reweights prediction error along action directions.

What we changed. The concerns fell into three groups, and we answered each with new experiments or revised text.

  • Clarity (Reviewers y7bg and 2Nva). We redesigned the architecture figure to make it clearer and moved it into the main text as Figure 1, where it now guides the reader through the method step by step. Each component opens with a plain-language explanation before its equations, secondary derivations moved to the appendix, and we reduced jargon throughout. We also renamed the method HPI-JEPA to avoid confusion with hierarchical JEPA.
  • Positioning (Reviewer Murs). A new comparison table places HPI-JEPA next to LeWM, Sub-JEPA, Delta-JEPA, physically grounded JEPA world models, HamJEPA, and BIA. No prior method combines a dedicated planning state, action-driven port-Hamiltonian rollouts, and a parameter-free inverse objective tied to the forward model.
  • Evidence (Reviewers y7bg and 2Nva). We added two DeepMind Control tasks, where HPI-JEPA improves on LeWM by 13.6 points on Cartpole and 15.0 points on Acrobot. We reran the key ablations over multiple training seeds, and every conclusion holds. We added the specific controls Reviewer 2Nva requested. Identity and learnable projections show the benefit of a separate planning state on OGB-Cube (93.9 ± 1.3 against 90.4 ± 0.5 percent for a single embedding), and removing only the energy-conserving term lowers success on all three tasks tested. A new blocked-push test shows that the model captures the difference between a blocked push and a free one without contact labels. We also clarified that the energy statements hold at a fixed velocity, and explained why the passive dynamics cannot take over the role of the action port.

These additions strengthen the paper's central claims rather than change them. The separate control state, the structured dynamics, and the tied inverse readout each contribute, and the gains hold across more tasks and more seeds.

We have incorporated all additional results and explanations presented in the rebuttal into the revised manuscript. We would be happy to provide any further clarification or address any remaining concerns. If we have fully addressed your concerns, we kindly ask you to consider reassessing your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.