- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces H-JEPA, a world model architecture for action planning. It separates visual representations from the control state used for planning by regularizing the visual representations and applying a fixed orthonormal projection to obtain the control state. The evolution of this state is predicted using port-Hamiltonian dynamics. The authors report evaluation results on four visual control tasks, Two-Room, Reacher, PushT, and OGB-Cube.
Critical strengths
- The authors explore a promising idea by using a Hamiltonian network, which remains relatively uncommon in robotic world models, as the predictor in a JEPA world model.
- Through mathematical derivation, the authors show that PIC reweights prediction errors along action-related directions.
- The ablation study is fairly extensive and examines several important choices, including the predictor, PIC, the representation prior, and the prediction horizon.
Critical weaknesses
-
The benefit of separating perception and control is not sufficiently isolated. On PushT, the selected configuration uses , so preserves all information and Euclidean distances. Although the authors acknowledge this, the result does not establish the benefit of using distinct perceptual and control features. Table 4 tests regularizer placement, while the rank sweep also changes predictor capacity and PIC’s relative weighting. A comparison with at full dimension would help clarify what the projection contributes. Furthermore, Table 1 reports collapse with learnable without clarifying whether orthonormality was maintained or providing quantitative analysis. A learnable orthonormal projection is therefore an important missing control. These omissions leave a central architectural contribution insufficiently distinguished from regularization and capacity choices, which materially limits my confidence in the paper’s explanation of its gains.
-
The experiments do not establish the specific contribution of the Hamiltonian component. Table 4 shows that the complete port-Hamiltonian predictor outperforms an MLP, but this comparison changes several structural properties together. It does not determine whether the conservative term contributes beyond the damping structure, constrained action-input map, and PIC objective. An ablation removing while retaining the other components, with matched training and evaluation settings, would directly address this question. The existing evidence supports the structured predictor as a package, but does not adequately substantiate the particular value of its Hamiltonian construction. Given the prominence of this construction in the paper’s framing, this unresolved attribution substantially weakens the methodological contribution.
Questions for authors
-
In lines 243, do the claims about energy conservation and dissipation assume that is fixed? Since also depends on , its derivative includes when changes. How do these claims apply to the actual rollout?
-
Lines 312 say that rotating the port away from prediction errors would make prediction worse. Since the port and passive dynamics are learned together, could the passive dynamics compensate for such a rotation? If not, what prevents this?
-
Equation 6 is linear in the action at a fixed phase, and the action map has unit singular values. How does the model handle an action that has little effect because an object is blocked, or whose effect changes with contact? Have the authors tested different actions from the same initial state?
5 comments
The Contribution of the Hamiltonian J Term
- Does the conservative term contribute beyond damping, the action port and PIC?
This is a fair distinction: the original MLP comparison tested the full predictor, not alone. We therefore removed only while keeping the learned energy, damping, orthonormal port, and PIC unchanged. Performance falls on OGB-Cube (93.9 ± 1.3% to 91.0 ± 2.5%), PushT (94.5% to 93.2%), and Reacher (88.4 ± 1.1% to 86.8%). The improvement is modest but consistent across tasks, the single-seed comparisons are not significance claims. The full design is also considerably simpler than the transformer baseline: on the same PushT representation and horizon, it reaches 92% with 0.86M dynamics parameters, versus 84% with 3.3M (Table 3). Appendix E shows that the learned balance of passive and action-driven motion changes across tasks, Appendix F finds that a learned history mixer largely ignored motion information, motivating our parameter-free phase. Together these experiments show both the value of the structured predictor and the additional contribution of .
| Task | Configuration | Success (%) |
|---|---|---|
| OGB-Cube | Full (with ) | 93.9 ± 1.3 |
| OGB-Cube | Without () | 91.0 ± 2.5 |
| PushT | Full (with ) | 94.5 |
| PushT | Without () | 93.2 |
| Reacher | Full (with ) | 88.4 ± 1.1 |
| Reacher | Without () | 86.8 |
Energy Conservation Under Changing Phase
- In lines 243, do the claims about energy conservation and dissipation assume that is fixed? Since also depends on , its derivative includes when changes. How do these claims apply to the actual rollout?
We thank the reviewer for pointing out this distinction. Yes, the energy argument assumes that is fixed, as specified in Section 3.4 when defining the state gradient. For the action-free, continuous-time dynamics at a fixed phase velocity ,
Thus, is energy-neutral and makes the passive dynamics dissipative relative to the current energy landscape .
During an actual rollout, is updated at each step. The full energy derivative then includes
where the second term has no guaranteed sign. The gradient in our update is the partial derivative with respect to , treating as fixed, it does not imply that the energy decreases over the full rollout. As already noted after Eq. (6) and in Section 5 (Limitations), explicit Euler integration also does not preserve continuous-time energy properties.
The port-Hamiltonian structure is therefore a useful bias for each transition: without actions, it separates an energy-neutral component from a dissipative one for the current . It is not a conservation law for the entire rollout, and neither planning nor training assumes one. We will clarify “for fixed ” at line 243 and explicitly mention the changing-phase term alongside the existing Euler qualification.
Perception and Control
We sincerely thank the reviewer for the careful and thorough assessment of our work. We appreciate the recognition of our core idea, mathematical analysis, and extensive ablations. More importantly, the reviewer raises several important questions about the contributions of individual components and the assumptions behind our formulation. We have taken these concerns seriously and conducted additional controlled experiments to directly address them. We believe the new results provide stronger evidence for our design choices and clarify the contributions of our work. We address each concern below.
On PushT, we deliberately kept the perceptual width at 192 to match LeWM, whose dimension ablation found performance saturates around this size (Appendix G). This is a strong practical baseline, not a claim that the true control state has 192 dimensions. We agree that at full rank U is a rotation, not compression. The separation is in purpose: we regularize the perceptual code h, while prediction and planning use s. The identity control is accordingly close on PushT (94.5% vs. 93.5%); we do not attribute the gains to rotation alone. On OGB-Cube, where s is smaller, the full model improves over a single 32-dimensional embedding (93.9 ± 1.3% vs. 90.4 ± 0.5%). A learnable orthonormal U also performs well (92.7%, one seed), while the existing prior-placement ablation favors regularizing h over s (93.3% vs. 86.3%; Table 4). These results support separating the roles of perception and control. We will clarify that the OGB-Cube identity control also reduces perceptual width.