acceptodds
Withdrawn from ICLR 2027

Hamiltonian World Model for Physically Consistent Embodied Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

17 views

Abstract

Embodied world models are increasingly expected to serve not only as video generators, but as internal simulators for action-conditioned prediction. However, most existing models parameterize latent transitions with generative modules, leaving physical structure to be learned implicitly from data. This design may accumulate physical drift over long horizons, especially in contact-rich manipulation. We propose Hamiltonian World Model (), a physically structured latent transition framework. The core concept of is to move physical inductive bias from auxiliary regularization or post-hoc evaluation into the transition law itself: backbone latents are mapped into a learned phase space of generalized coordinates and momenta, evolved through dissipative Hamiltonian dynamics. A bidirectional phase-space interface and a multi-step transition-consistency objective keep the structured rollout compatible with existing world-model encoders and decoders, making a drop-in replacement for the latent transition module. In simulation, improves speed, acceleration, and jerk prediction while better preserving system energy than unstructured transitions. We further instantiate across three embodied world-model settings: Ctrl-World for single-arm multi-view manipulation, EVAC for dual-arm single-view manipulation, and Cosmos-Predict2.5-2B for single-arm single-view task execution. On Robocasa, Cosmos-Predict2.5-2B with achieves relative success-rate gains of 2.7%; on real-robot AgiBot World data, delivers relative gains of 9.7% on nDTW with Ctrl-World and 8.4% on Logic with EVAC. These results suggest that physically structuring the latent transition is a practical path toward embodied world models whose imagined futures are more stable, more physically faithful, and more useful for action. The project page is available at https://hamilton-world-model.github.io/Hamiltonian_World_Model/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Hamiltonian

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tYchOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that physical structure belongs in the latent transition itself rather than in an auxiliary loss or a post-hoc metric. It keeps a backbone encoder and decoder fixed and inserts a bidirectional interface that maps each backbone latent ztz_t into a phase state (qt,pt)(q_t, p_t) and back through ψ\psi. The phase state evolves under a separable Hamiltonian of kinetic plus potential energy with a learned inverse mass matrix, plus a learned momentum-proportional damping, and is rolled out with a semi-implicit symplectic Euler step over KK future steps. Training adds a phase-space reconstruction loss and a multi-step transition-consistency loss to the backbone objective. The module is dropped into SimVP for a simulation study of position, velocity, acceleration and jerk errors and of energy drift, into Ctrl-World and EV AC for EWMBench generation metrics, and into Cosmos-Predict2.5-2B for LIBERO and Robocasa success rates. The reported effects span 0.08 to 19.3 percent error reduction in simulation, and relative gains of 2.7 percent on Robocasa, 9.7 percent on Ctrl-World nDTW and 8.4 percent on EV AC Logic.

Critical strengths
  1. The framing is clean and, more importantly, portable. Putting the structure in the transition law and keeping the interface through ϕ\phi and ψ\psi is what lets the same module be instantiated on four different world-model families, and reaching LIBERO and Robocasa task success at all is a harder test than another generation benchmark. Why it matters for acceptance is that this portability is the property other groups could build on.
  2. The evaluation probes dynamics rather than only appearance. Position, velocity, acceleration, jerk and energy drift are closer to the question the paper asks than FVD would be, and Table 6 reports per-task results over three seeds, so a reader can see that 15 of 24 Robocasa tasks improve instead of only the average.
Critical weaknesses
  1. The experiments do not isolate physical structure as the cause of the gains, and the one place where the physics is measured directly is not reported on a consistent scale. In the simulation study the only comparison is the same model with the Hamiltonian constraint removed, which also removes its parameters and its training signal, and the three backbone experiments compare against the untouched backbone. No parameter count, training budget or compute is reported for either arm, so a reader cannot tell whether the arms are capacity- or compute-matched, and none of the three mechanisms the introduction credits is ablated. In that study, position RMSE moves from 6.217 to 6.212 at 30 steps, the large gains sit on acceleration and jerk, at 15.8 and 19.3 percent, which come from finite differences of a tracked object centre, and one row of the long-horizon table prints MSE 2.76 beside MAE 23.88, which cannot hold on a common scale since MAE never exceeds RMSE and that MSE implies 1.66. A component ablation, a parameter-matched unstructured transition and a consistent metric scale in the same SimVP setting would settle this.
  2. The headline numbers are single runs reported as relative percentages, and they sit inside the seed variation that the paper reports elsewhere. Tables 1 to 4 give one number per cell with no seed, no deviation and no sample size, and no standard deviation or confidence interval appears anywhere in the paper. The one multi-seed table shows what that costs. The per-task Robocasa range across three seeds reaches 24 points on PnPCounterToStove and has a median of 8, both larger than the 1.8 point average that the abstract advertises as a 2.7 percent relative gain. Even counting each of the 3600 trials stated per arm as an independent sample, the most generous reading available, an exact test gives p=0.11p = 0.11. The EWMBench Logic column reads at one decimal as 28 of 30 against 29 of 30 on Ctrl-World and 24 of 30 against 26 of 30 on EV AC, for which exact tests give p=1.0p = 1.0 and p=0.73p = 0.73, while the neighbouring columns move by 0.001 to 0.03. Reporting seeds with intervals, absolute points beside relative ones, and the number of evaluated rollouts per metric would make these comparisons assessable. As printed, the reported gains cannot be separated from run-to-run noise, which is also why I cannot use them to support the mechanism claim above.
Questions for authors
  1. How many rollouts does each cell of Tables 1 to 3 average, and are those cells single runs? For the Logic column, please give the denominator.
  2. HWM assumes a backbone latent ztz_t and a matching decoder. How is it inserted into Cosmos-Predict2.5-2B, where the reported endpoint is task success rather than generated video, and what did each arm cost in training steps and GPU hours?
  3. The Hamiltonian is conservative, but the rollout adds damping γθ(p)p\gamma_\theta(p)p that removes energy. In the conservative case of Fig. 4, why should the damped model track ground-truth energy more closely than the undamped baseline?
  4. Which dataset and simulator produce the trajectories in Section 5.1, and how is ground-truth energy defined there? Please also give the SimVP reference, and the differentiation scheme for acceleration and jerk.
2 comments

We thank the reviewer for recognizing the transferability of the interface and the dynamics-based evaluation. Below, we focus on the ablations, metric definitions, three-seed results, and integration with Cosmos.

Response to W1 and Q4: Simulation ablations, error definitions, and physical evaluation

  • Appendix B explicitly compares the full HWM, a variant without the Hamiltonian constraint, and ground-truth trajectories. Table 1 shows lower position, velocity, acceleration, and jerk errors for both 30-frame and 50-frame prediction, with the largest improvements in higher-order motion quantities.
Prediction horizonMetricWithout Hamiltonian constraintHWMError reduction
30 framesAcceleration RMSE10.9669.23515.8%
30 framesJerk RMSE14.13511.40119.3%
50 framesAcceleration RMSE8.3327.48310.2%
50 framesJerk RMSE12.32910.75012.8%

Relative to the Vanilla baseline, Table 5 shows lower MSE and MAE across all seven training/test-horizon combinations. When trained to predict 10 frames and evaluated over 60 frames, the two errors decrease by 14.9% and 13.4%, respectively. The constraint-removal ablation in Table 1 and the long-horizon prediction results in Table 5 jointly support the advantages of the complete approach; they do not constitute separate, parameter-matched ablations of structure, capacity, and supervision objectives.

  • For the spatial-sum metrics used by SimVP/OpenSTL, the relationship between MAE and MSE must retain the spatial-dimensionality factor. Let en,t,ie_{n,t,i} denote the prediction error and D=CHWD=CHW. The default implementation averages over samples and time while summing over channels and spatial dimensions:
=\frac{1}{NT}\sum_{n,t}\sum_{i=1}^{D}e_{n,t,i}^{2}, \qquad \mathrm{MAE}_{\mathrm{sum}} =\frac{1}{NT}\sum_{n,t}\sum_{i=1}^{D}|e_{n,t,i}|.$$ The applicable inequality is therefore $$\mathrm{MAE}_{\mathrm{sum}} \leq\sqrt{D\,\mathrm{MSE}_{\mathrm{sum}}}.$$ Both metrics are divided by $D$ when converted to elementwise means. Under this reduction, an MSE of 2.76 and an MAE of 23.88 do not constitute the arithmetic contradiction raised in the review. - **Higher-order motion metrics are derived from object trajectories in the predicted videos, using the procedure described in Appendix B.** We estimate the object center in each frame, obtain velocity through finite differences between consecutive positions, and compute acceleration and jerk through successive temporal differences: $$v_t=\frac{x_{t+1}-x_t}{\Delta t},\qquad a_t=\frac{v_{t+1}-v_t}{\Delta t},\qquad j_t=\frac{a_{t+1}-a_t}{\Delta t}.$$ Fig. 3 presents velocity curves for constant-speed, accelerated, and bouncing motion. Fig. 4 compares the physical energy computed from the recovered motion states and its drift. For the projectile motion under uniform gravity discussed in Appendix B, for example, the mechanical energy is $E=\tfrac12m\|v\|^2+mgy$. This physical evaluation quantity is not required to be numerically equal to the network's internal latent Hamiltonian $H(q,p)$. We have added the citation to SimVP in the revised manuscript. ### Response to W2: Three-seed aggregation, absolute gains, and task-level distribution - **The downstream results in Table 4 are means over three seeds, following the Cosmos evaluation protocol.** The gains are 0.9 percentage points on LIBERO and 1.8 percentage points on RoboCasa. Absolute changes in prediction on real-robot data can also be read directly from Tables 2–3: | Evaluation | Metric | Vanilla | HWM | Absolute change | |---|---|---:|---:|---:| | LIBERO | Mean success rate | 97.8% | **98.7%** | +0.9 percentage points | | RoboCasa | Mean success rate | 66.7% | **68.5%** | +1.8 percentage points | | Ctrl-World | nDTW | 0.670 | **0.735** | +0.065 | | Ctrl-World | Logic | 93.3% | **96.7%** | +3.4 percentage points | | EVAC | Logic | 80.0% | **86.7%** | +6.7 percentage points | - **Table 6 provides per-task, per-seed results that characterize how broadly the gains are distributed.** HWM improves the three-seed mean on 15 tasks, ties on four, and declines on five. A paired bootstrap over tasks, retaining the three paired seed results within each task, gives a mean improvement of **1.81 percentage points** and a 95% interval of **[0.19, 3.42]** percentage points. This analysis characterizes variation across tasks conditional on the fixed seed results, while preserving the association among evaluations of the same task.

Response to Q2: Cosmos integration and computational scale

  • HWM is integrated into the visual-latent evolution path of Cosmos, and task success is evaluated through the original policy and planning pipeline. During training, current and future visual latents provide supervision for phase-space reconstruction and evolution, and action conditions can enter the momentum update through the control term. When post-generation correction is enabled, the evolved state is mapped back to the original latent space; the optional difference residual adds the predicted change to the current latent. The comparison in Appendix D keeps the remaining policy and planning components unchanged, so success rates assess the practical value of this integration for closed-loop execution.

  • The Cosmos interface preserves the original visual-latent shape, while structured evolution takes place in a smaller phase space.

ConfigurationVisual-latent dimensionDimension of each of qq and ppHWM parametersRelative to the nominal 2B backbone
LIBERO integration12,54412822.30MApproximately 1.1%
RoboCasa integration12,5446432.89MApproximately 1.6%

Hamiltonian training objectives operate on clean latents. Inference correction is applied after diffusion sampling is complete and is not repeated at every denoising step. Its additional computation therefore comes from the mappings and phase-space rollout, with both the parameter scale and the point of invocation explicitly specified.

Response to Q3: Conservative systems, the nonconservative branch, and energy drift

  • The model includes a nonconservative branch and does not require every predicted trajectory to continuously lose energy. Conservative dynamics correspond to D=0,u=0D=0,u=0. With external control and nonconservative effects, the energy change is

H˙=−∇pH⊤D(p)+∇pH⊤g(q)u.\dot H=-\nabla_pH^\top D(p)+\nabla_pH^\top g(q)u.

If D(p)=γ(p)pD(p)=\gamma(p)p with γ(p)≥0\gamma(p)\geq0, then, without control, H˙=−γ(p)p⊤M−1(q)p≤0\dot H=-\gamma(p)p^\top M^{-1}(q)p\leq0. For the vector-valued nonconservative branch, the energy change depends on ∇pH⊤D(p)\nabla_pH^\top D(p); its sign cannot be inferred simply from the name “damping.”

  • The advantage shown in Fig. 4 is that the physical energy of predicted trajectories is closer to the ground truth. The non-Hamiltonian baseline has no conservation constraint and can also produce spurious energy increases or oscillations. HWM's structured evolution and multi-step supervision improve motion prediction and can consequently reduce physical-energy deviations. This result supports the trajectory consistency of the complete model. It neither requires pointwise equality between the latent Hamiltonian and mechanical energy nor attributes the improvement solely to the damping term.
Reviewer_fJAwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HWM, which maps world-model latents into learned coordinates and momenta and evolves them using dissipative Hamiltonian dynamics. The method is integrated into SimVP, Ctrl-World, EVAC, and Cosmos. Experiments on controlled simulations, Ctrl-World, EVAC, and Cosmos-Predict2.5-2B demonstrate improved higher-order motion prediction, energy preservation, temporal and logical video consistency, and modest success-rate gains on the simulated LIBERO and RoboCasa benchmarks.

Critical strengths
  1. Introduces a physically motivated transition module compatible with several world-model backbones.
  2. Evaluates long-horizon motion, energy behavior, visual generation, and downstream task success.
  3. Tests across multiple embodiments and both simulation and real-robot datasets.
Critical weaknesses
  1. Limited novelty: The main contribution is adapting established Hamiltonian dynamics methods to large video-based embodied world models rather than introducing a new dynamics operator.
  2. The learned q and p do not necessarily correspond to physical positions and momenta, so the Hamiltonian form does not guarantee physically meaningful dynamics.
  3. There are no sufficient ablations separating Hamiltonian structure from the additional phase-space mappings, multi-step loss, damping, residual branch, and increased capacity.
  4. Several gains are small or mixed, and uncertainty estimates or significance tests are largely absent.
  5. Task execution is evaluated only in simulation; using real-robot datasets is not equivalent to real-robot deployment.
Questions for authors
  1. Can the authors ablate the symplectic Euler integrator, damping, multi-step loss, and residual module, and compare against a parameter-matched unstructured dynamics baseline?
  2. Since the learned q,p are abstract latents, can the authors analyze whether they encode meaningful motion or physical information?
  3. What training and inference overhead does HWM introduce? Does it create a latency bottleneck for online planning?
  4. Can the authors provide real-robot closed-loop experiments?
2 comments

We thank the reviewer for recognizing the compatibility across backbones, the evaluation at multiple levels, and the coverage of different embodied settings. Below, we address the methodological contributions, physical interpretation, and experimental evidence.

Response to W1, W3, and Q1: Methodological contributions and component evidence

  • HWM provides video world models with a trainable phase-space interface that supports state evolution and maps the resulting states back to visual representations. Its key designs include bidirectional latent mappings, controlled Hamiltonian evolution, and reconstruction and multi-step consistency objectives optimized jointly with the backbone. Together, these components address representation compatibility between structured dynamics and high-dimensional visual generation, enabling existing video models to use coordinate–momentum coupling for future prediction.
DesignRole
Bidirectional mappings ϕ,ψ\phi,\psiConnect backbone visual representations to a low-dimensional phase space while preserving compatibility with the decoding interface
Hamiltonian coreCouple coordinate and momentum changes through gradients of the same energy function
Action and nonconservative branchesRepresent control inputs and nonconservative interactions
Reconstruction objectiveEncourage the phase-space mappings to retain information from the backbone
Multi-step consistency objectiveAlign states after successive evolution steps with future visual latents
  • Table 1 and Appendix B already include an ablation that removes the Hamiltonian constraint, with gains concentrated in dynamics-related metrics. At a 30-frame prediction horizon, acceleration RMSE decreases from 10.966 to 9.235 and jerk RMSE from 14.135 to 11.401, corresponding to reductions of 15.8% and 19.3%, respectively. At 50 frames, the reductions remain 10.2% and 12.8%. Table 5 also shows lower MSE and MAE in all seven prediction-horizon settings.

These results support the advantages of the complete approach for motion prediction and long-horizon extrapolation. The component table above explains the mechanisms of the individual modules; the existing overall ablation is not presented as evidence of the independent benefit of damping, the integrator, the residual, or the multi-step objective.

Response to W2 and Q2: Physical interpretation of latent coordinates and momenta

  • The role of q,pq,p is to organize a predictive dynamical state, and their physical utility is assessed through the output trajectories. The reconstruction constraint encourages the phase space to retain visual information, while the multi-step constraint aligns its evolution with future observations. The variables q,pq,p need not correspond componentwise to physical positions and momenta; their utility is reflected in whether motion states recovered from predicted videos are more accurate.

  • The physical evaluation includes position, velocity, acceleration, jerk, and energy evolution. All four motion errors in Table 1 decrease at both prediction horizons; the HWM velocity curves in Fig. 3 more closely follow the ground-truth trajectories; and Fig. 4 shows smaller deviations in physical energy. These results directly evaluate dynamics prediction rather than relying only on visual similarity. The learned latent Hamiltonian H(q,p)H(q,p) and the mechanical energy computed from physical states belong to different representations and are not required to have the same numerical values or units.

Response to W4 and Q3: Reliability of the gains and computational overhead

  • The downstream results are means over three seeds under a protocol aligned with Cosmos, and can be interpreted together with the distribution of task-level gains.
BenchmarkVanillaHWMAbsolute improvement
LIBERO97.8%98.7%+0.9 percentage points
RoboCasa66.7%68.5%+1.8 percentage points

Of the 24 RoboCasa tasks, 15 improve, four remain unchanged, and five decline. A task bootstrap that preserves the within-task pairing across the three seeds gives a 95% interval of [0.19, 3.42] percentage points for the average improvement. Despite the high baseline, HWM improves three LIBERO subsets, including a gain of 1.8 percentage points on Goal.

  • The added module is small relative to the video backbone, and dynamics computation is concentrated in a low-dimensional phase space. The parameter counts for the corresponding integration configurations are:
ConfigurationMapping parametersDynamics-module parametersTotal HWM parameters
EVAC23.35M4.83M28.17M
Cosmos–LIBERO13.65M8.65M22.30M
Cosmos–RoboCasa28.07M4.83M32.89M

The additional parameters in the two Cosmos configurations are approximately 1.1% and 1.6% of the nominal 2B-parameter backbone, respectively. Training adds computation for mapping reconstruction and phase-space rollout. When inference correction is enabled, it is applied after diffusion sampling and is not repeated at every denoising step. At a fixed phase-space dimension, rollout cost grows linearly with the prediction horizon. These parameter ratios describe module size and do not directly equal the corresponding proportions of end-to-end planning latency.

Response to W5 and Q4: Scope of real-robot validation

  • We have not yet conducted large-scale physical-robot closed-loop experiments; the paper evaluates the method through prediction on real-robot data and simulated closed-loop tasks. The real manipulation data cover complex appearance, viewpoint variation, and contact interactions, while closed-loop success rates come from LIBERO and RoboCasa. The corresponding gains in these two types of evaluation are summarized below:
EvaluationMetricVanillaHWM
Ctrl-World, real-robot datanDTW0.6700.735
Ctrl-World, real-robot dataLogic93.3%96.7%
EVAC, real-robot dataLogic80.0%86.7%
LIBERO, simulated closed loopSuccess rate97.8%98.7%
RoboCasa, simulated closed loopSuccess rate66.7%68.5%
Reviewer_6MzQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Hamiltonian World Model (HWM), a structured latent dynamics module for embodied world models. Visual backbone representations are mapped into learned generalized coordinates and momenta, evolved using a learned Hamiltonian with momentum-dependent damping, and mapped back for decoding. Reconstruction and multi-step transition-consistency objectives connect the phase-space dynamics to the backbone representation. The method is evaluated through synthetic physical prediction, video generation using real-robot recordings, and simulated manipulation with LIBERO and RoboCasa. The results suggest benefits from structured latent transitions, although important methodological and experimental questions remain.

Critical strengths
  1. Encouraging empirical results. The method reports improvements across several aggregate metrics, including motion prediction, energy drift, video-generation quality, and downstream task success. The improvements in acceleration and jerk prediction are particularly encouraging given the motivation of improving physical consistency.

  2. Broad evaluation across different settings. The experiments cover synthetic dynamics and several embodied world-model backbones, including Ctrl-World, EVAC, and Cosmos. Evaluating both predictive quality and downstream task execution provides a broader assessment than visual reconstruction metrics alone.

  3. An elegant and economical treatment of damping. Extending Hamiltonian dynamics with a learned momentum-dependent damping term is intuitive and introduces relatively little additional complexity. This provides a practical mechanism for representing dissipative behavior, subject to clarification of the constraints imposed on the damping function.

  4. A useful interface with existing latent representations. The bidirectional mapping between backbone latents and phase-space states is appealing. It allows structured dynamics to operate alongside existing perception and rendering components, making the approach potentially useful for adapting pretrained visual world models.

Critical weaknesses
  1. The downstream improvements are modest and need stronger characterization. Average success increases from 97.8% to 98.7% on LIBERO and from 66.7% to 68.5% on RoboCasa. These improvements are positive, but uncertainty estimates and clarification of training versus evaluation randomness would help establish their reliability. LIBERO’s near-saturated baseline also complicates interpretation. This concern applies primarily to downstream success; some physical prediction and visual reasoning metrics improve more substantially.

  2. The methodological novelty is insufficiently distinguished from prior work. Learning Hamiltonian dynamics, inferring latent phase-space representations, and incorporating dissipation have substantial precedent. The paper discusses related structured world models but does not clearly establish which technical advances are necessary for its proposed integration with embodied video models. A more precise comparison with existing approaches would help distinguish a new modeling contribution from an application of established components.

  3. The experiments have breadth but limited diagnostic depth. Comparisons are primarily against vanilla backbones or a non-Hamiltonian variant. They do not adequately establish whether the improvements arise from Hamiltonian structure, additional model capacity, damping, or the multi-step training objectives. A matched unstructured transition using the same objectives, together with component ablations, would provide stronger evidence for the proposed mechanism. Relevant structured dynamics baselines are also missing.

  4. The representation and scalability of the phase space are unclear. The paper does not provide numerical values for the latent dimensions or explain how spatial and temporal video tokens are mapped into coordinates and momenta. Consequently, it is difficult to assess whether the representation introduces a restrictive bottleneck or preserves high-dimensional structure. The behavior and computational cost as the phase-space dimension increases are unexplored, particularly if the learned mass matrix is dense.

  5. Essential implementation details are missing. The architectures of the phase-space mappings, enforcement of positive-definite mass, and constraints ensuring nonnegative damping are unspecified. The precise integration with video denoising backbones is also unclear. Furthermore, action-dependent control and residual terms appear in the preliminary formulation but are absent from the main rollout equations. These omissions make the claimed plug-and-play integration difficult to assess and reproduce.

  6. The structure-preserving claim requires correction or justification. Equation 12 permits a position-dependent mass matrix, making the Hamiltonian generally nonseparable in coordinates and momenta. However, Equation 15 evaluates the momentum update using the old momentum, unlike the implicit symplectic Euler update in Equation 10. Thus, Equations 15–17 are not generally symplectic even after removing damping. The authors should clarify the implemented update and qualify the associated geometric claims.

  7. The real-world execution framing exceeds the demonstrated evidence. LIBERO and RoboCasa are simulation benchmarks, despite the “real-world” section heading. Experiments using real-robot recordings evaluate offline prediction rather than closed-loop execution on physical robots. The paper should clearly distinguish real-data modeling from physical deployment and restrict its execution claims accordingly.

Questions for authors

Have you compared scalar damping against diagonal or matrix-valued alternatives to assess the accuracy–complexity trade-off? What temporal context does the encoder receive for each backbone, and how does it provide sufficient information to infer momentum?

3 comments

We thank the reviewer for recognizing our physical prediction results, cross-backbone evaluation, and bidirectional phase-space interface. Below, we address related weaknesses and questions together.

Response to W1: Downstream gains and three-seed results

  • Table 4 reports means over three seeds under an evaluation protocol aligned with Cosmos, and HWM improves mean success rates on both closed-loop benchmarks. The average success rate on LIBERO increases from 97.8% to 98.7%, a gain of 0.9 percentage points; on RoboCasa, it increases from 66.7% to 68.5%, a gain of 1.8 percentage points. Three LIBERO suites improve, while Object remains at 99.8%.
Benchmark or suiteVanillaHWMAbsolute change
LIBERO-Spatial97.6%98.6%+1.0 percentage points
LIBERO-Object99.8%99.8%0
LIBERO-Goal96.2%98.0%+1.8 percentage points
LIBERO-Long97.6%98.2%+0.6 percentage points
LIBERO average97.8%98.7%+0.9 percentage points
RoboCasa average66.7%68.5%+1.8 percentage points
  • The RoboCasa improvement extends across most tasks and is not driven by a single task. Among the 24 tasks in Table 6, 15 improve, four remain unchanged, and five decline. OpenDrawer improves from 88.0% to 96.7%, and PnPMicrowaveToCounter from 32.7% to 44.0%. A paired bootstrap over tasks, retaining the three paired seed results within each task, yields a mean improvement of 1.81 percentage points and a 95% interval of [0.19, 3.42] percentage points. This interval characterizes variation across tasks conditional on the evaluated seed results.

Response to W2 and W3: Methodological advantages and ablation evidence

  • HWM's technical contribution is to connect structured dynamics to video world models, allowing physical inductive biases to operate on existing visual representations. Bidirectional mappings connect backbone latents to a low-dimensional phase space; the Hamiltonian core couples coordinate and momentum evolution through energy gradients; and reconstruction and multi-step consistency objectives keep dynamics predictions compatible with the backbone representation. This design reuses existing perception and generation capabilities while imposing structure on state changes over time. Instantiations with SimVP, Ctrl-World, EVAC, and Cosmos demonstrate the applicability of this interface across different models.

  • The main text and Appendix B already include an ablation that removes the Hamiltonian constraint, with particularly clear gains in higher-order motion prediction. Table 1 compares the full HWM with the variant without the Hamiltonian constraint under the same 10-frame training prediction horizon. HWM achieves lower errors on all four motion metrics at both the 30-frame and 50-frame prediction horizons.

Prediction horizonMetricWithout the Hamiltonian constraintHWMError reduction
30 framesAcceleration RMSE10.9669.23515.8%
30 framesJerk RMSE14.13511.40119.3%
50 framesAcceleration RMSE8.3327.48310.2%
50 framesJerk RMSE12.32910.75012.8%

Across all seven training/test-horizon combinations in Table 5, HWM achieves lower MSE and MAE. When trained to predict 10 frames and evaluated over 60 frames, MSE decreases from 62.32 to 53.03 and MAE from 95.32 to 82.58, reductions of 14.9% and 13.4%, respectively. The velocity curves in Fig. 3 and the physical-energy curves in Fig. 4 provide corresponding trajectory-level evidence. These results support the contribution of the complete Hamiltonian modeling scheme; this removal ablation does not constitute a separate parameter-matched comparison of each component.

Response to W4 and Q2: Phase-space dimensionality, mappings, and temporal information

  • The phase-space mappings operate on the visual latent at each temporal position, while the temporal axis forms the evolution sequence. The encoder mapping has widths L→e→e→2dL\rightarrow e\rightarrow e\rightarrow 2d, and the decoder mapping has widths 2d→e→e→L2d\rightarrow e\rightarrow e\rightarrow L, with Tanh activations in the hidden layers; q,p∈Rdq,p\in\mathbb{R}^{d}. The integration configurations are listed below. Parameter counts include only the HWM module, excluding the backbone and LoRA.
Integration configurationPer-frame latent shapeFlattened dimension LLDimension dd of each of q,pq,pMapping hidden width eeHWM parameters
EVAC4×40×644\times40\times6410,240641,02428.17M
Cosmos–LIBERO16×28×2816\times28\times2812,54412851222.30M
Cosmos–RoboCasa16×28×2816\times28\times2812,544641,02432.89M
  • The low-dimensional phase space models dynamics, while reconstruction and multi-step objectives jointly constrain it to retain information needed for prediction. The reconstruction objective constrains the round-trip mapping error, and the multi-step objective constrains the latent decoded after evolution to agree with future observations. For a fixed phase-space dimension, evolution cost grows linearly with the prediction horizon; storing a dense mass matrix requires O(d2)O(d^2) space, and directly forming LL⊤LL^\top costs O(d3)O(d^3).

  • Temporal conditioning is supplied by the backbone interface, and the momentum variables are learned through future-prediction supervision. The standard Ctrl-World configuration uses six conditioning-frame positions. The EVAC configuration uses four conditioning-frame positions and predicts 16 frames; at sequence initialization, these positions can be filled by repeating the first frame. The Cosmos policy uses current visual, proprioceptive, and action-related conditioning, and its multiple conditioning positions include different modalities or views. The variable pp is a latent momentum representation used for state evolution; it does not require uniquely recovering true physical momentum from a single image.

Response to W5, W6, and Q1: Mass matrix, damping, actions, and discrete updates

  • An explicitly positive-definite construction of the inverse mass matrix guarantees a nonnegative kinetic term. The network outputs a lower-triangular matrix LL, whose diagonal entries are set to ∣Lii∣+ϵ|L_{ii}|+\epsilon, with ϵ>0\epsilon>0, and we define
H(q,p)=\frac12p^\top M^{-1}(q)p+V(q).$$ This parameterization allows the effective inertia to vary with the state while directly predicting the inverse mass matrix, without requiring an additional matrix inversion. - **The nonconservative branch and action input allow the model to describe friction, contact, and external driving forces.** After dimension adaptation, actions enter the momentum update through $g(q)u$: $$\dot q=\nabla_pH,\qquad \dot p=-\nabla_qH-D(p)+g(q)u.$$ The implemented $D(p)$ is a vector-valued MLP with widths $d\rightarrow h\rightarrow d$. Its output size grows linearly with $d$, without requiring a full $d\times d$ damping matrix. Its energy relation is $$\dot H=-\nabla_pH^\top D(p)+\nabla_pH^\top g(q)u.$$ Dissipation requires $\nabla_pH^\top D(p)\geq0$. For the scalar form $D(p)=\gamma(p)p$, the condition $\gamma(p)\geq0$ is sufficient. The vector MLP itself does not enforce this sign constraint, so it does not justify claiming that the complete dynamics always dissipate energy. The existing ablation compares the full model with the variant without the Hamiltonian constraint; it does not include independent comparisons of scalar, diagonal, and matrix damping. - **Integration with the backbone uses latent-space training objectives and optional post-generation dynamics correction.** The Hamiltonian training objectives operate on clean latents. When inference correction is enabled, dynamics rollout is performed after diffusion sampling. The optional residual form used in Cosmos is $$\hat z_{t+1}=z_t+ \psi\!\left(F(\phi(z_t),u_t)\right)-\psi\!\left(\phi(z_t)\right),$$ where $F$ denotes the phase-space update. This form preserves the current visual representation and adds the predicted change. It is a latent-difference residual, not a separate phase-space residual-force network $r_\omega$. - **Eqs. 15–17 update momentum before coordinates, avoiding the solution of implicit equations.** When the mass matrix depends on $q$, the Hamiltonian in Eq. 12 is generally nonseparable, and the implemented update does not have an unconditional symplecticity guarantee. The long-horizon advantages reported in this work are supported by the prediction results in Table 1, Table 5, and Figs. 3–4, rather than relying on such a mathematical guarantee.

Response to W7: Real-robot data and the scope of closed-loop execution

  • Prediction on real-robot data and closed-loop simulation jointly evaluate HWM's practical utility; large-scale physical-robot closed-loop experiments have not yet been conducted. Ctrl-World and EVAC evaluate video prediction on real-robot manipulation data, covering complex appearances, viewpoint changes, and contact interactions. LIBERO and RoboCasa provide standardized simulated closed-loop tasks. In Table 2, Ctrl-World nDTW increases from 0.670 to 0.735; in Table 3, EVAC Logic increases from 80.0% to 86.7%. The improvements in closed-loop success rates come from the simulation evaluations in Table 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.