- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that physical structure belongs in the latent transition itself rather than in an auxiliary loss or a post-hoc metric. It keeps a backbone encoder and decoder fixed and inserts a bidirectional interface that maps each backbone latent into a phase state and back through . The phase state evolves under a separable Hamiltonian of kinetic plus potential energy with a learned inverse mass matrix, plus a learned momentum-proportional damping, and is rolled out with a semi-implicit symplectic Euler step over future steps. Training adds a phase-space reconstruction loss and a multi-step transition-consistency loss to the backbone objective. The module is dropped into SimVP for a simulation study of position, velocity, acceleration and jerk errors and of energy drift, into Ctrl-World and EV AC for EWMBench generation metrics, and into Cosmos-Predict2.5-2B for LIBERO and Robocasa success rates. The reported effects span 0.08 to 19.3 percent error reduction in simulation, and relative gains of 2.7 percent on Robocasa, 9.7 percent on Ctrl-World nDTW and 8.4 percent on EV AC Logic.
Critical strengths
- The framing is clean and, more importantly, portable. Putting the structure in the transition law and keeping the interface through and is what lets the same module be instantiated on four different world-model families, and reaching LIBERO and Robocasa task success at all is a harder test than another generation benchmark. Why it matters for acceptance is that this portability is the property other groups could build on.
- The evaluation probes dynamics rather than only appearance. Position, velocity, acceleration, jerk and energy drift are closer to the question the paper asks than FVD would be, and Table 6 reports per-task results over three seeds, so a reader can see that 15 of 24 Robocasa tasks improve instead of only the average.
Critical weaknesses
- The experiments do not isolate physical structure as the cause of the gains, and the one place where the physics is measured directly is not reported on a consistent scale. In the simulation study the only comparison is the same model with the Hamiltonian constraint removed, which also removes its parameters and its training signal, and the three backbone experiments compare against the untouched backbone. No parameter count, training budget or compute is reported for either arm, so a reader cannot tell whether the arms are capacity- or compute-matched, and none of the three mechanisms the introduction credits is ablated. In that study, position RMSE moves from 6.217 to 6.212 at 30 steps, the large gains sit on acceleration and jerk, at 15.8 and 19.3 percent, which come from finite differences of a tracked object centre, and one row of the long-horizon table prints MSE 2.76 beside MAE 23.88, which cannot hold on a common scale since MAE never exceeds RMSE and that MSE implies 1.66. A component ablation, a parameter-matched unstructured transition and a consistent metric scale in the same SimVP setting would settle this.
- The headline numbers are single runs reported as relative percentages, and they sit inside the seed variation that the paper reports elsewhere. Tables 1 to 4 give one number per cell with no seed, no deviation and no sample size, and no standard deviation or confidence interval appears anywhere in the paper. The one multi-seed table shows what that costs. The per-task Robocasa range across three seeds reaches 24 points on PnPCounterToStove and has a median of 8, both larger than the 1.8 point average that the abstract advertises as a 2.7 percent relative gain. Even counting each of the 3600 trials stated per arm as an independent sample, the most generous reading available, an exact test gives . The EWMBench Logic column reads at one decimal as 28 of 30 against 29 of 30 on Ctrl-World and 24 of 30 against 26 of 30 on EV AC, for which exact tests give and , while the neighbouring columns move by 0.001 to 0.03. Reporting seeds with intervals, absolute points beside relative ones, and the number of evaluated rollouts per metric would make these comparisons assessable. As printed, the reported gains cannot be separated from run-to-run noise, which is also why I cannot use them to support the mechanism claim above.
Questions for authors
- How many rollouts does each cell of Tables 1 to 3 average, and are those cells single runs? For the Logic column, please give the denominator.
- HWM assumes a backbone latent and a matching decoder. How is it inserted into Cosmos-Predict2.5-2B, where the reported endpoint is task success rather than generated video, and what did each arm cost in training steps and GPU hours?
- The Hamiltonian is conservative, but the rollout adds damping that removes energy. In the conservative case of Fig. 4, why should the damped model track ground-truth energy more closely than the undamped baseline?
- Which dataset and simulator produce the trajectories in Section 5.1, and how is ground-truth energy defined there? Please also give the SimVP reference, and the differentiation scheme for acceleration and jerk.
2 comments
Response to Q2: Cosmos integration and computational scale
-
HWM is integrated into the visual-latent evolution path of Cosmos, and task success is evaluated through the original policy and planning pipeline. During training, current and future visual latents provide supervision for phase-space reconstruction and evolution, and action conditions can enter the momentum update through the control term. When post-generation correction is enabled, the evolved state is mapped back to the original latent space; the optional difference residual adds the predicted change to the current latent. The comparison in Appendix D keeps the remaining policy and planning components unchanged, so success rates assess the practical value of this integration for closed-loop execution.
-
The Cosmos interface preserves the original visual-latent shape, while structured evolution takes place in a smaller phase space.
| Configuration | Visual-latent dimension | Dimension of each of and | HWM parameters | Relative to the nominal 2B backbone |
|---|---|---|---|---|
| LIBERO integration | 12,544 | 128 | 22.30M | Approximately 1.1% |
| RoboCasa integration | 12,544 | 64 | 32.89M | Approximately 1.6% |
Hamiltonian training objectives operate on clean latents. Inference correction is applied after diffusion sampling is complete and is not repeated at every denoising step. Its additional computation therefore comes from the mappings and phase-space rollout, with both the parameter scale and the point of invocation explicitly specified.
Response to Q3: Conservative systems, the nonconservative branch, and energy drift
- The model includes a nonconservative branch and does not require every predicted trajectory to continuously lose energy. Conservative dynamics correspond to . With external control and nonconservative effects, the energy change is
If with , then, without control, . For the vector-valued nonconservative branch, the energy change depends on ; its sign cannot be inferred simply from the name “damping.”
- The advantage shown in Fig. 4 is that the physical energy of predicted trajectories is closer to the ground truth. The non-Hamiltonian baseline has no conservation constraint and can also produce spurious energy increases or oscillations. HWM's structured evolution and multi-step supervision improve motion prediction and can consequently reduce physical-energy deviations. This result supports the trajectory consistency of the complete model. It neither requires pointwise equality between the latent Hamiltonian and mechanical energy nor attributes the improvement solely to the damping term.
We thank the reviewer for recognizing the transferability of the interface and the dynamics-based evaluation. Below, we focus on the ablations, metric definitions, three-seed results, and integration with Cosmos.
Response to W1 and Q4: Simulation ablations, error definitions, and physical evaluation
Relative to the Vanilla baseline, Table 5 shows lower MSE and MAE across all seven training/test-horizon combinations. When trained to predict 10 frames and evaluated over 60 frames, the two errors decrease by 14.9% and 13.4%, respectively. The constraint-removal ablation in Table 1 and the long-horizon prediction results in Table 5 jointly support the advantages of the complete approach; they do not constitute separate, parameter-matched ablations of structure, capacity, and supervision objectives.
- For the spatial-sum metrics used by SimVP/OpenSTL, the relationship between MAE and MSE must retain the spatial-dimensionality factor. Let en,t,i denote the prediction error and D=CHW. The default implementation averages over samples and time while summing over channels and spatial dimensions:
=\frac{1}{NT}\sum_{n,t}\sum_{i=1}^{D}e_{n,t,i}^{2}, \qquad \mathrm{MAE}_{\mathrm{sum}} =\frac{1}{NT}\sum_{n,t}\sum_{i=1}^{D}|e_{n,t,i}|.$$ The applicable inequality is therefore $$\mathrm{MAE}_{\mathrm{sum}} \leq\sqrt{D\,\mathrm{MSE}_{\mathrm{sum}}}.$$ Both metrics are divided by $D$ when converted to elementwise means. Under this reduction, an MSE of 2.76 and an MAE of 23.88 do not constitute the arithmetic contradiction raised in the review. - **Higher-order motion metrics are derived from object trajectories in the predicted videos, using the procedure described in Appendix B.** We estimate the object center in each frame, obtain velocity through finite differences between consecutive positions, and compute acceleration and jerk through successive temporal differences: $$v_t=\frac{x_{t+1}-x_t}{\Delta t},\qquad a_t=\frac{v_{t+1}-v_t}{\Delta t},\qquad j_t=\frac{a_{t+1}-a_t}{\Delta t}.$$ Fig. 3 presents velocity curves for constant-speed, accelerated, and bouncing motion. Fig. 4 compares the physical energy computed from the recovered motion states and its drift. For the projectile motion under uniform gravity discussed in Appendix B, for example, the mechanical energy is $E=\tfrac12m\|v\|^2+mgy$. This physical evaluation quantity is not required to be numerically equal to the network's internal latent Hamiltonian $H(q,p)$. We have added the citation to SimVP in the revised manuscript. ### Response to W2: Three-seed aggregation, absolute gains, and task-level distribution - **The downstream results in Table 4 are means over three seeds, following the Cosmos evaluation protocol.** The gains are 0.9 percentage points on LIBERO and 1.8 percentage points on RoboCasa. Absolute changes in prediction on real-robot data can also be read directly from Tables 2–3: | Evaluation | Metric | Vanilla | HWM | Absolute change | |---|---|---:|---:|---:| | LIBERO | Mean success rate | 97.8% | **98.7%** | +0.9 percentage points | | RoboCasa | Mean success rate | 66.7% | **68.5%** | +1.8 percentage points | | Ctrl-World | nDTW | 0.670 | **0.735** | +0.065 | | Ctrl-World | Logic | 93.3% | **96.7%** | +3.4 percentage points | | EVAC | Logic | 80.0% | **86.7%** | +6.7 percentage points | - **Table 6 provides per-task, per-seed results that characterize how broadly the gains are distributed.** HWM improves the three-seed mean on 15 tasks, ties on four, and declines on five. A paired bootstrap over tasks, retaining the three paired seed results within each task, gives a mean improvement of **1.81 percentage points** and a 95% interval of **[0.19, 3.42]** percentage points. This analysis characterizes variation across tasks conditional on the fixed seed results, while preserving the association among evaluations of the same task.