- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ViHaD, a framework that jointly learns a low-dimensional latent state representation and Hamiltonian dynamics directly from image sequences for accurate long-horizon prediction. ViHaD incorporates higher-order centered five-point local dynamics supervision, non-collapse latent regularization, and progressive closed-loop rollout training. Evaluated on pendulum, mass-spring, and two-body systems, ViHaD significantly improves visual prediction accuracy and maintains physical consistency over extended autonomous rollouts.
Critical strengths
Optimized Training and Learning Mechanisms for Visual Hamiltonian Models: The paper proposes a series of innovative learning and training techniques (including higher-order centered five-point local dynamics supervision, non-collapse regularization constraints, and progressive closed-loop rollout training strategies). These effectively overcome long-standing issues in traditional visual Hamiltonian models (such as PixelHNN), such as latent space collapse, sensitivity to initialization, and difficulty in long-horizon training. These techniques enable visual Hamiltonian models to be trained and learned more stably and robustly, thereby achieving unprecedented long-horizon prediction accuracy in complex visual dynamical systems.
Critical weaknesses
-
Lack of True Interpretability: Although the framework introduces Hamiltonian neural networks to model physical energy, the latent space coordinates inferred by its state encoder (such as generalized coordinates and conjugate momenta ) do not correspond to true physical simulator coordinates. These latent states are merely a set of abstract mathematical representations that "happen to work" under image reconstruction losses and vector field supervision. This introduces a severe "black-box" property: users cannot directly interpret or extract true physical laws (such as absolute physical scales or constants) from these latent states, nor is it easy to safely bridge such abstract latent spaces—devoid of physical anchors—into downstream embodied intelligence control or real-world robotic interactions.
-
Overly Simplistic Evaluation Scenarios and Lack of Real-World Video Validation: All experiments in the paper are limited to ideal synthetic datasets perfectly rendered by computers using mathematical formulas (including pendulum, mass-spring, and planar two-body systems). These scenarios feature clean backgrounds, no lighting variations, no occlusions, and strictly adhere to the assumption of friction-free conservative systems. However, the framework has never been validated on any real-world videos featuring complex background noise, lighting interference, motion blur, and frictional dissipation. This casts a huge question mark over its generalization capability and robustness when handling real-world visual noise.
Questions for authors
-
Have the authors tested this framework on real-world videos containing background noise, lighting interference, motion blur, or frictional dissipation? When facing real-world visual noise, can the encoder still reliably extract well-behaved phase-space states?
-
ViHaD is strictly built upon conservative Hamiltonian systems (assuming total mechanical energy conservation). However, the vast majority of real-world physical processes involve friction, damping, or energy dissipation. How do the authors plan to extend this framework to non-conservative, damped, or externally controlled systems?
-
Although the model employs a Hamiltonian structure to constrain dynamics, the inferred latent space coordinates ( and ) do not correspond to true physical simulator coordinates. How can the physical interpretability be verified in this case?
4 comments
Response to Reviewer W6uZ — Part 2/2: Real-world video, extensions, and physical interpretation
Q1. Have you tested real-world video or visual noise?
Not yet. The current experiments use clean rendered observations with known underlying dynamics so that representation learning and rollout behavior can be evaluated without confounding visual factors.
We therefore cannot claim that the current encoder will remain reliable under real-world background noise, lighting variation, motion blur, or occlusion. Testing and adapting ViHaD for such conditions is an important next step, and the conclusion already identifies learning from real-world video as a future direction.
Paper reference: real-world video is explicitly identified as a future direction in the conclusion, Lines 488–490. This point is a scope clarification rather than a newly added experiment.
Q2. How would ViHaD extend to dissipative or controlled systems?
The present vector-field parameterization assumes conservative autonomous Hamiltonian dynamics. Extending ViHaD to damped, forced, or controlled systems would require changing the dynamics model rather than treating those effects as if they were conservative.
A natural direction is to augment or replace the conservative Hamiltonian vector field with an appropriate dissipative, port-Hamiltonian, or controlled-Hamiltonian formulation; related examples are already discussed in the paper's Related Work. The representation-learning components, higher-order local supervision, non-collapse regularization, and closed-loop rollout-training strategy could then be used with that modified dynamics parameterization. We do not claim that the current model already handles these settings; they are future extensions.
Paper reference: the Related Work already points to controlled and non-conservative Hamiltonian extensions—Symplectic ODE-Net, port-Hamiltonian networks, and dissipative Hamiltonian models—in Lines 87–90. The current method itself is explicitly introduced for conservative mechanical systems in Lines 152–153. The proposed extension remains future work and is not claimed as part of the current model.
Q3. How can physical interpretability be verified if latent differ from simulator coordinates?
We distinguish physical-coordinate recovery from dynamical-structure recovery. ViHaD does not guarantee that its learned coincide numerically with the simulator coordinates. Hamiltonian dynamics are not unique to a single coordinate representation, and the model receives no simulator-state labels that would anchor the latent coordinates to the simulator's units or scale.
What we verify is that the learned representation contains and organizes physically meaningful state information. For the pendulum, the latent space separates the three motion families and is smoothly ordered by physical energy within each family. A single fixed post-hoc readout, fitted on training-split encodings and never used during ViHaD training, maps held-out encoded and rolled-out states to coherent physical phase portraits (Appendix A.5, Figure 7). For mass–spring, encoded and autonomous trajectories form nested energy-ordered closed orbits (Figure 6).
These analyses support the claim that the latent state is dynamically meaningful; they do not establish identifiable recovery of simulator coordinates or physical constants.
Paper reference: the non-identifiability of the learned coordinates is stated in Lines 155–161. Appendix A.5 gives the phase-space evidence in Lines 864–889, and Figure 7's held-out encoded/rollout/ground-truth comparison is described in Lines 946–952.
Response to rebuttal
Thank you for your explanation. However, I am not convinced that the proposed model will work beyond toy examples. In addition, as for its theoretical contribution, the physical interpretability is not clear, or at least not well proven. Thus, I maintain my initial rating.
Clarifying ViHaD’s dynamical interpretability
Thank you for the follow-up. We agree that the absence of real-world experiments is a limitation. We do not claim real-world robustness; extending ViHaD to real video is left as future work. Our evaluation uses controlled rendered systems, as is common for visual Hamiltonian methods, so that long-horizon rollouts can be checked against known dynamics.
To clarify the interpretability point: the paper does not present physical interpretability as a contribution, and makes no theoretical identifiability claim. Two notions should be separated.
Coordinate identifiability asks whether a latent coordinate equals the simulator's angle, momentum, scale, or physical parameters. ViHaD does not claim this: it is trained without simulator-state supervision, so its latent coordinates are not assumed to match the simulator coordinates (Lines 161, 173–174), nor do they need to. The same Hamiltonian dynamics can be written in many coordinate systems, and accurate prediction only requires that the model learn one of them consistently.
Recovery of dynamical structure asks whether the latent space arranges trajectories the way the physics does. This is what Appendix A.5 evaluates.
- Mass–spring (Figure 6). Physically, each trajectory is a closed loop, and higher-energy loops enclose lower-energy ones. The latent space shows the same picture: one closed loop per trajectory, nested in the same energy order, stretched in shape but never crossing. Autonomous 100-step rollouts, started from a single encoded state with no further observations, closely retrace these loops. No readout is involved.
- Pendulum (Figures 3 and 7). A pendulum can swing back and forth or spin fully around in either direction. Without any labels, the latent space places these three kinds of motion in three separate, non-overlapping regions. To view this in physical terms, we fit one readout to after training and apply it unchanged to both encoded observations and rollouts. Across 28 held-out trajectories and 192 steps, every swinging trajectory keeps swinging and every spinning one keeps spinning in its original direction, as in the ground truth.
Our claim is therefore dynamical-structure recovery, not coordinate-wise recovery of simulator variables or physical constants.
Response to Reviewer W6uZ — Part 1/2: Interpretability and evaluation scope
We thank the reviewer for the careful reading and for recognizing the long-horizon improvements over PixelHNN.
W1. Lack of true interpretability
We agree that ViHaD does not recover simulator coordinates, absolute physical units, or physical constants, and we do not claim that it does. The goal is to learn a dynamically useful Hamiltonian latent phase space rather than an identifiable simulator coordinate system.
The learned representation nevertheless preserves meaningful physical organization. Without regime labels, the pendulum latent space separates libration and both rotation directions, with physical energy varying smoothly within each family (Figure 3). On mass–spring, circular physical trajectories map to nested, energy-ordered closed latent orbits (Figure 6). Appendix A.5 further uses one fixed post-hoc readout, trained only after ViHaD, to map pendulum latents to physical-state variables for evaluation. These results support dynamical-structure recovery, not coordinate-wise identification.
Paper reference: the method explicitly states that the learned coordinates need not match simulator coordinates in Lines 155–161. The phase-space analysis is in Appendix A.5, Lines 864–889, with the held-out/readout rollout comparison in Figure 7, Lines 946–952.
W2. No real-world video validation
We agree that the current evaluation is limited to controlled conservative systems. This is intentional: the experiments are designed to isolate latent-state learning and error accumulation during long autonomous integration under known dynamics.
PixelHNN, HGN++, COMET and Hamiltonian GAN also evaluate their image-based models on rendered simulations.
We do not claim robustness to real-world video with complex backgrounds, lighting changes, occlusion, or motion blur. These settings are outside the present scope. The paper's conclusion already identifies real-world video as a future direction. Our present contribution is therefore narrower: establishing the behavior of the proposed representation-learning and rollout-training mechanisms in controlled visual Hamiltonian systems.
Paper reference: the conclusion limits the forward-looking claim to future work on more complex dynamics and real-world video in Lines 484–490. No real-world robustness claim is made elsewhere in the current revision.