Learning Hamiltonian Dynamics from Images for Long-Horizon Prediction

Neural networks can generate visually plausible frames while failing to preserve physical consistency over time. Physics-based models introduce physical inductive biases to address this limitation, but inferring a useful state representation from images while simultaneously learning the dynamics remains challenging. We propose ViHaD (Visual Hamiltonian Dynamics), a framework that jointly learns a low-dimensional state representation from images and a scalar Hamiltonian governing its evolution. To support accurate long-horizon prediction, we penalize latent collapse and minimize prediction error over autonomous rollouts. We evaluate ViHaD on a pendulum spanning libration and rotation, a mass–spring oscillator, and a planar two-body system. Across these systems, ViHaD substantially improves visual prediction accuracy compared with PixelHNN.