- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how physical structure should be built into predictive world models so that long-horizon rollouts stay physically coherent. It compares four ways of adding structures across two settings: JEPA-style visual models of mechanical systems (pendulum, damped pendulum, spring, and double pendulum) and next-token transformers trained on planetary orbits. The four are conserved-quantity supervision, relative and pairwise geometry, temporal-difference consistency, and a supervised energy-based mechanics head.
In the visual setting, a learned mechanics head that advances the decoded state with RK4 improves rollouts and phase portraits over a latent-only baseline. In the planetary setting, angular-momentum supervision and pairwise geometry reduce forecasting error. The effect of temporal consistency depends on the representation. The authors conclude that physical structures should be attached to the computation that generates future states.
Critical strengths
The paper asks a useful question: does it matter where physical structure enters a world model? It also evaluates the answer on long-horizon autonomous rollouts rather than one-step prediction. The comparison is careful and honest. Seeds and encodings are matched, and negative results are reported, such as TDV hurting the pairwise model, force supervision hurting NTP, and the structured head failing under parameter shifts. That makes the paper a useful map of which forms of structure help and which do not, even where its conclusions are limited.
Critical weaknesses
This paper does not engage with the large literature on physics-informed and structured neural networks (PINNs, Hamiltonian and Lagrangian networks, symplectic ODE models, Noether Networks, graph-network simulators). Building physical constraints such as energy and momentum conservation into a model is not a novel idea. The paper cites a few of these works in passing but does not discuss them in depth or explain what is new here. Several of them, such as HGN and Lagrangian dynamics learned from images, are close to Physics-JEPA and should be compared against directly.
To what extent are the built-in constraints "spoon-feeding" the correct prediction to the model? The mechanics head is given the true kinetic energy and damping form and is trained on simulator accelerations. Its latent forecasts are identical to the baseline's, so the visual model learns nothing extra. The planetary auxiliary labels come from velocities and masses that the model never sees as input. The constrained models might therefore simply perform better because they have access to richer inputs, in which case the observed improvement is nearly trivial. Several metrics also directly measure the imposed quantity (energy error, angular-momentum drift). The paper should evaluate improvement on metrics that are in some sense orthogonal to the physical constraints. Where it does so, the results are weak: frozen acceleration probes are near chance, and the gains largely vanish under parameter shifts.
Is data-driven learning needed at all when there is enough domain expertise to support such extensive constraints? Physics-informed methods are most useful when the constraint supplies the form of the dynamics and the data supply the unknowns. PINNs, for example, are most convincing in inverse problems, where parameters are estimated from observations, rather than in forward problems, where the network just acts as a solver. This paper sits in the forward regime: everything needed for next-state prediction is supplied, and nothing has to be inferred. The paper would be much stronger if the model had to infer unknown physical parameters from data and was evaluated on recovering them.
Questions for authors
How does this work differ from prior physics-informed and structured-dynamics models such as PINNs, HNN/LNN, HGN, and Lagrangian dynamics learned from images? Can the authors compare Physics-JEPA directly against the closest of these?
Do the improvements survive when the baseline gets the same information? For example, an unconstrained dynamics model could be trained with the same state and acceleration labels, and a planetary model could get the velocity and mass information as inputs. Those would be more apple-to-apple comparisons.
2 comments
I thank the authors for the response. I shall maintain my current rating.
Response to Reviewer 2ouq
We thank Reviewer 2ouq for the thoughtful questions on prior work, supervision and parameter inference.
1. Do gains survive equal information? (Weakness 2, Question 2)
Kinetic and dissipative forms are specified. Data determine states, potential and damping. Mechanics losses update the head, not the visual branch. Both heads share frozen observation features.
Energy/free heads share targets, decoded initial states, budget, shared-layer initialization and RK4. Simple-pendulum angle RMSE is 0.2147/0.2477 rad with acceleration targets and 0.2149/0.2574 with next-state targets (five seeds, 5/5 wins each). Decoded-transition error is 0.5117/0.6253 without simulator-state or acceleration targets for either head. The decoder remains state-supervised.
Takeaway: Decoded-transition simple-pendulum error falls 18.2% with equal head supervision and RK4, answering your equal-information concern.
The planetary control matches velocity/mass inputs, modules, initialization, data and budget. Only the angular-momentum (AM) weight changes. Velocity inputs stop after 32 frames, so autonomous prediction uses the same observed information.
Five seeds, AU squared. AM lowers mean MSE by 40.2% and median by 28.4%, with 3/5 wins. All seeds remain, including the baseline outlier influencing the mean. Shifts are mixed and conservation errors do not improve.
Takeaway: AM's nominal motion-error benefit survives equal velocity/mass inputs.
Trajectory error measures motion independently of the imposed quantity. Frozen probes remain weak and five-seed Plain/AM force adaptation is mixed. The probe reference of one is zero-vector regression, not classification chance. Fixed heads retain the nominal law under hidden-parameter shifts, motivating the inverse test below. Forecasting and probe performance are assessed separately.
2. How does this relate to prior structured dynamics? (Weakness 1, Question 1)
HNN/LNN mechanics, cited in Background and Related Work, underpin this study. Our controls separate energy parameterization from second-order dynamics, geometry from capacity, and trajectory accuracy from conservation. Reviewer mtJr also highlighted the value of evaluating motion and conservation together. Matched controls test which choices explain the gains.
PINNs constrain equation residuals. HNN/LNN and symplectic models structure dynamics. HGN learns Hamiltonian image-reconstruction rollouts. Lagrangian dynamics from images jointly learns coordinates and dynamics. Noether Networks learns a conservation loss. Our planetary auxiliary uses a specified target. We will expand these comparisons, including graph simulators, to clarify our tests of mechanics, training-only supervision and token geometry.
We ran the author Lag-caVAE architecture/image objective on our pendulum data (five seeds, 25 systems each, image-validation stopping):
Nonfinite forecasts are retained. All methods receive two images. Lag-caVAE uses image/variational training, inferred states and native Euler. Ours use a state-supervised decoder and RK4. Its angle calibration uses training labels after model training.
Takeaway: The requested image-pipeline comparison is complete. The matched test in section 1 isolates dynamics structure.
3. Can the model infer an unknown coefficient? (Weakness 3)
We estimate identifiable g/L through a bounded acceleration multiplier from 32 observed frames, with known nominal calibration and frozen neural weights. Neither the test coefficient nor future observations enter fitting.
Five observation/head seeds and five coefficient multipliers, with radians over 400 future steps. Energy-head error falls 16.8% with 5/5 wins, with similar gains for the alternatives. True-state coefficient errors are 0.0300%, 0.3884% and 0.00073%, respectively. The fitted multiplier can absorb model/observation error. Only g/L is identifiable, not gravity and length separately. Stationary controls are reported outside coefficient accuracy.
Takeaway: Observed motion supports g/L recovery and improves forecasts with frozen neural weights, directly testing the inverse inference you requested.