acceptodds
Under review as a conference paper at ICLR 2027

Where Structure Enters a World Model: From Predictive Fit to Physical Rollout

Abstract

Predictive world models learn to forecast observations, but accurate short-term predictions do not necessarily produce physically coherent behavior over longer horizons. We investigate how incorporating physical structure can strengthen world models while retaining their underlying predictive objectives. Across token-based models of planetary motion and JEPA-style visual models of mechanical systems, we study conserved-quantity supervision, symmetry-aware representations, temporal consistency, and learned structured dynamics. In planetary motion, angular-momentum supervision improves autonomous next-token forecasting, while the benefits of temporal regularization depend on the accompanying physical objective and representation. In visual models, generating rollouts through supervised energy-based dynamics heads produces more faithful trajectories and phase-space evolution. Together, these findings highlight the importance of how physical structure participates in prediction: improvements in an auxiliary physical readout or a local acceleration estimate do not automatically translate into more reliable long-horizon behavior. Evaluations across initial conditions and physical parameter shifts further reveal where these benefits persist and where generalization remains limited. Our study supports physical structure as a practical means of improving predictive world models and provides a unified framework for understanding its effects on representations, learned transitions, and autonomous rollouts. The central lesson is that effective physical inductive biases should help models both represent physical regularities and use them to generate future states.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Models, Inductive Bias, V-JEPA, Equivariance, Temporal Difference Learning, Hamiltonian Neural Networks, Lagrangian Neural Networks, Planetary Dynamics, Physics Informed ML, Autonomous Rollout, Self Supervised Learning, Predictive World Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_2ouqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how physical structure should be built into predictive world models so that long-horizon rollouts stay physically coherent. It compares four ways of adding structures across two settings: JEPA-style visual models of mechanical systems (pendulum, damped pendulum, spring, and double pendulum) and next-token transformers trained on planetary orbits. The four are conserved-quantity supervision, relative and pairwise geometry, temporal-difference consistency, and a supervised energy-based mechanics head.

In the visual setting, a learned mechanics head that advances the decoded state with RK4 improves rollouts and phase portraits over a latent-only baseline. In the planetary setting, angular-momentum supervision and pairwise geometry reduce forecasting error. The effect of temporal consistency depends on the representation. The authors conclude that physical structures should be attached to the computation that generates future states.

Critical strengths

The paper asks a useful question: does it matter where physical structure enters a world model? It also evaluates the answer on long-horizon autonomous rollouts rather than one-step prediction. The comparison is careful and honest. Seeds and encodings are matched, and negative results are reported, such as TDV hurting the pairwise model, force supervision hurting NTP, and the structured head failing under parameter shifts. That makes the paper a useful map of which forms of structure help and which do not, even where its conclusions are limited.

Critical weaknesses

This paper does not engage with the large literature on physics-informed and structured neural networks (PINNs, Hamiltonian and Lagrangian networks, symplectic ODE models, Noether Networks, graph-network simulators). Building physical constraints such as energy and momentum conservation into a model is not a novel idea. The paper cites a few of these works in passing but does not discuss them in depth or explain what is new here. Several of them, such as HGN and Lagrangian dynamics learned from images, are close to Physics-JEPA and should be compared against directly.

To what extent are the built-in constraints "spoon-feeding" the correct prediction to the model? The mechanics head is given the true kinetic energy and damping form and is trained on simulator accelerations. Its latent forecasts are identical to the baseline's, so the visual model learns nothing extra. The planetary auxiliary labels come from velocities and masses that the model never sees as input. The constrained models might therefore simply perform better because they have access to richer inputs, in which case the observed improvement is nearly trivial. Several metrics also directly measure the imposed quantity (energy error, angular-momentum drift). The paper should evaluate improvement on metrics that are in some sense orthogonal to the physical constraints. Where it does so, the results are weak: frozen acceleration probes are near chance, and the gains largely vanish under parameter shifts.

Is data-driven learning needed at all when there is enough domain expertise to support such extensive constraints? Physics-informed methods are most useful when the constraint supplies the form of the dynamics and the data supply the unknowns. PINNs, for example, are most convincing in inverse problems, where parameters are estimated from observations, rather than in forward problems, where the network just acts as a solver. This paper sits in the forward regime: everything needed for next-state prediction is supplied, and nothing has to be inferred. The paper would be much stronger if the model had to infer unknown physical parameters from data and was evaluated on recovering them.

Questions for authors

How does this work differ from prior physics-informed and structured-dynamics models such as PINNs, HNN/LNN, HGN, and Lagrangian dynamics learned from images? Can the authors compare Physics-JEPA directly against the closest of these?

Do the improvements survive when the baseline gets the same information? For example, an unconstrained dynamics model could be trained with the same state and acceleration labels, and a planetary model could get the velocity and mass information as inputs. Those would be more apple-to-apple comparisons.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2ouq

We thank Reviewer 2ouq for the thoughtful questions on prior work, supervision and parameter inference.

1. Do gains survive equal information? (Weakness 2, Question 2)

Kinetic and dissipative forms are specified. Data determine states, potential and damping. Mechanics losses update the head, not the visual branch. Both heads share frozen observation features.

Energy/free heads share targets, decoded initial states, budget, shared-layer initialization and RK4. Simple-pendulum angle RMSE is 0.2147/0.2477 rad with acceleration targets and 0.2149/0.2574 with next-state targets (five seeds, 5/5 wins each). Decoded-transition error is 0.5117/0.6253 without simulator-state or acceleration targets for either head. The decoder remains state-supervised.

Takeaway: Decoded-transition simple-pendulum error falls 18.2% with equal head supervision and RK4, answering your equal-information concern.

The planetary control matches velocity/mass inputs, modules, initialization, data and budget. Only the angular-momentum (AM) weight changes. Velocity inputs stop after 32 frames, so autonomous prediction uses the same observed information.

Equal-input objectiveNominal MSE mean +/- seed SDMedian
AM weight 00.2301 +/- 0.31550.1047
AM weight 0.10.1377 +/- 0.15150.0749

Five seeds, AU squared. AM lowers mean MSE by 40.2% and median by 28.4%, with 3/5 wins. All seeds remain, including the baseline outlier influencing the mean. Shifts are mixed and conservation errors do not improve.

Takeaway: AM's nominal motion-error benefit survives equal velocity/mass inputs.

Trajectory error measures motion independently of the imposed quantity. Frozen probes remain weak and five-seed Plain/AM force adaptation is mixed. The probe reference of one is zero-vector regression, not classification chance. Fixed heads retain the nominal law under hidden-parameter shifts, motivating the inverse test below. Forecasting and probe performance are assessed separately.

2. How does this relate to prior structured dynamics? (Weakness 1, Question 1)

HNN/LNN mechanics, cited in Background and Related Work, underpin this study. Our controls separate energy parameterization from second-order dynamics, geometry from capacity, and trajectory accuracy from conservation. Reviewer mtJr also highlighted the value of evaluating motion and conservation together. Matched controls test which choices explain the gains.

PINNs constrain equation residuals. HNN/LNN and symplectic models structure dynamics. HGN learns Hamiltonian image-reconstruction rollouts. Lagrangian dynamics from images jointly learns coordinates and dynamics. Noether Networks learns a conservation loss. Our planetary auxiliary uses a specified target. We will expand these comparisons, including graph simulators, to clarify our tests of mechanics, training-only supervision and token geometry.

We ran the author Lag-caVAE architecture/image objective on our pendulum data (five seeds, 25 systems each, image-validation stopping):

PipelineFinite 400-step forecastsFull-cohort angle RMSE
Lag-caVAE45/125Undefined
Matched energy / free heads125/125 each0.2147 / 0.2477

Nonfinite forecasts are retained. All methods receive two images. Lag-caVAE uses image/variational training, inferred states and native Euler. Ours use a state-supervised decoder and RK4. Its angle calibration uses training labels after model training.

Takeaway: The requested image-pipeline comparison is complete. The matched test in section 1 isolates dynamics structure.

3. Can the model infer an unknown coefficient? (Weakness 3)

We estimate identifiable g/L through a bounded acceleration multiplier from 32 observed frames, with known nominal calibration and frozen neural weights. Neither the test coefficient nor future observations enter fitting.

Decoded-state dynamicsRelative coefficient errorFixed / adapted angle RMSE
Energy23.63%0.7475 / 0.6223
Free acceleration23.58%0.7509 / 0.6237
Classical pendulum23.63%0.7476 / 0.6223

Five observation/head seeds and five coefficient multipliers, with radians over 400 future steps. Energy-head error falls 16.8% with 5/5 wins, with similar gains for the alternatives. True-state coefficient errors are 0.0300%, 0.3884% and 0.00073%, respectively. The fitted multiplier can absorb model/observation error. Only g/L is identifiable, not gravity and length separately. Stationary controls are reported outside coefficient accuracy.

Takeaway: Observed motion supports g/L recovery and improves forecasts with frozen neural weights, directly testing the inverse inference you requested.

I thank the authors for the response. I shall maintain my current rating.

Reviewer_ZS8eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the role of physical structure in long-horizon forecasting with predictive world models. It tests physical supervision, geometry, and temporal consistency in a transformer for planetary forecasting and the comparison results show how these different designs affect predictive accuracy, physical consistency, and generalization to changed physical conditions.

Critical strengths

The paper compares different ways of adding physical structure and examines their effects on model behavior. This offers guidance for designing world models that generate physically consistent trajectories. And the evaluations show that predictive accuracy and physical consistency respond differently to the design choices, which could helps identify which forms of physical structure support each objective and where trade-offs arise.

Critical weaknesses

Some comparisons do not fully isolate the contribution of physical structure and it is unclear how much of the improvement comes from the physical structure itself. For example, Physics-JEPA also adds acceleration supervision and changes the rollout procedure, without a baseline using the same supervision and procedure but no physical constraints. Further clarifications are needed.

The experiments focus on low-dimensional simulated systems and simplified planetary motion and it is unclear whether the findings extend to more complex dynamics and a wider range of physical conditions.

Questions for authors

Does Physics-JEPA outperform an unconstrained dynamics model trained with the same acceleration labels and evaluated with the same integration procedure? How do you explain the reduced benefits of physical structure when physical parameters or the number of planets change (Tables 13, 15, and 18)? Does the pairwise adapter outperform a similarly sized adapter without geometric constraints?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZS8e

We thank Reviewer ZS8e for highlighting the value of comparing physical designs and the need for clearer attribution. We address these concerns with matched controls and targeted shift tests.

1. Do gains survive the same supervision and integration? (Weakness 1, Question 1)

Energy/free heads share labels, examples, decoded initial states, shared-layer initialization, budget and RK4. Both keep dq/dt = v. Capacity is equal for simple/damped pendulum and spring, and within 0.19% for double pendulum. This tests energy parameterization with labels and integration held fixed.

System and targetsAngle RMSE, energy / freePaired angle wins
Simple pendulum, acceleration0.2147 / 0.24775/5
Simple pendulum, next state0.2149 / 0.25745/5
Double pendulum, next state, upper link0.17769 / 0.184295/5
Double pendulum, next state, lower link0.24621 / 0.255465/5

Five-seed means in radians over 400-step forecasts. Decoded-transition simple-pendulum training also lowers mean angle error by 18.2%, with 5/5 wins. The heads receive no simulator labels. The decoder remains state-supervised.

Takeaway: Simple/double-pendulum trajectory gains remain with matched supervision and RK4, answering your matched-baseline question.

Spring has 9.93/6.44-fold lower nominal drift under acceleration/next-state training with less than 0.03% difference in position error. Damped pendulum has nearly equal trajectory error and lower physical-balance error. Double-pendulum next-state energy-change error favors free dynamics despite the angle gains (common C1).

Takeaway: Conservation and motion are measured separately, so improvement in one is not used as evidence of improvement in the other.

2. Does geometry help beyond added parameters? (Question 3)

The geometric/generic adapters add 1,189,635/1,191,171 parameters with the same corrected relative encoding and budget. Across twelve rotations and five seeds:

PredictorMean position MSE, AU squared
Relative NTP0.1736
Generic relational adapter0.2413
Geometric adapter0.1128

Geometry lowers mean error by 53.3% against the generic control, with four paired wins. Its nominal mean is also lower, but with only two nominal paired wins. Our response to Reviewer mtJr gives the embedding/logit ablations.

Takeaway: Geometry's more consistent benefit is rotation robustness, and it survives the capacity-matched control you requested.

3. What happens under changed laws and additional bodies? (Weakness 2, Question 2)

Fixed nominal dynamics receive no new value when hidden pendulum length or spring coefficients change. Double-pendulum hidden-length errors are nearly tied. Spring's nominal consistency advantage does not persist comparably under hidden mass/stiffness changes. Decoded simple-pendulum hidden-length error favors free dynamics (0.6832 versus 0.9712 rad).

Takeaway: Conserving energy alone does not let a fixed nominal dynamics model infer a changed hidden coefficient.

The inverse test estimates identifiable g/L from 32 observed frames, with frozen neural weights and no future observations in fitting. Across five observation/head seeds, energy-head forecast RMSE falls from 0.7475 to 0.6223 rad, improving in every seed. Free dynamics and a classical pendulum reach 0.6237 and 0.6223.

Takeaway: Fitting the hidden coefficient from observed context improves forecasts for energy, free and classical dynamics.

The unchanged-orbit test confirms cross-body interference in the joint predictor. A new factorized inference variant applies each frozen model to independent star-planet pairs. Existing predictions are bitwise unchanged by addition or reordering. On a separate 16-system, three-seed cohort with one added planet, full-planet MSE is 0.5609 for geometry, 1.0929 for the generic adapter and 6.0806 for persistence. This uses independent central-force dynamics. The original joint failure remains reported (response to Reviewer mtJr, section 2).

Takeaway: Factorized inference removes added-body interference for this independent-orbit benchmark.

The tests cover conservative, dissipative and coupled mechanics, orbital geometry, hidden coefficients and body-count changes. They do not establish results for arbitrary interacting or high-dimensional dynamics. We propose adding the complete shifts and seeds to the existing Results and appendix.

Takeaway: The expanded tests address your breadth concern through coupled dynamics, hidden coefficients and added independent bodies within these evaluated systems.

Thank you for adding the experiments. The matched controls address my main concern. I’m keeping my original rating of 3.

Reviewer_rWzcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper performs a set of empirical comparisons to understand the role of physical structure in world modeling for better rollouts. They test visual models (e.g., I-JEPA) on physical systems like pendulum systes and spring-mass systems. They also test next token prediction on planetary models.

Critical strengths

S1: The fundamental question is interesting and the application to modern latent prediction models like JEPAs is timely.

Critical weaknesses

W1: To me, this paper is more of an empirical comparison of existing methods than an introduction of a new method, since all of the tested methods were introduced in previous work, but the rigor and completeness of the experiments is not sufficient. Physics-JEPA gets supervision signal from acceleration, but the baseline doesn't: without an unconstrained dynamics head trained on the same labels, you cannot attribute the gain to energy-based structure. The adapter in the planetary motion prediction task adds 1.9M parameters on top of the baseline, which confounds the results there.

W2: Some important related works are missing. [1] studies latent dynamics models on very similar set of systems as Part 1 of this work. [2] finds that HNN gains mainly come from the acceleration dynamics and not energy conservation, which is another reason to specifically disentangle these factors in this analysis.

[1] Botev et al. (2021), Which priors matter? Benchmarking models for learning latent dynamics (NeurIPS Datasets & Benchmarks) [2] Gruver et al. (2022), Deconstructing the inductive biases of Hamiltonian neural networks (ICLR)

Questions for authors

Q1: is there a way to make the physics-jepa vs jepa comparison without using the extra information from acceleration?

Q2: Plain and Relative use different encodings, and Absolute TDV uses an older encoding. The adapter adds 1.19M parameters with no capacity-matched control. Would it be possible to match as many of these factors as possible?

Q3: There is no hyperparameter sweep. All auxiliary and TDV weights are fixed at 0.1. So the finding that TDV hurts Pairwise can just be an effect of suboptimal weight.

Q4: Most of the results have one seed only, and a few have 3. Can we have at least 5 seeds for each result?

Q5: lines 1009-1010: text here says the Figure 3 Earth case was chosen “after examining the results" -- what does this entail? does this mean the other results do not support the claims? similar language is used in 977-978.

Q6: Botev et al., 2021 find that physical priors do not significantly beat standard techniques - can the authors look into their work and try to resolve the tension with the conclusions drawn in these experiments?

2 comments
AuthorsOpenReview ↗

Response to Reviewer rWzc

We thank Reviewer rWzc for the questions on experimental controls and related work. We address both central concerns with matched comparisons, five-seed replications and the suggested literature.

1. Do gains survive matched supervision and capacity? (Weakness 1, Questions 1, 2, 4)

Energy/free heads share examples, labels, decoded initial states, shared-layer initialization, budget and RK4. Both keep dq/dt = v, and the energy head receives no extra energy penalty. Holding second-order dynamics and integration fixed tests energy parameterization itself (common C1).

Simple-pendulum head targetsAngle RMSE, energy / freePaired accuracy wins
Acceleration0.2147 / 0.24775/5
Next state0.2149 / 0.25745/5
Decoded transitions0.5117 / 0.62535/5

Angles are radians, averaged over five seeds. Next-state training removes acceleration labels. Decoded-transition heads receive neither simulator states nor accelerations. The decoder remains state-supervised.

Takeaway: Simple-pendulum angle error falls 18.2% without simulator labels for dynamics-head training, answering your supervision question.

Both double-pendulum links improve in all five pairs, by about 1% with acceleration targets and 3.6% with next-state targets. Spring/damped pendulum show consistency gains with nearly equal trajectory errors (C1). Capacity is equal for simple/damped pendulum and spring, and within 0.19% for double pendulum.

Takeaway: Matched controls show trajectory gains for simple/double pendulum and mainly consistency gains for spring/damped pendulum.

The geometric and generic adapters have 1,189,635 and 1,191,171 parameters. With corrected relative encoding, examples and budget matched, geometry lowers twelve-rotation mean MSE from 0.2413 to 0.1128 AU squared (53.3%, 4/5 wins). Historical Absolute TDV uses the older encoding and will be labeled separately.

Takeaway: Geometry improves rotation robustness with capacity and encoding matched, answering the adapter-size concern.

Main visual/planetary controls, branch ablations, decoded heads, equal-input tests, recovery and the named baseline use five seeds. Intervention/correction diagnostics retain three-seed protocols. All valid runs remain included. Individual seeds and sample SDs will accompany the tables.

Takeaway: The primary comparisons now span five independent training seeds, retaining every valid run as requested.

2. How do these results relate to the suggested papers? (Weakness 2, Question 6)

We will discuss Botev et al., Which priors matter? and Gruver et al., Deconstructing the inductive biases of Hamiltonian neural networks. Botev finds that mechanics-based priors do not consistently outperform strong alternatives. Gruver identifies second-order dynamics and coordinates as explanations for apparent HNN gains. Our shared second-order/RK4 control directly tests that distinction.

Energy parameterization improves simple-pendulum accuracy under all three target regimes and gives smaller coupled-pendulum gains. Spring/damped pendulum separate consistency from accuracy, and double-pendulum next-state energy error favors free dynamics.

Takeaway: Energy structure helps specific systems and metrics under matched controls, consistent with Botev and directly testing Gruver's second-order explanation.

3. Does selecting the TDV weight improve results? (Question 3)

We swept 0, 0.01, 0.03 and 0.1. Validation token loss selected 0.03 for Relative and 0.1 for Pairwise at seed 1337. Four independent confirmation seeds gave:

RepresentationSelected weightNominal MSE, NTP / TDVTDV wins
Relative0.030.1253 / 0.23831/4
Pairwise0.10.0800 / 0.14590/4

At fixed weight 0.1, Pairwise TDV lowers five-seed 45-degree mean error by 44.0% while increasing nominal error in every pair. The sweep varies TDV. The physical-auxiliary weight remains 0.1.

Takeaway: Weight selection did not remove the nominal TDV penalty. The fixed-weight Pairwise model benefits under rotation.

4. Does the Earth example represent the full cohort? (Question 5)

Earth was selected after inspecting results as an illustration. The aggregate covers all 16 planet/cadence cases. Relative+TDV beats Plain NTP in 13/16 nominal, 12/16 eccentricity-shift and 12/16 stellar-mass-shift cases, with mean cosines 0.544/0.337, 0.508/0.308 and 0.413/0.271. These are correlated cases from one pretraining seed. A separate five-seed Plain/AM force-adaptation test is mixed and remains distinct from forecasting.

Takeaway: The force-direction evidence includes all 16 cases, beyond the post hoc Earth illustration, with its one-pretraining-seed scope explicit.

I thank the authors for the additional experiments and responses to my concerns. The matched supervision/capacity experiment helps alleviate some of my concern about experimental robustness, but my concerns on novelty, relation of this work to missing related works, and poor presentation/difficulty of understanding remain. Therefore I maintain my rating.

Reviewer_mtJrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors examine whether physical structure improves long-term forecasts in JEPA-based visual models and token-based models of planetary motion. The visual experiments use supervised mechanical dynamics to advance an image-derived initial state. The planetary experiments retain next-token prediction and add physical supervision, geometric adapters, or temporal losses. Learned mechanics improves visual trajectories, while angular-momentum supervision and pairwise geometry help planetary forecasting. Temporal losses give mixed results, and the gains do not consistently hold when physical parameters or the number of bodies change.

Critical strengths

1.The mechanics head has a sensible physical form. Conservative forces come from a learned potential, and the positive damping coefficient ensures that damping cannot inject energy into the system. This gives the learned transition an explicit energy constraint during rollout.

2.The pairwise adapter offers a practical way to incorporate geometry into a token predictor. Its continuous displacement proposal rotates and reflects with the input, and it guides both the embeddings and output logits while retaining the next-token prediction objective.

3.The angular-momentum result is encouraging. Forecasts improve in all three training runs, and the auxiliary head is discarded at inference, so the improvement carries over to the token predictor’s own rollouts. I found the phase portraits helpful for judging the rollouts. Reporting trajectory errors alongside conservation measures also makes it easier to see when a model gets the physical scores right but the motion wrong.​

Critical weaknesses

1.The novelty over prior physics-based dynamics models is unclear. Section 6.1 reports identical latent predictions for Physics-JEPA and the baseline; the rollout gains come from an acceleration-supervised mechanics head, while the physical loss leaves the visual branch unchanged. What does this add to existing approaches that infer initial states from images and integrate learned energy-based dynamics? The claim about where physical structure should enter the model also needs stronger support. The visual and planetary experiments differ in architecture, input modality, and supervision, so they do not isolate the effect of placement.

2.It is hard to tell how much the physical structure contributes to the gains. Physics-JEPA uses additional acceleration labels and a different rollout procedure. How well would a plain dynamics MLP of comparable size perform with the same labels, initial-state estimates, and RK4 solver? Pairwise NTP also adds parameters and biases the output logits, which could account for some of the improvement. I would like to see a similarly sized non-equivariant adapter and separate ablations of the embedding and logit branches to clarify the contribution of the geometric design.

3.I found Figure 1 too schematic to explain the method well. It should show more clearly which branches generate the rollouts and how the physical losses update the model. The writing needs a clearer thread. The visual and planetary parts often read like separate studies, and the transitions between trajectory prediction, force adaptation, and parameter transfer are abrupt. I would shorten the repeated general claims about physical structure and use that space to explain how the experiments fit together, especially where they give different results.

4.The four-planet result needs an explanation. Pairwise NTP has an MSE of about 50222 in Table 13, versus 104 for Relative NTP and 97 for persistence. This is a striking failure given that the planets follow independent Kepler orbits. The test also changes the mass distribution, making it hard to tell what went wrong. Could the authors keep an existing orbit unchanged, add one non-interacting planet, and check whether the original planet’s forecast deteriorates? This would help reveal whether the adapter introduces dependencies between bodies that should not affect each other.

Questions for authors

1.The novelty over prior physics-based dynamics models is unclear. Section 6.1 reports identical latent predictions for Physics-JEPA and the baseline; the rollout gains come from an acceleration-supervised mechanics head, while the physical loss leaves the visual branch unchanged. What does this add to existing approaches that infer initial states from images and integrate learned energy-based dynamics? The claim about where physical structure should enter the model also needs stronger support. The visual and planetary experiments differ in architecture, input modality, and supervision, so they do not isolate the effect of placement. 2.It is hard to tell how much the physical structure contributes to the gains. Physics-JEPA uses additional acceleration labels and a different rollout procedure. How well would a plain dynamics MLP of comparable size perform with the same labels, initial-state estimates, and RK4 solver? Pairwise NTP also adds parameters and biases the output logits, which could account for some of the improvement. I would like to see a similarly sized non-equivariant adapter and separate ablations of the embedding and logit branches to clarify the contribution of the geometric design. 3.I found Figure 1 too schematic to explain the method well. It should show more clearly which branches generate the rollouts and how the physical losses update the model. The writing needs a clearer thread. The visual and planetary parts often read like separate studies, and the transitions between trajectory prediction, force adaptation, and parameter transfer are abrupt. I would shorten the repeated general claims about physical structure and use that space to explain how the experiments fit together, especially where they give different results. 4.The four-planet result needs an explanation. Pairwise NTP has an MSE of about 50222 in Table 13, versus 104 for Relative NTP and 97 for persistence. This is a striking failure given that the planets follow independent Kepler orbits. The test also changes the mass distribution, making it hard to tell what went wrong. Could the authors keep an existing orbit unchanged, add one non-interacting planet, and check whether the original planet’s forecast deteriorates? This would help reveal whether the adapter introduces dependencies between bodies that should not affect each other.

3 comments
AuthorsOpenReview ↗

Response to Reviewer mtJr

We thank Reviewer mtJr for the constructive matched-control and unchanged-orbit suggestions. Both experiments are complete. We address the main concerns first and retain the original weakness and question numbers below.

1. Do gains survive matched controls? (Weakness 2, Question 2)

The free-acceleration baseline shares labels, decoded initial states, shared-layer initialization, budget and RK4 with the energy head. Both keep dq/dt = v. Simple-pendulum angle RMSE is 0.2147/0.2477 rad with acceleration targets, 0.2149/0.2574 with next-state targets, and 0.5117/0.6253 with decoded transitions (energy/free, five seeds). All five pairs favor the energy head in every regime. Decoded head training omits simulator-state and acceleration targets. The decoder remains state-supervised.

Takeaway: Simple-pendulum gains survive matched labels and RK4, including decoded-transition training, addressing your supervision concern (common comment, C1).

The generic and geometric adapters have 1,191,171 and 1,189,635 parameters. With relative encoding and budget matched, geometry reduces twelve-rotation mean MSE by 53.3% (0.1128 versus 0.2413 AU squared, 4/5 wins).

Takeaway: Geometry improves rotation robustness with matched adapter capacity, addressing the parameter-count confound.

We also trained the requested branch ablations with five seeds:

AdapterNominal MSE45-degree MSE
Generic, both branches0.2348 +/- 0.24160.2284 +/- 0.2339
Geometric, both branches0.0723 +/- 0.02600.1175 +/- 0.0530
Geometric, embeddings only0.1798 +/- 0.13270.0718 +/- 0.0380
Geometric, logits only0.1589 +/- 0.12690.1114 +/- 0.0967

Entries are mean +/- sample SD, AU squared. Both branches give the lowest nominal mean. Embeddings alone give the lowest 45-degree mean. This single-angle ranking differs from the broader twelve-rotation comparison.

Takeaway: The embedding-only result shows that rotation gains do not come solely from logit guidance.

2. Does adding a planet change the original forecast? (Weakness 4, Question 4)

We kept the original star, planet, history, cadence and forecast times fixed, then added a noninteracting planet at 90 degrees. Across three seeds and 32 source systems, original-planet Pairwise MSE rose from 0.621 to 12,794.888 AU squared. Relative NTP reached 5.040, close to persistence at 5.075. Message normalization reduced extremes without restoring accurate continuation.

Takeaway: The original forecast degrades despite unchanged target dynamics, confirming the cross-body interference tested by your intervention.

We then tested factorized inference, applying each frozen predictor separately to independent star-planet pairs. Existing forecasts remain bitwise unchanged under addition or reordering. On a separate 16-system confirmation cohort, three-seed full-planet MSE with one added planet is 0.5609 for geometry, 1.0929 for the generic adapter and 1.3095 for Relative NTP, against persistence at 6.0806. We retain the original joint failure and report this as a new inference variant.

Takeaway: For independent central-force systems, factorized inference makes existing forecasts invariant to added or reordered planets.

3. What is learned, and what is the contribution? (Weakness 1, Question 1)

The mechanics loss updates the transition head. Observation weights and latent forecasts remain unchanged. This tests energy parameterization on shared predictive features. Planetary controls separately test geometry and training-only physical supervision. Together, they provide controlled evidence about these choices, including when consistency and trajectory accuracy disagree.

Our response to Reviewer 2ouq, section 2, compares prior image-based mechanics and reports the direct Lag-caVAE experiment. We will clarify this relationship in Related Work. Attribution rests on within-family controls because the visual and planetary architectures do not isolate a causal placement effect.

Takeaway: Building on HNN/LNN mechanics, the contribution is a controlled comparison of physical choices in learned forecasting.

4. How will Figure 1 and the narrative change? (Weakness 3, Question 3)

We propose focused Figure 1 edits showing which branches generate rollouts, the gradient paths, the training-only auxiliary head and prediction-time adapter. Within the existing sections, the matched protocol will replace repeated general statements. Results will follow one sequence: autonomous forecasting, physical consistency, then changed physical conditions. Force adaptation will be identified as a separate downstream task on observed positions.

Takeaway: These edits address your presentation concern by showing which components generate forecasts and receive physical-loss gradients.

The responses provide additional experimental controls that address several of my concerns. The matched comparisons between energy-based and free-acceleration heads, using the same supervision, initialization, capacity, and RK4 integration, help distinguish the contribution of physical structure from supervision and numerical integration. The five-seed results support this distinction, although the improvements in trajectory accuracy are relatively modest or task-dependent for some systems. The capacity-matched adapter comparison and embedding/logit ablations also provide stronger evidence for the contribution of geometric structure. These additions substantially address my concerns about experimental fairness and attribution.

The unchanged-orbit experiment confirms that adding a non-interacting planet can substantially degrade the original forecast. Factorized inference avoids this interference for independent systems, but the limitation of joint multi-body prediction remains. Regarding novelty, the visual approach still relies on separately supervised mechanics, and the authors acknowledge that the two model families do not isolate the effect of structural placement across architectures. Overall, the additional experiments strengthen the empirical evidence, but my concerns about novelty and multi-body generalization remain. I therefore maintain my rating.

AuthorsOpenReview ↗

Clarification of the Contribution and Independent-Body Result

We thank mtJr for recognizing that the additional controls substantially address experimental fairness and attribution. We clarify what these results establish about the contribution and the distinct generalization requirements.

1. What physical structure adds to visual prediction. Both dynamics heads are fitted separately to transitions from the same state-supervised observation decoder. Neither receives simulator-state or acceleration targets during dynamics fitting, and both use the same decoded initial state and RK4 integration. Across five seeds, energy-based dynamics reduce simple-pendulum angle RMSE from 0.6253 to 0.5117 radians, an 18.2% reduction with all five pairs improving. Separate fitting and access to supervision are shared, so they do not explain the structured head's advantage. This isolates a benefit of energy parameterization in prediction from decoded observations without additional dynamics labels for the structured head.

Our two model families provide complementary demonstrations in visual and token-based prediction. Matched comparisons within each family isolate the structural contribution while controlling for supervision, capacity, and prediction procedure. These results support a structural contribution beyond additional supervision or model capacity.

2. Interacting dynamics and distinct generalization tests. Our evaluation includes finite-mass two-body gravitational motion, where both bodies move about their shared center of mass, and the double pendulum, whose coordinates have coupled dynamics. In the five-seed matched double-pendulum comparisons, the energy-based head improves both angular coordinates in every seed under acceleration and next-state supervision.

Across twelve rotation angles and five seeds, geometry reduces mean position MSE from 0.2413 to 0.1128 AU squared relative to the capacity-matched generic adapter, a 53.3% reduction with four of five paired wins. These results establish benefits for coupled-system prediction and geometric robustness.

The unchanged-orbit intervention complements these tests by isolating context-induced forecast interference: target motion is held fixed while system membership changes. For independent central-force systems, our separate factorized construction preserves existing forecasts exactly under body addition and reordering. On the 16-system cohort:

Added planetsGeometricGenericGeometric seed wins
One, 90 degrees0.56091.09292/3
One, 180 degrees0.38701.15863/3
Two, 90 and 180 degrees0.48881.07542/3

Entries are mean full-planet position MSE in AU squared across three seeds. Persistence MSE is 6.0806 in every row. These additions are rotated copies of the original orbit. Factorization therefore combines useful forecast accuracy with exact independence in this setting, without retraining. This demonstrates a practical way to preserve independent forecasts while retaining the geometric model's predictive advantage.

3. Relation to prior work. Gruver et al. (2022) identify coordinate choice and acceleration modeling as important explanations for apparent Hamiltonian-model advantages. Our comparisons hold coordinates, second-order dynamics, and RK4 integration fixed to test the additional contribution of energy parameterization. Botev et al. (2021) show that physical priors do not consistently improve latent prediction. Our matched decoded-transition and adapter experiments identify structural benefits in visual and token-based forecasting while evaluating trajectory accuracy alongside physical consistency.

Geometric symmetry and size generalization are also distinct in the literature. Satorras et al. (2021) formulate equivariance to rotations, translations, reflections and permutations. Adding a body changes system size, rather than applying one of those transformations. Yehudai et al. (2021) show that changes in local graph structure can obstruct size generalization. Our intervention separates this requirement from geometric robustness by varying membership while holding target physics fixed.

Together, the visual and token-based experiments show benefits of physical structure in complementary prediction settings. The evidence covers coupled-system prediction and rotational robustness, alongside a controlled body-addition diagnostic and a training-free construction for independent systems. We respectfully ask that this breadth of methodological and empirical evidence be reflected in the assessment of the contribution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Shared experimental controls and results

We sincerely thank the reviewers for their thoughtful and constructive feedback. We have clarified our contributions and added matched controls and five-seed replications to test the concerns about supervision, model capacity and physical structure.

C1. Do gains survive matched labels and integration?

Energy and free-acceleration heads share examples, targets, decoded initial states, shared-layer initialization, budget and RK4 for 400-step autonomous forecasts. Both keep dq/dt = v. Capacity is equal for simple/damped pendulum and spring, and within 0.19% for double pendulum. The energy head receives no extra energy penalty.

Takeaway: These controls isolate energy parameterization with labels and integration fixed, addressing the matched-supervision concerns of Reviewers mtJr, rWzc, ZS8e and 2ouq.

Simple-pendulum head targetsAngle RMSE, energy / freePhysical energy-drift RMSE, energy / free
Simulator acceleration0.2147 / 0.24770.000150 / 0.005856
Simulator next state0.2149 / 0.25740.000145 / 0.006771
Decoded transitions0.5117 / 0.62530.07246 / 4.27361

Five-seed means, with angles in radians and energy in simulator units. Decoded-transition heads receive neither simulator-state nor acceleration targets. The decoder remains state-supervised.

Takeaway: Simple-pendulum motion improves in all five pairs under every objective, including without simulator labels for head training, answering the supervision questions from Reviewers rWzc, ZS8e and 2ouq.

Both double-pendulum links improve in all five pairs, by about 1% with acceleration targets and 3.6% with next-state targets. Next-state energy-change error favors free dynamics (0.08831 versus 0.13381). Spring has 9.93/6.44-fold lower nominal drift under acceleration/next-state training, with position errors differing by less than 0.03%. Damped-pendulum state-trained heads have nearly equal angle error and about 5.9-fold lower balance error for the energy head.

Takeaway: Energy heads improve simple/double-pendulum motion. Spring/damped gains mainly concern consistency, answering Reviewers ZS8e and 2ouq on conservation versus predictive accuracy.

C2. Does geometry help beyond added parameters?

With capacity, corrected relative encoding and budget matched, geometry lowers twelve-rotation mean MSE from 0.2413 to 0.1128 AU squared (53.3%, 4/5 paired wins). Our response to Reviewer mtJr gives the embedding/logit ablations.

Takeaway: The rotation gain survives the capacity-matched generic control, addressing the parameter-count concern raised by Reviewers mtJr, rWzc and ZS8e.

C3. Does the physical target help beyond adding an auxiliary task?

We compare angular-momentum (AM) supervision with plain next-token prediction (NTP) and a nonconserved radial auxiliary. The control matches the scalar head, privileged training states, budget and training-only scale calibration.

Position-input objectiveNominal MSE mean +/- seed SDMedian
Plain NTP0.8596 +/- 1.22700.1769
NTP + AM0.1366 +/- 0.05330.1261
NTP + radial auxiliary2.0150 +/- 4.03220.2454

Five seeds, AU squared. AM wins 5/5 pairs against Plain NTP and 3/5 against the radial control. Auxiliary heads are discarded at inference. The equal-input test in our response to Reviewer 2ouq also gives lower nominal mean/median MSE, with 3/5 wins.

Takeaway: AM improves nominal autonomous motion beyond its conservation metric and generic auxiliary supervision. The benefit survives equal velocity/mass inputs, addressing Reviewer 2ouq's concern.

C4. What do these comparisons establish?

Visual observation weights stay frozen to isolate learned transitions. Planetary controls isolate geometry and training-only supervision within their model family. The two families do not isolate a causal placement effect across architectures.

Takeaway: The controls separate energy structure from supervision, geometry from capacity, and motion accuracy from consistency, addressing the attribution concerns of Reviewers mtJr, rWzc and 2ouq.

All valid seeds and high-error runs are retained. Main controls use five training seeds. Added-body and separate observation-correction diagnostics use three, and TDV selection has four independent confirmation seeds.

Takeaway: Five independent training runs test repeatability in the primary comparisons, addressing Reviewer rWzc's seed-variation concern.

We propose focused methods/results additions, clearer gradient paths in Figure 1, and fuller related-work comparisons. Complete shifts and individual seeds belong in the appendix.

Takeaway: These focused edits address Reviewer mtJr's presentation concern by making each component's forecasting role explicit without restructuring the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.