- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a method for state space prediction - the main observation is that measurement noise/error can propagate through to impact representation and physical model learning. The key idea is to decouple these processes e.g. through weighted latent incorporation. The results and experiments show that there is improvement in physical prediction tasks, including very complex and noisy kinematic tasks like Human3.6M, JEPA is used in the latent embedder to learn measurement noise resilient embeddings. Future state prediction is used to train the state-space prediction. The main contribution is Sec 3.2 for measurement latent assimilation. I did like reading this paper, although I found that parts of it were overly dense and could have just been moved to the (extensive) appendix without loss of detail.
Critical strengths
- The results are solid - good prediction across low complexity (pendulum) and high complexity (human motion) tasks, showing outperformance of competing baselines.
- The overall method is quite elegant - whilst neural state space is nothing new, I did like the approach of using blended (assimilated) latents between the forward model and the actual observations to map against observed state. I think this is a neat approach and is effectively a form of curriculum learning from a good physical prior.
- I found the ablation useful: particularly that JEPA is not the main source of improvement - this is a good finding, as it shows that there is strength in the wider method, rather than just learning a good system representation.
Critical weaknesses
- With a "collection of tricks" paper, it is sometimes hard to disentangle where the actual novelty/delta lies. The authors have done a reasonable job i.e. Table 6 does show a very useful breakdown of where the improvement is coming from. Interestingly, good results are obtained already without the R_stay phase 2 training. And it seems as though the key innovation is L_{fut}. Simpler breakdown would help more here e.g. to show this improvement as a MWE e.g. on a pendulum problem. This could help to provide deeper insight into how the proposed method works. The proposed block that supports L_{fut} is simple: this is good news as it means that this is a very transferrable method.
- E7 shows some detail on computational cost, but a greater breakdown/analysis would be useful here.
Questions for authors
- Extension: Does the assimilation block work as a generic technique that can be applied to other prior art?
- Unbounded forecasting: how much does this system "drift" compared with other works when observations are completely removed? What does this say about how well the model has learnt the underlying dynamics of the system?
4 comments
Response to Reviewer V9VY (2/2)
(Part 2 of 2.)
W2. Computational complexity.
We thank the reviewer for requesting a more detailed computational analysis. The human-motion model contains 178.73M parameters during training and 93.56M at inference, excluding the 85.17M training-only EMA teacher. The measurement Transformer accounts for most inference parameters (85.17M), while the physics-structured modules contribute 4.68M and assimilation and decoding contribute 2.66M. The bouncing-ball and pendulum models are substantially smaller, with 0.30M and 0.64M inference-active parameters, respectively. The analytical physics core introduces no trainable parameters.
On a single NVIDIA A100 40 GB GPU, training and evaluation across Human3.6M, Fit3D, and SportsPose require approximately 14 hours, while training the synthetic models takes approximately 28 minutes for the ball and 16 minutes for the pendulum. Human-motion inference averages 34.8 ms per frame across different sequences, whereas batched amortized inference on the synthetic systems requires approximately 1.3 ms per ball trajectory and 0.4 ms per pendulum trajectory. We will include the breakdown in the revised Appendix E.7.
Q1. Transferability of the assimilation mechanism.
We appreciate the reviewer's question regarding whether the assimilation block can be applied to other predictive models. The gain-based update
requires a propagated prior and contemporaneous measurement features in a compatible latent space, rather than a particular ODE formulation. Consequently, the assimilation module could be trained with another fixed predictive prior after establishing a compatible latent interface.
Table 10 provides supporting evidence within DePhy: with the prior held fixed, the learned gain outperforms direct teacher forcing across the evaluated measurement-noise and availability settings. While the assimilation mechanism is generic in form, our empirical claims are limited to the priors and datasets evaluated in this work; we do not claim zero-shot transfer to other architectures.
Q2. Forecasting drift without observations.
We thank the reviewer for raising the question of drift during long-horizon, observation-free forecasting. Table 3 directly evaluates this setting: the DePhy w/o-DA configuration performs a predictive-prior rollout without observations beyond the context window, and we will make this explicit in the manuscript.
On Human3.6M-P1, the observation-free prior achieves lower error than PhysMoP from 160 ms onward, reaching 97.7 mm MPJPE at 1000 ms compared with 103.9 mm for PhysMoP and 106.9–112.1 mm for the kinematic baselines. The same holds for the averaged P2 protocol, where the prior reaches 52.4 mm at 1000 ms versus 57.1 mm for PhysMoP and 63.7 mm for siMLPe. Recursive assimilation using only model-generated, rather than ground-truth, measurement evidence further reduces the 1000 ms error to 90.8 mm on P1 and 48.9 mm on P2.
Table 8 (supp.) addresses the second part of the question, namely what long-horizon drift reveals about the learned dynamics. It varies only the transition structure under the same forecasting setting (recursive assimilation of model-generated evidence, with no ground-truth observations). A purely neural transition is accurate at short horizons but drifts the most as the horizon grows, reaching 125.4 mm at 1000 ms. Purely analytical propagation drifts less over long horizons (119.2 mm at 1000 ms, and lower error than the neural transition from 320 ms onward), but is too restrictive at short and medium horizons (12.4 versus 8.6 mm at 160 ms). Combining the analytical dynamics with a learned residual of limited capacity, as in DePhy (), gives the lowest error at every horizon, reaching 91.7 mm at 1000 ms, whereas giving the residual more capacity increases long-horizon error again (101.7 mm at and 123.1 mm with an unrestricted residual). This indicates that the analytical dynamics account for long-horizon stability, while the learned residual corrects model discrepancy at shorter horizons.
The synthetic benchmarks provide complementary evidence beyond the training horizon. On the bouncing-ball task, DePhy achieves 0.0317 MSE at , compared with 0.0512 for MoNODE. Together, these results indicate that the physics-structured prior retains useful dynamical information and exhibits reduced forecasting error within the evaluated horizons. We do not claim stability for rollouts beyond the horizons evaluated here.
Dear Reviewer,
Thank you again for the positive and constructive review. We have uploaded the revised manuscript, with all changes marked in blue, and have tightened the main text while keeping implementation details in the supplement (Supp. C.7 and G). Here is where each point is addressed:
- W1 (minimal working example of ): new Supp. F.1 (Table 14) ablates the Stage 1 training signal on the damped pendulum: alone, with the state-consistency term , alone, and future-measurement reconstruction. The main-text ablation in Sec. 4.3 summarizes it. The role of as an auxiliary Stage 2 term is stated in Sec. 3.6 and Supp. E.5.
- W2 (computational cost): Supp. E.7 now reports module-wise parameter counts (Table 11), encoder-size scaling with runtime (Table 12), module-wise inference runtime (Table 13; ODE integration accounts for 32.2 of 34.8 ms per frame), and training times for all systems.
- Q1 (transferability of assimilation): Sec. 3.5 and Supp. A.4 state the interface the update requires and the scope of our claims. Supp. E.4 (Table 10, Fig. 5) compares the learned gain with direct teacher forcing and with PhysMoP's learned fusion under noisy and sparse measurements.
- Q2 (drift without observations): Sec. 4.2 and Supp. C.10 define w/o DA as a rollout with no observations beyond the context window. Supp. E.2 (Table 8) analyzes how drift depends on the transition structure (neural-only, physics-only, and bounded-residual variants). Supp. E.8 states that we make no stability claim beyond the evaluated horizons.
We are grateful that you found the method elegant, the assimilated-latent approach neat and transferable, and the ablations informative. We hope the added pendulum example, compute breakdown, and drift analysis address your remaining points, and if so, we would be grateful if you would consider raising your score. We are happy to clarify anything further.
Thank you for your extremely detailed feedback - it helps to clarify where the gains come from in your work. The pendulum ablation is helpful as it is a simple toy example. And the information about computational complexity is useful Reading the other reviewer's comments as well, and taking into account the wider context of this work, I remain on the positive side, but I am content to remain on my current score. I hope that the review comments have helped guide your manuscript and boosted its clarity/coherence.
Response to Reviewer V9VY (1/2)
We thank the reviewer for their thoughtful and constructive assessment of our work, particularly for finding the overall method elegant and the ablation study informative. We appreciate the suggestions for clarifying the methodological contribution, computational cost, transferability of assimilation, and long-horizon forecasting behavior.
W1. Novelty and future prediction.
We appreciate the reviewer's observation that LFUT is a central mechanism in DePhy, and the suggestion to demonstrate its contribution through a minimal working example. The analytical dynamics fK, amortized physical factors ϕ^, and bounded residual fU are jointly supervised through future measurement-latent prediction, rather than measurement reconstruction. The predictive prior is subsequently frozen before assimilation training. The stay regularizer Rstay supports assimilation (50.2 to 47.3 mm MPJPE; Table 6) and acts as an auxiliary loss in the Stage 2 assimilation step.
Following the reviewer's suggestion, we conducted a minimal ablation on the damped pendulum, where ground-truth physical factors are available for evaluation. The predictive prior is trained using the future-latent objective of Eq. 7,
LFUT=Tf1∑k=1Tfzt+kp−zt+k⋆22,zt+kp=Πz(st+kp),
where z⋆ denotes the stop-gradient EMA target representation. We additionally examine a physical-state consistency regularizer,
Rs=Tf1∑k=1Tfst+kp−st+k22,st+k=Estate(zt+km),
and a reconstruction-supervised alternative,
Lrecfut=Tf1∑k=1TfDmeas(Πz(st+kp))−y~t+k22.
Unlike LFUT, the reconstruction objective directly supervises future measurements and permits reconstruction gradients to update the predictive prior. Table R6 compares these training objectives under otherwise identical settings.
Table R6. Pendulum ablation of the Stage 1 training signal. Decoded MSE measures the error of Dmeas(Πz(st+kp)) against clean observations; state MSE measures the error of the propagated physical state against the clean simulated state.
The results support three conclusions. First, future-latent prediction is substantially more effective than measurement reconstruction for training the predictive prior. Replacing LFUT with reconstruction increases forecast error by one to two orders of magnitude, reduces FoV R2 from 0.909 to 0.401, and increases parameter error from 1.7% to 3.4%. This is consistent with the human-motion ablation in Table 6, where allowing reconstruction gradients to update the prior increases acceleration error from 3.9 to 8.8 mm/s².
Second, Rs provides complementary physical-state supervision, but LFUT accounts for most of the improvement: relative to the reconstruction-trained prior, LFUT alone achieves approximately 98–99.8% of the forecast-error reduction obtained with both objectives. Third, Rs alone produces accurate physical-state predictions but substantially worse decoded forecasts, because the state-to-latent projection Πz is not supervised to align propagated states with measurement representations.
These findings clarify the central role of LFUT: it supervises the physics-structured transition while aligning the propagated latent zp with the measurement latent zm, enabling subsequent assimilation without reconstruction gradients updating the predictive prior. The ablation therefore provides a useful demonstration of the mechanism and its potential transferability. We will include these results in the revised manuscript.