acceptodds
Under review as a conference paper at ICLR 2027

DePhy: Decoupled Physics-Aware Predictive Models for State Estimation, Forecasting and Data Assimilation

14 views

Abstract

Physics-aware state estimation from noisy observations requires both robust measurement representations and dynamics that support physically plausible reconstruction. Existing physics-aware methods typically rely on reconstruction-driven representation learning, with physical consistency introduced through additional regularization or post-hoc correction. As a result, measurement-related nuisance variation can remain entangled with the states used for physical propagation. We introduce DePhy, a decoupled latent state-space framework that learns physical state propagation from measurement latent without reconstruction supervision. DePhy uses mask-based self-supervised learning to obtain representations that are robust to corrupted or incomplete measurements, while physics-mediated future latent prediction preserves information needed for physical state evolution. The resulting representations are mapped to a physical state space governed by an augmented physics-based model that combines analytical mechanics with bounded learned residuals. Amortized inference of trajectory-specific physical factors further enables instance-specific adaptation without per-instance optimization. To reconcile physics-based predictions with new observations, a Kalman-inspired neural assimilation module fuses the propagated state prior with current measurement evidence, yielding measurement-consistent states for reconstruction. By decoupling predictive representation and dynamics learning from reconstruction, DePhy learns states that are both robust to noisy measurements and suitable for stable physical propagation. Experiments on monocular human motion benchmarks demonstrate improved state estimation and long-horizon forecasting, with DePhy reducing Human3.6M MPJPE from 54.8 to 47.3 mm and acceleration error from 8.4 to 3.9 mm/s relative to OSDCap, while improving 1-s forecasting MPJPE from 103.9 to 90.8 mm relative to PhysMoP.

Primary area:
learning on time series and dynamical systems
Keywords:
State Estimation, Physics Aware, Data Assimilation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V9VYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for state space prediction - the main observation is that measurement noise/error can propagate through to impact representation and physical model learning. The key idea is to decouple these processes e.g. through weighted latent incorporation. The results and experiments show that there is improvement in physical prediction tasks, including very complex and noisy kinematic tasks like Human3.6M, JEPA is used in the latent embedder to learn measurement noise resilient embeddings. Future state prediction is used to train the state-space prediction. The main contribution is Sec 3.2 for measurement latent assimilation. I did like reading this paper, although I found that parts of it were overly dense and could have just been moved to the (extensive) appendix without loss of detail.

Critical strengths
  • The results are solid - good prediction across low complexity (pendulum) and high complexity (human motion) tasks, showing outperformance of competing baselines.
  • The overall method is quite elegant - whilst neural state space is nothing new, I did like the approach of using blended (assimilated) latents between the forward model and the actual observations to map against observed state. I think this is a neat approach and is effectively a form of curriculum learning from a good physical prior.
  • I found the ablation useful: particularly that JEPA is not the main source of improvement - this is a good finding, as it shows that there is strength in the wider method, rather than just learning a good system representation.
Critical weaknesses
  • With a "collection of tricks" paper, it is sometimes hard to disentangle where the actual novelty/delta lies. The authors have done a reasonable job i.e. Table 6 does show a very useful breakdown of where the improvement is coming from. Interestingly, good results are obtained already without the R_stay phase 2 training. And it seems as though the key innovation is L_{fut}. Simpler breakdown would help more here e.g. to show this improvement as a MWE e.g. on a pendulum problem. This could help to provide deeper insight into how the proposed method works. The proposed block that supports L_{fut} is simple: this is good news as it means that this is a very transferrable method.
  • E7 shows some detail on computational cost, but a greater breakdown/analysis would be useful here.
Questions for authors
  • Extension: Does the assimilation block work as a generic technique that can be applied to other prior art?
  • Unbounded forecasting: how much does this system "drift" compared with other works when observations are completely removed? What does this say about how well the model has learnt the underlying dynamics of the system?
4 comments
AuthorsOpenReview ↗

Response to Reviewer V9VY (1/2)

We thank the reviewer for their thoughtful and constructive assessment of our work, particularly for finding the overall method elegant and the ablation study informative. We appreciate the suggestions for clarifying the methodological contribution, computational cost, transferability of assimilation, and long-horizon forecasting behavior.

W1. Novelty and future prediction.

We appreciate the reviewer's observation that LFUT\mathcal{L}_{\mathrm{FUT}} is a central mechanism in DePhy, and the suggestion to demonstrate its contribution through a minimal working example. The analytical dynamics fKf_K, amortized physical factors ϕ^\hat\phi, and bounded residual fUf_U are jointly supervised through future measurement-latent prediction, rather than measurement reconstruction. The predictive prior is subsequently frozen before assimilation training. The stay regularizer Rstay\mathcal{R}_{\mathrm{stay}} supports assimilation (50.2 to 47.3 mm MPJPE; Table 6) and acts as an auxiliary loss in the Stage 2 assimilation step.

Following the reviewer's suggestion, we conducted a minimal ablation on the damped pendulum, where ground-truth physical factors are available for evaluation. The predictive prior is trained using the future-latent objective of Eq. 7,

LFUT=1Tf∑k=1Tf∥zt+kp−zt+k⋆∥22,zt+kp=Πz(st+kp),\mathcal{L}_{\mathrm{FUT}} = \frac{1}{T_f}\sum_{k=1}^{T_f} \big\Vert z^p_{t+k}-z^\star_{t+k}\big\Vert_2^2, \qquad z^p_{t+k}=\Pi_z(s^p_{t+k}),

where z⋆z^\star denotes the stop-gradient EMA target representation. We additionally examine a physical-state consistency regularizer,

Rs=1Tf∑k=1Tf∥st+kp−st+k∥22,st+k=Estate(zt+km),\mathcal{R}_s = \frac{1}{T_f}\sum_{k=1}^{T_f} \big\Vert s^p_{t+k}-s_{t+k}\big\Vert_2^2, \qquad s_{t+k}=E_{\mathrm{state}}(z^m_{t+k}),

and a reconstruction-supervised alternative,

Lrecfut=1Tf∑k=1Tf∥Dmeas(Πz(st+kp))−y~t+k∥22.\mathcal{L}^{\mathrm{fut}}_{\mathrm{rec}} = \frac{1}{T_f}\sum_{k=1}^{T_f} \big\Vert D_{\mathrm{meas}}\big(\Pi_z(s^p_{t+k})\big) -\tilde y_{t+k} \big\Vert_2^2.

Unlike LFUT\mathcal{L}_{\mathrm{FUT}}, the reconstruction objective directly supervises future measurements and permits reconstruction gradients to update the predictive prior. Table R6 compares these training objectives under otherwise identical settings.

Table R6. Pendulum ablation of the Stage 1 training signal. Decoded MSE measures the error of Dmeas(Πz(st+kp))D_{\mathrm{meas}}(\Pi_z(s^p_{t+k})) against clean observations; state MSE measures the error of the propagated physical state against the clean simulated state.

ConfigurationLFUT\mathcal{L}_{\mathrm{FUT}}Rs\mathcal{R}_sRec.Decoded MSE Tf=20T_f{=}20 ↓Decoded MSE Tf=40T_f{=}40 ↓State MSE Tf=20T_f{=}20 ↓State MSE Tf=40T_f{=}40 ↓FoV R2R^2 ↑ϵϕ\epsilon_\phi (%) ↓
DePhy (LFUT\mathcal{L}_{\mathrm{FUT}})✓0.00140.00570.00270.00310.9091.7
LFUT+Rs\mathcal{L}_{\mathrm{FUT}}+\mathcal{R}_s✓✓0.00110.00250.000890.00150.9111.68
Rs\mathcal{R}_s only✓0.04060.03760.000600.00120.9091.7
Reconstruction only✓0.1730.1530.1360.1140.4013.4

The results support three conclusions. First, future-latent prediction is substantially more effective than measurement reconstruction for training the predictive prior. Replacing LFUT\mathcal{L}_{\mathrm{FUT}} with reconstruction increases forecast error by one to two orders of magnitude, reduces FoV R2R^2 from 0.909 to 0.401, and increases parameter error from 1.7% to 3.4%. This is consistent with the human-motion ablation in Table 6, where allowing reconstruction gradients to update the prior increases acceleration error from 3.9 to 8.8 mm/s².

Second, Rs\mathcal{R}_s provides complementary physical-state supervision, but LFUT\mathcal{L}_{\mathrm{FUT}} accounts for most of the improvement: relative to the reconstruction-trained prior, LFUT\mathcal{L}_{\mathrm{FUT}} alone achieves approximately 98–99.8% of the forecast-error reduction obtained with both objectives. Third, Rs\mathcal{R}_s alone produces accurate physical-state predictions but substantially worse decoded forecasts, because the state-to-latent projection Πz\Pi_z is not supervised to align propagated states with measurement representations.

These findings clarify the central role of LFUT\mathcal{L}_{\mathrm{FUT}}: it supervises the physics-structured transition while aligning the propagated latent zpz^p with the measurement latent zmz^m, enabling subsequent assimilation without reconstruction gradients updating the predictive prior. The ablation therefore provides a useful demonstration of the mechanism and its potential transferability. We will include these results in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer V9VY (2/2)

(Part 2 of 2.)

W2. Computational complexity.

We thank the reviewer for requesting a more detailed computational analysis. The human-motion model contains 178.73M parameters during training and 93.56M at inference, excluding the 85.17M training-only EMA teacher. The measurement Transformer accounts for most inference parameters (85.17M), while the physics-structured modules contribute 4.68M and assimilation and decoding contribute 2.66M. The bouncing-ball and pendulum models are substantially smaller, with 0.30M and 0.64M inference-active parameters, respectively. The analytical physics core introduces no trainable parameters.

On a single NVIDIA A100 40 GB GPU, training and evaluation across Human3.6M, Fit3D, and SportsPose require approximately 14 hours, while training the synthetic models takes approximately 28 minutes for the ball and 16 minutes for the pendulum. Human-motion inference averages 34.8 ms per frame across different sequences, whereas batched amortized inference on the synthetic systems requires approximately 1.3 ms per ball trajectory and 0.4 ms per pendulum trajectory. We will include the breakdown in the revised Appendix E.7.

Q1. Transferability of the assimilation mechanism.

We appreciate the reviewer's question regarding whether the assimilation block can be applied to other predictive models. The gain-based update

zt∗=ztp+αt⊙(ztm−ztp)z_t^* = z_t^p+\alpha_t\odot(z_t^m-z_t^p)

requires a propagated prior and contemporaneous measurement features in a compatible latent space, rather than a particular ODE formulation. Consequently, the assimilation module could be trained with another fixed predictive prior after establishing a compatible latent interface.

Table 10 provides supporting evidence within DePhy: with the prior held fixed, the learned gain outperforms direct teacher forcing across the evaluated measurement-noise and availability settings. While the assimilation mechanism is generic in form, our empirical claims are limited to the priors and datasets evaluated in this work; we do not claim zero-shot transfer to other architectures.

Q2. Forecasting drift without observations.

We thank the reviewer for raising the question of drift during long-horizon, observation-free forecasting. Table 3 directly evaluates this setting: the DePhy w/o-DA configuration performs a predictive-prior rollout without observations beyond the context window, and we will make this explicit in the manuscript.

On Human3.6M-P1, the observation-free prior achieves lower error than PhysMoP from 160 ms onward, reaching 97.7 mm MPJPE at 1000 ms compared with 103.9 mm for PhysMoP and 106.9–112.1 mm for the kinematic baselines. The same holds for the averaged P2 protocol, where the prior reaches 52.4 mm at 1000 ms versus 57.1 mm for PhysMoP and 63.7 mm for siMLPe. Recursive assimilation using only model-generated, rather than ground-truth, measurement evidence further reduces the 1000 ms error to 90.8 mm on P1 and 48.9 mm on P2.

Table 8 (supp.) addresses the second part of the question, namely what long-horizon drift reveals about the learned dynamics. It varies only the transition structure under the same forecasting setting (recursive assimilation of model-generated evidence, with no ground-truth observations). A purely neural transition is accurate at short horizons but drifts the most as the horizon grows, reaching 125.4 mm at 1000 ms. Purely analytical propagation drifts less over long horizons (119.2 mm at 1000 ms, and lower error than the neural transition from 320 ms onward), but is too restrictive at short and medium horizons (12.4 versus 8.6 mm at 160 ms). Combining the analytical dynamics with a learned residual of limited capacity, as in DePhy (γmax⁡=0.30\gamma_{\max}{=}0.30), gives the lowest error at every horizon, reaching 91.7 mm at 1000 ms, whereas giving the residual more capacity increases long-horizon error again (101.7 mm at γmax⁡=0.50\gamma_{\max}{=}0.50 and 123.1 mm with an unrestricted residual). This indicates that the analytical dynamics account for long-horizon stability, while the learned residual corrects model discrepancy at shorter horizons.

The synthetic benchmarks provide complementary evidence beyond the training horizon. On the bouncing-ball task, DePhy achieves 0.0317 MSE at T=40T{=}40, compared with 0.0512 for MoNODE. Together, these results indicate that the physics-structured prior retains useful dynamical information and exhibits reduced forecasting error within the evaluated horizons. We do not claim stability for rollouts beyond the horizons evaluated here.

Dear Reviewer,

Thank you again for the positive and constructive review. We have uploaded the revised manuscript, with all changes marked in blue, and have tightened the main text while keeping implementation details in the supplement (Supp. C.7 and G). Here is where each point is addressed:

  • W1 (minimal working example of LFUT\mathcal{L}_{\mathrm{FUT}}): new Supp. F.1 (Table 14) ablates the Stage 1 training signal on the damped pendulum: LFUT\mathcal{L}_{\mathrm{FUT}} alone, with the state-consistency term Rs\mathcal{R}_s, Rs\mathcal{R}_s alone, and future-measurement reconstruction. The main-text ablation in Sec. 4.3 summarizes it. The role of Rstay\mathcal{R}_{\mathrm{stay}} as an auxiliary Stage 2 term is stated in Sec. 3.6 and Supp. E.5.
  • W2 (computational cost): Supp. E.7 now reports module-wise parameter counts (Table 11), encoder-size scaling with runtime (Table 12), module-wise inference runtime (Table 13; ODE integration accounts for 32.2 of 34.8 ms per frame), and training times for all systems.
  • Q1 (transferability of assimilation): Sec. 3.5 and Supp. A.4 state the interface the update requires and the scope of our claims. Supp. E.4 (Table 10, Fig. 5) compares the learned gain with direct teacher forcing and with PhysMoP's learned fusion under noisy and sparse measurements.
  • Q2 (drift without observations): Sec. 4.2 and Supp. C.10 define w/o DA as a rollout with no observations beyond the context window. Supp. E.2 (Table 8) analyzes how drift depends on the transition structure (neural-only, physics-only, and bounded-residual variants). Supp. E.8 states that we make no stability claim beyond the evaluated horizons.

We are grateful that you found the method elegant, the assimilated-latent approach neat and transferable, and the ablations informative. We hope the added pendulum example, compute breakdown, and drift analysis address your remaining points, and if so, we would be grateful if you would consider raising your score. We are happy to clarify anything further.

Thank you for your extremely detailed feedback - it helps to clarify where the gains come from in your work. The pendulum ablation is helpful as it is a simple toy example. And the information about computational complexity is useful Reading the other reviewer's comments as well, and taking into account the wider context of this work, I remain on the positive side, but I am content to remain on my current score. I hope that the review comments have helped guide your manuscript and boosted its clarity/coherence.

Reviewer_4XC4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DePhy, a decoupled latent state-space framework for physics-aware state estimation and forecasting from noisy observations, where a predictive measurement representation is learned through masked and future latent prediction without reconstruction supervision, mapped to an explicit physical state and propagated by an augmented physics-based model that conditions analytical mechanics on amortized trajectory-specific factors and adds a bounded learned residual. This predictive prior is then frozen, and a Kalman-inspired neural assimilation module learns a latent gain that fuses the propagated prior with current measurement evidence. Experiments on bouncing-ball, damped-pendulum dynamics, and monocular human motion benchmarks report gains over baselines in state estimation and long-horizon forecasting, together with improved recovery of physical factors on the synthetic systems.

Critical strengths

S1: The central design principle - keeping reconstruction gradients out of predictive-prior learning - is well-motivated and directly tested.

S2: The evaluation is well controlled, with all human-motion methods using the same fixed upstream TRACE estimator (Tables 1-2), and the ablation studies provide evidence supporting the design choices (Tables 6-8). The supplementary analyses are also detailed and add transparency to the behavior of the learned residual and assimilation gain as well as to the method’s limitations.

Critical weaknesses

W1: The reproducibility of this work would be impaired if no code is available. Uploading the anonymous code including any preprocessing and hyperparameter & seed documentation could help ensure reproducibility.

W2: The paper reports variance only for DePhy in Tables 1-2 but not for the remaining results; including such information would be relevant to assessing whether the margins we see in the results reflect genuine improvements. Some claims in this work warrant more careful qualification; for example, physical-factor recovery is stated across "synthetic and real-world benchmarks" (Conclusion) but evaluated only on synthetic data, and the "domain shifts" in Table 2 are evaluated after in-domain training or fine-tuning. The manuscript would also benefit from proofreading to address typographical errors and inconsistencies in notation.

Questions for authors

Q1: Could the authors report the number of seeds and the variance for the human-motion results in Tables 3 and 6-10? Besides, as Table 6 only switches LJEPAL_{JEPA} on or off, a sensitivity analysis on the masking and noising curriculum for LJEPAL_{JEPA} would help clarify how robust the reported gains are.

Q2: At present, incomplete measurements and noise sensitivity are examined only at forecast time against prior-only variant (Table 10). Have the authors also tested for baseline comparison on whether DePhy retain advantage over the baselines under different levels of missing measurements and injected noise?

Q3: There seem to be typos and editorial issues in the text, for example, Section C.10 refers to “GRE” while Table 2 uses “GRP”; some notations are inconsistent (e.g. ϕ^\hat{\phi} vs. ψ^\hat{\psi} in the main text vs. Section C, Πz\Pi_z vs. Πθ\Pi_\theta in Eq. 6 vs Fig. 1; undefined z′z' in Fig. 1); the two-stage training is described three times (Secs. 3.6, C.7, C.8) with differing loss weights that could be better explained; and there are LaTeX referencing artifacts such as “Eq. equation 16”. Please fix where needed.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4XC4 (1/3)

We thank the reviewer for the careful and constructive review, and for recognizing that the central design principle of DePhy, keeping reconstruction gradients out of predictive-prior learning, is well motivated and directly tested (S1), and that the evaluation is well controlled, with all human-motion methods sharing the same fixed upstream TRACE estimator and with ablations and supplementary analyses supporting the design choices (S2). Following the reviewer's suggestions, we have (a) documented the reproducibility details in the revised manuscript and committed to releasing code and synthetic data; (b) qualified the claims on physical-factor recovery and domain shift, and expanded the statistical reporting; (c) added a sensitivity study of the corruption curriculum (Table R4); (d) added an external-baseline comparison with PhysMoP under noisy and sparse measurements (Table R8); and (e) corrected the editorial and notational issues. We address each point below.

W1. Reproducibility.

Thank you for emphasizing reproducibility. The revised supplement documents the full training procedure and the implementation details for every experiment, including preprocessing, hyperparameters, curricula, seeds, and evaluation protocols. Table R1, in our response to Reviewer 1v9B (W2), also reports module-wise parameter counts for all three systems: 93.56M inference-active parameters for human motion (178.73M with the training-only EMA teacher), and 0.30M and 0.64M for the bouncing ball and pendulum, respectively.

We will release full code upon acceptance, together with the synthetic bouncing-ball and pendulum datasets and their generation scripts. Human3.6M, Fit3D, and SportsPose cannot be redistributed under their licenses but are publicly available from their authors; we document the access procedure and all preprocessing steps applied to the TRACE outputs.

W2. Variance and claim precision.

We appreciate the request for clearer statistical reporting. Tables 1–2 report variance for DePhy; comparator values are taken from published reports, many of which do not provide corresponding variances, and we have clarified this limitation in the revised manuscript. For our own remaining results, we will report seed counts and standard deviations where available: the synthetic results in Tables 4a and 5 are averaged over three seeds, and we will add their standard deviations, as already reported in Table 4b; similarly, the human-motion results in Tables 3, 6 and 7 are averaged over five seeds, and we will add their standard deviations where applicable. Table 8 and10 are single-seed analyses and we will specify them as such.

We have also qualified the two claims the reviewer identified. Physical-factor recovery is now stated only for the synthetic systems, where ground-truth factors are available (Conclusion). For the human-motion benchmarks, ground truth for the physical factors, such as actuation, contact forces, and body parameters, does not exist, so their recovery cannot be evaluated directly. We therefore evaluate human motion quantitatively through state-estimation and reconstruction metrics (MPJPE, global and Procrustes-aligned MPJPE, root error, acceleration error, and foot-skating rate), consistent with the evaluation protocols of the human-motion baselines such as OSDCap and QuaMo. The synthetic systems serve precisely to evaluate the recoverability of physical factors under controlled conditions. In addition, the Fit3D and SportsPose evaluations are now described as involving in-domain training or fine-tuning rather than zero-shot generalization (Section 4.1 and the caption of Table 2). Finally, we have corrected notation and typographical inconsistencies throughout the manuscript, including those listed in Q3.

AuthorsOpenReview ↗

Response to Reviewer 4XC4 (2/3)

Q1. Seeds and corruption curriculum.

We thank the reviewer for this question. The revised Tables 3, 6, and 7 report five-seed standard deviations, and Tables 8 and 10 are labeled as single-seed.

We agree that Table 6 evaluates the contribution of LJEPA\mathcal{L}_{\mathrm{JEPA}} but does not isolate sensitivity to the corruption schedule. We therefore ran an additional experiment on Human3.6M comparing the default curriculum with fixed corruption and with alternative final masking ratios (Table R4).

Table R4. Sensitivity to the corruption curriculum on Human3.6M.

CurriculumMPJPE (mm) ↓Accel (mm/s²) ↓
Default curriculum (DePhy, final p=0.6p=0.6)47.33.9
Fixed corruption (no curriculum)49.55.6
Weaker final masking (p=0.2p=0.2)48.75.3
Stronger final masking (p=0.8p=0.8)47.64.4

MPJPE varies by at most 1.4 mm across the tested final masking ratios, with the default curriculum achieving the lowest errors. Acceleration error is more sensitive to insufficient corruption: weaker final masking raises it from 3.9 to 5.3 mm/s², whereas stronger masking has a smaller effect (4.4 mm/s²). Compared with fixed corruption, the default curriculum reduces MPJPE from 49.5 to 47.3 mm (4.4%) and acceleration error from 5.6 to 3.9 mm/s² (30%). These results show that the progressive corruption curriculum improves both positional accuracy and temporal consistency, with the larger gain in temporal consistency, and that sufficiently strong final masking matters mainly for acceleration error.

AuthorsOpenReview ↗

Response to Reviewer 4XC4 (3/3)

Q2. Missing and noisy measurements.

We appreciate the suggestion to test robustness against baselines under corruption. Table 10 compares assimilation with direct teacher forcing of the same frozen prior under identical sparse and noisy inputs: at 10% availability and 10 dB SNR, error is 83.1 versus 131.5 mm. Since this is an internal comparison, we have extended it with an external baseline. We use PhysMoP, the strongest physics-informed forecasting baseline in Table 3 and the closest to DePhy in structure: it combines a physics-based prediction with a data-driven prediction through a learned fusion weight. We feed each available forecast-time measurement to PhysMoP's data-driven branch, whose prediction is then combined with the physics-based prediction by PhysMoP's fusion module, under the same measurement-quality and availability settings as Table 10 (Table R8). In this configuration, PhysMoP mirrors the structure of DePhy's assimilation: its physics branch plays the role of the propagated prior, its data-driven branch provides measurement-based evidence, and its fusion weight plays the role of the learned gain.

Table R8. Robustness to noisy and sparse forecast-time measurements: MPJPE (mm) at 1000 ms on Human3.6M. All methods receive identical measurements. TF: measurements directly replace the predicted state. w/ DA: measurements are fused with the prior through the learned gain. PhysMoP (fusion): available measurements are fed to PhysMoP's data-driven branch and then passed through its pretrained fusion module, used as released. Lower is better.

SNRMethod10%20%40%60%
CleanPhysMoP (fusion)94.883.674.767.3
DePhy prior only (TF)98.490.278.570.9
DePhy w/ DA78.569.260.852.6
30 dBPhysMoP (fusion)104.399.591.186.8
DePhy prior only (TF)104.9100.390.783.6
DePhy w/ DA79.671.162.853.7
20 dBPhysMoP (fusion)128.8120.5116.1111.5
DePhy prior only (TF)114.2111.7106.397.5
DePhy w/ DA80.773.467.458.2
10 dBPhysMoP (fusion)145.7138.2130.9129.4
DePhy prior only (TF)131.5125.2117.6108.1
DePhy w/ DA83.179.673.267.5

DePhy with assimilation retains its advantage at every noise level and availability. With clean measurements, PhysMoP makes use of the available evidence and improves on DePhy's teacher-forced prior (e.g., 94.8 versus 98.4 mm at 10% availability). As noise increases, however, PhysMoP degrades the fastest: at 10% availability, its error rises from 94.8 to 145.7 mm between clean and 10 dB measurements, compared with 98.4 to 131.5 mm for teacher forcing and only 78.5 to 83.1 mm for DePhy with assimilation; from 20 dB downward, it is the least accurate of the three methods. Since both methods share this prediction–correction structure, the difference lies in what the weighting has learned: PhysMoP's fusion weight was learned to balance two model predictions from clean motion histories, and therefore does not discount measurement evidence as it becomes unreliable, whereas DePhy's assimilation module is trained on a frozen prior to weigh measurement evidence, and its gain decreases as measurement quality drops (Fig. 5). Measurement-tracking estimators such as OSDCap and QuaMo require a measurement at every step and were not designed for this sparse forecast-time setting. For state estimation, the closest external evidence remains SportsPose (Table 2), which has the noisiest upstream measurements among our benchmarks and is where the acceleration advantage over OSDCap is largest (10.9 to 4.8 mm/s²). We will include Table R8 in the revised manuscript.

Q3. Editorial consistency.

Thank you for identifying these issues; all are fixed in the revised manuscript. We now use one name for global-root error throughout, unify the trajectory-factor notation (ϕ^\hat\phi in the main text and ψ^\hat\psi in Section C) and the prior notation between Eq. 6 and Fig. 1, and define every symbol shown in Fig. 1. Section 3.6 gives the conceptual objective, and a single implementation section (formerly C.7–C.8) gives the scheduled loss weights λFUT\lambda_{\mathrm{FUT}} and λres\lambda_{\mathrm{res}} and how they relate to Eq. 10. Duplicate descriptions and cross-reference artifacts such as "Eq. equation 16" have been removed.

Reviewer_1v9BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The authors propose DePhy, a physics-aware state estimator and forecaster for noisy measurement streams primarily coming from monocular estimates of human motion. The proposed model consists of two parts. First, a predictive prior is learned using a JEPA-style architecture with a masked latent loss and an EMA teacher. From the latent variables, a map to an explicit physical state is constructed, where the model learns an Euler-Lagrange dynamical system with a neural ODE (similar to PhysPT with MoNODE). In the second stage, this prior is frozen, and an element-wise gain with a decoder is learned on top of these dynamics with a reconstruction loss. The paper reports empirical improvements over OSDCap and QuaMo on a number of datasets, including Human3.6M, Fit3D, and SportsPose for forecasting, as well as other datasets for bouncing-ball / damped-pendulum synthetic experiments.

Critical strengths
  • S1. Good motivation. The paper argues that a dynamics prior for physical parameter estimation should be learned without reconstruction gradients in a latent representation, in the style of JEPA-flavor world models. The authors ablate this directly and show that variants roughly double the acceleration error (Table 6).
  • S2. Good performance on selected benchmarks. The method performs well across multiple metrics and beats, e.g., PhysMoP by a decent margin for long-horizon forecasting, suggesting robustness of the learned dynamics.
Critical weaknesses
  • W1 Method Scope. Physical forecasting using deep neural networks has a long history by now, and I find the scope of the paper somewhat odd and mispositioned. In the abstract, the authors discuss general physical priors (see minor notes below on how to improve this) and only later specify that their model is designed primarily for monocular vision settings. However, I don’t see why their model as more generally applicable across different settings; adjusting the scope would make their paradigm more relevant to a broad audience. Nevertheless, each component of the method is close to existing work. The authors essentially extend OSDCap/QuaMo style models with a JEPA-style architecture; this needs to come across clearly in the paper, along with the experimental scope.
  • W2. Experimental comparison. The authors compare against latent ODE methods such as MoNODE, but do not compare to the important baseline GOKU-net (Linial, Ori, et al 2021), which infers physical parameters and feeds them into a known ODE. Strong JEPA-style encoders can certainly be justified for encoding noisy real-world measurements, but the dynamics ultimately occur in physical space. Phys-VAE (Takeishi & Kalousis, NeurIPS 2021) takes a similar approach. Moreover, model capacity is not justified. The trade-off between prediction capacity and computation must be presented in the paper. It appears as though DePhy has ~178M parameters, which is orders of magnitude higher than the baselines. This suggests much of the gain comes from capacity, not from the decoupled physics design space. How does the performance of the model scale with the capacity, and why is such a massive model required for these relatively light tasks?
  • W3. Model Design. The method opts for a predictive representation using an EMA teacher with stop-gradient (I-JEPA/V-JEPA style). Current JEPA architectures replace these with an explicit distributional regularizer which is more effective in avoiding collapse, for instance SIGReg in LeJEPA (Balestriero & LeCun, 2025), and VICReg (Bardes et al., 2022). This regularization provides JEPA-type models with a mechanism to prevent collapse without requiring an additional EMA teacher with non-stationary training targets. This is particularly relevant to this problem setting because Stage 1 has no supervision and the physics transition is trained against non-stationary EMA targets under a custom training paradigm. Please (i) justify the EMA design, (ii) report collapse diagnostics for zᵐ, and (iii) ablate replacing the EMA teacher with SIGReg or VICReg. If it turns out that an isotropic regularizer hurts, this would be interesting and would support the authors' claims.

Minor

  • The abstract is written in a way that it was not at all clear to me that this is a computer vision paper until the last sentence. This should be rewritten more appropriately for a (general) machine learning audience.
  • L196-198: these are experimental conditions that are being mixed in with the method contribution. They should be properly separated; it’s better to describe the target as a general parameter vector and how this changes for different problem settings in the appropriate section.
  • Table 1: bold the best model in the MPJPE-PA column for fairness
  • What is w/ DA, and w/o DA? The authors never define it. Please define this in the main text as well as in table captions.
  • Using a root rotation as “a minimal coordinate Rt∈R3R_t \in \mathrm{R}^3 is not appropriate for dynamics; I encourage the authors to refer to QuaMO or SO(3)-nCDEs (ICCV 2025) on representation learning for rotational forecasting. Please clarify the parameterization and whether it affects rollout stability.
Questions for authors
  • How does performance scale with model size, and how many of the 178M parameters are used at inference? Does this include the EMA teacher? Please add some capacity-matched experiments and runtime experiments/ablations.
  • Please compare against baselines that use known physical, and not latent, models.
  • Please justify the EMA teacher and report collapse diagnostics on the latent representations. Also discuss how SO(3) representations affect learning.
  • Justify the positioning of the paper for monocular settings more specifically.
8 comments
AuthorsOpenReview ↗

Response to Reviewer 1v9B (1/6)

We thank the reviewer for the detailed and constructive review, and for recognizing both the motivation for learning the dynamics prior without reconstruction gradients (S1) and the long-horizon forecasting performance of DePhy (S2). Following the reviewer's suggestions, we have (a) clarified the contribution of DePhy relative to OSDCap, QuaMo, and PhysPT, together with its experimental scope; (b) added GOKU-net to the pendulum benchmark and Phys-VAE to the factor-recovery comparison; (c) provided a full parameter breakdown, showing that 93.60M of the 178.77M parameters are active at inference, and started an encoder-size scaling study; (d) reported collapse diagnostics for zmz^m; (e) compared the EMA teacher with VICReg and SIGReg; and (f) clarified the root-rotation parameterization and its effect on rollout stability. We address each point below.

W1. Scope and novelty.

Methodological contribution. The novelty of DePhy is not limited to applying a JEPA-style encoder to monocular-video-based human state estimation methods such as OSDCap and QuaMo. Its central motivation is to learn a predictive physical prior independently of reconstruction supervision, while retaining sufficient flexibility to accommodate incomplete physical models. This distinguishes our formulation from OSDCap and QuaMo, which employ measurement-tracking controllers trained with reconstruction-based supervision, and from PhysPT, which incorporates physical consistency through additional training losses on a reconstruction objective. Specifically, DePhy extends reconstruction-free predictive representation learning, as explored in S-JEPA, GFP, and Path-JEPA, to physics-aware state propagation. While these methods learn representations primarily for skeleton-based action recognition, DePhy uses future latent prediction to directly supervise an augmented physics-based model (APBM). We would also like to point that DePhy presents a unified framework for forecasting, data assimilation and physics aware state estimation and reconstruction. While QuaMo, OSDCap and PhyPT only allows for state estimation and reconstruction.

This distinction is supported by the ablations in Table 6. Allowing reconstruction gradients to update the predictive prior increases acceleration error from 3.9 to 8.8 mm/s² on Human3.6M and from 4.8 to 12.8 mm/s² on SportsPose. Furthermore, the learned prior supports open-loop forecasting without incoming measurements, achieving 97.7 mm MPJPE at 1 s compared with 103.9 mm for PhysMoP (Table 3). Finally, while we adopt PhysPT's Phys-SMPL body model, the formulation remains effective without PhysPT-style physical warm initialization, achieving 49.8 mm MPJPE on Human3.6M (Table 7), compared with 52.7 mm for PhysPT and 54.8 mm for OSDCap.

Experimental scope. The primary real-world application of DePhy is physics-aware state estimation and forecasting from noisy kinematic measurements derived from monocular video. We will make this explicit at the beginning of the abstract and clarify the distinctions from related methods in the introduction and related work. Although the framework is formulated for dynamical systems with a specified analytical component, each instantiation requires system-specific physical models and inference heads. The bouncing-ball and damped-pendulum experiments (Tables 4–5) therefore serve as controlled evaluations of physical-factor recovery and extrapolation, rather than as evidence of generalization to arbitrary dynamical systems. Monocular human motion remains the primary real-world application studied in this work.

AuthorsOpenReview ↗

Response to Reviewer 1v9B (2/6)

W2. Known-physics baselines and capacity.

We thank the reviewer for requesting stronger physical-model comparisons and clearer cost accounting, and for pointing us to GOKU-net, which we will discuss in the revised related work.

Relation to GOKU-net and Phys-VAE. We agree that JEPA-style encoders suit noisy measurements while the dynamics should evolve in physical space, and DePhy follows this principle: the JEPA encoder provides only the measurement latent, which EstateE_{\mathrm{state}} maps to an explicit physical state propagated by the analytical ODE (Eq. 4). GOKU-net and Phys-VAE are relevant known-physics baselines, and DePhy shares certain conceptual elements with them: the notion of known unknowns with GOKU-net, and regularized neural components for incomplete physics with Phys-VAE. Both, however, rely on reconstruction-based variational training, whereas DePhy uses future latent supervision and decouples the predictive prior from reconstruction. Its formulation differs in four respects. (i) DePhy supervises its instance-adaptive APBM through future latent prediction, without reconstruction gradients updating the predictive prior; allowing reconstruction to update the prior increases acceleration error from 3.9 to 8.8 mm/s² on Human3.6M (Table 6). (ii) DePhy learns measurement assimilation separately, with the predictive prior frozen. (iii) Beyond trajectory-level physical factors, it infers time-varying quantities with known mechanical roles, including actuation, contact forces, and contact states. (iv) Unlike GOKU-net, DePhy accounts for incomplete physics through a bounded residual, whose removal reduces mean factor-recovery R2R^2 on the pendulum from 0.89 to 0.76 (Table 5). Together, these distinctions establish the central contribution of DePhy: learning an instance-adaptive, physics-structured predictive prior through latent supervision, decoupled from downstream assimilation and reconstruction.

Known-physics baselines. We have added GOKU-net to the damped-pendulum benchmark as a known-physics parameter-inference baseline, using the same damped-pendulum ODE as the analytical branch of DePhy (θf=[ℓ,g,c]\theta_f = [\ell, g, c]) and the same observations, context length, forecast horizons, data splits, validation-based grid search, and three seeds as all other methods; factor recovery uses the Table 5a linear probe on its inferred parameters θ^f\hat\theta_f. We also report factor recovery for Phys-VAE by probing the posterior mean of its physics-grounded latent zPz_P. As Table R2 shows, DePhy outperforms both on every metric, consistent with the difference in training signal. The correctly specified fitted ODE in Table 5 remains best at Tf=40T_f{=}40 (0.0038); this is expected, because the fitted ODE has the correct simulator form, while DePhy learns the transition from noisy measurements through the same representation-learning pipeline used in the other experiments. For human motion, the physics-only variant in Table 8 (γmax⁡=0\gamma_{\max}{=}0) reaches 119.2 mm at 1 s versus 91.7 mm for DePhy. The revised Table 5 includes GOKU-net and adds Phys-VAE to Table 5a.

Table R2. GOKU-net and Phys-VAE on the damped-pendulum benchmark, under the same protocol as Table 5.

MethodMean R2R^2 ↑MSE (Tf=20T_f{=}20) ↓MSE (Tf=40T_f{=}40) ↓ϵϕ\epsilon_\phi (%) ↓
GOKU-net0.700.00380.01422.9
Phys-VAE0.720.00270.01182.8
DePhy0.890.00150.00591.7
AuthorsOpenReview ↗

Response to Reviewer 1v9B (3/6)

Capacity. The 178.73M figure in E.7 includes the training-only EMA teacher (85.17M); DePhy uses 93.56M parameters at inference (Table R1; TRACE excluded), of which 85.17M form the measurement Transformer, configured as in PhysPT (6 layers, width 1024, 8 heads). The physics-structured modules total 4.68M, assimilation and decoding 2.66M, and fKf_K has no learned parameters; the bouncing-ball and pendulum models use only 0.30M and 0.64M. OSDCap (7.2M) and QuaMo (0.62M) have no measurement encoder. As a diagnostic decomposition, the non-encoder components of DePhy total 8.39M parameters, comparable to OSDCap; however, we agree that the inference-active encoder is part of the deployed model, so we additionally report encoder-size scaling below. The encoder alone does not explain the gains: with the same encoder, training with only LJEPA\mathcal{L}_{\mathrm{JEPA}} gives 76.2 mm MPJPE, close to the 78.1 mm of the TRACE input, and a reconstruction-supervised prior gives 52.6 mm, versus 47.3 mm for DePhy (Table 6); a neural-only transition gives 125.4 mm at 1 s versus 91.7 mm (Table 8). The encoder also enables open-loop forecasting (Table 3) and operation under sparse measurements (Table 10). OSDCap and QuaMo are designed around sequential correction from incoming measurements, and were not designed or evaluated for the same open-loop and sparse-measurement forecasting protocol. PhysPT uses a similar Transformer configuration to our measurement encoder (6 layers, width 1024, 8 heads), within a full encoder–decoder architecture, and obtains 52.7 mm MPJPE versus 47.3 mm for DePhy. To address scaling directly, we are training DePhy with measurement encoders of width 256 and 512 (approximately 10M and 27M inference-active parameters), keeping all else fixed, and will report MPJPE, acceleration error, and runtime as a function of encoder size, together with Table R1, in the revised E.7.

Table R1. Parameter counts by module. The EMA target encoder is used only during training; TRACE is excluded.

ModuleHuman3.6MBouncing ballPendulum
Measurement encoder EmeasE_{\mathrm{meas}}85.17M76.5K76.1K
EMA target encoder (training only)85.17M76.5K76.1K
Trajectory-factor inference MtrajM_{\mathrm{traj}}0.98M164.7K513.9K
Bounded residual and gate1.28M12.1K10.8K
Assimilation AηA_\eta2.10M16.5K16.5K
Other neural modules†4.03M27.1K25.9K
Analytical dynamics fKf_K0 (analytical)0 (analytical)0 (analytical)
Total (training)178.73M373.4K719.3K
Inference-active (excl. EMA)93.56M296.9K643.2K

†Human3.6M: Predmeas\mathrm{Pred}_{\mathrm{meas}} 1.05M, EstateE_{\mathrm{state}} 0.87M, MdynM_{\mathrm{dyn}} 0.95M, Πz\Pi_z 0.60M, DmeasD_{\mathrm{meas}} 0.56M.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response from Authors

Central clarification. We thank the reviewers for their careful reading and detailed reviews. DePhy targets physics-aware state estimation and forecasting from corrupted measurement streams, with monocular human motion as the primary real-world setting. The key design is to learn a predictive physical prior without reconstruction gradients and freeze it before learning measurement assimilation, thereby separating measurement representation learning, physics-based propagation, and reconstruction and assimilation. The synthetic systems are used as controlled diagnostics for factor recovery and extrapolation, not as evidence of arbitrary-system generalization. Its contribution has two parts.

  • A decoupled system design. DePhy separates three roles that physics-aware estimators often optimize jointly: (a) measurement representation learning, (b) state estimation and physics-based state propagation (forecasting), and (c) measurement reconstruction and assimilation. For (a), JEPA-style objectives learn the measurement latent by predicting in latent space from masked and corrupted context, without reconstructing the input, which suits the noisy kinematics produced by monocular estimators. Unlike prior JEPA uses for action recognition or generic forecasting, DePhy uses latent prediction to supervise a physics-based transition for state estimation. For (b), the physical state is defined explicitly, separately from the measurement latent, and is propagated by analytical physics. For (c), reconstruction and assimilation are trained only after the prior is frozen, so reconstruction gradients never reshape the representation or the dynamics. Table 6 tests this decoupling with a fixed architecture and fixed TRACE inputs: letting reconstruction update the prior raises acceleration error from 3.9 to 8.8 mm/s² on Human3.6M.
  • An amortized augmented physics-based model (APBM) for state propagation. From the measurement latent, the transition learns both kinds of missing physics: amortized inference estimates known unknowns, quantities with a known role in the equations whose values are unobserved, and a bounded residual absorbs unknown unknowns, effects the analytical model omits. Monocular human motion has both. The Newton–Euler model of the SMPL body requires actuation, contact forces, and contact states that cannot be observed from video, and it simplifies contact and omits musculoskeletal effects. Tables 4, 5, and 8 test this design: removing the residual lowers mean factor-recovery R2R^2 from 0.82 to 0.70 (bouncing ball) and from 0.89 to 0.76 (pendulum), and physics-only propagation gives 119.2 mm at 1 s versus 90.8 mm for DePhy.

Beyond the framework itself, DePhy also contributes to monocular human motion estimation, where it couples a JEPA-style measurement encoder with physics-based human dynamics for state estimation and forecasting, and adds a decoupled, data-assimilation-style stage for measurement reconstruction; prior JEPA work on human motion has focused on recognition and generic forecasting.

Thus, \DePhy\ provides a principled method for structuring physics-guided learning from corrupted measurements, in which measurement nuisance is kept out of the physical prior by construction rather than through additional regularization. Our evaluation focuses on monocular human motion estimation and forecasting.

Dear Reviewer,

Thank you for the follow-up. We are glad that the sensitivity study, the PhysMoP comparison, and the tightened claims address several of your concerns.

We have now uploaded the revised manuscript with all changes marked in blue. Regarding reproducibility, we have committed in the Reproducibility Statement (after the Conclusion) to release the full code upon acceptance, together with the synthetic bouncing-ball and pendulum datasets and their generation scripts; Human3.6M, Fit3D, and SportsPose cannot be redistributed under their licenses, so the release will document how to access them and all preprocessing applied to the TRACE outputs.

Additionally, the supplement specifies every setting needed to reproduce each experiment: Supp. G (Table 18) lists preprocessing, splits, optimizers and schedules, epochs, curricula, loss weights, solver settings, and seeds for all three systems; Supp. C.7 gives the architecture, layer sizes, and two-stage training schedule; and Supp. C.5, C.8–C.10, D.1, and E.7 cover warm initialization, datasets and evaluation protocols, synthetic training, and compute, parameter counts, and runtime.

We have also addressed all the promised changes in the manuscript. DePhy results are now reported as mean ± std over five seeds for human motion and three seeds for the synthetic systems (Sec. 4, Tables 1–7); single-seed analyses are labeled in their captions (Tables 8, 10, 12, 14, 16, 17), and Sec. 4.3 notes that the MPJPE differences in Table 6 are several times the standard deviations. Physical-factor recovery is claimed only for the synthetic systems (Sec. 4.1, Supp. C.9, E.8). For Fit3D and SportsPose, Sec. 4.1 and the Table 2 caption state the training protocol, and Supp. C.8 and C.11 state that neither evaluation is zero-shot. The corruption curriculum is specified in Supp. C.7 and Table 18, with the sensitivity study in Supp. F.3 (Table 17), and the PhysMoP comparison under noisy and sparse measurements is in Supp. E.4 (Table 10). Finally, we have also fixed all the editorial inconsistencies that the reviewer pointed out.

We appreciate the reviewer's constructive feedback, which has helped us strengthen the manuscript. Given that we have now incorporated the requested revisions and addressed the concerns raised, we respectfully ask the reviewer to reconsider their assessment and consider increasing their score. We would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank the reviewers and the AC for their time, thoughtful comments, and constructive suggestions. We particularly appreciate the reviewers recognizing the motivation for learning the dynamics prior without reconstruction gradients as well-founded and directly tested (1v9B, 4XC4), finding the overall method elegant (V9VY), and highlighting the well-controlled evaluation and informative ablations (4XC4, V9VY).

Paper summary. DePhy is a decoupled state-space framework for physics-aware state estimation and forecasting from noisy measurements, with monocular human motion as the primary application. A joint-embedding predictive architecture learns a measurement representation and an instance-adaptive physical transition (analytical mechanics with amortized physical quantities and a bounded learned residual) through masked and future latent prediction, without reconstruction supervision. This predictive prior is then frozen and used for latent data assimilation, forecasting, and physically plausible reconstruction. DePhy improves state estimation on Human3.6M, Fit3D, and SportsPose and long-horizon forecasting on Human3.6M, while recovering trajectory-specific physical factors on synthetic systems.

We've uploaded the revised manuscript with all changes marked in blue.

Revisions:

  • Positioning: Sharpened the focus on monocular human motion and clarified the differences from OSDCap, QuaMo, PhysPT, skeleton-based JEPAs, GOKU-net, and Phys-VAE.
  • Additional baselines and analyses: Added comparisons with GOKU-net and Phys-VAE; parameter counts, encoder-size scaling, and runtime analysis; collapse diagnostics and comparisons of EMA with VICReg and SIGReg; a minimal pendulum example isolating the future-prediction loss; corruption curriculum sensitivity; and a PhysMoP comparison under noisy and sparse measurements.
  • Statistics and reproducibility: Added standard deviations for the main results, explicitly identified single-seed analyses, and consolidated experimental settings, architectural details, parameter counts and computational details in supplementary, alongside our commitment to release the code and synthetic data upon acceptance.
  • Clarity and notation: Refined the scope of our claims, clarified forecasting variants and root-rotation parameterization, standardized notation, and consolidated the training description.

We are grateful for the reviewers' suggestions, which have helped us make the paper clearer and more complete. While the underlying method, training procedure, and experimental protocol remain unchanged, we have strengthened the manuscript with the requested experiments, additional analyses, and clearer explanations. We hope the updated version makes it easier to assess our contributions and addresses the concerns raised during the review. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.