acceptodds
Under review as a conference paper at ICLR 2027

Bayesian Embedding of Physical Knowledge within Continuous-Time Latent Neural Models for Dynamic Systems

Abstract

Neural differential equations provide a flexible framework for learning dynamical systems from data, but can struggle with limited, noisy, and sparse data. Physical relations can provide a valuable inductive bias in these settings, yet existing soft constraints provide no inference-time guarantees, while hard state-space projections can be brittle under noise or misspecified physics. In this paper, we propose a Bayesian framework that embeds physical knowledge as functions of a vector field within a learned latent space, enforced via differentiable weighted projections. Performing joint Bayesian inference over both the model and physics-based parameters propagates uncertainty across the full framework. We show that, at each instant, latent projection approaches the same correction to the observed dynamics as state-space projection in the vanishing-regularization limit, but leaves the null space of the decoder Jacobian unconstrained. Across six forced dynamical systems, latent projection improves accuracy and calibration over unconstrained, soft-constrained and state-space-projected models when the physics is correct, and degrades far less than state-space projection when the embedded physics is misspecified.

Primary area:
learning on time series and dynamical systems
Keywords:
neural differential equations, physics informed learning, bayesian inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dK8kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper lifts partial physical relations in the observation space into constraints on the latent vector field through the decoder Jacobian, and enforces them at each vector-field evaluation using a weighted projection. The method combines continuous-time variational inference over neural network weights with inference over physical parameters, and analyzes the effect of projection on local uncertainty. Experiments cover six controlled dynamical systems under observation corruption, partial observability, and physical misspecification.

Critical strengths
  1. The problem is clearly defined and practically relevant. The core method is clearly formulated, and the theoretical results provide useful interpretations of its behavior.

  2. The proposed method achieves the lowest mean MSE on five of the six systems. The experiments further cover various types of data corruption and physical misspecification, providing reasonably broad initial evidence of its practical value.

Critical weaknesses
  1. Limited novelty and insufficient validation of the robustness mechanism. The core projection formula (Eq. 7) is mathematically equivalent to the general weighted projection for linear equality constraints in ProbHardE2E (Utkarsh et al., End-to-End Probabilistic Framework for Learning with Hard Constraints, ICLR 2026, Appendix C.1, Eq. 15). Specifically, setting z = f_theta, A = C J_phi, b = g, and Q = J_phi^T J_phi + lambda I in the general projection formula recovers Eq. 7. Therefore, the overall method appears largely to integrate existing techniques. Furthermore, although the experiments cover six dynamical systems, five of them impose essentially the same position–velocity kinematic relation. Thus, the diversity of embedded physical constraints is rather limited, and it remains unclear whether the proposed method generalizes effectively to more complex forms of partial physical knowledge.

  2. Discrepancies between the paper and the supplementary code. The provided code appears inconsistent with the manuscript in several important respects. First, a nonlinear tanh bounding operation is applied after projection, which can violate the claimed exact satisfaction of physical constraints. Second, the saved prediction trajectories are generated from a training trajectory rather than held-out test trajectories, although the script separately computes test losses. Third, the claimed Bayesian posterior over physical parameters does not appear to be implemented: physical parameters are represented as deterministic arrays, and the code does not appear to include the variational posterior q_eta(Lambda) or its corresponding KL term in the ELBO. These discrepancies raise concerns about the reproducibility of the reported results. The authors should clarify whether they arise from a code version mismatch or from important implementation details not described in the paper.

Questions for authors
  1. Could the authors clarify the discrepancies between the supplementary code and the manuscript? Was a different version of the code used to produce the reported results?

  2. Could the authors provide experiments involving more diverse physical constraints beyond simple position–velocity relations to better demonstrate the generality of the proposed framework?

3 comments
AuthorsOpenReview ↗

Rebuttal to reviewer dk8k

1. Novelty relative to ProbHardE2E, and diversity of the embedded constraints (Weakness 1, Question 2)

"The core projection formula (Eq. 7) is mathematically equivalent to the general weighted projection for linear equality constraints in ProbHardE2E [...] Therefore, the overall method appears largely to integrate existing techniques."

The reviewer has noted our projection is mathematically identical to one in ProbHardE2E - this is because both projections are derived as KKT solutions to a minimum distance optimization problem. Our related work cites projection based approaches as a popular method for embedding constraints, which encompasses ProbHardE2E (we note we did not cite this). However, the contexts used between them are completely different - the ProbHardE2E utilises this as an output conditioning layer on a probabilistic forecaster, that projects an output location scale distribution onto a constraint manifold, using a covariance weighting.

In our work, we utilize a projection inside a latent space, with our weighting instead being the pullback through a learned decoder. Our approach acts on a vector field at every solver evaluation, that allows us to constrain dynamics along a full trajectory, not outputs at one time. Both are results of projections derived from differentiable optimization layers, we are happy to reference ProbHardE2E as an alternative projection method in related work.

"five of them impose essentially the same position–velocity kinematic relation [...] it remains unclear whether the proposed method generalizes effectively to more complex forms of partial physical knowledge."

We appreciate the reviewers concerns regarding generalisability. We have added two experiments that address this directly: a new case study with physics that are not kinematic structures, and an ablation over the systems with multiple relations, number of embedded and type of physics.

Dynamic Chemical Reactor Firstly, we have added a non-isothermal CSTR with series, parallel and side reactions, comprising 6 states and 3 inputs with Arrhenius kinetics. The embedded knowledge is a set of species mole balances. We test it under full observation and under partial observation, where two species are unable to be accessed. Scoring the MSE / CRPS and WIS on the held out test set:

SettingLatent projection (ours)State-space projection
Full observation, 1 balance0.13 ± 0.017 / 0.72 ± 0.0490.14 ± 0.11 / 0.85 ± 0.34
Full observation, 2 balances0.12 ± 0.019 / 0.65 ± 0.0260.23 ± 0.35 / 0.93 ± 1.1
Partial observation (c_D, c_E hidden), 1 balance0.0079 ± 0.0033 / 0.12 ± 0.0320.011 ± 0.0039 / 0.15 ± 0.014
Partial observation (c_D, c_E hidden), 2 balances0.017 ± 0.0029 / 0.17 ± 0.0190.011 ± 0.0032 / 0.14 ± 0.018
Misspecified balances, 10.14 ± 0.017 / 0.77 ± 0.0670.18 ± 0.052 / 1.1 ± 0.22
Misspecified balances, 20.14 ± 0.028 / 0.72 ± 0.0370.24 ± 0.28 / 1.2 ± 1
Setting (observation WIS, normalised)Latent projection (ours)State-space projection
Full observation, 1 balance0.45 ± 0.0280.68 ± 0.16
Full observation, 2 balances0.4 ± 0.0140.73 ± 0.45
Partial observation (c_D, c_E hidden), 1 balance0.12 ± 0.00540.19 ± 0.031
Partial observation (c_D, c_E hidden), 2 balances0.13 ± 0.00460.17 ± 0.029
Misspecified balances, 10.48 ± 0.0370.72 ± 0.15
Misspecified balances, 20.43 ± 0.0260.81 ± 0.44

Secondly, on the systems that admit several relations (the reactor and the chained pendulum), we sweep the number and type of embedded relations. This shows how the method behaves as more physics is added, and when that physics is wrong.

For the reactor: we embed one or two species balances. Our latent approach under full observation improved as more balances are added and this remains ahead of state-space projection over the vast majority of settings.

This remains the case for the high pendulum, the latent projection is ahead. On the pendulum it also stays stable under misspecified relations, where state-space projection diverges on all 9 runs.

AuthorsOpenReview ↗

Response to reviewer dk8k

We embed 12 separate constraint sets as combinations of the kinematics and dynamic rows, and misspecified variants of each. The dynamic rows go beyond the position–velocity relation. The table reports MSE and CRPS on the held out test set

Embedded relationsLatent projection (ours)State-space projection
1 kinematic0.031 ± 0.0052 / 0.74 ± 0.0121.1 ± 0.44 / 5.5 ± 1.7
2 kinematic0.03 ± 0.0053 / 0.73 ± 0.0181 ± 0.39 / 3.8 ± 1.5
3 kinematic0.027 ± 0.0024 / 0.67 ± 0.0350.16 ± 0.21 / 1.9 ± 0.9
1 dynamic0.017 ± 0.0015 / 0.57 ± 0.0170.81 ± 0.32 / 4.2 ± 0.61
2 dynamic0.017 ± 0.0026 / 0.52 ± 0.0490.62 ± 0.46 / 3.2 ± 1.3
3 dynamic0.015 ± 0.0011 / 0.46 ± 0.0220.49 ± 0.0089 / 3 ± 0.13
3 kinematic + 1 dynamic0.014 ± 0.0017 / 0.53 ± 0.0120.29 ± 0.34 / 2 ± 1.4
3 kinematic + 2 dynamic0.011 ± 0.0014 / 0.45 ± 0.0130.032 ± 0.013 / 0.65 ± 0.12
Misspecified kinematic (3 wrong)0.031 ± 0.0056 / 0.75 ± 0.016all seeds diverged
Misspecified kinematic (1 of 3 wrong)0.032 ± 0.0073 / 0.74 ± 0.013all seeds diverged
Misspecified kinematic (2 of 3 wrong)0.031 ± 0.0052 / 0.74 ± 0.014all seeds diverged
Misspecified dynamic0.014 ± 0.0044 / 0.45 ± 0.0280.49 ± 0.054 / 3 ± 0.069
Embedded relations (observation WIS, normalised)Latent projection (ours)State-space projection
1 kinematic0.51 ± 0.00462.9 ± 0.91
2 kinematic0.51 ± 0.00972.4 ± 0.81
3 kinematic0.49 ± 0.0221.8 ± 0.64
1 dynamic0.43 ± 0.0132.3 ± 0.25
2 dynamic0.42 ± 0.0472.1 ± 0.41
3 dynamic0.39 ± 0.021.9 ± 0.13
3 kinematic + 1 dynamic0.4 ± 0.0121.7 ± 0.26
3 kinematic + 2 dynamic0.37 ± 0.0311.5 ± 0.15
Misspecified kinematic (3 wrong)0.51 ± 0.002all seeds diverged
Misspecified kinematic (1 of 3 wrong)0.52 ± 0.0037all seeds diverged
Misspecified kinematic (2 of 3 wrong)0.51 ± 0.0063all seeds diverged
Misspecified dynamic0.38 ± 0.00451.9 ± 0.16

2. Supplementary code versus the manuscript (Weakness 2, Question 1)

The supplementary code was a standalone extract of a larger codebase, which we reduced to a minimal working repository with an LLM coding agent. All results were generated from a larger respository which we will release subject to publication.

  • The tanh bound was applied as a numerical safeguard, and only acts when a drift increases too much in magnitude. We have verified that held-out trajectories the bound has a neglible impact on the constraint residuals.

  • The extract's plotting function use training trajectory 0. Both the plots in Figure 1 and the rest of the paper, and all metrics are computed on held-out test trajectories.

  • The posterior over Λ\Lambda is implemented in the full codebase. It is used when we make the parameters learnable (i.e. across the various misspecification regimes).

Thank you for the detailed response and additional experiments. The new experiments with more diverse physical constraints have addressed my concern about the generalisability. However, I still have concerns about the novelty and the reproducibility. Therefore, I will keep my original rating for now.

Reviewer_rj34OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Summary

This paper proposes a Bayesian framework for incorporating partial physical knowledge into continuous-time latent neural differential equations. The key idea is to enforce physical relations through a differentiable weighted projection of the latent vector field, where observation-space constraints are lifted into latent space using the decoder Jacobian. The authors derive a closed-form projection operator and show that it enforces the specified physical relations while preserving the component of the latent vector field in the null space of the decoder Jacobian. The proposed method is combined with a continuous-time Bayesian formulation that places a stochastic-process posterior over the neural vector-field parameters and performs joint inference over uncertain physical parameters. The paper also characterizes the effect of projection on local predictive uncertainty. Experiments on six synthetic dynamical systems demonstrate improvements over several unconstrained, soft-constrained, and state-space projection baselines, particularly under noisy observations, partial observability, and physical misspecification.

Critical strengths
  • Strengths
  1. Interesting problem formulation. Incorporating imperfect physical knowledge into probabilistic dynamical models is an important problem. The paper addresses the practical tension between enforcing physical consistency and retaining flexibility when the available physical knowledge is incomplete or inaccurate.

  2. Well-motivated latent projection framework. The use of the decoder Jacobian to lift observation-space constraints into latent dynamics is mathematically natural. The closed-form projection provides an interpretable mechanism for enforcing physical relations during integration.

Critical weaknesses
  • Weaknesses
  1. The connection between latent null-space freedom and robustness to physical misspecification is not sufficiently established. A central motivation of the paper is that latent projection is less brittle than state-space projection because it preserves directions in the null space of the decoder Jacobian. Proposition 1 provides a useful local characterization of this property. However, the proposition does not directly establish that preserving these directions leads to improved robustness under misspecified physics. In particular, both latent and state-space projections enforce the specified physical relation exactly in the decoded dynamics. If this relation is structurally incorrect, the decoded derivative remains constrained by the misspecified relation regardless of the latent dimension. The null-space directions may affect future trajectories through the state-dependent decoder geometry, but they cannot directly remove the instantaneous error in the constrained observable derivatives. Therefore, the claimed robustness appears to depend on how the learned decoder and latent dynamics compensate for the imposed constraint over time, rather than following directly from Proposition 1. I would appreciate a more precise characterization of this mechanism. In particular, it would be useful to distinguish the benefits of the additional latent degrees of freedom from those arising from the projection formulation itself.

  2. The novelty of the proposed framework relative to existing projection-based latent dynamical models could be clarified. The proposed method combines several established ingredients: latent neural differential equations, equality-constrained quadratic projections, decoder-Jacobian-based mappings, and SDE-based variational Bayesian inference. While their integration is useful, the projection formula follows from standard constrained quadratic optimization, and the continuous-time Bayesian inference construction largely builds upon existing stochastic-weight neural differential equation methods. The paper would benefit from a clearer explanation of which components are fundamentally new and which are adaptations of existing techniques. In particular, the distinction from previous approaches to latent conservation laws, projected neural differential equations, and partially stochastic Bayesian neural networks should be made more explicit. I would also encourage the authors to isolate the contribution of latent projection from that of the Bayesian inference machinery through controlled comparisons.

  3. The Bayesian treatment is incomplete in an important component of the proposed framework. Although Section 3.2 introduces a variational distribution over the neural model and physical parameters, the practical implementation treats the encoder and decoder parameters as point estimates. This approximation is particularly consequential in the proposed setting because the decoder Jacobian determines the projection operator and the directions constrained by the embedded physical knowledge. Consequently, uncertainty in the learned representation can directly affect both the physical correction and the predicted dynamics. The predictive covariance decomposition in Eq. (17) includes an encoder–decoder uncertainty contribution, but this term is absent in the implemented point-estimate formulation. The authors acknowledge this limitation and include an ablation in Appendix B.6 showing instability when Bayesian encoder–decoder inference is introduced. Nevertheless, this raises an important question about how the proposed uncertainty characterization should be interpreted in practice. More discussion is needed to clarify which sources of epistemic uncertainty are captured, whether the reported calibration is sensitive to decoder uncertainty, and what guarantees remain when uncertainty in the projection geometry is ignored.

Questions for authors
  • Questions for the Authors
  1. Mechanism of robustness: Since latent projection still enforces the potentially misspecified physical relation exactly in the decoded dynamics, could the authors explain more precisely how null-space freedom mitigates structural misspecification? Is it possible to quantify the residual prediction bias induced by an incorrect constraint?

  2. Role of latent dimensionality: How does performance under misspecified physics change as the latent dimension increases? In particular, does robustness correlate with the dimension of the decoder Jacobian's null space? A systematic sweep over latent dimensions would help substantiate the main explanation.

  3. Uncertainty in the projection geometry: Since the encoder and decoder are point-estimated in the implemented method, how sensitive are predictive uncertainty and physical-parameter inference to the learned decoder? Is there a practical approximation that captures decoder uncertainty without the instability reported in Appendix B.6?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer rj34

1. Mechanism of robustness, and the role of latent dimension (Weakness 1, Questions 1–2)

"[...] the proposition does not directly establish that preserving these directions leads to improved robustness under misspecified physics. [...] it would be useful to distinguish the benefits of the additional latent degrees of freedom from those arising from the projection formulation itself." "Is it possible to quantify the residual prediction bias induced by an incorrect constraint?" "How does performance under misspecified physics change as the latent dimension increases? [...] A systematic sweep over latent dimensions would help."

Our paper presents casting the dynamics within the latent space leading to greater robustness as a hypothesis. The proposition just states that the projection of the vector field within the state and latent space is tends to equivalence as λ→0\lambda \to 0. When we then numerically integrate the vector field in the latent space, this projected vector field has nh−nxn_h - n_x additional free latent dimensions that are unconstrained via the projection; whereas the state space does not have these extra dimensions. As a result we hypothesise using the latent state leads to the greater robustness, and this is shown empirically in the results.

We understand the reviewers concerns with how the performance is affected with increasing latent dimension. We have added additional ablations, where we have systematic misspecification with learned physical parameters Λ\Lambda, across different latent space dimension, nhn_h. The tables show the MSE, CRPS and WIS on held out test set

Systemnh=nxn_h = n_x (state-space projection)nh=8n_h = 8nh=32n_h = 32nh=64n_h = 64
Duffing0.062 ± 0.012 / 0.8 ± 0.210.066 ± 0.013 / 0.9 ± 0.130.052 ± 0.006 / 0.79 ± 0.0650.053 ± 0.011 / 0.78 ± 0.054
Forced oscillatorall seeds diverged3 ± 2.2 / 0.35 ± 0.0673.9 ± 3.8 / 0.45 ± 0.222.9 ± 1.3 / 0.33 ± 0.0026
Lotka–Volterra1.3e+03 ± 38 / 1.1e+02 ± 210.066 ± 0.022 / 0.8 ± 0.0530.048 ± 0.0069 / 0.74 ± 0.0430.046 ± 0.013 / 0.65 ± 0.082
Pendulum0.065 ± 0.027 / 1.4 ± 0.40.014 ± 0.01 / 0.51 ± 0.320.027 ± 0.009 / 0.72 ± 0.0220.021 ± 0.003 / 0.74 ± 0.027
Van der Pol1.8 ± 0.23 / 0.81 ± 211.3 ± 0.21 / 0.57 ± 0.121.6 ± 0.74 / 0.62 ± 0.221.8 ± 0.79 / 0.66 ± 0.19
Chained pendulum0.38 ± 0.16 / 2.2 ± 0.220.03 ± 0.0074 / 0.8 ± 0.0210.034 ± 0.0062 / 0.79 ± 0.0180.034 ± 0.0054 / 0.81 ± 0.029
System (observation WIS, normalised)nh=nxn_h = n_x (state-space projection)nh=8n_h = 8nh=32n_h = 32nh=64n_h = 64
Duffing0.68 ± 0.0740.53 ± 0.0690.47 ± 0.0480.46 ± 0.054
Forced oscillatorall seeds diverged0.22 ± 0.0330.27 ± 0.120.2 ± 0.0019
Lotka–Volterra79 ± 150.5 ± 0.0490.51 ± 0.0220.4 ± 0.044
Pendulum0.98 ± 0.170.38 ± 0.180.5 ± 0.0340.5 ± 0.045
Van der Pol0.47 ± 0.0840.32 ± 0.0490.34 ± 0.110.37 ± 0.092
Chained pendulum1.2 ± 0.210.49 ± 0.0130.51 ± 0.0160.51 ± 0.019

On all six systems, adding the latent space leads to improved performance

An incorrect relation introduces an error Δg=gwrong−gtrue\Delta g = g_{\text{wrong}} - g_{\text{true}} in the constrained vector field. At any instant this error is the same for latent and state-space projection: by Proposition 1, both pass it into the decoded dynamics unchanged, so its size is bounded by ∣Δg∣|\Delta g|. The difference in approach is how much of this bias carries through to the prediction. In state space the error acts directly on the state at every step and accumulates along the trajectory. In latent space, the extra free directions and the learned decoder can re-route the trajectory, so the error need not accumulate.

In each trained model we replace the incorrect relation with the true one, keeping the weights and posterior noise fixed, and record how far the prediction moves. A large change means the trajectory was being driven by the incorrect relation; a small change means it is carried by the learned dynamics. Swapping in the true relation generally moves the latent prediction by a much smaller magnitude, compared to state-space projection. The instantaneous error along the latent trajectories is also consistently smaller, showing that the latent model settles more where the wrong and true relations nearly agree.

AuthorsOpenReview ↗

Rebuttal to Reviewer rj34

SystemMethodInstantaneous bias ∥bt∥\|b_t\|AmplificationSwap to true relationTotal error
DuffingLatent (ours)0.0360.98110.29
DuffingState-space0.131.11.1e+030.3
Forced oscillatorLatent (ours)0.762.31.51.5
Forced oscillatorState-spaceall seeds diverged–––
Lotka–VolterraLatent (ours)0.0370.370.140.28
Lotka–VolterraState-space9.16.34243
PendulumLatent (ours)0.050.820.130.21
PendulumState-space0.240.562.30.31
Van der PolLatent (ours)0.510.819.21.7
Van der PolState-space0.420.227.51.7
Chained pendulumLatent (ours)0.0560.970.630.42
Chained pendulumState-space1.30.19851.5

2. Novelty relative to existing components (Weakness 2)

"[...] which components are fundamentally new and which are adaptations of existing techniques [...] isolate the contribution of latent projection from that of the Bayesian inference machinery through controlled comparisons."

Existing hard-constraint methods for neural ODEs enforce relations in the observed state space: projected and stabilized NDEs [6, 13], neural DAEs [8, 25] and FINDE [7]. The latent approach of Deep Conservation [23] embeds global conservation laws in a low dimensional latent space. Our construction allows us to enforces partially known relations on a subset of variables inside a learned latent vector field. This allows for more flexible embedding of physics for forced, dissipative systems, which typically do not have invariants. By combining our new latent embedding and Bayesian inference over both model and physics parameters, our approach remains more robust when the embedded physics is incorrect.

"In particular, the distinction from previous approaches to latent conservation laws, projected neural differential equations, and partially stochastic Bayesian neural networks should be made more explicit. I would also encourage the authors to isolate the contribution of latent projection from that of the Bayesian inference machinery through controlled comparisons."

Existing latent approaches only embed global knowledge, whereas approaches that embed partial knowledge, such as projected neural differential equations embed physics in the state space, which can be brittle under misspecification. Our approach instead embeds partial physical knowledge, within a learned latent space. Our latent formulation outperforms the existing state-space projection appraoches in both the nominal case, and when the known physical laws are misspecified. The Partially stochastic extension of infinitely deep Bayesian neural networks is our chosen method of uncertainty quantification over the model predictions.

The comparison of projection with and without projection is already in place. All baselines use an SDE-based posterior and Gaussian likelihood, so the inference approach is fixed and we vary how the constraint is enforced -- i.e. none / soft / state-space / latent, as well as allowing for learned and fixed constraint parameters.

3. Point-estimated encoder and decoder (Weakness 3, Question 3)

"[...] which sources of epistemic uncertainty are captured, whether the reported calibration is sensitive to decoder uncertainty, and what guarantees remain when uncertainty in the projection geometry is ignored." "Is there a practical approximation that captures decoder uncertainty without the instability reported in Appendix B.6?"

This approximation is stated in the Limitations. It follows the standard partially Bayesian NDE practice [18].

Our posterior inference quantifies aleatoric uncertainty through likelihood term, whilst epistemic uncertainty is propagated via the posterior through θt\theta_t and the physics-parameter term (qη(Λ)q_\eta(\Lambda)), but we do not practically do inference with the encoder and decoder parameters.

JϕJ_\phi is evaluated at the point-estimated decoder, but uncertainty in θt\theta_t and Λ\Lambda still propagates through the projection. Proposition 1 holds for any fixed decoder, so it does not depend on decoder uncertainty.

We tested ablation directly, performing inference on the encoder and decoder as well makes training unstable. With point estimates, 95% coverage remains reasonable at 0.89–0.98 across systems. Post-hoc options that do not impact the training, such as a Laplace approximation of the decoder or ensembles, could quantify this uncertainty, with a reduced risk of instability. We are happy to acknowledge this and add it to future work.

Reviewer_iVo9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper incorporates physical relations into latent neural differential equations through a weighted projection. It also estimates uncertainty in the neural dynamics and physical parameters using Bayesian inference. Experiments on six synthetic systems examine prediction accuracy and uncertainty under noisy observations, missing measurements, and incorrect physics.

Critical strengths
  1. The problem is meaningful. Physical knowledge is often incomplete or uncertain, so understanding how to use it without hurting prediction is useful. The projection formulation and its connection to state-space projection are clearly presented.

  2. The numerical analysis covers several relevant conditions, including noisy measurements, partial observations, and incorrect physics. Results across six systems and five random seeds provide useful evidence of the method’s performance.

Critical weaknesses
  1. The technical innovation needs stronger justification. Embedding physics into neural models is already well studied. The contribution should therefore be clearer than combining physical projection, latent dynamics, and Bayesian inference. What does this particular construction enable that existing methods cannot? Also, what physical knowledge is needed for it to work? The experiments specify relations for each system but do not systematically vary their number or choice. They show that selected relations help, but provide limited guidance on which physics to embed and when it is sufficient.

  2. The explanation of robustness to incorrect physics is not fully established. Proposition 1 shows that the decoded dynamics still satisfy the imposed relation exactly. If that relation is wrong, moving the projection to latent space does not remove the error in the constrained dynamics. The free latent directions may affect future predictions, but the current analysis does not establish when this reduces prediction error. Controlled comparisons should separate this effect from additional model capacity and numerical stability. This is particularly important because several state-space baselines diverge, and the default evaluation uses only three test trajectories per system.

Questions for authors
  1. What is the main technical advance over existing latent physics-constrained models? Which benefit specifically requires the proposed projection and Bayesian formulation?

  2. How does performance change with the number and choice of physical relations? Can you identify when adding physics helps or hurts?

  3. When an incorrect relation is enforced exactly, how do the free latent directions reduce prediction error? Can you demonstrate this while controlling latent dimension and model capacity?

  4. Do the large improvements over state-space projection remain with smaller integration steps or an adaptive solver? Please also clarify why the main text reports no decoder degeneracy while Appendix B.6 discusses rank-deficient decoder Jacobians.

4 comments
AuthorsOpenReview ↗

Rebuttal to reviewer iVo9

1. Technical advance, and which physics to embed (Weakness 1, Questions 1–2)

"What does this particular construction enable that existing methods cannot? Also, what physical knowledge is needed for it to work? The experiments [...] do not systematically vary their number or choice." "What is the main technical advance over existing latent physics-constrained models? Which benefit specifically requires the proposed projection and Bayesian formulation?" "How does performance change with the number and choice of physical relations? Can you identify when adding physics helps or hurts?"

Existing hard-constraint methods for neural ODEs enforce relations in the observed state space: projected and stabilized NDEs [6, 13], neural DAEs [8, 25] and FINDE [7]. The latent approach of Deep Conservation [23] embeds global conservation laws in a low dimensional latent space. Our construction enforces partially known relations on a subset of observed variables inside a learned latent vector field. This allows for more flexible embedding of physics for forced, dissipative systems, which typically do not have invariants. We also find that within our latent embedding and combined with Bayesian inference over both model and physics parameters, our approach is more robust when the embedded physics is incorrect.

We embed physical knowledge that is only defined over a subset of variables, rather than global approaches. Existing projected neural differential equations only embed physics in the state space, which can brittle under misspecification. Our latent formulation outperforms this in both the nominal case, and when the known physical laws are misspecified. The Partially stochastic extension of infinitely deep Bayesian neural networks is our chosen method of uncertainty quantification over the model predictions.

The comparison of projection with and without projection is already in place. All baselines use an SDE-based posterior and Gaussian likelihood, so the inference machinery is fixed and we vary how the constraint is enforced - -i.e. none / soft / state-space / latent, as well as allowing for learned and fixed constraint parameters.

We have added an ablation that varies the number of physical relations embedded into a model. Firstly, we add a dynamic chemical reactor, as an additional case study, comprising 6 states and 3 inputs, of which we can embed 3 separate species molar balances. We then use this and the high pendulum case study, to embed different numbers of constraints. For the pendulum, comparing our latent approach and the state space projection, we report the held out test MSE / CRPS below:

Embedded relationsLatent projection (ours)State-space projection
1 kinematic0.031 ± 0.0052 / 0.74 ± 0.0121.1 ± 0.44 / 5.5 ± 1.7
2 kinematic0.03 ± 0.0053 / 0.73 ± 0.0181 ± 0.39 / 3.8 ± 1.5
3 kinematic0.027 ± 0.0024 / 0.67 ± 0.0350.16 ± 0.21 / 1.9 ± 0.9
1 dynamic0.017 ± 0.0015 / 0.57 ± 0.0170.81 ± 0.32 / 4.2 ± 0.61
2 dynamic0.017 ± 0.0026 / 0.52 ± 0.0490.62 ± 0.46 / 3.2 ± 1.3
3 dynamic0.015 ± 0.0011 / 0.46 ± 0.0220.49 ± 0.0089 / 3 ± 0.13
3 kinematic + 1 dynamic0.014 ± 0.0017 / 0.53 ± 0.0120.29 ± 0.34 / 2 ± 1.4
3 kinematic + 2 dynamic0.011 ± 0.0014 / 0.45 ± 0.0130.032 ± 0.013 / 0.65 ± 0.12
Misspecified kinematic (3 wrong)0.031 ± 0.0056 / 0.75 ± 0.016all seeds diverged
Misspecified kinematic (1 of 3 wrong)0.032 ± 0.0073 / 0.74 ± 0.013all seeds diverged
Misspecified kinematic (2 of 3 wrong)0.031 ± 0.0052 / 0.74 ± 0.014all seeds diverged
Misspecified dynamic0.014 ± 0.0044 / 0.45 ± 0.0280.49 ± 0.054 / 3 ± 0.069
Embedded relations (observation WIS, normalised)Latent projection (ours)State-space projection
1 kinematic0.51 ± 0.00462.9 ± 0.91
2 kinematic0.51 ± 0.00972.4 ± 0.81
3 kinematic0.49 ± 0.0221.8 ± 0.64
1 dynamic0.43 ± 0.0132.3 ± 0.25
2 dynamic0.42 ± 0.0472.1 ± 0.41
3 dynamic0.39 ± 0.021.9 ± 0.13
3 kinematic + 1 dynamic0.4 ± 0.0121.7 ± 0.26
3 kinematic + 2 dynamic0.37 ± 0.0311.5 ± 0.15
Misspecified kinematic (3 wrong)0.51 ± 0.002all seeds diverged
Misspecified kinematic (1 of 3 wrong)0.52 ± 0.0037all seeds diverged
Misspecified kinematic (2 of 3 wrong)0.51 ± 0.0063all seeds diverged
Misspecified dynamic0.38 ± 0.00451.9 ± 0.16
AuthorsOpenReview ↗

Rebuttal to iVo9 Continued

Then for the CSTR reactor

ReactorLatent projection (ours)State-space projection
1 species balance0.13 ± 0.017 / 0.72 ± 0.0490.14 ± 0.11 / 0.85 ± 0.34
2 species balances0.12 ± 0.019 / 0.65 ± 0.0260.23 ± 0.35 / 0.93 ± 1.1
Reactor (observation WIS, normalised)Latent projection (ours)State-space projection
1 species balance0.45 ± 0.0280.68 ± 0.16
2 species balances0.4 ± 0.0140.73 ± 0.45

As expected, embedding more correct physical relations makes the models better. Although embedding structurally misspecified physics reduces performance, but this sweep reinforces that the latent approach remains more robust, and degrades more gracefully than state space, remaining consistent with the other ablations in the paper.

AuthorsOpenReview ↗

Rebuttal to Reviewer iVo9

2. Robustness to incorrect physics, capacity, stability and evaluation size (Weakness 2, Question 3)

"If that relation is wrong, moving the projection to latent space does not remove the error in the constrained dynamics. [...] Controlled comparisons should separate this effect from additional model capacity and numerical stability. [...] the default evaluation uses only three test trajectories per system." "When an incorrect relation is enforced exactly, how do the free latent directions reduce prediction error? Can you demonstrate this while controlling latent dimension and model capacity?"

Our paper presents casting the dynamics within the latent space leading to greater robustness as a hypothesis. The proposition just states that the projection of the vector field within the state and latent space is equivalent. When we then numerically integrate the vector field in the latent space, this projected vector field has additionally nh−nxn_h - n_x free latent dimensions that are unconstrained via the projection; whereas the state space does not have these extra dimensions. As a result we hypothesise using the latent state leads to the greater robustness, and this is shown empirically in the results.

We understand the reviewers concerns with how the performance is affected with increasing latent dimension. We have added additional ablations, where we have systematic misspecification with learned physical parameters Λ\Lambda, across different latent space dimension, nhn_h. nh=nxn_h = n_x leaves no null space, so it isolates the projection formulation from the extra degrees of freedom. The table reports held-out MSE and CRPS

Systemnh=nxn_h = n_x (state-space projection)nh=8n_h = 8nh=32n_h = 32nh=64n_h = 64
Duffing0.062 ± 0.012 / 0.8 ± 0.210.066 ± 0.013 / 0.9 ± 0.130.052 ± 0.006 / 0.79 ± 0.0650.053 ± 0.011 / 0.78 ± 0.054
Forced oscillatorall seeds diverged3 ± 2.2 / 0.35 ± 0.0673.9 ± 3.8 / 0.45 ± 0.222.9 ± 1.3 / 0.33 ± 0.0026
Lotka–Volterra1.3e+03 ± 38 / 1.1e+02 ± 210.066 ± 0.022 / 0.8 ± 0.0530.048 ± 0.0069 / 0.74 ± 0.0430.046 ± 0.013 / 0.65 ± 0.082
Pendulum0.065 ± 0.027 / 1.4 ± 0.40.014 ± 0.01 / 0.51 ± 0.320.027 ± 0.009 / 0.72 ± 0.0220.021 ± 0.003 / 0.74 ± 0.027
Van der Pol1.8 ± 0.23 / 0.81 ± 211.3 ± 0.21 / 0.57 ± 0.121.6 ± 0.74 / 0.62 ± 0.221.8 ± 0.79 / 0.66 ± 0.19
Chained pendulum0.38 ± 0.16 / 2.2 ± 0.220.03 ± 0.0074 / 0.8 ± 0.0210.034 ± 0.0062 / 0.79 ± 0.0180.034 ± 0.0054 / 0.81 ± 0.029
System (observation WIS, normalised)nh=nxn_h = n_x (state-space projection)nh=8n_h = 8nh=32n_h = 32nh=64n_h = 64
Duffing0.68 ± 0.0740.53 ± 0.0690.47 ± 0.0480.46 ± 0.054
Forced oscillatorall seeds diverged0.22 ± 0.0330.27 ± 0.120.2 ± 0.0019
Lotka–Volterra79 ± 150.5 ± 0.0490.51 ± 0.0220.4 ± 0.044
Pendulum0.98 ± 0.170.38 ± 0.180.5 ± 0.0340.5 ± 0.045
Van der Pol0.47 ± 0.0840.32 ± 0.0490.34 ± 0.110.37 ± 0.092
Chained pendulum1.2 ± 0.210.49 ± 0.0130.51 ± 0.0160.51 ± 0.019

On all six systems, adding the latent space leads to improved performance under misspecification. The additional dimensions are able to absorb this misspecified error more robust compared to state space approaches.

In regards to training with alternative ODE solver parameters, we also added a further two ablations that ran the latent and state-space projection with half the step size, as well as with an adaptive PID step-size controller, which made training and inference much longer (rtol 3×10−23\times10^{-2}, atol 10−210^{-2}). Tighter tolerances hit the step budget. The table reports the MSE and CRPS of each approach over the held out test trajectories.

SystemSolverOursState-space projection
Duffingdefault0.00098 / 0.0920.018 / 0.39
Duffinghalf step0.0014 / 0.10.017 / 0.34
Duffingadaptive0.0021 / 0.110.023 / 0.39
Pendulumdefault0.0049 / 0.290.015 / 0.55
Pendulumhalf step0.00096 05 / 0.120.015 / 0.57
Pendulumadaptive0.00098 / 0.130.057 / 1.2
Van der Poldefault0.23 / 0.160.46 / 0.3
Van der Polhalf step0.2 / 0.140.54 / 0.31
Van der Poladaptive0.17 / 0.140.7 / 0.35
Forced oscillatordefault0.18 / 0.15.7 / 0.83
Forced oscillatorhalf step0.21 / 0.0893.7 / 0.57
Forced oscillatoradaptive0.29 / 0.0890.31 / 0.093
AuthorsOpenReview ↗

Rebuttal to reviewer iVo9

System (observation WIS, normalised)SolverOursState-space projection
Duffingdefault0.140.49
Duffinghalf step0.140.45
Duffingadaptive0.150.41
Pendulumdefault0.310.65
Pendulumhalf step0.20.65
Pendulumadaptive0.20.8
Van der Poldefault0.120.24
Van der Polhalf step0.120.25
Van der Poladaptive0.110.29
Forced oscillatordefault0.0950.49
Forced oscillatorhalf step0.0780.36
Forced oscillatoradaptive0.0750.088

Smaller step sizes and adaptive controllers do not seem to noticeably improve state-space projection performance, relative to the latent space. Note: we were unable to run these ablations over all case studies due to computational resource limitations.

3. Decoder degeneracy (Question 4, second part)

"Please also clarify why the main text reports no decoder degeneracy while Appendix B.6 discusses rank-deficient decoder Jacobians."

We thank the reviewer for spotting this. These are two different objects:

  • The main text states that as Qt=Jϕ⊤JϕQ_t = J_\phi^\top J_\phi is nh×nhn_h \times n_h with rank at most nx<nhn_x < n_h, hence is deficient. That is why Tikhonov regularization (or a pseudoinverse) is needed at all, and it is what B.6 ablates (Table 45).
  • JϕJ_\phi however has full row rank, as a result well-posedness condition in Sec. 5

Instead, "where JϕJ_\phi is rank-deficient" in B.6 is related to the Q metric instead ("QtQ_t"), and will correct it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.