acceptodds
Under review as a conference paper at ICLR 2027

Identifying ODEs from Unstructured Data with Causal Representation Learning

Abstract

We study the problem of recovering the governing ODE of a dynamical system from unstructured, high-dimensional observations such as images. Existing methods for ODE discovery typically assume direct measurements of the variables, or do not provide theoretical guarantees on the learned variables and equations. While Causal Representation Learning (CRL) methods provide guarantees on identifying variables from high-dimensional observations up to component-wise diffeomorphisms, we show that in general these variables cannot be used directly as input to equation discovery methods, which typically assume that the variables will lead to sparse equations. So we introduce SParse Equivalent Equation Discovery AutoEncoder (SPEED-AE), a framework that combines a pretrained CRL method with a component-wise autoencoder that learns transformations of variables that are amenable to sparse ODE discovery. We show that for polynomial ODEs, this additional step allows us to restrict the identifiability of each variable from polynomial to monomial diffeomorphisms. Experiments on Lotka-Volterra, Lorenz, and a two-pendulum system show that SPEED-AE improves on the disentanglement of the CRL methods and that it recovers ODEs that are closest to the ground truth, while achieving state-of-the-art forecasting performance.

Primary area:
learning on time series and dynamical systems
Keywords:
dynamical systems, ordinary differential equations, equation discovery, causal representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_P95HOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies recovering the governing ODE of a system from high-dimensional observations such as images. CRL methods identify latents only up to permutation and component-wise diffeomorphism. The authors argue that such distortions destroy the sparsity that equation discovery e.g., SINDy needs. They propose SPEED-AE, which is a frozen, pretrained CRL encoder followed by component-wise autoencoders trained jointly with a sparse ODE objective (SINDy or MNN). The paper states that for polynomial ODEs and polynomial diffeomorphisms, sparsity restricts the component-wise maps to monomials. The experiments on Lotka-Volterra, Lorenz, and two pendulums show better disentanglement and recovered equations closer to the ground truth than SINDyAE, MNNAE, and CRL latents fed directly to SINDy/MNN.

Critical strengths

S1: Even with perfect disentanglement, the factor [hi′(z^i)]−1[h_i'(\hat z_i)]^{-1} in the conjugate ODE can destroy sparsity. The point is simple, new, and well illustrated.

S2: It is shown that component-wise autoencoders preserve the CRL disentanglement, and that the pipeline works with different CRL and equation-discovery back-ends.

Critical weaknesses

W1: SPEED-AE's CRL stage is pretrained on a separate dataset with interventions and known intervention targets. SINDyAE and MNNAE appear to be trained only on standard trajectories and are described as learning latents and equations "without additional information". Poor disentanglement of the baselines is therefore partly expected, and the gap in CorrD and ODE recovery says little about the equation-discovery objective. The paper discloses the interventional pretraining but never discusses this asymmetry. Baselines with access to the same interventional data are missing.

W2: Lemma 3.1 assumes polynomial diffeomorphisms and two non-cancellation assumptions. CRL guarantees only a general diffeomorphism, and polynomial encoders and decoders were proven unstable by the authors even in the oracle setting. The final model therefore uses MLPs, which is a reasonable next approach, but the identifiability result does not apply to what is evaluated. The theoretical analysis does not apply to the implementation.

The theory also does not explain the observed gains. For Lotka-Volterra, f1=z1(α−βz2)f_1 = z_1(\alpha-\beta z_2), so q1(z1)=z1q_1(z_1)=z_1 is non-constant, and as I read Lemma 3.1 any monomial map z=az~pz=a\tilde z^{p} is allowed, with the same number of terms for every pp. The sparsity objective therefore does not favor p=1p=1. The experiments nevertheless show near-affine maps (Figs. 10, 11), and the evaluation protocol rewards them. The observed behavior is not predicted by the lemma and may come from the MLPs' inductive bias or from the interventional CRL stage and no ablation tests whether sparsity is the cause. The ablations vary the CRL stage or drop the autoencoder, but none removes or varies the L1L_1 penalty or tests a case where the lemma predicts a non-affine monomial. As a result, the link between the theory and the empirical advantage is not shown.

W3: The proof of Lemma 3.1 argues about real roots ζ\zeta of hi′h_i'. The domain of hih_i is never stated. For a polynomial diffeomorphism of all of R\mathbb{R} (e.g., h(u)=u3+uh(u)=u^3+u), h′h' has no real roots, so the argument would need to be made over C\mathbb{C}. The authors' own counterexample uses a diffeomorphism "restricted to an appropriate domain", where h′h' has real roots outside it. Likewise, the conclusion zi=aiz~piz_i=a_i\tilde z^{p_i} with pi>1p_i>1 is a diffeomorphism only on restricted domains (e.g., positive values). Please state the domain assumptions explicitly and restate the argument accordingly.

W4: From my understanding applying SPEED-AE to a new system from images requires (i) the true number of latents, which is set to dd in the CRL models and never estimated, (ii) the ability to intervene on every latent, with known intervention targets, via arbitrary state resets, and (iii) a CRL stage that actually disentangles, since the 1D autoencoders cannot repair cross-latent mixing. The authors explain why they need interventions, but this is equal to knowing the system's structure and being able to manipulate each of its degrees of freedom, which excludes passively observed systems. The paper describes the method as "agnostic" to the CRL method, yet both CRL methods tested need interventional data and the framing as discovery from "unstructured data such as images" is therefore misleading and does not reflect what the method supports.

Questions for authors

Q1: How do SINDyAE and MNNAE perform when trained with the same interventional data (and targets) used for CITRIS/DMSVAE? Q2: What domain do you assume for hih_i in Lemma 3.1, and does the divisibility argument extend to complex roots of hi′h_i'? Q3: How do you justify "state-of-the-art forecasting" given that in Table 6 LSTM, Transformer, and MNNAE are better in several settings? Q4: How would SPEED-AE be applied to a system where the number of latents is unknown or interventions are unavailable? What happens if dd is over- or under-specified in the CRL stage? Q5: What happens to the learned maps and recovered equations when the L1L_1 penalty is removed or varied, and in a system where Lemma 3.1 predicts a non-affine monomial map? Without this, how do you attribute the gains to sparsity?

3 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for your careful review. We updated the manuscript, highlighting the main changes in blue. We address your concerns one by one in the following.

W1, Q1 and W4 (Interventional data). There is a misunderstanding here, we do not require interventional data. SPEED-AE is CRL-agnostic, and several non-interventional CRL methods exist, which can also be used in our framework (among the ones we cite, for example, Kügelgen et al., 2021; Xu et al., 2024; Yao et al., 2023; and several others). In particular, DMSVAE is not a method that requires known interventions, but it can leverage either actions without known targets or, in the absence of actions, the sparsity in the transition functions. In our experiments, we wanted to use the same data for DMSVAE and CITRIS (which requires known intervention targets), so we used interventional data, but we did not provide the intervention targets to DMSVAE (since the method does not use them). In the revised manuscript, we added an ablation on the Pendulum experiment (App. G.6), where we use SPEED-AE with DMSVAE and SINDy on only observational data, showing that our method provides similar results also in this setting.

We also disagree that no baselines use the same data. The naive combinations of CRL and equation discovery, which we report as our main baselines, use exactly the same interventionally pretrained CRL models as SPEED-AE. So the improvement of SPEED-AE over these baselines is therefore not explained by the interventional data, and is due to the component-wise autoencoders and the sparse ODE objective.

Finally, there is no meaningful way to integrate interventional data into SINDyAE and MNNAE, as they focus on how well the learned ODE fits the data, not on the structure. Moreover, these methods might perform even worse if we add interventional data, since they see them as a mixture of distributions. We show an ablation in App. G.5 on Lotka-Volterra, where we train SINDyAE on the same data as SPEED-AE, including interventions, showing that there is no improvement in any disentanglement metrics. Furthermore, the trajectories predicted are very unstable, and most of them on most test seeds diverge. We cannot even run this ablation for MNN, since its mechanistic core does not fit the derivative at each time step, but instead predicts subsequences of the trajectories in batches. Therefore, these subsequences might have jumps in them due to interventions, and no continuous ODE can fit them accurately.

W2 (Theory for polynomials) First, we wanted to clarify a small point: while our new identifiability results are indeed limited to polynomials, the SPEED-AE framework can be applied to other settings, while still providing theoretical guarantees on the latents (which is in itself novel). In this case, we get the identifiability results inherited by the specific CRL method, which is typically identifiability up to permutations and element-wise diffeomorphisms. So although our specific identifiability results might not apply to the general case, SPEED-AE as a framework is still theoretically valid also in the general case, which is our main contribution, and this is also what we aim to show with our empirical validation. Just for completeness, we did already report on experiments with polynomial encoders/decoders in the original submission (now in App. G.2). The results were not as stable as with general encoders/decoders, also because, as you mention, most CRL methods, including the ones we use, cannot guarantee that the learned variables are polynomial transformations of the true ones, even with polynomial encoders/decoders.

W2 and Q5 (L1-Penalty). We agree that in practice SPEED-AE often recovers approximately affine transformations in general cases, while the theory can only guarantee monomials up to shift and only for the polynomial case. We hypothesize that this is due to higher-order monomials being much more affected by small errors in the learned variables (so they overfit on the noise) than lower-order monomials, leading the model to prefer simpler monomials, but a rigorous study of these effects is out of scope for this paper. This effect is also not due to the CRL stage, since the learned transformations for the naive CRL+equation discovery baselines are not affine.

In order to consider the effect of sparsity in isolation, we run an additional ablation, where we do not enforce sparsity and report the results in App. G.3. Removing sparsity affects both sequence prediction and, more importantly, the identification of the ground-truth ODE. Therefore, as proved in Theorem 3.2 (the new name for the old Lemma 3.1), sparsity is fundamental to reducing the class of possible diffeomorphisms between recovered and ground-truth variables.

AuthorsOpenReview ↗

Response (2/2)

W3 and Q2 (Complex Roots and diffeomorphisms domain). Thank you for identifying relevant aspects of the Lemma that need further clarification. In the revised manuscript, we now specify the domain of the polynomial diffeomorphisms. In particular, we consider a non-empty real interval UiU_i on which hih_i is a diffeomorphism onto its image Zi\mathcal{Z}_i. We also add more details on the domain and image of such diffeomorphisms throughout the proofs. With regard to real or complex roots, the proof of the new Lemma 3.1 now considers the roots of h’ih’_i over C\mathbb{C}, which covers diffeomorphisms such as the one reported by the reviewer. However, even in the revised manuscript, all of the polynomials are defined over real coefficients, while we prove all the results by allowing complex roots and recovering the real case as a consequence. Thus, the final statements still relate to polynomials with real coefficients.

W4 and Q4 (Interventions). As already mentioned in the answer to W1, we are agnostic to CRL and do not require interventions (and one of the methods we use, DMSVAE, actually already works without interventions under assumptions on the sparsity in the transitions). Regardless of using interventions, the inputs are still images, both in CRL and for our framework, and having interventions does not equate to knowing the system structure (otherwise there would be no need for CRL in general).

W4 and Q4 (Number of latents). We remark that the baselines SINDy-AE and MNNAE also receive the ground-truth number of variables dd as input; hence, the empirical comparison is fair. Moreover, while the theory of CRL methods often assumes a known number of latents, most methods empirically work quite well as long as this number is overestimated, ultimately not learning any information in the excess dimensions, as shown, for example, by [Lippe et al. 2022]. This is the same for SPEED-AE, since the unused latents will not be used in the learned equations. We added an ablation in App. G.7 where CITRIS and SPEED-AE start from an overestimated number of latents, and the extra latents do not capture any of the true ones. We show that SPEED-AE with CITRIS and SINDy is able to recover the ODEs after a simple post-processing that identifies and removes from the equations the latent variables that are constant.

W4 and Q4 (Disentanglement). We agree that SPEED-AE relies on a CRL stage to disentangle the latents and cannot correct cross-latent mixing. We discuss these limitations, inherited by the CRL backends, in the conclusions of our manuscript.

Q3 (Forecasting). We rephrased “state-of-the-art” into “competitive forecasting performance”. As reported in Table 7 (App. H), unstructured models such as LSTM and Transformer perform well on the short Lorenz trajectories, but their errors on the longer Lotka-Volterra trajectories are up to an order of magnitude higher than those of SPEED-AE. SPEED-AE consistently outperforms SINDyAE, while MNNAE achieves lower forecasting errors in three settings, but fails on chaotic Lorenz. As discussed in Sec. 6, the forecasting performance of MNNAE comes at the cost of non-identifiable latents and poor ODE recovery. Our main goal is the identification of the latents and their governing equations, and SPEED-AE is the only method among the compared ones that achieves this while maintaining competitive forecasting.

Thank you for the revisions. The new ablations address several of my other concerns but the corrected proof of Lemma 3.1 only partly resolves W3. It's a 2.5 page proof and I didn't have the time to thoroughly check it but I do understand the idea of the proof. Theorem 3.2 seems to have another issue since it depends on a non-cancellation assumption on h itself, which Lorenz already violates under a simple shift. This means the theorem doesn't show that sparsity selects monomial maps. The theory still cannot explain the near-affine maps observed for Lotka–Volterra. I raise my score slightly.

Reviewer_ck8gOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SPEED-AE, a method that combines methods from the field of Causal Representation Learning (CRL) with the field of ODE discovery with the aim of discovering ODEs from high dimensional observations. The basic premise is quite elegant and simple. ODE discovery is considered a fruitful avenue for ML based scientific discovery, but insight from a discovered ODE is only really useful if the variables the ODE operates upon are semantically meaningful. Yet, fitting a simple ODE in a latent space does not guarantee learning a useful representation. In contrast, CRL methods learn semantically meaningful representations yet these are not guaranteed to admit simple ODE representations. SPEED-AE combines these two approaches, learning semantically meaningful variables using off-the-shelf CRL approaches, and then fitting a component-wise autoencoder onto this representation to learn a simple ODE in the latent space whilst preserving semantic meaning. The work comes with theoretical guarantees that shows their method can recover the ground truth ODE up to some indeterminacies given fairly restrictive structural assumptions, such as operating solely on polynomial ODEs.

Overall I feel very positively about this work. The motivation and the novelty is clear. The primary weaknesses come from overly restrictive assumptions on the data generative processes and interesting/unclear experimental results.

Note that I only skimmed the proof in the appendix, and did not find any soundness issues, but was not able to verify it in depth.

Critical strengths

CS1: The paper identifies and clearly motivates a real gap in the literature which serves to benefit both the fields of ODE system identification and causal representation learning.

CS2: The proposed method is simple and allows for easy integration of various CRL and ODE discovery method, allowing the implications of the work to grow as the fields grow.

CS3: In the same light, the method enables the refinement and further constraint of solutions found by CRL methods, thereby improving the identifiability guarantees by incorporating information about a systems dynamics. An interesting direction which should be explored further in the field.

CS4: The addition of a rigorous identifiability analysis makes it very clear when the theory is applicable and its limitations. As said before I was not able to verify it thoroughly.

Critical weaknesses

CW1: Experimental section and evaluation. The evaluation metrics used in the experiments are not intuitive and not entirely faithful to the arguments presented. First of all, the measure used for measuring disentanglement uses the pearson correlation, and therefore only measures affine disentanglement, whereas your theory permits monomial but not necessarily affine disentanglement. This is the reason why the field commonly uses the nonlinear variant of the MCC metric which either uses spearman correlation, or tried to fit the nonlinearity from learnt to ground truth using an MLP before using the MLP predictions for Pearson MCC to measure disentanglement [Song2024, Lippe2023, Lachapelle2023]. Furthermore, the zErr metric seems sensitive to the prediction errors encountered by an MLP , especially when an evaluation trajectory traverses into OOD areas. Therefore the xErr (image space) metric seems much more relevant and faithful as a metric, but it is not reported in the main paper apart from the pendulum experiment.

CW2: Limits of the theory. Requiring the encoder, decoder, and the ODE to be polynomial functions severly limits the applicability of the theory to more complicated or real world data. It is appreciated that more general theoretical analysis is challenging if not nigh on impossible, but to make the claim that the concepts hold in more general cases requires much more than just the pendulum experiment in my opinion. Furthermore, there needs to be a much stronger discussion on the implications of replacing the encoder and decoder with an MLP , which might help explain the strange results in figures 10 and 15 (see question section).

CW3: Weakness on the Theory Presentation front. Not critical, but definitely a major weakness. The theory is fairly difficult to follow in its current presentation format, and is missing a theorem block entirely. Ideally, a sketch proof would be presented which compartamentalises the theory into the lemmas, and explains intuitively how they combine to lead to the proof, withholding mathematical detail. Thereby allowing a reader to verify the function of each lemma more easily and understand the big picture. The sketch proof should culminate in a proof block (see for example Theorem 5 in Lachapelle et al., 2022b, Disentanglement via Mechanism Sparsity Regularisation: A New Principle for Nonlinear ICA).

Minor comments. These should be considered but do not affect the score:

  • Line 322 (Disentangelment, 410 (80/20/20 split), (a any) 476.
  • Text in figure 2 is too small to read when printed out.
  • There was a oral at CLeaR this year on CRL and dynamical systems that might be relevant related work, 'Disentangling Dynamical Systems: Causal Representation Learning Meets Local Sparse Attention, Baumgartner et al. 2026'.

Song2024 - Continual Learning of Nonlinear Independent Representations, Song et al., 2024

Lippe2023 - Causal Representation Learning from Binary Interactions, Lippe et al., 2023.

Lachapelle2023 - Additive Decoders for Latent Variable Identification and Cartesian Product Extrapolation, Lachepelle et al., 2023.

Questions for authors

Q1: Figure 2 and the xErr results show that SPEED-AE can fit the data much better than SINDyAE can. I find this surprising, why do you think this is?

Q2: Relatedly, in the SINDyAE baseline, you change the methods setting relative to the original paper by providing empirical derivatives to the image input as opposed to true ones. How much better (if at all) does the model perform when providing true derivatives instead?

Q3: Figures 10,15,16 appear to show the SPEEDY-AE method learn representations with nonlinear (and maybe non-monomial?) mappings to the ground truth variables. Is this a failure of the SPEEDY-AE method or expected given the theory?

Q4: How do you believe, and have you run experiments on, how the method scales to general more complex non-polynomial environments beyond the pendulum case?

3 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for the positive feedback. We updated the manuscript, highlighting the main changes in blue, including a reference to Baumgartner et al. (2026) in the related work. Similarly to Yao et al. (2024), they identify trajectory-specific parameters from directly observed states, while our work also recovers the governing equations. We address each of your other concerns in the following.

CW1 (MCC, R2R^2, xErr). We had used Pearson correlation, since empirically our results show approximately affine identifiability, but we have now revised all results to use MCC and R2R^2 using an MLP. In most experiments, we report as an aggregate measure the R2R^2 matrix discrepancy R2DR^2D, i.e., the average difference between the R2R^2 for the ground truth variables and the R2R^2 between the ground truth variables and the learned variables. This is a more fine-grained measure of disentanglement than MCC and a better aggregation than averaging the values of the R2R^2 matrix on the diagonal and off-the-diagonal, as in [Lippe et al. 2022] or [Lippe et al. 2023], which assume that the test data has independent variables. Since we allow latent variables to be dependent in the test data, the R2R^2 matrix might not be a diagonal matrix even with perfect disentanglement due to dependencies between ground truth variables; hence, we consider this discrepancy. We also report the actual R2R^2 matrices in Figure 12.

We revised all of our results, including the results in Figure 4 in the main paper, and show that our conclusions are unchanged. These additional results highlight how SPEED-AE reaches approximately perfect MCC (1.0) in all four considered settings and achieves a smaller R2DR^2D than the baselines, thus outperforming them in terms of disentanglement. We also added the trajectory error on x\mathbf{x} (xErr) to Figure 4 in the main paper (these results were already in App. H of the original submission), and report the complete results, including the LSTM and Transformer baselines, in Table 8, App. H.

CW2 (Limits of the theory). First, we wanted to clarify a small point: while our new identifiability results are indeed limited to polynomials, the SPEED-AE framework can be applied to other settings, while still providing theoretical guarantees on the latents (which is in itself novel). In this case, we get the identifiability results inherited by the specific CRL method, which is typically identifiability up to permutations and element-wise diffeomorphisms. So although our specific identifiability results might not apply to the general case, SPEED-AE as a framework is still theoretically valid also in the general case, which is our main contribution, and this is also what we aim to show with our empirical validation. Just for completeness, we did already report on experiments with polynomial encoders/decoders in the original submission in App. G.2. The results were not as stable as with general encoders/decoders, also because most CRL methods, including the ones we use, cannot guarantee that the learned variables are polynomial transformations of the true ones, even with polynomial encoders/decoders.

CW3 (Presentation). In the revised manuscript, we restructured Section 3: we introduce the decomposition fi=qirif_i = q_i r_i before the results, state the main result in a theorem block (the previous Lemma 3.1 is now divided into a lemma on divisibility, the new Lemma 3.1, and the result on sparsity, Theorem 3.2), and add proof sketches in the main paper.

AuthorsOpenReview ↗

Response (2/2)

Q1. SPEED-AE can fit and forecast data better than SINDyAE, because it has the right inductive bias encouraging disentangled representations that can be used to learn sparse equations based on the correct library of functions, while SINDyAE doesn’t, so its representations tend to be entangled (as can be seen in Fig. 2). Using sparse regression on entangled inputs (so heavily correlated/dependent inputs) is unstable in terms of their coefficients due to multicollinearity, while disentangled representations mitigate this issue, by reducing these dependences as much as possible to the ground truth ones. This can be seen clearly also in the error bars in Figure 4. Finally, estimating accurate equations in the latent space allows for better forecasting, even in image space, as we also show in App. H when we compare to LSTMs and Transformers.

Q2. We report the results of the additional experiments with SINDy-AE using true derivatives in App. G.4. The results show small or no improvement in absolute terms of SINDy-AE on all metrics, leading to the same conclusions as in the original submission.

Q3. We first clarify that in Figures 10, 15, and 16 (Figures 12, 13, 17, and 18 in the revised manuscript), most of the plots for SPEED-AE show affine disentanglement. Most of the nonlinear plots are for the baselines and the CRL latents before SPEED-AE. The few exceptions in which SPEED-AE shows nonlinearity are based on DMSVAE, which, in these specific cases, has poor disentanglement, which cannot be fixed by SPEED-AE, as we can expect based on the theory.

Q4. We believe our general framework can deal with more complex non-polynomial environments, given appropriate causal representation learning and equation discovery methods. We have performed some of these experiments in the paper; e.g., Section 6.2 already reports the two-pendulum experiment, a non-polynomial system where SPEED-AE uses a library with trigonometric terms. In general, equation discovery methods like SINDy can be applied to non-polynomial settings by providing an appropriate library of functions, e.g., with trigonometric functions. Scalability is limited by the CRL and equation discovery, which is currently often limited to a few tens of variables at most. We believe our current setting already matches the difficulty of current ODE benchmarks on observed system variables, e.g., ODEbench [1], which generally considers systems with at most three variables and mostly polynomial functions with a few trigonometric and rational functions.

[1] d’Ascoli et al., “ODEFormer: Symbolic Regression of Dynamical Systems with Transformers”, ICML 2024

The authors’ thorough response is appreciated.

All concerns raised in the initial review have been adequately addressed, and my overall assessment leans toward acceptance.

However, I will maintain my score remains at 3, as the principal limitation of the paper persists. The theoretical results rest on polynomial autoencoders in a restrictive setting, which the newly introduced assumptions make more restrictive still, and this setting does not correspond to what is evaluated experimentally. This mismatch limits how far the experiments corroborate the theory. It also limits the theory itself, since the results appear difficult to verify faithfully even in toy domains. However, the view expressed in the initial review stands, exploring the synergy between CRL and ODE discovery is an interesting and promising research direction.

Reviewer_RXkBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper works on the problem of inferring ODEs from high-dimensional observations where the underlying state variables are not directly observed. The authors first use causal representation learning to recover disentangled latent variables and then learn coordinate transformations to try to obtain sparser equations. The main theoretical contribution is an identifiability result that claims that, under certain assumptions, sparsity further restricts the possible coordinate transformations.

Critical strengths

S1. I think that the idea to use causal representation learning for equation discovery is interesting.

S2. The paper includes a relatively extensive empirical evaluation across different dynamical systems and the reported results are promising.

Critical weaknesses

W1. I think that the main theoretical result (Lemma 3.1) is incorrect as stated:

Consider the polynomial system

z˙1=(27z12+4)(1+z2),z˙2=z2.\dot z_1=(27z_1^2+4)(1+z_2),\qquad \dot z_2=z_2.

Under the component-wise polynomial diffeomorphism

z1=u+u3,z2=v,z_1=u+u^3,\qquad z_2=v,

the transformed dynamics are

u˙=(1+3u2)(4+3u2)(1+v),v˙=v.\dot u=(1+3u^2)(4+3u^2)(1+v),\qquad \dot v=v.

Following the factorization in Lemma 3.1, the residual factors of the first equation are r1(z)=1+z2r_1(z)=1+z_2 and r~1(u,v)=1+v\tilde r_1(u,v)=1+v, both containing two monomial terms. The residual factors of the second equation are constant in both coordinate systems. Thus, the residual sparsity counts are preserved for all equations. The non-cancellation assumptions also appear to be satisfied, yet z1=u+u3z_1=u+u^3 is not a monomial transformation.

So the claimed identifiability guarantee does not hold even under the stated assumptions.

W2: The theoretical analysis assumes polynomial ODEs, polynomial coordinate transformations, and non-cancellation conditions. The experiments are however run with MLP and sparsity is approximated via L1 regularization.

W3: The main methodological idea is closely related to SINDy-AE (Champion et al., 2019), which already combines coordinate learning with sparse equation discovery. Lachapelle et al. (2022, 2026) also established sparsity-based identifiability principles, although for a different notion of sparsity. The main novelty therefore seems to be the specific theoretical result and the combination of existing methods. Given W1 and W2, the extent of this contribution is unclear.

Questions for authors

Please see my weaknesses.

2 comments

Thank you for finding our idea to combine CRL with equation discovery interesting and for appreciating our extensive validation. We updated the manuscript, highlighting the main changes in blue. We address your other points in the following.

W1 (Fixing old Lemma 3.1). We address the counterexample by adding a technical assumption. Both the Lotka-Volterra and the Lorenz experiments fulfill it, so this does not have any impact on our empirical results. For clarity, Lemma 3.1 in the original version is now mostly Theorem 3.2, and the current Lemma 3.1 only deals with the divisibility of the polynomial ODE and the necessary form of the diffeomorphism hh.

The counterexample works because the first variable z1z_1 never appears in any of the remainder polynomials r1,r2r_1, r_2, as they both depend only on z2z_2. Thus, the variable z1z_1 and the transformed uu never influence the remainder polynomials of the transformed ODEs. Then the only constraint that limits the form of h1h_1 is that h1′h'_1 must divide q1(h1(u))q_1(h_1(u)). Since q1(z1)=27z12+4q_1(z_1) = 27z_1^2 + 4 is neither constant nor of the form (z1−ρ1)m1(z_1-\rho_1)^{m_1}, neither special case of Lemma 3.1 applies, and h1h_1 is not forced to be a monomial.

We solve the issue by adding to Thm. 3.2 the new Ass. A.4:

Let SiS_i be the set of indices jj of the variables zjz_j that appear in the remainder polynomial rir_i. We assume that ∪iSi=[d]\cup_i S_i = [d], or, in other words, all variables appear in at least one remainder polynomial.

Previously, when we counted the number of terms in the transformed remainder r~\tilde{r}, we implicitly assumed this, as the expansion accounted for each transformed variable and the number of terms it contributes to. With this additional assumption, each variable influences at least one transformed remainder. Hence, if the transformed ODE has the same number of remainder terms as the original one for every component, no hjh_j can be a general polynomial transformation, and each must be a monomial up to a shift.

W2 (Polynomial experiments already in submission). We already report on experiments with polynomial encoders/decoders in the original submission in App. G (now appendix G.2). The results were not as stable as with general encoders/decoders, also because most CRL methods, including the ones we use, cannot guarantee that the learned variables are polynomial transformations of the true ones, even with polynomial encoders/decoders. So most of our evaluation focuses on the general setting. As discussed in Sec. 3 and 4, the theoretical analysis only covers the special case of polynomials, in which we can get even stronger identifiability guarantees, but our framework is more general and in the general case still maintains the identifiability guarantees of the underlying CRL methods.

W2 (L1 for sparsity penalty vs L0). Using the L1L_1 penalty is standard in equation discovery, since L0L_0 is not differentiable. We take inspiration from SINDyAutoencoder, which combines the sequential thresholding of SINDy with an L1L_1 penalty. Similarly, CRL methods such as DMS-VAE optimize relaxations of the L0L_0 penalty. We ran an additional ablation on the importance of sparsity on Lotka-Volterra and CITRIS, where we turn off the L1L_1 penalty and sequential thresholding. The results are in App. G.3 of the revised paper, showing that not only are the recovered ODEs less sparse than with L1 sparsity, but also the trajectory prediction deteriorates.

W3 (Novelty). As discussed in the paper, SINDy-AE (Champion et al. 2019) does not provide any guarantee on learning the ground truth variables, even up to any identifiability class, and hence also no guarantees on the learned equations. Similarly, CRL methods, including DMS-VAE (Lachapelle et al. 2022, Lachapelle et al. 2026), which is one of our baselines, do not learn equations at all (so sparsity cannot be applied to the learned equations). Moreover, using the CRL methods directly as preprocessing for an equation discovery method (which we use as a baseline in our experiments and which, to the best of our knowledge, has not been done before) provides overly complicated equations. In Figure 2, we show exactly this comparison with CITRIS (we also show the comparison with DMSVAE in App.H), and showcase how our approach is better suited to learning equations that are both better at forecasting and closer to the ground truth ones.

Moreover, our framework is agnostic to the equation discovery and CRL methods, and can be combined also with other types of methods beyond the ones mentioned by the reviewer, including MNN and CITRIS as we show in our evaluation. Finally, although limited to polynomials, our theoretical analysis shows that we can recover the ground truth variables up to permutations and monomial transformations up to a shift, which is a promising result that goes beyond the standard CRL identifiability.

Thanks for the detailed response and the additional experiments. While my concern regarding the original theorem has been addressed, I am still sceptical about the relevance of the theoretical results to the actual method and the extent of the methodological novelty. I therefore maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their insightful feedback and for helping us improve the presentation and quality of our contributions. We updated the manuscript based on the discussion with the reviewers, highlighting the changes in blue.

Clarity and theoretical analysis

Following the suggestions of Reviewer ck8g, we restructured Section 3 to improve its clarity and better distinguish the main claims and theoretical contribution. We split the previous Lemma 3.1 into two main results, both of which now have proof sketches in the main paper:

  1. Lemma 3.1, characterizing which polynomial diffeomorphisms hih_i are admissible given the structure of fif_i, including a clearer definition of the domains of the diffeomorphisms (as suggested by Reviewer P95H).

  2. Theorem 3.2, proving the identifiability of the latents up to monomial transformations and a constant shift based on the sparsity of the recovered ODE.

In particular, we also clarified the identifiability class in our final results in Theorem 3.2, which now specifies that there might also be indeterminacy up to a constant shift, and discussed a special case where the identifiability is up to affine transformations.

Moreover, we added an assumption (Assumption A.4) to Theorem 3.2, requiring every variable zjz_j to appear in at least one remainder polynomial rir_i, as otherwise sparsity would not constrain the form of the diffeomorphism hjh_j related to it. This assumption covers the counterexample reported by Reviewer RXkB. This assumption holds in all of our experimental scenarios, so the empirical results are unaffected.

Additional experiments and metrics

As suggested by Reviewer ck8g, we added two metrics to all our results instead of the Pearson correlation matrices:

  • the mean correlation coefficients (MCC), and
  • R2 discrepancy (R2D), measuring how far the true R2 scores between pairs of true variables are compared to the R2 scores between true and recovered ones, which allows us to take into account nonlinear dependencies between the ground truth variables and how close our recovered latent variables are in terms of these.

We also performed an additional ablation where we trained SINDyAE with true derivatives (the setting of its original paper) instead of empirical ones (our more realistic setting). The results in Figure 8, Appendix G.4 in the revised paper, either improve very slightly or stay approximately the same: SINDyAE still underperforms compared to SPEED-AE.

In response to Reviewers P95H and RXkB, we performed an ablation in which we turned off the L1L_1 penalty and sequential thresholding and reran the whole model selection on the other parameters for SPEED-AE (C+S) and (C+M). For both the SINDy-based and MNN-based versions, the removal of the sparsity constraint strongly affects the performance of SPEED-AE, affecting both the forecasting and the recovered ODE, as shown in Table 6, Appendix G.3, in the revised paper.

As suggested by reviewer P95H, we confirmed that SINDyAE cannot take any advantage from training with interventional data, as also expected based on our theoretical analysis, as we show in Figure 9, App. G.5 in the revised paper. Furthermore, we ran an additional experiment on the Pendulums experiment, where we used DMSVAE and SINDy on only observational data, without interventions. The results, available in Table 7, Appendix G.6, show that SPEED-AE’s pipeline works even in this case, with results that are in line with the rest of the experiments.

Finally, as suggested by reviewer P95H, we considered a setting where the true number of latents is not available, and the system is overspecified. We show the results in App.G7, where we used the Lotka-Volterra system and ran the whole pipeline with d’=4>2=dd’=4>2=d latents. We show that SPEED-AE with CITRIS and SINDy is able to recover the ODEs after a simple post-processing that identifies and removes from the equations the latent variables that are constant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.