- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies recovering the governing ODE of a system from high-dimensional observations such as images. CRL methods identify latents only up to permutation and component-wise diffeomorphism. The authors argue that such distortions destroy the sparsity that equation discovery e.g., SINDy needs. They propose SPEED-AE, which is a frozen, pretrained CRL encoder followed by component-wise autoencoders trained jointly with a sparse ODE objective (SINDy or MNN). The paper states that for polynomial ODEs and polynomial diffeomorphisms, sparsity restricts the component-wise maps to monomials. The experiments on Lotka-Volterra, Lorenz, and two pendulums show better disentanglement and recovered equations closer to the ground truth than SINDyAE, MNNAE, and CRL latents fed directly to SINDy/MNN.
Critical strengths
S1: Even with perfect disentanglement, the factor in the conjugate ODE can destroy sparsity. The point is simple, new, and well illustrated.
S2: It is shown that component-wise autoencoders preserve the CRL disentanglement, and that the pipeline works with different CRL and equation-discovery back-ends.
Critical weaknesses
W1: SPEED-AE's CRL stage is pretrained on a separate dataset with interventions and known intervention targets. SINDyAE and MNNAE appear to be trained only on standard trajectories and are described as learning latents and equations "without additional information". Poor disentanglement of the baselines is therefore partly expected, and the gap in CorrD and ODE recovery says little about the equation-discovery objective. The paper discloses the interventional pretraining but never discusses this asymmetry. Baselines with access to the same interventional data are missing.
W2: Lemma 3.1 assumes polynomial diffeomorphisms and two non-cancellation assumptions. CRL guarantees only a general diffeomorphism, and polynomial encoders and decoders were proven unstable by the authors even in the oracle setting. The final model therefore uses MLPs, which is a reasonable next approach, but the identifiability result does not apply to what is evaluated. The theoretical analysis does not apply to the implementation.
The theory also does not explain the observed gains. For Lotka-Volterra, , so is non-constant, and as I read Lemma 3.1 any monomial map is allowed, with the same number of terms for every . The sparsity objective therefore does not favor . The experiments nevertheless show near-affine maps (Figs. 10, 11), and the evaluation protocol rewards them. The observed behavior is not predicted by the lemma and may come from the MLPs' inductive bias or from the interventional CRL stage and no ablation tests whether sparsity is the cause. The ablations vary the CRL stage or drop the autoencoder, but none removes or varies the penalty or tests a case where the lemma predicts a non-affine monomial. As a result, the link between the theory and the empirical advantage is not shown.
W3: The proof of Lemma 3.1 argues about real roots of . The domain of is never stated. For a polynomial diffeomorphism of all of (e.g., ), has no real roots, so the argument would need to be made over . The authors' own counterexample uses a diffeomorphism "restricted to an appropriate domain", where has real roots outside it. Likewise, the conclusion with is a diffeomorphism only on restricted domains (e.g., positive values). Please state the domain assumptions explicitly and restate the argument accordingly.
W4: From my understanding applying SPEED-AE to a new system from images requires (i) the true number of latents, which is set to in the CRL models and never estimated, (ii) the ability to intervene on every latent, with known intervention targets, via arbitrary state resets, and (iii) a CRL stage that actually disentangles, since the 1D autoencoders cannot repair cross-latent mixing. The authors explain why they need interventions, but this is equal to knowing the system's structure and being able to manipulate each of its degrees of freedom, which excludes passively observed systems. The paper describes the method as "agnostic" to the CRL method, yet both CRL methods tested need interventional data and the framing as discovery from "unstructured data such as images" is therefore misleading and does not reflect what the method supports.
Questions for authors
Q1: How do SINDyAE and MNNAE perform when trained with the same interventional data (and targets) used for CITRIS/DMSVAE? Q2: What domain do you assume for in Lemma 3.1, and does the divisibility argument extend to complex roots of ? Q3: How do you justify "state-of-the-art forecasting" given that in Table 6 LSTM, Transformer, and MNNAE are better in several settings? Q4: How would SPEED-AE be applied to a system where the number of latents is unknown or interventions are unavailable? What happens if is over- or under-specified in the CRL stage? Q5: What happens to the learned maps and recovered equations when the penalty is removed or varied, and in a system where Lemma 3.1 predicts a non-affine monomial map? Without this, how do you attribute the gains to sparsity?
3 comments
Response (2/2)
W3 and Q2 (Complex Roots and diffeomorphisms domain). Thank you for identifying relevant aspects of the Lemma that need further clarification. In the revised manuscript, we now specify the domain of the polynomial diffeomorphisms. In particular, we consider a non-empty real interval on which is a diffeomorphism onto its image . We also add more details on the domain and image of such diffeomorphisms throughout the proofs. With regard to real or complex roots, the proof of the new Lemma 3.1 now considers the roots of over , which covers diffeomorphisms such as the one reported by the reviewer. However, even in the revised manuscript, all of the polynomials are defined over real coefficients, while we prove all the results by allowing complex roots and recovering the real case as a consequence. Thus, the final statements still relate to polynomials with real coefficients.
W4 and Q4 (Interventions). As already mentioned in the answer to W1, we are agnostic to CRL and do not require interventions (and one of the methods we use, DMSVAE, actually already works without interventions under assumptions on the sparsity in the transitions). Regardless of using interventions, the inputs are still images, both in CRL and for our framework, and having interventions does not equate to knowing the system structure (otherwise there would be no need for CRL in general).
W4 and Q4 (Number of latents). We remark that the baselines SINDy-AE and MNNAE also receive the ground-truth number of variables as input; hence, the empirical comparison is fair. Moreover, while the theory of CRL methods often assumes a known number of latents, most methods empirically work quite well as long as this number is overestimated, ultimately not learning any information in the excess dimensions, as shown, for example, by [Lippe et al. 2022]. This is the same for SPEED-AE, since the unused latents will not be used in the learned equations. We added an ablation in App. G.7 where CITRIS and SPEED-AE start from an overestimated number of latents, and the extra latents do not capture any of the true ones. We show that SPEED-AE with CITRIS and SINDy is able to recover the ODEs after a simple post-processing that identifies and removes from the equations the latent variables that are constant.
W4 and Q4 (Disentanglement). We agree that SPEED-AE relies on a CRL stage to disentangle the latents and cannot correct cross-latent mixing. We discuss these limitations, inherited by the CRL backends, in the conclusions of our manuscript.
Q3 (Forecasting). We rephrased “state-of-the-art” into “competitive forecasting performance”. As reported in Table 7 (App. H), unstructured models such as LSTM and Transformer perform well on the short Lorenz trajectories, but their errors on the longer Lotka-Volterra trajectories are up to an order of magnitude higher than those of SPEED-AE. SPEED-AE consistently outperforms SINDyAE, while MNNAE achieves lower forecasting errors in three settings, but fails on chaotic Lorenz. As discussed in Sec. 6, the forecasting performance of MNNAE comes at the cost of non-identifiable latents and poor ODE recovery. Our main goal is the identification of the latents and their governing equations, and SPEED-AE is the only method among the compared ones that achieves this while maintaining competitive forecasting.
Thank you for the revisions. The new ablations address several of my other concerns but the corrected proof of Lemma 3.1 only partly resolves W3. It's a 2.5 page proof and I didn't have the time to thoroughly check it but I do understand the idea of the proof. Theorem 3.2 seems to have another issue since it depends on a non-cancellation assumption on h itself, which Lorenz already violates under a simple shift. This means the theorem doesn't show that sparsity selects monomial maps. The theory still cannot explain the near-affine maps observed for Lotka–Volterra. I raise my score slightly.
Response (1/2)
Thank you for your careful review. We updated the manuscript, highlighting the main changes in blue. We address your concerns one by one in the following.
W1, Q1 and W4 (Interventional data). There is a misunderstanding here, we do not require interventional data. SPEED-AE is CRL-agnostic, and several non-interventional CRL methods exist, which can also be used in our framework (among the ones we cite, for example, Kügelgen et al., 2021; Xu et al., 2024; Yao et al., 2023; and several others). In particular, DMSVAE is not a method that requires known interventions, but it can leverage either actions without known targets or, in the absence of actions, the sparsity in the transition functions. In our experiments, we wanted to use the same data for DMSVAE and CITRIS (which requires known intervention targets), so we used interventional data, but we did not provide the intervention targets to DMSVAE (since the method does not use them). In the revised manuscript, we added an ablation on the Pendulum experiment (App. G.6), where we use SPEED-AE with DMSVAE and SINDy on only observational data, showing that our method provides similar results also in this setting.
We also disagree that no baselines use the same data. The naive combinations of CRL and equation discovery, which we report as our main baselines, use exactly the same interventionally pretrained CRL models as SPEED-AE. So the improvement of SPEED-AE over these baselines is therefore not explained by the interventional data, and is due to the component-wise autoencoders and the sparse ODE objective.
Finally, there is no meaningful way to integrate interventional data into SINDyAE and MNNAE, as they focus on how well the learned ODE fits the data, not on the structure. Moreover, these methods might perform even worse if we add interventional data, since they see them as a mixture of distributions. We show an ablation in App. G.5 on Lotka-Volterra, where we train SINDyAE on the same data as SPEED-AE, including interventions, showing that there is no improvement in any disentanglement metrics. Furthermore, the trajectories predicted are very unstable, and most of them on most test seeds diverge. We cannot even run this ablation for MNN, since its mechanistic core does not fit the derivative at each time step, but instead predicts subsequences of the trajectories in batches. Therefore, these subsequences might have jumps in them due to interventions, and no continuous ODE can fit them accurately.
W2 (Theory for polynomials) First, we wanted to clarify a small point: while our new identifiability results are indeed limited to polynomials, the SPEED-AE framework can be applied to other settings, while still providing theoretical guarantees on the latents (which is in itself novel). In this case, we get the identifiability results inherited by the specific CRL method, which is typically identifiability up to permutations and element-wise diffeomorphisms. So although our specific identifiability results might not apply to the general case, SPEED-AE as a framework is still theoretically valid also in the general case, which is our main contribution, and this is also what we aim to show with our empirical validation. Just for completeness, we did already report on experiments with polynomial encoders/decoders in the original submission (now in App. G.2). The results were not as stable as with general encoders/decoders, also because, as you mention, most CRL methods, including the ones we use, cannot guarantee that the learned variables are polynomial transformations of the true ones, even with polynomial encoders/decoders.
W2 and Q5 (L1-Penalty). We agree that in practice SPEED-AE often recovers approximately affine transformations in general cases, while the theory can only guarantee monomials up to shift and only for the polynomial case. We hypothesize that this is due to higher-order monomials being much more affected by small errors in the learned variables (so they overfit on the noise) than lower-order monomials, leading the model to prefer simpler monomials, but a rigorous study of these effects is out of scope for this paper. This effect is also not due to the CRL stage, since the learned transformations for the naive CRL+equation discovery baselines are not affine.
In order to consider the effect of sparsity in isolation, we run an additional ablation, where we do not enforce sparsity and report the results in App. G.3. Removing sparsity affects both sequence prediction and, more importantly, the identification of the ground-truth ODE. Therefore, as proved in Theorem 3.2 (the new name for the old Lemma 3.1), sparsity is fundamental to reducing the class of possible diffeomorphisms between recovered and ground-truth variables.