acceptodds
Under review as a conference paper at ICLR 2027

Neural Autonomous Differential Equations: Integration-Free Learning for Time-Invariant Dynamics

Abstract

We consider the problem of learning autonomous dynamics from observed trajectories. Neural ODEs, which model the vector field with a neural network, have emerged as a popular framework for this task. These methods backpropagate through adaptive ODE solvers, which makes them computationally expensive and numerically sensitive. Recently proposed variants accelerate training, but either weaken the ability to constrain autonomous dynamics, introduce periodicity assumptions, or require trajectory truncation. We propose Neural Autonomous Differential Equations (NADE), a simple solver-free training method. NADE separates trajectory reconstruction from vector-field estimation by (i) fitting the trajectory using differentiable splines, and (ii) training the velocity field using analytic derivatives of the fitted trajectory at sampled states. At inference time, the learned autonomous field can be integrated with any standard numerical solver. This preserves the inductive bias of autonomous dynamics while avoiding solver-based training. We evaluate NADE on multistable, periodic, and chaotic systems. Under equal wall-clock training budgets, it achieves substantially faster convergence and lower training error than standard Neural ODEs and accelerated variants.

Primary area:
learning on time series and dynamical systems
Keywords:
Learning Unknown Dynamics, Neural ODE, Autonomous Dynamics, Dynamical Systems

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_D1PiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Neural Autonomous Differential Equations (NADE) to reduce the computational cost of learning autonomous dynamical systems. The authors argue that conventional Neural ODEs (NODEs) require numerical integration during training, which increases computational expense and numerical sensitivity.

To address this issue, NADE separates trajectory reconstruction from vector-field learning. It first fits cubic splines to observed trajectories and analytically differentiates them to obtain velocity targets. Standard multilayer perceptrons (MLPs) are then trained to approximate the state-to-velocity mapping without numerical integration during optimization. The paper introduces two implementations, NADE-j and NADE-c, and presents a consistency analysis under regularity and identifiability assumptions.

Experiments on several synthetic dynamical systems, including Duffing, Van der Pol, Lotka–Volterra, Lorenz, and Lorenz–96, report substantial improvements in vector-field accuracy and computational efficiency compared with NODE, FNODE, and NFLOW under the authors' evaluation protocol.

Critical strengths
  1. The paper provides a clear explanation of the relationship among basic concepts in the ODE learning field, such as numerical integration, derivative supervision, and autonomous vector-field learning. The overall workflow is straightforward to follow.

  2. The figures and experimental visualizations are carefully prepared. These aspects reflect good technical writing and engineering practice.

Critical weaknesses
  1. The proposed methodology lacks substantial independent novelty

The central contribution of NADE is presented as separating vector-field learning from numerical integration during training. However, I do not find this to be a fundamentally new learning paradigm.

One of the important features of conventional NODEs is precisely the incorporation of numerical integration into the training process. Although this increases the computational cost of backpropagation, it serves a clear methodological purpose. By optimizing trajectory errors directly, NODEs avoid relying on externally estimated derivative targets, which can introduce additional errors and noise.

NADE essentially removes this integration-based training mechanism and replaces it with standard supervised state-to-velocity regression. As described in Section 3, NADE-j uses an ordinary MLP with a conventional MSE loss to approximate the vector field. NADE-c further introduces coordinate-wise MLPs and learnable output scaling, which are also relatively standard neural network design choices.

Moreover, the cubic spline fitting, analytical differentiation, and derivative-target preparation used in NADE are established numerical techniques. The underlying training principle is closely related to conventional gradient matching, derivative matching, and neural vector-field regression. FNODE already follows a similar paradigm, with the main difference being its use of Fourier differentiation rather than spline differentiation.

Therefore, the most identifiable methodological difference appears to be the choice of derivative estimator rather than a new vector-field learning mechanism.

The consistency analysis provides some theoretical justification for the two-stage estimator, but it relies on conventional parametric assumptions and does not clearly establish a substantial theoretical advance beyond existing estimation principles.

In my view, the proposed method is primarily a straightforward combination of mature numerical preprocessing and standard MLP regression. While such a combination may be useful in practice, its independent algorithmic contribution appears insufficient for a conference at the level of ICLR, even if it is proposed before the original NODE method.

  1. The experimental comparisons raise concerns about baseline adequacy and the interpretation of the reported improvements

Section 5 reports substantial improvements over NODE, FNODE, and NFLOW. However, considering that NADE essentially performs conventional MLP vector-field regression, the magnitude of these improvements warrants closer examination.

In particular, NODE performs surprisingly poorly on several classical low-dimensional systems, including Van der Pol and Lotka–Volterra. These systems are well-established benchmarks, and a properly configured and sufficiently optimized NODE should generally be capable of approximating their dynamics.

Although the trajectories labeled "within training horizon" in Figures 3–5 are generated from held-out initial conditions rather than the original training trajectories, these initial conditions are relatively close to those used for training. The poor NODE predictions are therefore concerning, particularly because the reported vector-field errors at the observed states are also large.

Appendix B further demonstrates that NODE can learn the Lorenz dynamics more successfully when trained on a shorter trajectory. This suggests that the poor performance in the main experiments may be strongly influenced by long-horizon optimization difficulties rather than insufficient model capacity.

Unfortunately, the paper does not provide enough detail about the NODE solver configuration, numerical tolerances, and training-window strategy to establish whether the baseline was adequately optimized. A comparison using multiple shorter training windows extracted from the same complete trajectories would be particularly important.

Another concern is the use of identical wall-clock budgets for methods with fundamentally different training objectives. NADE performs direct supervised regression, whereas NODE optimizes trajectory errors through numerical integration. Equal training time does not necessarily imply equally effective optimization, particularly when one method is substantially more computationally demanding per iteration.

Furthermore, the reported vector-field metric, MSE, is evaluated at the observed training-trajectory states rather than on an independent set of test states. Although the paper includes trajectory predictions from unseen initial conditions, the vector-field results do not directly establish generalization to unobserved regions of the state space.

The selected experimental conditions may also favor spline-based derivative regression. Several benchmarks involve smooth, densely sampled, noise-free trajectories, for which cubic splines can provide highly accurate derivative estimates. In contrast, Fourier differentiation may be disadvantaged by boundary artifacts when applied to nonperiodic trajectories.

A more informative comparison would include conventional derivative estimators, such as Savitzky–Golay filtering or local polynomial differentiation, followed by the same MLP regression architecture. Without such controls, it is difficult to establish whether the reported gains represent a meaningful new contribution.

Overall, the experiments demonstrate that spline-based MLP regression performs well under the selected conditions. However, they do not convincingly establish that NADE offers a substantial methodological advantage over established derivative-matching approaches or adequately optimized NODE baselines.

More importantly, even if the reported improvements were fully confirmed, the underlying methodological novelty would remain limited.

Questions for authors
  1. Independent methodological contribution

Could the authors clarify what fundamentally new learning mechanism NADE introduces beyond conventional spline-based derivative estimation and supervised MLP vector-field regression?

In particular, how does NADE differ from established gradient-matching and derivative-matching approaches, apart from the choice of derivative estimator and the coordinate-wise parameterization?

Could the authors also clarify the additional theoretical insight provided by the consistency analysis compared with existing results for two-stage parametric estimation?

  1. Adequacy of the NODE baseline

The NODE baseline performs poorly even on relatively simple low-dimensional systems. Could the authors provide the complete NODE training configuration, including the numerical solver, error tolerances, gradient computation, and training-window length?

It would be important to compare against a NODE trained on multiple short windows extracted from the same complete training trajectories, while maintaining comparable model capacity and computational budgets.

Such a comparison would help determine whether the reported performance gap reflects the intrinsic limitations of NODE or the particular optimization protocol used in this study.

Reviewer_6yFCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes NADE, a two-stage method for learning autonomous dynamics without numerical integration during training. It estimates trajectory derivatives using cubic interpolation or smoothing splines, then trains a state-only neural vector field against these targets. Joint and coordinate-wise network variants are considered, with numerical integration used at inference. The paper provides a consistency result for a well-specified parametric formulation and evaluates the method on synthetic dynamical systems, including noisy observations and ten-dimensional Lorenz–96.

Critical strengths

The method is straightforward and computationally attractive. Separating derivative estimation from vector-field regression avoids differentiating through a solver while retaining an explicit autonomous model.

The reported clean-data results show substantial improvements under the stated per-trial training budgets. The three-seed comparisons and Lorenz–96 extension provide useful evidence of practical effectiveness in densely observed settings.

Critical weaknesses

The central estimation procedure is an established two-stage gradient-matching strategy: reconstruct trajectories with splines, then fit a parameterized vector field to their derivatives. This procedure appears directly in Varah [1], with subsequent consistency analysis of two-stage estimators in Brunel [2]. Neither work is discussed. Replacing the vector-field parameterization with a standard MLP does not, by itself, establish a substantive methodological advance. Comparisons against other inexpensive derivative estimators using the same neural network are also missing.

The consistency theorem does not cover the estimator used in the experiments. It assumes a fixed-dimensional compact trajectory family containing the exact solution, whereas the implemented spline changes with the observation grid. The proof therefore provides neither spline approximation control nor conditions ensuring derivative consistency under noise. Moreover, the theoretical objective uses fitted states as regression inputs, while the implementation description appears to retain observed states, potentially introducing errors-in-variables bias. The theorem may hold under its stated assumptions, but its relevance to the implemented pipeline remains unestablished.

References

[1] J. M. Varah. A Spline Least Squares Method for Numerical Parameter Estimation in Differential Equations. SIAM Journal on Scientific and Statistical Computing, 3(1):28–46, 1982. DOI: 10.1137/0903003.

[2] N. J.-B. Brunel. Parameter Estimation of ODE's via Nonparametric Estimators. Electronic Journal of Statistics, 2008. arXiv:0710.4190.

Questions for authors
  1. What substantive advance does NADE provide over classical spline-based derivative matching? Please compare alternative derivative estimators using the same neural vector field, optimizer, and evaluation protocol, particularly across sampling densities and noise levels.

  2. Under noisy observations, are the regression inputs the observed states xkx_k or the fitted states sη^(tk)s_{\hat{\eta}}(t_k)? Please reconcile Equations (4)–(7) with Equation (11), and explain how the consistency result applies to the sample-dependent spline estimator used in the implementation.

  3. What shooting horizon, trajectory batching, solver, tolerances, and differentiation scheme are used for NODE? Could the authors include a tuned short-window NODE baseline trained on the same long records? The separate T=1T=1 experiment does not establish competitiveness against this training strategy.

Reviewer_WjgUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes the Neural Autonomous Differential Equations (NADE) framework, which aims to effectively train neural ordinary differential equations (Neural ODEs) without using numerical ODE solvers. The key idea is that training an autonomous Neural ODE becomes a supervised learning problem if one has access to the velocity vectors at each measured state. To obtain this velocity information from potentially noisy and irregular measurements, the authors fit smoothing cubic splines on the measured trajectories and utilize the analytical derivative of the cubic spline. Experiments performed against Neural ODEs, Fourier Neural ODEs (FNODEs) and Neural Flows (NFLOWs) on various low dimensional systems (highest system tested is 10) demonstrate the enhanced accuracy of the approach.

Critical strengths

Developing alternative training strategies for Neural ODEs to circumvent the heavy costs and training pathologies associated with the numerical ODE solver is an important problem in the field. The paper's exposition is clean with visualizations that clearly showcase the behavior of the learned vector fields.

Critical weaknesses

Significant overlaps with existing results

The biggest problem with this paper is that training Neural ODEs through supervised learning via pre-estimated velocity vectors is already an well-established result in the field. For parameter estimation of ODE models, there exists classical literature describing the same idea (cubic spline smoothing for derivative estimation, followed by supervised regression on the parameters of the vector field) [1]. For Neural ODEs specifically, the proposed approach is extremely similar to the collocation method for training Neural ODEs [2], which is from 2021 and have been incorporated into Julia's ecosystem for training neural ordinary differential equations [3].

In [2], systematic comparisons between using an numerical solver and using estimated derivatives for various noise strengths has already been reported; the difference between [2] and this paper is the choice of the derivative estimator (local polynomial regression vs smoothing cubic splines), which seems more like an implementation choice than a methodological novelty.

Given these existing results, simply proposing a supervised training strategy via estimated derivatives and comparing performances is too much of an overlap to warrant a new publication. Furthermore, given its similarity, the collocation based training method must also be incorporated into the baseline methods for comparison.

  • [1]: J. Varah. A spline least squares method for numerical parameter estimation in differential equations. SIAM Journal on Scientific and Statistical Computing 3, 1 (1982).
  • [2]: E. Roesch et al. Collocation based training of neural ordinary differential equations. Statistical Applications in Genetics and Molecular Biology 20, 2 (2021).
  • [3]: Smoothed Collocation. DiffEqFlux.jl documentation. https://docs.sciml.ai/DiffEqFlux/dev/utilities/Collocation/

The effect of noise on performance is not properly investigated & Smoothing leverages inaccessible information

The key shortcoming of Neural ODE supervised training schemes that use estimated trajectory derivatives is how much it is affected by noise. While the paper uses smoothing cubic splines to obtain more robust derivative estimations, couple serious concerns still remain.

Tested noise strength is too weak

The tested noise strength is 2% of the maximum of each coordinate (Appendix I), which is in the weaker end of noise strengths used in Neural ODE studies. For example, [2] uses noise standard deviations of 0.0, 0.1, 0.2, 0.5 for the Van der Pol oscillator with a maximum coordinate value of roughly 2. The weak penalty Neural ODE framework (WP-NeuralODE) [4], which is another framework that avoids ODE solvers by utilizing the ODE weak form, uses noise standard deviations up to 20% of the RMS of each coordinate. To properly verify if the proposed method has comparable robustness to these other methods, much higher noise strengths (up to 20%) should be tested.

Smoothing parameter of the smoothings splines are set based on the added noise strength, which is unavailable information in practice

As mentioned above, a key procedure in these types of training methods is how to estimate the derivative values from noisy data. With cubic smoothing splines, this boils down to the problem of how to set an appropriate smoothing parameter value without prior knowledge on the added noise, which is another nontrivial problem. However, Appendix E states that the smoothing parameter is set using exactly this unavailable information, making the paper's noise robustness claims very weak.

Neural ODE baseline seems to not have been adequately trained

In the reported results, Neural ODEs are seen to perform extremely badly compared to other baselines, with completely erroneous vector fields. For the simple systems considered in this study, this is not an expected behavior. One strongly suspected reason is that the Neural ODEs were trained using very long rollouts (for each initial point, produce long rollouts, obtain the trajectory-wise MSE, and optimize this loss). The fact that Neural ODEs struggle when trained on long rollouts is a well-reported phenomenon in the Neural ODE literature, as the compounding compositions of neural network evaluations can cause optimization pathologies [5, 6, 7]. If this is indeed the case, the Neural ODE baseline should be retrained using one-step or few-step rollouts instead for a more fair comparison.

  • [4]: X. Li et al. A weak penalty neural ODE for learning chaotic dynamics from noisy time series. arXiv:2511.06609 (2025).
  • [5]: J. Ko et al. Homotopy-based training of NeuralODEs for accurate dynamics discovery. NeurIPS (2023).
  • [6]: F. Hess et al. Generalized teacher forcing for learning chaotic dynamics. ICML (2023).
  • [7]: L. Eisenmann et al. Bifurcations and loss jumps in RNN training. NeurIPS (2023).
Questions for authors

Q1.

The proposed algorithm is very closely related to the collocation-based training methods mentioned above. What advantage does the proposed cubic smoothing splines have over the local polynomial regression used in collocation-based methods, or other types of derivative estimation schemes (ex. total variation regularization)?

Q2.

How was the plain Neural ODE baseline trained? How many temporal rollouts were used to form its loss function?

Q3.

How does the proposed method compare against closely related baselines such as collocation-based training or weak-penality Neural ODE?

Comments on the paper

1 comment

We thank the reviewers for their time and effort, and in particular for pointing us to the collocation method [1]. We were indeed not aware of [1], and our literature search earlier in this project, with the tools available then, did not bring it up. We nonetheless think our work is valuable, and preliminary experiments in fact suggest that NADE outperforms collocation, but evaluating this properly is clearly beyond the scope of a rebuttal. We will resubmit to another venue.

[1] E. Roesch et al. Collocation based training of neural ordinary differential equations, 2021.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.