- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Neural Autonomous Differential Equations (NADE) to reduce the computational cost of learning autonomous dynamical systems. The authors argue that conventional Neural ODEs (NODEs) require numerical integration during training, which increases computational expense and numerical sensitivity.
To address this issue, NADE separates trajectory reconstruction from vector-field learning. It first fits cubic splines to observed trajectories and analytically differentiates them to obtain velocity targets. Standard multilayer perceptrons (MLPs) are then trained to approximate the state-to-velocity mapping without numerical integration during optimization. The paper introduces two implementations, NADE-j and NADE-c, and presents a consistency analysis under regularity and identifiability assumptions.
Experiments on several synthetic dynamical systems, including Duffing, Van der Pol, Lotka–Volterra, Lorenz, and Lorenz–96, report substantial improvements in vector-field accuracy and computational efficiency compared with NODE, FNODE, and NFLOW under the authors' evaluation protocol.
Critical strengths
-
The paper provides a clear explanation of the relationship among basic concepts in the ODE learning field, such as numerical integration, derivative supervision, and autonomous vector-field learning. The overall workflow is straightforward to follow.
-
The figures and experimental visualizations are carefully prepared. These aspects reflect good technical writing and engineering practice.
Critical weaknesses
- The proposed methodology lacks substantial independent novelty
The central contribution of NADE is presented as separating vector-field learning from numerical integration during training. However, I do not find this to be a fundamentally new learning paradigm.
One of the important features of conventional NODEs is precisely the incorporation of numerical integration into the training process. Although this increases the computational cost of backpropagation, it serves a clear methodological purpose. By optimizing trajectory errors directly, NODEs avoid relying on externally estimated derivative targets, which can introduce additional errors and noise.
NADE essentially removes this integration-based training mechanism and replaces it with standard supervised state-to-velocity regression. As described in Section 3, NADE-j uses an ordinary MLP with a conventional MSE loss to approximate the vector field. NADE-c further introduces coordinate-wise MLPs and learnable output scaling, which are also relatively standard neural network design choices.
Moreover, the cubic spline fitting, analytical differentiation, and derivative-target preparation used in NADE are established numerical techniques. The underlying training principle is closely related to conventional gradient matching, derivative matching, and neural vector-field regression. FNODE already follows a similar paradigm, with the main difference being its use of Fourier differentiation rather than spline differentiation.
Therefore, the most identifiable methodological difference appears to be the choice of derivative estimator rather than a new vector-field learning mechanism.
The consistency analysis provides some theoretical justification for the two-stage estimator, but it relies on conventional parametric assumptions and does not clearly establish a substantial theoretical advance beyond existing estimation principles.
In my view, the proposed method is primarily a straightforward combination of mature numerical preprocessing and standard MLP regression. While such a combination may be useful in practice, its independent algorithmic contribution appears insufficient for a conference at the level of ICLR, even if it is proposed before the original NODE method.
- The experimental comparisons raise concerns about baseline adequacy and the interpretation of the reported improvements
Section 5 reports substantial improvements over NODE, FNODE, and NFLOW. However, considering that NADE essentially performs conventional MLP vector-field regression, the magnitude of these improvements warrants closer examination.
In particular, NODE performs surprisingly poorly on several classical low-dimensional systems, including Van der Pol and Lotka–Volterra. These systems are well-established benchmarks, and a properly configured and sufficiently optimized NODE should generally be capable of approximating their dynamics.
Although the trajectories labeled "within training horizon" in Figures 3–5 are generated from held-out initial conditions rather than the original training trajectories, these initial conditions are relatively close to those used for training. The poor NODE predictions are therefore concerning, particularly because the reported vector-field errors at the observed states are also large.
Appendix B further demonstrates that NODE can learn the Lorenz dynamics more successfully when trained on a shorter trajectory. This suggests that the poor performance in the main experiments may be strongly influenced by long-horizon optimization difficulties rather than insufficient model capacity.
Unfortunately, the paper does not provide enough detail about the NODE solver configuration, numerical tolerances, and training-window strategy to establish whether the baseline was adequately optimized. A comparison using multiple shorter training windows extracted from the same complete trajectories would be particularly important.
Another concern is the use of identical wall-clock budgets for methods with fundamentally different training objectives. NADE performs direct supervised regression, whereas NODE optimizes trajectory errors through numerical integration. Equal training time does not necessarily imply equally effective optimization, particularly when one method is substantially more computationally demanding per iteration.
Furthermore, the reported vector-field metric, MSE, is evaluated at the observed training-trajectory states rather than on an independent set of test states. Although the paper includes trajectory predictions from unseen initial conditions, the vector-field results do not directly establish generalization to unobserved regions of the state space.
The selected experimental conditions may also favor spline-based derivative regression. Several benchmarks involve smooth, densely sampled, noise-free trajectories, for which cubic splines can provide highly accurate derivative estimates. In contrast, Fourier differentiation may be disadvantaged by boundary artifacts when applied to nonperiodic trajectories.
A more informative comparison would include conventional derivative estimators, such as Savitzky–Golay filtering or local polynomial differentiation, followed by the same MLP regression architecture. Without such controls, it is difficult to establish whether the reported gains represent a meaningful new contribution.
Overall, the experiments demonstrate that spline-based MLP regression performs well under the selected conditions. However, they do not convincingly establish that NADE offers a substantial methodological advantage over established derivative-matching approaches or adequately optimized NODE baselines.
More importantly, even if the reported improvements were fully confirmed, the underlying methodological novelty would remain limited.
Questions for authors
- Independent methodological contribution
Could the authors clarify what fundamentally new learning mechanism NADE introduces beyond conventional spline-based derivative estimation and supervised MLP vector-field regression?
In particular, how does NADE differ from established gradient-matching and derivative-matching approaches, apart from the choice of derivative estimator and the coordinate-wise parameterization?
Could the authors also clarify the additional theoretical insight provided by the consistency analysis compared with existing results for two-stage parametric estimation?
- Adequacy of the NODE baseline
The NODE baseline performs poorly even on relatively simple low-dimensional systems. Could the authors provide the complete NODE training configuration, including the numerical solver, error tolerances, gradient computation, and training-window length?
It would be important to compare against a NODE trained on multiple short windows extracted from the same complete training trajectories, while maintaining comparable model capacity and computational budgets.
Such a comparison would help determine whether the reported performance gap reflects the intrinsic limitations of NODE or the particular optimization protocol used in this study.
We thank the reviewers for their time and effort, and in particular for pointing us to the collocation method [1]. We were indeed not aware of [1], and our literature search earlier in this project, with the tools available then, did not bring it up. We nonetheless think our work is valuable, and preliminary experiments in fact suggest that NADE outperforms collocation, but evaluating this properly is clearly beyond the scope of a rebuttal. We will resubmit to another venue.
[1] E. Roesch et al. Collocation based training of neural ordinary differential equations, 2021.