acceptodds
Under review as a conference paper at ICLR 2027

Higher-Order Kolmogorov-Arnold Networks for Dynamics

Abstract

The ability to derive the governing equations of a dynamical system from data is essential for prediction and control across many scientific fields. Two challenges, however, are the expansion of candidate terms when reducing higher-order equations to first-order form and noise from estimating derivatives. We propose Higher-Order-KANDy, which addresses both problems by using delay and derivative (“jet”) embeddings of the original data with derivative-ordered Kolmogorov-Arnold Networks for Dynamics (KANDy) layers stacked into a machine learning architecture. Compared with SINDy and WSINDy on noisy data, Higher-Order-KANDy learns governing equations without a candidate library (e.g., recovering the full-period with nothing named in advance), natively for higher-order systems, and holds the correct term structure as noise grows, selecting the exact active set on every seed at noise levels up to 7.5% where an oracle weak-form regression admits a spurious term—though the oracle remains the more accurate coefficient estimator. We leverage the relationship between delay and differential embeddings, along with the Kolmogorov–Arnold representation theorem, to construct a delay-to-jet map and learn it as a Kolmogorov–Arnold network to preserve derivative structure and provide meaningful terms that appear in the governing equation. In our synthetic benchmarks, this construction recovers equations in the presence of coarse sampling and noise.

Primary area:
learning on time series and dynamical systems
Keywords:
Governing Equation Discovery, Kolmogorov-Arnold Networks, Higher-Order Differential Equations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TWy4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Higher-Order-KANDy for discovering higher-order ordinary and partial differential equations from observed data. The work addresses two challenges in conventional equation discovery methods. First, converting higher-order differential equations into first-order state-space systems may introduce additional state variables and considerably expand the candidate function library. Second, estimating higher-order derivatives directly from noisy observations can significantly amplify measurement noise.

To address these challenges, the authors combine delay embeddings, derivative (jet) embeddings, and derivative-ordered KANDy layers. The proposed framework first reconstructs derivative coordinates from delayed observations or obtains them through smooth function fitting. KANDy layers with learnable spline functions are then used to identify the right-hand side of the governing equation. The authors argue that this formulation allows higher-order equations to be identified directly while reducing the need to specify fixed nonlinear candidate functions in advance.

The theoretical analysis draws on delay embedding theory and the Kolmogorov–Arnold representation theorem to establish properties of the delay-to-jet mapping. The method is evaluated on several synthetic ODEs, PDEs, and systems with hidden states. The experiments examine equation recovery, coarse sampling, and noise robustness, with comparisons against SINDy, WSINDy, and other KAN-based methods.

Critical strengths
  1. A potentially useful formulation for higher-order dynamics discovery

The paper attempts to identify governing equations directly in their native higher-order form rather than relying entirely on first-order state-space reformulations. This may help avoid unnecessary candidate terms and preserve a representation that is more closely related to the original physical equation.

The theoretical discussion of the delay-to-jet mapping also provides some justification for the proposed architecture. In particular, the analysis of additive and compositional representations helps explain why a more expressive lifting map may be needed under certain sampling conditions.

This formulation may be useful for problems where preserving the original differential order is important.

  1. Promising results on selected nonlinear PDEs and coarsely sampled systems

The proposed method shows encouraging results on several synthetic benchmarks. For instance, it recovers the nonlinear restoring force in the sine-Gordon equation without explicitly including sin⁡(u)\sin(u) in the candidate functions. Table 1 reports good support recovery under the authors' evaluation criteria.

The learned delay-to-jet mapping also performs better than conventional differentiation or smoothing approaches in some coarsely sampled settings. These results suggest that the proposed framework may be useful for certain additively separable nonlinear equations and undersampled dynamical systems.

Critical weaknesses
  1. The library-free claim is overstated, and the benefits in interpretability and performance are not sufficiently established

The authors emphasize that Higher-Order-KANDy can discover governing equations without a candidate library, as illustrated by recovering the nonlinear function −sin⁡(u)-\sin(u) without explicitly providing it in advance.

However, this capability largely follows from the flexible function approximation provided by KAN spline edges. Such flexibility also comes at a cost to interpretability. A learned spline may have an irregular shape that does not correspond directly to a recognizable analytical expression. Additional symbolic fitting or regression may still be needed to determine the underlying functional relationship. This process may itself require candidate function families or other assumptions about the symbolic structure.

Therefore, avoiding a predefined symbolic library during training does not necessarily mean that the method directly produces interpretable governing equations.

Furthermore, Eq. (3) in Section 2.1 still introduces a feature library. Although the authors use free-form edges in some experiments, composite features must still be specified for equations involving multiplicative or non-additively separable interactions. The library-free capability is therefore restricted to particular equation structures and does not directly extend to general nonlinear dynamical systems.

In my view, the method mainly replaces fixed nonlinear basis functions with more flexible learnable splines. This may improve representation flexibility, but it can also introduce additional costs in symbolic interpretation, model complexity, and training. These trade-offs are not adequately evaluated.

More importantly, the experiments in Supplementary Section S6 show that PySINDy-STLSQ achieves better functional-form estimation accuracy when the correct sin⁡(u)\sin(u) term is included in its library and the same derivative data are used. Although Higher-Order-KANDy shows better exact-support recovery under the authors' evaluation protocol, trigonometric functions are common candidate functions in SINDy and can be included without much difficulty. The advantage over a polynomial library that excludes sin⁡(u)\sin(u) is therefore not sufficient to demonstrate a general performance improvement over a reasonably configured SINDy method.

There is also a concern about the support evaluation itself. Higher-Order-KANDy identifies symbolic terms through post-hoc shape matching of learned spline functions, whereas WSINDy selects explicit candidate functions through their coefficients. These are not fully equivalent criteria. A small residual component within a learned spline may not be counted in the same way as an explicitly selected spurious term in WSINDy. A common symbolic read-out and support evaluation procedure would make the comparison more convincing.

Overall, I find the claims regarding library-free discovery, interpretability, and performance improvement stronger than what the current evidence supports.

  1. The treatment of derivative noise relies substantially on existing techniques, while the necessity and practical relevance of the proposed architecture remain unclear

The amplification of noise during higher-order derivative estimation is presented as one of the main motivations for this work. The authors propose a delay-to-jet embedding strategy to address this problem.

However, the proposed method does not eliminate the need for higher-order derivative estimation. Instead, it appears to move a substantial part of this difficulty to the construction of derivative supervision during training or calibration.

According to Eq. (9), the training objective still requires first-order and highest-order derivative targets, which are obtained using conventional techniques such as smoothed finite differentiation and Savitzky–Golay filtering. In the PDE experiments, the derivative coordinates are similarly obtained by fitting smooth functions to the observed data.

Consequently, it is not clear how much of the reported improvement in noise robustness comes from the proposed Higher-Order-KANDy architecture rather than from established preprocessing and derivative estimation techniques.

Some coarse-sampling experiments also rely on additional densely sampled calibration trajectories or high-quality derivative targets. This raises concerns about how well the method would perform when only noisy or sparsely sampled observations are available throughout both training and deployment.

Another important limitation is the absence of a direct KAN baseline. Specifically, the paper does not compare the proposed architecture against a standard KAN that directly learns the mapping from the delay embedding to the target highest-order derivative x(n)x^{(n)}, without an explicit jet embedding stage.

Such an ablation is important for establishing whether the derivative-ordered architecture itself improves modeling accuracy and robustness. Although the authors compare linear, additive KAN, and compositional KAN lifting maps, these experiments do not directly establish the benefit of the complete Higher-Order-KANDy framework over a capacity-matched direct KAN.

The general applicability of the method is also insufficiently demonstrated. The main formulation focuses on scalar observations, and most experiments involve second- or third-order systems and selected synthetic PDEs. It remains unclear how the method would scale to high-dimensional vector-valued states or dynamical systems with strong interactions among multiple variables.

Finally, the practical advantage of the compositional delay-to-jet mapping appears to be restricted to a particular sampling regime. The observations must be too coarse for conventional differentiation to remain accurate, but still sufficiently informative for the delay mapping to preserve the information needed for state reconstruction.

The paper's own theoretical analysis and experiments show that conventional linear differentiation or smoothing splines can be competitive or even superior when sufficiently dense observations are available.

If practical measurement systems can readily provide samples that are sufficiently dense relative to their characteristic dynamical timescales, the benefit of the proposed nonlinear lifting strategy may be considerably reduced.

Although the experimental study covers several systems, many of the governing equations have relatively simple or special structures. The current results do not convincingly establish the practical value of the proposed approach for more general and complex dynamical systems.

Questions for authors
  1. Candidate-library-free discovery and interpretability

Could the authors clarify the distinction between avoiding a predefined library of analytical functions and avoiding predefined candidate feature structures?

For systems involving non-additively separable interactions, how are the feature library and composite terms determined?

When a learned KANDy spline has a complicated or irregular shape, how is it reliably converted into an explicit symbolic expression? Does this post-hoc procedure require a candidate function family, and what are its computational and interpretability costs?

  1. Derivative noise and supervision requirements

To what extent does the proposed jet embedding actually address the high-order differentiation problem, rather than transferring it to the construction of derivative supervision during calibration?

In particular, how would the method perform if only the noisy or coarsely sampled observations available at deployment were also available during training, without additional densely sampled calibration trajectories or high-quality derivative targets?

It would be helpful if the authors could clarify the sources of derivative supervision and calibration data used in the main experiments, as well as the conditions under which the learned mappings generalize to new observations.

  1. Necessity of the explicit jet embedding architecture

Why is a standard KAN that directly maps the delay embedding to the target highest-order derivative x(n)x^{(n)} not included as a baseline?

I suggest comparing a direct KAN against Higher-Order-KANDy using identical input data, derivative targets, comparable model capacity, and similar training budgets.

This comparison would help establish whether the explicit jet representation provides measurable benefits in prediction accuracy, noise robustness, or training efficiency beyond making the model structure more interpretable.

  1. Fairness of the SINDy comparisons and the practical performance advantage

Supplementary Section S6 shows that STLSQ achieves better functional-form estimation accuracy when the correct sin⁡(u)\sin(u) term is provided in the candidate library.

Since trigonometric functions are commonly used in SINDy libraries, could the authors clarify under which conditions Higher-Order-KANDy offers a meaningful advantage over a reasonably configured SINDy or WSINDy model?

Could a common post-hoc symbolic decomposition and support evaluation procedure also be applied to both the free-form KANDy functions and the explicit SINDy candidate terms?

  1. Model dimensionality and practical application scenarios

The formulation in Section 2 mainly considers a scalar observable x(t)x(t). How would the framework extend to high-dimensional vector-valued states or systems with strong interactions among multiple state variables?

More importantly, the main advantage of the compositional delay-to-jet mapping appears to arise within an intermediate sampling regime where conventional differentiation becomes inaccurate but the delay observations remain sufficiently informative.

Could the authors clarify how commonly this regime occurs in realistic scientific measurement settings? Is obtaining denser observations typically difficult or costly in the applications targeted by this work?

If sufficiently dense observations are readily available, the practical value of the learned lifting approach over conventional smoothing and derivative estimation methods may be limited.

  1. Minor Comment

Please check the figure references throughout the main text and supplementary material. Every figure should be explicitly referenced and discussed in the corresponding text.

2 comments
AuthorsOpenReview ↗

Thank you for your review!

Dear Reviewer TWy4,

Thank you for the thorough review. We agree with several of your points.

Q1. Candidate-library-free discovery and interpretability.

In the Genesio–Tesi and sine-Gordon experiments, the equation layer sees only the raw coordinates, and terms such as x2x^2 and sin⁡(u)\sin(u) are read from the edges. We supplied uuxuu_x for KdVKdV and ∣x˙∣z3|\dot{x}|z^3, x˙∣z∣3\dot{x}|z|^3 for the Bouc–Wen memory law. A suggested default for an unknown oscillatory system is a Duffing-style oscillator library as shown in Equation (4). Applying a function approximation to a learned spline requires practitioner domain knowledge, and it is computationally efficient to try many combinations. When applied to data, the splines can be irregular in shape. For example, we tested our method on biomedical data where ground truth was unknown, and the existence of a dynamical system is not necessarily guaranteed. This testbed motivated many of our algorithm’s design principles, including the jet for denoising and reintroducing the library for non-additive dynamics.

Q2. Derivative noise and supervision requirements.

The proposed jet embedding addressed the high-order differentiation problem surprisingly well. The jet embedding is consistently more accurate than the training targets: 0.981 vs. 0.964 (Δ\Delta= 0.05, 5% noise), 0.918 vs. 0.884 (Δ\Delta = 0.2, 5% noise), and 0.987 vs. 0.970 (Δ\Delta = 0.4, 1% noise). This experiment will appear in the final version.

Q3. Necessity of the explicit jet embedding architecture.

We benchmarked a KAN without the intermediate layers, and the performance was roughly the same. The additional architecture delivers extra interpretability. Accuracy is the same: held-out R2R^2 of 0.9993 for our model against 0.9991 to 0.9998 for the direct models. The jet stage added equational interpretability. Our model returns x’‘’=−ax¨−bx˙−cx+qx2x’‘’=-a\ddot{x}-b\dot{x} -cx +qx^2 with coefficients within 7%.

Q5. Fairness of the SINDy comparisons and the practical performance advantage.

Given the correct column, sparse regression and WSINDy estimate coefficients better. The method helps when the law has no sparse representation in the library. Two cases are in the paper. A polynomial library cannot span sin⁡(u)\sin(u) over the full period of the collision (forming RMS 0.707 at every noise level, including zero). A damping term κtanh⁡(νut)\kappa\tanh(\nu u_t) with unknown ν\nu does not lie in a finite set of columns, and the free-form edges recover it correctly.

In Messenger & Bortz, “Weak SINDy for partial differential equations,” Journal of Computational Physics 443:110525 (2021), doi: 10.1016/j.jcp.2021.110525 (not the arXiv version), there are circumstances where WSINDy can return spurious terms.

In Appendix B.6, Eq. (B.9), p. 24 of the published JCP version, it is described that with standard STLS thresholding instead of their modified MSTLS, WSINDy returns a mixture of the true term and its Taylor expansion, roughly:

α(u−16u3+⋯ )+(1−α)sin⁡(u)\alpha\left(u - \tfrac{1}{6}u^3 + \cdots\right) + (1-\alpha)\sin(u)

So the polynomial terms partly stand in for sin⁡(u)\sin(u), even though sin⁡(u)\sin(u) is in the library and can be resolved through a choice of thresholding methods.

In our benchmark, we didn’t use the MSTLS thresholds reported in their paper; we used some of PySINDy’s threshold settings, which may have skewed this benchmark. Likewise, “re-sparsifying” both KANDy and Higher-Order-KANDy models by applying MSLS or STLSQ changes the results and often returns machine-precision coefficients. We will re-evaluate this benchmark in the final version of the paper. As KAN-based models and data-driven dynamical systems methods converge, we believe a more coherent comparison will contribute positively to the literature.

We will tone down our claim in the abstract and introduction regardless. We additionally added trig libraries to the STQLS benchmarks. Although these remain accurate estimates, the trig terms do not always return the correct function.

Our core thesis is that Higher-Order-KANDy performs under noise with fewer hyperparameters (in the data-driven dynamical systems sense, e.g., thresholds and dictionary terms), and this benchmark shows that. And, per your comment, unknowns remain, such as spline interpretability and architecture choice. This work attempts to apply proofs to motivate the choice of architecture, an underutilized property of KANs: constraints on the spline functions translate directly into hypotheses used to derive theoretical proofs about architectural capabilities.

Q5. Model dimensionality and practical application scenarios.

Our framework extends naturally to higher-dimensional oscillators and coupled systems. We tested it on up to 16 coupled Duffing oscillators and still recovered the equations without specifying the coupling term in advance. This additional benchmark will be in the final version of the paper.

Thanks for the clarifications and the additional experiments (including the direct KAN ablation, the improved derivative estimates and the extension to the coupled systems). That all helps, but my fundamental concerns are still there.

  • Architecture and interpretability (Q1, Q3). The direct KAN achieves similar prediction accuracy so the jet architecture's advantage is largely confined to interpretability. Composite features are still required for non-additive dynamics and symbolic interpretation is still dependent on domain informed fitting. The reliability and additional value of this interpretation is not yet established.
  • SINDy comparisons (Q4). The authors do admit that the threshold choices may have biased the benchmark and that sparse methods will in fact do a better job of estimating the coefficients if the correct terms are available. A corrected comparison using consistent symbolic support evaluation is still to be carried out and so the claimed advantage is not settled.
  • Derivative supervision (Q2). The denoising results are quite promising, but exceeding the training targets doesn't prove anything in the absence of dense calibration trajectories or any real derivative supervision during training.

These responses do clarify the approach in terms of its scope but still don't provide enough to allay my concerns about its further contribution and practical benefit. I therefore remain at my current rating.

Reviewer_Z5AfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Higher-Order-KANDy, which is an extension of the KANDy algorithm that combines the SINDy algorithm for symbolic equation discovery from times series data with Kolmogorov-Arnold networks (KANs). KANDy aims to improve SINDy's dependence on an adequate candidate term library by expressing the target dynamics as a linear combination of the terms passed through a single layer KAN. The single layer KAN acts as an interpretable learnable activation, and allows one to discover terms like sin⁡(x2)\sin(x^2) even when the candidate library only contains monomial terms of xx. The paper's Higher-Order-KANDy expands on this architecture by making it operate on not just the state variables themselves, but also their higher-order derivatives. Since the algorithm is supplied with only (noisy) state trajectories, the paper proposes two avenues for obtaining the higher-order derivative information: 1) based on time-delay embedding techniques, use an 1-layer KAN to map a vector of the original state and m−1m-1 time delays to the m−1m-1 jet; 2) fitting smoothing interpolants to the trajectory data, and recovering derivatives from the fitted interpolant. Experiments on ODE and PDE systems show that Higher-Order-KANDy can discover unknown equations from data, can express nonlinear functions not present in the candidate library through KAN activations, and these nonlinear functions can be recovered from the learned KANs.

Critical strengths

The idea of leveraging the interpretability of single layer KANs to improve upon the candidate library dependency of SINDy is an interesting idea and merits further exploration. The paper also expands on the original KANDy paper through experiments on various additional systems.

Critical weaknesses

Limited novelty over KANDy algorithm; lack of comparisons against KANDy

Higher-Order-KANDy is essentially KANDy algorithm operating on a mm-jet of the state variables instead of just the state variables themselves. The KANDy algorithm structure, its ability to discover nonlinear terms outside the candidate library, method to interpret the learned KAN activations are all present in the original paper. Therefore, the significance of this work has to be assessed by the following criteria, which I believe are insufficiently addressed in the current submission.

Motivation / justification for running KANDy on higher-order derivatives than just the state variables themselves.

The authors' motivation is that they aim to discover nn-th order differential equations, which is a function of the state xx and its higher-order derivatives - hence their usage of the nn-jets. However, an nn-th order differential equation can be simply expressed as nn coupled first-order differential equations, which is exactly the reason why architectures such as neural ordinary differential equations are formulated using first derivatives only. The authors briefly state in the introduction that converting to first-order form "expands the dictionary and the room for error". However, this is not a rigorous argument and never justified experimentally.

Comparisons against KANDy modeling nn first order equations is needed to validate this claim, but is absent from the benchmarks.

Novelty of identifying derivative information from data

If the derivative identification scheme was highly nontrivial, this could also serve as another point of novelty. However, both time-delay embedding and differentiating a fitted spline or Savitsky-Golay filter are well established techniques in the field, and cannot be deemed original. Furthermore, each of these techniques have further potential concerns, which I address in the rest of the Critical Weaknesses and Questions sections.

Strong dependence of the algorithm on smoothing interpolants

To estimate the higher-order derivatives of the state trajectories, one of the strategies used is to fit smoothing interpolants (splines, Savitsky-Golay filter) on the data and use the analytical derivatives of these interpolants instead. The authors claim their resulting technique is robust, but such scheme means that the derivative estimations are strongly affected by the hyperparameters of the smoothing algorithm. In particular, the smoothing interpolants used in the study are of the polynomial type - so their order directly affects the maximum expressible derivative order. The smoothing strength will also have a direct consequence on the accuracy of the obtained derivatives. The authors do not provide adequate information on how to determine these quantities given only noisy trajectory data, which greatly discounts their noise robustness claims.

I am also concerned about the way higher order derivatives are estimated - by repeatedly differentiating the first-derivative (velocity) estimator. It is well-known that a model well-fitting a function may result in inaccurate derivative predictions when differentiated, especially when noise is present or data is sparse [1, 2]. A proper characterization of the model's sensitivity and the accuracy of estimated higher-order derivatives with respect to smoothing parameters and noise strengths is required to dispel this concern.

  • [1] W. Czarnecki et al. Sobolev training for neural networks. NeurIPS (2017).
  • [2] F. Latremoliere et al. Estimating the Jacobian matrix of an unknown multivariate function from sample values by means of a neural network. arXiv: 2204.00523 (2022).
Questions for authors

Q1.

As the authors also state, the 1-KANDy equation layer for the jet estimation is essentially learning a sophisticated finite difference scheme. Is there a reason to use KANs for this purpose? What about 1D convolutional neural networks (or other neural network types used in the literature to estimate derivative operators)?

Q2.

Please provide details on the derivative estimation procedure. For delay embedding, how was the delay lag and the number of delays determined? For smoothing, how were the smoothing hyperparameters (order of the piecewise polynomial, smoothing strength, etc.) determined? Can these be done using only noisy data?

Q3.

Related to Q2, is there a way to choose the maximum derivative order of the target equation using data only? Can one set it to be sufficiently high and expect the algorithm to discard unnecessarily high orders?

2 comments
AuthorsOpenReview ↗

Thank you reviewer!

Dear Reviewer Z5Af,

Thank you for the careful review. We ran two new experiments to address your points for the final version.

We agree that an nth-order ODE has an equivalent first-order companion form. Our motivation is to reduce the reliance on the symbolic dictionary and a priori reductions when discovering dynamical systems from data. Fitting higher-order ODEs through a first-order reduction requires imposing known derivative identities, which is hard in practice.

The contribution over KANDy is constructing coordinates from delay and jet embeddings that capture the dynamical system from partial and noisy observations. Likewise, the inclusion of higher-order terms yields better interpretability and generalization, as observed in neural ODEs and Scientific Machine Learning; in Norcliffe et al. [1], the authors find that first-order models can capture higher-order dynamics, “but at the cost of interpretability,” and in Gruver et al. [2], Hamiltonian Neural Networks (HNNs), better generalization is “the result of modeling acceleration directly.” We will keep this statement in the introduction and abstract.

Q1:

The KAN is required for the jet embedding because we sometimes want the model's symbolic readout for delay differential equations. In the jet embedding for the velocity of the Duffing oscillator, the splines in the 1-KANDy layer took a known form: each edge was a finite-difference weight plus a multiple of the restoring force, and one linear convolution is the case where the force term dropped. This level of interpretability is not available with CNNs or other opaque models.

Q2:

Table S5 reports the delay settings, including m=7m=7, τ=6\tau=6 for Genesio–Tesi. §2.2 specifies a degree-(5,5)(5,5) PDE spline with smoothing strength Nσ^2N\hat{\sigma}^2, where the noise estimate comes from temporal second differences. In general, prior art exists for choosing the optimal delay and embedding dimension using false nearest neighbors and the first minimum of mutual information.

Reinsch (1967) and Dierckx (1981, 1993), which are in our bibliography, demonstrate prior art for spline smoothing, and we do not claim to invent it.

Q3:

One can test the correct order of an ODE by performing a saturation test; the value of R2R^2 at saturation indicates the correct order.

[1] Norcliffe, Bodnar, Day, Simidjievski, and Liò, “On Second Order Behavior in Augmented Neural ODEs,” NeurIPS 2020, arXiv:2006.07220.

[2] Gruver, Finzi, Stanton, and Wilson, “Deconstructing the Inductive Biases of Hamiltonian Neural Networks,” ICLR 2022, arXiv:2202.04836.

Reviewer_Z5AfOpenReview ↗

Reply to the authors

Thank you for your answers. Here are some of my thoughts regarding each.

Motivation

I thank the authors for the clarification. The new references makes their stance clearer than the more vague phrase "expands the dictionary and the room for error" used in the manuscript. Since this is a key motivation of the paper, I think replacing that sentence with the arguments above will better outline the paper's philosophy.

Regarding KANDy, it seems that the original paper did not have a method to address higher order systems that are only partially observed. If that is the case, this would also be worth emphasizing in the paper as well, it is clear what weaknesses of the original algorithm this paper is trying to improve.

Q1

The response regarding the duffing oscillator is intriguing, but raises more questions.

  • I can't seem to find this result in the paper. Is it in the manuscript?
  • Each edge being a finite difference weight makes sense, but this can be captured with conventional neural networks (for example, [1] is an excellent paper). The novel KAN-specific contribution is the "multiple of the restoring force". How did the authors identity this decomposition? Why does this multiple of the restoring force help derivative estimation?

[1] Y. Bar-Sinai et al. Learning data-driven discretizations for partial differential equaitons. PNAS 116, 31 (2019).

Q2

I thank the authors for the information. However, my concerns regarding the sensitivity of the method's performance to the accuracy of the estimation results remain. Note that it is possible for the proposed method to be quite sensitive to the quality of the estimated derivatives since these are directly used in the loss function. Proper derivative estimation for real-world data with varying noise characteristics is quite non-trivial, so a sensitivity analysis would clarify the level of applicability of the proposed approach to real world data.

Q3

I am intrigued by the saturation test, but this does not seem to be present in the manuscript. I believe a proper discovery method should be able to gauge the most likely derivative order of the data.

Reviewer_1wJHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a delay embedding method for expanding KAN to recover the dynamical systems with high order terms, The method includes different cases for handling ODEs/PDEs separately, and the results show the robustness and validation of the proposed method. But the current method requires the knowledge of the highest order in the underlying dynamics and the knowledge of the non-linear interaction term between different orders, which restrict the application potential of the proposed method.

Critical strengths
  1. The whole framework is complete and adapted to different differential equation scenarios, which makes it a useful integrated package to be invoked in the machine learning community.

  2. The numerical results clearly show the advantages of the proposed method compared with the existing methods, such as SINDy or WSINDy.

Critical weaknesses

The results in the main text mainly focus on toy models where the models are well-known physical equations, and none of new properties of these equations are discoverd by the current method. The advantages of the proposed method compared to SINDy et al are clearly expected given the ability of KAN. Right now the proposed method is more like a combination of delay embedding and KAN, it is a solid work but does not provide novel or interesting ideas.

Questions for authors
  1. How to use the current method to discover mathematical laws from known equation, such as new conservation laws? Or How to apply the current method to better reconstruct the model from data?

  2. Can the authors provide theroetical analysis/explanation for the robustness of the proposed method when dealing with noisy perturbation?

1 comment
AuthorsOpenReview ↗

Thank you for your review

Dear reviewer,

Thank you for your review.

Q1:

The order of the equation does not have to be known. The final version of our paper includes a saturation test to determine the order of the derivative. If a third-order system has a higher R2R^2 than a second-order system, a practitioner would use a third-order system as Higher-Order-KANDy's target.

Although, applying these methods to unknown equations requires domain knowledge. We present our findings on known equations for validation purposes.

Given your feedback, we included an experiment for the final version that tests discovery of conservation laws. We applied a test to the sine-Gordon system where we trained our method on $u_{tt} = \varphi_1(u) + \varphi_2(u_{xx}) + \varphi_3(u_t) + \varphi_4(u_x) + \varphi_5(u_{xt})$, with five free-form edges. After pruning, only the conservative cux+R(u)cu_x + R(u) remains. The edges on utu_t, uxu_x, and uxtu_{xt} are zeroed out by the method, which leaves the conservative law utt=cuxx+R(u)u_{tt} = cu_{xx} + R(u). After refitting the two surviving edges, the conserved energy follows by integrating the learned edge V(u)=−∫0uR(s)dsV(u) = -\int_0^u R(s)ds and E=∫[12ut2+12cux2+V(u)]dxE = \int \left[\tfrac12 u_t^2 + \tfrac12 cu_x^2 + V(u)\right] dx. Because the system is known, the result can be scored: VV matches 1−cos⁡u1 - \cos u. Neither sin⁡\sin nor cos⁡\cos were provided to the method beforehand. However, per your concern, this is on a known system.

Q2

Yes. The supplement gives a bound for the delay and the jet embedding construction.

Proposition S3.3 (iv) shows that measurement noise is multiplied by a fixed constant when differentiation is applied to an estimate of a truncated delayed system. Using a finite-difference formula to compute derivatives amplifies noise by a factor of Δ−k\Delta^{-k}, where kk is the order of the derivative. The KAN delay-to-jet embedding map does not difference samples, so it has no such factor. So the noise in the estimated derivatives stays proportional to the measurement noise, and the constant is computable from the trained network.

The supplement's harmonic-oscillator remark (Remark S3.4) shows this more concretely. With two delays,

x˙(t)=ωx(t)cos⁡ωΔ−x(t−Δ)sin⁡ωΔ,\dot x(t) = \omega\frac{x(t)\cos\omega\Delta - x(t-\Delta)}{\sin\omega\Delta},

exactly, at any Δ\Delta. The noise amplification is about ωsin⁡(ωΔ)\omega\sin(\omega\Delta). At ωΔ=π/2\omega\Delta = \pi/2 it is ω\omega, with no dependence on Δ\Delta. It grows only as sin⁡(ωΔ)→0\sin(\omega\Delta) \to 0, where the two delays no longer determine the state.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.