acceptodds
Under review as a conference paper at ICLR 2027

Overcoming Stiffness Barriers in Neural ODE Training

Abstract

Vanishing sensitivities to fast decay rates can hinder shooting based training of stiff neural ODEs. We investigate how established representation and training techniques can be combined to address this difficulty. Specifically, we avoid differentiation through an initial value solve by combining a production rate parameterization with a derivative matching initialization and simultaneous collocation. Analytically, we show for a scalar decay model that transformed derivative matching retains a restoring gradient for overestimated rates and further that a joint collocation step maintains a nonvanishing log rate correction while the corresponding shooting step vanishes. Experimentally, we observe on six stiff benchmarks benefits from pairing the architecture and training procedure, including improved spectral recovery on Van der Pol and the Oregonator, although recovery remains mixed across systems. We also show via counterexample a separate identification limit: exact trajectory and derivative fits don't imply identification of fast Jacobian eigenvalues. Together, these results present an avenue for addressing vanishing sensitivities through training formulation, while showing why improvements in optimization must be evaluated separately from exact recovery of the underlying dynamics.

Primary area:
learning on time series and dynamical systems
Keywords:
Neural Ordinary Differential Equations, Stiff Dynamical Systems, Scientific Machine Learning, Collocation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_XxcyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new training strategy for stiff neural ODEs. Essentially, it combines several existing approaches from the field of (stiff) neural ODEs. A production/rate representation is chosen for the ODE, both to efficiently represent/learn large decay rates and because these rates are directly reflected in the Jacobian matrices. Further, the straight-forward scaling strategy from (Kim et al., 2021) is applied. The resulting model is referred to as StiffNet. Three training modes are investigated: (1) [D]erivative matching (training to fit derivatives, instead of the ODE solution), (2) [C]ollocation with Radau IIA (optimization of a piecewise-polynomial trajectory together with the network parameters, instead of step-wise integration), which comes with "shorter" and less vanishing-prone gradients, and (3) common [S]ingle Shooting.

Critical strengths

The scientific analysis is clearly written and sound. The topic is interesting and very relevant to the SciML community. The “Limitations” section is comprehensive and covers obvious points and more.

Critical weaknesses

The main point of criticism is that this work merely combines existing architectures, and the training section also compares three different existing strategies. To be fair, it doesn't claim anything else either. That's not necessarily a bad thing, since many areas of science work that way ("combining existing things"). Still, the contribution would be much more significant, for example, by filling at least one of the methological gaps that naturally occur when combining things. For example proposing a metric or sub-algorithm that defines the switching point between the training methods. In my opinion, the core contribution therefore shifts more toward an experimental comparison of the approaches (then proposing a new thing), although there are still some weaknesses in the experiment (see below). For these reasons, I rate it as "weak reject" with the prospect of acceptance if the experimental issues are addressed in the rebuttal.

Major points

  • The paper never says when the D training mode ends or how it hands over to S or C. This is essential for reproducibility and further because it affects the performance - and more important even the ranking - of the various approaches. Further, the information density with respect to relevant experiment parameters varies, like the optimizer for the experiment, tolerances, iteration counts, etc. are missing for some experiments. This needs to be improved to ensure the credibility and validity of the experimental validation.
  • StiffNet D -> C has median errors > 1 on VanDPOL, ROBER, HIRES and DS. This is worse than just predicting a flat zero line. Even though StiffNet performs better in 4 out of 6 cases here, the result isn't good, and this makes my first point mentioned above even more relevant.
  • Results discussion starts with Tab. 1, that says in the caption "Rober training trajectory". In the text the table is referenced by "ROBER test trajectory". This is an important difference. Also, keep "Rober" uppercase like in the rest of the document. Further, the mentioned values in the text don't fit the ones in the table.

Minor points

  • Statistical relevance is weak. Varied is only the random seed for network initialization, if I got this correctly. Fixed are the data, the initial condition, hyperparameters, and so on. At least varying initial conditions (to obtain different solution trajectories) could be implemented with little effort. Of course, you have to draw the line somewhere, but things that could have large impact (e.g. hyperparameters) should be mentioned or discussed, if not varied.
Questions for authors

When discussing (and experimenting with) collocation and single shooting, one might wonder why the "in-between" multiple shooting wasn't included in the comparison. Why is this?

3 comments

Thank you for the valuable feedback. We are glad you (likewise) found the topic interesting and relevant, and we address your concerns and suggestions below.


Contribution: Thank you for raising the question of what the combination contributes. The use of established components is deliberate. Fronk and Petzold (2025b) explicitly call for “fundamentally new tools or theoretical breakthroughs” to address vanishing sensitivities. As a result, we investigate how even existing training formulations can retain corrective signals despite this attenuation. One of our contributions is the constructive analysis of that possibility, including a nonvanishing initial log rate correction even when the trajectory variables can move, together with evaluation of the combined formulation on nonlinear benchmarks. Therefore, new training formulations are not necessary to obtain a nonvanishing corrective signal under the stated assumptions. We have made this motivation and scope of the contribution explicit in the introduction. Furthermore, we are careful to separate the optimization benefit from information required for spectral identification, which comes with its own distinct difficulties.


Point 1: Thank you for the concern about the experimental details. We have increased the amount of detail in Appendix C accordingly.

Derivative matching:

  • POLLU: 30000 Adam iterations
  • Others: 10000 Adam iterations

LR Schedule:

  • 1e-2 first 34%
  • 3e-3 second 25%,
  • 1e-3 third 25%,
  • 3e-4 final 16%.

Shooting solvers/tolerances

  • Rodas5 for ROBER (abstol = reltol = 10−1010^{-10}), POLLU (abstol = 10−2510^{-25}, reltol = 10−1010^{-10}), HIRES and OREGO (abstol = 10−1010^{-10}, reltol = 10−810^{-8});
  • Kvaerno5 for VDPOL (abstol = reltol = 10−710^{-7});
  • Rodas5P for Davis–Skodje (abstol = 10−1010^{-10}, reltol = 10−810^{-8}).

LM iterations: 150.

All tolerances, schedules, and budgets were fixed heuristically during development and the listed settings were fixed within each benchmark across seeds. In particular, the handoff uses the same warm start and fixed Adam iteration budget above: i.e. not a convergence test.


Point 2: Our interpretation of the results performing worse than the zero estimate is addressed in response https://openreview.net/forum?id=gh5KgPuULf&noteId=RS90HDdp4h.


Point 3: Thank you for catching the inconsistencies in Table 1. This is addressed in response https://openreview.net/forum?id=gh5KgPuULf&noteId=RS90HDdp4h.


Initial Conditions: We additionally evaluated ten perturbed initial conditions per benchmark, with five network seeds per condition, using the original training hyperparameters. Full median and minimum error tables are provided in Appendix D. Below are StiffNet D$\to$C’s success counts, comparing each run against the zero predictor for its one initial condition.

Table 17: https://imgur.com/a/mKQkBzP

The POLLU and VDPOL successes span all perturbed initial conditions while ROBER exhibits unexpected sensitivity. Five of the ten perturbed ROBER initial conditions lie near the quasi steady state value of y2y_2, reducing their initial displacement from the slow manifold. We suspect that fast transient exposure may contribute to the changed learning behavior (see Q4 of https://openreview.net/forum?id=gh5KgPuULf&noteId=3LGMTIGjkG).


Multiple Shooting: Thank you for bringing up this comparison. Our original comparison focused on whether simultaneous collocation can produce corrective signals when differentiation through an integrated trajectory produces vanishing rate sensitivities. Multiple shooting (MS) still differentiates through integrated trajectories, although over shorter intervals. For example, for scalar decay u˙=−au\dot u = -au, the flow over a segment of length HH has log-rate sensitivity ∂log⁡a(ze−aH)=−aHze−aH,\partial_{\log a}\big(ze^{-aH}\bigr) = -aHze^{-aH}, which vanishes as aH→∞aH \to \infty for fixed segment initial state zz. Thus, shortening the segments can mitigate attenuation but does not automatically remove the underlying mechanism.

Nevertheless, we agree that MS is a useful common alternative to shooting and should have noted why it was left out. We therefore evaluated it both directly (MS) and after derivative matching (D →\to MS) with the same network sizes, seed counts, iterations, timeouts, and architecture comparisons as the main experiments. We used a fixed segmentation rule across problems and 1,000 multiple shooting iterations. We evaluated five of the paper’s six benchmarks (POLLU was omitted due to computational time constraints).

The following table, https://imgur.com/a/WCI6r5e, reports the additional results, with StiffNet D →\to C included as a reference. StiffNet D →\to C retains the lowest median and minimum vector field errors on all five tested benchmarks. However, multiple shooting does improve on single shooting in several settings. Trajectory performance remains problem dependent. The full comparison and discussion are included in Appendix D.

I view the response positively, as meaningful improvements were made to the experiments (e.g., inclusion of additional hyperparameters, revision of the tables, and addition of the MS comparison). Nevertheless, there are still a few minor areas that need attention to make the experiment flawless, such as statistical analysis (quantiles?) and a justification for the choice of hyperparameters (as we know, this choice can skew a benchmark in one direction or the other). Even though not everything is perfect, I see many positive revisions and am adjusting my rating to “weak accept.” I expect the linked resources to be corrected and incorporated into the final version.

Thank you for the thoughtful follow up and for recognizing the improvements made during the discussion. We agree that reporting variability and explaining hyperparameter choices more clearly would strengthen the paper. We will include interquartile ranges alongside the median errors in the camera-ready and expand the appendix to better explain how we chose hyperparameters, which settings were shared across problems, and which were problem dependent. We will also make sure the corrected results in the linked resources are incorporated consistently into the final manuscript.

Thank you again for the constructive discussion.

Reviewer_zzMtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines a production–rate parameterization, derivative-matching initialization, and collocation to handle vanishing sensitivities in stiff Neural ODEs (NODE). The analysis shows how the proposed formulation can retain corrective signals when shooting sensitivities vanish. Experiments on six stiff NODE benchmarks show benefits, while a counterexample demonstrates that accurate trajectory and derivative fits do not guarantee spectral recovery.

Critical strengths
  • The paper addresses optimization difficulties, showing why accurate trajectory fitting alone is insufficient to establish modeling of fast dynamics.

  • The experiments cover 6 stiff dynamical systems with 50 runs per benchmark, providing a broad assessment of performance.

Critical weaknesses
  • Proposition 2 has a narrow scope, analyzing a scalar linear decay problem on a single Radau interval under a specific initialization and fixed positive damping. It remains unclear whether this mechanism extends to subsequent iterations or nonlinear, multidimensional neural ODE training, limiting the generality of the result.

  • The paper is somewhat difficult to follow on a first reading. A concise statement of the scope of the results would improve readability. For example, tiℓt_{i\ell} (Ln.203) and bℓb_\ell in Figure 2 are not clearly defined, which hinders understanding and requires readers to infer their meaning.

Questions for authors
  1. Can the authors justify the choice of sinh⁡\sinh and exp⁡\exp in the production-rate parameterization and explain their advantages over alternative transformations? (Eq. 4)
  2. What does “X” denote in Table 3? What is the difference between "-"?
  3. What motivates sharing the MLP between the production and rate branches mentioned in Eq.3 and Figure 1? How would using separate MLPs affect performance?
4 comments

Thank you for reviewing our paper and for the helpful comments!


Q1: The exponential parameterizes a positive rate magnitude, so the default rate term is negative and additive log rate changes produce multiplicative rate changes. The sinh map permits signed production, is approximately linear near zero, and represents large magnitudes with moderate outputs which is useful for stiff systems of varying magnitudes see Doppel and Votsmeier (2023): https://pubs.rsc.org/re/article/8/10/2620/826342/Efficient-neural-network-models-of-chemical.


Q2: In the submitted tables, “-” denotes that the run did not finish within the allotted eight hour time limit, and “X” denotes ill-defined Jacobian metrics due to null space degeneracies. This distinction appears at lines 376-377 in the original manuscript. We made an edit to the table that replaces the metric distinction by instead correcting the metric to exclude known conserved directions. The modified metric definition and the updated tables 2-3 are provided in response: https://openreview.net/forum?id=gh5KgPuULf&noteId=RS90HDdp4h.


Q3: Both branches depend on the same state, so a shared network provides common features without duplicating the feature extractor. Separate networks would change feature coupling and parameter allocation and we do not attribute reported improvements specifically to parameter sharing or lack thereof. Parameter counts are provided in the original document in lines 1007-1017.

Scope and notation: Proposition 2 guarantees a nonvanishing initial joint correction and objective decrease under the stated assumptions. It does not establish subsequent iteration convergence. We will state the scope concisely and define the Radau stage time, tiℓt_{i\ell}, and the quadrature weight, bℓb_\ell, alongside their first use.

Thank you for the clarifications, but I feel that the main weaknesses raised in my review remain insufficiently addressed. In particular, the response acknowledges the limited scope of Proposition 2 without explaining whether its insights apply to more general training settings. I appreciate the planned improvements to notation and readability, though these only partially address my concerns.

Ah yes, I think I understand the question about Proposition 2 better now. Thank you for following up. Our previous response emphasized the scope of the guarantee but did not sufficiently explain which insights extend to more general settings. Allow us to expand on our response.


Appendix E.3 of the original submission first considers the general nonlinear, multidimensional collocation problem. With the stage states held fixed, the parameter Jacobian of the dynamics residual depends directly on the local derivative ∂θfθ\partial_\theta f_\theta, rather than on sensitivities propagated through an integrated trajectory. This is a structural distinction that directly addresses the question of generality. Indeed, for nonlinear, multidimensional models, the collocation residual Jacobian avoids the propagation through an integrated trajectory responsible for the sensitivity attenuation analyzed by Fronk and Petzold (2025b). It therefore bypasses the specific mechanism that they identify as “a significant barrier for training and parameter identification,” and this property does not depend on the scalar specialization. However, by itself it does not guarantee a useful parameter update because joint optimization can reduce the residual by instead changing the trajectory variables. It is therefore important to derive the reduced parameter system after eliminating those variables (Equations (39)-(40)), which makes explicit how state parameter coupling and damping determine the remaining correction. Proposition 2 resolves this competition in the scalar decay setting. In particular, it establishes a nonvanishing initial log rate correction even when the trajectory variables can move, together with a decrease in the nonlinear collocation objective.


The choice of scalar decay follows the classical use of the Dahlquist test equation to isolate individual fast modes. As explained by Hairer and Lubich, this approach is motivated by linearization and diagonalization (4.2 of https://www.unige.ch/~hairer/preprints/pcam-ode.pdf). Specifically, infinitesimal perturbations along a nonlinear trajectory satisfy a linear variational equation. Freezing its Jacobian locally and (when possible) diagonalizing it gives scalar modal equations, with modes corresponding to negative real eigenvalues obeying the decay equation analyzed here. This explains the relevance of the scalar mechanism to fast modes in larger systems, although the neural parameterization can (of course) couple those modes and their training updates. Fronk and Petzold (2025b) use the same test equation to analyze the neural ODE sensitivity attenuation, further motivating our choice of this setting for a direct comparison of training corrections.


We also examine the relevance of this mechanism on the nonlinear benchmarks through the new diagnostics in Section 4.2 (see the response to Q1 of reviewer dwFb: https://openreview.net/forum?id=gh5KgPuULf&noteId=3LGMTIGjkG). For ROBER, collocation moves initially overestimated fast-rate estimates closer to the truth at 88% of the retained points, compared with 45% for shooting. The added results provide empirical evidence of the rate correction beyond the scalar example without establishing that the mechanism explains every improvement in the benchmarks.


Finally, Appendix E.3 also outlines a possible continuation argument for convergence of subsequent iterations beyond the initial step for the scalar fixed mesh setting. This would require additional control of the state configurations, damping, and sufficient decrease, including an argument for entry into the required region. We considered pursuing this extension while developing the paper, but decided that establishing these conditions and evaluating their practical implications warranted a separate, more extensive investigation. We nevertheless included the sketch to explain how the analysis might proceed and what remains to be established.


Thank you again for following up, these are important points and we appreciate the opportunity to clarify them.

Thank you for the follow-up. The structural argument is helpful, but the formal guarantee covers only the first step of a scalar linear problem and does not establish continued correction or convergence in nonlinear, multidimensional training in practice. I will therefore maintain my rating.

Reviewer_u1mbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the challenges of stiffness in neural ODE training. Instead of developing a better ODE solver, the authors propose reformulating the optimization process to overcome the difficulties caused by stiff dynamics. Specifically, the authors divide the problem into two challenges: the representation barrier, which determines what the vector field can express, and the sensitivity barrier, which determines whether training can reach the desired parameters.The proposed framework consists of three main components. First, the production–rate parameterization reformulates a standard MLP-based vector field as fθ(u)=Pθ(u)+Rθ(u)⊙uf_\theta(u) = P_\theta(u) + R_\theta(u) \odot u, where Rθ(u)R_\theta(u) is a state-dependent rate term and Pθ(u)P_\theta(u) is an additive term. This formulation provides a mechanism for representing large decay rates, while the additive term allows changes that are not proportional to the state. Second, the derivative-matching warm start directly uses finite differences to estimate derivatives from observed trajectories and initialize the network. Third, the authors employ simultaneous collocation, which jointly optimizes the normalized mesh states UU, normalized stage increments KK, and network parameters θ\theta. This allows the entire trajectory and network parameters to be optimized simultaneously, rather than relying on differentiation through an ODE solver.The authors demonstrate the effectiveness of their approach through both theoretical analysis and experiments on six stiff ODE benchmarks.

Critical strengths

1:The motivation of the paper is clear. Stiff dynamics in neural ODEs can lead to vanishing sensitivities, making training difficult. Addressing this problem is an important and meaningful research direction.

2: The authors provide insightful theoretical analysis of their proposed methods. Both propositions help explain why derivative matching and simultaneous collocation can mitigate the vanishing sensitivity problem.

3: The experiments are thorough, covering six stiff ODE benchmarks. The authors also compare different architectures and training strategies to evaluate the effectiveness of their framework.

Critical weaknesses

1: In the experiments, the authors mainly compare their method with Kim21 and a standard MLP. However, several important methods discussed in the related work section are missing from the experimental comparisons. Including these methods would provide a more comprehensive evaluation.

2: The current theoretical analysis is mainly based on scalar linear decay dynamics, while real neural ODE systems are often more complex and nonlinear. It is unclear whether the theoretical conclusions can be generalized to more complex systems.

3: The experimental results are not consistently strong across different benchmarks. Some error values are extremely large, indicating that the proposed method still struggles with certain stiff systems. This raises concerns about the robustness and reliability of the proposed framework.

Questions for authors

1: Can the authors provide comparisons with more baselines, especially methods that share similar components with the proposed framework?

2: Can the authors provide further analysis of how their methods perform in more general and complex scenarios?

3: Can the authors provide a clearer analysis of the extremely large error values reported in the experiments?

2 comments

Thank you for your comments and helpful questions.


Q1: Thank you for the suggestion. The closest method sharing our components is the collocation approach of Shapovalova & Tsay (2025). We implemented this method and ran it on the stiff benchmarks in the following table (https://imgur.com/a/9DvhzxQ), and found little improvement after the derivative matching stage, along with failure to converge on POLLU runs.

We also implemented and ran Multiple Shooting both directly (MS) and after derivative matching (D →\to MS) with the same network sizes, seed counts, iterations, timeouts, and architecture comparisons as the main experiments. Please see the response to reviewer Xxcy’s question 1 and for full details see Appendix D.3 (https://openreview.net/forum?id=gh5KgPuULf&noteId=czfyDDJliP). We have additionally attached the table, https://imgur.com/a/WCI6r5e, for reference.


Q2: Following Fronk and Petzold (2025b), we analyze the scalar decay equation which is a classical setting for stability analysis (Dahlquist, 1963). The scalar decay equation is commonly used for stiff ODE analysis because it isolates the fast rate sensitivity mechanism. Here, it additionally allows for an explicit comparison of the training corrections applied to fast rates. The six nonlinear benchmarks assess the combined method’s practical benefits and limitations in coupled systems.


Q3: This is a good point. Conserved zero modes make the original relative spectral metric ill-defined, and small normalization factors can inflate trajectory errors. To avoid this, we revised the tables to use mixed tolerances for trajectory errors so that error values are absolute near zero, and we omit eigenvalues that are identically zero. Not all values were significantly reduced under the change of metric. In particular, Kim21 ROBER trajectory error for both training methods presented a minimum error magnitude of 10^{17}. Full definitions and the updated tables 2-3 are provided in https://openreview.net/forum?id=gh5KgPuULf&noteId=RS90HDdp4h.

I really appreciate the detailed response. I will improve my rating to "week acceptance".

Reviewer_dwFbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses training neural ODEs on stiff systems. When a candidate decay rate is overestimated, shooting-based gradients with respect to that rate can vanish. The authors combine several existing components:

  • a production–rate vector field fθ(u)=Pθ(u)+Rθ(u)⊙uf_\theta(u) = P_\theta(u) + R_\theta(u)\odot u with a log-parameterized rate and data-derived output scaling (following Kim et al., 2021);
  • a derivative-matching warm start with an asinh-transformed loss;
  • simultaneous two-stage Radau IIA collocation solved with Levenberg–Marquardt, which avoids differentiating through an initial value solve.

For a scalar decay model, they show two things. First, the transformed derivative loss keeps a restoring gradient in the log rate. Second, the initial collocation step gives a non-vanishing log-rate correction, while the corresponding shooting step decays like (ha)−1(ha)^{-1}. A two-dimensional counterexample shows that exact trajectory and derivative fits need not identify a fast Jacobian eigenvalue.

Experiments on six stiff benchmarks (Van der Pol, POLLU, ROBER, Oregonator, HIRES, Davis–Skodje) compare the proposed network with an MLP and the Kim21 network. Each is trained with derivative matching followed by either shooting or collocation, and the paper reports trajectory, vector-field and spectral errors.

Critical strengths

I appreciate how the problem is framed. The paper separates three questions: what the vector field can represent, whether training can reach it, and whether the data determine it at all. This is a useful way to think about stiff neural ODEs, and the paper backs it up by reporting vector-field and spectral errors alongside trajectory error. The discussion is also candid about where the approach does not work.

The appendix analysis is careful. E.1 shows that the plateau is already present in the exact shooting objective, not only in the numerical solver. E.2 extends the derivative-matching result to inexact targets with explicit bounds (Prop. 4 and Eq. 29), and openly notes that the implemented filter does not guarantee the required accuracy.

When the combination works, it is also much cheaper. Table 9 reports about a minute for StiffNet with collocation on Van der Pol versus over 20 minutes for Kim21 with shooting, and POLLU is solved only by the proposed configuration.

Critical weaknesses

1. The analysis is careful, but no experiment shows that it describes what actually happens.

The components themselves are established (production–rate form, Kim21 scaling, derivative matching, Radau collocation), so the contribution rests on the analysis and on evidence that this pairing helps.

The causal story the paper suggests has three links:

  • centered differences overestimate fast rates (E.2 notes that the implemented filter admits aCD/a∗≈10.9a_{CD}/a_* \approx 10.9), so the warm start can land on the overestimated side;
  • shooting then stalls on the plateau of E.1;
  • collocation keeps a non-vanishing correction (Prop. 5).

Each link is plausible, and Prop. 5 is nicely matched to the initialization used in practice (C.3). However, none of them is checked experimentally. The paper does not report the learned rates after the warm start, whether they are overestimated, or the size of the shooting updates in D→S runs. Without this, statements such as Table 1 being "consistent with Proposition 2" remain interpretation.

Some of the paper's own results point the other way:

  • In Fig. 4, Kim21 with D+S reaches a median training error of around 10−710^{-7} on Davis–Skodje at ε=0.01\varepsilon = 0.01, so shooting does not stall there.
  • On HIRES and OREGO, StiffNet's median extrapolation error is lower with D→S than with D→C.

It is also not stated which optimizer trains the shooting stage. Prop. 5 compares both formulations under the same LM step. If the experiments use LM for collocation but a first-order method for shooting, the comparison conflates the formulation with the optimizer. Finally, Prop. 5 covers only the first LM step, as the authors acknowledge.

2. The reported numbers don't support clear conclusions.

Table 1. This is the main evidence for the combination claim, but it is inconsistent with both the text and Table 3. Its D→S and D→C columns coincide with Table 3's ROBER row (minimum extrapolation error), with two exceptions: MLP and Kim21 are swapped under D→S, and StiffNet's D→C entry is 0.00436 here but 4.361e-02 in Table 3. The caption says training trajectory, the text says test trajectory, and the text reports a third set of values (0.113 for D→C, 1.293 for D→S).

Metrics. Eqs. 13–15 normalize each species, and each eigenvalue index, by its own norm over the evaluation interval. On the extrapolation intervals many species have decayed to near zero, so tiny absolute deviations become enormous relative errors. Davis–Skodje makes this concrete. At ε=0.01\varepsilon = 0.01, Fig. 4 reports a median training error of around 10−1110^{-11} for StiffNet with D+C. Yet its median extrapolation error in Table 2 is 41.7, and only 4/50 runs count as successful in Table 9, on an interval where both states are below about 10−610^{-6}.

ROBER and HIRES also have linear conservation laws (y1+y2+y3y_1+y_2+y_3 and y7+y8y_7+y_8), so their Jacobians carry a zero eigenvalue along the whole trajectory. This is the same degeneracy flagged with "X" for POLLU, and it likely explains the spectral errors of 102110^{21}–102910^{29} seen for every architecture. Tables 2–3 are therefore hard to read in either direction.

Reliability and selection. Even taking the metric at face value, Table 9's success rates are low: 16/50 on Van der Pol, 18/50 on ROBER, 11/50 on OREGO, 4/50 on Davis–Skodje and 0/50 on HIRES. Table 3 reports per-metric minima over 50 seeds. This selects on test performance and may combine different runs for different metrics. The clean μ=100\mu=100 fit in Fig. 3 is presumably one of the 16 successful seeds, whereas Fig. 4 shows the median training error at that stiffness close to the baseline's.

Attribution. StiffNet with shooting is missing from Table 9, and Figs. 3–4 compare StiffNet with collocation against Kim21 with shooting. Architecture and training therefore can't be separated.

Setting.

  • The data are noise-free, and derivatives come from finite differences.
  • For Van der Pol and OREGO, the observations sit at the adaptive steps of a reference stiff solver, i.e. densely placed at the fast transitions.
  • Each model is trained on a single trajectory, which by E.4 can leave the fast modes unidentifiable.
  • The networks have 268–4,744 parameters and are trained with dense LM.
  • None of the stiff-specific training methods from the related work is included as a baseline.
Questions for authors
  1. After the warm start, are the learned rates overestimated, as the finite-difference bias in E.2 suggests? In D→S runs, are the shooting updates small while D→C corrects the rates? And which optimizer trains the shooting stage? These diagnostics are the result most likely to change my assessment.
  2. Could you reconcile Table 1 with the text and with Table 3 (training vs. test trajectory, the swapped entries, and 0.00436 vs. 4.361e-02)? Do Table 3's minima for different metrics come from the same run?
  3. How do Tables 2, 3 and 9 change with absolute or log-scale errors, and with conserved directions removed from the spectral metric?
  4. In an identifiable setting (several initial conditions, or perturbations as in E.4), with noisy data and non-adaptive sampling, does collocation recover the fast eigenvalues where shooting does not?
1 comment

Thank you for the thoughtful comments and for recognizing the distinction between representation, optimization, and identifiability. We address your questions below.


Q1: Thank you for suggesting these useful diagnostics. The following Table 4 (https://imgur.com/a/albMxAH) shows that the StiffNet overestimates ROBER’s fast rate after derivative matching in 49/50 runs. After shooting, 49/50 remain classified as overestimated, whereas after collocation, 30/50 fall within the ±0.05\pm 0.05 decade (i.e. 100.0510^{0.05}) tolerance, compared with 1/50 after D and D$\to$ S. The signed log rate error changes from +0.52 decades after D to +0.42 after shooting and +0.01+0.01 after collocation. On initially overestimated visible points, collocation moves the rate closer to the truth at 88% of points, compared with 45% for shooting.

This gives direct empirical examples of stronger rate correction from collocation. The behavior is problem dependent, and moreover we agree that these results and the analysis in the appendix establish a potential mechanism rather than the cause of every benchmark gain. We therefore remove “consistent with proposition 2” from the Table 1 discussion and added these new diagnostics to the main paper as Table 4, with accompanying analysis in Section 4.2. Shooting uses Adam (lr 5×10−35 \times 10^{-3}, 1000 iterations) with forward mode AD while collocation uses Levenberg-Marquart. Importantly, we note that Proposition 5 takes care to hold the LM rule, Radau formula, log rate coordinate, and positive damping fixed. This allows for a direct comparison between the initial collocation correction which tends to a nonzero negative constant, whereas the shooting correction is O(s−1)O(s^{-1}). We have attached a figure for reference (now Figure 3 in the main body): https://imgur.com/a/T7ApUb2. Thus, the theoretical distinction remains when the optimizer is held fixed, although we recognize that optimizer choice can influence the benchmark results.


Q2: Table 3 takes the minimum independently for each metric, so its entries need not come from the same run. We have made this explicit in the table caption. For updated tables and table corrections please see response https://openreview.net/forum?id=gh5KgPuULf&noteId=RS90HDdp4h.


Q3: The revised metric definitions, zero baselines, and updated tables are provided in response: https://openreview.net/forum?id=gh5KgPuULf&noteId=RS90HDdp4h. In addition to the tables, we have attached the additional spectral fit plots that were not included in the original submission (https://imgur.com/a/pV8mMKs). These plots further illustrate our original claim on lines 86-87: “We demonstrate, through a counterexample and numerical experiments, that accurate trajectory reconstruction can coexist with poor recovery of fast Jacobian eigenvalues.” In particular, removing the zero eigenvalues does not change that takeaway.


Q4: Thank you for suggesting the additional experiment. We added a ROBER comparison to examine how exposure to an identifiable fast transient affects eigenvalue recovery under derivative matching, shooting, and collocation. Recovery improves substantially when the training data exposes the fast relaxation, but we do not observe a consistent spectral advantage of collocation over shooting. This is consistent with the distinction between useful optimization updates and having sufficient information to identify fast modes.

Each initial condition uses five seeds on a fixed non adaptive sampling grid. At three fixed values of y3(0)y_3(0), we set y2(0)=(1+δ)y2∗y_2(0) = (1+\delta)y_2^* with offsets δ=0\delta = 0 and 22. At δ=0\delta=0, production and consumption is approximately balanced, placing the initial state near the slow manifold. The displaced start δ=2\delta =2 exposes a fast relaxation that is largely absent from the δ=0\delta = 0 trajectory.

During the visible transient all three training approaches recover the fast rate closely, with much of their recovery already achieved by derivative matching.

Table 18: https://imgur.com/a/d4z5JP9

At the later quasi-steady training times, the models trained from δ=0\delta = 0 substantially underestimate the fast rate. Exposure to the initial transient improves the subsequent estimates across all three approaches, although accurate recovery during the transient does not extend uniformly to these later states.

Table 19: https://imgur.com/a/O8r3cdP

The results support the importance of fast transient exposure for spectral recovery. They do not show that collocation consistently recovers the fast eigenvalues that shooting misses. The results therefore reinforce the paper’s distinction between accurate trajectory fitting and accurate Jacobian recovery.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback. The questions raised several directions we were eager to investigate, motivating additional experiments and analysis that deepen our understanding of the method’s behavior. We have aimed to address each question as fully as possible, and the results below extend the evaluation and clarify the scope of the paper’s central claims.


The inconsistencies in Table 1 and its description as both a training trajectory and a test trajectory was determined to be a result of human error when transferring the results into TeX. We have corrected the table, the caption, and accompanying discussion with the verified results. See the corrected table (https://imgur.com/a/wYyQqHP).


We have also revised the metrics in Tables 2, 3, and 9 (https://imgur.com/a/eC0gBek) to reduce sensitivity to near zero reference values. The denominators now include a floor based on the training data scale, and the spectral metric excludes eigenvalues that are identically zero by conservation. Definitions can be found in Appendix D.1. For completeness, consider the following metric definitions used in the updated tables:

\qquad a_j = 10^{-3}\max_{t\in[T_0,T_1]} |q_j(t)|.$$ The floor $a_j$ acts as a scale dependent tolerance to mitigate divide-by-zero errors. For trajectory error, vector field error, and spectral error the inputs to $E(\cdot, \cdot)$ are true vs. estimated trajectories, vector fields and eigenvalues, respectively. \ **On the reliability of extrapolation (was Table 9 now Table 11)**. StiffNet achieves very low training errors, as shown in the corrected Table 1 and trajectory plots. Kim et al. (2021) likewise demonstrated successful fitting of stiff trajectories. Our evaluation additionally examines extrapolation beyond the observed interval. StiffNet with collocation improves on the zero prediction in 36/50 POLLU runs and 26/50 VDPOL runs, compared with 0/50 and 1/50, respectively, for Kim21 with the same training. These are substantial improvements on difficult extrapolation tasks even though they do not reach a 100% success rate. These improvements should be considered in the context of the difficulty of training stiff neural ODEs. Kim et al. (2021) describe learning these systems as “very challenging,” while Caldana and Hesthaven (2025) state that “it is a well known fact that neural ODE training may fail due to stiffness”. Fronk and Petzold’s (2025b) analysis further explains how sensitivities to fast decay rates can vanish during shooting based training. They state that vanishing sensitivities are a “universal feature rooted in the mathematics of stable stiff integration methods,” concluding that their findings pose “a significant barrier for training and parameter identification in stiff neural ODEs”. Our contribution provides both a constructive analysis of how existing formulations retain corrective signals and empirical improvements in trajectory fitting and extrapolation. Success remains problem dependent, but this does not negate the demonstrated improvements over the evaluated baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.