- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a new training strategy for stiff neural ODEs. Essentially, it combines several existing approaches from the field of (stiff) neural ODEs. A production/rate representation is chosen for the ODE, both to efficiently represent/learn large decay rates and because these rates are directly reflected in the Jacobian matrices. Further, the straight-forward scaling strategy from (Kim et al., 2021) is applied. The resulting model is referred to as StiffNet. Three training modes are investigated: (1) [D]erivative matching (training to fit derivatives, instead of the ODE solution), (2) [C]ollocation with Radau IIA (optimization of a piecewise-polynomial trajectory together with the network parameters, instead of step-wise integration), which comes with "shorter" and less vanishing-prone gradients, and (3) common [S]ingle Shooting.
Critical strengths
The scientific analysis is clearly written and sound. The topic is interesting and very relevant to the SciML community. The “Limitations” section is comprehensive and covers obvious points and more.
Critical weaknesses
The main point of criticism is that this work merely combines existing architectures, and the training section also compares three different existing strategies. To be fair, it doesn't claim anything else either. That's not necessarily a bad thing, since many areas of science work that way ("combining existing things"). Still, the contribution would be much more significant, for example, by filling at least one of the methological gaps that naturally occur when combining things. For example proposing a metric or sub-algorithm that defines the switching point between the training methods. In my opinion, the core contribution therefore shifts more toward an experimental comparison of the approaches (then proposing a new thing), although there are still some weaknesses in the experiment (see below). For these reasons, I rate it as "weak reject" with the prospect of acceptance if the experimental issues are addressed in the rebuttal.
Major points
- The paper never says when the D training mode ends or how it hands over to S or C. This is essential for reproducibility and further because it affects the performance - and more important even the ranking - of the various approaches. Further, the information density with respect to relevant experiment parameters varies, like the optimizer for the experiment, tolerances, iteration counts, etc. are missing for some experiments. This needs to be improved to ensure the credibility and validity of the experimental validation.
- StiffNet D -> C has median errors > 1 on VanDPOL, ROBER, HIRES and DS. This is worse than just predicting a flat zero line. Even though StiffNet performs better in 4 out of 6 cases here, the result isn't good, and this makes my first point mentioned above even more relevant.
- Results discussion starts with Tab. 1, that says in the caption "Rober training trajectory". In the text the table is referenced by "ROBER test trajectory". This is an important difference. Also, keep "Rober" uppercase like in the rest of the document. Further, the mentioned values in the text don't fit the ones in the table.
Minor points
- Statistical relevance is weak. Varied is only the random seed for network initialization, if I got this correctly. Fixed are the data, the initial condition, hyperparameters, and so on. At least varying initial conditions (to obtain different solution trajectories) could be implemented with little effort. Of course, you have to draw the line somewhere, but things that could have large impact (e.g. hyperparameters) should be mentioned or discussed, if not varied.
Questions for authors
When discussing (and experimenting with) collocation and single shooting, one might wonder why the "in-between" multiple shooting wasn't included in the comparison. Why is this?
3 comments
I view the response positively, as meaningful improvements were made to the experiments (e.g., inclusion of additional hyperparameters, revision of the tables, and addition of the MS comparison). Nevertheless, there are still a few minor areas that need attention to make the experiment flawless, such as statistical analysis (quantiles?) and a justification for the choice of hyperparameters (as we know, this choice can skew a benchmark in one direction or the other). Even though not everything is perfect, I see many positive revisions and am adjusting my rating to “weak accept.” I expect the linked resources to be corrected and incorporated into the final version.
Thank you for the thoughtful follow up and for recognizing the improvements made during the discussion. We agree that reporting variability and explaining hyperparameter choices more clearly would strengthen the paper. We will include interquartile ranges alongside the median errors in the camera-ready and expand the appendix to better explain how we chose hyperparameters, which settings were shared across problems, and which were problem dependent. We will also make sure the corrected results in the linked resources are incorporated consistently into the final manuscript.
Thank you again for the constructive discussion.
Thank you for the valuable feedback. We are glad you (likewise) found the topic interesting and relevant, and we address your concerns and suggestions below.
Contribution: Thank you for raising the question of what the combination contributes. The use of established components is deliberate. Fronk and Petzold (2025b) explicitly call for “fundamentally new tools or theoretical breakthroughs” to address vanishing sensitivities. As a result, we investigate how even existing training formulations can retain corrective signals despite this attenuation. One of our contributions is the constructive analysis of that possibility, including a nonvanishing initial log rate correction even when the trajectory variables can move, together with evaluation of the combined formulation on nonlinear benchmarks. Therefore, new training formulations are not necessary to obtain a nonvanishing corrective signal under the stated assumptions. We have made this motivation and scope of the contribution explicit in the introduction. Furthermore, we are careful to separate the optimization benefit from information required for spectral identification, which comes with its own distinct difficulties.
Point 1: Thank you for the concern about the experimental details. We have increased the amount of detail in Appendix C accordingly.
Derivative matching:
LR Schedule:
Shooting solvers/tolerances
LM iterations: 150.
All tolerances, schedules, and budgets were fixed heuristically during development and the listed settings were fixed within each benchmark across seeds. In particular, the handoff uses the same warm start and fixed Adam iteration budget above: i.e. not a convergence test.
Point 2: Our interpretation of the results performing worse than the zero estimate is addressed in response https://openreview.net/forum?id=gh5KgPuULf¬eId=RS90HDdp4h.
Point 3: Thank you for catching the inconsistencies in Table 1. This is addressed in response https://openreview.net/forum?id=gh5KgPuULf¬eId=RS90HDdp4h.
Initial Conditions: We additionally evaluated ten perturbed initial conditions per benchmark, with five network seeds per condition, using the original training hyperparameters. Full median and minimum error tables are provided in Appendix D. Below are StiffNet D$\to$C’s success counts, comparing each run against the zero predictor for its one initial condition.
Table 17: https://imgur.com/a/mKQkBzP
The POLLU and VDPOL successes span all perturbed initial conditions while ROBER exhibits unexpected sensitivity. Five of the ten perturbed ROBER initial conditions lie near the quasi steady state value of y2, reducing their initial displacement from the slow manifold. We suspect that fast transient exposure may contribute to the changed learning behavior (see Q4 of https://openreview.net/forum?id=gh5KgPuULf¬eId=3LGMTIGjkG).
Multiple Shooting: Thank you for bringing up this comparison. Our original comparison focused on whether simultaneous collocation can produce corrective signals when differentiation through an integrated trajectory produces vanishing rate sensitivities. Multiple shooting (MS) still differentiates through integrated trajectories, although over shorter intervals. For example, for scalar decay u˙=−au, the flow over a segment of length H has log-rate sensitivity ∂loga(ze−aH)=−aHze−aH, which vanishes as aH→∞ for fixed segment initial state z. Thus, shortening the segments can mitigate attenuation but does not automatically remove the underlying mechanism.
Nevertheless, we agree that MS is a useful common alternative to shooting and should have noted why it was left out. We therefore evaluated it both directly (MS) and after derivative matching (D → MS) with the same network sizes, seed counts, iterations, timeouts, and architecture comparisons as the main experiments. We used a fixed segmentation rule across problems and 1,000 multiple shooting iterations. We evaluated five of the paper’s six benchmarks (POLLU was omitted due to computational time constraints).
The following table, https://imgur.com/a/WCI6r5e, reports the additional results, with StiffNet D → C included as a reference. StiffNet D → C retains the lowest median and minimum vector field errors on all five tested benchmarks. However, multiple shooting does improve on single shooting in several settings. Trajectory performance remains problem dependent. The full comparison and discussion are included in Appendix D.