acceptodds
Under review as a conference paper at ICLR 2027

Cosine Tolerance Annealing for Neural ODE Training: A Study of Cost and Accuracy

Abstract

Neural Ordinary Differential Equations (Neural ODEs) are usually trained with adaptive solvers whose tolerances stay constant for the whole run. Loose tolerances keep the number of function evaluations (NFE) low but give less accurate solutions; tight tolerances are accurate but expensive at every iteration. We study Cosine Tolerance Annealing (CTA), which tightens the relative and absolute tolerances from loose initial values to tight final values along a cosine curve over the first 60% of training, and compare it with constant tolerances and other schedules. On a synthetic cubic system with the Dormand-Prince solver and ten seeds, CTA has on every seed a lower error, averaged over the last ten checkpoints, than each of the four constant tolerances we tested; relative to the tight final tolerance (Fixed-Final), it lowers this error by 62% and the training NFE by 21%. Across five synthetic settings, CTA needs 13-40% fewer training NFE than Fixed-Final, with an error that is lower in two settings and not significantly different in the other three; a linear schedule is not significantly different from the cosine one in four of them. On MNIST (three seeds) and Jena Climate forecasting (ten seeds), CTA reaches an accuracy similar to that of Fixed-Final with 22.8% and 17.8% fewer function evaluations per training iteration, but a constant tolerance between the initial and final values (Fixed-Geometric) reaches a similar accuracy at a lower average cost. On the two synthetic systems, however, this constant tolerance is significantly less accurate than Fixed-Final, and CTA has a lower error than it in four of the five settings, so CTA is useful when it is not known in advance whether a loose constant tolerance is accurate enough.

Primary area:
learning on time series and dynamical systems
Keywords:
Neural ODEs, adaptive ODE solvers, tolerance scheduling, numerical optimization, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9X2DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors observe, that, under convex optimization, the earlier gradient errors matter geometrically less than the final ones. Therefore, one should focus the compute on the final optimization steps. This is possible to implement directly for Neural ODEs, via scheduling the tolerance of the method.

Empirically, the authors demonstrate the superiority of their simpler scheduling approaches over fixed-tolerance ones -- on a cubic synthetic dataset. Then, the authors compare these approaches on real data -- Jena weather and MNIST. On real data, the results are mixed, with a fixed-tolerance approach (FixedGeom) taking the lead (training time being equal).

Critical strengths

This is a very interesting application niche for Neural ODEs. Traditional methods do not allow such explicit & simple regulation of a method's capacity, meaning that they cannot utilize the optimization theory insight into an error weight, that changes throughout training. Simple synthetic cubic data shows it well: in the absence of noise and the complexity of real data, scheduling mostly helps.

Critical weaknesses

Contribution. The tolerance scheduling idea already exists, as you state yourself in related work. Ott et al. (2021) already proposes to adaptively reduce tolerances throughout training, but they aim to achieve a different goal -- consistency with different solvers; to make the ODE an actual ODE -- not inspired by the "geometrically decaying loss weights" bound. That makes your bound-based motivation the main contribution and the novelty of this paper, which suffices, in my opinion. However, this bound motivation is only weakly supported by the presented evidence (theoretical and practical). Therefore, I believe that this work is incomplete without at least some confirmation of this mechanism, and some application of it. I would like to see some practical benefit which comes from this mechanism, turning it from just an interesting observation into a portable finding.

Then, some other observations:

  • Your motivation does not explain, why the cosine schedule leads over the geometric one. For a schedule-proposing paper, I would like to see some schedule comparison and explanation there.
  • Since tolerance is a hyperparameter, it should be selected & tuned per-dataset, so, at the moment, comparison to a fixed method which is not the leader (fixed-final, for instance) is not really fair: comparison should be against fixed-best, wherever you claim to outperform fixed methods.
Questions for authors

Contribution. I would like to provide an example of what I mean by "deeper analysis of the optimiser mechanism": measuring errors at different states during training, comparing that curve to what the theory predicts, making conclusions about the given datasets. Because, on real data, we observe that your scheduling loses to a carefully-tuned tolerance. Then, it would be interesting to see some additional modification of the integration procedure, inspired by that analysis.

A positive result here would make me raise my mark.

Besides, it would be preferable to fix the other observed weaknesses:

  • Analyze why, how and where the schedules differ -- to produce some portable finding here as well.
  • Compare your schedules to the fixed-best model to the text and abstract -- a fixed-final comparison is not fair.
Reviewer_JXD2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new method called Cosine Tolerance Annealing (CTA) whose purpose is to change the absolute and relative tolerance of the numerical adaptive solver that is used for training a Neural Ordinary Differential Equation (NeuralODE). The goal is to combine the advantages of cheaper training steps with looser tolerances and accurate results with tighter ones by scheduling the tolerances during a parameter-defined range of training. The paper reports that this approach needs 13-40% fewer function evaluations across five synthetic settings in comparison to training with the fixed tight tolerances and 22.8/17.8% fewer function evaluations on MNIST and Jena with no significant loss of accuracy.

Critical strengths
  • The paper covers a wide range of experiments from synthetic Cubic and Lotka-Volterra to public benchmarks MNIST and Jena with multiple seeds used for creating the corresponding results with results for each seed in the appendix.
  • Results are reported for all experiments even when they show that CTA is not always the correct approach to solving the problem. For MNIST and Jena the Fixed-Geometric approach is cheaper at a similar accuracy.
  • Since the algorithm for CTA and the used model and hyperparameters are provided the results can be easily reproduced.
Critical weaknesses
  • The choice of cosine annealing is not supported by the data:
    • CTA and Linear flip between being better than the other one in the four Cubic experiments (Table 1, 10, 11) with no significant difference. One could conclude that there is no conclusive advantage to using CTA over Linear.
  • The baselines Ott et al. (2021) and InfoCNF are named but never run, so there is no comparison between these three adaptive tolerance methods with regard to the accuracy of the results.
  • On Cubic with Dopri5 CTA performs significantly better than Fixed-Final which uses the final tolerances of CTA, but this finding is, although reported, never explained. It would be very interesting why training with an initial loose tolerance leads to better results than training with a tighter tolerance for the whole training.
  • There are several points in the paper that point to a weak comparison of accuracy and cost saving:
    • In section 3.2 it is mentioned that for the synthetic systems the training trajectory is used for evaluating the system, so the accuracy improvement of CTA could be due to fitting to the data. Although the model is not trained on it directly it is still the same trajectory.
    • For Jena, on 5 of 10 seeds the best checkpoint of CTA is before the end of annealing when its tolerances are still looser than Fixed-Final and the NFE are reported only up to that checkpoint. This means that on those seeds the speedup comes from the run ending during the annealing phase which is cheap due to loose tolerances rather than from the scheduling itself. This is partially addressed with averaging over all epochs trained but the more favorable number is the one mentioned first in the text.
Questions for authors
  • Why is the final error of CTA lower on Cubic with Dopri5 than Fixed-Final (see above)?
  • Can you run the Ott et al. (2021) algorithm or InfoCNF and report the numbers in comparison to CTA?
  • On Jena, what are the numbers for CTA if no early stopping is done and a fixed training length is used?
Reviewer_ATbuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Under review is a pre-specified cosine schedule that progressively tightens the solver tolerances of a Neural ODE during the first 60% of training. The authors benchmark it against constant tolerances as well as other schedule shapes on two synthetic systems, on MNIST, and on Jena Climate forecasting. Where the schedule shows its clearest benefit is the cubic system solved with Dopri5: there, average trajectory error at late checkpoints is lower and training requires fewer function evaluations. For both real-data tasks, by contrast, a constant tolerance set at the geometric midpoint reaches comparable reported accuracy at lower cost. The authors state outright that the cosine shape offers no consistent edge over the other schedules.

Critical strengths

S1. Rather than stopping at a comparison between the loose and tight endpoints, the experiments are built out with some care: an informative intermediate constant tolerance, multiple schedule shapes, extra tolerance pairs and solvers, and ten seeds for the principal synthetic and Jena configurations (Tables 1–4; Appendix B).

S2. The headline synthetic finding holds up across seeds. In the cubic/Dopri5 case, Mean Final Error drops from 0.1725 to 0.0652 while training NFE per iteration falls from 34.90 to 27.45 (Table 1). Two further checks—re-evaluating under a tighter tolerance and varying the window over which late checkpoints are averaged—speak to numerical and reporting sensitivities that matter (Section 4.3.2).

S3. Training cost is kept separate from the cost of inference with the final model; on the adjoint tasks both forward and backward NFE are counted; and timing measurements are given for MNIST. The authors are also candid about the settings in which simpler baselines are the better choice, and about the weaknesses of the Jena validation protocol.

Critical weaknesses

W1. Set against prior work on controlling numerical accuracy while training Neural ODEs, the methodological contribution is a modest one. Ott et al. and InfoCNF receive appropriate discussion in Section 2, and the inexact-gradient bound in Section 3 is an existing result that serves only to motivate the approach. What remains as a contribution is, in the main, an empirically chosen default schedule. Yet in four of the five synthetic settings a linear schedule cannot be told apart from the cosine one, and on Lotka-Volterra the step schedule does better. As it stands, the paper needs a practical finding that goes beyond a preference for this one schedule.

W2. On real data, no advantage over a sensible constant setting is demonstrated: Fixed-Geometric delivers comparable accuracy to CTA at lower cost on both MNIST and Jena Climate (Tables 3–4). The largest accuracy gain, meanwhile, is measured on the very synthetic trajectory from which the training subsequences are drawn, and it varies with the solver and the tolerance regime. An empirical study can be valuable without winning on every task; even so, the benefit claimed for the situation in which the right tolerance is not known in advance remains insufficiently supported by these results.

W3. The claim that accuracy is preserved needs to be qualified more precisely. Not being able to reject a difference in accuracy is not the same as showing equivalence or non-inferiority. On Lotka-Volterra, for instance, Mean Final Error is 1.474 under CTA against 1.080 under Fixed-Final, and CTA’s variability is high (Tables 2 and 10). On Jena, the validation set used to select checkpoints is also the one on which error is reported, and half of the best checkpoints reached under CTA occur before annealing is complete (Sections 4.5 and 5). Such issues bear directly on whether CTA can be read as a dependable cost-saving default.

Questions for authors

Q1. Could the accuracy-preservation claim be expressed in terms of paired effect-size intervals together with a meaningful tolerance for degradation—above all for the high-variance Lotka-Volterra setting? It would help to keep differences that are merely inconclusive apart from actual evidence of non-inferiority. This paper lacks theoretical justifications overall.

Q2. Based on the evidence as it stands, what practical rule would tell a user to pick CTA rather than a loose constant tolerance or a linear schedule? A targeted comparison against a constant tolerance picked by a modest validation procedure would speak directly to the unknown-tolerance scenario the paper claims to address.

Q3. Is it possible to disentangle results that reflect optimization fit from those that reflect held-out generalization, and to say whether any results already exist on an independent chronological Jena test split or on held-out synthetic initial conditions? Doing so would make the scope of the empirical conclusion clearer.

Q4. Do you have more experiments? The current experiments are close to toy experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.