acceptodds
Under review as a conference paper at ICLR 2027

Solver-Skipping Backpropagation: Efficient Training of Continuous-Time Neural Networks

Abstract

Continuous-time neural networks (CTNNs) provide an expressive and flexible framework for learning and modeling complex dynamics. However, computing gradients associated with ODE-solver-based state evolution introduces substantial computational and memory overhead, making training a key bottleneck. In this work, we propose Solver-Skipping Backpropagation (SSBP), an efficient CTNN training method that keeps forward integration unchanged while bypassing backpropagation through the solver’s internal steps by employing a one-step gradient approximation constructed from the integration interval’s initial state, thereby substantially reducing computational and memory costs during training. Relative to conventional backpropagation through time (BPTT), this solver-level decoupling reduces backward-pass computational and memory costs from to , where denotes the number of function evaluations in the forward solve. We provide a theoretical explanation for the effectiveness of SSBP by characterizing how its update direction approximates the BPTT gradient. Experiments across diverse CTNN models, tasks, and solvers show that SSBP achieves performance comparable to conventional BPTT while substantially reducing training time and memory usage. Numerical experiments on a spin-torque oscillator, a typical neuromorphic device, further demonstrate its effectiveness and practical utility.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Continuous-time neural networks, neural ordinary differential equations, efficient training, approximate gradients

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TMNsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Solver-Skipping Backpropagation (SSBP) for training continuous-time neural networks. It retains the original numerical forward integration while replacing differentiation through the solver’s internal trajectory with a single local proxy evaluated at the beginning of each integration interval. Neural ODEs and CT-RNNs use an Euler-based proxy, while LTCs use a proxy derived from their fused semi-implicit solver. Experiments cover several sequence tasks, fixed-step and adaptive solvers, and a coupled spin-torque oscillator simulator. The results demonstrate substantial reductions in training time and memory, with generally competitive task performance.

Critical strengths

The method is simple and offers a useful efficiency trade-off. Preserving the forward integration while simplifying its backward computation allows accurate state evolution without retaining the solver’s full internal computation graph.

The evaluation reports measured training runtime and peak memory across multiple model–solver combinations, with five-run results. The experiments varying the number of solver substeps clearly illustrate where the efficiency gains arise.

The appendix provides explicit gradient formulas, pseudocode, and implementation details. The paper also acknowledges limitations involving long integration intervals, rapidly varying dynamics, and continued reliance on BPTT across intervals.

Critical weaknesses

W1. The methodological distinction from Euler Gradient Approximation is insufficiently established.

The closest prior work appears to be Ouala et al., Online calibration of deep learning sub-models for hybrid numerical modeling systems (Communications Physics, 2024). EGA already uses Euler-based gradient approximations while retaining the numerical forward solver, and explicitly discusses reducing backward computation to a single function evaluation through a first-order Jacobian approximation.

SSBP cites this work, but does not provide a sufficiently explicit mathematical comparison. In particular, the Neural ODE and CT-RNN rules in Appendix E appear closely related to EGA with a first-order flow-Jacobian approximation, specialized to a neural vector field. The paper should establish whether these rules are equivalent under matched assumptions and interval definitions, or identify the precise additional approximation or algorithmic contribution. If they are equivalent, the contribution should be framed around the CTNN extension, model-specific adaptations, and empirical findings. Comparisons against BPTT and adjoint methods demonstrate efficiency but do not establish novelty relative to existing approximate-gradient approaches.

W2. The evaluation protocol weakens the evidence for performance preservation and improvement.

Appendix I states that classification results use the highest test accuracy across all epochs, while Traffic results use the lowest test MSE. Selecting the reported result using test performance introduces optimistic bias and can affect comparisons when methods have different training variability. Checkpoints should instead be selected using validation performance and then evaluated on the test set.

Appendix H also describes randomly splitting overlapping windows for Gesture, Traffic, and PA. This permits shared observations across training and test windows and weakens the independence of the evaluation. Recording-level or chronological splits, constructed before windowing and with appropriate separation, would provide stronger evidence of generalization. Applying the same split to both methods does not eliminate this concern.

W3. A task-specific modification complicates the interpretation of the generality results.

Appendix I reports that the standard LTC proxy produces an approximately two-percentage-point accuracy gap on Gesture and less stable optimization. The reported experiment therefore uses a decay-preserving variant that replaces the single-step decay factor with an accumulated L-step expression.

This is a meaningful methodological modification, but it is not clearly identified in the main results table. The paper should report the standard proxy and the modified variant side by side, explain the complete gradient rule, and clarify how the choice between them is made. This would distinguish the performance of the general prescription from that of a task-specific adaptation.

Questions for authors

Under matched interval definitions and first-order Jacobian approximations, how do the Neural ODE and CT-RNN versions of SSBP differ mathematically from EGA?

Do the main conclusions hold with validation-selected checkpoints and recording-level or chronological splits that prevent overlapping observations across partitions?

What are the standard and decay-preserving LTC results on Gesture, and can the modified proxy be selected using a general criterion?

How do gradient alignment and task performance change as the integration interval and sequence length increase?

Reviewer_SbyyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method for training continuous-time neural networks (CTNNs) called solver-skipping backpropagation (SKBP), aiming at efficiency in training. The main idea is that in the backward pass, the gradient through the steps of the solver is approximated using the gradient of one step. Thereby, the backward pass has a cost that is independent of the number of steps in the solver, hence alleviating the cost in terms of memory. The paper gives a theoretical motivation rooted in the angles between the approximation and the correct gradient. The experimental analysis covers several tasks (6 benchmarks, a simulated spin-torque oscillator), and models (NeuralODE, CT recurrent neural nets, and LTC).

Critical strengths
  • The proposed methodology is simple and can be applied to several different CT neural network models and solvers. To the best of my understanding, applicability is surely a strong point in the submission.

  • Efficiency: the proposed approach actually enables substantial (measured and reported in the manuscript) efficiency gains in terms of GPY memory and training time.

Critical weaknesses
  • The fairness of the comparison is not really clear to me. Both BPTT and SSBP have been experimented with using the same learning rate, but to the best of my understanding, it is not reported how this was chosen. Moreover, in Appendix I, it is said that the best test accuracy over all epochs is reported, and this points out a risk that model selection was performed using test set information. In addition, several differences in the results are within one std, which leads to the possibility that the accuracy gains reported for SSBP might be due to the specific setup rather than the novel method itself.
  • The baselines reported in the experiments seem to be limited. In particular, SSBP is mostly compared with BPTT and with the adjoint method in only part of the experiments. For example, in my understanding, a natural and not expensive baseline might be included, i.e., training with a coarse solver (using a single step per interval). Other efficient training alternatives, discussed in the related work section, are also absent.
Questions for authors
  • Please, clarify for every dataset the protocol adopted for model selection. Have you used a sepatare validation set? Or did you select the results on the test set (as it appears to my reading of Appendix I)? Please, report all the achieved results when using a selection only on a validation set.
  • Can you please clarify how you choose the learning rate? Could you report the results with a learning rate value separately tuned for SSBP and BPTT?
  • Can you please provide a quick comparison with BPTT with one single solver step per interval? And possibly, with at least one other more among the efficient alternative training methods from the literature (several are mentioned in the related works)? If a simple baseline gives a similar performace: what would it be the advantage of the accurate forward solver on the used tasks?
  • From my understanding, the given theory is based on a 1st order approximation for short intervals, and the sequences used in training also seem rather short. E.g., the SMNIST version used in the paper translates images to sequences in a row-wise fashion, while the conventionally used sequential version of MNIST is a pixel-by-pixel classification task (28 steps in your approach vs 784 steps in the usual sMNIST case). Can you show how SSBP performs on longer sequences, where the approximation errors accumulate over many intervals?
  • Can you please clarify which 1-step approximation is adopted for the spin-torque oscillator case? As the system naturally physically works as a reservoir, perhaps a missing baseline would be the one where only a readout layer is trained, and the recurrent part is left fixed under the constraints of reservoir computing.
Reviewer_DMZxOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Continuous-time neural networks are typically trained by backpropagating through time, either via discrete differentiation (BPTT) or by solving adjoint ODEs backward in time (Adjoint sensitivity method). However, both approaches suffer from clear efficiency bottlenecks: BPTT requires storing intermediate computation graphs, while the adjoint method incurs significant computational overhead from additional ODE solves. To address these limitations, this paper proposes Solver-Skipping Backpropagation (SSBP), a local gradient approximation method that bypasses backpropagation through internal solver steps by employing a model-specific one-step operator, drawing inspiration from techniques like phantom gradients in DEQs or OTTT in SNNs. Theoretical analysis demonstrates that SSBP yields an effective update direction as long as the alignment angle with the exact BPTT gradient remains under 90 degrees. Empirical evaluations show that SSBP achieves performance on par with (and occasionally superior to) BPTT and adjoint methods, while maintaining constant memory consumption and reducing training time.

Critical strengths
  • A simple yet highly effective approach with solid empirical and theoretical foundations. The underlying idea is very promising and holds strong potential for extension.

  • The paper is well-written and structured with intuitive visualizations for readability.

Critical weaknesses
  • Limited in-depth analysis on the underlying mechanisms through which SSBP outperforms BPTT and adjoint methods in multiple benchmarks.
  • Some important details are omitted from the main text and relegated to the appendix, including concrete definitions and examples of the one-step approximation operators, formal theoretical assumptions, and a simple proof sketch of the gradient alignment property.
Questions for authors

Under what practical conditions does SSBP fail to maintain positive gradient alignment with BPTT and hinder training? Could the authors provide a concrete empirical example where SSBP underperforms?

Is it possible to empirically analyze why SSBP frequently outperforms BPTT? What would be an appropriate experimental setup to support the authors' conjecture regarding implicit regularization effect?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.