Solver-Skipping Backpropagation: Efficient Training of Continuous-Time Neural Networks

Continuous-time neural networks (CTNNs) provide an expressive and flexible framework for learning and modeling complex dynamics. However, computing gradients associated with ODE-solver-based state evolution introduces substantial computational and memory overhead, making training a key bottleneck. In this work, we propose Solver-Skipping Backpropagation (SSBP), an efficient CTNN training method that keeps forward integration unchanged while bypassing backpropagation through the solver’s internal steps by employing a one-step gradient approximation constructed from the integration interval’s initial state, thereby substantially reducing computational and memory costs during training. Relative to conventional backpropagation through time (BPTT), this solver-level decoupling reduces backward-pass computational and memory costs from $\mathcal{O}(Ls)$ to $\mathcal{O}(1)$, where $Ls$ denotes the number of function evaluations in the forward solve. We provide a theoretical explanation for the effectiveness of SSBP by characterizing how its update direction approximates the BPTT gradient. Experiments across diverse CTNN models, tasks, and solvers show that SSBP achieves performance comparable to conventional BPTT while substantially reducing training time and memory usage. Numerical experiments on a spin-torque oscillator, a typical neuromorphic device, further demonstrate its effectiveness and practical utility.