- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Solver-Skipping Backpropagation (SSBP) for training continuous-time neural networks. It retains the original numerical forward integration while replacing differentiation through the solver’s internal trajectory with a single local proxy evaluated at the beginning of each integration interval. Neural ODEs and CT-RNNs use an Euler-based proxy, while LTCs use a proxy derived from their fused semi-implicit solver. Experiments cover several sequence tasks, fixed-step and adaptive solvers, and a coupled spin-torque oscillator simulator. The results demonstrate substantial reductions in training time and memory, with generally competitive task performance.
Critical strengths
The method is simple and offers a useful efficiency trade-off. Preserving the forward integration while simplifying its backward computation allows accurate state evolution without retaining the solver’s full internal computation graph.
The evaluation reports measured training runtime and peak memory across multiple model–solver combinations, with five-run results. The experiments varying the number of solver substeps clearly illustrate where the efficiency gains arise.
The appendix provides explicit gradient formulas, pseudocode, and implementation details. The paper also acknowledges limitations involving long integration intervals, rapidly varying dynamics, and continued reliance on BPTT across intervals.
Critical weaknesses
W1. The methodological distinction from Euler Gradient Approximation is insufficiently established.
The closest prior work appears to be Ouala et al., Online calibration of deep learning sub-models for hybrid numerical modeling systems (Communications Physics, 2024). EGA already uses Euler-based gradient approximations while retaining the numerical forward solver, and explicitly discusses reducing backward computation to a single function evaluation through a first-order Jacobian approximation.
SSBP cites this work, but does not provide a sufficiently explicit mathematical comparison. In particular, the Neural ODE and CT-RNN rules in Appendix E appear closely related to EGA with a first-order flow-Jacobian approximation, specialized to a neural vector field. The paper should establish whether these rules are equivalent under matched assumptions and interval definitions, or identify the precise additional approximation or algorithmic contribution. If they are equivalent, the contribution should be framed around the CTNN extension, model-specific adaptations, and empirical findings. Comparisons against BPTT and adjoint methods demonstrate efficiency but do not establish novelty relative to existing approximate-gradient approaches.
W2. The evaluation protocol weakens the evidence for performance preservation and improvement.
Appendix I states that classification results use the highest test accuracy across all epochs, while Traffic results use the lowest test MSE. Selecting the reported result using test performance introduces optimistic bias and can affect comparisons when methods have different training variability. Checkpoints should instead be selected using validation performance and then evaluated on the test set.
Appendix H also describes randomly splitting overlapping windows for Gesture, Traffic, and PA. This permits shared observations across training and test windows and weakens the independence of the evaluation. Recording-level or chronological splits, constructed before windowing and with appropriate separation, would provide stronger evidence of generalization. Applying the same split to both methods does not eliminate this concern.
W3. A task-specific modification complicates the interpretation of the generality results.
Appendix I reports that the standard LTC proxy produces an approximately two-percentage-point accuracy gap on Gesture and less stable optimization. The reported experiment therefore uses a decay-preserving variant that replaces the single-step decay factor with an accumulated L-step expression.
This is a meaningful methodological modification, but it is not clearly identified in the main results table. The paper should report the standard proxy and the modified variant side by side, explain the complete gradient rule, and clarify how the choice between them is made. This would distinguish the performance of the general prescription from that of a task-specific adaptation.
Questions for authors
Under matched interval definitions and first-order Jacobian approximations, how do the Neural ODE and CT-RNN versions of SSBP differ mathematically from EGA?
Do the main conclusions hold with validation-selected checkpoints and recording-level or chronological splits that prevent overlapping observations across partitions?
What are the standard and decay-preserving LTC results on Gesture, and can the modified proxy be selected using a general criterion?
How do gradient alignment and task performance change as the integration interval and sequence length increase?