acceptodds
Under review as a conference paper at ICLR 2027

Training Neural ODEs by Path Matching

Abstract

The prohibitive training cost of Neural ODEs (NODEs) stems from backpropagation through the ODE solver, whose cost scales with the number of function evaluations (NFE). Flow Matching (FM) removes this backpropagation by reframing training as field regression, but supervising the field on pre-defined trajectories (e.g. straight lines) creates a path-crossing conflict: on paired data the field simply averages conflicting velocities. We present Path Matching (PM), which resolves this conflict by matching the field on-policy: rather than regressing onto a prescribed path, we roll out the model and supervise the field at the states it actually visits, with a bridge target that realigns a drifted rollout back onto a target path (a straight line between endpoints by default). On image classification and a real, irregularly-sampled clinical time-series task, PM reaches accuracy on par with end-to-end NODE training at a fraction of the per-step cost and with no backprop through the solver, an efficiency we characterize with a runtime scaling law. A latent-dimension ablation then explains why the simulation-free alternative is fragile: it fails when a straight coupling cannot avoid path crossings—a limitation PM's curvable, on-policy objective escapes.

Primary area:
learning on time series and dynamical systems
Keywords:
Neural ODEs, flow matching, on-policy training, efficient training, continuous-depth models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_924mOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Path Matching (PM), an on-policy approach to training Neural ODEs (NODEs) that learns the velocity field at states visited during model rollouts while resolving path-crossing conflicts which enables training on the paired data setting. They evaluate PM on classification and clinical time-series tasks, comparing its performance with that of other NODE baselines.

Critical strengths
  • PM demonstrates robustness across architectural variations, including network width and depth.
Critical weaknesses
  • The claim that PM resolves path-crossing conflicts lacks sufficient theoretical justification. It remains largely unclear why on-policy supervision would yield compatible trajectories when the prescribed paths contain conflicting velocity targets. For instance, a mathemetical supports establishing conditions under which training loss goes to zero guarantees non-crossing trajectories would help substantiate this claim.
  • The per-iteration runtime comparison in Table 4 excludes PM’s pretraining cost and compares field-only training against other baselines. These differences make it difficult to attribute the observed speedup to PM’s training objective alone or to establish an overall training-efficiency advantage.
Questions for authors
  1. Can the authors justify the choice of k(t)=1/(1−t)k(t)=1/(1-t)? (Ln.151) Ablating with other options will help understand the effectivness of the proposed method.
  2. Can the authors provide the total and trainable parameter counts for each baseline and PM, also spcifying with 'non-NODE stack' (mentioned in Ln.200)?
  3. Since the authors claim that PM learns interpretable transport (Ln.429), can the authors visualize the learned transport trajectories and compare them with those of the baselines? This would help clarify what the on-policy framework learns and how the resulting transport is really interpretable.
  4. Can the authors mention which solver is used to meausre MSE on Table 7?
2 comments

Thank you. I really appreciate your feedback.

  1. for 1/(1-t), it's because the end point matters. We care the endpoint reach the target while keeping trajectory flexible in the mid run.
  2. for all the node family baselines, numbers of param are the same.
  3. fig 3
  4. dorpri-5

Also, thank you for pointing out the reference issues. It's quite important to cite other papers properly! I will definitely fix them in the final version.

I thank the authors for their response, but I will maintain my score.

The response does not address either critical weakness: the lack of theoretical justification for resolving path-crossing conflicts, and the exclusion of pretraining cost from the Table 4 runtime comparison.

Reviewer_sG8bOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper considers a more efficient way to train Neural ODEs for tasks like classification or time series data. Specifically it avoids the computational overhead of differentiation through the ODE solver, while also avoiding the training instabilities of competitive approaches. The paper focuses on introducing a new training procedure, and training loss approach, and demonstrates empirically the benefits.

Critical strengths

To the best of my knowledge, the 'on-policy' approach of the paper is novel and in my opinion very creative. I especially appreciate the training loss introduced in (3). My understanding is that the approach wouldn't have worked if it wasn't for this.

The numerical results are very strong.

Critical weaknesses

My opinion is that the paper is a bit unfair in criticizing the approach that relies on mapping the problem in a latent space. To my understanding, the latent space approach is computationally efficient. The paper states that it can maintain interpretability instead. However, the main benchmark of image classification relies on converting the input to a latent space anyway, so the advantage of interpretability does not seem valid there.

Along this, I want to also mention the paper 'Augmented Neural ODEs Emilien Dupont, Arnaud Doucet, Yee Whye Teh' which emphasizes why this augmentation to latent space is necessary. I would have liked to understand the present paper in comparison with that one.

Questions for authors
  • As above, please clarify the contribution to augmented/latent space methods.

  • At the end of training, do I understand correctly that there will still be paths that go close to each other and still have different velocities/fields but the training is sufficient to match these different velocities? Or is it that because of the training objective, the paths after training have slight curves that separate them slightly?

  • Do the numerical results also contain the time it took for the pre-training phase? My understanding is that this pre-training phase is a standard NODE training with NFEs, etc.

  • Another related concern is how much pre-training is needed, this may be hard to know a priori.

  • There are other approaches in the literature to improve the training efficiency on NODEs. For example, 'LyaNet: A Lyapunov Framework for Training Neural ODEs Ivan Dario Jimenez Rodriguez, Aaron D. Ames, Yisong Yue' ' How Deep Do We Need: Accelerating Training and Inference of Neural ODEs via Control Perspective Keyan Miao, Konstantinos Gatsis ' These are based on different approaches, like changing the training objective, but they could be competitive. Could you compare with them?

1 comment

Thanks for acknowledging the novelty. I really appreciate your feedback.

  1. PM doesn't rely on augmented/latent space. If latent dim is larger, less conflict, PM is less important. latent space is good. What i criticize is doing FM in high dim latent space, while crossing issue is already solved by encoder.

  2. PM won't match target perfectly. If we have perfect non-cross target, FM can easily solve it. but having perfect non-cross target is the problem itself. So it is a chicken-egg problem. In my plan of future work, this should be solved by iteratively self-improvement. In this work we use fixed, straight target, and the the trajectory is slight curved to mitigate the conflict. However, when latent dim is really small, non-straight target is still important (see latent dim ablation).

Reviewer_y5BQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at the cost of training Neural ODEs and proposes Path Matching (PM). Instead of regressing the field on points of a prescribed path as in flow matching, PM rolls out the current model without gradients and regresses the field at the states the rollout actually visits. The target is w˙(t)+κ(t)(w(t)−xt)\dot w(t) + \kappa(t)(w(t)-x_t) with κ(t)=1/(1−t)\kappa(t)=1/(1-t), where ww is the straight line between the paired endpoints. The claim is that this avoids the path-crossing problem flow matching has on paired data, without backprop through the solver.

On MNIST/SVHN/CIFAR-10, PM is used to distill a frozen pretrained block into a NODE. It reaches accuracy comparable to the solver-backprop baselines (better on CIFAR-10) at a lower per-step cost. There is also a PhysioNet mortality experiment, an architecture sweep showing that SFNO becomes unstable at small capacity, and a few small 3-D transport tasks.

Critical strengths

The practical side is appealing. The recipe is simple, the backward pass is a single regression with constant memory, and on all three image datasets plus PhysioNet the resulting NODE is about as accurate as one trained end-to-end through the solver, at lower per-step cost. The load-bearing probe is a nice touch; it convinced me the node is actually doing work rather than being bypassed.

I also found the architecture sweep useful. It shows fairly clearly that SFNO-style training gets unstable across seeds once the latent, the field or the depth gets small, which is good to know for anyone choosing a simulation-free recipe.

Critical weaknesses

1. The proposed mechanism does not do what the paper says, and the evidence offered for it points elsewhere.

With the default straight-line ww and κ=1/(1−t)\kappa=1/(1-t), Eq. 3 simplifies to w˙+κ(w−xt)=x1−xt1−t.\dot w + \kappa(w-x_t)=\frac{x_1-x_t}{1-t}. So x0x_0 drops out, and the target is the usual conditional flow-matching velocity evaluated at the rollout state. If the field matches this target, the rollout satisfies e˙=−κe\dot e=-\kappa e with e(0)=0e(0)=0 and follows the straight line exactly. The same holds for κ≡0\kappa\equiv 0. The fixed point of PM is therefore the same set of trajectories that flow matching already asks for, and PM cannot bend around a crossing at convergence.

The ability to curve that the abstract credits to PM seems to come instead from the height bootstrap described in Section 5 and Appendix B, which the authors present as future work. This would also explain two things in Fig. 3: the trajectories lift well out of the plane even though Section 3 says all results use the straight-line path, and the MSEs printed in the figure differ from Table 7 for the swap and merge tasks. Lifting into an extra dimension to escape a topological obstruction is also the idea behind Augmented Neural ODEs (Dupont et al., 2019), which is not discussed.

The architecture sweep does not support the crossing explanation either. The latent width LL is the axis on which the crossing argument applies, and on that axis SFNO degrades smoothly and is not below PM at L=8L=8 (roughly 69% vs. 63%). SFNO's collapses occur on the field-width and depth axes, where L=64L=64 is fixed and the room for non-crossing straight segments does not change. These collapses look like a capacity problem in fitting a many-to-one target onto shared class anchors. On the same axes PM falls well behind FW (around 74% vs. 92% at w=16w=16), which the text does not mention.

Finally, Section 5 says PM's advantage is specific to conflict-heavy transport. In the main experiments, however, the targets are per-sample features in a 64- or 128-dimensional latent, where exact crossings should be rare. By the paper's own account, the mechanism should therefore be inert where the main results are reported.

2. The reported gains can't be attributed to PM.

Accuracy. PM distills a frozen teacher trained with a tuned SGD recipe (92.3% on CIFAR-10), while the baselines train from scratch through the solver; the authors acknowledge that the CIFAR-10 gain is a distillation by-product. The comparison with SFNO also changes the target, from a shared class anchor to a per-sample teacher feature. The ablation I would most like to see is plain off-policy flow matching on the same frozen teacher and the same (z0,mid(z0))(z_0,\mathrm{mid}(z_0)) pairs. Table 9, where κ≡0\kappa\equiv0 stays within 0.5%, already suggests the correction term does little.

Cost. In phase 2, PM trains only the field, with the encoder and head frozen, while every baseline trains the full stack at each step. Table 4 shows this matters. On SVHN and CIFAR-10, SFNO with one field evaluation per step is about twice as slow as PM with seven (25.5 vs. 11.9 and 21.8 vs. 10.9 ms/iter), so per-step cost there is dominated by encoder training rather than by solver backprop. The "structural" explanation in "Why PM is faster" therefore doesn't hold as stated.

  • On MNIST, where the encoder is cheap, PM is about 2× faster than FW, which uses the same 3-step solver.
  • The order-of-magnitude numbers are against dopri5 with the adjoint.
  • Since PM still does a full forward rollout each step, its saving over direct backprop with the same solver is bounded by a constant factor, and the many-NFE regime that motivates the paper is not tested.

Baseline fidelity. SFNO reaches 69% on CIFAR-10 here, well below what the original paper reports. The authors attribute this to running SFNO outside its native architecture, which makes conclusions about SFNO drawn from this comparison hard to rely on.

Questions for authors
  1. Under the default setting, is Eq. 3 equal to (x1−xt)/(1−t)(x_1-x_t)/(1-t)? If so, how does it differ from the endpoint-aligned target in Appendix A? And which variant produced Fig. 3 and Table 7: the straight-line default or the height bootstrap?
  2. On the same frozen teacher and pairs, what does off-policy flow matching achieve, and what do NODE/FW achieve when trained to regress mid(z0)\mathrm{mid}(z_0) through the solver? This is the result most likely to change my score.
  3. What exactly does one PM step include compared with one baseline step? Could you report ms/iter with only the field trainable for every arm, and put the total wall-clock including teacher pretraining in the main table?
1 comment

Thanks. I really appreciate your feedback.

In Fig. 3 and Table 7 (toy cases), we use non-straight target and PM has some convincing results.

However, in classification tasks, we suffer from 2 challenges:

  1. too many datapoints making analytical non-straight target inefficient, so we have to use straight target
  2. with encoder and decoder, we need to make lots of effort to explain contribution is from PM

For the straight target, the loss never goes to 0, so the learnt trajectories never become straight and behave different from CFM. But I agree with your concern about straight target. Straight target performs surprisingly better than I expect. I plan to replace it with non-straight target in an efficient way in future work.

For the second point, that is the reason why I prefer frozen pairs without changing encoder and decoder. the result is just like the result from toy cases: PM is less accurate but faster than FW and other NODE method --- which is what we want to achieve.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.