acceptodds
Withdrawn from ICLR 2027

SoF: Second-Order Flow Matching with Learning Acceleration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continuous-time generative models are bottlenecked by the number of function evaluations (NFE). We introduce Second-Order Flow Matching (SoF), which equips flow-based models with an explicit acceleration field via the squared-generator identity ( mathcalH_t = mathcalG_t^2 + partial_t mathcalG_t ) and a phase-space equivalence, realized by a dual-head backbone enabling single-NFE second-order sampling at ( < 2 % ) overhead. Combined with a schedule-matched sampling formula and an endpoint-refined inference grid, second-order sampling with the trained acceleration head yields large gains over same-model first-order solvers: 50-NFE FID drops from 21.24 to 2.48 on CIFAR-10 and from 32.03 to 3.40 on CelebA-HQ with 2nd-order Heun, reaching 2.41 at 100 NFE on CIFAR-10 without distillation, while DPM-Solver++ (2V) halves the 20-NFE FID relative to the best first-order solver (3.31 versus 7.16). At very low NFE ( (N leq 4 )), first-order solvers remain competitive. Inpainting on CelebA-HQ and LSUN-Church confirms the benefit extends to conditional generation. Code will be released upon acceptance.

Primary area:
generative models
Keywords:
Second-order flow matching, acceleration field learning, few-step generation, DPM-Solver++, flow matching, generative modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zUJ8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Second-Order Flow Matching (SoF) for learning the second-order time derivative, i.e., acceleration, of the flow ODE. The theoretical motivation is that integrating the flow ODE with quadratic approximation using acceleration reduces local truncation error to O(h3)O(h^3) from O(h2)O(h^2) of tangent-line approximation. To support efficient training and inference, the paper proposes a shared-decoder dual head network, where the first head predicts velocity and the second head predicts acceleration. This architectural design enables velocity and acceleration inference with a single forward pass. In additional to these core contributions, the paper also discusses various design choices such as interpolant parametrization (Sec. 2.3), time sampling (Sec. 2.5), and adaptive integrator variants (Sec. 2.6). The combination of the choices is shown to provide a non-trivial gain in FID scores on standard generative benchmarks such as CIFAR10, CelebA, and LSUN.

Critical strengths

None.

Critical weaknesses

Inconsistency between theory and practice. While the theory regarding the quadratic integration x(t+h)=x(t)+hv(t)+12h2a(t)+O(h3)x(t+h) = x(t) + h v(t) + \frac{1}{2} h^2 a(t) + O(h^3) where a(t)=dv(t)/dt=∂tvt+(∇vt)vta(t) = d v(t) / dt = \partial_t v_t + (\nabla v_t) v_t in Section 2.1 and 2.2 are sound, I am not sure whether the SoF acceleration learned by solving min⁡θE[∥∥aθ(Xt,t)−Xt¨∥∥22]\min_\theta \mathbb{E}[ \|\| a_\theta(X_t,t) - \ddot{X_t} \|\|_2^2] proposed in Section 2.4 Eq. (10) actually converges to the right quantity a(t)a(t). In short, my concern is that E[Xt¨∣Xt]≠dE[Xt˙∣Xt]/dt\mathbb{E}[\ddot{X_t} | X_t] \neq d \mathbb{E}[\dot{X_t} | X_t] / dt in general. The right-hand side is a(t)a(t) discussed in Sections 2.1 and 2.2, whereas the left-hand side is the acceleration learned and used in practice within the paper.

Insufficient empirical evidence. Using learned acceleration shows inconsistent benefits across number of function evaluations (NFEs). For instance, in Table 2, CIFAR10, 2nd Euler starts under-performing 1st Euler for NFE >= 20. Likewise for DPM++ in the same table. Furthermore, the experiment settings are unclear, so it is difficult to comprehend exactly how much the learned acceleration contributes to the final performance. For instance, while Table 2 in the main paper implies 1st Heun vs. 2nd Heun is the comparison isolating the effect of learned acceleration, Table 10 in the Appendix reports 2nd Heun (C,S) vs. 2nd Heun (C,S,Z) where "Z = zero acceleration". The paper should clarify the exact training and inference settings for reported numbers, so one may disentangle the gain from auxiliary training and inference techniques from the gain from core idea of second-order integration with learned acceleration.

Insufficient discussion of closely related works. The paper does not adequately acknowledge closely related prior works such as [1,2,3]. In particular, GENIE already introduced a lightweight neural head for predicting higher-order ODE derivatives and using them for single-evaluation Taylor sampling. It also demonstrated the approach on realistic image-generation benchmarks.

[1] GENIE: Higher-Order Denoising Diffusion Solvers, NeurIPS, 2022

[2] NRFlow: Towards Noise-Robust Generative Modeling via High-Order Mechanism, UAI, 2025

[3] High-Order Matching for One-Step Shortcut Diffusion Models, arXiv, 2025

Questions for authors
  • Can the authors verify whether the optimum of the SoF acceleration loss equals a(t)=dv(t)/dt=∂tvt+(∇vt)vta(t) = d v(t) / dt = \partial_t v_t + (\nabla v_t) v_t?
  • Can the authors provide an ablation table which clearly disentangles the effects of training techniques, inference techniques, and using acceleration for ODE integration?
  • Can the authors explain the methodological novelty and empirical strengths compared to prior related works such as GENIE?
Reviewer_8ARSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a second-order formulation for flow matching (FM) using a dual-head backbone. The method estimates second-order terms during training to improve FM performance. The experimental results demonstrate the method's validity.

Critical strengths

Strengths

  1. The proposed method cancels the Hessian matrix during the derivation of the second-order term.

  2. The experimental results show that the proposed SoF can further improve the generation quality in 50 and 100 NFEs.

Critical weaknesses
  1. The overall writing is not friendly to the audience. 1) The mathematical symbols are not explained when they first appear. For example, Eq. 1 uses mathematical symbols without explanation, so I have to check the details in the appendix. 2) Some claims are unclear. For example, the author states: "50-NFE FID drops from 21.24 to 2.48 on CIFAR-10 and from 32.03 to 3.40 on CelebA-HQ with 2nd-order Heun." This is confusing because I would assume 21.24 comes from the baseline, but it actually comes from the ablation study of their own method. 3) Some important equations have been moved to the appendix. For example, the calculation of this loss function.

  2. The theoretical proof in this paper seems unreliable. The main concern is that the acceleration SoF learns is not generally the material acceleration required by a second-order ODE solver.

Eq. (10) trains the acceleration head using the conditional path acceleration:

aθ∗(x,t)=E[X¨t∣Xt=x].a_\theta^*(x,t)=\mathbb{E}[\ddot X_t\mid X_t=x].

However, Theorem 2.2 defines the acceleration of the marginal velocity field as

αt(x)=∂tut(x)+(∇xut(x))ut(x).\alpha_t(x)=\partial_tu_t(x)+(\nabla_xu_t(x))u_t(x).

These two quantities are generally not equivalent. In fact,

αt(x)=E[X¨t∣Xt=x]−1pt(x)∇x⋅(pt(x)Cov⁡(X˙t∣Xt=x)).\alpha_t(x)=\mathbb{E}[\ddot X_t\mid X_t=x]-\frac{1}{p_t(x)}\nabla_x\cdot \left(p_t(x)\operatorname{Cov}(\dot X_t\mid X_t=x)\right).

The proposed objective ignores the additional covariance term.

  1. The experiments fail to convincingly demonstrate the benefits of explicit acceleration learning, as the acceleration-aware DPM-Solver++(2A) consistently underperforms its velocity-only counterpart (2V), particularly at low NFE. The reported improvements appear to be largely attributable to changes in the inference grid and sampling schedule rather than second-order acceleration, while properly configured first-order baselines achieve nearly identical FID scores. Furthermore, the negligible contribution of the acceleration correction (about 0.2% of the velocity update), along with inconsistent experimental configurations and no statistical uncertainty analysis, raises concerns about the effectiveness and reliability of the reported improvements.
Questions for authors
  1. Effectiveness of Acceleration Learning: Why does the acceleration-aware DPM-Solver++(2A) consistently underperform the velocity-only DPM-Solver++(2V), especially at low NFE, despite the claimed benefits of explicit second-order information?

  2. Attribution of Experimental Improvements: How can the authors demonstrate that the reported FID improvements originate from learned acceleration rather than changes in the inference grid, sampling schedule, or solver configuration, given that well-tuned first-order baselines achieve nearly identical results?

  3. Actual Contribution of the Acceleration Head: Given that the acceleration correction contributes only approximately 0.2% of the velocity update and is largely insensitive to perturbations, does the acceleration head genuinely improve numerical integration, or does it primarily serve as an auxiliary training regularizer?

Reviewer_Vtq5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Second-Order Flow Matching (SoF), a framework that equips flow-based models with a lightweight dual-head architecture to jointly predict velocity and acceleration. By replacing the degenerate linear path with a quadratic polynomial schedule, SoF achieves second-order sampling accuracy at a single function evaluation (1 NFE) per step while adding less than 2% computational overhead. Furthermore, the authors diagnose and resolve two critical inference-time bottlenecks: (1) the mismatch between standard DPM-Solver++ formulas and polynomial schedules, and (2) corrector distortion caused by uniform time grids near the endpoints. With the corrected schedule-matched formula and an endpoint-refined cosine grid, SoF outperforms same-model first-order baselines at moderate-to-high NFE budgets—achieving FIDs of 2.41 on CIFAR-10 and 2.56 on CelebA-HQ at 100 NFE, while its DPM-Solver++ (2V) variant surpasses first-order solvers from 20 NFE onward. The effectiveness of acceleration supervision is also validated on image inpainting tasks across CelebA-HQ and LSUN-Church.

Critical strengths
  1. Natural Motivation and Rigorous Theoretical Foundation: The idea of incorporating acceleration to characterize the trajectory curvature within the FM framework is very intuitive and well-motivated. Furthermore, the mathematical derivations are thorough and complete, providing a solid theoretical foundation for second-order dynamics.

  2. Diverse Datasets and Impressive Claimed Performance: The experimental evaluation spans a diverse set of benchmarks, including CIFAR-10, CelebA-HQ, and LSUN-Church. The claimed generation quality is impressive, reaching FIDs of 2.41 on CIFAR-10 and 2.56 on CelebA-HQ at 100 NFE.

  3. Comprehensive Coverage of Training-Free Solvers: The paper provides an extensive discussion and integration of various training-free ODE solvers, thoroughly covering classical and modern schemes such as Heun, RK4, and DPM-Solver++.

  4. Generalization Beyond Unconditional Generation: Beyond standard unconditional and class-conditional generation across different step budgets, the authors also explore conditional downstream scenarios such as image inpainting, demonstrating the broader applicability of their approach.

Critical weaknesses
  1. Over-Complicated Theoretical Packaging: The core idea of the paper is simply introducing acceleration into the FM framework. However, the theoretical formulation is heavily over-packaged with functional analysis and measure-theoretic terminology (e.g., Polish spaces, Radon-Nikodym densities, and operator semigroups), which is unnecessary for the actual Euclidean ODE algorithms and severely hinders the readability of the paper.

  2. Unrigorous and Cherry-Picked Performance Claims: Several headline claims in the abstract and main text are misleading. For example, the authors highlight that "50-NFE FID drops from 21.24 to 2.48 on CIFAR-10". Here, 21.24 is cherry-picked from the worst-performing 1st-order Heun solver evaluated on the SoF-trained backbone (1st Heun (SoF)). To make a fair and meaningful claim, the method should be compared against the best-performing standard 1st-order training-free baseline at 50 NFE (such as standard 1st RK4 at 4.73 FID or 1st Euler (SoF) at 2.52 FID in Table 2), rather than a degraded baseline.

  3. Unfair Experimental Comparisons (Training-Based vs. Training-Free): SoF is fundamentally a train-from-scratch method that requires bending the straight CondOT path into a curved polynomial trajectory to obtain non-zero acceleration targets. However, the empirical evaluation only compares SoF against training-free ODE solvers. To justify the additional training modifications, the authors should compare SoF against other train-from-scratch acceleration methods (e.g., MeanFlow [1], Improved MeanFlow [2], ShortCut models [3], AYF [4] and so on), or at least lightweight training-based methods (e.g., BA-Solver [5]).

  4. Factual Errors in Related Work and Missing Baselines: In Section 4 ("Flow matching variants and few-step distillation", Lines 455–460), the authors cite Shortcut Models [3] and MeanFlow [1] and claim that "all distill a pretrained teacher; SoF instead trains from scratch". This is factually incorrect: both Shortcut Models and MeanFlow are explicitly designed to train from scratch without any pretrained teacher. Mischaracterizing these works as distillation methods and omitting them from the experimental comparisons weakens the paper's positioning.

[1] Geng, Zhengyang, et al. "Mean flows for one-step generative modeling." Advances in Neural Information Processing Systems 38 (2026): 75460-75482.

[2] Geng, Zhengyang, et al. "Improved mean flows: On the challenges of fastforward generative models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[3] Frans, Kevin, et al. "One step diffusion via shortcut models." International Conference on Learning Representations. Vol. 2025. 2025.

[4] Sabour, Amirmojtaba, Sanja Fidler, and Karsten Kreis. "Align your flow: Scaling continuous-time flow map distillation." Advances in Neural Information Processing Systems 38 (2026): 146459-146512.

[5] Chen, Hongxu, et al. "Bi-Anchor Interpolation Solver for Accelerating Generative Modeling." In Forty-third International Conference on Machine Learning.

Questions for authors

Please refer to the Weaknesses section above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.