acceptodds
Under review as a conference paper at ICLR 2027

Gradient-Descent Based Convergence and Error Analysis of Flow Matching in Deep ReLU Networks

Abstract

Flow matching has proven to be a highly efficient technique for generative modeling of complex data, and works by learning the constructed velocity fields of a continuous normalizing flow via deep least-squares regression. However, most existing theoretical investigations on the accuracy of flow models, while significant, assume that the velocity function has been approximated to a certain accuracy, and then use this a priori bound to control the error of generation. We provide a quantitative understanding of the whole generation process, from training to sampling. More precisely, we provide a non-asymptotic convergence analysis of flow matching under gradient descent for a general class of deep ReLU neural networks, which elucidates how to design the training and sampling process for effective generation. For instance, our theory indicates the explicit effect of increasing the depth and width of the network, as well as the gradient-descent hyperparameters, on the final target accuracy. By propagating the convergence error across the generation process, we derive end-to-end error bounds on the Wasserstein-2 distance between the generated and target distributions.

Primary area:
learning theory
Keywords:
flow matching, generative modeling, deep learning theory, nonconvex optimization, generalization bounds, Wasserstein convergence

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FW8MOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the end-to-end convergence of flow matching with deep ReLU networks. It analyzes the convergence of gradient descent and combines optimization, approximation, generalization, and discretization errors to derive Wasserstein-2 bounds between the generated and target distributions.

Critical strengths

The paper addresses a theoretical problem by explicitly incorporating gradient-descent optimization error into the end-to-end convergence analysis of flow matching.

Critical weaknesses
  1. Limited novelty and significance. The optimal velocity field obtained by minimizing the population L2 objective is already known to preserve the prescribed marginal distributions in flow matching. Moreover, existing studies have established generation error bounds given an approximate velocity field. Therefore, once convergence to the optimal velocity field is guaranteed, the resulting distributional convergence is largely expected. It is unclear what fundamentally new theoretical insights this work provides beyond combining existing neural network optimization theory with established flow matching error analysis.

  2. Insufficient justification of the end-to-end convergence rate. The convergence rate claimed in Theorem 5.1 does not appear to follow directly from the preceding error bounds. In particular, Lemma 2.3 provides a Lipschitz bound of order (O((1−T)−3O((1-T)^{-3})), while Equation (16) contains an exponential amplification factor (eLνTe^{L_\nu T}). Under the proposed choice (T→1T\to1), this factor is not uniformly bounded and may overwhelm the polynomial decay of the estimation error. Moreover, substituting the Lipschitz bound into Equation (15) does not yield a vanishing error bound. Therefore, additional assumptions or a more rigorous argument seem necessary to establish the claimed Wasserstein convergence rate.

Questions for authors
  1. Theorem 3.5 establishes convergence for the empirical loss defined using fixed time discretization points in Eq. (4), whereas Section 4 considers a different Monte Carlo-based empirical objective in Eq. (10). How is convergence for the former sufficient to control the optimization error of the latter? Is an additional approximation or discretization error bound required?
  2. The network defined in Eq. (5) does not explicitly include bias terms, despite the claim that biases are absorbed. In particular, at (t=0t=0), this architecture is positively homogeneous in (x), whereas the optimal flow-matching velocity is generally affine: (v∗(x,0)=E[X1]−xv^*(x,0)=\mathbb E[X_1]-x). How does the proposed architecture satisfy the universal approximation guarantee in Theorem 3.1?
  3. The learning-rate and iteration-complexity conditions in Theorem 3.5 appear different from those in Theorem E.45, which additionally involves the minimum input separation and time quadrature weights. Could the authors clarify how the convergence guarantee stated in the main text follows from the appendix?
  4. The analysis relies on full-batch projected gradient descent, fixed input and output layers, and sufficiently overparameterized ReLU networks. To what extent do these guarantees provide meaningful insights into practical flow-matching models trained with mini-batch optimization and more expressive architectures?
  5. Theorem 3.5 establishes geometric convergence of the empirical training loss, but the experiments mainly examine the final Wasserstein error. Could the authors provide direct empirical verification of the claimed convergence rate, particularly under different network widths and training configurations?
Reviewer_cvauOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops an optimization-aware, end-to-end analysis of flow matching with deep ReLU networks. It proves geometric convergence of projected gradient descent under overparameterization and controlled initialization, then combines optimization, approximation, statistical, and numerical errors to bound generation quality in Wasserstein-2 distance.

Critical strengths
  • The paper bridges gradient-based optimization and end-to-end generation guarantees for flow matching, addressing an important gap in existing theory.
  • The analysis develops flow-matching-specific gradient lower bounds to handle dependencies among interpolated training samples.
  • Theorem 3.5 provides geometric empirical-loss decay under explicit assumptions, making training accuracy a quantitatively analyzable part of the generation pipeline.
Critical weaknesses
  • The optimization analysis closely follows [1] and [2], with innovations primarily adapting existing techniques to flow matching. Although the analysis provides explicit convergence bounds, it offers relatively few new flow-matching-specific insights for improving practical training or sampling.
  • The Wasserstein convergence rate deteriorates with ambient dimension, decreasing its relevance to high-dimensional generative modeling.
  • The experiments mainly demonstrate qualitative trends without directly validating the predicted convergence rates or required network-width conditions.

References:
[1] Allen-Zhu et al. A Convergence Theory for Deep Learning via Over-Parameterization. ICML 2019.
[2] Wang et al. Evaluating the Design Space of Diffusion-Based Generative Models. NeurIPS 2024.

Questions for authors

See Weaknesses and below:

  • Which specific challenges in flow matching cannot be addressed by Wang et al. (2024), and which newly developed proof techniques are essential?
  • Can the convergence guarantees extend to standard training settings, including mini-batch SGD, Adam, and jointly optimized network layers?
  • Could the authors experimentally investigate how network width affects convergence rates and whether the predicted overparameterization requirements are necessary?
Reviewer_AttdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims at an optimization-aware, end-to-end guarantee for flow matching (Gaussian source, linear interpolant). Thm 3.5 (App. E) adapts Allen-Zhu et al. (2019) and Wang et al. (2024). Projected GD on the hidden layers of a wide, spectrally constrained ReLU net, with the outer layers frozen, drives the empirical loss on a fixed set of n pairs and N times below any ϵₜᵣₐᵢₙ at a linear rate. Eq. (15) adds ϵₜᵣₐᵢₙ to the approximation and covering bounds of Zhou & Liu (2025). App. F then adds Euler and time-clipping errors, giving W₂=Õ(n^{-1/(3(d+5))}) (Thm 5.1) and a KL bound after Gaussian smoothing. App. H has small experiments in d≤ 7.

Critical strengths

S1. The question is a good one. Current flow-matching theory either takes the velocity error as given or assumes exact ERM, and a correct training-to-sampling bound would be useful. Sec. 4.1 and Fig. 1 lay out clearly what has to be connected.

S2. App. E is serious work. Constants are explicit, the projections are shown to stay inactive, and the separation lemma for interpolated inputs (Lemma E.35) is specific to flow matching and could be reused. I also appreciated the candor of Remark 3.6 and the caveat on p. 59.

Critical weaknesses

W1. The optimization-to-generalization step (Eq. 15) is not established, and I don't think this decomposition can establish it.

  • Thm 3.5/E.45 drives the raw empirical loss to zero (the PL-type bound (116) is relative to 0). Its population counterpart is at least L(v^{*})>0, the time-averaged conditional variance of X₁-X₀ given Xₜ. I get about 0.43d0.43d for π₁=Unif[0,1]ᵈ, T=0.99. So training to ϵₜᵣₐᵢₙ≲ n^{-2/(3(d+5))}, as Thm 5.1 asks, means interpolating noisy targets.
  • For an interpolant, L(v̂ₜᵣₐᵢₙ)-Lₑₘₚ(v̂ₜᵣₐᵢₙ)≥ L(v^{*})-ϵₜᵣₐᵢₙ, so no uniform deviation bound over a class containing v̂ₜᵣₐᵢₙ can vanish. The class in Lemma D.1 is in any case the sparse width-M class of Thm 3.1. It does not contain a dense width-poly(n) ReLU net with no sup-norm bound. Also, the training term in (11) uses L̂_{n,m̄} (fresh (t,x0)(t,x_0) draws), not the fixed-grid objective of Thm 3.5.
  • Eq. (122) needs Γ≥ 32/(√(ν) ω₀) with ν∝ R^{-2L}δᵢₙβₘᵢₙ/(n³N²), so Γ grows at least like n^{3/2}NR^{L}. But Remark 3.2 fixes Γ Bₓ=10dζ, and acceptance uses Bₓ>√(2). Page 59 says the theorem then "does not apply as stated". Taking the large Γ instead puts exp(poly(n)) into e^{γ₁ T} in Lemma F.2.

This link is the main contribution, so W1 decides my score.

W2. Even granting (15), Thm 5.1 does not give a meaningful rate under the paper's own assumptions.

  • (a) The GD result needs δ_{data}=4ne^{-d/32}+…<1, i.e. d>32ln(4n) (cf. Assumption 3.4). Then n^{-1/(3(d+5))}>e^{-1/96}≈ 0.99. So the clipping term √(2d)(1-T) alone is about 0.99√(2d), which is the bound Lemma F.3's argument gives for W₂(π₀,π₁), i.e. for outputting noise.
  • (b) The rate treats ζ as a constant, which needs Assumption 2.2. Lemma G.8 uses 2.1 and 2.2 together, but e^{-V} with V∈ C²(ℝᵈ) has full support, so both cannot hold. With ζ≤ d/(1-T)³ you get e^{L_{ν} T}=exp(Θ(dn^{1/(d+5)})), and ζ^{d/2}(1-T)⁻⁴n^{-2/(d+5)} grows with n for d≥2. Even under 2.2, the factor e^{γ₁ T} with γ₁=10dζ>20d² is hidden in the Õ.
  • (c) Unless I misread γ₂, γ2≍(1−T)−4=n4/(3(d+5))≍N\gamma_2\asymp(1-T)^{-4}=n^{4/(3(d+5))}\asymp N. The Euler term in (16) is then Θ̃(1), and it would need N≍n5/(3(d+5))N\asymp n^{5/(3(d+5))} to decay at the claimed rate.

Minor:

  • The main-text Thm 3.5 (h=Θ(nN/(mwₘᵢₙ)), a₀, K=Θ(d^{(1-a₀)/2}n²Nlog)) is not what App. E proves. Thm E.45 has h=R²/(80Γ²dL), no a₀, and K with R^{2L-2}n³N³, which is exponential in depth.
  • Lemma F.1 needs Lip(ν_{θ}), not L_{ν}.
  • Frobenius rescaling is not the spectral-ball projection, and RK4 is explicit (Remark 5.4).
  • The released code trains a width-128 SiLU MLP with Adam and fresh minibatch noise, so App. H does not probe the analyzed regime.
  • App. A has a dangling sentence, and some citations don't match their claims (e.g., the KL-convexity step in Prop. F.4 cites work on pseudorandom walks).
Questions for authors
  1. (Clarification) How does v̂ₜᵣₐᵢₙ enter a class covered by Lemma D.1, given that it interpolates and L(v^{*})>0? Is ϵₜᵣₐᵢₙ meant as an excess empirical loss?
  2. (Clarification) Is Assumption 2.2 in force in Thm 5.1, and how does it coexist with 2.1? If not, what rate do you get with ζ≍d/(1−T)3\zeta\asymp d/(1-T)^3?
  3. (Clarification) How do you reconcile (122) with Γ Bₓ=10dζ? And what does Thm 5.1 give once d>32ln(4n)?
  4. (Experiment) Please train a wide ReLU net by full-batch GD on a fixed (x0i,x1i,tj)(x_0^i,x_1^i,t_j) set and plot training and held-out loss against L(v^{*}).

To raise my score I would need a fix for W1 (e.g., early stopping or regularization, with a generalization argument for the trained network itself) and a corrected Thm 5.1. Repositioning the paper around the GD result alone, with a clear comparison to Wang et al. (2024), may be a cleaner path.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.