acceptodds
Under review as a conference paper at ICLR 2027

On the Stability and Convergence of Flow Map Training

23 views

Abstract

Flow maps are a recent method of probabilistic generation that amortize diffusion trajectories into one-step samplers, unlocking efficient generation. While progress has been made into improving their empirical performance, the understanding of the mechanisms underlying their success remains largely unknown. Specifically, flow maps employ stop gradient operators as a crucial part of their training target, which seem necessary to obtain convergent methods. Adaptive weighting also drastically improves the generation capabilities of flow maps. Although many interpretations have been proposed to support the inclusion of these heuristics, none have analyzed the impact of these assumptions on the training dynamics of flow maps themselves. In this work, we address this issue by providing grounded theory explaining why such tricks are required. Analyzing the functional gradient flow induced by the MeanFlow and Lagrangian objective, we show their convergence to the true flow map, settling the validity of these objectives. For finite updates, we highlight a severe roadblock to the usage of these losses in practice; the possible unboundedness of the update. Adaptive normalization not only provides bounded updates, but combined with a specific weighting, additionally provides convergence guarantees towards a neighborhood of the solution. On synthetic data, we show that our functional gradient flow analysis exactly describes the true training dynamics for flow maps. Experiments on standard image datasets demonstrate that a proper reweighting scheme is essential for strong empirical performance, which is only achieved for a specific range of reweighting strengths.

Primary area:
generative models
Keywords:
Flow maps, Generative models, Optimization theory, Consistency models, Flow matching

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_2vBZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the stability and convergence of flow map training, especially for Mean Flow. Even though this one-step generations uses some well-known empirical tricks such as stop gradient and adaptive loss normalization, the full picture of optimization dynamics is not yet fully understood in the previous work.

The authors analyze the training of MeanFlow and Lagrangian models and show that both can converge to the true flow map under some assumptions and give the upper boundary for convergence rates. In addition, they also analyze the instability of the training dynamics as well. They find that using a fixed learning rate can cause training to diverge, even when the initial error is very small. To address this problem, they study adaptive normalization and show that chose of different $ p $ can result in different convergence guarantee, which has been showed in the experiments

Critical strengths

1: The paper partially addresses a fundamental question in mean flow training: why do stop-gradient-based objectives work, and what makes their optimization stable? Given the recent burst of one-step generative models, I think understanding these mechanisms are valuable.

2: The paper provides solid theoretical analysis. The authors establish convergence guarantees under relatively mild assumptions and provide useful insights into training stability and also provide adaptive normalization as the solution.

4: The theoretical findings are supported by synthetic experiments and image generation experiments on CIFAR-10 and FFHQ, where appropriate normalization indeed improves FID.

Critical weaknesses

1: Although the paper provides solid theoretical guarantees for the stability of mean flow training, but the experiments mainly focus on generation quality (FID). It would be helpful for me to see more direct evidence of training stability, such as gradient norms, to better support the theoretical findings.

2: The other assumptions seem make sense. However, the convergence guarantee in Theorem 4 relies on the additional assumption that the squared residual norm grows at most linearly with training iterations,

∣uk−utgt,k∣L2(ρ)2≤M(k+1)\\|u_k-u_{\mathrm{tgt},k}\\|_{L^2(\rho)}^2 \leq M(k+1)

but it is unclear whether this assumption holds in practice.

Questions for authors

1: In Proposition 1, it seems that the authors assume a fixed learning rate when analyzing training instability during the training dynamics? However, practical MeanFlow training typically uses adaptive learning-rate schedules. Could the authors clarify whether the same instability would occur under these more realistic training settings (e.g changing learning rate)?

2: As shown the weakness above that you assume

∣uk−utgt,k∣L2(ρ)2≤M(k+1)\\|u_k-u_{\mathrm{tgt},k}\\|_{L^2(\rho)}^2 \leq M(k+1)

Could the authors give us more intuition and explain whether this assumption holds in practice?

3: Theorem 4 provides an additional convergence guarantee for p=1, but the experiments show that p=0.75 often achieves better FID (under NFE=1). Could the authors explain this difference between the theoretical results and empirical performance?

6 comments

(1/2)

We would like to thank Reviewer 2vBZ for their detailed review, highlighting the contribution of our analysis to the understanding of flow maps.

W1: Although the paper provides solid theoretical guarantees for the stability of mean flow training, but the experiments mainly focus on generation quality (FID). It would be helpful for me to see more direct evidence of training stability, such as gradient norms, to better support the theoretical findings.

We thank the reviewer for their suggestion: we agree that FID is mainly a metric of performance. In order to better monitor stability, we track gradient statistics across training.

To the paper, we have added an additional experiment, in Figure 4, where we track the gradient norms of MeanFlow on FFHQ for values of pp in {0.0,0.25,0.5,0.75,1.0}\{0.0, 0.25, 0.5, 0.75, 1.0\}. We record both the raw gradient norm, and the norm of the gradient after Adam’s preconditioning, over 100k steps of training. Different values of the normalization exponent pp have different raw gradient scales, so to make the statistics comparable, we scale each value of pp independently by the inverse of the median gradient norm. Adam preconditioned gradients remain unscaled. Using these statistics, we observe three phenomena.

  • Qualitatively large gradients for p≤0.25p \leq 0.25 (Figure 4). We plot the maximum of the gradient norms observed in 100-step windows. We observe that for p≤0.25p \leq 0.25 the gradient magnitudes spike to higher relative levels than for p≥0.5p \geq 0.5 for the raw gradients, and for p≥0.75p \geq 0.75 for the Adam preconditioned gradients.
  • Heavy-tailed gradient norm distributions (Table 5). We collect statistics obtained from the gradient norm distributions for each value of pp. In particular, we list the variance and interquartile range (IQR) of the normalized distributions, and the skewness and excess kurtosis. The variance and IQR give a notion of the average spread of the distributions, whereas skewness and kurtosis are scale-invariant measures of the asymmetry and heavy-tailedness of the distributions, respectively. The skewness and excess kurtosis of the raw gradients are much larger for p≤0.25p \leq 0.25 than for p≥0.5p \geq 0.5, which is consistent with our observation of large gradient norms in Figure 4, unboundedness in Proposition 1 and the theoretical boundedness of updates for p≥0.5p \geq 0.5 in Theorem 4. p≥0.75p \geq 0.75 is seen to have more concentrated and less heavy-tailed Adam gradient norms than p≤0.25p \leq 0.25.
  • Persistence to learning rate. These same statistics are tabulated and plotted for various learning rates under p=0.0,0.25p = 0.0, 0.25. Even when the learning rate is decreased to 1/81/8’th of the base learning rate, variance, skewness and excess kurtosis for p≤0.25p \leq 0.25 still exceed the corresponding statistics for p≥0.75p \geq 0.75 at the base LR.

These provide an alternative measure of the difference in stability between values of pp.

W2 and Q2: The other assumptions seem make sense. However, the convergence guarantee in Theorem 4 relies on the additional assumption that the squared residual norm grows at most linearly with training iterations, ∥uk−utgt,k∥L2(ρ)2≤M(k+1),\|u_k-u_{\mathrm{tgt},k}\|_{L^2(\rho)}^2 \leq M(k+1), but it is unclear whether this assumption holds in practice.

Thank you for raising this point; it is a very pertinent question towards understanding the guarantees that gradient descent for flow map training has. We tackle this assumption in multiple parts.

  • Intuition. The quantity ∥uk−utgt,k∥L2(ρ)2\|u_k-u_{\mathrm{tgt},k}\|_{L^2(\rho)}^2 corresponds to the unnormalized training loss during iterations, hence this assumption relates to a condition that the training loss grows at most linearly.
  • Empirical growth. In Appendix E.3, we track the unnormalized loss during the first 100k iterations of MeanFlow FFHQ training, to track whether it diverges and at what speed; these losses are tracked only on the sub-batch of points (x,s,t)(x, s, t) with s≠ts \neq t, i.e. off-diagonal points, since on the diagonal the loss corresponds to matching the teacher and the target loss does not come into play. Figure 12 and Table 7 describe how the average losses over windows of size 2k and 20k change during the iterations. The squared residuals are seen to be increasing as training advances, and the average squared residuals between 20k-40k steps and 80k-100k steps have standard-error intervals that do not overlap. However, the rate of this growth is smaller than the increase in the number of steps, increasing by at most 1919% after around triple the step count, aligned with the assumption.

(2/2)

W2/Q2 continued.

  • Mathematical guarantees. We analyse this more in Appendix B.5. The assumption of the bounded growth of the unnormalized training loss is needed only for the condition that ∑i=0∞1∥ri∥L2(ρ)2+c=∞.\sum_{i = 0}^{\infty} \frac{1}{\|r_{i}\|^2_{L^2(\rho)} + c} = \infty. Letting ci=(∥ri∥L2(ρ)2+c)−1c_i = (\|r_i\|^2_{L^2(\rho)} + c)^{-1}, cic_i is the adaptive factor that we multiply our gradient update by when p=1p = 1, and so can be viewed as an adaptive step size. The condition that ∑ici=∞\sum_i c_i = \infty is then an assumption that the sum of the step sizes we take is infinite, which is a classical assumption on step size schedules for gradient descent methods [3]. In our case, the step sizes are not chosen at the beginning and instead depend on the residual error of the map at step kk, hence this condition is taken to be an assumption. As is shown in Appendix B.4, gradient descent with p=1.0p = 1.0 cannot move uku_k further away from u⋆u^\star unless it is already within an O(η1/2)\mathcal O(\eta^{1/2}) neighborhood of it, in L2(ps/ρ)L^2(p_s/\rho), without any bounded growth assumption.

    Assuming boundedness of psp_s and ρ\rho above and below on our domain, this means uku_k is always within some L2(ρ)L^2(\rho) ball of u⋆u^\star, and so if ∥u⋆∥L2(ρ)2<∞\|u^\star\|^2_{L^2(\rho)} < \infty, ∥uk∥L2(ρ)2\|u_k\|^2_{L^2(\rho)} is uniformly bounded. By the triangle inequality, the condition that ∥rk∥L2(ρ)2≤M(k+1)\|r_k\|^2_{L^2(\rho)} \leq M(k+1) is then implied by a condition on the target ∥utgt,k∥L2(ρ)2≤M(k+1)\|u_{\mathrm{tgt}, k}\|^2_{L^2(\rho)} \leq M(k+1). As utgt,k=v+(t−s)Dvuku_{\mathrm{tgt}, k} = v + (t - s) \mathcal D_v u_k, this is a condition that the derivatives of uku_k do not grow overly fast (quicker than k\sqrt k) in the direction of the velocity.

Q1: In Proposition 1, it seems that the authors assume a fixed learning rate when analyzing training instability during the training dynamics? However, practical MeanFlow training typically uses adaptive learning-rate schedules. Could the authors clarify whether the same instability would occur under these more realistic training settings (e.g changing learning rate)?

This is an interesting question that depends heavily on the choice of learning-rate schedule used. Our analysis assumes a fixed η\eta which matches the current best practice for training flow map models, such as Improved Mean Flow [1] and Terminal Velocity Matching [2] which take a constant learning rate throughout training, apart from a warm-up near the start for iMF. For learning rate methods which anneal to a positive constant as number of steps tends to infinity, our counterexample exhibiting blow up (Appendix B.4) still diverges by taking mm to be greater than 2/ηmin2/\eta_{\mathrm{min}}, where ηmin\eta_{\mathrm{min}} is the minimum learning rate encountered during training and after any warmup. For any practical training run over a finite number KK of optimization steps, ηmin\eta_{\mathrm{min}} exists and is positive, hence such a counterexample would increase in size by a factor of (ηmin(m+1)−1)K(\eta_{\mathrm{min}}(m+1) - 1)^K. The only case which this does not cover is the theoretical convergence of adaptive schemes in which ηk→0\eta_k \to 0, over an infinite number of steps. In this case, a counterexample would likely be constructed by piecing together different eigenmodes of the update map in different spatiotemporal parts of the domain; we leave this for future work.

Q3: Theorem 4 provides an additional convergence guarantee for p=1, but the experiments show that p=0.75 often achieves better FID (under NFE=1). Could the authors explain this difference between the theoretical results and empirical performance?

Theorem 4 provides sufficient conditions for bounded updates and convergence to within some O(η1/2)\mathcal O(\eta^{1/2}) neighborhood of the ground truth. We do not claim that p=1p = 1 is necessary for convergence. In practice many factors could influence the convergence of flow map training methods, such as architecture, optimizer, and problem setting. In our case, we use a neural network parametrization which does not follow the exact gradient dynamics (Limitations, Section 6), and the image scale runs use Adam instead of the analyzed gradient descent. Practical training also uses per-sample weights; lower pp could result in quicker convergence and better performance as it may shift the relative weights of training samples. Our experiments aim to test the qualitative predictions of the theory, that p≥0.5p \geq 0.5 achieves stable training while pp in {0.0,0.25}\{0.0, 0.25\} exhibits forms of instability (Figure 3, Table 1, Figure 4, Appendix E), instead of showing that a specific value of pp is optimal.

[1] Z. Geng, Y. Lu, Z. Wu, E. Shechtman, J. Z. Kolter, and K. He. Improved mean flows: On the challenges of fastforward generative models. In CVPR, 2026.

[2] L. Zhou, M. Parger, A. Haque, and J. Song. Terminal velocity matching. In ICLR, 2026.

[3] H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22(3):400–407, 1951.

We would like to thank again the reviewer for their feedback, and we would be more than happy to answer any lingering questions.

Reviewer_bCa5OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzes the functional gradient flow induced by the MeanFlow and Lagrangian objectives, shows their convergence to the true flow map. It also constructs initial errors for which every fixed positive gradient-descent step size can be unstable, and analyzes a globally normalized update.

Critical strengths
  • The analysis is rigorous and the exact dynamics offer an interpretable explanation about flow map training.
  • The experiment results are strong and complement the theory well.
Critical weaknesses

None

Questions for authors
  • Equation (21) bounds the squared norm by O(η)O(\eta). It therefore establishes an O(η)O(\sqrt{\eta}) norm-radius neighborhood, whereas p. 7, lines 336–340 describes an O(η)O(\eta) ball.
  • Could factors other than normalization-induced training instability explain the poor FID at low pp in Table 1? Did you check whether simply reducing the learning rate resolves these failures without normalization?
3 comments

We would like to thank Reviewer bCa5 for their positive feedback highlighting the rigor of the analysis, the informativeness of the manuscript and the strength of the experiments.

Equation (21) bounds the squared norm by O(η)\mathcal O(\eta). It therefore establishes an O(η)\mathcal O(\sqrt{\eta}) norm-radius neighborhood, whereas p. 7, lines 336–340 describes a O(η)\mathcal O(\eta) ball.

We thank the reviewer for pointing out this typographical error. The lines after theorem 4 now state an O(η1/2)\mathcal O(\eta^{1/2}) neighborhood. Typos have been fixed, and other changes appear in blue in the updated manuscript.

Could factors other than normalization-induced training instability explain the poor FID at low pp in Table 1? Did you check whether simply reducing the learning rate resolves these failures without normalization?

We agree that many training hyperparameters can influence the performance of flow maps, in particular the choice of the learning rate. In order to rule out a potentially too large learning rate as the source of instability at low values of pp, we provide multiple metrics with lower values of the learning rate at normalization exponents p=0.0,0.25p = 0.0, 0.25 on FFHQ with the MeanFlow objective. We report two additional metrics for this learning rate ablation:

  • FID over training (Figure 11 and Table 6 in Section E.2). Reducing the learning rate worsens FID at almost every training interval, compared to the base learning rate.
  • Gradient norm statistics (Figure 4 in Section 5 and Table 5 in Section E.1). Lowering the learning rate is seen to decrease the gradient spikes, however, compared to p≥0.5p \geq 0.5 the raw-gradient norms of the normalization exponents p=0.00,0.25p = 0.00, 0.25 remain far heavier-tailed for all values of learning rates.

The result that lowering the learning rate does not remove gradient spikes is consistent with Proposition 1. In addition, a lower learning rate slows down training speed, and for a fixed compute budget of 100k steps does not improve FID.

The other hyperparameters were chosen from publicly available flow map training procedures; in particular, the choice of time sampling distribution was taken from the popular Pytorch MeanFlow implementation on CIFAR-10 [https://github.com/Gsunshine/py-meanflow/blob/1f6d72d94247c8fdeb89489acca1a8007a6baf6c/meanflow/scripts/cifar10_v1.sh#L6].

This has been clarified, and the learning rate ablation experiment has been added to the manuscript in Figure 4 in the main text, and in Appendix E.

Thank you for the response. I am happy to keep my positive score.

We are grateful for the reviewer’s positive assessment of the paper. Their suggestion to examine the learning rate has strengthened our empirical claims and improved the final version. Thank you for your time.

Reviewer_rh81OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper theoretically analyzes training flow map models, which contains bootstrapped supervision with stopgrads. It examines whether training dynamics can achieve the true flow map despite using stopgrads/self-referential targets, also analyze when learning can become unstable. Authors handles two types of flow map objectives, i.e., MeanFlow and Lagrangian Flow, and prove convergence to the true flow map Φ⋆\Phi^{\star} under ideal continuous time functional optimization. They derive closed-form expressions for the evolving maps such as ΦτMF(x,s,t)=Φ0(Φ⋆(x,s,tτ),tτ,t),\Phi_\tau^{MF}(x,s,t)=\Phi_0\bigl(\Phi^{\star}(x,s,t_\tau),t_\tau,t\bigr), and ΦτL(x,s,t)=Φ⋆(Φ0(x,s,sτ),sτ,t).\Phi_\tau^{L}(x,s,t)=\Phi^{\star}\bigl(\Phi_0(x,s,s_\tau),s_\tau,t\bigr). MeanFlow is expressed as the true map followed by the initial map, while Lagrangian Flow is expressed as the initial map followed by the true map. Fig2 visualizes this Thrm1 with synthetic dataset, which is the agreement between the learned one and theoretically predicted one. Under additional assumptions, the authors also establish L∞L^\infty convergence bounds for both flow map objectives and weighted-L2L^2 contraction of the average velocity for MeanFlow. However, with finite-step updates, arbitrarily small initial errors can be amplified and cause divergence (Proposition1). The continuous-time analysis also shows greater sensitivity to initial errors for Lagrangian training under expansive flows (Thrm2). The authors then analyze existing adaptive normalization and convergence to a neighborhood of the true solution under additional conditions with MeanFlow (Thrm4). The main table results (CIFAR10, FFHQ) show the effect of normalization across different values of pp.

Critical strengths

The manuscript is already a high quality paper that able to make a meaningful contribution to Generative AI domain. It provides a theoretical lens on widely used points, such as stopgrads and adaptive weighting.

The main strength is that the analysis makes the training process itself interpretable. Under idealized assumptions, the authors express the MeanFlow and Lagrangian Flow maps as compositions of the initial and true flow maps, and establish continuous-time convergence. Then, they distinguish the ideal analysis from finite-step stability, where arbitrarily small initial errors can still lead to divergence with finite updates. The analysis of adaptive normalization gives a concrete account of how update magnitudes can be controlled and, under additional conditions, how MeanFlow can approach a neighborhood of the true solution. This separation between convergence of the continuous dynamics and stability of their discrete updates gives useful viewpoint. Authors also provide toy examples to make the theoretical results easier to follow. Image-scale experiments show benefits from adaptive weighting, which is consistent with observations in existing MeanFlow papers (MeanFlow, AlphaFlow, ...etc). I also appreciate authors to discuss and compare with the concurrent work (How I Learned to Stop Worrying and Love StopGrads, Neurips2026) given ICLR submission timeline. Overall, I strongly recommend acceptance.

Critical weaknesses

No major concerns about the current manuscript. The paper reads well even though it is a theoretical heavy paper. One minor comment on Fig2 is, it would be helpful to include the details like learning rate and the number of updates corresponding to τ\tau in the caption, so the figure to be more self-contained. Also, since Fig2 is about Thrm 1, could it also be placed closer to that part?

Questions for authors

None

1 comment

We would like to thank Reviewer rh81 for their positive comments on the clarity of the manuscript and the usefulness of the theoretical insights.

As suggested, empirical details such as the learning rate and number of steps have been added to the caption of Figure 2, and the figure has been placed after Theorem 1 as an illustration.

Other major changes have been highlighted in blue in the revised paper.

Comments on the paper

1 comment

We would like to thank all the reviewers for their positive feedback, that highlighted the importance and timeliness of the question of flow map stability (Reviewer 2vBZ), the clarity (Reviewer rh81) and rigor (Reviewer bCa5) of the manuscript, as well as the insights provided by the theoretical results (Reviewer bCa5 and Reviewer rh81).

Reviewers' feedback significantly improved the quality of the manuscript. Reviewers bCa5 and 2vBZ highlighted extensions of the empirical evaluation of stability during training. We thus conducted further experiments assessing the stability of flow map learning methods at different normalization exponents:

  • First, we provide additional metrics (statistics on the gradient norm) to assess the stability of the learning dynamics (Figure 4, Table 5, Appendix E.1). Gradient norms are shown to be heavier-tailed and spike higher if p≤0.25p \leq 0.25 compared to p≥0.5p \geq 0.5.
  • Second, we provide ablations over the learning rate for smaller values of p∈{0.0,0.25}p \in \{0.0, 0.25\} in Appendix E.1 and E.2, showing that decreasing learning rate does not prevent heavy-tailed gradient norms nor improve performance at matched compute.
  • Third, in Appendix E.3 we show empirical evidence that the squared residual does not grow faster than linearly, aligning with the assumed condition for convergence in Theorem 4.

Changes from the original submission are highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.