- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the stability and convergence of flow map training, especially for Mean Flow. Even though this one-step generations uses some well-known empirical tricks such as stop gradient and adaptive loss normalization, the full picture of optimization dynamics is not yet fully understood in the previous work.
The authors analyze the training of MeanFlow and Lagrangian models and show that both can converge to the true flow map under some assumptions and give the upper boundary for convergence rates. In addition, they also analyze the instability of the training dynamics as well. They find that using a fixed learning rate can cause training to diverge, even when the initial error is very small. To address this problem, they study adaptive normalization and show that chose of different $ p $ can result in different convergence guarantee, which has been showed in the experiments
Critical strengths
1: The paper partially addresses a fundamental question in mean flow training: why do stop-gradient-based objectives work, and what makes their optimization stable? Given the recent burst of one-step generative models, I think understanding these mechanisms are valuable.
2: The paper provides solid theoretical analysis. The authors establish convergence guarantees under relatively mild assumptions and provide useful insights into training stability and also provide adaptive normalization as the solution.
4: The theoretical findings are supported by synthetic experiments and image generation experiments on CIFAR-10 and FFHQ, where appropriate normalization indeed improves FID.
Critical weaknesses
1: Although the paper provides solid theoretical guarantees for the stability of mean flow training, but the experiments mainly focus on generation quality (FID). It would be helpful for me to see more direct evidence of training stability, such as gradient norms, to better support the theoretical findings.
2: The other assumptions seem make sense. However, the convergence guarantee in Theorem 4 relies on the additional assumption that the squared residual norm grows at most linearly with training iterations,
but it is unclear whether this assumption holds in practice.
Questions for authors
1: In Proposition 1, it seems that the authors assume a fixed learning rate when analyzing training instability during the training dynamics? However, practical MeanFlow training typically uses adaptive learning-rate schedules. Could the authors clarify whether the same instability would occur under these more realistic training settings (e.g changing learning rate)?
2: As shown the weakness above that you assume
Could the authors give us more intuition and explain whether this assumption holds in practice?
3: Theorem 4 provides an additional convergence guarantee for p=1, but the experiments show that p=0.75 often achieves better FID (under NFE=1). Could the authors explain this difference between the theoretical results and empirical performance?
6 comments
(2/2)
W2/Q2 continued.
-
Mathematical guarantees. We analyse this more in Appendix B.5. The assumption of the bounded growth of the unnormalized training loss is needed only for the condition that Letting , is the adaptive factor that we multiply our gradient update by when , and so can be viewed as an adaptive step size. The condition that is then an assumption that the sum of the step sizes we take is infinite, which is a classical assumption on step size schedules for gradient descent methods [3]. In our case, the step sizes are not chosen at the beginning and instead depend on the residual error of the map at step , hence this condition is taken to be an assumption. As is shown in Appendix B.4, gradient descent with cannot move further away from unless it is already within an neighborhood of it, in , without any bounded growth assumption.
Assuming boundedness of and above and below on our domain, this means is always within some ball of , and so if , is uniformly bounded. By the triangle inequality, the condition that is then implied by a condition on the target . As , this is a condition that the derivatives of do not grow overly fast (quicker than ) in the direction of the velocity.
Q1: In Proposition 1, it seems that the authors assume a fixed learning rate when analyzing training instability during the training dynamics? However, practical MeanFlow training typically uses adaptive learning-rate schedules. Could the authors clarify whether the same instability would occur under these more realistic training settings (e.g changing learning rate)?
This is an interesting question that depends heavily on the choice of learning-rate schedule used. Our analysis assumes a fixed which matches the current best practice for training flow map models, such as Improved Mean Flow [1] and Terminal Velocity Matching [2] which take a constant learning rate throughout training, apart from a warm-up near the start for iMF. For learning rate methods which anneal to a positive constant as number of steps tends to infinity, our counterexample exhibiting blow up (Appendix B.4) still diverges by taking to be greater than , where is the minimum learning rate encountered during training and after any warmup. For any practical training run over a finite number of optimization steps, exists and is positive, hence such a counterexample would increase in size by a factor of . The only case which this does not cover is the theoretical convergence of adaptive schemes in which , over an infinite number of steps. In this case, a counterexample would likely be constructed by piecing together different eigenmodes of the update map in different spatiotemporal parts of the domain; we leave this for future work.
Q3: Theorem 4 provides an additional convergence guarantee for p=1, but the experiments show that p=0.75 often achieves better FID (under NFE=1). Could the authors explain this difference between the theoretical results and empirical performance?
Theorem 4 provides sufficient conditions for bounded updates and convergence to within some neighborhood of the ground truth. We do not claim that is necessary for convergence. In practice many factors could influence the convergence of flow map training methods, such as architecture, optimizer, and problem setting. In our case, we use a neural network parametrization which does not follow the exact gradient dynamics (Limitations, Section 6), and the image scale runs use Adam instead of the analyzed gradient descent. Practical training also uses per-sample weights; lower could result in quicker convergence and better performance as it may shift the relative weights of training samples. Our experiments aim to test the qualitative predictions of the theory, that achieves stable training while in exhibits forms of instability (Figure 3, Table 1, Figure 4, Appendix E), instead of showing that a specific value of is optimal.
[1] Z. Geng, Y. Lu, Z. Wu, E. Shechtman, J. Z. Kolter, and K. He. Improved mean flows: On the challenges of fastforward generative models. In CVPR, 2026.
[2] L. Zhou, M. Parger, A. Haque, and J. Song. Terminal velocity matching. In ICLR, 2026.
[3] H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22(3):400–407, 1951.
We would like to thank again the reviewer for their feedback, and we would be more than happy to answer any lingering questions.
(1/2)
We would like to thank Reviewer 2vBZ for their detailed review, highlighting the contribution of our analysis to the understanding of flow maps.
We thank the reviewer for their suggestion: we agree that FID is mainly a metric of performance. In order to better monitor stability, we track gradient statistics across training.
To the paper, we have added an additional experiment, in Figure 4, where we track the gradient norms of MeanFlow on FFHQ for values of p in {0.0,0.25,0.5,0.75,1.0}. We record both the raw gradient norm, and the norm of the gradient after Adam’s preconditioning, over 100k steps of training. Different values of the normalization exponent p have different raw gradient scales, so to make the statistics comparable, we scale each value of p independently by the inverse of the median gradient norm. Adam preconditioned gradients remain unscaled. Using these statistics, we observe three phenomena.
These provide an alternative measure of the difference in stability between values of p.
Thank you for raising this point; it is a very pertinent question towards understanding the guarantees that gradient descent for flow map training has. We tackle this assumption in multiple parts.