- Rating
- 3(−1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies stability, diversity collapse, and optimization efficiency in online reinforcement learning for diffusion and flow-based generative models. The authors first reinterpret DiffusionNFT and Reinforce Adjoint Matching as closely related reward-guided velocity-regression methods and show that both can become unstable under prolonged optimization. FlowSRL addresses this with two main changes: deviation clipping, which constrains each reward-guided update around the pretrained velocity field, and a learned flow-matching baseline estimated from the model’s own rollout distribution. The paper argues that these changes provide a more stable update direction without requiring several stabilization heuristics used by prior methods.
The paper then considers two related problems. To mitigate diversity collapse, it studies a passive refinement strategy that leaves the high-noise part of generation to the pretrained model and an active entropy bonus that rewards rare samples within the rollout group. Finally, it combines FlowSRL with flow-map-based rollout search for sparse or difficult rewards where standard sampling rarely discovers successful examples.
Experiments on SD3.5-Medium cover GenEval, OCR, PickScore, and UniGenBench++. FlowSRL remains stable over longer training runs where DiffusionNFT and RAM frequently collapse, and the advantage becomes more pronounced when the update batch is reduced. On the difficult layout task, adding search changes a setting with zero successful samples under standard rollout into one reaching a high success rate. The diversity experiments also show a clear improvement over simple regularization on the controlled 2D problem, although the gains are smaller on text-to-image generation.
Critical strengths
-
The stability story is strong and supported by several complementary experiments. FlowSRL is not merely slightly better at a final checkpoint. The important result is that prior regression-based methods frequently improve initially and then become unstable, whereas FlowSRL continues training across GenEval, OCR, and PickScore. The small-update-batch experiments make this distinction considerably sharper: DiffusionNFT and RAM become substantially less stable, while FlowSRL remains competitive with its own full-batch result.
-
The paper studies failure modes beyond benchmark reward and connects them to practical training behavior. The diversity experiments are particularly useful because the 2D setting provides an exact reference distribution rather than evaluating diversity through reward alone. The hard-reward experiment also illustrates a qualitatively different failure mode: standard online RL cannot improve if useful samples are never discovered, whereas rollout search can provide the necessary training signal. In the single-prompt setting, the standard rollout never obtains a successful sample over 1000 steps, while search reaches a hit rate of roughly 0.9 within 200 steps.
Critical weaknesses
-
The empirical generality is narrower than the title and framing suggest. The main large-scale experiments use a single backbone, SD3.5-Medium, and remain within text-to-image generation. The proposed arguments are framed broadly in terms of diffusion and flow-model RL, and the stability mechanism is presented as a general property of reward-guided velocity matching, but there is no second model family, model scale, or modality demonstrating that the same failure mode and remedy persist.
-
Some of the “scalability” and theoretical stability claims should be stated more precisely. In the small-batch experiment, the number of rollout samples is not actually reduced: all methods still generate the full rollout group, and only the subset used for the parameter update is reduced. The experiment therefore demonstrates a substantial reduction in update compute, which is useful, but it does not reduce rollout or reward-evaluation cost. The paper itself notes that the rollout stage is kept unchanged.
Questions for authors
-
Have the authors evaluated FlowSRL on another flow or diffusion backbone? Since the main claim concerns the stability of reward-guided velocity matching, validation on a second model family would materially affect my assessment of generality.
-
In the reduced-update-batch experiment, can the authors report total training cost broken down into rollout, reward evaluation, and optimization? The current result convincingly reduces update compute, but the rollout group remains unchanged, so I would like to better understand the end-to-end scaling benefit.
-
How sensitive is FlowSRL to the deviation-clipping radius across tasks and model configurations? The paper states that one value works well across most experiments, but a broader sensitivity analysis would help establish whether this is genuinely less tuning-sensitive than the methods it replaces.
-
How should the theoretical invariant-region result be interpreted under finite-step neural-network optimization and parameter sharing? Does the authors’ guarantee extend in any approximate form to the actual LoRA training setting, or is it intended primarily as an idealized explanation of the clipping mechanism?
4 comments
Thanks for the follow-up. The Wan experiment addresses my concern about relying on a single backbone, and the timing breakdown answers my question about the smaller update batch. Fig. H.1 makes me more confident that the stability advantage carries over to video. The new analysis in H.2 is helpful, though it still assumes exact gradients, so my question about the practical LoRA training setup is only partly resolved. I remain at weak accept (3).
Dear Reviewer 1gzb,
Thank you for your response and highlighting the remaining question about stochastic optimization. We agree that distinguishing our analysis from practical implementations is important. However, the argument in Section H.2 can actually be extended to SGD with bounded gradient errors. Specifically, consider
\quad \|\xi_k\|_2\le\delta.$$ Writing $w_k=\theta_k-\theta_{\mathrm{ref}}$ and $\kappa=1+\lambda$, the same reasoning as Eq. (24) gives $$\||w_{k+1}\||_2 \le (1-\eta\kappa\ell)\||w_k\||_2+\eta(S\rho+\delta).$$ Under the same assumptions, holding on a neighborhood of radius at least $(S\rho+\delta)/(\kappa\ell)$, this yields $$\sup_{x,t}\|v_{\theta_k}(x,t)-v^{\mathrm{ref}}(x,t)\| \le \frac{S(S\rho+\delta)}{\kappa\ell} \left[1-(1-\eta\kappa\ell)^k\right].$$ Thus, exact gradients are not essential to our argument, i.e., bounded gradient errors yield a larger bound that remains uniform over training. Moreover, setting $\delta=0$ recovers the original result. The separate question of establishing sensitivity and curvature assumptions for nonlinear LoRA is discussed at the end of Section H.2. We hope that clarifies your remaining concerns and we are happy to include this argument in our final version.Thanks for the authors' response. I have decided to maintain my score of weak accept (3).
Dear reviewer 1gzb,
We are glad you found our stability results convincing and our 2D experiments helpful. Below we’ll aim to address your concerns.
Regarding scaling our approach to other settings: We appreciate the reviewer raising this point. In the original submission, we aimed to follow prior works and showcase FlowSRL’s applicability to a variety of rewards, using the fixed SD3.5-M model. However, we agree that applying FlowSRL to other settings can help strengthen the generality and scalability arguments. To address this, we post-train Wan-2.1-1.3B, which is a popular text-to-video generation model, following the setup of Reward Velocity Matching (RVM). We evaluate FlowSRL, DiffusionNFT, RAM, and RVM on this setting and observe that during extended training, only FlowSRL maintains stable training and achieves the highest final reward. We’ve included complete details of these experiments in Section H.1 of the appendix. This section currently includes all additional results requested during the rebuttal period, and we aim to incorporate the information more seamlessly in the final version.
Regarding exact timings for the small-batch experiments: This was briefly discussed in Section 4.1, but we’re happy to include all the timings in detail as well. We summarize all the timing values in a table below:
Note that RAM has roughly double the rollout generation time since the original paper uses 20 sampling steps for rollouts compared to the default 10 used by DiffusionNFT and FlowSRL.
Regarding the dev clip radius ablation: In our experiments, we observed that the clipping radius has a fairly large range of values within which training is stable. To demonstrate this, we ablate the clipping radius on pickscore as well as on our 2d toy settings, and provide the figures in Section H.4 of the revised appendix.
Regarding the assumptions of Theorem 1: As the reviewer pointed out, in Theorem 1, we make the assumption of doing continuous functional optimization. The goal of that theorem was to primarily give a mostly intuitive theoretical understanding on how clipping the target leads to local refinement of the policy model around the reference. However, by itself, it isn’t directly applicable to neural network training in practice. To address this, we add an additional theorem in Section H.2 in the revised appendix that replaces continuous functional optimization with finite-step gradient descent and shows that under some regularity conditions, the clipped targets still result in a global deviation bound.
We hope these clarifications are helpful. Please let us know if there are any other concerns we can address. Otherwise, we would be grateful if you’d consider raising your score accordingly. Thank you.