acceptodds
Under review as a conference paper at ICLR 2027

FlowSRL: Stable and Scalable Reinforcement Learning for Diffusion and Flow Models

Abstract

Online reinforcement learning has established itself as the standard approach for post-training diffusion and flow-based generative models with arbitrary reward signals. Despite their limited success, they often rely on various training tricks to remain stable, and inevitably collapse the diversity of the generated samples. In this work, we revisit both problems. To address the instability, we introduce deviation clipping, a simple trust region on the velocity update that stabilizes training. Anchoring the update to a learned flow-matching baseline rather than to the policy itself stabilizes it further and speeds up convergence. The resulting algorithm, FlowSRL, matches or outperforms prior diffusion RL methods on standard text-to-image benchmarks at significantly smaller batch sizes and roughly 2× faster convergence. To address the diversity collapse, we study two complementary mechanisms, a passive one that restricts the policy to refining trajectories the pretrained model begins, and an active entropy reward bonus that incentivizes ex- ploration beyond the base model’s distribution. Finally, we apply FlowSRL to hard reward problem and show that, combined with search during the rollout stage, it is able to solve them reliably where the existing baselines without search struggled.

Primary area:
generative models
Keywords:
Image generation, Diffusion finetuning, Few-step generation, Flow maps

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1gzbOpenReview ↗
Rating
3(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies stability, diversity collapse, and optimization efficiency in online reinforcement learning for diffusion and flow-based generative models. The authors first reinterpret DiffusionNFT and Reinforce Adjoint Matching as closely related reward-guided velocity-regression methods and show that both can become unstable under prolonged optimization. FlowSRL addresses this with two main changes: deviation clipping, which constrains each reward-guided update around the pretrained velocity field, and a learned flow-matching baseline estimated from the model’s own rollout distribution. The paper argues that these changes provide a more stable update direction without requiring several stabilization heuristics used by prior methods.
The paper then considers two related problems. To mitigate diversity collapse, it studies a passive refinement strategy that leaves the high-noise part of generation to the pretrained model and an active entropy bonus that rewards rare samples within the rollout group. Finally, it combines FlowSRL with flow-map-based rollout search for sparse or difficult rewards where standard sampling rarely discovers successful examples. Experiments on SD3.5-Medium cover GenEval, OCR, PickScore, and UniGenBench++. FlowSRL remains stable over longer training runs where DiffusionNFT and RAM frequently collapse, and the advantage becomes more pronounced when the update batch is reduced. On the difficult layout task, adding search changes a setting with zero successful samples under standard rollout into one reaching a high success rate. The diversity experiments also show a clear improvement over simple regularization on the controlled 2D problem, although the gains are smaller on text-to-image generation.

Critical strengths
  1. The stability story is strong and supported by several complementary experiments. FlowSRL is not merely slightly better at a final checkpoint. The important result is that prior regression-based methods frequently improve initially and then become unstable, whereas FlowSRL continues training across GenEval, OCR, and PickScore. The small-update-batch experiments make this distinction considerably sharper: DiffusionNFT and RAM become substantially less stable, while FlowSRL remains competitive with its own full-batch result.

  2. The paper studies failure modes beyond benchmark reward and connects them to practical training behavior. The diversity experiments are particularly useful because the 2D setting provides an exact reference distribution rather than evaluating diversity through reward alone. The hard-reward experiment also illustrates a qualitatively different failure mode: standard online RL cannot improve if useful samples are never discovered, whereas rollout search can provide the necessary training signal. In the single-prompt setting, the standard rollout never obtains a successful sample over 1000 steps, while search reaches a hit rate of roughly 0.9 within 200 steps.

Critical weaknesses
  1. The empirical generality is narrower than the title and framing suggest. The main large-scale experiments use a single backbone, SD3.5-Medium, and remain within text-to-image generation. The proposed arguments are framed broadly in terms of diffusion and flow-model RL, and the stability mechanism is presented as a general property of reward-guided velocity matching, but there is no second model family, model scale, or modality demonstrating that the same failure mode and remedy persist.

  2. Some of the “scalability” and theoretical stability claims should be stated more precisely. In the small-batch experiment, the number of rollout samples is not actually reduced: all methods still generate the full rollout group, and only the subset used for the parameter update is reduced. The experiment therefore demonstrates a substantial reduction in update compute, which is useful, but it does not reduce rollout or reward-evaluation cost. The paper itself notes that the rollout stage is kept unchanged.

Questions for authors
  1. Have the authors evaluated FlowSRL on another flow or diffusion backbone? Since the main claim concerns the stability of reward-guided velocity matching, validation on a second model family would materially affect my assessment of generality.

  2. In the reduced-update-batch experiment, can the authors report total training cost broken down into rollout, reward evaluation, and optimization? The current result convincingly reduces update compute, but the rollout group remains unchanged, so I would like to better understand the end-to-end scaling benefit.

  3. How sensitive is FlowSRL to the deviation-clipping radius across tasks and model configurations? The paper states that one value works well across most experiments, but a broader sensitivity analysis would help establish whether this is genuinely less tuning-sensitive than the methods it replaces.

  4. How should the theoretical invariant-region result be interpreted under finite-step neural-network optimization and parameter sharing? Does the authors’ guarantee extend in any approximate form to the actual LoRA training setting, or is it intended primarily as an idealized explanation of the clipping mechanism?

4 comments

Dear reviewer 1gzb,

We are glad you found our stability results convincing and our 2D experiments helpful. Below we’ll aim to address your concerns.

  • Regarding scaling our approach to other settings: We appreciate the reviewer raising this point. In the original submission, we aimed to follow prior works and showcase FlowSRL’s applicability to a variety of rewards, using the fixed SD3.5-M model. However, we agree that applying FlowSRL to other settings can help strengthen the generality and scalability arguments. To address this, we post-train Wan-2.1-1.3B, which is a popular text-to-video generation model, following the setup of Reward Velocity Matching (RVM). We evaluate FlowSRL, DiffusionNFT, RAM, and RVM on this setting and observe that during extended training, only FlowSRL maintains stable training and achieves the highest final reward. We’ve included complete details of these experiments in Section H.1 of the appendix. This section currently includes all additional results requested during the rebuttal period, and we aim to incorporate the information more seamlessly in the final version.

  • Regarding exact timings for the small-batch experiments: This was briefly discussed in Section 4.1, but we’re happy to include all the timings in detail as well. We summarize all the timing values in a table below:

    MethodUpdate batchRolloutRewardOptimizationTotal
    DiffusionNFTFull10.050.4339.8450.3
    DiffusionNFTSmall10.040.4310.0920.6
    RAMFull19.180.4335.4455.0
    RAMSmall19.170.438.8928.5
    FlowSRLFull10.060.4336.9747.5
    FlowSRLSmall10.040.4310.1220.6

    Note that RAM has roughly double the rollout generation time since the original paper uses 20 sampling steps for rollouts compared to the default 10 used by DiffusionNFT and FlowSRL.

  • Regarding the dev clip radius ablation: In our experiments, we observed that the clipping radius has a fairly large range of values within which training is stable. To demonstrate this, we ablate the clipping radius on pickscore as well as on our 2d toy settings, and provide the figures in Section H.4 of the revised appendix.

  • Regarding the assumptions of Theorem 1: As the reviewer pointed out, in Theorem 1, we make the assumption of doing continuous functional optimization. The goal of that theorem was to primarily give a mostly intuitive theoretical understanding on how clipping the target leads to local refinement of the policy model around the reference. However, by itself, it isn’t directly applicable to neural network training in practice. To address this, we add an additional theorem in Section H.2 in the revised appendix that replaces continuous functional optimization with finite-step gradient descent and shows that under some regularity conditions, the clipped targets still result in a global deviation bound.

We hope these clarifications are helpful. Please let us know if there are any other concerns we can address. Otherwise, we would be grateful if you’d consider raising your score accordingly. Thank you.

Thanks for the follow-up. The Wan experiment addresses my concern about relying on a single backbone, and the timing breakdown answers my question about the smaller update batch. Fig. H.1 makes me more confident that the stability advantage carries over to video. The new analysis in H.2 is helpful, though it still assumes exact gradients, so my question about the practical LoRA training setup is only partly resolved. I remain at weak accept (3).

Dear Reviewer 1gzb,

Thank you for your response and highlighting the remaining question about stochastic optimization. We agree that distinguishing our analysis from practical implementations is important. However, the argument in Section H.2 can actually be extended to SGD with bounded gradient errors. Specifically, consider

\quad \|\xi_k\|_2\le\delta.$$ Writing $w_k=\theta_k-\theta_{\mathrm{ref}}$ and $\kappa=1+\lambda$, the same reasoning as Eq. (24) gives $$\||w_{k+1}\||_2 \le (1-\eta\kappa\ell)\||w_k\||_2+\eta(S\rho+\delta).$$ Under the same assumptions, holding on a neighborhood of radius at least $(S\rho+\delta)/(\kappa\ell)$, this yields $$\sup_{x,t}\|v_{\theta_k}(x,t)-v^{\mathrm{ref}}(x,t)\| \le \frac{S(S\rho+\delta)}{\kappa\ell} \left[1-(1-\eta\kappa\ell)^k\right].$$ Thus, exact gradients are not essential to our argument, i.e., bounded gradient errors yield a larger bound that remains uniform over training. Moreover, setting $\delta=0$ recovers the original result. The separate question of establishing sensitivity and curvature assumptions for nonlinear LoRA is discussed at the end of Section H.2. We hope that clarifies your remaining concerns and we are happy to include this argument in our final version.

Thanks for the authors' response. I have decided to maintain my score of weak accept (3).

Reviewer_tavXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FlowSRL, a method aiming at improving the stability and efficiency of reinforcement learning for diffusion and flow models. Its main contribution is to clip the reward-guided regression target to get closer to the pre-trained velocity field, and improve the update direction based on the learned flow-matching baseline. Experiments on toy distributions and text-to-image generation with Stable Diffusion 3.5 Medium validate improved training stability and sampling diversity.

Critical strengths

The improvements in training stability and efficiency are validated by experimental results and have important practical value. Across multiple different tasks, FlowSRL remains stable where baseline methods collapse. FlowSRL also reduces the computation by using smaller update batches, which makes RL post-training more practical to use. The authors also unifiy previous methods as reward-guided velocity regression, and uses theoretical analysis and experimental results to validate the performance gain.

Critical weaknesses

The stability proof seems to assume an ideal update process, that is, continuous updates in function space, which does not cover the finite-step AdamW optimization of LoRA parameters. Therefore, the clipping regression targets may not directly guarantee the same bound on the trained model's velocity. This limits the theoretical support for the central claim on stability.

Also, the search experiments mainly compare training steps. Is there any extra cost introduced by searching itself? Key search parameters are also marked as TBD in Table F.1, which makes reproduction difficult.

Questions for authors

Please refer to the 'Critical Weaknesses' part and address the concerns listed.

1 comment

Dear reviewer tavX,

We are glad you found our practical contributions important and our paper sound. Below we’ll aim to address your concerns.

  • Regarding the assumptions of Theorem 1: As the reviewer pointed out, in Theorem 1, we make the assumption of doing continuous functional optimization. The goal of that theorem was to primarily give a somewhat intuitive theoretical understanding on how clipping the target leads to local refinement of the policy model around the reference. However, by itself, it isn’t sufficient to establish bounded deviations on standard neural network training. To address this, we add an additional theorem in Section H in the revised appendix that replaces continuous functional optimization with finite-step gradient descent and shows that under some regularity conditions, the clipped targets still result in a global deviation bound.

  • Regarding the x-axis is Figure 8: We thank the reviewer for pointing this out. Performing search during rollouts does indeed introduce an additional overhead. We’ve performed a timing analysis and reported the time it takes to generate the samples, score them with the reward, and optimize the network in the table below. As seen below, performing search roughly doubles the rollout time but since one iteration is primarily optimization-bound in these experiments, overall it results in a ~15% total overhead. Naturally, this will change depending on the task/reward and harder rewards may incur a higher search cost.

    RolloutsRewardOptimizationTotal
    Without search2.491.2820.7624.53
    With search5.711.8920.2727.87

    Additionally, we’ve added a variant of Figure 8 with the x-axis being wall clock time of the entire run as requested in Section H of the revised appendix. As seen in the results, search leads to faster convergence in the multi prompt/multi mask setting, and remains crucial for any improvement in the single prompt/single mask setting. We believe we are the first to show this combination of gradient-based search + diffusion/flow RL and hope this can lead to new avenues of research in the future.

  • Regarding missing details in the appendix: We thank the reviewer for pointing this out. We’ve filled in the missing entries in Table F.1. We’ve also added a paragraph in the previous section for the search-only experimental details that needed more discussion.

We hope these clarifications are helpful. Please let us know if there are any other concerns we can address. Otherwise, we would be grateful if you’d consider raising your score accordingly. Thank you.

Reviewer_tKbyOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FlowSRL, which introduces deviation clipping to constrain reward-guided velocity targets to a trust region around the pretrained velocity field, aiming to stabilize training. It further incorporates a learned flow-matching baseline to improve stability and accelerate convergence, along with two complementary mechanisms to mitigate diversity collapse. Experiments on SD3.5-M demonstrate competitive reward performance, improved stability with smaller update batches, and approximately twice the training efficiency.

Critical strengths
  • The paper addresses practically relevant challenges in diffusion RL, including training instability and diversity collapse. The 2D toy experiments are particularly compelling and insightful, clearly illustrating the failure modes and the effects of the proposed mechanisms.

  • The learned flow-matching baseline is well motivated by the distinction between the policy’s sampling velocity and the flow-matching field of its generated samples. The analysis and ablation provide theoretical motivation and empirical support for this design.

Critical weaknesses
  • The scalability claim is supported primarily by training with smaller update batches while keeping the rollout budget unchanged. Evidence of scaling to larger models or more demanding generation settings remains limited, and the intended scope of “scalable” could be clarified.

  • Some implementation details of the hard-reward search experiments remain unspecified: Table F.1 lists the search configuration as “TBD.”

Questions for authors
  1. Could you clarify the intended scope of “scalable”? Beyond reducing the update batch size while keeping the rollout budget unchanged, is there evidence that FlowSRL scales to larger models or more demanding generation settings?

  2. How much of the improvement on hard rewards comes from search versus the FlowSRL update rule? Would the advantage persist if DiffusionNFT or RAM used the same search procedure and computational budget?

  3. Could you provide an image-scale ablation evaluating deviation clipping and the learned flow-matching baseline both individually and jointly? While Figure G.1 isolates the baseline’s contribution with clipping enabled, a full comparison would clarify their respective effects and whether they are complementary?

3 comments

Dear reviewer tKby,

We are glad you found our paper easy to follow and our 2D experiments insightful. Below we’ll aim to address your concerns.

  • Regarding scaling our approach to other settings: As you pointed out, in the original manuscript, we called our method “scalable” due to it being able to achieve stable training and comparable reward optimization with smaller training batch sizes at reduced cost on various rewards. However, we agree that applying FlowSRL to other training settings and larger models, and showing it remains stable can help strengthen the argument. To address this, we post-train Wan-2.1-1.3B, which is a popular text-to-video generation model, following the setup of Reward Velocity Matching (RVM). We evaluate FlowSRL, DiffusionNFT, RAM, and RVM on this setting and observe that during extended training, only FlowSRL maintains stable training and achieves the highest final reward. We’ve included complete details of these experiments in Section H of the appendix. This section currently includes all additional results requested during the rebuttal period, and we aim to reorganize the information more seamlessly in the final version.

  • Regarding missing details in the appendix: We thank the reviewer for pointing this out. We’ve filled in the missing entries in Table F.1. We’ve also added a paragraph in the previous section for the search-only experimental details that needed more discussion.

  • Regarding the hard reward experiments: In our hard reward experiments, we aimed to mainly demonstrate the impact of applying gradient-based test-time search methods in the rollout generation stage. And the shown improvements should be solely attributed to this search mechanism rather than the optimization algorithm itself. For example, Figure 8 shows a comparison between runs using the same optimization algorithm (FlowSRL) with and without search. Additionally, during the rebuttal stage, we tested both DiffusionNFT and RAM combined with search, and we found that although RAM was unstable in both cases, DiffusionNFT with proper tuning exhibits similar behaviour as the one shown in Figure 8. For details, please see Section H.3 in the revised appendix. We believe we are the first to show this combination of gradient-based search + diffusion/flow RL and hope this can lead to new avenues of research in the future.

  • Regarding additional image-scale ablations: We appreciate the reviewers suggestion. We’ve added the requested 4-way ablation (fm baseline on/off, deviation clipping on/off) as a new figure in the revised appendix, for both the toy and image settings. As requested by other reviewers, we’ve also added an ablation over the deviation clipping radius as well as an additional toy experiment showing the stabilization effect of clipping when turned on vs off. Looking at the results, both mechanisms are important for stable training. However, which is more important depends on the problem setting, data modality, and training horizon.

We hope these clarifications are helpful. Please let us know if there are any other concerns we can address. Otherwise, we would be grateful if you’d consider raising your score accordingly. Thank you.

Thank you for the detailed responses and additional experiments. The text-to-video results, clarification of the search contribution, and four-way ablation of deviation clipping and the flow-matching baseline have addressed my concerns. I consider this work worthy of acceptance and will maintain my score of 3.

Dear Reviewer tKby,

Thank you for your response. We are pleased that our rebuttal has addressed your concerns and we appreciate your support for acceptance. If the additional results and clarifications have also strengthened your overall assessment of the paper, we would be grateful if you would consider increasing your score. We appreciate your time and constructive feedback.

Comments on the paper

1 comment

Dear all,

We are happy that our paper was well received by the reviewers. During the rebuttal phase, we performed several additional experiments and expanded our theoretical analysis, which we summarize below:

  • Text-to-video experiments: We’ve applied FlowSRL to the text-to-video generation setting, in which we post-trained Wan2.1-1.3B following concurrent work (RVM). The results show that compared to the baselines (DiffusionNFT, RAM, RVM), our FlowSRL is the only method that achieves stable optimization across long training horizons and obtains the highest final reward. For details and qualitative examples, please see Section H.1.
  • Extended Theorem 1 to gradient descent: As pointed out by multiple reviewers, our original Theorem 1 assumed continuous functional optimization, which does not directly reflect neural network training. To address this, we added a new theorem in Section H.2 that extends our analysis to gradient descent. We show that, under certain regularity conditions or with weight-decay-like regularization, clipped targets still provide a global bound on the deviation from the reference model.
  • Additional figures and ablations: We’ve added ablations for the deviation clipping radius, a four-way ablation with clipping (on/off) + fm baselines (on/off), a detailed timing analysis for the hard-reward search experiments, and extra figures for the search experiments. We believe these help further clarify the impact of each design decision. These results can be seen in Sections H.3-H.5.
  • Missing details and appendix corrections: We corrected two typos and added missing information to the hyperparameter table (Table F.1). We also added a paragraph clarifying the rollout procedure used in the layout search experiments.

These additions are currently all in Section H of the appendix. We plan to seamlessly integrate them into the paper in the final version. We thank the reviewers for their valuable feedback and believe these additions further strengthen our submission.

Thank you,
The authors of FlowSRL

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.