acceptodds
Under review as a conference paper at ICLR 2027

FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification

Abstract

Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods. Existing density-approximated policy gradient methods rely on stochastic SDE samplers to construct tractable transition kernels, which introduce training-inference inconsistencies and necessitates Classifier-Free Guidance (CFG). DiffusionNFT bypasses trajectory likelihoods through implicit velocity regression, but uses fixed residual coefficients in its branch targets, with rewards controlling their relative loss weights. We propose Flow Advantage-Weighted Rectification (FlowAWR), a paradigm that recasts continuous generative policy optimization as supervised regression toward a theoretically optimal velocity field. Starting from the optimal policy of a KL-constrained reward maximization, FlowAWR derives the optimal velocity field that admits a magnitude-aware, advantage-weighted rectification form, yielding SDE-free optimization. In comparative evaluations on SD3.5-Medium, FlowAWR achieves improved alignment performance alongside a 2 to 5 convergence acceleration over DiffusionNFT (e.g., reaching a 24.12 PickScore in 1.2k steps, versus 23.82 in 2.0k steps for DiffusionNFT and 23.50 in 4k steps for FlowGRPO). Under multi-reward constraints, FlowAWR sustains quality, satisfying structural rules with stable out-of-domain performance.

Primary area:
generative models
Keywords:
Generative Models, Reinforcement Learning, Flow Matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EHywOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FlowAWR, an online reinforcement learning framework that does not require SDE sampling. Starting from a KL-regularized reward maximization objective, the method derives the theoretically optimal velocity field and transforms policy optimization into Advantage-Weighted Velocity Regression. Compared with traditional policy gradient methods, it does not require explicit estimation of the likelihood of generation trajectories. Compared with DiffusionNFT, it adaptively adjusts the direction and magnitude of velocity field corrections based on rewards, thereby improving training efficiency and generation quality. In addition, extensive experiments demonstrate the effectiveness of the proposed method. Experiments on SD3.5-Medium show that FlowAWR achieves significantly faster convergence than DiffusionNFT in single-reward optimization tasks.

Critical strengths
  1. The design of FlowAWR is very interesting. Compared with the heuristically designed positive and negative velocity regression targets in DiffusionNFT, FlowAWR starts from KL-regularized reward maximization, derives the theoretically optimal velocity field, and proves that the corresponding Advantage-Weighted Regression has the same optimization gradients as the ideal objective under ideal conditions.
  2. DiffusionNFT uses fixed residual correction coefficients and adjusts the weights of different regression terms through rewards. In contrast, FlowAWR directly adjusts the direction and magnitude of the velocity regression target using Advantage, allowing dynamic updates for different samples.
  3. FlowAWR achieves notable improvements in metrics such as PickScore and HPSv2.1
Critical weaknesses
  1. The ablation experiments are insufficient. For example, (1) Does the performance improvement mainly come from the adaptive advantage or the change in the regression target formulation? (2) How large is the difference between the Group-based Advantage and a more accurate approximation of the conditional Advantage?
  2. Although the method provides detailed mathematical derivations, the core algorithm is not fundamentally different from DiffusionNFT. For example, the paper emphasizes that DiffusionNFT uses fixed residual correction coefficients, while FlowAWR employs adaptive Advantage. However, the main difference between the two methods in practical optimization may lie more in the design of the reward-to-advantage mapping, rather than a fundamental shift from fixed to adaptive correction coefficients.
Questions for authors
  1. Does the performance improvement primarily come from the adaptive Advantage or the change in the regression target formulation? Could the authors clarify this point or provide additional experiments to disentangle their contributions?
  2. How significant is the difference between the Group-based Advantage and a more accurate approximation of the Conditional Advantage? If the authors can adequately address these concerns, I would consider increasing my score.
Reviewer_EaZtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FlowAWR, an SDE-free online RL framework for flow-matching generative models. Starting from the closed-form optimal policy under KL-regularized reward maximization, the authors map the optimal terminal distribution to intermediate probability paths and derive that the optimal velocity field equals the reference field plus an advantage-weighted residual expectation. They then cast policy optimization as supervised regression toward the optimal velocity and prove that the AWR loss and the ideal loss share the same global optimum. Experiments on SD3.5-Medium cover single-reward and multi-reward settings, reporting roughly 2×–5× convergence speedups over DiffusionNFT and better alignment metrics.

Critical strengths
  • The theoretical derivation is reasonably clear: it connects the KL-regularized optimal policy, the optimal velocity field, and advantage-weighted regression, and gives a unified interpretation of DiffusionNFT’s positive/negative velocity fields.
  • The method avoids the mismatch between SDE sampling trajectories and deterministic ODE inference while retaining CFG-free generation, offering a practical efficiency advantage.
  • Compared with DiffusionNFT’s fixed-magnitude linear reward coefficients, FlowAWR uses sample-dependent exponentially tilted advantages to determine rectification magnitudes, making the regression target closer to a KL-regularized optimal field. This plausibly contributes to faster convergence and improved stability.
Critical weaknesses
  • The core algorithmic difference from DiffusionNFT is relatively limited. From the perspective of the regression target, DiffusionNFT’s two-branch loss can be folded into an equivalent single-target regression whose rectification coefficient is a linear function of normalized reward with fixed magnitude. FlowAWR mainly replaces this coefficient with a group-relative exponentially tilted advantage and rewrites the two-branch weighted loss as a single-target regression. Both share the EMA reference policy, CFG-free generation, group sampling, and other outer-loop components. The main contribution is therefore better characterized as a more principled reparameterization of the advantage/rectification interface for NFT-style forward-process RL, rather than a structurally different framework.
  • **The paper does not address the most critical issue of NFT-style forward-process RL:stable training still relies on an EMA old policy which needs to be tuned per task. Claims about stability and efficiency should therefore be framed as improvements within this framework rather than as resolutions of its structural limitations.
  • Empirical comparisons remain dominated by DiffusionNFT; FlowGRPO/AWM/TempFlow-GRPO appear only in parts of the appendix or curves. The column structure and repeated rows in Table 1 are hard to interpret, and OCR mixes task-specific and general configurations, weakening the conclusions. Multi-reward training uses a hand-designed staged schedule, with insufficient sensitivity analysis on stage boundaries, reward weights, and random seeds. Statistical significance of some gains is not fully reported.
Questions for authors
  1. Beyond the computation of the advantage/rectification coefficient and the loss form (two-branch weighting → single-target regression), what structural changes does FlowAWR introduce relative to DiffusionNFT? Please list them explicitly.

  2. The method still uses an EMA reference policy. Did the authors evaluate a variant without EMA, or with an alternative stabilization mechanism? If not, can they clarify the extent to which FlowAWR still depends on this off-policy stabilization?

  3. Were the stage divisions and weights in multi-reward sequential training systematically tuned? Is the method sensitive to stage order?

Reviewer_rwkbOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Online RL for flow models either relies on SDE samplers to get tractable likelihoods (FlowGRPO, DanceGRPO) or, like DiffusionNFT, uses velocity regression with fixed residual coefficients. The paper derives the optimal velocity field of KL-regularized reward maximization, writes it as the reference field voldv^{old} rectified by a centered advantage AA (the exponentiated reward, with temperature γ\gamma, divided by a value function ϕ\phi, minus 1), and trains vθv_\theta to regress toward vold+A(ut−vold)v^{old} + A (u_t - v^{old}) using a group-based estimate of ϕ\phi and γ\gamma set to the reward standard deviation.

Critical strengths
  1. The method is simple and fits existing pipelines. It needs no SDE sampling, no trajectory likelihoods, and no CFG at inference, and per-update cost is close to DiffusionNFT, so it can replace the NFT loss with little change.
  2. The derivation is clear and easy to check. It goes from the tilted optimal policy to intermediate marginals, then through Tweedie's formula to a sample-wise regression target. It also gives a clean reading of DiffusionNFT as the same residual form with fixed coefficients ANFTA_{NFT} equal to plus or minus one over β\beta, and of SFT as the equal-reward case.
  3. The empirical study is fairly broad for this line of work. It covers single-reward runs on GenEval, OCR, PickScore and HPSv2.1, a multi-reward curriculum, out-of-domain DrawBench metrics, GPU-hour comparisons with AWM and TempFlow-GRPO, a FLUX.1-dev run with HPSv3, and ablations on GG, γ\gamma, the advantage form, and the EMA schedule. The paper is also open about the OCR configuration difference for DiffusionNFT (Appendix C.5).
Critical weaknesses
  1. In Theorem 1 the advantage depends on xtx_t through ϕ(xt,c,t)\phi(x_t, c, t), a posterior expectation given xtx_t. Eq. 11 replaces it with the group mean of the exponentiated rewards, which does not depend on xtx_t or tt and estimates the partition function Z(c)Z(c) instead. With this change the gradient equivalence of Theorem 2 no longer holds, so the claim that the loss "drives" vθv_\theta to v∗v^* does not carry over to the practical method. On top of that, Appendix C.4 adds logit clipping to [-5, 5], and for GenEval and OCR a rescaling by the within-group standard deviation. This produces advantages below -1 (down to -1.76 for OCR and -1.54 for GenEval in Figure 9), which the definition in Eq. 9 can never give, since its first term is positive. The reference policy is an EMA of θ\theta, and it is not stated whether rollouts come from θ\theta or from θold\theta^{old}, while the theory requires x1x_1 to be drawn from poldp^{old}. The theory itself also overlaps with known results: tilted-distribution velocity and score fields, and reward-weighted flow matching, have been derived before. Zhang et al. (2025) is cited, but these are not: Lu et al., "Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning" (ICML 2023); Domingo-Enrich et al., "Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control" (arXiv 2024); Fan et al., "Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization" (ICLR 2025); Uehara et al., "Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control" (arXiv 2024).
  2. The abstract compares FlowAWR's in-domain PickScore of 24.12 (Table 1) with FlowGRPO's 23.50, but 23.50 is the DrawBench PickScore of FlowGRPO in Table 2. Its in-domain PickScore there is 23.33. The 5.2x OCR speedup in Figure 3(c) is measured against DiffusionNFT's general configuration, which only reaches 0.89 after 520 updates. The official OCR configuration reaches 0.98 after 400 updates, and in Table 1 FlowAWR gets 0.975 at 260 updates, slightly lower. Final gains over DiffusionNFT are small: GenEval is tied at 0.98, PickScore is 24.12 vs 23.82, and in the multi-reward setting OCR is 0.91 vs 0.90 and GenEval is tied at 0.93. All results come from single runs, with no seeds or error bars, so it is not clear whether these gaps are real.
  3. Sec. 4.2 says FlowAWR "exhibits reduced reward hacking" based on out-of-domain metrics, but each FlowAWR row in Table 1 is an earlier checkpoint (0.5k vs 1k, 0.26k vs 0.4k, 1.2k vs 2.0k). Earlier checkpoints usually drift less, so this is not a matched comparison. The "Adaptive Advantage" ablation compares against a single-branch NFT-style target that collapses (Appendix C.4). This is not DiffusionNFT, which does not collapse, so the ablation does not show what the adaptive coefficient adds over the real two-branch NFT loss. The γ\gamma and EMA ablations are run only on OCR, for at most about 250 iterations. The FLUX.1-dev result is a single learning curve with no table, no out-of-domain metrics, and CFG turned on in rollouts and evaluation, unlike the main setting.
Questions for authors
  • How do you justify replacing ϕ(xt,c,t)\phi(x_t, c, t) with an estimate that does not depend on xtx_t? Does any part of Theorem 2 still hold for the Eq. 11 estimator, for example as a bound?
  • Are rollouts drawn from θ\theta or from the EMA θold\theta^{old}? If from θ\theta, how does this fit the requirement that x1x_1 is drawn from poldp^{old}?
  • How much of the gain comes from the std rescaling and clipping in Appendix C.4? Please show GenEval and OCR runs with the pure Eq. 11 advantage, and explain advantages below -1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Our General Response: Starting Point, Significance, and Additional Results

We thank all reviewers for their constructive feedback. They find the derivation clear and easy to check, with a unified interpretation of DiffusionNFT (rwkb, EaZt, EHyw), and the empirical study broad, with notable gains on preference rewards (rwkb, EHyw).

Starting point

The reward scores only the final sample, so the KL-regularized optimum is a terminal distribution, p∗(x1∣c)∝pold(x1∣c),eR(c,x1)/γp^*(x_1|c)\propto p^{\rm old}(x_1|c)\\,e^{R(c,x_1)/\gamma} (Eq. 6). We ask which velocity field reaches it while keeping the deployment advantages of flow models, namely deterministic ODE sampling and, on SD3.5-M, CFG-free generation. This faces three obstacles: p∗p^* alone gives no guidance along the generation process, the resulting correction ∇xtlog⁡ϕ\nabla_{x_t}\log\phi is intractable, and the exact normalizer ϕ(xt,c,t)\phi(x_t,c,t) requires branching rollouts from every xtx_t. FlowAWR addresses them in turn:

(i) Propagating p∗p^* through the forward noising kernel gives pt∗p_t^* (Prop. 1) and v∗=vold+1−tt∇xtlog⁡ϕv^*=v^{\rm old}+\frac{1-t}{t}\nabla_{x_t}\log\phi (Eq. 8).

(ii) The identity ∇xtlog⁡pold(x1∣xt,c)=t1−t(ut−vold)\nabla_{x_t}\log p^{\rm old}(x_1|x_t,c)=\frac{t}{1-t}(u_t-v^{\rm old}) rewrites this correction exactly as a signed residual expectation, v∗=vold+E[A,(ut−vold),∣,xt]v^*=v^{\rm old}+\mathbb{E}[\mathcal{A}\\,(u_t-v^{\rm old})\\,|\\,x_t], which can be regressed per sample from scalar rewards (Theorems 1–2).

(iii) The group mean over the GG rollouts of θold\theta^{\rm old} estimates ϕ0=ϕ(x0,c,0)\phi_0=\phi(x_0,c,0) at no extra cost, and Proposition 2 characterizes the resulting target exactly.

The CFG-free rollouts, the EMA reference (both sampler and KL anchor), and group sampling therefore all follow from the same objective rather than added heuristics.

Significance

  • Exact characterization of the practical estimator (rwkb, EHyw). With ρt=ϕ(xt,c,t)/ϕ0\rho_t=\phi(x_t,c,t)/\phi_0, the group-based loss learns v~=vold+ρt(v∗−vold)\tilde v=v^{\rm old}+\rho_t(v^*-v^{\rm old}), where ρt>0\rho_t>0, E[ρt]=1\mathbb{E}[\rho_t]=1, and ρ0≡1\rho_0\equiv1. The practical target thus keeps the direction and fixed points of the KL-regularized optimum, and only the step size varies.
  • Principled direction and adaptive magnitude (EaZt, EHyw). Under squared error, DiffusionNFT's two branches fold into the same residual form, vold+2r−1β(ut−vold)v^{\rm old}+\frac{2r-1}{\beta}(u_t-v^{\rm old}). Our derivation thus gives a principled explanation of why its targets rectify voldv^{\rm old} along ut−voldu_t-v^{\rm old}, whereas the magnitude (2r−1)/β(2r-1)/\beta is set by hand through β\beta. FlowAWR derives the magnitude as well, as the sample-adaptive coefficient A=eR/γ/eR/γ‾−1\mathcal{A}=e^{R/\gamma}/\overline{e^{R/\gamma}}-1, so that both the direction and the strength of the rectification follow from the KL-regularized optimum.
  • Difference from prior tilted-field methods (rwkb). CEP trains an auxiliary energy network, while Uehara et al. and Adjoint Matching require differentiable rewards and stochastic dynamics. Reward-weighted flow matching (ORW-CFM-W2, EFM) places nonnegative weights outside the loss, so every sample pulls vθv_\theta toward itself. FlowAWR places a signed coefficient inside the target, centered at voldv^{\rm old}, so low-reward samples induce a reverse rectification.

Additional results

ExperimentReviewersKey result
Only A\mathcal{A} replaced by (2r−1)/β(2r-1)/\beta (Fig. R1(a,b))EaZt, EHyw, rwkbPickScore 23.30 vs. 23.84 (500 updates); OCR 0.81 (β=1\beta=1) or collapse (β=0.1\beta=0.1) vs. 0.975
DiffusionNFT single-target vs. two-branch (Fig. R1(e))EHyw, rwkbOCR 0.94 vs. 0.98
Pure Eq. 11 advantagerwkbGenEval 0.976, OCR 0.97; 9 of 10 DrawBench metrics above DiffusionNFT
Matched 0.5k updatesrwkbAll five DrawBench metrics higher on GenEval and PickScore
γ\gamma and EMA on PickScore (Fig. R1(c,f))rwkb, EaZtAdaptive γ\gamma and default EMA best (23.84 at 500 updates)
Seeds 42 / 0 / 1234rwkb, EaZt23.96 / 23.98 / 23.98 at 800 updates, all above DiffusionNFT's 23.82 at 2.0k
AWM and TempFlow-GRPOEaZtBest on 6 of 8 metrics (PickScore); 0.98 GenEval at DiffusionNFT's cost
FLUX.1-dev on DrawBenchrwkbHighest HPSv3 (15.69) with about 38% of TempFlow-GRPO's GPU-hours

Regression-based methods share rollout steps, group size, LoRA, and hardware, and FlowAWR keeps DiffusionNFT's default EMA without task-specific tuning. Baselines use their official configurations, and for OCR we report both DiffusionNFT configurations (App. C.5, Table S1). The revised PDF includes Proposition 2 (Sec. 7.1) and all additional ablations (Sec. 7.2, Fig. R1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.