- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FlowAWR, an online reinforcement learning framework that does not require SDE sampling. Starting from a KL-regularized reward maximization objective, the method derives the theoretically optimal velocity field and transforms policy optimization into Advantage-Weighted Velocity Regression. Compared with traditional policy gradient methods, it does not require explicit estimation of the likelihood of generation trajectories. Compared with DiffusionNFT, it adaptively adjusts the direction and magnitude of velocity field corrections based on rewards, thereby improving training efficiency and generation quality. In addition, extensive experiments demonstrate the effectiveness of the proposed method. Experiments on SD3.5-Medium show that FlowAWR achieves significantly faster convergence than DiffusionNFT in single-reward optimization tasks.
Critical strengths
- The design of FlowAWR is very interesting. Compared with the heuristically designed positive and negative velocity regression targets in DiffusionNFT, FlowAWR starts from KL-regularized reward maximization, derives the theoretically optimal velocity field, and proves that the corresponding Advantage-Weighted Regression has the same optimization gradients as the ideal objective under ideal conditions.
- DiffusionNFT uses fixed residual correction coefficients and adjusts the weights of different regression terms through rewards. In contrast, FlowAWR directly adjusts the direction and magnitude of the velocity regression target using Advantage, allowing dynamic updates for different samples.
- FlowAWR achieves notable improvements in metrics such as PickScore and HPSv2.1
Critical weaknesses
- The ablation experiments are insufficient. For example, (1) Does the performance improvement mainly come from the adaptive advantage or the change in the regression target formulation? (2) How large is the difference between the Group-based Advantage and a more accurate approximation of the conditional Advantage?
- Although the method provides detailed mathematical derivations, the core algorithm is not fundamentally different from DiffusionNFT. For example, the paper emphasizes that DiffusionNFT uses fixed residual correction coefficients, while FlowAWR employs adaptive Advantage. However, the main difference between the two methods in practical optimization may lie more in the design of the reward-to-advantage mapping, rather than a fundamental shift from fixed to adaptive correction coefficients.
Questions for authors
- Does the performance improvement primarily come from the adaptive Advantage or the change in the regression target formulation? Could the authors clarify this point or provide additional experiments to disentangle their contributions?
- How significant is the difference between the Group-based Advantage and a more accurate approximation of the Conditional Advantage? If the authors can adequately address these concerns, I would consider increasing my score.
Our General Response: Starting Point, Significance, and Additional Results
We thank all reviewers for their constructive feedback. They find the derivation clear and easy to check, with a unified interpretation of DiffusionNFT (rwkb, EaZt, EHyw), and the empirical study broad, with notable gains on preference rewards (rwkb, EHyw).
The reward scores only the final sample, so the KL-regularized optimum is a terminal distribution, p∗(x1∣c)∝pold(x1∣c),eR(c,x1)/γ (Eq. 6). We ask which velocity field reaches it while keeping the deployment advantages of flow models, namely deterministic ODE sampling and, on SD3.5-M, CFG-free generation. This faces three obstacles: p∗ alone gives no guidance along the generation process, the resulting correction ∇xtlogϕ is intractable, and the exact normalizer ϕ(xt,c,t) requires branching rollouts from every xt. FlowAWR addresses them in turn:
(i) Propagating p∗ through the forward noising kernel gives pt∗ (Prop. 1) and v∗=vold+t1−t∇xtlogϕ (Eq. 8).
(ii) The identity ∇xtlogpold(x1∣xt,c)=1−tt(ut−vold) rewrites this correction exactly as a signed residual expectation, v∗=vold+E[A,(ut−vold),∣,xt], which can be regressed per sample from scalar rewards (Theorems 1–2).
(iii) The group mean over the G rollouts of θold estimates ϕ0=ϕ(x0,c,0) at no extra cost, and Proposition 2 characterizes the resulting target exactly.
The CFG-free rollouts, the EMA reference (both sampler and KL anchor), and group sampling therefore all follow from the same objective rather than added heuristics.
Regression-based methods share rollout steps, group size, LoRA, and hardware, and FlowAWR keeps DiffusionNFT's default EMA without task-specific tuning. Baselines use their official configurations, and for OCR we report both DiffusionNFT configurations (App. C.5, Table S1). The revised PDF includes Proposition 2 (Sec. 7.1) and all additional ablations (Sec. 7.2, Fig. R1).