acceptodds
Withdrawn from ICLR 2027

AdvantageFlow: Regularized Advantage-Weighted RL in Flow Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

We present AdvantageFlow, a forward-process reinforcement learning (RL) algorithm for rectified flow models. The algorithm minimizes an advantage-weighted prediction loss, which maximizes reward, regularized by the rollout policy, which convexifies the objective and makes its optimization stable. Our objective can be viewed as fitting a local reward-improving target distribution. The rollout regularization arises as a variance reduction step. We evaluate AdvantageFlow empirically on text-to-image generation with Stable Diffusion 3.5 Medium and FLUX.1, and compare it to both forward- and reverse-process RL algorithms.

Primary area:
generative models
Keywords:
RL, flow models, text-to-image generation, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9MzJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

AdvantageFlow performs reward-related regression-based prediction on images generated by a rollout flow model and incorporates a regularization term involving the rollout and reference model predictions. Essentially, AdvantageFlow builds upon AWM by adding a regularization term and provides a series of mathematical proofs to support this approach.

Critical strengths
  1. The paper involves a significant number of mathematical formulas and provides closed-form proofs, although these proofs are somewhat verbose;
  2. Experiments were conducted on multiple datasets to demonstrate the effectiveness of the proposed method;
Critical weaknesses
  1. The writing style feels heavily AI-generated; the introduction and abstract fail to clearly distinguish AdvantageFlow from DiffusionNFT and AWM, or to highlight the core improvements. Regarding lines 64–66, the core concept described is already shared across existing Diffusion RL methods and cannot be considered a unique contribution of this paper. Furthermore, the explanation of why AdvantageFlow qualifies as a least-squares algorithm (lines 77–78) is unclear.
  2. Regarding the methodological contribution, my understanding is that the authors simply added a regularization term to AWM to impose constraints and prevent gradient divergence; there appear to be no other significant innovations.
  3. Regarding the experiments: how were λ\lambda and γ\gamma determined? Why are the numerical results for the GenEval evaluation so low?
Questions for authors

In my view, this is an incremental study—essentially adding a regularization term and providing some additional mathematical proofs—without empirical backing (such as demonstrating the specific conditions under which gradient divergence occurs). I strongly suspect that AI was used to generate much of the verbose mathematical proof. See the Weaknesses.

Reviewer_EiSTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes advantageflow, which minimizes an advantage-weighted prediction loss and achieves superior performance mainly on SD3.5 Medium.

Critical strengths
  1. The paper proposes AdvantageFlow, a least-squares algorithm for reinforcement learning in flow models using the forward process. It derives the advantage-weighted term from a linear tilt of the reward, uses the rollout prediction to reduce variance, and shows that DiffusionNFT is a special case with the coefficient γNFT=β(β−A)\gamma_{NFT} =\beta(\beta-A).
  2. Experiments show that, on SD3.5 Medium and with only the loss changed, AdvantageFlow surpasses other methods on PickScore, HPSv2.1, OCR, and the sum of multiple objectives, using less GPU time.
Critical weaknesses
  1. I think the largest experimental gap is that GenEval is not used as a training reward. Table 1 only evaluates models trained on PickScore, HPSv2.1, and CLIPScore, whereas DiffusionNFT’s main results include directly optimizing GenEval, improving it from about 0.24 to 0.98, and also giving the training curve. Preference rewards and compositional generation rewards measure different capabilities, and OCR cannot replace GenEval either. Therefore, training with GenEval as the reward, as well as the curve of the overall score against GPU time, needs to be added.
  2. The abstract says that the method is validated on FLUX, but there is only one task. I think this is far from enough. Compared with related papers, experiments should be added to show that it also works on FLUX.
  3. I think the methodological ablations are insufficient and do not fully demonstrate the effectiveness of each component. Figure 4 only compares adaptive and fixed regularization, which changes the regularization strength rather than the loss structure. The claim that replacing x0x_0 with the rollout prediction foldf_{\mathrm{old}} reduces variance is proved only when foldf_{\mathrm{old}} equals the conditional mean, and it is not verified experimentally.
Questions for authors
  1. Can you provide training curves with GenEval itself as the reward, plotting the overall score against GPU time, to verify the effectiveness of AdvantageFlow?
  2. Do you have results on other FLUX tasks, such as OCR or GenEval?
  3. Can you show an ablation of replacing x0x_0 with foldf_{\mathrm{old}} , or a measurement of gradient variance, to test the claim that the rollout prediction reduces variance?
Reviewer_RVETOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

AdvantageFlow is a forward-process reinforcement learning method for rectified flow models. It combines an advantage-weighted prediction loss with rollout-policy and reference-model regularization. The method introduces a prompt-dependent rollout regularization coefficient and motivates the objective through a linearly tilted target distribution and a conditional variance-reduction argument. Experiments on SD3.5 Medium and FLUX.1 report improved reward optimization and computational efficiency relative to DiffusionNFT.

Critical strengths
  1. AdvantageFlow revisits the rollout regularization coefficient in an existing advantage-weighted quadratic objective. Its interpretation through a linearly tilted target distribution and conditional variance reduction helps clarify the relationship between reward-weighted regression and rollout-policy regularization.

  2. The reported experiments show better reward optimization across the evaluated tasks, including substantial reductions in the computation required to reach comparable reward levels.

Critical weaknesses

Limited novelty relative to DiffusionNFT. Appendix D establishes that the simplified DiffusionNFT objective can already be expressed as an advantage-weighted prediction loss with rollout-policy regularization. After removing an overall positive scaling factor, its rollout coefficient is β−A\beta-A, whereas AdvantageFlow uses a prompt-dependent coefficient γ(c)\gamma(c). Thus, the principal objective-level difference is the regularization schedule. Reference-model L2 regularization is standard in RL fine-tuning and is also present in the DiffusionNFT implementation; it does not establish novelty.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.