acceptodds
Under review as a conference paper at ICLR 2027

Stable Signed Flow Reinforcement

27 views

Abstract

Reward-based finetuning of flow models requires promoting high-reward samples and suppressing low-reward ones while maintaining training stability. Reward-weighted regression provides a simple starting point, but nonnegative weights give no explicit unfitting signal, and repeated reweighting can concentrate the distribution and cause collapse. We propose Stable Signed Flow Reinforcement (SIFR), which uses signed advantages to explicitly reinforce high-reward samples and unfit low-reward ones, while constraining velocity drift from a reference model to stabilize the updates. Alternating updates of the model and a Lagrange multiplier adapt the regularization strength to a prescribed drift budget. Experiments show that SIFR achieves higher rewards than existing methods in fewer training rounds on SD3.5 Medium (2.5B), FLUX.2 Klein (9B), and Cosmos 3 Super Text2Image (64B), using preference rewards as well as rule-based rewards. These gains further transfer to held-out rewards and prompt sets. Ablations show that signed advantages and the drift constraint each contribute to stable training, and that together they make flow RL faster and raise its reward ceiling.

Primary area:
generative models
Keywords:
Rectified Flow, Alignment, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MQcmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies reward-based fine-tuning of flow models using signed advantages. The authors explain that reward weighted regression suffers from distributional collapse, and does not provide a signal to suppress low-reward samples, and how signed advantages can overcome the last issue but can make the objective unbounded. They provide a unified perspective of previous methods under this formulation, and they propose Stable Signed Flow Reinforcement (SIFR) to stabilize signed advantage regression. SIFR is formulated as a constrained optimization problem, where the constraint limits the velocity change relative to a reference model and the regularization multiplier is adapted from the measured change. SIFR is tested across three model backbones, several reward settings, and evaluation settings. The experiments show a consistent improvement over DiffusionNFT and Reinforce Adjoint Matching.

Critical strengths
  1. Theoretical unification and proposed improvement: Table 1 provides an insightful theoretical unification of several different reward fine-tuning methods, and can be used to develop further insights. Beyond this, the authors tackle a known problem of signed-advantage regression, and the solution is neat.

  2. Broad empirical validation: They test three different model backbones, with both model-based and rule-based rewards, evaluating them on the training reward objectives, held-out rewards, and held-out prompts. The results are extensive and positive. The ablations help clarify the contributions of the methodological choices, for example multiplier adaptation through feedback control.

Critical weaknesses
  1. Clarity of the contribution: The presentation of the first three sections makes it difficult to distinguish the problem setting, motivation, background, and the specific contribution. I would expect a clearer introduction to the problem and the problem setting, and the background to be separated from the contribution. The authors should refer to previous works whenever the stated properties and statements have been previously discussed and developed. Additionally I would expect a more detailed comparison with AdvantageFlow and Adaptive Divergence Regularized Policy Optimization (ADRPO), and how SIRF improves upon them. This should be accompanied by an experimental validation.

  2. Experimental settings and presentation: The multi-reward settings are unclear. If rewards are combined as is, the variance of PickScore could dominate the other two rewards, potentially being very close to the single-reward setting. Additionally, I would expect the speed comparison to use GPU hours as the x-axis, as the number of gradient updates does not directly quantify the computational cost. Moreover, a fair comparison would be between methods that achieve the same level of reward, not different levels.

Questions for authors
  1. Contribution: Can you better clarify throughout the manuscript which is your contribution, and separate it to the established background?

  2. Comparison to additional methods: Could you clarify the relationship to ADRPO and provide empirical comparison between SIRF, AdvantageFlow, and ADRPO?

  3. Multi-reward settings: Are the 3 rewards (PickScore, ClipScore, and HPSv2.1) scaled or normalized before being combined? What is the scale for each single reward at training time?

  4. Computational efficiency: Could you report the computational budget required for each method to reach a specific reward level, alongside the training rounds?

Reviewer_PzcZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Stable Signed Flow Reinforcement (SIFR) for reward-based fine-tuning of rectified-flow models. SIFR uses signed advantages to promote above-average samples and suppress below-average ones, while adaptively controlling velocity drift through a constrained objective and Lagrange multiplier. The paper relates existing flow-RL methods to regularized signed regression and evaluates SIFR on three models ranging from 2.5B to 64B parameters. Reported results show faster reward improvement, higher final scores, and transfer to held-out rewards, prompt distributions, and rule-based objectives.

Critical strengths
  • The signed-curvature analysis clearly explains why negative advantages can destabilize flow regression.
  • Experiments span three model scales, four objectives, held-out rewards, and prompt-distribution transfer.
Critical weaknesses
  • The signed-flow derivation is internally inconsistent. Equation (4) centers advantages, implying E[A]=0\mathbb{E}[A]=0, while Section 2 assumes ZA=E[A]>0Z_A=\mathbb{E}[A]>0 to normalize a target density. Moreover, the saddle-point analysis concerns unrestricted population fields and does not establish stability for neural SGD using the heuristic multiplier controller.

  • Experimental breadth is strong, but evidence comes from single training runs without uncertainty or human evaluation. “2–4× faster” measures rounds rather than computation, while samplers and selected checkpoints differ across baselines. The supplied PDF also omits cited appendices and several tables and figures containing essential derivations and results.

Questions for authors
  • How can ZA=E[A(X1)]>0Z_A=\mathbb{E}[A(X_1)]>0 hold when Equation (4) defines a centered advantage with E[A]=0\mathbb{E}[A]=0? If the intended normalizer is E[A1Ωreach]\mathbb{E}[A\mathbf{1}_{\Omega_{\mathrm{reach}}}], what target distribution and RL objective does it represent?

  • What assumptions guarantee strong duality, existence of a finite saddle point, and m(x,t)+λ>0m(x,t)+\lambda>0 almost everywhere? Do these properties continue to hold under neural parameterization and finite minibatch updates?

  • What theoretical relationship connects the clipped log-space update in Equation (9) to dual gradient ascent? How frequently and by how much does the empirical drift violate its prescribed budget?

  • Can the authors report multiple independent training runs with uncertainty, along with GPU-hours, generated samples, NFEs, optimizer updates, and wall-clock time? Do the efficiency conclusions remain under matched samplers, checkpoint rules, and baseline-tuning budgets?

  • Can the authors provide a same-setting 2×22\times2 ablation of signed versus nonnegative advantages and adaptive versus fixed/no drift constraints, together with blinded human evaluation of quality, faithfulness, artifacts, and diversity?

Reviewer_W9sqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Stable Signed Flow Reinforcement (SIFR) for online reward finetuning of rectified-flow models. It interprets existing flow-RL objectives as signed advantage-weighted velocity regression with implicit reference regularization, then makes the stability mechanism explicit: optimize the signed objective under a velocity-drift budget and adapt the corresponding Lagrange multiplier. Across SD3.5 Medium, FLUX.2 Klein, and Cosmos 3 Super Text2Image, the paper reports better reward metrics, faster attainment of baseline-relative targets, and transfer to held-out prompts/rewards than DiffusionNFT and RAM.

Critical strengths

S1: The signed-regression view usefully exposes a mechanism that is often obscured by nonnegative-looking implementations: successful online flow RL needs both an explicit way to suppress low-return samples and a constraint preventing destructive velocity drift.

S2: The constrained formulation is clean and practical. Updating a dual multiplier is a principled alternative to holding a reference penalty fixed, and the paper provides a helpful unification table for existing methods.

S3: The scale of the experiments is a real strength. Evaluation spans 2.5B, 9B, and 64B backbones, multiple preference and rule-based objectives, held-out reward models, and multiple prompt suites. The FLUX collapse observed for RAM is also a useful stress case.

Critical weaknesses

W1: The ablation of signed advantages is not a sufficient baseline for the central stability claim. It compares SIFR with a nonnegative version that removes the explicit unlearning signal and collapses, but does not compare equally tuned alternatives such as advantage clipping, bounded/robust signed regression, trust-region/KL constraints, or a broad fixed-penalty sweep. Thus it remains unclear whether the gain is specific to the proposed primal-dual formulation.

W2: The method replaces a fixed regularization coefficient with a drift budget deltadelta, but the key new hyperparameter is not characterized. No sweep reports the reward-versus-drift/diversity frontier, feasibility behavior, or sensitivity of SIFR to deltadelta across backbones and objectives. An adaptive dual variable does not itself establish robustness to the chosen constraint.

W3: The headline evidence largely consists of reward-model scores, including several models closely related to the optimized preference objectives. Held-out reward models are helpful but are not independent human evidence, and no reward-hacking/quality failure analysis or human comparison is provided. This is especially important when increasing reward is framed as a higher “ceiling.”

Questions for authors

Q1: How does SIFR compare with a strong sweep of fixed reference penalties, clipped or bounded signed losses, and a conventional KL/trust-region update under exactly matched rollout and compute budgets?

Q2: Please provide deltadelta sweeps and reward/diversity/drift Pareto curves for representative backbone-objective pairs. How transferable is a budget selected on one task?

Q3: Do human raters, or an evaluator deliberately held out from both reward training and method selection, prefer SIFR samples? Are there qualitative examples where the reward increases but fidelity, prompt adherence, or diversity degrades?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.