acceptodds
Withdrawn from ICLR 2027

Optimizing Stochastic Policy Geometry for Few-Step Flow Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-step flow models generate samples through a handful of large denoising transitions. Applying GRPO to such models requires introducing stochasticity into the denoising transitions, and when a single transition spans a wide range of noise levels, the choice of the resulting stochastic policy becomes a major factor in optimization. Recent constructions constrain the policy to preserve the target flow coefficients, but this constraint admits a family of policies and offers no criterion for choosing among them. We introduce Geo-FSRL (Geometry-Optimized Few-Step Reinforcement Learning), built on a selection principle: at a fixed conditioning state and timestep pair, choose the admissible policy that minimizes the conditional KL divergence induced by any given nonzero velocity-output update. Within a coefficient-preserving family of isotropic Gaussian transitions, this criterion reduces to minimizing a single scalar geometry factor that determines the conditional KL for every velocity-output update. The same factor governs velocity-space Fisher information and conditional transition log-likelihood-ratio variance. We derive the unique closed-form optimizer within this family, shared across all nonzero update magnitudes and directions, with transition coefficients determined by both the source and target noise levels. Because the minimized geometry factor still varies across timesteps, we further adopt gradient-scale matching from prior work and derive closed-form policy-gradient and KL weights that normalize explicit score scales and reference-KL curvature. Geo-FSRL trains on the original few-step schedule and retains the deterministic solver at inference. On four-step Qwen-Image and Wan2.1 models, it reaches higher final reward scores than Flow-GRPO and Flow-CPS.

Primary area:
generative models
Keywords:
Diffusion, Reinforcement Learning, Few Step

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K8SjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduced a reinforcement learning framework Geo-FSRL for few-step generation. The idea of the proposed method is to minimize the conditional KL divergence (Eq. 8) under the restrictions in Eq. 6. The experimental results demonstrate the effectiveness of the proposed method compared to Flow-GRPO and Flow-CPS under the same budget of sampling steps.

Critical strengths
  1. The proposed methods achieve the best performance on both image and video generation tasks, compared to Flow-GRPO and Flow-CPS.
  2. The policy reweighting ablation shows the effectiveness of the policy reweighting strategy.
Critical weaknesses
  1. The paper seems to be hard to follow (at least for me); e.g., I did not know what the velocity-output update, coefficient-preserving family and scalar geometry are (in the abstract) when I first read the abstract of the paper. Additionally, Section 3.4 is hard to understand and would benefit from a more intuitive explanation of the main steps.
  2. To compare with the baseline Flow-GRPO, I think it is necessary to use the standard datasets used in Flow-GRPO, such as PickScore, GenEval and text rendering evaluated by OCR (these datasets can be found in the official GitHub repo of Flow-GRPO), which are also commonly used in recent RL papers like "Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models" and "On-Policy Self-Distillation in Diffusion Models".
Questions for authors
  1. I am curious about the performance of the proposed methods under the same time budgets. Some rewards vs. training time figures might show that the proposed methods can achieve higher rewards than Flow-GRPO under the same time budget. Could the authors report the training time for different RL methods?
  2. For generalization, could the authors present more generalization experiments in the paper? For example, you can train models using Flow-GRPO and your proposed frameworks on PickScore, and evaluate the models' performance on other rewards, similar to the experiments in Flow-GRPO.
  3. Some terms appear for the first time without sufficient discussion or explanation. Could you add more explanation and discussion for these terms (see Weakness 1)? A more intuitive explanation of Section 3.4 would also be helpful.
Reviewer_hmcCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Geo-FSRL for reinforcement learning of few-step flow models. It derives transition coefficients that minimize the conditional KL divergence induced by a fixed velocity-output update within a coefficient-preserving Gaussian policy family. The selected transition is combined with policy and KL loss reweighting. Experiments on four-step image and video generation report higher final rewards than Flow-GRPO and Flow-CPS.

Critical strengths
  1. The conditional KL minimization criterion yields a unique closed-form coefficient setting within the specified policy family, providing a principled alternative to empirical selection of stochasticity parameters.
  2. Geo-FSRL achieves higher final reward scores than the evaluated Flow-GRPO and Flow-CPS configurations on both image and video generation tasks.
Critical weaknesses
  1. Limited novelty of the sampling method. As the authors acknowledge, the derived optimal transition is equivalent to the existing first-order SDE-DPM-Solver++. Thus, the paper does not introduce a new sampling rule; its methodological novelty primarily lies in the conditional KL optimality characterization and its integration with reweighting for few-step RL.

  2. Insufficiently justified DiffusionNFT comparison. In Figure 1, the tested DiffusionNFT adaptation exhibits early training degradation across the reported settings, but the paper does not explain its cause or provide sufficient adaptation and tuning details. The results do not rule out inadequate hyperparameter tuning or adaptation issues. Consequently, the fairness of this comparison remains unclear; the authors should provide diagnostic experiments and demonstrate that DiffusionNFT receives an adequate tuning budget.

Questions for authors

Why minimizing the conditional KL divergence is the best way for RL?

Reviewer_TBKxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Geo-FSRL, a stochastic-policy design for GRPO training of few-step flow models. Within a coefficient-preserving isotropic Gaussian family, it minimizes the conditional KL induced by a fixed velocity-output update and derives a unique closed-form transition equivalent to the posterior-variance/SDE-DPM-Solver++ rule. It then rescales policy-gradient and reference-KL terms using the resulting geometry factor. Experiments on four-step image and video models report improved reward-model scores over Flow-GRPO and Flow-CPS, with an ablation and small blinded image preference study.

Critical strengths
  • The exact finite-update KL characterization yields a unique, interpretable transition rule without empirical stochasticity tuning.
  • Consistent image and video gains suggest practical value across modalities without changing the deterministic inference schedule.
Critical weaknesses
  • The optimality result assumes a fixed state, candidate-independent velocity update, and restricted Gaussian family. Actual GRPO updates depend on sampled states, rewards, advantages, clipping, Jacobians, and optimizer dynamics. Because the method does not measure realized KL, likelihood-ratio variance, clipping, or gradient dispersion, the claimed mechanism remains unvalidated.

  • The empirical evidence relies on one seed and training reward models as evaluation metrics. Image gains are small, while the 20-pair human study lacks confidence intervals. For video, Flow-CPS uses ϕ=0.3\phi=0.3 specifically for greater geometric separation rather than reported tuning, making the largest claimed improvements difficult to interpret.

Questions for authors
  • Why was video Flow-CPS evaluated at ϕ=0.3\phi=0.3 rather than an empirically tuned value? Please report an equally budgeted sweep, including ϕ=0.7\phi=0.7, and multi-seed results with uncertainty estimates.

  • Can the authors independently vary transition coefficients, policy weighting, and KL weighting while reporting realized KL, likelihood-ratio variance, clipping frequency, and cross-timestep gradient dispersion?

  • Can the authors provide evaluation on an untouched prompt set using metrics not optimized during training, together with a statistically powered human evaluation, particularly for video?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.