acceptodds
Withdrawn from ICLR 2027

STEP: Post-Training Few-Step Flow Models by Propagating Reward Gradients Through Every Generation Step

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study reward fine-tuning of few-step Gaussian-mixture (GM) flow policies. Each network evaluation defines a velocity field over a sampling interval, allowing analytic estimates of the clean latent at intermediate states. We introduce STEP, which uses these estimates as probes for constructing surrogate reward gradients. One probe is sampled within each deployment interval, and a straight-through chain retains dependencies between intervals. Weighted fusion connects all probes to a shared terminal reward while preserving the sampled image in the forward pass. The probes require no additional transformer evaluations. The full training objective combines this terminal loss with auxiliary rewards and frozen-teacher velocity regularization. After optimization steps on FLUX.1-dev, STEP achieves GenEval, OCR, and held-out HPSv3 with four neural function evaluations. The official distilled student scores , , and , respectively. Within the full objective, removing the chain or replacing interior probes with zero-depth estimates reduces OCR and GenEval. These results support using the interval structure of GM policies for reward fine-tuning at a fixed four-step deployment budget.

Primary area:
generative models
Keywords:
image generation, RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RUzcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STEP, a reward fine-tuning method designed for few-step Gaussian-mixture flow models. The authors address the challenge of reward optimization in highly compressed steps by caching the deployment trajectory and introducing analytic clean-latent probes. A key technical contribution is the combination of a straight-through chain and distance-weighted fusion. This design elegantly decouples the forward and backward passes, maintaining the original sampling trajectory for inference while effectively propagating reward gradients. The method is further stabilized via intermediate rewards and frozen-teacher velocity regularization. The paper evaluates STEP on FLUX.1-dev with 4 NFEs against strong baselines, supported by thorough and convincing ablation studies.

Critical strengths

STEP reuses analytic posterior estimates within sampling intervals to distribute a shared terminal reward over multiple steps, and the probes do not require extra evaluations. This is a practical way to adapt straight-through gradients to few-step GM policies, and it clearly separates forward values from backward paths.

Critical weaknesses
  1. Empirical gains are modest and scope is narrow. While STEP improves GenEval and OCR by 0.03 over adapted Flow-GRPO, it still trails LeapAlign on GenEval (0.530 vs. 0.550). The evaluation is limited to a single backbone and four NFEs. More concerning is the missing baseline: we see results for the distilled π-Flow reference, but not for STEP's own base GM initialization before reward training. This leaves the actual improvement over the starting policy unclear.

  2. The surrogate gradient and fusion weights lack sufficient validation. The coefficient bound doesn't establish gradient alignment or guarantee improvement of the terminal objective. Comparisons with sampler gradients and weight-sensitivity analysis are lacked.

Questions for authors
  1. Appendix C.1 separates the relative fusion weights from the overall scale γ. What evidence supports inverse-distance weighting over uniform fusion with the same total coefficient (c_i = γ/N)? Does the sample-dependent γ offer benefits beyond a fixed overall scale?

  2. Table 1 includes DRaFT, but its objective differs from STEP's. With auxiliary losses and teacher regularization matched, how does STEP compare with backpropagation through the actual sampler? Could you compare their update directions and resulting changes in terminal reward, including cases where the surrogate update is detrimental?

Reviewer_pVHxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reward-based fine-tuning + pi-flow few-step distillation of flow-based models, starting from an undistilled conversion of a Flux model. The proposed STEP method first performs 4-step no-grad rollout, then recompute the heads (with grad) and attenuates the gradient with ρ=0.3\rho = 0.3 following LeapAlign. In each interval, it also draws a probe (clean latent prediction at a random intermediate step) and fuses them with trajectory-distance weighting cic _ i. The final loss has three parts: terminal term, auxiliary supervision term, and teacher anchoring term. The paper compares with multiple RL and/or distillation baselines and shows comparable or improved few-step performance

Critical strengths
  • Improving few-step generation performance of diffusion/flow-based models with external human-preference reward functions is an important and very timely topic
  • Doing joint RL and distillation simultaneously is a promising direction. As is shown in section 4.3, RL-then-distillation and distill-then-RL both have some shortcomings and may hurt performance
  • The improvement for few-step performance is not limited to the training reward (section 4.5), showing that the method might be less prone to reward hacking (though no diversity metric is reported directly for the generated samples)
Critical weaknesses
  1. I feel the paper's novelty in methodology is limited compared to LeapAlign, if I understand correctly. The paper inherits many components from them, such as gradient attenuation, hinge reward loss, and trajectory-distance weighting, with minor modifications. The major new design seems to be the adaptation of the framework to GM policies, and to do so, the paper introduces random-depth probes and probe fusion weights
  2. The proposed method introduces many design choices and hyperparameters (e.g., in equations (7, 8, 10)), but ablation studies are not enough to justify the robustness of the method with respect to them. High-level intuition for these choices would also be helpful
  3. As the authors acknowledge, the method is only validated on one model (FLUX.1-dev); also only one training reward (HPSv3) is used. Whether the method is generalizable to other models/rewards/inference budgets remains unknown
  4. The proposed method requires a differentiable reward, which is not always available in practice, while existing diffusion RL methods like FlowGRPO and DiffusionNFT only rely on the reward values
  5. The paper writing can be improved in many places:
  • Section 3.1 assumes the reader is familiar with GM policy enough, which makes me difficult to follow when first reading the paper. I think it's better to provide a high-level picture of what GM policy is doing. Also, I'm curious why the authors didn't choose the simpler DX parameterization in the pi-flow paper?
  • The two-argument integrator Fθ,i(t,a)F _ {\theta, i}(t, a) is not clearly explained. If I understand correctly, the transformer input and initial state have the same value and differ only in their gradient?
  • The LeapAlign+substep adaptation is not explained clearly
Questions for authors
  1. Have the authors also considered flexible NFEs for both training and inference? Ideally this can be done to trade off quality and speed
  2. STEP needs a gradient-free rollout and then a gradient-enabled recomputation of the heads. I think this is inherited from LeapAlign (whose rollout involves 25 steps, and for training, needs to randomly select two steps on the trajectory to construct a leap), but STEP's rollout is only 4 steps and no skipping is involved. I'm wondering if two passes is still needed for other reasons?
  3. (Minor issue) The paper seems to have imported some LaTeX packages that change the font of the math formulas, which is not standard from the submission template
Reviewer_38PDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper is targeting an important computational limitation of trying to use gradient-based RL generally w/ flow models, i.e. circumventing backprop through the rollout without resorting to simple scalar reward algos. It relies on this gaussian mixture flow approach from 2504.05304, and learns the few step map for the clean latents from the various posteriors to provide an approximation to the reward gradient, e.g. a "surrogate reward gradient." It ultimately targets the same objective as FlowGRPO but with a different gradient estimator (bias variance tradeoff). The size of the bias depends on the accuracy of the shortcut approximation used in the few-step surrogate reward estimator. This sort of checks out given the similarity in its performance to FlowGRPO.

They benchmark against FlowGRPO, LeapAlign, and variants, and show that STEP can have a modest improvement over existing baselines

Critical strengths

The paper targets one of the main limitations of differentiable reward alignment for flow models, in that it avoids backprop through simulation using the STEP surrogate gradient estimator. This allows it to access lower variance estimators, though it is not clear that this is a compute-matched comparison with the other methods.

Critical weaknesses

I see two main weaknesses with the approach

  1. Fundamentally, it is very very similar to LeapAlign. As I understand it, the straight-through chain is also used in that paper, as well as fusion distance weights, so it seems like the new part is limited to the gaussian mixture? Are LeapAlign and STEP given the same requisite auxiliary loss terms? They seem important.

  2. Given that it targets the same objective as FlowGRPO, and only modestly outperforms it but requires a differentiable reward, it seems that the applicability of the method is strictly less than FlowGRPO but for only modest gains. In particular, I also don't think the comparison is fairly compute matched. Is the extra gradient costs in the backward pass accounted for in the cost comparison with Flow GRPO?

Questions for authors

See above, with regards to questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.