acceptodds
Withdrawn from ICLR 2027

OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post training via GRPO has demonstrated remarkable effectiveness in improving the generation quality of flow-matching models. However, GRPO suffers from inherently low sample efficiency due to its on-policy training paradigm. To address this limitation, we present OP-GRPO, the first Off-Policy GRPO framework tailored for flow-matching models. First, we maintain a replay buffer that retains high-quality trajectories and reuses them by mixing a small fraction of off-policy samples into each rollout batch. Second, to mitigate the distribution shift introduced by off-policy samples, we propose a sequence-level importance sampling correction that keeps GRPO's clipping anchored to the current policy, reducing the clipped off-policy samples while ensuring stable policy updates. Third, we theoretically and empirically show that late denoising steps yield ill-conditioned off-policy ratios, and mitigate this by truncating trajectories at late steps. Across image and video generation benchmarks, OP-GRPO achieves comparable or superior performance to Flow-GRPO with only 34.2% of the training steps on average, yielding substantial gains in training efficiency while maintaining quality.

Primary area:
generative models
Keywords:
Diffusion RL, Flow GRPO, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Sb4yOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper targets the cost of GRPO post-training for flow-matching models: rollouts are discarded after every policy iteration, so sampling is never amortized, and on hard prompts where all samples fail the group advantages collapse. OP-GRPO adds a replay buffer that keeps the highest-reward trajectory per prompt and decays stored rewards each epoch to keep entries fresh; at rollout a fraction of prompt groups are seeded from it, with G−1 trajectories sampled fresh and one replayed. The resulting mismatch is handled by keeping a GSPO-style geometric-mean ratio inside the clipping term and moving the distribution correction into a separate unnormalized sequence-level factor outside it. Replayed latents are reused only down to a truncation step, with the low-noise tail regenerated by the current policy.

Critical strengths

The replay mechanism addresses a real and under-treated failure mode in GRPO: groups in which every sample fails, reward variance is zero, and the advantage collapses to zero.

Critical weaknesses

1.My main concern is that all three contributions sit close to existing work, and the paper is aware of this in several places.

Reward-filtered replay with a distribution correction is well covered on the LLM side. RePO, ReMix and BAPO are all cited in Sec. 2.2, and between them they already do buffer-based advantage estimation, KL-constrained reuse of stale data, and selective reuse of high-value samples. The buffer here — keep the group argmax, one entry per prompt, decay the stored reward each epoch — reads as a sensible implementation of that recipe rather than something new. The sequence ratio is GSPO's geometric mean, and B.1 more or less says so when it lists the method alongside PPO/GRPO/GSPO. The 1/σ² blow-up at low noise is not a new finding either; B.3 credits earlier on-policy work for it and adds only that the drift gap is larger when the sample is stale.

What I would count as the authors' own is the separation of the clipping ratio from the correction factor, so that clipping still measures movement away from p_old, and the decision to truncate during rollout, which has the nice consequence that the retained product remains an exact ratio. Both are correct, and the second is cleaner than it first appears. But they are refinements.

2.The claim in Sec. 5.2 that OP-GRPO reaches Flow-GRPO's final performance in only 34.2% of the steps is read off the training curves in Fig. 3, and 2 things make that reading unsafe.

First, the two methods are not training on the same distribution. By Alg. 1, buffer entries are admitted by reward argmax, so the prompts held in the buffer are selected for being ones on which the model already scores well, and a fraction α of OP-GRPO's groups are seeded from that subset — including the G−1 freshly sampled trajectories in those groups. A gap in mean training reward therefore mixes prompt enrichment with policy improvement, and the paper neither discusses nor bounds the former.

Second, "the number of steps needed to reach Flow-GRPO's final performance" is well defined only if the baseline has converged. In Fig. 3(c) the Flow-GRPO curve is still clearly rising at 2,200 steps, so its "final performance" is simply where the run was stopped, and the reported ratio moves with that choice.

Questions for authors

Both are clarifications, not suggestions for future work.

First, diversity. The buffer keeps the group argmax and replays it with a large positive advantage, and truncation reuses the high-noise part of the trajectory, the part that fixes layout. Both push toward one realisation per prompt, and none of the reported metrics would catch that — they are all per-sample quality scores, and a model collapsing to a single good image per prompt would score higher on all of them. Could you report a diversity measure against Flow-GRPO at matched task metric, e.g. pairwise perceptual distance across samples for the same prompt?

Second, the training curves. The 34.2% figure comes from Fig. 3, but buffer entries are admitted by reward argmax, so buffer prompts are ones the model already does well on and a share of OP-GRPO's groups is drawn from them; the gap in mean training reward then mixes prompt enrichment with real improvement. It also depends on where the baseline stops, and Flow-GRPO is still rising at 2,200 steps in Fig. 3(c). The step counts disagree too: 1,150/1,600 in Sec. 5.2, past 2,000 in Fig. 3(c), 1,500/2,200+ in App. C. Appendix C says the test set is evaluated every 50 steps — could you plot test metric against step and recompute, including the 30.1% for video?

Two smaller points on the same figures. Fig. 6(b) has an unlabelled "OP-GRPO" curve the text never defines; Sec. 5.5 says 25% is used throughout, yet OP-GRPO-25 is a separate and the weakest curve there, and Table 3 gives 0.125. What does the main setting use, and does "proportion" mean samples or prompt groups? And in Fig. 9 the curves start apart, 0.805 against 0.790, about 40% of the final gap, though the buffer is empty at step 0 and both should start from the same checkpoint.

Reviewer_rRqtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OP-GRPO to address the sample efficiency of GRPO post-training for flow-matching models. The method has three main components: a replay buffer that selects trajectories by quality and refreshes them over time; a mixed rollout scheme that combines one historical trajectory with G−1 newly generated trajectories in a subset of prompt groups; and an objective that separates off-policy importance weights from a sequence-level clipping ratio, together with truncation of the low-noise denoising steps for stability. The authors report improvements on SD3.5-M image generation, Wan2.1-1.4B video text rendering, and FLUX.1-dev preference alignment. The central claim is that OP-GRPO matches or exceeds Flow-GRPO's final performance using only 34.2% of its training steps on average.

Critical strengths
  1. The problem is well defined and the engineering motivation is clear. Improving the efficiency of on-policy sampling for expensive generative models is a worthwhile research direction.
  2. The method includes designs motivated by the properties of flow matching. Separating the update constraint from off-policy correction, and analyzing the sensitivity of low-noise transitions to policy mismatch, are both well motivated. Even if the current theoretical guarantees require revision, these designs may still constitute an effective empirical method.
Critical weaknesses

W1. The proof that off-policy reuse introduces no additional bias does not cover the actual algorithm. Equations (11)–(12) in Appendix B.1 state a valid, standard change-of-measure identity. However, this identity requires samples to actually follow the distribution represented by the denominator, and the integrand to be defined consistently with the target objective. The actual algorithm introduces at least two additional factors:

  • Section 4.1 and Alg 1 select the highest-reward trajectory within each group, replace entries across samples, apply age-based decay, and enforce prompt uniqueness. The distribution of trajectories admitted to and subsequently sampled from the buffer is generally no longer their original generating policy $P_{\mathrm{off}}$. Using only $P_{\mathrm{old}}/P_{\mathrm{off}}$ does not correct for this selection process.
  • The advantage in Eq. (3) depends on the reward mean and standard deviation of the entire mixed group. The actual surrogate is therefore $F_i(\tau_1,\ldots,\tau_G;\theta)$, rather than a function $F(\tau_i,\theta)$ of a single trajectory. The off-policy trajectory also changes the advantages of newly generated trajectories in the same group. These concerns address the applicability of the proof, rather than making a generic criticism that PPO clipping itself is biased; the authors already distinguish these two sources of bias in B.1. Please specify the actual proposal distribution, target prompt distribution, joint group distribution, and advantage estimator, and provide a derivation that applies to them. Alternatively, narrow the unbiasedness claim to a restricted setting and explicitly discuss the bias–stability trade-off of the implemented method.

W2. The evidence for computational efficiency and causal attribution remains insufficient. The reported average of 34.2% of the training steps is important, but it does not automatically imply a proportional reduction in GPU-hours, wall-clock time, or network computation. The supplementary pipeline still executes a transformer forward pass on the entire batch at each denoising step, and replayed trajectories require rescoring. The sequence-level objective also changes the backward-pass and activation-storage scheme. Please report, for each task, the actual time, GPU-hours, rollout count, forward/backward evaluation counts, and peak memory required to reach a common held-out performance threshold, and how the average ratio is calculated.

The current comparison changes replay, the stepwise versus sequence-level objective, and truncation simultaneously. A useful control would use the same sequence-level objective, normalization, and retained timesteps as OP-GRPO but no replay, together with a comparison between random replay and high-quality replay. The ablations in Figure 6 do not yet isolate the contribution of each component, and their task and ratio labels are themselves ambiguous. Multiple random seeds and uncertainty intervals are particularly important because the improvements in the best GenEval, OCR, and mean CompBench scores are only 0.01, 0.01, and 0.006, respectively.

W4. Some conclusions are stronger than the tabulated evidence, and key experimental settings need to be reconciled. Section 5.3 claims that intermediate checkpoints improve all cross-task metrics. However, for the GenEval model at 1020 steps, Table 1 shows Aesthetic, DeQA, and ImageReward decreasing from 5.27/4.05/1.37 to 5.22/4.02/1.21. The evidence therefore supports faster improvement on certain task metrics, but not an unconditional claim of no quality trade-off. In addition, Table 3 specifies a hybrid ratio of 0.125, whereas Section 5.5 states that 25% is always used; the training budgets in Section 5.2 and Appendix C differ; and the vertical axis in Figure 6(a) is OCR, although the text states that all ablations use GenEval. Please reconcile the configurations and the sources of the curves. The FLUX appendix primarily presents training rewards and qualitative examples; held-out metrics are needed to support claims of generalization on the larger backbone.

Questions for authors
  1. What exactly is the replay proposal distribution after reward-based selection? How does Eq. (11) account for this selection distribution and group-dependent advantages? If the claim is only a conditional, single-trajectory result, please specify what is held fixed and whether these conditions hold in the actual training procedure.
  2. Is log_prob summed or averaged over latent dimensions in the experiments? Please give the complete normalization used in both $s$ and $w$, and explain its relationship to B.1.
  3. Did the runs producing FLUX Figures 8–9 regenerate the tail? Where exactly does the sampling policy switch? Are 4, 5, or some other number of steps retained?
  4. Please provide per-task common performance thresholds, the steps and wall-clock time/GPU-hours required to reach them, at least one same-objective/no-replay control, and variability across seeds.
Reviewer_6ihAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the efficiency problem in Flow-GRPO. Authors propose OP-GRPO, with off-policy GRPO that integrates replay-based sampling, sequence-level distribution correction, and truncated denoising. The main ides is resuing high-reward trajectories from a replay buffer. Experiements on SD 3.5 and WAN2.1 show faster training covergence over FLow-GRPO.

Critical strengths
  1. The motivation - reusing good trajectories - is reasonable. Flow-GRPO uses many generated samples, in its training, many samples has limited value. especially for difficult prompts, the successful samples are rare.
  2. Results on image and video generation is clear. OP-GRPO accelerates convergence while preserving generation quality competitive with fully on-policy baselines.
Critical weaknesses
  1. The comparison is not sufficient. The paper only compares over Flow-GRPO. recent work, sush as mixgrpo, flashgrpo, brnchgrpo also improve the training efficiency of Flow-GRPO. however, the paper does not compre the proposed mehtod against those methods.
  2. The experiments on vidoe diffusion only study text rendering, which is limited. Suggest to also evalute motion quality and temporal consistency for videos. Morevoer, quantative results on video has not been provided.
  3. Paper claims the off-policy correction is unbiased (Eq. 9). However, the buffer only keeps selected high-reward trajectories, which do not follow the original policy distriubtion. It seems that the proposed importance ratio does not consider the selection.
Questions for authors
  1. Are the results sensitive to the truncation strategy? the paper does not study it. Besides, does removing late denoising steps affect image details or video quality?
  2. The off-policy ratio is inconsistency. Section 5.5 says 25% is used throughout, but Table 3 reports 0.125. which value is used for the main experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.