- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper targets the cost of GRPO post-training for flow-matching models: rollouts are discarded after every policy iteration, so sampling is never amortized, and on hard prompts where all samples fail the group advantages collapse. OP-GRPO adds a replay buffer that keeps the highest-reward trajectory per prompt and decays stored rewards each epoch to keep entries fresh; at rollout a fraction of prompt groups are seeded from it, with G−1 trajectories sampled fresh and one replayed. The resulting mismatch is handled by keeping a GSPO-style geometric-mean ratio inside the clipping term and moving the distribution correction into a separate unnormalized sequence-level factor outside it. Replayed latents are reused only down to a truncation step, with the low-noise tail regenerated by the current policy.
Critical strengths
The replay mechanism addresses a real and under-treated failure mode in GRPO: groups in which every sample fails, reward variance is zero, and the advantage collapses to zero.
Critical weaknesses
1.My main concern is that all three contributions sit close to existing work, and the paper is aware of this in several places.
Reward-filtered replay with a distribution correction is well covered on the LLM side. RePO, ReMix and BAPO are all cited in Sec. 2.2, and between them they already do buffer-based advantage estimation, KL-constrained reuse of stale data, and selective reuse of high-value samples. The buffer here — keep the group argmax, one entry per prompt, decay the stored reward each epoch — reads as a sensible implementation of that recipe rather than something new. The sequence ratio is GSPO's geometric mean, and B.1 more or less says so when it lists the method alongside PPO/GRPO/GSPO. The 1/σ² blow-up at low noise is not a new finding either; B.3 credits earlier on-policy work for it and adds only that the drift gap is larger when the sample is stale.
What I would count as the authors' own is the separation of the clipping ratio from the correction factor, so that clipping still measures movement away from p_old, and the decision to truncate during rollout, which has the nice consequence that the retained product remains an exact ratio. Both are correct, and the second is cleaner than it first appears. But they are refinements.
2.The claim in Sec. 5.2 that OP-GRPO reaches Flow-GRPO's final performance in only 34.2% of the steps is read off the training curves in Fig. 3, and 2 things make that reading unsafe.
First, the two methods are not training on the same distribution. By Alg. 1, buffer entries are admitted by reward argmax, so the prompts held in the buffer are selected for being ones on which the model already scores well, and a fraction α of OP-GRPO's groups are seeded from that subset — including the G−1 freshly sampled trajectories in those groups. A gap in mean training reward therefore mixes prompt enrichment with policy improvement, and the paper neither discusses nor bounds the former.
Second, "the number of steps needed to reach Flow-GRPO's final performance" is well defined only if the baseline has converged. In Fig. 3(c) the Flow-GRPO curve is still clearly rising at 2,200 steps, so its "final performance" is simply where the run was stopped, and the reported ratio moves with that choice.
Questions for authors
Both are clarifications, not suggestions for future work.
First, diversity. The buffer keeps the group argmax and replays it with a large positive advantage, and truncation reuses the high-noise part of the trajectory, the part that fixes layout. Both push toward one realisation per prompt, and none of the reported metrics would catch that — they are all per-sample quality scores, and a model collapsing to a single good image per prompt would score higher on all of them. Could you report a diversity measure against Flow-GRPO at matched task metric, e.g. pairwise perceptual distance across samples for the same prompt?
Second, the training curves. The 34.2% figure comes from Fig. 3, but buffer entries are admitted by reward argmax, so buffer prompts are ones the model already does well on and a share of OP-GRPO's groups is drawn from them; the gap in mean training reward then mixes prompt enrichment with real improvement. It also depends on where the baseline stops, and Flow-GRPO is still rising at 2,200 steps in Fig. 3(c). The step counts disagree too: 1,150/1,600 in Sec. 5.2, past 2,000 in Fig. 3(c), 1,500/2,200+ in App. C. Appendix C says the test set is evaluated every 50 steps — could you plot test metric against step and recompute, including the 30.1% for video?
Two smaller points on the same figures. Fig. 6(b) has an unlabelled "OP-GRPO" curve the text never defines; Sec. 5.5 says 25% is used throughout, yet OP-GRPO-25 is a separate and the weakest curve there, and Table 3 gives 0.125. What does the main setting use, and does "proportion" mean samples or prompt groups? And in Fig. 9 the curves start apart, 0.805 against 0.790, about 40% of the final gap, though the buffer is empty at step 0 and both should start from the same checkpoint.