- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper analyze how GRPO can allow off-policy training in LLM RL training.
Traditional GRPO: need swap frequently for rollout gen. & policy optimization, which lead to high cost in budget.
Core: propose mu-grpo, change training process to less and large-scale generation-optimization stages, which reduce the swap from rollout and policy update. In addition, it combined with relaxed clipping and Negative-Adavantage Veto.
experiments on five benchmarks shows the effectiveness of the method, and reduce the rollout generation time and overall training time.
Critical strengths
-
the research question is valuable, and method is good in effiency.
-
anaysis on high-staleness training lead failure of trianing is deep and recognizable.
-
experiments covers mutiple scenes, including coding tasks. ablation studies also seem fine.
Critical weaknesses
-
evaluation bias during main results. since authors said they evaluate per 200 steps, then select the ckpt who has the highest avg performance on all benchmarks. This may cause bias. authors may need to distinguish the validation set on ckpt selections, with the final test benchmarks.
-
improvements are not consistent, and seems limited.
-
authors use prefix-support mismatch and chi-square divergence to explain why post-trigger suffix updates may cause training unstable. but the therotical support I think only explain that under some specific conditions, bbehavior policy and current policy prefix distributions may contains distribution difference, but why this divergence must/will cause the collapse of GRPO trianing.
Questions for authors
see above
5 comments
Thank you
Thank you for your rebuttal, it partially solves my concerns.
In addition, I recently find this paper also discuss the use of off-policy GRPO, therefore I suggest authors may consider to include this paper and some other papers into your discussion.
Wang et al. Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning. ICLR 2026
Response to Reviewer r97H’s follow-up
Thank you for the reply and for the pointer. We are glad our rebuttal helped address some of your concerns.
Both SFPO (Wang et al., ICLR 2026) and our work study how to reuse rollouts across policy updates. SFPO takes a few gradient steps on the same batch and controls the resulting drift by moving the policy parameters back toward their starting point before a final correction step. Our work studies why GRPO collapses when rollouts are refreshed far less frequently (once per 1024 updates) and how the negative-advantage veto stabilizes training in that regime, so the two are complementary. We have added this connection to the related-work discussion ( and of the updated PDF), and we are happy to include any other related papers you have in mind.
If there is a particular point from your review that you would like us to clarify further, we would be happy to address it here.
Response to Reviewer 3 (r97H) (1/1)
We thank Reviewer r97H for the comments. We are glad the reviewer found the research question valuable, the method efficient, the analysis of the high-staleness failure mode deep, and the experiments broad. We address the three weaknesses below. A revised PDF has been uploaded, with all changes marked in blue. All accuracies below are final-checkpoint numbers unless noted.
We agree that taking the best of 21 evaluations on the test benchmarks carries an optimistic selection bias. The protocol follows M2PO (Zheng et al., ICLR 2026) and was applied identically to every method; we now also report the final checkpoint (update 4096) for every run, which involves no selection. (We cannot re-select on a validation set retroactively, since only the best and final checkpoints were retained.) The conclusions hold: μ-GRPO matches or exceeds standard GRPO on five of six models, including the newly added Qwen3-Base-4B, and is 0.8 points behind on Qwen3-Base-8B (2.1 at the best checkpoints). The selection did not favor our method: the mean best−final gap is 0.28 Avg.5 points for μ-GRPO and 1.08 for the baselines. The full best-vs-final table is in our response to Reviewer ahme (1/6).
Avg.5 at the final checkpoint; GRPO (μ=1024) and μ-GRPO run in the same staged pipeline.
Our claim is comparable accuracy at lower cost, not higher accuracy than standard GRPO, and the revision states this explicitly. Against standard GRPO (μ=4), accuracy differences are small (−0.8 to +1.3 Avg.5 on five models, +3.4 on DeepSeek-7B), while end-to-end training time is 1.2–2.25× lower on every model.
Where the stabilization should appear as an accuracy gain, it is consistent. Against GRPO at the same staleness in the same pipeline, μ-GRPO is +4.2 to +10.9 points on all six models, +14.0 on LiveCodeBench, and +5.8 in the asynchronous setup (the last two are new controls). Against M2PO and BAPO ported into the same pipeline (new; cost within 10% of μ-GRPO), it is ahead of BAPO on all four models tested (+1.3 to +3.9) and of M2PO on three (+1.5 to +1.8), and 0.6 behind M2PO on Qwen3-Base-8B.
The reviewer is right that the theory does not show this, and we have corrected the wording. Theorem 1 shows only that token-level GRPO over-weights post-trigger tokens by the inverse prefix ratio and that clipping on the local ratio cannot detect it; it does not prove collapse. That these updates cause the collapse is an empirical claim, supported by three experiments: (i) the volume-matched control (Table 9): retaining the same number of negative-advantage tokens before the trigger is stable (41.2 at update 400), after it collapses (17.6); (ii) a new measurement in the collapsing run: 97–99% of post-trigger tokens have a prefix ratio below 10⁻⁴ although their local ratios are ordinary (median 0.92–0.99), against at most 2.9% of pre-trigger tokens; (iii) a new control that vetoes directly on the prefix ratio also stabilizes training (47.1 vs. 47.6 Avg.5). Details are in our response to Reviewer ahme (5/6) and the revised Sec. 3.3 and App. E.
Revision. W1: Table 1 has an added final-checkpoint column, the best-vs-final table is added as App. G.1, Sec. 5 and App. B.1 state the selection protocol and its bias, and Qwen3-Base-4B is added as App. G.3. W2: Sec. 1 and Sec. 5.1 state the claim as comparable accuracy at lower cost, and the abstract reports per-model speedups; the same-pipeline M2PO/BAPO comparison is added as App. G.2 and the coding/asynchronous GRPO (μ=1024) controls as App. G.5. W3: Sec. 3.3 and App. E describe Theorem 1 as explaining why token-level clipping cannot detect the prefix mismatch, not as proving the collapse; App. E.1 derives the assumption from the trigger, and the prefix-ratio measurement and prefix-ratio veto are added as App. G.4.
We hope these results address the concerns raised; if so, we would be grateful if the reviewer would consider updating the score, and we are happy to provide further details.