acceptodds
Under review as a conference paper at ICLR 2027

How Off-Policy Can GRPO Be? Efficient -GRPO with Negative-Advantage Veto

Abstract

Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose -GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation–optimization stages. This design induces high rollout staleness while greatly reducing rollout–optimization switching overhead. To stabilize learning under stale data, -GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, -GRPO matches or exceeds the performance of standard GRPO while achieving around 2× speedup in wall-clock training time, establishing a substantially improved performance–efficiency trade-off for LLM reinforcement learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Off-policy RL, LLM, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r97HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper analyze how GRPO can allow off-policy training in LLM RL training.

Traditional GRPO: need swap frequently for rollout gen. & policy optimization, which lead to high cost in budget.

Core: propose mu-grpo, change training process to less and large-scale generation-optimization stages, which reduce the swap from rollout and policy update. In addition, it combined with relaxed clipping and Negative-Adavantage Veto.

experiments on five benchmarks shows the effectiveness of the method, and reduce the rollout generation time and overall training time.

Critical strengths
  1. the research question is valuable, and method is good in effiency.

  2. anaysis on high-staleness training lead failure of trianing is deep and recognizable.

  3. experiments covers mutiple scenes, including coding tasks. ablation studies also seem fine.

Critical weaknesses
  1. evaluation bias during main results. since authors said they evaluate per 200 steps, then select the ckpt who has the highest avg performance on all benchmarks. This may cause bias. authors may need to distinguish the validation set on ckpt selections, with the final test benchmarks.

  2. improvements are not consistent, and seems limited.

  3. authors use prefix-support mismatch and chi-square divergence to explain why post-trigger suffix updates may cause training unstable. but the therotical support I think only explain that under some specific conditions, bbehavior policy and current policy prefix distributions may contains distribution difference, but why this divergence must/will cause the collapse of GRPO trianing.

Questions for authors

see above

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (r97H) (1/1)

We thank Reviewer r97H for the comments. We are glad the reviewer found the research question valuable, the method efficient, the analysis of the high-staleness failure mode deep, and the experiments broad. We address the three weaknesses below. A revised PDF has been uploaded, with all changes marked in blue. All accuracies below are final-checkpoint numbers unless noted.

W1: "…authors may need to distinguish the validation set on ckpt selections, with the final test benchmarks."

We agree that taking the best of 21 evaluations on the test benchmarks carries an optimistic selection bias. The protocol follows M2PO (Zheng et al., ICLR 2026) and was applied identically to every method; we now also report the final checkpoint (update 4096) for every run, which involves no selection. (We cannot re-select on a validation set retroactively, since only the best and final checkpoints were retained.) The conclusions hold: μ-GRPO matches or exceeds standard GRPO on five of six models, including the newly added Qwen3-Base-4B, and is 0.8 points behind on Qwen3-Base-8B (2.1 at the best checkpoints). The selection did not favor our method: the mean best−final gap is 0.28 Avg.5 points for μ-GRPO and 1.08 for the baselines. The full best-vs-final table is in our response to Reviewer ahme (1/6).

ModelGRPO μ=4GRPO μ=1024μ-GRPOT_total GRPO μ=4 (h)T_total μ-GRPO (h)Speedup
Llama-3.2-3B22.117.322.322.718.61.22×
Qwen2.5-Math-7B47.141.247.623.718.11.31×
Qwen3-Base-1.7B29.125.229.420.512.41.65×
Qwen3-Base-4B (new)45.841.347.143.821.72.02×
Qwen3-Base-8B48.939.148.158.225.92.25×
DeepSeek-7B65.257.768.661.951.31.21×

Avg.5 at the final checkpoint; GRPO (μ=1024) and μ-GRPO run in the same staged pipeline.

W2: "improvements are not consistent, and seems limited."

Our claim is comparable accuracy at lower cost, not higher accuracy than standard GRPO, and the revision states this explicitly. Against standard GRPO (μ=4), accuracy differences are small (−0.8 to +1.3 Avg.5 on five models, +3.4 on DeepSeek-7B), while end-to-end training time is 1.2–2.25× lower on every model.

Where the stabilization should appear as an accuracy gain, it is consistent. Against GRPO at the same staleness in the same pipeline, μ-GRPO is +4.2 to +10.9 points on all six models, +14.0 on LiveCodeBench, and +5.8 in the asynchronous setup (the last two are new controls). Against M2PO and BAPO ported into the same pipeline (new; cost within 10% of μ-GRPO), it is ahead of BAPO on all four models tested (+1.3 to +3.9) and of M2PO on three (+1.5 to +1.8), and 0.6 behind M2PO on Qwen3-Base-8B.

W3: "…why this divergence must/will cause the collapse…"

The reviewer is right that the theory does not show this, and we have corrected the wording. Theorem 1 shows only that token-level GRPO over-weights post-trigger tokens by the inverse prefix ratio and that clipping on the local ratio cannot detect it; it does not prove collapse. That these updates cause the collapse is an empirical claim, supported by three experiments: (i) the volume-matched control (Table 9): retaining the same number of negative-advantage tokens before the trigger is stable (41.2 at update 400), after it collapses (17.6); (ii) a new measurement in the collapsing run: 97–99% of post-trigger tokens have a prefix ratio below 10⁻⁴ although their local ratios are ordinary (median 0.92–0.99), against at most 2.9% of pre-trigger tokens; (iii) a new control that vetoes directly on the prefix ratio also stabilizes training (47.1 vs. 47.6 Avg.5). Details are in our response to Reviewer ahme (5/6) and the revised Sec. 3.3 and App. E.

Revision. W1: Table 1\color{blue}{\text{Table 1}} has an added final-checkpoint column, the best-vs-final table is added as App. G.1\color{blue}{\text{App. G.1}}, Sec. 5\color{blue}{\text{Sec. 5}} and App. B.1\color{blue}{\text{App. B.1}} state the selection protocol and its bias, and Qwen3-Base-4B is added as App. G.3\color{blue}{\text{App. G.3}}. W2: Sec. 1\color{blue}{\text{Sec. 1}} and Sec. 5.1\color{blue}{\text{Sec. 5.1}} state the claim as comparable accuracy at lower cost, and the abstract reports per-model speedups; the same-pipeline M2PO/BAPO comparison is added as App. G.2\color{blue}{\text{App. G.2}} and the coding/asynchronous GRPO (μ=1024) controls as App. G.5\color{blue}{\text{App. G.5}}. W3: Sec. 3.3\color{blue}{\text{Sec. 3.3}} and App. E\color{blue}{\text{App. E}} describe Theorem 1 as explaining why token-level clipping cannot detect the prefix mismatch, not as proving the collapse; App. E.1\color{blue}{\text{App. E.1}} derives the assumption from the trigger, and the prefix-ratio measurement and prefix-ratio veto are added as App. G.4\color{blue}{\text{App. G.4}}.


We hope these results address the concerns raised; if so, we would be grateful if the reviewer would consider updating the score, and we are happy to provide further details.

Reviewer_r97HOpenReview ↗

Thank you

Thank you for your rebuttal, it partially solves my concerns.

In addition, I recently find this paper also discuss the use of off-policy GRPO, therefore I suggest authors may consider to include this paper and some other papers into your discussion.

Wang et al. Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning. ICLR 2026

AuthorsOpenReview ↗

Response to Reviewer r97H’s follow-up

Thank you for the reply and for the pointer. We are glad our rebuttal helped address some of your concerns.

Both SFPO (Wang et al., ICLR 2026) and our work study how to reuse rollouts across policy updates. SFPO takes a few gradient steps on the same batch and controls the resulting drift by moving the policy parameters back toward their starting point before a final correction step. Our work studies why GRPO collapses when rollouts are refreshed far less frequently (once per 1024 updates) and how the negative-advantage veto stabilizes training in that regime, so the two are complementary. We have added this connection to the related-work discussion (Sec. 6\color{blue}{\text{Sec. 6}} and App. F\color{blue}{\text{App. F}} of the updated PDF), and we are happy to include any other related papers you have in mind.

If there is a particular point from your review that you would like us to clarify further, we would be happy to address it here.

Reviewer_ahmeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how far Group Relative Policy Optimization (GRPO) can be pushed into a highly off-policy regime by increasing rollout staleness μ=Btrain/Bmini\mu = B _ {\mathrm{train}} / B _ {\mathrm{mini}} from standard μ=4\mu = 4 to μ=1024\mu = 1024 (4,096 optimizer updates organized into 4 sequential rollout–optimization stages, Sections 2 and 4, Figures 4–5). In Section 3 (Figures 2–3), the authors diagnose why high-staleness GRPO fails: standard clipping [0.8,1.2][0.8, 1.2] clips too many tokens and stalls early, whereas relaxed clipping [0,5][0, 5] learns quickly but collapses mid-stage. By decomposing negative-advantage responses around the first extremely low-ratio trigger token κ(y)=min⁡{t:At<0,ρt(θ)<τc}\kappa(y) = \min\lbrace t : A _ t \lt 0, \rho _ t(\theta) \lt \tau _ c \rbrace (τc=10−4\tau _ c = 10^{-4}), Section 3.3 and Appendix C.2–C.3 (Tables 7–9) show that masking only the trigger tokens T(y)\mathcal{T}(y) still collapses, whereas masking the post-trigger suffix Hκ(y)\mathcal{H} _ \kappa(y) or the entire triggered response Sseq(y)\mathcal{S} _ {\mathrm{seq}}(y) stabilizes training.

Based on this diagnosis, the paper proposes μ\mu-GRPO (Section 4, unnumbered objective Jμ-GRPO(θ)J _ {\mu\text{-GRPO}}(\theta) on page 6), which combines relaxed clipping [0,5][0, 5] with Negative-Advantage Veto (NAV) (Sseq\mathcal{S} _ {\mathrm{seq}} by default). Across five 1.7B–8B models trained on DeepScaleR and evaluated on five primary mathematical benchmarks (AMC23/24, AIME24/25, MATH500, Table 1) plus three held-out math benchmarks (Appendix Table 15), a coding task (Qwen2.5-Coder-7B, Section 5.1), and an asynchronous setup (Table 2), μ\mu-GRPO preserves or slightly improves peak accuracy on 4 of 5 models while reducing wall-clock training time by an unweighted mean of 1.53×1.53\times (median 1.31×1.31\times). While the post-trigger suffix diagnosis (especially the volume-matched control in Table 9) is genuinely insightful, I lean toward weak rejection because (1) the evaluation selects the best checkpoint directly on the reported test benchmarks every 200 updates, (2) the wall-clock efficiency comparisons do not control for generated token counts or response lengths (which vary drastically even among methods sharing the exact same μ=1024\mu = 1024 staged pipeline), (3) accuracy drops by 2.1 points on the only 8B general base model (Qwen3-Base-8B) alongside an unexplained late-stage rise in NAV vetoing, and (4) Theorem 1 assumes low prefix support rather than deriving it from the local NAV trigger.

Critical strengths

1. Well-controlled empirical diagnosis of why extreme off-policy GRPO collapses under relaxed clipping.
Rather than proposing an ad-hoc heuristic in isolation, Section 3 systematically narrows down the source of collapse at μ=1024\mu = 1024: Figure 2 shows the trade-off between over-clipping under [0.8,1.2][0.8, 1.2] and mid-stage collapse under [0,5][0, 5]; Figure 3 and Table 7 compare five masking scopes (∅,T,Sκ,Hκ,Sseq\emptyset, \mathcal{T}, \mathcal{S} _ \kappa, \mathcal{H} _ \kappa, \mathcal{S} _ {\mathrm{seq}}) to show that post-trigger non-trigger suffix tokens Hκ(y)\mathcal{H} _ \kappa(y)—not the trigger tokens T(y)\mathcal{T}(y) themselves—drive the collapse; and Table 9 provides a clean volume-matched control showing that keeping pre-trigger negative-advantage tokens maintains 41.2% accuracy whereas keeping the exact same number of post-trigger tokens collapses to 17.6%.

2. Practical staged training design evaluated across multiple backbones, held-out math sets, coding, and async RL.
Co-designing NAV with 4-stage sequential execution (μ=1024\mu = 1024) reduces rollout refreshes and weight synchronizations from 1,024 to 4 over 4,096 updates while vetoing only 0.17%–4.03% of tokens on average (Table 5), outperforming high-staleness GRPO (μ=1024\mu=1024), M2PO, BAPO (Table 1), and GPPO/CISPO/GSPO (Appendix Table 16) on Qwen2.5-Math-7B.

Critical weaknesses

1. Uncontrolled response-length / token-count confounds in wall-clock timing and unexplained accuracy degradation on Qwen3-Base-8B.

  • Missing token-count and response-length normalization in Table 1: In Table 1, GRPO (μ=1024\mu=1024), M2PO (μ=1024\mu=1024), BAPO (μ=1024\mu=1024), and μ\mu-GRPO (μ=1024\mu=1024) all use the exact same 4-stage μ=1024\mu = 1024 system pipeline (4 rollout phases, 4 weight synchronizations, identical prompt counts and G=8G=8 responses per prompt). Yet their rollout generation times TrolloutT _ {\mathrm{rollout}} differ by up to 3×3\times: on Qwen3-Base-8B, TrolloutT _ {\mathrm{rollout}} is 7.9h for μ\mu-GRPO vs. 23.2h for M2PO and 23.9h for BAPO (21.3h for standard GRPO μ=4\mu=4); on Qwen3-Base-1.7B, TrolloutT _ {\mathrm{rollout}} is 5.2h for μ\mu-GRPO vs. 12.9h for M2PO and 12.8h for BAPO. Because system orchestration and synchronization counts are identical across all μ=1024\mu = 1024 methods, these massive rollout-time differences must be driven by differences in generated response lengths / total generated tokens (e.g., sequence-level NAV suppressing longer failed trajectories) rather than by staged orchestration alone. However, the paper never reports total generated tokens, mean/percentile response lengths, truncation rates, or tokens/s throughput.
  • Accuracy drop and rising late-stage veto rate on Qwen3-Base-8B (and omission of Qwen3-Base-4B): Within the Qwen3-Base family, the paper tests 1.7B and 8B (skipping Qwen3-Base-4B), and Qwen3-Base-8B is the only model where μ\mu-GRPO degrades accuracy relative to both standard GRPO (μ=4\mu=4) (48.2% vs. 50.3%, -2.1 pp on Avg.5\mathrm{Avg.} _ 5 with MATH500 dropping from 86.3% to 81.8%; 50.7% vs. 51.6% on Avg.8\mathrm{Avg.} _ 8 in Table 15) and M2PO (μ=1024\mu=1024) (48.2% vs. 50.1%, -1.9 pp on Avg.5\mathrm{Avg.} _ 5; 50.7% vs. 51.9% on Avg.8\mathrm{Avg.} _ 8). At the same time, Appendix Table 5 (p. 15) shows that Qwen3-Base-8B is the only model whose NAV veto rate increases in later stages (Stage 1 0.81% to Stage 2 2.83% to Stage 3 4.72%), and it is also the single outlier driving the headline average speedup (2.25×2.25\times total speedup vs. 1.21×–1.65×1.21\times\text{--}1.65\times on the other four models). Because all diagnostic ablations (Tables 2–4, 7–14, 16) are restricted to 4k-context Qwen2.5-Math-7B, this 8k-context Qwen3-Base-8B degradation and response-length interaction remain unexamined.

2. Test-benchmark checkpoint selection (test-set peeking) and incomplete high-staleness baselines on coding and async RL.

  • Checkpoint selection directly on the reported test sets: In Section 5 (p. 7, lines 351–354), models are evaluated every 200 updates directly on the five reported test benchmarks (AMC23/24, AIME24/25, MATH500), and Table 1 reports the checkpoint that achieves the highest average score on those same test benchmarks (while pairing it with the wall-clock time of the full 4,096-update run). Because AIME24/25 (30 problems each) and AMC23/24 have small sample sizes, taking the maximum across roughly 20 evaluation checkpoints on the test set introduces optimistic selection bias and obscures final-checkpoint (step 4,096) performance—especially on Qwen3-Base-8B where NAV vetoing peaks in Stage 3 (4.72%).
  • Slight regressions and missing off-policy baselines in coding and async execution: On both the coding task (Qwen2.5-Coder-7B on LiveCodeBench: 38.6% vs. 39.3% for standard GRPO, Section 5.1, p. 8) and asynchronous training (47.7% vs. 47.9% for GRPO, Table 2), μ\mu-GRPO slightly trails standard GRPO, and neither experiment includes high-staleness baselines (GRPO μ=1024\mu=1024, M2PO, or BAPO).
Questions for authors
  1. Assessment-critical (Token counts, response lengths, and the Qwen3-Base-8B anomaly):

    • For each model and method in Table 1 (GRPO μ=4\mu=4, GRPO μ=1024\mu=1024, M2PO, BAPO, μ\mu-GRPO), what are the total generated tokens across the 4 stages, mean/P95 response lengths, truncation rates, and rollout throughput (tokens/s)? Specifically, why does μ\mu-GRPO take 7.9h of rollout time on Qwen3-Base-8B while M2PO and BAPO take 23.2h and 23.9h under the exact same 4-stage μ=1024\mu = 1024 schedule?
    • Why does μ\mu-GRPO lose 2.1 points vs. GRPO (μ=4\mu=4) and 1.9 points vs. M2PO on Qwen3-Base-8B (while its NAV veto rate climbs from 0.81% in Stage 1 to 4.72% in Stage 3 in Table 5), and how does μ\mu-GRPO perform on Qwen3-Base-4B?
  2. Assessment-critical (Final-checkpoint and validation-selected accuracy without test-set peeking):
    What are the Avg.5\mathrm{Avg.} _ 5 and Avg.8\mathrm{Avg.} _ 8 accuracies in Table 1 and Table 15 (and the actual wall-clock time-to-checkpoint) when evaluating (a) the final checkpoint at update 4,096 and (b) a checkpoint selected strictly via an independent validation split rather than taking the best average over AMC23/24, AIME24/25, and MATH500?

  3. Assessment-critical (Empirical verification of prefix-occupancy mismatch and zero-variance group handling):

    • Since Theorem 1 assumes qH≤rmHq _ {\mathcal{H}} \le r m _ {\mathcal{H}} rather than deriving it from the local trigger ρk(θ)<10−4\rho _ k(\theta) \lt 10^{-4}, what is the empirical distribution of the cumulative log-prefix-ratio log⁡Rt−1(θ)=∑j=1t−1log⁡ρj(θ)\log R _ {t-1}(\theta) = \sum _ {j=1}^{t-1} \log \rho _ j(\theta) on post-trigger non-trigger suffix tokens t∈Hκ(y)t \in \mathcal{H} _ \kappa(y) during training? How would a direct prefix-ratio veto (Rt−1(θ)<τpreR _ {t-1}(\theta) \lt \tau _ {\mathrm{pre}}) perform compared to the local token-ratio trigger?
    • How does your implementation handle homogeneous groups where all G=8G = 8 binary rewards are identical (std=0\mathrm{std} = 0 in Eq. 1)?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (ahme) (1/6)

We thank Reviewer ahme for the exceptionally careful review. We are glad the reviewer found the diagnosis of why high-staleness GRPO collapses well controlled, including the volume-matched control, and the staged training design practical. We address each point below in six comments, grouped by topic; the labels (W1.1, Q1a, …) follow the review. A revised PDF has been uploaded, with all changes marked in blue.

Summary reasons → comments: (1) → A (1/6); (2) → B (2/6, 3/6); (3) → C (4/6); (4) → D (5/6).


A. Checkpoint selection on test benchmarks (W2.1, Q2; summary reason 1)

Summary, reason (1): "the evaluation selects the best checkpoint directly on the reported test benchmarks every 200 updates"

W2.1: "…taking the maximum across roughly 20 evaluation checkpoints on the test set introduces optimistic selection bias and obscures final-checkpoint (step 4,096) performance…"

Q2: "What are the Avg.5 and Avg.8 accuracies … when evaluating (a) the final checkpoint at update 4,096 and (b) a checkpoint selected strictly via an independent validation split…?"

We agree that taking the best of 21 evaluations on the test benchmarks carries an optimistic selection bias. The protocol follows M2PO (Zheng et al., ICLR 2026) and other prior work (1-shot RLVR, Wang et al., NeurIPS 2025; GRESO, Zheng et al., NeurIPS 2025); it was applied identically to every method, but the reviewer is right that final-checkpoint numbers are needed to judge its effect. Table R1 (below) reports Avg.5 and Avg.8 at the final checkpoint (update 4096) for all 25 runs of Table 1, next to the best checkpoint.

On Llama-3.2-3B, Qwen2.5-Math-7B, and Qwen3-Base-1.7B the best−final reduction is similar across methods (Avg.5: 0.3–0.6 for μ-GRPO, 0.2–1.0 for the baselines). On Qwen3-Base-8B and DeepSeek-7B the baselines drop by 0.9–2.5 points and μ-GRPO by 0.0–0.1: μ-GRPO's best checkpoint is the final one or within 0.1 of it, whereas the baselines' best checkpoints are earlier peaks (Fig. 1 for DeepSeek-7B). On average the gap is 0.28 Avg.5 points for μ-GRPO and 1.08 for the baselines, so the selection did not favor our method.

Two things follow. (i) The conclusions of Tables 1 and 15 hold at the final checkpoint: μ-GRPO matches or exceeds standard GRPO on four of five models (+0.2 to +3.4 Avg.5), leads GRPO (μ=1024) by 4.2–10.9 points in the same staged pipeline, leads BAPO on all five models and M2PO on four, and has the best Avg.8 on four of five models. (ii) On Qwen3-Base-8B the gap to GRPO (μ=4) shrinks from 2.1 to 0.8 Avg.5 points (48.1 vs. 48.9) and from 0.9 to 0.7 Avg.8 points (50.5 vs. 51.2); MATH500 is 83.4 vs. 85.2 (81.8 vs. 86.3 at the best checkpoints); the gap to M2PO is 0.3 / 0.5. The Avg.5 and Avg.8 gaps are within 1 point, at 2.25× lower cost (25.9 h vs. 58.2 h).

Q2 (a): Table R1; since the final checkpoint is the last update, its wall-clock time-to-checkpoint is the full T_total of Table 1. Q2 (b): We cannot provide this retroactively: we retained the evaluation logs and only the best and final checkpoints of each run, so a checkpoint cannot be re-selected on a validation set without retraining all runs. The final checkpoint involves no selection.

Revision. Table 1\color{blue}{\text{Table 1}} has an added final-checkpoint Avg. column, Table R1 is added as App. G.1\color{blue}{\text{App. G.1}}, and Sec. 5\color{blue}{\text{Sec. 5}} states the selection protocol and its bias.

Table R1. Best vs. final checkpoint for every run in Table 1.

MethodμAvg.5 bestAvg.5 finalΔ5Avg.8 bestAvg.8 final
Llama-3.2-3B-Instruct
GRPO422.622.10.525.224.2
GRPO102418.317.31.020.420.1
M2PO102421.420.70.724.023.1
BAPO102421.521.10.424.123.7
μ-GRPO102422.922.30.625.625.2
Qwen2.5-Math-7B
GRPO447.347.10.246.946.4
GRPO102441.841.20.641.741.1
M2PO102446.846.10.746.546.2
BAPO102444.243.21.043.942.9
μ-GRPO102448.047.60.447.046.5
Qwen3-Base-1.7B
GRPO429.529.10.432.432.1
GRPO102425.525.20.328.127.4
M2PO102428.127.60.532.331.7
BAPO102426.526.20.331.330.8
μ-GRPO102429.729.40.333.032.8
Qwen3-Base-8B
GRPO450.348.91.451.651.2
GRPO102441.239.12.143.343.1
M2PO102450.148.41.751.951.0
BAPO102447.545.91.649.248.7
μ-GRPO102448.248.10.150.750.5
DeepSeek-7B
GRPO466.165.20.964.364.1
GRPO102460.257.72.558.858.1
M2PO102467.865.32.565.064.1
BAPO102467.164.92.264.263.7
μ-GRPO102468.668.60.066.065.9

best: highest Avg.5 among the 21 evaluations of a run (every 200 updates and at update 4096), as reported in Tables 1 and 15; final: update 4096; Δ5 = best − final (Avg.5). Avg.8 adds Gaokao, Minerva Math, and OlympiadBench, which were never used for selection. M2PO/BAPO rows are the released-pipeline runs (see B); Qwen3-Base-4B is reported in C.

AuthorsOpenReview ↗

Response to Reviewer 2 (ahme) (2/6)

B1. Why T_rollout differs among the μ=1024 methods (W1.1, Q1a; summary reason 2)

Summary, reason (2): "the wall-clock efficiency comparisons do not control for generated token counts or response lengths (which vary drastically even among methods sharing the exact same μ=1024 staged pipeline)"

W1.1: "…these massive rollout-time differences must be driven by differences in generated response lengths / total generated tokens … rather than by staged orchestration alone."

Q1a: "…why does μ-GRPO take 7.9h of rollout time on Qwen3-Base-8B while M2PO and BAPO take 23.2h and 23.9h under the exact same 4-stage μ=1024 schedule?"

How the rows of Table 1 were executed. The submission stated that GRPO (μ=1024) shares the staged pipeline with μ-GRPO; for M2PO and BAPO it said only that we used their recommended configurations and released code, and the text could be read as if every μ=1024 row shared one pipeline. They did not:

  • GRPO (μ=4): standard verl interleaved execution; training state resident during generation; behavior log-probs recomputed by the trainer.
  • GRPO (μ=1024), μ-GRPO: our staged pipeline; rollout engine released between phases; behavior log-probs taken from the engine; Stage-0 rollouts (base model) generated once and shared by staged runs of the same model, with their time counted in every run.
  • M2PO, BAPO: their released codebases, configured for the same regime (four rollout phases of 32,768 prompts × 8 responses, each used once over 1024 updates; not the lagged stale-k protocol of the M2PO paper) with the same data, budget, hardware, evaluation, and recommended hyperparameters, but with training state resident and log-probs recomputed, as in standard GRPO.

Accuracy was thus compared in a matched data regime, but T_rollout/T_total for M2PO and BAPO reflected their native orchestration. The efficiency statements against them in Sec. 5.1\color{blue}{\text{Sec. 5.1}} (1.87×/1.49×, 1.59×) were drawn across orchestrations and we have removed them.

Why 7.9 h vs. 23 h (Q1a). Orchestration, not response length. Table R2 shows M2PO and BAPO in their released pipelines and ported into the staged pipeline with only the objective changed. In the released pipelines they generate about as many tokens as μ-GRPO (0.89–1.18×) but at the throughput of standard GRPO; on Qwen3-Base-8B, M2PO generates 11% fewer tokens than μ-GRPO yet takes 2.9× longer. In the staged pipeline their throughput is comparable to that of the other staged runs, T_rollout drops to 8.2/8.5 h (μ-GRPO: 7.9 h), and T_total is within −1% to +10% of μ-GRPO on all three models and 1.1–1.9× shorter than in the released pipelines, with accuracy within 1.2 points of the released-code runs. On Qwen3-Base-4B (see C), staged M2PO and BAPO likewise run within 1.5% of μ-GRPO's T_total.

Two conclusions. (i) The staged pipeline is objective-agnostic: its wall-clock gain is available to any objective that remains stable at large staleness. (ii) The comparison with M2PO/BAPO is an accuracy comparison at matched cost: μ-GRPO is ahead of BAPO on all three models (+1.5 to +3.9 Avg.5) and of M2PO on two (+1.8, +1.7), and 0.6 behind M2PO on Qwen3-Base-8B.

Table R2. M2PO/BAPO: released vs. staged pipeline (final checkpoint).

ModelMethodPipelineAvg.5Tokens (B)tok/sT_roll (h)T_total (h)
Math-7BGRPO μ=4interleaved47.10.8627.1k8.823.7
GRPO μ=1024staged41.20.6446.8k3.817.9
M2POreleased46.10.7832.3k6.720.2
M2POstaged45.80.8347.1k4.917.9
BAPOreleased43.20.9135.1k7.224.3
BAPOstaged43.70.8746.5k5.219.4
μ-GRPOstaged47.60.7745.5k4.718.1
Qwen3-1.7BGRPO μ=4interleaved29.10.8621.3k11.220.5
GRPO μ=1024staged25.20.9038.5k6.513.3
M2POreleased27.60.7616.4k12.924.2
M2POstaged27.70.7940.6k5.413.1
BAPOreleased26.20.8718.9k12.823.9
BAPOstaged27.40.8240.0k5.713.4
μ-GRPOstaged29.40.8143.3k5.212.4
Qwen3-8BGRPO μ=4interleaved48.91.4318.6k21.358.2
GRPO μ=1024staged39.11.1543.2k7.423.6
M2POreleased48.41.2014.4k23.249.7
M2POstaged48.71.3144.4k8.226.6
BAPOreleased45.91.5117.6k23.954.1
BAPOstaged46.61.5651.0k8.528.5
μ-GRPOstaged48.11.3547.5k7.925.9

released: the method's released codebase; staged: our pipeline. Tokens = generated response tokens over all rollout phases; tok/s = Tokens / T_roll. Staged runs of a model share Stage-0 rollouts (Math-7B: 0.25 B tokens, 1.32 h; Qwen3-1.7B: 0.23 B, 1.73 h; Qwen3-8B: 0.31 B, 1.94 h), counted in each run's times.

AuthorsOpenReview ↗

Response to Reviewer 2 (ahme) (3/6)

B2. Token statistics; the speedup over standard GRPO is throughput, not shorter responses (W1.1, Q1a)

W1.1: "…(e.g., sequence-level NAV suppressing longer failed trajectories) … the paper never reports total generated tokens, mean/percentile response lengths, truncation rates, or tokens/s throughput."

Q1a: "For each model and method in Table 1 …, what are the total generated tokens across the 4 stages, mean/P95 response lengths, truncation rates, and rollout throughput (tokens/s)?"

Table R2b (below) reports these for all 25 runs of Table 1.

Decomposition. Writing T_rollout = tokens / throughput, the GRPO (μ=4) → μ-GRPO rollout speedup is (length factor × throughput factor): Llama 0.95 × 1.25, Math-7B 1.12 × 1.68, Qwen3-1.7B 1.06 × 2.03, Qwen3-8B 1.06 × 2.55, DeepSeek-7B 0.92 × 1.31. μ-GRPO generates between 10% fewer (Math-7B) and 8% more (DeepSeek-7B) tokens than standard GRPO, and truncation rates are within 1.5 points across GRPO (μ=4), GRPO (μ=1024), and μ-GRPO on every model.

NAV cannot shorten rollouts. NAV is a loss mask applied during optimization: every response of a stage is generated by the frozen β_k before any veto is computed, and vetoed responses are still generated, counted in T_rollout, and passed through the trainer's forward pass. It can affect response length only through the policy that becomes β_{k+1}; over the full run μ-GRPO generates −10% to +20% tokens relative to GRPO (μ=1024) in the same pipeline, i.e., no systematic shortening.

Beyond rollout. Of the 32.3 h saved on Qwen3-Base-8B, 18.9 h is outside rollout generation, consistent with fewer weight synchronizations (4 instead of 1024 synchronizations) and no trainer-side behavior log-probability recomputation (Sec. 4).

Revision. Sec. 5\color{blue}{\text{Sec. 5}} and App. B.1\color{blue}{\text{App. B.1}} now state how every method is executed (pipeline, source of behavior log-probabilities, shared Stage-0 rollouts), and Table 1\color{blue}{\text{Table 1}} has a pipeline column. The efficiency statements against M2PO/BAPO in Sec. 5.1\color{blue}{\text{Sec. 5.1}} are replaced by the same-pipeline comparison. Tables R2 and R2b, with the decomposition, are added as App. G.2\color{blue}{\text{App. G.2}}. On the original five models, the speedups over standard GRPO (1.53× T_total, 1.83× T_rollout on average) are unchanged.

Table R2b. Rollout token statistics for every run in Table 1.

ModelMethodPipelineT_roll (h)Tokens (B)Mean lenP95 lenTrunc. (%)tok/s
Llama-3BGRPO μ=4interleaved15.11.10104523281.7420.2k
GRPO μ=1024staged13.21.19113926532.6625.0k
M2POreleased15.31.11106323122.3120.2k
BAPOreleased15.91.26120427333.1222.0k
μ-GRPOstaged12.71.16110225702.4525.4k
Math-7BGRPO μ=4interleaved8.80.8682017031.8327.1k
GRPO μ=1024staged3.80.6460814791.3746.8k
M2POreleased6.70.7874218862.1332.3k
BAPOreleased7.20.9186619372.5535.1k
μ-GRPOstaged4.70.7773917661.7345.5k
Qwen3-1.7BGRPO μ=4interleaved11.20.8681820540.5921.3k
GRPO μ=1024staged6.50.9086123930.6638.5k
M2POreleased12.90.7672419630.5416.4k
BAPOreleased12.80.8783020350.8718.9k
μ-GRPOstaged5.20.8177620220.5143.3k
Qwen3-8BGRPO μ=4interleaved21.31.43136135702.2318.6k
GRPO μ=1024staged7.41.15110130221.2443.2k
M2POreleased23.21.20114129641.1514.4k
BAPOreleased23.91.51144334712.2117.6k
μ-GRPOstaged7.91.35129132241.6747.5k
DeepSeek-7BGRPO μ=4interleaved26.02.67254866568.3428.5k
GRPO μ=1024staged22.32.68255666569.7533.4k
M2POreleased27.92.70257566568.0926.9k
BAPOreleased23.62.592470665610.0430.5k
μ-GRPOstaged21.42.89275866569.6337.5k

Every run generates 1,048,576 responses. Trunc. = fraction of responses reaching the maximum response length (6656 tokens; 2560 for Math-7B); tok/s = Tokens / T_roll.

Reviewer_UwdgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses rollout staleness in GRPO by proposing μ\mu-GRPO, which drastically scales up the number of optimization steps per rollout. The authors identify training instability as stemming from suffix updates on improbable prefixes and resolve this by combining relaxed clipping with a negative-advantage veto that masks affected responses. Empirical results across five models demonstrate preserved reasoning performance on most benchmarks alongside a 1.5-fold training speedup, supported by additional coding and asynchronous experiments.

Critical strengths
  • Targeted masking experiments provide convincing evidence that harmful updates concentrate after low-probability trigger tokens.
  • The method is simple and delivers practical efficiency gains.
  • Extensive ablations, additional benchmarks, and coding and asynchronous experiments strengthen the evaluation.
Critical weaknesses
  • Could the authors clarify how reliably the proposed trigger identifies harmful updates, and whether it may miss harmful updates or suppress useful ones?
  • Multiple-seed validation covers only one model. Some accuracy gains are small, and Qwen3-Base-8B shows a noticeable decline.
  • Guidance on choosing the rollout refresh interval across different training settings remains limited.
Questions for authors
  1. How reliably does the trigger identify prefixes that the current policy is unlikely to generate?
  2. Do all baselines use identical staging and memory management?
  3. How should practitioners choose the refresh interval when changing the model, learning rate, or response length?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (Uwdg) (1/2)

We thank Reviewer Uwdg for the positive assessment. We are glad the reviewer found the masking experiments convincing evidence that harmful updates concentrate after low-probability trigger tokens, the method simple and practically efficient, and the evaluation extensive. A revised PDF has been uploaded, with all changes marked in blue. Following the other reviewers' comments on checkpoint selection, accuracies below are final-checkpoint numbers (update 4096), except for the ablation tables (Tables 4, 7, 11), which follow the submission's protocol.

W1 / Q1: "Could the authors clarify how reliably the proposed trigger identifies harmful updates, and whether it may miss harmful updates or suppress useful ones?"

Does it identify prefixes the current policy is unlikely to generate? Yes. In the relaxed-clipping run without NAV we measured, before the collapse, the prefix ratio R_{t−1} = π_θ(a_<t)/β(a_<t) of negative-advantage tokens. After a trigger, 97–99% of tokens have R_{t−1} < 10⁻⁴ although their own ratios are ordinary (median 0.92–0.99); before the trigger this holds for at most 2.9%, and in untriggered responses for at most 0.8%.

Does it miss harmful updates? We tested the direct alternative, vetoing whenever R_{t−1} < 10⁻⁴. It additionally removes responses whose prefix ratio decays cumulatively without any single low-ratio token (token veto rate 20.1/1.3/2.6/2.1% over Stages 0–3 vs. 15.2/0.3/0.3/0.3%), but it is neither more stable nor more accurate (47.1 vs. 47.6 Avg.5). What the trigger misses is therefore not harmful in practice.

Does it suppress useful updates? The default scope also vetoes the pre-trigger tokens of a triggered response, which are not harmful by themselves (Table 9). The scope ablation (Table 7) shows this costs nothing measurable relative to vetoing only the post-trigger suffix, and after Stage 0 only about 0.3% of tokens are vetoed on Qwen2.5-Math-7B. Over-suppression does occur if the threshold is too large (Table 11: τ_c = 10⁻¹ or 10⁻² loses 1.6–2.8 points), which is why we use 10⁻⁴.

W2: "Multiple-seed validation covers only one model. Some accuracy gains are small, and Qwen3-Base-8B shows a noticeable decline."

Qwen3-Base-8B. At the final checkpoint the gap to standard GRPO is 0.8 Avg.5 points (2.1 at the best checkpoints), and at matched updates there is no systematic deficit (−1.6, +1.1, +0.6, −0.8 at updates 1000/2000/3000/4096). On the newly added Qwen3-Base-4B, μ-GRPO is 1.3 points ahead at 2.0× lower training time.

Size of the gains. Our claim is comparable accuracy at lower cost: against standard GRPO the differences are −0.8 to +1.3 Avg.5 on five models (+3.4 on DeepSeek-7B) at 1.2–2.25× lower T_total. The large and consistent gains are against GRPO at the same staleness in the same pipeline: +4.2 to +10.9 points on all six models.

Seeds. We agree that three seeds on one model is limited. We could not add seeds on further models within the discussion period and state this as a limitation; the conclusions above do not rest on sub-point differences.

AuthorsOpenReview ↗

Response to Reviewer 1 (Uwdg) (2/2)

W3 / Q3: "How should practitioners choose the refresh interval when changing the model, learning rate, or response length?"

What the data support: (i) Table 4 varies the interval at a fixed budget on Qwen2.5-Math-7B: 8 and 4 refreshes (μ = 512, 1024) are within 0.7 points of each other, while 2 and 1 refreshes lose about 2 points. (ii) The same setting, four stages of 1024 updates, was used without tuning for all six models (1.7B–8B), 4k and 8k contexts, and both math and coding. Since a refresh costs a single weight synchronization in the staged pipeline, the practical rule is to err toward more stages. We did not vary the learning rate; policy drift per update grows with it, so we would expect shorter stages at higher learning rates. This is untested and stated as a limitation.

Q2: "Do all baselines use identical staging and memory management?"

No, and the submission did not state this explicitly. GRPO (μ=1024) and μ-GRPO use our staged pipeline. M2PO and BAPO were run in their released codebases in the same four-phase regime, but with training state resident during generation, as in standard GRPO. Their timings therefore reflected their native orchestration, and we have removed the efficiency comparison against them. We then ported both into the staged pipeline on four models: their T_total is within −1% to +10% of μ-GRPO (on Qwen3-Base-8B their rollout time falls from 23.2/23.9 h to 8.2/8.5 h, vs. 7.9 h for μ-GRPO). The comparison is thus an accuracy comparison at matched cost: μ-GRPO is ahead of BAPO on all four models (+1.3 to +3.9 Avg.5) and of M2PO on three (+1.5 to +1.8), and 0.6 behind M2PO on Qwen3-Base-8B. Full tables are in our response to Reviewer ahme (2/6, 3/6).

Revision. W1/Q1: the prefix-ratio measurement and the prefix-ratio veto are added as App. G.4\color{blue}{\text{App. G.4}}. W2: final-checkpoint results are added to Table 1\color{blue}{\text{Table 1}} and App. G.1\color{blue}{\text{App. G.1}}, and the Qwen3-Base-8B trajectory and Qwen3-Base-4B results as App. G.3\color{blue}{\text{App. G.3}}. Q2: Sec. 5\color{blue}{\text{Sec. 5}} and App. B.1\color{blue}{\text{App. B.1}} state how every method is executed, Table 1\color{blue}{\text{Table 1}} has a pipeline column, the efficiency comparison against M2PO/BAPO in Sec. 5.1\color{blue}{\text{Sec. 5.1}} is replaced by the same-pipeline comparison, and the staged M2PO/BAPO runs are added as App. G.2\color{blue}{\text{App. G.2}}. W2/W3: App. G.6\color{blue}{\text{App. G.6}} (Limitations) states that the component ablations and multi-seed runs are on one model and that the learning rate was not varied.


We hope these answers are helpful, and we are happy to provide further details on any point.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

Dear all reviewers,

We sincerely thank all reviewers for their careful and constructive reviews.

We are grateful that the reviewers recognized several strengths of μ-GRPO, including: a "well-controlled empirical diagnosis of why extreme off-policy GRPO collapses under relaxed clipping" (ahme), "targeted masking experiments that provide convincing evidence that harmful updates concentrate after low-probability trigger tokens" (Uwdg), an "analysis on high-staleness training" that is "deep and recognizable" (r97H), a research question that "is valuable" (r97H), a method that "is simple and delivers practical efficiency gains" (Uwdg), a "practical staged training design" (ahme), and "extensive ablations, additional benchmarks, and coding and asynchronous experiments" (Uwdg).

We have posted point-by-point responses to each reviewer and revised the manuscript accordingly; we summarize the key changes below. All modifications are highlighted in blue in the revised PDF. All new results are collected in a new Appendix G, so existing section, table, and figure numbers are unchanged. Overall, the revision adds 6 pages and 8 new tables.

Two points affect how the results should be read, and we state them up front:

  • Checkpoint selection (ahme, r97H). The submission reported the best of 21 evaluations per run, following prior work. We now also report the final checkpoint (update 4096) for every run. The conclusions hold, the gap to standard GRPO on Qwen3-Base-8B shrinks from 2.1 to 0.8 Avg.5 points, and the selection did not favor our method (mean best−final gap: 0.28 for μ-GRPO vs. 1.08 for the baselines). All new experiments are reported at the final checkpoint.
  • Execution pipelines (ahme, Uwdg). M2PO and BAPO were run in their released codebases, not in our staged pipeline. We have removed the efficiency comparison against them and ported both into the staged pipeline on four models. At matched cost (T_total within −1% to +10%), μ-GRPO is ahead of BAPO on all four models (+1.3 to +3.9 Avg.5) and of M2PO on three (+1.5 to +1.8), and 0.6 behind M2PO on Qwen3-Base-8B. The speedups over standard GRPO are unchanged.

Paper Revision Summary

1. Additional evaluation and analysis:

  1. Table 18 in Appendix G.1\color{blue}{\text{Table 18 in Appendix G.1}}: Best vs. final checkpoint for all 25 runs of Table 1. (ahme, r97H)
  2. Table 19 in Appendix G.2\color{blue}{\text{Table 19 in Appendix G.2}}: M2PO and BAPO in their released vs. our staged pipeline: accuracy, rollout time, and total training time. (ahme, Uwdg)
  3. Table 20 in Appendix G.2\color{blue}{\text{Table 20 in Appendix G.2}}: Rollout token statistics for every run; the rollout speedup over standard GRPO comes mainly from throughput (1.25–2.55×), not shorter responses. (ahme)
  4. Table 21 in Appendix G.3\color{blue}{\text{Table 21 in Appendix G.3}}: Qwen3-Base-8B accuracy trajectory and per-stage veto statistics. (ahme, Uwdg)
  5. Table 22 in Appendix G.3\color{blue}{\text{Table 22 in Appendix G.3}}: Qwen3-Base-4B: μ-GRPO is +1.3 Avg.5 over standard GRPO at 2.02× lower training time. (ahme)
  6. Table 23 in Appendix G.4\color{blue}{\text{Table 23 in Appendix G.4}}: Prefix-ratio measurement: 97–99% of post-trigger tokens have a prefix ratio below 10⁻⁴, against at most 2.9% of pre-trigger tokens. (ahme, r97H, Uwdg)
  7. Table 24 in Appendix G.4\color{blue}{\text{Table 24 in Appendix G.4}}: A direct prefix-ratio veto is also stable (47.1 vs. 47.6 Avg.5 for the local trigger) while vetoing more tokens. (ahme, Uwdg)
  8. Table 25 in Appendix G.5\color{blue}{\text{Table 25 in Appendix G.5}}: GRPO (μ=1024) controls for the coding and asynchronous experiments: the control loses 14.4 and 5.4 points, while μ-GRPO stays within 0.4 of standard GRPO. (ahme)

2. Writing:

  1. Update the Abstract\color{blue}{\text{Abstract}}, Section 1\color{blue}{\text{Section 1}}, and Section 5.1\color{blue}{\text{Section 5.1}} to report per-model speedups (1.21–2.25×) and to state the claim as comparable accuracy at lower cost.
  2. Update Section 5\color{blue}{\text{Section 5}} and Appendix B.1\color{blue}{\text{Appendix B.1}} to state how every method is executed, the checkpoint-selection protocol and its bias, and the handling of zero-variance groups.
  3. Update Table 1\color{blue}{\text{Table 1}} with final-checkpoint and pipeline columns, and Table 5\color{blue}{\text{Table 5}} with Qwen3-Base-4B.
  4. Update Section 5.1\color{blue}{\text{Section 5.1}} and Appendix D.2\color{blue}{\text{Appendix D.2}} to replace the efficiency comparison against M2PO/BAPO with the same-pipeline comparison.
  5. Update Section 3.3\color{blue}{\text{Section 3.3}} and Appendix E\color{blue}{\text{Appendix E}} to describe Theorem 1 as explaining why token-level clipping cannot detect the prefix mismatch, not as proving the collapse; add Appendix E.1\color{blue}{\text{Appendix E.1}} to derive its assumption from the trigger.
  6. Add Appendix G.6\color{blue}{\text{Appendix G.6}} to discuss limitations of the evaluation scope.

We thank all reviewers again for their constructive feedback, which has substantially improved the paper. We hope that our responses and the revised paper address the reviewers' concerns, and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.