acceptodds
Under review as a conference paper at ICLR 2027

Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis

Abstract

Asynchronous reinforcement learning (RL) improves the efficiency of large language model post-training but introduces stale rollouts generated by earlier policies. Theoretical understanding of how this staleness affects convergence and how to mitigate its impact remains limited. We derive a convergence bound for GRPO-style algorithms that explicitly characterizes the tradeoff between the gradient estimator's second moment and bias. For trajectory-level importance-weighted estimators, our analysis shows that once the second moment is uniformly controlled, delay enters the bound through the bias introduced by clipping or rescaling. Guided by this insight, we propose a novel group mass capping GRPO (GMC-GRPO) method, which minimizes a ratio-based bias bound within a class of weighted estimators sharing a common second-moment guarantee. We establish convergence guarantees for asynchronous GMC-GRPO and show that, compared with TIC-GRPO, it improves the threshold dependence of the fourth-order delay term from to as , where is the ratio threshold. Under local policy overlap, the delay-dependent term decreases as after tuning the step size, where is the group size. For fixed behavior and current policies, the bias introduced by group rescaling also vanishes as , whereas the bias from trajectory-wise clipping can persist. Experiments across Qwen3 models and reasoning benchmarks demonstrate improved robustness to stale rollouts, with GMC-GRPO achieving the best performance among the evaluated baselines under large rollout delays.

Primary area:
learning theory
Keywords:
Reinforcement Learning, Large Language Models, Asynchronous System, Data Staleness.

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_CV87OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies GRPO-style RL post-training of LLMs on stale asynchronous rollouts. Trajectory-level importance-weighted updates stay conditionally unbiased regardless of delay, so staleness enters only through the bias of the variance-control correction. Minimizing a bias envelope under a cap on the group-mean importance ratio yields GMC-GRPO (capped self-normalized importance sampling) with a delay-explicit convergence bound whose delay coefficient scales as ε⁻² rather than TIC-GRPO's ε⁻⁴. Experiments on three Qwen3-Base sizes show GMC-GRPO matching or beating TIC-GRPO under stale rollouts.

Critical strengths
  • Clean, correct delay-explicit convergence analysis. Theorem 3 appears to be the first bound for GRPO-style trajectory-IS methods in which stochastic rollout delay appears explicitly, with a transparent removed-mass bias mechanism (Proposition 1).
  • Consistent empirical robustness across sizes and delays. GMC-GRPO never trails TIC-GRPO on macro-avg@n in Table 1, clearly beats vanilla GRPO, GAC and VCPO, and leads TIC at delays 128–512.
Critical weaknesses
  • ε⁻⁴→ε⁻² is bound-vs-bound; the realized variance cost is unmeasured. The shared envelope of Lemmas 5 and 8 ignores averaging over G and the mass that shrinks TIC's second moment. With G=8 and ε=0.28, one trajectory at ρ=50 plus seven near zero gives GMC roughly 13× the on-policy second moment while TIC's update is ≈0; no ρ̂ or second-moment statistics are reported.
  • Single-run experiments with greedy margins inside noise. No seeds or variance are reported. In Table 1 the greedy macro-average margin of GMC over TIC is at most +1.32 and negative (−0.83) at 4B, so only the avg@n margins plausibly exceed run-to-run noise.
  • "Best among stable baselines" depends on excluding A-3PO. A-3PO beats GMC at 1.7B by 3.8 avg@n points (45.27 vs 41.47, Table 5) yet is relegated to an appendix after one collapsed 8B run, so the abstract's framing holds at 1.7B only by this exclusion.
Questions for authors
  • For D = 128 and 512, report ρ and ρ̂ distributions and per-update gradient norms for GMC versus TIC.
  • Was ε tuned for TIC-GRPO, and can you sweep ε for both methods at delays 128 and 512?
  • How many seeds underlie Tables 1 and 3–5, and does A-3PO's 8B collapse reproduce across seeds?
  • State in the main text that at D = 512 only the last 288 of 800 updates run at steady-state delay.
6 comments

We thank the reviewer for the thoughtful feedback and positive assessment. Below we address the raised questions.

01 Bound comparison

We agree that Theorems 1–2 compare upper bounds and do not establish that GMC-GRPO converges faster. Such comparisons are standard for stochastic methods on nonconvex problems, where algorithm-specific lower bounds are rarely available. However, the gap between the two bounds is not an artifact of loose bounding. It comes from the bias bounds in Lemmas 6 and 9: Theorems 1–2 retain their quadratic-KL branches, which take the form ∥b∥22≤K(ϵ)(Dn(b))2\Vert b\Vert_2^2\le K(\epsilon)(D_n^{(b)})^2 with KGMC(ϵ)∝log⁡−2(1+2ϵ)=Θ(ϵ−2)K_{\rm GMC}(\epsilon)\propto\log^{-2}(1+2\epsilon)=\Theta(\epsilon^{-2}) and KTIC(ϵ)∝cϵ−2=Θ(ϵ−4)K_{\rm TIC}(\epsilon)\propto c_\epsilon^{-2}=\Theta(\epsilon^{-4}). Both orders are attained by the exact biases on a simple instance.

Consider a single prompt with T=1T=1, G=2G=2, two responses y∈{0,1}y\in\{0,1\} with reward r(y)=yr(y)=y, and a logistic policy πθ(y=1)=σ(θ)\pi_\theta(y=1)=\sigma(\theta). Let the behavior and current policies select y=1y=1 with probabilities q=1/4q=1/4 and p=1/4+ϵp=1/4+\epsilon, respectively. Then Assumptions 1–2 hold with L=1/4L=1/4 and C=4\mathcal C=4 for every ϵ∈(0,1/4)\epsilon\in(0,1/4). The ratios are ρ(1)=1+4ϵ\rho(1)=1+4\epsilon and ρ(0)=1−43ϵ\rho(0)=1-\frac43\epsilon. A group contributes to the update only if it contains both responses, and then ρˉ=1+43ϵ>1+ϵ\bar\rho=1+\frac43\epsilon>1+\epsilon. TIC-GRPO therefore drops the rewarded response, whereas GMC-GRPO rescales the group by (1+ϵ)/(1+43ϵ)(1+\epsilon)/(1+\frac43\epsilon). A direct computation gives the exact biases bTIC=−34hb_{\rm TIC}=-\frac34h and bGMC=−ϵ3+4ϵhb_{\rm GMC}=-\frac{\epsilon}{3+4\epsilon}h, where h=p(1−p)h=p(1-p) is the true gradient, while Dn(b)=83ϵ2+O(ϵ3)D_n^{(b)}=\frac83\epsilon^2+O(\epsilon^3). Hence bTIC2/(Dn(b))2=Θ(ϵ−4)b_{\rm TIC}^2/(D_n^{(b)})^2=\Theta(\epsilon^{-4}) and bGMC2/(Dn(b))2=Θ(ϵ−2)b_{\rm GMC}^2/(D_n^{(b)})^2=\Theta(\epsilon^{-2}), so neither coefficient can be improved in its dependence on ϵ\epsilon. This is the mechanism described in Section 4.3: TIC-GRPO removes the full contribution of every trajectory above the threshold, whereas GMC-GRPO removes only the excess of the group average.

We will add this example to the final version of our paper.

02 Realized second moment of the importance weights

We agree that the shared envelope of Lemmas 5 and 8 is a worst-case bound: it does not exploit averaging over GG, and the mass that TIC-GRPO discards can make its realized second moment smaller than that of GMC-GRPO. The reviewer's example is the extreme case of this asymmetry. Since ρ^n(b,i)≥0\hat\rho_n^{(b,i)}\ge0 and 1G∑iρ^n(b,i)≤1+ϵ\frac1G\sum_i\hat\rho_n^{(b,i)}\le1+\epsilon (Proposition 3), we have 1G∑i(ρ^n(b,i))2≤G(1+ϵ)2≈13.1\frac1G\sum_i(\hat\rho_n^{(b,i)})^2\le G(1+\epsilon)^2\approx13.1, with equality when a single trajectory carries the whole capped mass. Even then, the group update of GMC-GRPO equals 1+ϵ=1.281+\epsilon=1.28 times that trajectory's score–advantage term, compared with 50/8=6.2550/8=6.25 times for the raw IS update, and in general GMC-GRPO never amplifies the raw update since ρ^n(b,i)≤ρn(b,i)\hat\rho_n^{(b,i)}\le\rho_n^{(b,i)}. The update of TIC-GRPO is close to zero in this example only because it discards the response that the current policy favors by a factor of 50, which is the bias that Theorems 1–2 account for. This trade-off favors GMC-GRPO under large delays. The delay enters our bounds only through the correction bias, whereas the second moments of both methods share the same delay-independent envelope (Lemmas 5 and 8). As the delay grows, the key is therefore to minimize the bias while keeping the second moment controlled, which is exactly what the group mass cap of GMC-GRPO does.

Empirically, the realized second moment of GMC-GRPO remains moderate. Table R9 reports the mean and mean square of the effective importance weights, ρ^\hat\rho for GMC-GRPO and ρ1{ρ≤1+ϵ}\rho\mathbf 1\{\rho\le1+\epsilon\} for TIC-GRPO, in the fixed-delay runs; on-policy weights have mean and mean square equal to one. GMC-GRPO keeps every trajectory and retains about 90% of the importance mass, while its mean squared weight of 2.6–2.8 stays far below the worst-case value of 13.1 in the reviewer’s example. TIC-GRPO has a smaller mean squared weight, but only because it discards every trajectory whose ratio exceeds 1+ϵ1+\epsilon. These trajectories carry most of the importance mass, so TIC-GRPO retains only about 30% of it, and its average update weight is roughly one-third that of GMC-GRPO. The simulated asynchronous runs (3 seeds, updates 400–800) show the same pattern: the batch average of ρ^2\hat\rho^2 has median 2.57, compared with 5.51 for the raw ratios, and TIC-GRPO assigns zero weight to a median of 25% of the trajectories per update.

TIC-GRPO’s smaller second moment therefore comes at the cost of substantial update attenuation. GMC-GRPO’s better preservation of importance mass helps explain its stronger performance, reported in the last column of Table R9.

Table R9. Effective importance weights in the fixed-delay runs with ϵ=0.28\epsilon=0.28 on Qwen3-1.7B-Base, computed over the updates at steady-state delay (672 updates for D=128D=128 and 288 for D=512D=512). On-policy weights have mean and mean square equal to one. MATH500 accuracy (%) is reported as greedy / sampled Pass@1 (16 samples per problem) at the final checkpoint.

Delay DMethodMeanMean squareMATH500 (greedy / avg@16)
128GMC-GRPO0.9092.56168.40 / 59.03
128TIC-GRPO0.2970.21663.20 / 55.41
512GMC-GRPO0.8852.75156.40 / 42.01
512TIC-GRPO0.3320.24553.80 / 38.17

We will add this discussion and report these statistics in the revision.

03 Multiple seed experiment

We agree that single-run greedy margins can fall within run-to-run noise. We have therefore rerun the simulated asynchronous experiment on Qwen3-1.7B-Base with three seeds for all methods in Table 1 (Table R10). GMC-GRPO attains the best avg@n on every benchmark, and its average exceeds that of TIC-GRPO by 4.5 points (41.67 ± 0.37 vs. 37.21 ± 0.59), well beyond the variation across seeds. In the revision, we will report means and standard deviations over three seeds and base our comparisons primarily on avg@n, which averages over 8–16 samples per problem.

Table R10. avg@n (%) under simulated asynchronous delays on Qwen3-1.7B-Base, mean ± standard deviation over three seeds. Avg. Delay is measured in learner updates.

MethodAvg. DelayMATH500GSM8KGaokaoOlympiadMinervaAMC23Avg. perf.
GRPO158.56 ± 0.7531.36 ± 0.6047.99 ± 1.1526.04 ± 0.426.74 ± 0.238.71 ± 0.469.79 ± 0.7421.77 ± 0.56
GAC158.82 ± 1.0931.62 ± 0.5748.07 ± 0.9925.92 ± 0.926.76 ± 0.108.79 ± 0.7210.21 ± 1.0421.90 ± 0.52
VCPO158.42 ± 0.7947.27 ± 6.3264.73 ± 5.0439.87 ± 4.6215.47 ± 4.3716.28 ± 1.9923.23 ± 3.1034.48 ± 3.99
TIC-GRPO158.97 ± 0.9152.49 ± 0.9967.99 ± 0.7541.76 ± 1.1318.77 ± 0.7617.37 ± 0.3024.90 ± 1.5037.21 ± 0.59
GMC-GRPO160.06 ± 0.2757.63 ± 0.5073.04 ± 1.3947.73 ± 0.4922.28 ± 0.6820.21 ± 0.1029.11 ± 0.5941.67 ± 0.37

04 Seeds and the stability of A-3PO

Tables 1 and 3–5 in the submission each report a single run. In addition to the three-seed rerun of Table 1 on Qwen3-1.7B-Base, we evaluated A-3PO on Qwen3-8B-Base over three seeds, including the run in the paper. A-3PO collapses in one of the three seeds, whereas GMC-GRPO remains stable in all of our runs. When A-3PO does not collapse, GMC-GRPO’s greedy MATH500 accuracy (78.80%) is quite comparable to that of A-3PO (79.60% and 78.40%).

The risk of collapse comes from the design of A-3PO. It clips the ratio between the current policy and a proximal anchor log⁡πprox=αlog⁡πθbehav+(1−α)log⁡πθ\log\pi_{\rm prox}=\alpha\log\pi_{\theta_{\rm behav}}+(1-\alpha)\log\pi_\theta, where α=1/max⁡{1,τ}\alpha=1/\max\{1,\tau\} (Appendix B.2.3). This clipping ratio equals ρtα\rho_t^{\alpha}, where ρt=πθ/πθbehav\rho_t=\pi_\theta/\pi_{\theta_{\rm behav}} is the token-level importance ratio, so clipping is active only when ρt∉[(1−ϵ)τ,(1+ϵ)τ]\rho_t\notin[(1-\epsilon)^{\tau},(1+\epsilon)^{\tau}]. At the average delays in our experiments (about 140–160 updates), this interval spans more than 20 orders of magnitude. A-3PO thus applies the full weight ρt\rho_t to essentially every token and reduces to unclipped token-level IS. Keeping all importance mass explains its fast early progress and its strong result on Qwen3-1.7B, but nothing bounds the update when a ratio becomes extreme. The same prosperity-before-collapse pattern has been reported when clipping is removed or relaxed on stale rollouts (Zheng et al., 2026; Tian et al., 2026). GMC-GRPO instead caps the importance mass of each group at 1+ϵ1+\epsilon, which bounds the second moment for every delay (Proposition 1).

We will add this discussion and the experiment to the revision of our paper.

References:

[1] H. Zheng, J. Zhao, and B. Chen. Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs? ICLR, 2026.

[2] M. Tian, Y. Xie, and C. Wei. How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning. arXiv:2605.17570, 2026.

Reviewer_TGyfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how stale rollouts in async RL training affect the convergence of GRPO style methods. Its key observation is that the raw importance-weighted gradient is always unbiased regardless of the staleness. The source of bias comes from the clipping, rescaling, etc. The authors define a family of weighted estimators that all share a second-moment bound. In this family, The authors derive GMC-GRPO, a method that rescales the importance ratio for a group with a common factor, when that group's average importance ratio goes over a threshold. The authors also provide a convergence bound dependent on the staleness. Experiments on three Qwen models on math benchmarks show GMC-GRPO is more robust than GRPO and TIC-GRPO under large staleness and under simulated length-dependent staleness.

Critical strengths

S1. The paper explains simply why staleness matters: the uncorrected estimator stays unbiased, so staleness hurts only through the bias of the correction. It also puts several existing methods into one shared framework, which makes them easy to compare.

S2. The proposed method is derived and simple. GMC-GRPO follows from maximizing the kept importance mass under the shared constraint, plus the requirement that each group's gradient direction is preserved. It's only a few lines of code.

Critical weaknesses

W1. The theory claims more than it shows. The advantage over TIC-GRPO is a comparison of upper bounds instead of proving GMC converges faster. The claim of "Staleness instead enlarges the spread of the importance ratios" is not supported by either empirical evidence or theoretical proofs. 

W2. All staleness is simulated, no real measurement of wall-clock time or throughput, even though speed is the reason to train with async RL. The setup also assumes every trajectory in a group comes from a fixed policy version. A trajectory can be generated by more than one behavior policies under the partial rollout setting. Systems that update weights while a response is still being generated don't fit this assumption, which is not discussed in the paper.

W3. A-3PO beats GMC-GRPO on Qwen3-1.7B but appears only in the appendix B2. On Qwen3-8B it is reported as collapsed, even though Figure 6 shows it reaching a similar peak reward first. Please include A-3PO in the main body's comparison table, and discuss the tradeoff between learning speed, stability, and best accuracy achieved.

Questions for authors

Q1. The proof bounds the order of the average squared gradient norm, but the experiments show indirect evidence like reward. Do we have any empirical evidence about the order of the average of the squared gradient norm to verify Theorem 1 and 2?

Q2. Assumption 3 assumes stationarity of parameter update on average. I am concerned whether this assumption is too strong. Since the length-dependent delay experiments violate this assumption, which empirical results support this assumption?

7 comments

We thank the reviewer for the valuable comments and suggestions. Please find our response below. If our response has addressed your concern, we would appreciate that you could consider to raise your rating.

W1(a). Bound comparison

We agree that Theorems 1–2 compare upper bounds and do not establish that GMC-GRPO converges faster. Such comparisons are standard for stochastic methods on nonconvex problems, where algorithm-specific lower bounds are rarely available. However, the gap between the two bounds is not an artifact of loose bounding. It comes from the bias bounds in Lemmas 6 and 9: Theorems 1–2 retain their quadratic-KL branches, which take the form ∥b∥22≤K(ϵ)(Dn(b))2\Vert b\Vert_2^2\le K(\epsilon)(D_n^{(b)})^2 with KGMC(ϵ)∝log⁡−2(1+2ϵ)=Θ(ϵ−2)K_{\rm GMC}(\epsilon)\propto\log^{-2}(1+2\epsilon)=\Theta(\epsilon^{-2}) and KTIC(ϵ)∝cϵ−2=Θ(ϵ−4)K_{\rm TIC}(\epsilon)\propto c_\epsilon^{-2}=\Theta(\epsilon^{-4}). Both orders are attained by the exact biases on a simple instance.

Consider a single prompt with T=1T=1, G=2G=2, two responses y∈{0,1}y\in\{0,1\} with reward r(y)=yr(y)=y, and a logistic policy πθ(y=1)=σ(θ)\pi_\theta(y=1)=\sigma(\theta). Let the behavior and current policies select y=1y=1 with probabilities q=1/4q=1/4 and p=1/4+ϵp=1/4+\epsilon, respectively. Then Assumptions 1–2 hold with L=1/4L=1/4 and C=4\mathcal C=4 for every ϵ∈(0,1/4)\epsilon\in(0,1/4). The ratios are ρ(1)=1+4ϵ\rho(1)=1+4\epsilon and ρ(0)=1−43ϵ\rho(0)=1-\frac43\epsilon. A group contributes to the update only if it contains both responses, and then ρˉ=1+43ϵ>1+ϵ\bar\rho=1+\frac43\epsilon>1+\epsilon. TIC-GRPO therefore drops the rewarded response, whereas GMC-GRPO rescales the group by (1+ϵ)/(1+43ϵ)(1+\epsilon)/(1+\frac43\epsilon). A direct computation gives the exact biases bTIC=−34hb_{\rm TIC}=-\frac34h and bGMC=−ϵ3+4ϵhb_{\rm GMC}=-\frac{\epsilon}{3+4\epsilon}h, where h=p(1−p)h=p(1-p) is the true gradient, while Dn(b)=83ϵ2+O(ϵ3)D_n^{(b)}=\frac83\epsilon^2+O(\epsilon^3). Hence bTIC2/(Dn(b))2=Θ(ϵ−4)b_{\rm TIC}^2/(D_n^{(b)})^2=\Theta(\epsilon^{-4}) and bGMC2/(Dn(b))2=Θ(ϵ−2)b_{\rm GMC}^2/(D_n^{(b)})^2=\Theta(\epsilon^{-2}), so neither coefficient can be improved in its dependence on ϵ\epsilon. This is the mechanism described in Section 4.3: TIC-GRPO removes the full contribution of every trajectory above the threshold, whereas GMC-GRPO removes only the excess of the group average.

We will add this example to the final version of our paper.

W1(b). Theoretical and empirical evidence of staleness enlarges the spread of importance ratios

Our statement in Remark 4 that staleness enlarges the spread of the importance ratios follows from results given in the paper, and we will state the argument explicitly in the revision. Conditional on Hn\mathcal H_n, each ratio has mean one for every delay, E[ρn(b,i)∣Hn]=1\mathbb E[\rho_n^{(b,i)}\mid\mathcal H_n]=1, so staleness does not shift the ratios on average. Their conditional variance equals the chi-square divergence χn2,(b)\chi_n^{2,(b)}, and Jensen's inequality gives χn2,(b)≥eDn(b)−1\chi_n^{2,(b)}\ge e^{D_n^{(b)}}-1. A larger mismatch between the current and behavior policies therefore forces a larger spread of the ratios, and this mismatch is governed by the parameter drift accumulated over the delay window: by Lemmas 3 and 7, Dn(b)≤TL2∥θn−θn−τn(b)∥22D_n^{(b)}\le\frac{TL}{2}\Vert\theta_n-\theta_{n-\tau_n^{(b)}}\Vert_2^2 and E[Dn(b)∣τn(b)]=O(η2(τn(b))2)\mathbb E[D_n^{(b)}\mid\tau_n^{(b)}]=O(\eta^2(\tau_n^{(b)})^2).

Table R7 confirms this in our fixed-delay runs. As the delay grows from 32 to 512, the 99th percentile of the ratios grows by a factor of 1.9–2.7, and the chi-square divergence grows by a factor of 4–9. Since the ratios keep conditional mean one, this growth reflects a heavier right tail rather than a shift in level, which is precisely the regime where a second-moment correction is needed. We will add Table R7 to the appendix.

Table R7. Raw trajectory-level importance ratios, before any correction, in the fixed-delay runs on Qwen3-1.7B-Base. The chi-square divergence is estimated by n∑iρi2/(∑iρi)2−1n\sum_i\rho_i^2/(\sum_i\rho_i)^2-1.

DelayGMC-GRPO 99th pct. ρGMC-GRPO χ²TIC-GRPO 99th pct. ρTIC-GRPO χ²
325.00.676.20.94
645.00.668.11.50
1286.61.29.62.3
25612.65.611.24.3
51213.76.111.73.8

W2. Real asynchronous systems, prompts generated by mixed polices

Due to the lack of access to the distributed hardware platform, our experiments therefore focuses on the optimization side of asynchronous training: the throughput gain comes from decoupling generation from optimization, which existing systems already provide (Noukhovitch et al., 2025; Fu et al., 2025), and we study how to optimize reliably under the staleness that this decoupling introduces. GMC-GRPO only changes the weights in the loss. Given the stored behavior log-probabilities, it computes one average ratio and one scaling factor per group (Algorithm 2, lines 16–19), and it requires no additional forward or backward pass compared with TIC-GRPO. Its throughput is therefore that of the underlying system.

Regarding multiple behavior policies under the partial rollout setting, our analysis does not require a single behavior version per trajectory or per group as we will justify below. Suppose that the snapshot θst\theta_{s_t} used to generate token tt is determined before that token is sampled, as in systems that update weights between decoding steps. The trajectory is then drawn from μ(y∣x)=∏tπθst(yt∣x,y<t)\mu(y\mid x)=\prod_{t}\pi_{\theta_{s_t}}(y_t\mid x,y_{<t}), and computing ρ\rho from the behavior log-probabilities recorded during generation gives exactly πθn(y∣x)/μ(y∣x)\pi_{\theta_n}(y\mid x)/\mu(y\mid x). Our proofs use the behavior policy only through this change of measure and the conditional independence of the responses within a group, so all results continue to hold, with Dn(b)D_n^{(b)} and χn2,(b)\chi_n^{2,(b)} replaced by their averages over the responses of the group. Moreover, applying the token-wise argument of Lemma 3 with a different snapshot at each position gives DKL(πθn∥μ)≤L2 Ey∼πθn∑t∥θn−θst∥22D_{\rm KL}(\pi_{\theta_n}\Vert\mu)\le\frac L2\,\mathbb E_{y\sim\pi_{\theta_n}}\sum_{t}\Vert\theta_n-\theta_{s_t}\Vert_2^2. The relevant delay is thus the per-token delay, which is at most the delay of the launch version, so partial rollouts can only reduce the delay that enters our bounds. The one requirement is that the recorded log-probabilities come from the snapshot that actually generated each token; if a system recomputes them under a single version, the ratio is no longer a likelihood ratio and falls outside our analysis. We will add this discussion to the final version of our paper.

References:

[1] M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. ICLR, 2025.

[2] W. Fu, J. Gao, X. Shen, C. Zhu, Z. Mei, C. He, S. Xu, G. Wei, J. Mei, J. Wang, T. Yang, B. Yuan, and Y. Wu. AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. NeurIPS, 2025.

W3. Discussion about the tradeoff between learning speed, stability, and best accuracy achieved.

We agree that A-3PO belongs to the main comparison, and we will move it from Appendix B.2.3 into Table 1 in the revision. We also evaluated A-3PO on Qwen3-8B-Base over three seeds, including the run in the paper. A-3PO collapses in one of the three seeds, whereas GMC-GRPO remains stable in all of our runs. When A-3PO does not collapse, GMC-GRPO’s greedy MATH500 accuracy (78.80%) is quite comparable to that of A-3PO (79.60% and 78.40%).

The risk of collapse comes from the design of A-3PO. It clips the ratio between the current policy and a proximal anchor log⁡πprox=αlog⁡πθbehav+(1−α)log⁡πθ\log\pi_{\rm prox}=\alpha\log\pi_{\theta_{\rm behav}}+(1-\alpha)\log\pi_\theta, where α=1/max⁡{1,τ}\alpha=1/\max\{1,\tau\} (Appendix B.2.3). This clipping ratio equals ρtα\rho_t^{\alpha}, where ρt=πθ/πθbehav\rho_t=\pi_\theta/\pi_{\theta_{\rm behav}} is the token-level importance ratio, so clipping is active only when ρt∉[(1−ϵ)τ,(1+ϵ)τ]\rho_t\notin[(1-\epsilon)^{\tau},(1+\epsilon)^{\tau}]. At the average delays in our experiments (about 140–160 updates), this interval spans more than 20 orders of magnitude. A-3PO thus applies the full weight ρt\rho_t to essentially every token and reduces to unclipped token-level IS. Keeping all importance mass explains its fast early progress and its strong result on Qwen3-1.7B, but nothing bounds the update when a ratio becomes extreme. The same prosperity-before-collapse pattern has been reported when clipping is removed or relaxed on stale rollouts (Zheng et al., 2026; Tian et al., 2026). GMC-GRPO instead caps the importance mass of each group at 1+ϵ1+\epsilon, which bounds the second moment for every delay (Proposition 1).

The trade-off between learning speed, stability and best accuracy is therefore governed by how much importance mass a correction retains and whether it bounds the update. TIC-GRPO bounds the update by discarding mass, which keeps training stable but biases and shrinks the update. A-3PO retains all mass without a bound, which accelerates early learning and can yield the best accuracy, as on Qwen3-1.7B, but leaves training unstable even exposed to collapse. GMC-GRPO retains as much mass as the cap 1+ϵ1+\epsilon allows (eq. (8)), and thus combines the stability of TIC-GRPO with most of the learning signal of A-3PO. Its accuracy is also not uniformly below that of A-3PO: on Qwen3-4B, GMC-GRPO is better in both mean greedy and mean avg@n accuracy (60.63/56.20 vs. 56.47/54.65, Table 5). Since the clipping interval of A-3PO widens with the delay, A-3PO remains an effective choice when delays are short, whereas GMC-GRPO is preferable when delays are long or a failed run is costly. Finally, the peak reward of A-3PO in Figure 6 is a training reward reached before the collapse around step 430. Exploiting it would require stopping training before a collapse whose timing is not known in advance.

References:

[1] H. Zheng, J. Zhao, and B. Chen. Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs? ICLR, 2026.

[2] M. Tian, Y. Xie, and C. Wei. How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning. arXiv:2605.17570, 2026.

Reviewer_qyE3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper replaces conventional trajectory-wise importance ratio clipping with group-level uniform rescaling, reducing correction-induced bias while maintaining gradient stability, thereby improving the utilization of stale rollouts in asynchronous GRPO training.

Critical strengths
  1. Innovative Group-Level Importance Sampling Correction (GMC-GRPO): Compared with conventional trajectory-wise clipping, GMC preserves more training signals through group-level uniform rescaling, reducing correction-induced bias while controlling the gradient's second moment and improving the utilization of stale rollouts.

  2. Rigorous Theoretical Analysis and Experimental Validation: The paper derives convergence bounds for asynchronous GRPO, explicitly characterizes the impact of rollout delays on convergence, and demonstrates the stability and performance advantages of GMC-GRPO under large delays through experiments across different model sizes, delay levels, and group sizes.

Critical weaknesses
  1. Potential Loss of Valuable Training Signals: GMC-GRPO applies uniform rescaling to all trajectories within a group when the average importance ratio exceeds the threshold. This unnecessarily reduces the contributions of trajectories that would not otherwise require clipping, potentially weakening useful gradient updates. The paper lacks direct ablation studies to quantify this negative effect.

  2. Limited Theoretical Optimality and Experimental Validation: GMC-GRPO minimizes an upper bound on correction-induced bias rather than the actual gradient bias. Moreover, uniform rescaling preserves the relative importance weights within a group, allowing trajectories with extreme importance ratios to dominate gradient updates. The paper does not sufficiently investigate whether alternative group-level weighting strategies could achieve better stability and performance.

Questions for authors

Q1.Impact of Uniform Rescaling on Unclipped Trajectories

GMC-GRPO uniformly rescales all trajectories within a group when the average importance ratio exceeds the threshold. However, this also reduces the gradient contributions of trajectories whose individual importance ratios are already within the clipping threshold. Have the authors investigated whether this unintended downweighting negatively affects training performance? An ablation comparing uniform group rescaling with a strategy that preserves the weights of otherwise unclipped trajectories would help clarify whether the benefits outweigh this potential loss of useful training signals.

Q2. Optimality of Group-Level Rescaling

The theoretical analysis establishes that GMC-GRPO minimizes an upper bound on correction-induced bias, rather than the actual gradient bias. Furthermore, uniform rescaling preserves the relative importance weights within each group, potentially allowing trajectories with extreme importance ratios to dominate gradient updates. Could the authors clarify why preserving the group gradient direction is preferable to alternative non-uniform weighting strategies under the same group-level importance-mass constraint? Empirical comparisons of gradient bias, variance, or training performance would help establish whether uniform rescaling is the most effective choice.

5 comments

We thank the reviewer for the thoughtful feedback and positive assessment. Below we address the raised questions.

01 Uniform rescaling: within-threshold responses and extreme ratios

We agree that uniform rescaling also downweights responses whose individual ratios are within the threshold. In practice, this downweighting is mild for most groups. Table R5 reports how strongly GMC-GRPO rescales the groups in the simulated asynchronous runs: although 31.0% of the groups are rescaled, only 7.3% are scaled by a factor of at most 1/2, and only 1.2% by a factor of at most 1/4.

Table R5. Distribution of the GMC scaling factor αn(b),⋆=min⁡{1,(1+ϵ)/ρˉn(b)}\alpha_n^{(b),\star}=\min\{1,(1+\epsilon)/\bar\rho_n^{(b)}\} over all groups (Qwen3-1.7B-Base, simulated asynchronous delays, ϵ=0.28\epsilon=0.28). A factor of at most 1/k1/k corresponds to ρˉn(b)≥k(1+ϵ)\bar\rho_n^{(b)}\ge k(1+\epsilon).

Scaling factorGroup-average ratioFraction of groups
< 1 (rescaled)> 1.2831.0%
≤ 1/2≥ 2.567.3%
≤ 1/4≥ 5.121.2%
≤ 1/8≥ 10.240.4%

Uniform rescaling multiplies the raw importance-sampling update of a group by a single factor, Un,bGMC=αn(b),⋆Un,bISU^{\rm GMC}_{n,b}=\alpha_n^{(b),\star}U^{\rm IS}_{n,b}. GMC-GRPO can thus be viewed as the unbiased IS update with a group-adaptive step size: it keeps the relative contributions of the responses to the current-policy gradient and never reverses the sign of the IS update along any direction. A large ratio means that the current policy assigns the response a much higher probability than the behavior policy did, so its large weight correctly reflects its contribution to the current-policy gradient. However, the group update remains bounded: its norm is at most 1+ϵ1+\epsilon times that of its largest per-response term T−1∇log⁡πθn(y(b,i)∣x(b))A^(b,i)T^{-1}\nabla\log\pi_{\theta_n}(y^{(b,i)}\mid x^{(b)})\widehat A^{(b,i)}. In contrast, a rule that trims only high ratios shifts the relative weight toward responses that the behavior policy over-represents. It reduces the variance within the group, but it changes the direction of the group update. Since neither rule dominates the other in exact bias, we compare them empirically below.

02 Ablation: uniform rescaling versus trimming high ratios

As suggested, we compare GMC-GRPO with a variant that preserves the weights of within-threshold responses and trims only high ratios. When ρˉn(b)>1+ϵ\bar\rho_n^{(b)}>1+\epsilon, it sets wn(b,i)ρn(b,i)=min⁡{ρn(b,i),λn(b)}w_n^{(b,i)}\rho_n^{(b,i)}=\min\{\rho_n^{(b,i)},\lambda_n^{(b)}\}, where λn(b)\lambda_n^{(b)} is chosen so that the average weighted ratio of the group equals 1+ϵ1+\epsilon. The two rules remove the same importance mass, (ρˉn(b)−(1+ϵ))+(\bar\rho_n^{(b)}-(1+\epsilon))_+, and differ only in where the mass is removed. The trimming variant therefore belongs to the class in Definition 1 and also solves eq. (8), so Proposition 1, Theorem 1 and Corollaries 1–2 hold for it without modification.

Table R6. MATH500 results (%) on Qwen3-1.7B-Base under fixed delays DD. Pass@kk is estimated from 16 samples per problem; the TIC-GRPO and uniform-rescaling rows are taken from Table 3. The best result for each delay is in bold.

DelayMethodGreedyPass@1Pass@8Pass@16
32TIC-GRPO66.4061.7084.2488.80
32GMC-GRPO (uniform rescaling)64.8060.8483.8988.60
32GMC-GRPO (trimming)65.6062.3984.3288.60
128TIC-GRPO63.2055.4182.5987.80
128GMC-GRPO (uniform rescaling)68.4059.0383.4588.00
128GMC-GRPO (trimming)63.8060.0583.1987.40
512TIC-GRPO53.8038.1772.4780.60
512GMC-GRPO (uniform rescaling)56.4042.0176.9183.40
512GMC-GRPO (trimming)61.0043.3179.1386.20

The trimming variant attains a slightly higher sampled Pass@1 at every delay (by 1.0–1.6 points), while greedy accuracy and Pass@k are mixed across delays. This supports the reviewer's intuition that preserving within-threshold weights can help. At the same time, both group-level rules improve the sampled Pass@1 of TIC-GRPO by 3.6–5.1 points at D∈{128,512}D\in\{128,512\}, which indicates that the gain comes mainly from capping the group-level importance mass rather than from the particular way the mass is removed. We will add this variant and the discussion above to the final version of our paper.

03 Optimization over the bias upper bound

We agree that GMC-GRPO minimizes the ratio-based bound in eq. (7) rather than the exact bias. However, no implementable weighting rule can minimize the exact bias. To see this, fix a group and minimize ∥bn,bw∥22\Vert b^w_{n,b}\Vert_2^2 over all weighting rules satisfying eq. (5). Since bn,bwb^w_{n,b} is affine in ww and the constraints are convex, this is a convex problem, and its first-order optimality conditions decouple across realized groups. These optimality conditions show that the optimal weights are of threshold form, wi⋆=1κi>μw_i^\star=\mathbf 1\\{\kappa_i>\mu\\} up to ties, where κi=⟨−b⋆,∇log⁡πθn(y(b,i)∣x(b))A^(b,i)⟩\kappa_i=\langle -b^\star,\nabla\log\pi_{\theta_n}(y^{(b,i)}\mid x^{(b)})\widehat A^{(b,i)}\rangle, b⋆b^\star is the optimal bias, and μ≥0\mu\ge0 is the multiplier of the mass constraint, which is zero when the constraint is inactive.

This rule cannot be implemented for two reasons. First, b⋆=E[Un,bw⋆∣Hn]−T−1∇Jx(b)(θn)b^\star=\mathbb E[U^{w^\star}_{n,b}\mid\mathcal H_n]-T^{-1}\nabla J_{x^{(b)}}(\theta_n) involves the unknown policy gradient that the update is meant to estimate. It also depends on w⋆w^\star itself, so w⋆w^\star is defined only implicitly through a fixed point. Second, b⋆b^\star is an expectation over the distribution of the group, whereas the learner observes only a single group per prompt.

The ratio-based bound is the natural target for rules that cannot observe these directions. For any admissible ww and any M>0M>0, sup⁡∥vi∥2≤M∥1G∑i=1G(1−wi)ρn(b,i)vi∥2=M⋅1G∑i=1G(1−wi)ρn(b,i)\sup_{\Vert v_i\Vert_2\le M}\Vert\frac1G\sum_{i=1}^G(1-w_i)\rho_n^{(b,i)}v_i\Vert_2=M\cdot\frac1G\sum_{i=1}^G(1-w_i)\rho_n^{(b,i)}, attained when all viv_i point in the same direction with norm MM. With vi=T−1∇log⁡πθn(y(b,i)∣x(b))A^(b,i)v_i=T^{-1}\nabla\log\pi_{\theta_n}(y^{(b,i)}\mid x^{(b)})\widehat A^{(b,i)} and M=2R2Llog⁡CM=2R\sqrt{2L\log\mathcal C} from eq. (15), the right-hand side is exactly the bound in eq. (7). Minimizing this bound therefore minimizes the worst-case bias over all score–advantage vectors satisfying the norm bound of eq. (15), and the bound is attained rather than loose. Its minimizers are exactly the rules that retain the mass min⁡{ρˉn(b),1+ϵ}\min\{\bar\rho_n^{(b)},1+\epsilon\}, among which GMC-GRPO is the unique direction-preserving one (Proposition 2); we compare it with the other minimizers below. We will add this analysis to the final version of our paper.

We hope that our response above has addressed your concern, and we would appreciate that you could consider to raise your rating if your concern has been resolved.

Dear Reviewer qyE3,

As the author–reviewer discussion period is approaching its end, could you please check our responses and let us know if you have further questions? If our response has resolved your concerns, we would be grateful if you could kindly consider to raise your rating.

Thank you very much for your thoughful review and your effort.

Reviewer_7CptOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the convergence of asynchronous GRPO-style algorithms under stale rollouts. The authors identify a fundamental tradeoff between controlling the second moment of importance-weighted gradient estimators and minimizing the bias introduced by clipping. To address this issue, they propose Group Mass Capping GRPO (GMC-GRPO), which caps the average importance ratio at the group level rather than clipping individual trajectories. The authors show that GMC-GRPO minimizes a ratio-based bias upper bound within a class of weighted gradient estimators under a common second-moment guarantee. They further establish convergence guarantees with explicit dependence on rollout delays and show improved theoretical delay dependence compared with TIC-GRPO. Experiments on Qwen3 models ranging from 1.7B to 8B demonstrate improved robustness under fixed and simulated asynchronous delays.

Critical strengths
  • The paper provides a useful theoretical perspective on asynchronous GRPO. Rather than treating rollout staleness solely as an empirical stability issue, the authors explicitly connect the gradient estimator's second moment, correction bias, and rollout delay to convergence.
  • The proposed group-level correction is simple and well-motivated.
  • The paper is generally well-written, and the motivation and methodology are easy to follow.
Critical weaknesses
  • The maximum response length is too short to adequately evaluate training stability in realistic asynchronous RL settings. All experiments use a maximum response length of only 4,096 tokens. However, in practical reasoning RL training, instability often becomes more pronounced with longer responses, where trajectory-level importance ratios can become increasingly extreme and the mismatch between the behavior and current policies can be amplified.
  • The experimental evaluation omits several important existing methods, like M2PO and GSPO.
  • All experiments are conducted on relatively small models, ranging from Qwen3-1.7B to Qwen3-8B, and are restricted to math reasoning tasks. It would be good to include setting like coding or other agentic tasks and larger models to prove the generalizability of the proposed methods.
Questions for authors

See weakness.

7 comments

We thank the reviewer for the valuable comments and suggestions. Please find our response below. If our response has addressed your concern, we would appreciate that you could consider to raise your rating.

01 Response length and heavy-tailed importance ratios

The 4,096-token limit is not binding in our setting. As illustrated in Table R1 below, across all runs in Table 1 in the paper, only 0.71% of rollouts reach the limit. The per-run mean response length ranges from 315 to 542 tokens, and the per-run 90th percentile from 638 to 947 tokens. A larger limit would affect fewer than 1% of rollouts.

Table R1. Truncation rate and response length of the Qwen3-1.7B runs in Table 1 (simulated asynchronous delays).

ModelMethodTrunc. (%), all stepsTrunc. (%), last 100 stepsMean lengthp90 length
Qwen3-1.7BGRPO0.921.25415896
GAC1.030.62413861
A-3PO1.151.72542947
VCPO0.310.31315638
TIC-GRPO0.460.70401806
GMC-GRPO0.370.31502927

We agree that longer responses increase the dispersion of trajectory-level importance ratios. However, as illustrated in Table R2 below, our experiments already operate in a regime where this dispersion is severe. Over learner updates 400–800 of the Qwen3-1.7B runs, where the median staleness is 170–188 updates, the raw trajectory-level ratios (before any correction) have a median normalized effective sample size (ESS) between 0.20 and 0.30 and a median χ2\chi^2 divergence between 2.9 and 3.5, and a 99th percentile between 7.6 and 8.2, depending on the method. By contrast, at the same staleness the token-level ratios used by GRPO, GAC and A-3PO stay close to one, with an ESS of at least 0.99 and a χ2\chi^2 divergence of at most 0.008. The heavy tail of the trajectory-level ratios therefore comes from accumulating a small per-token mismatch over a few hundred tokens, which is the mechanism the reviewer describes. Our conclusions are therefore drawn in the heavy-tailed regime the reviewer describes, rather than in a mildly off-policy one.

Table R2. Raw importance ratios of each method, before any clipping or rescaling (Qwen3-1.7B). Each entry is the median over learner updates 400–800 (81 logged updates per run). ESS is normalized by the number of samples, (∑iρi)2/(n∑iρi2)(\sum_i\rho_i)^2/(n\sum_i\rho_i^2), and staleness is measured in learner updates.

MethodRatio levelStalenessESSχ²Ratio p99
GRPOtoken1790.9970.00251.19
GACtoken1700.9980.00231.19
A-3POtoken1770.9920.00841.22
VCPOtrajectory1860.2052.86827.57
TIC-GRPOtrajectory1880.2923.21418.06
GMC-GRPOtrajectory1810.3013.47188.18

Further, our theory suggests that the advantage of GMC-GRPO will persist at longer response lengths. To see this, first note that our theory treats response length and staleness in the same way. The delay enters the bounds only through the correction bias, which is controlled by Dn(b)D_n^{(b)}, and Lemma 3 gives Dn(b)≤TL2∥θn−θn−τn(b)∥22D_n^{(b)}\le\frac{TL}{2}\Vert\theta_n-\theta_{n-\tau_n^{(b)}}\Vert_2^2. Longer responses (larger TT) and larger delays (larger ∥θn−θn−τn(b)∥2\Vert\theta_n-\theta_{n-\tau_n^{(b)}}\Vert_2) therefore act on the bias through the same quantity. Moreover, the ratio (cϵ/log⁡(1+2ϵ))2(c_\epsilon/\log(1+2\epsilon))^2 between the delay coefficients of GMC-GRPO and TIC-GRPO in Theorems 1–2 does not depend on TT. For every TT, GMC-GRPO also removes no more importance mass than TIC-GRPO on each sample path (Appendix E). We therefore expect the advantage of GMC-GRPO to persist at longer response lengths.

We will add Tables R1–R2 to Appendix B and discuss response length in the revision of our paper.

02 Baselines M2PO and GSPO

We thank the reviewer for the suggestion. We have added M2PO (Zheng et al., 2026) and GSPO (Zheng et al., 2025) as baselines on Qwen3-1.7B-Base, under both the simulated asynchronous delays of Table 1 and the fixed delay D=512D=512, with all other settings as in Appendix B.1 (Table R3).

Table R3. MATH500 accuracy (%) on Qwen3-1.7B-Base, reported as greedy / avg@16.

GMC-GRPOM2POGSPO
Simulated asynchronous delays66.00/ 57.2965.40 / 62.5049.00 / 32.24
Fixed delay D=512D=51256.40 / 42.01Collapsed48.60 / 27.31

Table R3 shows that GMC-GRPO well outperforms GSPO in both settings. GSPO was designed to stabilize large-scale RL training. Its length-normalized sequence ratio is the geometric mean of the per-token ratios rather than the likelihood ratio between the current and behavior policies. It therefore does not correct the distribution shift introduced by stale rollouts.

Compared to M2PO, GMC-GRPO’s advantage is its robustness to severe staleness. Under simulated delays averaging approximately 160 updates, M2PO is competitive: it matches GMC-GRPO’s greedy accuracy and achieves a higher avg@16. At D=512D=512, however, M2PO training collapses, while GMC-GRPO remains stable. This contrast demonstrates that M2PO’s competitive performance at moderate delays does not extend to the high-staleness regime in which GMC-GRPO continues to train reliably.

To explain why M2PO can collapse, it keeps the batch average of the squared token-level log-ratio (log⁡ρt)2(\log\rho_t)^2 below a fixed threshold τM2\tau_{M_2} by masking the tokens with the largest values. This second moment grows rapidly with the delay: in our D=512D=512 run, it increases by more than four orders of magnitude (from 0.019 to 465) as the delay grows from 1 to 512. With a fixed τM2\tau_{M_2}, M2PO must therefore mask an increasing fraction of the tokens, and eventually almost all of them. The gradient norm then drops from 0.28 to at most 0.002 while the reward signal is still present. In other words, the learning signal is masked out rather than misdirected. The performance of M2PO is therefore sensitive to the choice of τM2\tau_{M_2} relative to the delay. GMC-GRPO instead rescales a group only when its average ratio exceeds 1+ϵ1+\epsilon, and then keeps an importance mass of exactly 1+ϵ1+\epsilon. It never discards the learning signal of a group, while its second moment remains bounded for every delay (Proposition 1).

We will add both baselines to Section 5 and discuss them in the related work.

References:

[1] H. Zheng, J. Zhao, and B. Chen. Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs? ICLR, 2026.

[2] C. Zheng, S. Liu, M. Li, X.-H. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin. Group Sequence Policy Optimization. arXiv:2507.18071, 2025.

03 Coding/agentic tasks or larger models

We thank the reviewer for this suggestion. To examine whether the benefit of GMC-GRPO extends beyond mathematical reasoning, we trained Qwen3-1.7B-Base on 8,000 Python problems from KodCode, under the same simulated asynchronous protocol as Table 1. We evaluate on HumanEval, the MBPP test split and the easy tier of LiveCodeBench v6 (Table R4).

Table R4. Coding results (%) of Qwen3-1.7B-Base under simulated asynchronous delays, reported as greedy / avg@8. Avg. is the mean over the three benchmarks. In each column, the best greedy and avg@8 results are in bold.

MethodHumanEvalMBPP (test)LiveCodeBench v6 (easy)Avg.
Base53.05 / 21.8043.69 / 23.4532.56 / 5.8143.10 / 17.02
GRPO60.98 / 34.3848.70 / 30.5432.56 / 9.5947.41 / 24.84
GAC60.98 / 36.2046.89 / 30.0634.88 / 11.6347.58 / 25.96
TIC-GRPO64.02 / 48.2547.70 / 39.0032.56 / 13.6648.09 / 33.64
VCPO63.41 / 50.6948.70 / 40.0832.56 / 11.3448.22 / 34.04
GMC-GRPO67.68 / 52.9049.30 / 38.9837.21 / 13.0851.40 / 34.99

The results are consistent with our math experiments. GMC-GRPO achieves the best greedy accuracy on all three benchmarks and the best avg@8 on HumanEval; on MBPP and LiveCodeBench, its avg@8 is within 1.1 and 0.6 points of the best method, respectively. Averaged over the three benchmarks, GMC-GRPO attains the highest greedy accuracy and avg@8 (51.40 / 34.99). The trajectory-level corrections (TIC-GRPO, VCPO and GMC-GRPO) also improve avg@8 over GRPO and GAC, by 12.1–18.5 points on HumanEval and 8.4–10.0 points on MBPP.

Regarding model scale, our experiments span Qwen3-1.7B, 4B and 8B, and GMC-GRPO attains the highest mean avg@n among the methods in Table 1 at all three sizes. Training larger models under the same asynchronous protocol exceeds our current computational budget. Our analysis does not depend on a particular model size, and we will discuss larger models and agentic tasks as a limitation and a direction for future work.

We hope that our response above has addressed your concern, and we would appreciate that you could consider to raise your rating. Thanks again for your time and effort.

Comments on the paper

1 comment

Dear Reviewers,

I hope you're doing well.

Please respond to the authors' rebuttal if you have not already done so.

Thank you for your valuable time and engagement.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.