- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies GRPO-style RL post-training of LLMs on stale asynchronous rollouts. Trajectory-level importance-weighted updates stay conditionally unbiased regardless of delay, so staleness enters only through the bias of the variance-control correction. Minimizing a bias envelope under a cap on the group-mean importance ratio yields GMC-GRPO (capped self-normalized importance sampling) with a delay-explicit convergence bound whose delay coefficient scales as ε⁻² rather than TIC-GRPO's ε⁻⁴. Experiments on three Qwen3-Base sizes show GMC-GRPO matching or beating TIC-GRPO under stale rollouts.
Critical strengths
- Clean, correct delay-explicit convergence analysis. Theorem 3 appears to be the first bound for GRPO-style trajectory-IS methods in which stochastic rollout delay appears explicitly, with a transparent removed-mass bias mechanism (Proposition 1).
- Consistent empirical robustness across sizes and delays. GMC-GRPO never trails TIC-GRPO on macro-avg@n in Table 1, clearly beats vanilla GRPO, GAC and VCPO, and leads TIC at delays 128–512.
Critical weaknesses
- ε⁻⁴→ε⁻² is bound-vs-bound; the realized variance cost is unmeasured. The shared envelope of Lemmas 5 and 8 ignores averaging over G and the mass that shrinks TIC's second moment. With G=8 and ε=0.28, one trajectory at ρ=50 plus seven near zero gives GMC roughly 13× the on-policy second moment while TIC's update is ≈0; no ρ̂ or second-moment statistics are reported.
- Single-run experiments with greedy margins inside noise. No seeds or variance are reported. In Table 1 the greedy macro-average margin of GMC over TIC is at most +1.32 and negative (−0.83) at 4B, so only the avg@n margins plausibly exceed run-to-run noise.
- "Best among stable baselines" depends on excluding A-3PO. A-3PO beats GMC at 1.7B by 3.8 avg@n points (45.27 vs 41.47, Table 5) yet is relegated to an appendix after one collapsed 8B run, so the abstract's framing holds at 1.7B only by this exclusion.
Questions for authors
- For D = 128 and 512, report ρ and ρ̂ distributions and per-update gradient norms for GMC versus TIC.
- Was ε tuned for TIC-GRPO, and can you sweep ε for both methods at delays 128 and 512?
- How many seeds underlie Tables 1 and 3–5, and does A-3PO's 8B collapse reproduce across seeds?
- State in the main text that at D = 512 only the last 288 of 800 updates run at steady-state delay.
6 comments
02 Realized second moment of the importance weights
We agree that the shared envelope of Lemmas 5 and 8 is a worst-case bound: it does not exploit averaging over , and the mass that TIC-GRPO discards can make its realized second moment smaller than that of GMC-GRPO. The reviewer's example is the extreme case of this asymmetry. Since and (Proposition 3), we have , with equality when a single trajectory carries the whole capped mass. Even then, the group update of GMC-GRPO equals times that trajectory's score–advantage term, compared with times for the raw IS update, and in general GMC-GRPO never amplifies the raw update since . The update of TIC-GRPO is close to zero in this example only because it discards the response that the current policy favors by a factor of 50, which is the bias that Theorems 1–2 account for. This trade-off favors GMC-GRPO under large delays. The delay enters our bounds only through the correction bias, whereas the second moments of both methods share the same delay-independent envelope (Lemmas 5 and 8). As the delay grows, the key is therefore to minimize the bias while keeping the second moment controlled, which is exactly what the group mass cap of GMC-GRPO does.
Empirically, the realized second moment of GMC-GRPO remains moderate. Table R9 reports the mean and mean square of the effective importance weights, for GMC-GRPO and for TIC-GRPO, in the fixed-delay runs; on-policy weights have mean and mean square equal to one. GMC-GRPO keeps every trajectory and retains about 90% of the importance mass, while its mean squared weight of 2.6–2.8 stays far below the worst-case value of 13.1 in the reviewer’s example. TIC-GRPO has a smaller mean squared weight, but only because it discards every trajectory whose ratio exceeds . These trajectories carry most of the importance mass, so TIC-GRPO retains only about 30% of it, and its average update weight is roughly one-third that of GMC-GRPO. The simulated asynchronous runs (3 seeds, updates 400–800) show the same pattern: the batch average of has median 2.57, compared with 5.51 for the raw ratios, and TIC-GRPO assigns zero weight to a median of 25% of the trajectories per update.
TIC-GRPO’s smaller second moment therefore comes at the cost of substantial update attenuation. GMC-GRPO’s better preservation of importance mass helps explain its stronger performance, reported in the last column of Table R9.
Table R9. Effective importance weights in the fixed-delay runs with on Qwen3-1.7B-Base, computed over the updates at steady-state delay (672 updates for and 288 for ). On-policy weights have mean and mean square equal to one. MATH500 accuracy (%) is reported as greedy / sampled Pass@1 (16 samples per problem) at the final checkpoint.
| Delay D | Method | Mean | Mean square | MATH500 (greedy / avg@16) |
|---|---|---|---|---|
| 128 | GMC-GRPO | 0.909 | 2.561 | 68.40 / 59.03 |
| 128 | TIC-GRPO | 0.297 | 0.216 | 63.20 / 55.41 |
| 512 | GMC-GRPO | 0.885 | 2.751 | 56.40 / 42.01 |
| 512 | TIC-GRPO | 0.332 | 0.245 | 53.80 / 38.17 |
We will add this discussion and report these statistics in the revision.
03 Multiple seed experiment
We agree that single-run greedy margins can fall within run-to-run noise. We have therefore rerun the simulated asynchronous experiment on Qwen3-1.7B-Base with three seeds for all methods in Table 1 (Table R10). GMC-GRPO attains the best avg@n on every benchmark, and its average exceeds that of TIC-GRPO by 4.5 points (41.67 ± 0.37 vs. 37.21 ± 0.59), well beyond the variation across seeds. In the revision, we will report means and standard deviations over three seeds and base our comparisons primarily on avg@n, which averages over 8–16 samples per problem.
Table R10. avg@n (%) under simulated asynchronous delays on Qwen3-1.7B-Base, mean ± standard deviation over three seeds. Avg. Delay is measured in learner updates.
| Method | Avg. Delay | MATH500 | GSM8K | Gaokao | Olympiad | Minerva | AMC23 | Avg. perf. |
|---|---|---|---|---|---|---|---|---|
| GRPO | 158.56 ± 0.75 | 31.36 ± 0.60 | 47.99 ± 1.15 | 26.04 ± 0.42 | 6.74 ± 0.23 | 8.71 ± 0.46 | 9.79 ± 0.74 | 21.77 ± 0.56 |
| GAC | 158.82 ± 1.09 | 31.62 ± 0.57 | 48.07 ± 0.99 | 25.92 ± 0.92 | 6.76 ± 0.10 | 8.79 ± 0.72 | 10.21 ± 1.04 | 21.90 ± 0.52 |
| VCPO | 158.42 ± 0.79 | 47.27 ± 6.32 | 64.73 ± 5.04 | 39.87 ± 4.62 | 15.47 ± 4.37 | 16.28 ± 1.99 | 23.23 ± 3.10 | 34.48 ± 3.99 |
| TIC-GRPO | 158.97 ± 0.91 | 52.49 ± 0.99 | 67.99 ± 0.75 | 41.76 ± 1.13 | 18.77 ± 0.76 | 17.37 ± 0.30 | 24.90 ± 1.50 | 37.21 ± 0.59 |
| GMC-GRPO | 160.06 ± 0.27 | 57.63 ± 0.50 | 73.04 ± 1.39 | 47.73 ± 0.49 | 22.28 ± 0.68 | 20.21 ± 0.10 | 29.11 ± 0.59 | 41.67 ± 0.37 |
04 Seeds and the stability of A-3PO
Tables 1 and 3–5 in the submission each report a single run. In addition to the three-seed rerun of Table 1 on Qwen3-1.7B-Base, we evaluated A-3PO on Qwen3-8B-Base over three seeds, including the run in the paper. A-3PO collapses in one of the three seeds, whereas GMC-GRPO remains stable in all of our runs. When A-3PO does not collapse, GMC-GRPO’s greedy MATH500 accuracy (78.80%) is quite comparable to that of A-3PO (79.60% and 78.40%).
The risk of collapse comes from the design of A-3PO. It clips the ratio between the current policy and a proximal anchor , where (Appendix B.2.3). This clipping ratio equals , where is the token-level importance ratio, so clipping is active only when . At the average delays in our experiments (about 140–160 updates), this interval spans more than 20 orders of magnitude. A-3PO thus applies the full weight to essentially every token and reduces to unclipped token-level IS. Keeping all importance mass explains its fast early progress and its strong result on Qwen3-1.7B, but nothing bounds the update when a ratio becomes extreme. The same prosperity-before-collapse pattern has been reported when clipping is removed or relaxed on stale rollouts (Zheng et al., 2026; Tian et al., 2026). GMC-GRPO instead caps the importance mass of each group at , which bounds the second moment for every delay (Proposition 1).
We will add this discussion and the experiment to the revision of our paper.
References:
[1] H. Zheng, J. Zhao, and B. Chen. Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs? ICLR, 2026.
[2] M. Tian, Y. Xie, and C. Wei. How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning. arXiv:2605.17570, 2026.
We thank the reviewer for the thoughtful feedback and positive assessment. Below we address the raised questions.
01 Bound comparison
We agree that Theorems 1–2 compare upper bounds and do not establish that GMC-GRPO converges faster. Such comparisons are standard for stochastic methods on nonconvex problems, where algorithm-specific lower bounds are rarely available. However, the gap between the two bounds is not an artifact of loose bounding. It comes from the bias bounds in Lemmas 6 and 9: Theorems 1–2 retain their quadratic-KL branches, which take the form ∥b∥22≤K(ϵ)(Dn(b))2 with KGMC(ϵ)∝log−2(1+2ϵ)=Θ(ϵ−2) and KTIC(ϵ)∝cϵ−2=Θ(ϵ−4). Both orders are attained by the exact biases on a simple instance.
Consider a single prompt with T=1, G=2, two responses y∈{0,1} with reward r(y)=y, and a logistic policy πθ(y=1)=σ(θ). Let the behavior and current policies select y=1 with probabilities q=1/4 and p=1/4+ϵ, respectively. Then Assumptions 1–2 hold with L=1/4 and C=4 for every ϵ∈(0,1/4). The ratios are ρ(1)=1+4ϵ and ρ(0)=1−34ϵ. A group contributes to the update only if it contains both responses, and then ρˉ=1+34ϵ>1+ϵ. TIC-GRPO therefore drops the rewarded response, whereas GMC-GRPO rescales the group by (1+ϵ)/(1+34ϵ). A direct computation gives the exact biases bTIC=−43h and bGMC=−3+4ϵϵh, where h=p(1−p) is the true gradient, while Dn(b)=38ϵ2+O(ϵ3). Hence bTIC2/(Dn(b))2=Θ(ϵ−4) and bGMC2/(Dn(b))2=Θ(ϵ−2), so neither coefficient can be improved in its dependence on ϵ. This is the mechanism described in Section 4.3: TIC-GRPO removes the full contribution of every trajectory above the threshold, whereas GMC-GRPO removes only the excess of the group average.
We will add this example to the final version of our paper.