- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies asynchronous RL post-training of LLMs on rollouts from a stale behavior policy. Existing corrections reweight only the actor's ratio while the critic target and GAE advantage still bootstrap on behavior-policy continuations ("advantage staleness"); COPC instead reweights TD residuals with a two-sided clipped token ratio and couples the actor-side and advantage-side correction strengths, motivated by a bias/variance decomposition. On tool-integrated math and search, COPC beats AReaL, 50.29 versus 48.26 on math.
Critical strengths
- Cleanly isolated advantage-side effect. The "COPC w/o RC" ablation (η_A=0) is the right control: +2.18 average on math and +3.77 on search (Tables 1–2), and its training score collapses late, COPC's does not (Fig. 2).
- Real interaction in the joint sweep. In Fig. 1 (s=8), raising η_A from 0.25 to 0.75 changes AIME 2025 avg@32 by +4.06 at η_P=0.25 but −2.16 at η_P=0.75, a sign reversal no additive model produces (single seed).
Critical weaknesses
- Single runs; hyperparameters apparently test-set-selected; baselines untuned. No seeds or standard deviations are reported. Table 1's AIME 2025 entry (53.58) is exactly the (η_P, η_A)=(0.5, 0.5), s=8 cell of Fig. 1, the best of nine configurations, while baselines use untuned defaults; at s=8 five of nine COPC cells fall below untuned AReaL (51.25), undercutting the claimed "broad high-performing region".
- The 64-step-staleness claim has no baselines. Gains "persist at 64-step policy staleness" (Abstract) only in Fig. 2 (right), which shows COPC alone; no baseline runs above s=8 on math or s=14 on search, so relative persistence is untested.
- The V-trace comparison does not test COPC's distinguishing choices. COPC's advantage side differs from V-trace only in the lower clip at 1−ε_ℓ and in replacing the Π c_i products by λ=1, neither ablated; the sole V-trace baseline sits outside the shared recipe (no advantage normalization, value clipping or trust region, App. C.2.1), so two-sided clipping is never tested against one-sided truncation like-for-like.
Questions for authors
- How many seeds underlie Tables 1–2 and Figs. 1–2, and was AIME 2025 used to choose (η_P, η_A)?
- Which checkpoint do Tables 1–2 report, given that SAO and AReaL collapse near step 400 in Fig. 2 yet AReaL is the best search baseline?
- Within the COPC recipe, ablate one-sided truncation min(1, ρ) with and without Π c_i products, and plain GAE with λ=0.95/0.99.
4 comments
Q2: Checkpoint selection for Tables 1–2.
Tables 1–2 do not necessarily report the final training checkpoint. For runs without late-stage performance degradation, we report the final saved checkpoint. For runs exhibiting performance collapse, we report the best-performing checkpoint among those saved before the collapse. This explains why SAO and AReaL retain competitive performance in Tables 1–2 despite the late-stage degradation observed during training.
Q3: Ablations of one-sided truncation, trace products, and GAE decay.
We conducted the requested ablations under the same COPC training recipe to examine the effects of one-sided TD-residual weighting, cumulative trace products, and the GAE parameter . As shown below, removing V-trace's cumulative trace products improves average performance by 0.85 and 1.20 points on mathematical reasoning and search, respectively, suggesting that attenuating the contributions of distant TD residuals can hinder long-horizon credit assignment. The resulting variant still trails full COPC by 2.08 and 1.88 points, respectively, supporting the additional benefit of two-sided importance-ratio clipping over one-sided truncation when weighting TD residuals.
We also evaluate standard GAE with under COPC w/o RC. Reducing consistently degrades performance on both tasks, particularly at , suggesting that longer-horizon, Monte Carlo-style estimates are more effective than shorter-horizon bootstrapped estimates in our setting. Together, these results support COPC's design choices of avoiding cumulative trace products and adopting two-sided TD-residual weighting, alongside its use of for long-horizon RLVR.
The corresponding ablation results are summarized below, with detailed per-dataset results in Tables 6–7 and further analysis in Appendix D (Effects of Trace Decay and Trace Coefficients). Here, COPC w/ VT and its w/o TC variant denote the V-trace-based ablations with and without cumulative trace products, respectively. COPC w/o RC uses standard GAE with .
| Method | Math Avg. | Search Avg. |
|---|---|---|
| COPC w/ VT | 47.36 | 41.31 |
| ↳ w/o TC | 48.21 | 42.51 |
| COPC w/o RC | 48.11 | 41.08 |
| ↳ | 45.79 | 40.14 |
| ↳ | 36.53 | 36.51 |
| COPC (ours) | 50.29 | 44.39 |
Thank you for your response! The three concerns in my review are now addressed, and I appreciate how much was added in a short window. I am raising my rating to 3.
Thank you for your positive feedback and for recognizing our revisions. We sincerely appreciate your constructive comments and are glad that our responses and additional experiments have addressed your concerns.
Thank you for the insightful review. Here are our response to your comments.
W1 & Q1: Statistical reliability, hyperparameter selection, and broad performance claims.
Our mathematical reasoning experiments used three random seeds (42, 3407, and 1234). We additionally conducted independent training runs using the same seeds for COPC, COPC w/o RC, SAO, and AReaL on search tasks. COPC achieves the highest average performance on both tasks, reaching 50.29 ± 0.22 and 44.39 ± 1.05, respectively. We have updated Tables 1–2 to report the mean and sample standard deviation for the corresponding three-seed experiments.
We clarify that the default configuration (ηP,ηA)=(0.5,0.5) was fixed before the sensitivity analysis in Figure 1 and was not selected by maximizing AIME 2025 performance. Given the base margins (ϵℓ,ϵh)=(0.4,0.56), this configuration yields effective clipping margins of (0.2,0.28), consistent with commonly used settings in LLM-RL, such as DAPO. Moreover, we also adjusted baseline hyperparameters following established training recipes, including setting PPO's clipping margins to (0.2,0.28).
We acknowledge that not all configurations in Figure 1 outperform AReaL. Nevertheless, four of the nine configurations exceed AReaL (51.25), including (ηP,ηA)=(0.25,0.75) at 52.60. Varying either control from 0.25 to 0.75 changes the effective clipping margins from (0.10,0.14) to (0.30,0.42), spanning absolute differences of (0.20,0.28). For comparison, DAPO increases the upper clipping margin from 0.20 in GRPO to 0.28, a difference of only 0.08. These results suggest that COPC maintains competitive performance over a relatively broad range of correction strengths.
W2: Lack of baseline comparisons at 64-step policy staleness.
To address this concern, we conducted additional experiments with AReaL and COPC w/o RC under 64-step policy staleness (s=64), using the same training settings and evaluation protocol as COPC. As shown below, COPC outperforms both methods across all five mathematical reasoning benchmarks, achieving average improvements of 2.14 and 1.66 points over AReaL and COPC w/o RC, respectively. These results provide direct evidence that COPC's performance advantage over the evaluated baselines persists under substantially increased policy staleness, further supporting the effectiveness of its correction mechanisms.
The corresponding results are summarized below, with detailed per-dataset results in Table 13 and further analysis in Appendix F (Additional Robustness and Generalization Experiments).
W3: V-trace comparison and ablation of distinguishing design choices.
Thank you for highlighting these concerns. We retrained V-trace using the same strengthened training setup as COPC where applicable, including critic pretraining, advantage normalization, and value clipping, while retaining its original actor objective. We also conducted additional ablations to isolate COPC's distinguishing design choices. Specifically, COPC w/ VT replaces COPC's advantage correction with V-trace-style advantages, while its w/o TC variant further removes cumulative trace products but retains V-trace's one-sided TD-residual weight min(1,ρ). All COPC variants use the same training settings.
As shown below, COPC outperforms the retrained V-trace by 3.87 and 2.99 points on mathematical reasoning and search, respectively, showing that its improvements persist under matched training settings. Within the COPC ablations, removing cumulative trace products improves performance by 0.85 and 1.20 points, suggesting that attenuating the contributions of distant TD residuals can hinder long-horizon credit assignment. The w/o TC variant still trails COPC by 2.08 and 1.88 points, respectively. Since both variants omit cumulative trace products, this comparison more directly isolates the effect of replacing V-trace's one-sided truncation with COPC's two-sided importance-ratio clipping when weighting TD residuals. The latter mitigates excessive suppression through a lower bound while controlling large importance weights through an upper bound. Together, these results support the complementary benefits of COPC's two distinguishing design choices.
The corresponding results are summarized below, with detailed per-dataset results in the revised Tables 1–2, additional ablations in Tables 6–7, and further analysis in Appendix D (Effects of Trace Decay and Trace Coefficients).