acceptodds
Under review as a conference paper at ICLR 2027

COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning

Abstract

Asynchronous RL improves throughput in large language model post-training by decoupling rollout generation from optimization, at the cost of training on stale trajectories. Existing methods primarily address token-level policy mismatch through importance-ratio control in the actor objective, providing policy-side correction. We show that this alone is insufficient because advantage estimates inherit mismatch from behavior-policy continuations, a phenomenon we term advantage staleness. This motivates a complementary advantage-side correction, raising the question: how do the two correction channels interact, and how should their strengths be coordinated? We derive exact bias and variance decompositions for a general two-channel actor update, revealing a nonseparable coupling between policy-weight and advantage-estimation errors: their interaction induces multiplicative bias terms, while squared policy weights amplify advantage uncertainty in gradient variance. This coupling motivates a testable hypothesis that the preferred settings of the two correction channels should be coordinated rather than chosen independently. Guided by this analysis, we introduce Coupled Off-Policy Correction (COPC), an actor–critic method combining token-level ratio masking with a dedicated advantage-side correction based on two-sided clipped-ratio weighting of TD residuals for return and advantage estimation. Joint parameter sweeps across staleness levels provide empirical support for this hypothesis: the effect of varying one correction parameter depends on, and can even reverse with, the setting of the other. COPC achieves the highest reported performance on both tool-integrated mathematical reasoning and search, outperforming the strongest reported asynchronous baseline in each setting. COPC further exhibits strong practical robustness, with a broad high-performing parameter region and substantially improved training stability. In particular, in the search setting, it remains stable throughout training, while most evaluated asynchronous baselines exhibit late-stage performance collapse. These performance gains persist at 64-step policy staleness, while COPC incurs minimal step-time overhead over asynchronous PPO and retains a step-time speedup over synchronous PPO.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Off-policy correction, asynchronous LLM-RL

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8dEvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies asynchronous RL post-training of LLMs on rollouts from a stale behavior policy. Existing corrections reweight only the actor's ratio while the critic target and GAE advantage still bootstrap on behavior-policy continuations ("advantage staleness"); COPC instead reweights TD residuals with a two-sided clipped token ratio and couples the actor-side and advantage-side correction strengths, motivated by a bias/variance decomposition. On tool-integrated math and search, COPC beats AReaL, 50.29 versus 48.26 on math.

Critical strengths
  • Cleanly isolated advantage-side effect. The "COPC w/o RC" ablation (η_A=0) is the right control: +2.18 average on math and +3.77 on search (Tables 1–2), and its training score collapses late, COPC's does not (Fig. 2).
  • Real interaction in the joint sweep. In Fig. 1 (s=8), raising η_A from 0.25 to 0.75 changes AIME 2025 avg@32 by +4.06 at η_P=0.25 but −2.16 at η_P=0.75, a sign reversal no additive model produces (single seed).
Critical weaknesses
  • Single runs; hyperparameters apparently test-set-selected; baselines untuned. No seeds or standard deviations are reported. Table 1's AIME 2025 entry (53.58) is exactly the (η_P, η_A)=(0.5, 0.5), s=8 cell of Fig. 1, the best of nine configurations, while baselines use untuned defaults; at s=8 five of nine COPC cells fall below untuned AReaL (51.25), undercutting the claimed "broad high-performing region".
  • The 64-step-staleness claim has no baselines. Gains "persist at 64-step policy staleness" (Abstract) only in Fig. 2 (right), which shows COPC alone; no baseline runs above s=8 on math or s=14 on search, so relative persistence is untested.
  • The V-trace comparison does not test COPC's distinguishing choices. COPC's advantage side differs from V-trace only in the lower clip at 1−ε_ℓ and in replacing the Π c_i products by λ=1, neither ablated; the sole V-trace baseline sits outside the shared recipe (no advantage normalization, value clipping or trust region, App. C.2.1), so two-sided clipping is never tested against one-sided truncation like-for-like.
Questions for authors
  • How many seeds underlie Tables 1–2 and Figs. 1–2, and was AIME 2025 used to choose (η_P, η_A)?
  • Which checkpoint do Tables 1–2 report, given that SAO and AReaL collapse near step 400 in Fig. 2 yet AReaL is the best search baseline?
  • Within the COPC recipe, ablate one-sided truncation min(1, ρ) with and without Π c_i products, and plain GAE with λ=0.95/0.99.
4 comments

Thank you for the insightful review. Here are our response to your comments.

W1 & Q1: Statistical reliability, hyperparameter selection, and broad performance claims.

Our mathematical reasoning experiments used three random seeds (42, 3407, and 1234). We additionally conducted independent training runs using the same seeds for COPC, COPC w/o RC, SAO, and AReaL on search tasks. COPC achieves the highest average performance on both tasks, reaching 50.29 ± 0.22 and 44.39 ± 1.05, respectively. We have updated Tables 1–2 to report the mean and sample standard deviation for the corresponding three-seed experiments.

We clarify that the default configuration (ηP,ηA)=(0.5,0.5)(\eta_P,\eta_A)=(0.5,0.5) was fixed before the sensitivity analysis in Figure 1 and was not selected by maximizing AIME 2025 performance. Given the base margins (ϵℓ,ϵh)=(0.4,0.56)(\epsilon_\ell,\epsilon_h)=(0.4,0.56), this configuration yields effective clipping margins of (0.2,0.28)(0.2,0.28), consistent with commonly used settings in LLM-RL, such as DAPO. Moreover, we also adjusted baseline hyperparameters following established training recipes, including setting PPO's clipping margins to (0.2,0.28)(0.2,0.28).

We acknowledge that not all configurations in Figure 1 outperform AReaL. Nevertheless, four of the nine configurations exceed AReaL (51.25), including (ηP,ηA)=(0.25,0.75)(\eta_P,\eta_A)=(0.25,0.75) at 52.60. Varying either control from 0.25 to 0.75 changes the effective clipping margins from (0.10,0.14)(0.10,0.14) to (0.30,0.42)(0.30,0.42), spanning absolute differences of (0.20,0.28)(0.20,0.28). For comparison, DAPO increases the upper clipping margin from 0.20 in GRPO to 0.28, a difference of only 0.08. These results suggest that COPC maintains competitive performance over a relatively broad range of correction strengths.

W2: Lack of baseline comparisons at 64-step policy staleness.

To address this concern, we conducted additional experiments with AReaL and COPC w/o RC under 64-step policy staleness (s=64s=64), using the same training settings and evaluation protocol as COPC. As shown below, COPC outperforms both methods across all five mathematical reasoning benchmarks, achieving average improvements of 2.14 and 1.66 points over AReaL and COPC w/o RC, respectively. These results provide direct evidence that COPC's performance advantage over the evaluated baselines persists under substantially increased policy staleness, further supporting the effectiveness of its correction mechanisms.

The corresponding results are summarized below, with detailed per-dataset results in Table 13 and further analysis in Appendix F (Additional Robustness and Generalization Experiments).

MethodMath Avg.
AReaL47.07
COPC w/o RC47.55
COPC (ours)49.21

W3: V-trace comparison and ablation of distinguishing design choices.

Thank you for highlighting these concerns. We retrained V-trace using the same strengthened training setup as COPC where applicable, including critic pretraining, advantage normalization, and value clipping, while retaining its original actor objective. We also conducted additional ablations to isolate COPC's distinguishing design choices. Specifically, COPC w/ VT replaces COPC's advantage correction with V-trace-style advantages, while its w/o TC variant further removes cumulative trace products but retains V-trace's one-sided TD-residual weight min⁡(1,ρ)\min(1,\rho). All COPC variants use the same training settings.

As shown below, COPC outperforms the retrained V-trace by 3.87 and 2.99 points on mathematical reasoning and search, respectively, showing that its improvements persist under matched training settings. Within the COPC ablations, removing cumulative trace products improves performance by 0.85 and 1.20 points, suggesting that attenuating the contributions of distant TD residuals can hinder long-horizon credit assignment. The w/o TC variant still trails COPC by 2.08 and 1.88 points, respectively. Since both variants omit cumulative trace products, this comparison more directly isolates the effect of replacing V-trace's one-sided truncation with COPC's two-sided importance-ratio clipping when weighting TD residuals. The latter mitigates excessive suppression through a lower bound while controlling large importance weights through an upper bound. Together, these results support the complementary benefits of COPC's two distinguishing design choices.

The corresponding results are summarized below, with detailed per-dataset results in the revised Tables 1–2, additional ablations in Tables 6–7, and further analysis in Appendix D (Effects of Trace Decay and Trace Coefficients).

MethodMath Avg.Search Avg.
V-trace46.4241.40
COPC w/ VT47.3641.31
↳ w/o TC48.2142.51
COPC (ours)50.2944.39

Q2: Checkpoint selection for Tables 1–2.

Tables 1–2 do not necessarily report the final training checkpoint. For runs without late-stage performance degradation, we report the final saved checkpoint. For runs exhibiting performance collapse, we report the best-performing checkpoint among those saved before the collapse. This explains why SAO and AReaL retain competitive performance in Tables 1–2 despite the late-stage degradation observed during training.

Q3: Ablations of one-sided truncation, trace products, and GAE decay.

We conducted the requested ablations under the same COPC training recipe to examine the effects of one-sided TD-residual weighting, cumulative trace products, and the GAE parameter λ\lambda. As shown below, removing V-trace's cumulative trace products improves average performance by 0.85 and 1.20 points on mathematical reasoning and search, respectively, suggesting that attenuating the contributions of distant TD residuals can hinder long-horizon credit assignment. The resulting variant still trails full COPC by 2.08 and 1.88 points, respectively, supporting the additional benefit of two-sided importance-ratio clipping over one-sided truncation when weighting TD residuals.

We also evaluate standard GAE with λ∈{1,0.99,0.95}\lambda\in\{1,0.99,0.95\} under COPC w/o RC. Reducing λ\lambda consistently degrades performance on both tasks, particularly at λ=0.95\lambda=0.95, suggesting that longer-horizon, Monte Carlo-style estimates are more effective than shorter-horizon bootstrapped estimates in our setting. Together, these results support COPC's design choices of avoiding cumulative trace products and adopting two-sided TD-residual weighting, alongside its use of λ=1\lambda=1 for long-horizon RLVR.

The corresponding ablation results are summarized below, with detailed per-dataset results in Tables 6–7 and further analysis in Appendix D (Effects of Trace Decay and Trace Coefficients). Here, COPC w/ VT and its w/o TC variant denote the V-trace-based ablations with and without cumulative trace products, respectively. COPC w/o RC uses standard GAE with λ=1\lambda=1.

MethodMath Avg.Search Avg.
COPC w/ VT47.3641.31
↳ w/o TC48.2142.51
COPC w/o RC48.1141.08
↳ λ=0.99\lambda=0.9945.7940.14
↳ λ=0.95\lambda=0.9536.5336.51
COPC (ours)50.2944.39

Thank you for your response! The three concerns in my review are now addressed, and I appreciate how much was added in a short window. I am raising my rating to 3.

Thank you for your positive feedback and for recognizing our revisions. We sincerely appreciate your constructive comments and are glad that our responses and additional experiments have addressed your concerns.

Reviewer_CTPCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies advantage correction under the async RL post-training for LLMs. The authors argue that existing methods mainly focus on policy-side correction such as importance ratios in the actor objective, but advantage estimation still depends on the stale behavior policy. To address the advantage staleness, they propose an advantage-side correction that weights TD errors with two-sided clipping ratios, and they show that the corresponding expected return operator has a unique fixed point. Then, they derive bias and variance decompositions for a generic two-channel actor update. The coupling term in bias and variance motivates a coordination hypothesis: stronger correction in one channel should be paired with a weaker correction in the other. The proposed method COPC combines SAO-style token-level ratio masking with advantage-side correction, and trains the critic on the corrected returns. As a result, COPC achieves comparable accuracy to baselines on math and search tasks.

Critical strengths

S1. This paper connects the classical off-policy return correction (Retrace/V-trace) with async RL for LLM post-training. It identifies a real gap.

S2. COPC is simple by changing the way of advantage computations. Very little computational overhead  added. Easy to integrate into existing pipelines.

S3. Fair amount of evaluation. Math and search tasks on both Qwen and Llama models, six baseline methods.

S4. Ablation shows that advantage-side correction improves the accuracy.

Critical weaknesses

W1. The advantage-side correction is very similar to Retrace/V-trace, but the difference is not well discussed. 1) The fixed point in Prop. 1 is similar to V-trace, and the main diff here is two-sided clipping. 2) Eq. 3 uses single-step weight to multiply the TD error, unlike V-trace's product of trace coefficients. 3) Hence, the author should discuss the design choice: single-step weight vs product of trace coefficients, and its implications for the claim of correcting advantage staleness.

W2. V-trace uses ρˉ=cˉ=1\bar{\rho}=\bar{c}=1 without advantage normalization or value clipping. This differs from the strengthened PPO recipe used for COPC (Clip-Higher, critic pretraining, advantage normalization, value clipping), so it is hard to tell whether gains come from the advantage-side correction or from the recipe. A strengthened PPO recipe + V-trace advantages baseline would be the most direct comparison.

Minor point: The notation of ηp\eta_p and ηA\eta_A are abused in the manuscript. In line 201 and 205, they are defined as tuples, but in line 303, they are defined as scalars.

Questions for authors

Q1. Why Eq. 3 does not use a Retrace/V-trace-style product of trace coefficients?

Q2. How would a variant of COPC that replaces Eq. 3 with V-trace style advantages perform?

Q3. Can the advantage-side correction be applied to critic-free methods (e.g., group-relative advantages)?

4 comments

Thank you for the insightful review. Here are our response to your comments.

W1 & Q1: Differences between COPC and Retrace/V-trace in advantage correction.

We agree that COPC shares similarities with Retrace/V-trace, particularly in its fixed-point characterization. The key differences lie in COPC's use of per-step importance weighting without cumulative trace products and two-sided importance-ratio clipping rather than V-trace's one-sided truncation when weighting TD residuals. Per-step weighting accounts for local behavior–target policy mismatch in each TD residual, thereby mitigating advantage staleness without requiring cumulative trace products. However, it does not provide the same multi-step off-policy correction as Retrace/V-trace.

Our design is motivated by long-horizon RLVR, where rewards are typically sparse and outcome-based. We adopt λ=1\lambda=1 to favor Monte Carlo-style estimates over shorter-horizon bootstrapped estimates. Consistent with this choice, reducing λ\lambda from 1 to 0.99 and 0.95 under standard GAE (COPC w/o RC) substantially degrades performance. Similarly, removing V-trace's cumulative trace products improves average performance by 0.85 and 1.20 points on mathematical reasoning and search, respectively, suggesting that attenuating the contributions of distant TD residuals can hinder long-horizon credit assignment.

However, removing cumulative trace products alone is insufficient. The resulting variant retains V-trace's one-sided TD-residual weight min⁡(1,ρ)\min(1,\rho) and still trails COPC by 2.08 and 1.88 points on mathematical reasoning and search, respectively. One-sided truncation can excessively suppress TD-residual contributions when importance ratios are small, potentially weakening the correction of stale advantages. In contrast, COPC's two-sided clipping limits such suppression through a lower bound while controlling large importance weights through an upper bound. Together, these ablations support the complementary benefits of avoiding cumulative trace products and adopting two-sided importance-ratio clipping.

The corresponding ablation results are summarized below, with detailed per-dataset results in Tables 6–7 and further analysis in Appendix D (Effects of Trace Decay and Trace Coefficients). Here, COPC w/ VT and its w/o TC variant denote the V-trace-based ablations with and without cumulative trace products, respectively. COPC w/o RC uses standard GAE with λ=1\lambda=1.

MethodMath Avg.Search Avg.
COPC w/ VT47.3641.31
↳ w/o TC48.2142.51
COPC w/o RC48.1141.08
↳ λ=0.99\lambda=0.9945.7940.14
↳ λ=0.95\lambda=0.9536.5336.51
COPC (ours)50.2944.39

W2 & Q2: Comparison with V-trace under the strengthened PPO recipe.

Thank you for highlighting this concern. We retrained V-trace using the same strengthened training setup as COPC where applicable, including critic pretraining, advantage normalization, and value clipping, while retaining its original actor objective. We also evaluated COPC w/ VT, which replaces COPC's advantage correction with V-trace-style advantages while keeping the remaining training settings unchanged.

As shown below, COPC outperforms the retrained V-trace by 3.87 and 2.99 points on mathematical reasoning and search, respectively, and COPC w/ VT by 2.93 and 3.08 points, respectively. These results demonstrate that COPC's improvements persist under matched training recipes, while the comparison with COPC w/ VT more directly supports the effectiveness of COPC's advantage-correction design.

The corresponding results are summarized below, with detailed per-dataset results provided in the revised Tables 1–2.

MethodMath Avg.Search Avg.
V-trace46.4241.40
COPC w/ VT47.3641.31
COPC (ours)50.2944.39

Minor: Inconsistent notation for ηP\eta_P and ηA\eta_A.

Thank you for pointing out this inconsistency. We have revised the manuscript to consistently use ηP\eta_P and ηA\eta_A as scalar correction controls, while their corresponding pairs of ϵ\epsilon parameters denote masking and clipping margins. This is purely a notational clarification and does not affect the method, experimental results, or theoretical conclusions.

Q3: Applicability of advantage correction to critic-free methods.

Our current advantage correction relies on TD residuals constructed from learned value estimates and therefore cannot be directly applied to critic-free methods using group-relative advantages. Nevertheless, we believe the underlying idea of advantage correction can be extended to critic-free settings, and we plan to explore this direction in future work.

We are very grateful for your thoughtful and detailed feedback on our manuscript. We believe that incorporating your suggestions has substantially improved the quality and clarity of the manuscript.

As the discussion period is drawing to a close, please do not hesitate to let us know if there are any further questions or areas for clarification. We are ready and willing to discuss any remaining concerns. We thank you once more for your invaluable contribution to our work.

Thank you for the detailed response and new experiments. The ablation (strengthened PPO + Vtrace), and COPC with VT, COPC without TC addressed my W2 and mostly W1.

A small follow-up: Be careful when attributing the credits to two-sided clip only, when comparing COPC with VT to COPC, because they are also different in the upper bound: the former is min⁡(1,ρ)\min (1, \rho), while the latter is 1.28.

Thanks and I will keep my positive rating.

We sincerely appreciate your positive assessment and continued support. We are glad that our responses and additional experiments have addressed your W2 concern and most of W1.

Regarding the small follow-up, we have begun the corresponding controlled experiments. However, with only about a day available for these experiments, we will not be able to complete them before the discussion deadline and will report the results in the next revision.

Reviewer_F4LcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies policy staleness in asynchronous LLM reinforcement learning. The authors argue that existing off-policy correction methods primarily focus on token-level importance ratios in the actor objective, while overlooking the mismatch in advantage estimates caused by stale behavior-policy continuations. To address this issue, they propose Coupled Off-Policy Correction (COPC), which combines token-level importance-ratio masking with clipped-ratio reweighting of TD residuals for advantage and return estimation. The paper also analyzes the interaction between policy-side and advantage-side correction and proposes jointly tuning their correction strengths. Experiments on tool-integrated mathematical reasoning and search tasks demonstrate improved performance and training stability compared to existing asynchronous RL baselines.

Critical strengths
  1. The paper studies an important practical problem. Policy staleness is a significant challenge in asynchronous LLM RL, especially for long trajectories involving tool interactions.

  2. The motivation of advantage staleness is interesting. The paper highlights that correcting the actor's importance ratios alone does not address the mismatch in return and advantage estimation caused by stale trajectories.

  3. The experimental results are promising. COPC outperforms several relevant baselines, including SAO, AReaL, and V-trace, on both mathematical reasoning and search tasks. The improvements over COPC without return correction also provide evidence for the usefulness of the proposed advantage-side correction.

Critical weaknesses
  1. The theoretical analysis does not fully justify the proposed algorithm. The bias coupling term between policy-weight errors and advantage-estimation errors follows directly from expanding the product of two approximate quantities. Similarly, the variance coupling through squared policy weights is relatively straightforward. While these observations motivate joint tuning, the analysis does not establish why the particular clipped-ratio TD reweighting strategy should be preferred over existing alternatives or provide a principled way to choose the two correction strengths.

  2. The ablation studies do not sufficiently isolate the contributions of actor and critic correction. COPC uses corrected advantages for actor updates and corrected returns for critic training. However, the COPC w/o RC ablation removes both modifications simultaneously. Since advantages are treated as constants during actor optimization, the proposed advantage correction effectively changes the token-level gradient coefficients, while critic correction additionally changes future value estimates. It would be helpful to separately evaluate actor-only advantage correction and critic-only return correction to understand which component contributes most to the improvements.

  3. The role of group sampling is unclear. COPC uses 8 responses per prompt for mathematical reasoning and 4 for search, despite relying on a learned critic rather than group-relative rewards for advantage estimation. In contrast, SAO uses a single rollout per prompt. This difference in sampling strategies complicates the comparison between the two methods. I would like to see COPC evaluated with a single rollout per prompt, along with an ablation on the number of rollouts, to better understand its robustness and sample efficiency.

  4. The experimental evaluation remains somewhat limited in model scale and statistical robustness. The experiments use Qwen3-4B-Instruct for mathematical reasoning and Llama-3.2-3B-Instruct for search. Although the two task settings provide useful diversity, it remains unclear whether the findings generalize to larger reasoning models. Additionally, the main results do not report multiple random seeds or uncertainty estimates, making it difficult to assess the statistical significance of the reported improvements.

Questions for authors

Listed in weakness.

4 comments

Thank you for the insightful review. Here are our response to your comments.

W1: Limited theoretical justification for COPC's design choices.

We agree that the decompositions themselves are mathematically straightforward. Their value, however, lies in making the interaction between the two correction channels explicit: the effect of adjusting one channel depends on the setting of the other, motivating joint rather than independent calibration. Motivated by our theoretical decompositions, we hypothesize that tighter ratio control in one channel may favor looser control in the other, a trend supported empirically by Figure 1. The optimal correction strengths can vary with critic quality, importance-ratio distributions, cross-token correlations, and the current training state. Deriving a closed-form optimum would require additional assumptions about these factors and the training dynamics, making the result dependent on specific modeling choices. We therefore focus on providing qualitative guidance for jointly calibrating the two correction channels without relying on a particular model of the training dynamics.

Our return-correction design is motivated by long-horizon credit assignment in RLVR, where rewards are typically sparse and outcome-based. We adopt λ=1\lambda=1 to favor Monte Carlo-style estimates over shorter-horizon bootstrapped estimates. As shown in the ablations below, reducing λ\lambda under standard GAE (COPC w/o RC) substantially degrades performance, suggesting that longer-horizon estimates are more effective in our setting. Similarly, V-trace's cumulative trace products can attenuate the contributions of distant TD residuals. Removing these products improves average performance by 0.85 and 1.20 points on mathematical reasoning and search, respectively, further supporting the importance of maintaining long-range credit assignment.

However, removing cumulative trace products alone is insufficient. The resulting variant retains V-trace's one-sided TD-residual weight min⁡(1,ρ)\min(1,\rho) and still trails COPC by 2.08 and 1.88 points on mathematical reasoning and search, respectively. Such one-sided truncation can excessively suppress TD-residual contributions when importance ratios are small. In contrast, COPC's two-sided importance-ratio clipping introduces a lower bound to mitigate this suppression while retaining an upper bound to control large weights. Together, these results support our choices of longer-horizon advantage estimation, avoiding cumulative trace products, and two-sided TD-residual weighting.

The corresponding ablation results are summarized below, with detailed per-dataset results in Tables 6–7 and further analysis in Appendix D (Effects of Trace Decay and Trace Coefficients). Here, COPC w/ VT and its w/o TC variant denote the V-trace-based ablations with and without cumulative trace products, respectively. COPC w/o RC uses standard GAE with λ=1\lambda=1.

MethodMath Avg.Search Avg.
COPC w/ VT47.3641.31
↳ w/o TC48.2142.51
COPC w/o RC48.1141.08
↳ λ=0.99\lambda=0.9945.7940.14
↳ λ=0.95\lambda=0.9536.5336.51
COPC (ours)50.2944.39

W2: Contributions of advantage and critic return correction.

To address this concern, we conducted additional ablations to disentangle the contributions of advantage correction and critic return correction. As shown below, both corrections independently improve performance over COPC w/o RC, with advantage correction yielding larger gains on both mathematical reasoning and search tasks. Combining both corrections further improves performance over advantage correction alone by 0.57 and 0.85 points, respectively. Advantage correction directly modifies the advantages used in policy updates, whereas critic return correction influences policy learning indirectly through updated value estimates. This distinction may explain the larger gains from advantage correction, while the full COPC results support the complementary benefits of combining both corrections.

The corresponding ablation results are summarized below, with detailed per-dataset results in Tables 8–9 and further analysis in Appendix E (Disentangling Advantage and Critic Return Correction). Here, w/o Critic corr. removes critic return correction while retaining advantage correction, whereas w/o Adv. corr. removes advantage correction while retaining critic return correction.

MethodMath Avg.Search Avg.
COPC (ours)50.2944.39
↳ w/o Critic corr.49.7243.54
↳ w/o Adv. corr.48.2442.69
↳ w/o RC48.1141.08

W3: Robustness and sample efficiency under different group sampling sizes.

We conducted additional ablations with K∈{1,4,8}K \in \{1,4,8\} for mathematical reasoning and K∈{1,2,4}K \in \{1,2,4\} for search. We fix the rollout batch size at 512 trajectories per training step and adjust the number of prompts accordingly, ensuring the same rollout trajectory budget across group sizes. The K=1K=1 setting also allows a direct comparison with SAO under the same number of responses per prompt.

As shown below, COPC maintains comparable performance across group sizes, including K=1K=1. Notably, at K=1K=1, COPC outperforms SAO by 2.24 and 3.45 points on mathematical reasoning and search, respectively. These results indicate that COPC's gains do not rely on larger group sizes or additional rollout trajectories per training step, supporting its robustness under a fixed sampling budget. Detailed per-dataset results and further analysis are provided in Tables 10–11 and Appendix F (Additional Robustness and Generalization Experiments).

MethodMath Avg.Search Avg.
SAO (K=1K=1)47.8541.61
COPC (K=1K=1)50.0945.06
COPC (K=2K=2)—44.49
COPC (K=4K=4)49.9844.39
COPC (K=8K=8)50.29—

W4: Generalization to larger models and reliability across random seeds.

To assess generalization to larger models, we conducted additional experiments on Qwen3-30B-A3B-Instruct, a larger Mixture-of-Experts (MoE) model. As shown below, COPC achieves the highest average performance, outperforming AReaL and COPC w/o RC by 1.96 and 2.01 points, respectively. Detailed per-dataset results and further analysis are provided in Table 12 and Appendix F (Additional Robustness and Generalization Experiments).

MethodAvg.
AReaL58.21
COPC w/o RC58.16
COPC (ours)60.17

Regarding statistical reliability, our mathematical reasoning experiments already used three random seeds (42, 3407, and 1234). We additionally conducted independent training runs with the same seeds for COPC, COPC w/o RC, SAO, and AReaL on search tasks. Across three seeds, COPC achieves 50.29 ± 0.22 on mathematical reasoning and 44.39 ± 1.05 on search, outperforming the strongest asynchronous baseline, AReaL, by 2.03 and 2.51 points, respectively. We have updated Tables 1–2 to report the mean and sample standard deviation across three seeds.

We are very grateful for your thoughtful and detailed feedback on our manuscript. We believe that incorporating your suggestions has substantially improved the quality and clarity of the manuscript.

As the discussion period is drawing to a close, please do not hesitate to let us know if there are any further questions or areas for clarification. We are ready and willing to discuss any remaining concerns. We thank you once more for your invaluable contribution to our work.

We sincerely appreciate your positive assessment of our work and your constructive feedback. For ease of reference, we summarize below our clarifications and additional experiments, which we hope adequately address the four concerns you raised:

  1. Motivation and justification of COPC’s design: We clarified that our theoretical analysis exposes the coupling between policy-side and advantage-side corrections, motivating joint rather than independent calibration. Deriving an exact optimum would require additional assumptions about the underlying training dynamics, making the result dependent on modeling choices; we therefore focus on qualitative guidance for joint calibration. We further motivated return correction from long-horizon credit assignment in sparse-reward RLVR. With λ=1\lambda=1, COPC w/o RC achieves 48.11/41.08 on mathematical reasoning/search; these scores drop to 45.79/40.14 at λ=0.99\lambda=0.99 and 36.53/36.51 at λ=0.95\lambda=0.95. Removing V-trace’s cumulative trace products improves performance by 0.85/1.20 points, while full COPC provides a further 2.08/1.88-point improvement, supporting the overall return-correction design.

  2. Contributions of the two correction components: We added separate advantage-only and critic-only ablations. Relative to COPC w/o RC, they improve mathematical reasoning/search by 1.61/2.46 and 0.13/1.61 points, respectively. Combining both produces larger gains of 2.18/3.31 points, demonstrating their complementary contributions.

  3. Robustness to group sampling: Under the same budget of 512 trajectories per step, COPC remains stable across group sizes, with scores ranging from 49.98–50.29 on mathematical reasoning and 44.39–45.06 on search. At K=1K=1, COPC outperforms SAO by 2.24 and 3.45 points, confirming that its gains do not rely on group size or additional rollout trajectories.

  4. Model scale and statistical reliability: On Qwen3-30B-A3B-Instruct, COPC achieves 60.17, compared with 58.21 for AReaL and 58.16 for COPC w/o RC. Across three random seeds, COPC achieves 50.29±0.2250.29\pm0.22 on mathematical reasoning and 44.39±1.0544.39\pm1.05 on search, outperforming AReaL by 2.03 and 2.51 points, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.