acceptodds
Under review as a conference paper at ICLR 2027

Estimation Debiasing for Asynchronous Policy Optimization

Abstract

In reinforcement learning for large language models, asynchronous pipelines sample a batch of rollouts once and use them for several updates. This reduces generation overhead but degrades performance due to rollout staleness: the behavior policy gradually diverges from the target policy, introducing off-policy errors that bias the estimated objective. Existing methods control this error with PPO-style clipping by limiting the policy shift. However, our derivation shows that this restriction also limits objective improvement, making clipping alone insufficient. In addition to clipping, the bias should be directly corrected in the objective estimator. We propose Bias-Corrected Yielded Policy Optimization (BYPO), which constructs a more accurate objective estimator through sequence-level importance weighting and second-order Taylor correction, adaptively switches between them to control variance, and falls back to a first-order approximation when both corrections become excessively large. Experiments on mathematical reasoning and preference-alignment tasks show consistent improvements in asynchronous training. Applied to DAPO, BYPO mitigates 83.70% of the performance degradation caused by rollout staleness, with negligible runtime overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reinforcement learning, reasoning, alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mc63OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the mismatch between the policy that generates rollouts and the policy being updated during LLM reinforcement learning. It proposes BYPO, which corrects the training objective using either sequence-level importance weighting or a second-order Taylor correction. The method switches between these corrections based on their magnitude and falls back to the original objective when both are too large.

The authors evaluate BYPO on mathematical reasoning and preference alignment, compare it with several relevant baselines, and show that it can improve multiple existing optimization methods. In the reported Qwen2.5-7B comparison, BYPO recovers about 84% of the accuracy gap between their synchronous and asynchronous DAPO settings, with little additional training time.

Critical strengths

The correction addresses a relevant problem and can be incorporated into existing training objectives without requiring additional rollouts. I find the second-order correction particularly interesting because it offers a practical way to account for policy mismatch while avoiding the instability of full sequence-level importance weighting. The improvements across several underlying algorithms make this contribution more convincing.

Overall, I find the baseline selection and experimental design adequate for evaluating the proposed correction within the settings studied. The paper includes both reasoning and preference-alignment tasks, comparisons with other methods that address policy mismatch, and useful ablations. BYPO achieves the highest average reasoning score in all three model settings.

Critical weaknesses

My main concern is that the paper conflates asynchronous training with taking several updates on a previously generated batch. Asynchronous training decouples rollout generation from optimization; it does not inherently require reusing the same experience. Rollouts can become stale while being generated or waiting in a queue as the trainer continues updating, even when each rollout is consumed only once. Section 6 appears to describe sequential minibatch updates on a previously generated rollout batch, but does not clearly establish whether generation and training run concurrently. Please clarify whether these operations overlap, how rollout-worker weights are synchronized, and what policy lag is observed. The results support the usefulness of the correction for policy mismatch, but its effectiveness in the asynchronous setting used to motivate the paper remains unclear.

Overall, I appreciate the quality of the work and the effort put into the experiments, and I would be inclined toward acceptance if this concern is resolved. Since asynchronous training is central to the paper’s motivation and claims, the evaluation should demonstrate the method in that setting, or the claims should be revised to match the form of policy mismatch actually studied.

Additionally, the evaluation is limited to Qwen2.5-7B, Qwen3-4B, Llama3.2-1B-Instruct, and Llama3.1-8B-Instruct. Although these experiments provide useful coverage across model families and tasks, they leave applicability to newer or larger models untested. I see this as a limitation of the paper’s broader claims rather than a reason to dismiss the reported improvements.

Questions for authors
  1. Can you clarify the actual execution of the asynchronous experiments? Do rollout workers generate new samples concurrently with training, how often are their weights synchronized, and are individual responses consumed once or reused? What policy lag do the samples have when they reach the trainer?
  2. For the synchronous DAPO comparison underlying the 83.70% recovery claim, which quantities are held fixed (training prompts, generated responses, optimizer updates, and minibatch size)? This would help establish how much of the gap can be attributed specifically to rollout staleness.
  3. Are the improvements over DAPO, and especially the difference between full BYPO and the bounded second-order-only variant, consistent across independent training seeds?
  4. Which bias and variance conclusions apply to the implemented objective after adaptive selection, correction scaling, and PPO clipping? In particular, can you clarify why the reported small selection-correction correlations are sufficient to conclude that the additional bias is negligible?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mc63 (1/2)

Thank you for your detailed feedback! We have responded to the weaknesses and questions.

Weakness 1: whether asynchronous RL

Thank you for pointing out the need to accurately describe the application scenario. In fact, our work does not target "asynchronous RL"; our previous use of this term was inaccurate. Instead, our method addresses rollout staleness or policy mismatch, which introduces bias into the objective function because the behavior policy is not updated in a timely manner to match the target policy. This issue also arises in synchronous training with verl, where rollouts for multiple update steps are typically generated in a single batch and subsequently reused for multiple consecutive parameter updates. Consequently, after the first update, the target policy already deviates from the behavior policy. Nevertheless, this is also an important problem that deserves attention, and our baselines, such as TOPR and VESPO, are specifically designed to address this issue.

We acknowledge that the title of our paper and our use of the term "asynchronous RL" may be misleading. In the next revision, we will remove all references to "asynchronous" and replace them with "rollout staleness." We will also change the title to "Debiasing Estimation for Rollout Staleness in Policy Optimization." Given that rollout staleness is an important research problem in its own right, and that BYPO directly addresses this problem, we hope that these revisions will alleviate the reviewer's concerns regarding the applicability and positioning of our work.

We have revised these sections, with the changes highlighted in blue in the latest version of the PDF.

Weakness 2: newer or larger models

Thank you for raising the consideration of larger-scale models. We acknowledge that this is a limitation of the broader claims supported by our experiments, and we will explicitly discuss it in the Limitations section of the revised paper.

Question 1: experiment details

As explained in our response to Weakness 1, we previously used the term "asynchronous RL" inaccurately. In fact, BYPO addresses "rollout staleness" and "policy mismatch," which are also important problems. In our experiments, we use a synchronous but stale RL workflow implemented with verl. The rollout workers generate responses to 256 prompts in each rollout batch. In the Qwen7B experiments, these responses are used for 8 consecutive parameter updates, while in the other experiments, they are used for 4 consecutive updates. Therefore, the parameter synchronization intervals are 8 and 4 updates, respectively. We have also reported additional experimental results under more frequent synchronization in our response to reviewer wPMY's question.

Each response is used only once, and policy lag accumulates as the number of parameter updates within each rollout cycle increases. For example, in the Qwen7B experiments, the KL divergence between the target and behavior policies is typically on the order of 10−410^{-4} to 10−310^{-3} at the final update step.

Question 2: "synchronous DAPO"

The number of samples used for each parameter update (including both the number of prompts and the number of generated responses), the total number of parameter updates, the minibatch size (which, in verl, corresponds to the number of samples used per parameter update), and the microbatch size (which, in verl, corresponds to the number of samples processed per GPU for each gradient computation) are all kept identical. Therefore, both the number of parameter updates and the number of samples used per update are the same across all methods. The only difference is whether the rollout workers generate samples for multiple parameter-update steps togather or regenerate samples after each parameter update.

Thank you for the concern and we will include all these details in a future revision of the paper.

Question 3: independent training seeds

We have added results from multiple seeds for both the bounded second-order-only variant and full BYPO in the Qwen7B experiments. The following results are based on three runs in total:

full BYPObounded second-order-only
40.07 ± 0.1939.94 ± 0.16
As can be seen, although there is some overlap, full BYPO achieves better performance on average.

Moreover, we have already conducted multiple runs for several key experiments, with the following results:

MethodQwen7BQwen4BLlama1B
DAPO38.91 ± 0.2460.74 ± 0.1414.77 ± 0.30
TOPR38.77 ± 0.2560.52 ± 0.2515.23 ± 0.19
BYPO40.07 ± 0.1961.61 ± 0.1316.38 ± 0.12

where BYPO consistently outperforms DAPO and other baselines.

AuthorsOpenReview ↗

Response to Reviewer mc63 (2/2)

Question 4: conclusions apply to the implemented objective

In conclusion, the bias-reduction guarantees remain valid for the implemented objective, whereas the variance analysis may no longer strictly hold due to the presence of clipping. However, since clipping is triggered in less than 1% of cases in our experiments, the variance conclusions are likely to remain valid as well. Furthermore, the variance analysis is primarily intended to motivate the switching between different correction terms, rather than to characterize the final objective function. We will clarify and emphasize this distinction in a future revision of the paper.

The final objective function of BYPO (Eqs. 14–17) is derived from the two fully corrected formulations in Eqs.6 and 12, by successively introducing three steps: scaling, selection, and clipping. As we show below, Steps 1 and 3 strictly guarantee bias reduction, while Step 2 may introduce additional error. However, our experimental measurements show that this additional bias is negligible. Therefore, BYPO also guarantees error reduction in practical applications.

We provide a more detailed explanation below, which will be incorporated into the appendix in a future revision.

1. Scaling (Lines 289–291 of Section 5.3)

This step still strictly guarantees bias reduction.

The fully corrected objective (denoted as L2L_2, corresponding to Eq. 7 or Eq. 11) has a smaller error relative to the actual objective (L\mathcal{L} in Eq. 1) than the completely uncorrected objective (denoted as L1L_1, Eq. 2). Therefore, the error of their convex combination is also smaller than that of the completely uncorrected objective:

$| \mathcal{L} - (\alpha L_1 + (1-\alpha) L_2 ) | \le \alpha |\mathcal{L} - L_1 | + (1-\alpha) | \mathcal{L} - L_2 | < \alpha |\mathcal{L} - L_1 | + (1-\alpha) | \mathcal{L} - L_1 | = | \mathcal{L} - L_1 | $

Therefore, scaling also guarantees error reduction.

2. In particular: why the reported small selection-correction correlations are sufficient to conclude that the additional bias is negligible (Lines 292–293 and 296–297 of Section 5.3)

This step introduces additional error. However, our experiments show that the additional error is very small, so the overall error is still reduced.

We select among three objective formulations (importance weighting / Taylor correction / no correction), which means that the actual objective function is:

$ \mathbb{E} [ (1 - \mathbb{I}_1- \mathbb{I}_2) L _1 + \mathbb{I}_1 L_2 + \mathbb{I}_2 L_3 ] = -\mathbb{E} [L _ 1] + \mathbb{E} [(1 - \mathbb{I}_1) L _ 1 + \mathbb{I} _ 1 L _ 2 ] + \mathbb{E} [(1 - \mathbb{I}_2) L _ 1 + \mathbb{I} _ 2 L _ 3 ], $

where the two I\mathbb{I} cannot simultaneously equal 1.

We observe that both terms have the structure $ \mathbb{E}[\mathbb{I}X + (1 - \mathbb{I}) Y] $, so we first analyze this structure:

$ \mathbb{E}[\mathbb{I}X + (1 - \mathbb{I}) Y] = \mathbb{E} [\mathbb{I}] \mathbb{E}[X] + \mathbb{E}[(1-\mathbb{I})]\mathbb{E}[Y] + \mathrm{Cov}(\mathbb{I}, X-Y) $

This shows that the difference between dynamically selecting between XX and YY and directly taking their linear combination (scaling) is only Cov(I,X−Y)\mathrm{Cov}(\mathbb{I}, X-Y), which represents the correlation between the selection operator and the difference between the two objective functions. In other words, if the selection operator is independent of the correction term, it does not introduce additional bias.

In our experiments, we measured the correlations between the two selection operators (determining whether to apply second-order correction and whether to apply importance-weighting correction, respectively) and their corresponding correction terms. We found that the absolute values of both correlation coefficients were below 0.02, indicating extremely weak correlations. Therefore, the additional error introduced by the selection mechanism is very small.

3. PPO-clip (Lines 278–280 of Section 5.3)

This step does not introduce bias because the actual optimization objective must also account for the confidence-region constraint.

Even after correction, offline estimation error still exists, although it has been reduced. Therefore, as in TRPO, our goal is not simply to solve max⁡L\max {L}, but rather $ \max_{\mathbb{D} \le \epsilon} L $, where LL is the surrogate objective (including the corrected objective), and D\mathbb{D} represents the policy deviation.

PPO-clip provides an unconstrained approach to solving this constrained optimization problem, based on the principle of: $ \max \mathrm{PPOclip}(\nabla L, \theta) $.

This is precisely what BYPO does: it derives the gradient of the correction term (Eqs. 16–17) and then incorporates it into PPO-clip. Therefore, with respect to the objective of optimizing within the confidence region, this step does not introduce additional error.

We will incorporate the above discussion into a future revision of the paper.

Thank you for the clarification and additional results. Reframing the paper around rollout staleness addresses my concern about the asynchronous-training claims. However, the response does not sufficiently justify the bias-reduction guarantees for the final objective after adaptive selection and clipping. The additional experiments also show little separation between full BYPO and the simpler second-order-only variant. Overall, I remain unconvinced by the contribution and maintain my weak rejection.

Reviewer_CKj8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes Bias-Corrected Yielded Policy Optimization (BYPO) to reduce the bias caused by stale rollouts when one batch of rollouts is reused for several policy updates in RL training of LLMs. This work argus that the existing PPO-style clipping alone cannot fix this staleness issue. By reviewing the existing method such as DAPO as a first-order Taylor approximation of the true objective, this work proposes to add the second-order term, or use the full sequence-level importance weight when the policy has barely changed, and turn the correction off and reduce to the original DAPO when the either term is large. To validate this design, this work trains Qwen2.5-7B, Qwen3-4B, and Llama3.2-1B on math reasoning datasets and Llama-3.1-8B on preference alignment, mitigating 83.70% of the performance degradation with negligible runtime overhead.

Critical strengths

I think this work provides an interesting perspective of viewing the token-level RL surrogate objectives such as DAPO as first-order Taylor approximations of the sequence-level objective. The corresponding analysis about the gap between the optimized surrogate and the true objective under policy drift is convincing and the resultant higher-order corrections are making sense. Also, the proposed BYPO is easy to integrate to the existing training frameworks.

Critical weaknesses

(1) The asynchronous RL training setting targeted by this work: generating one batch of rollouts and splits it into mini-batch updates, is actually standard synchronous PPO-style training as used by default in verl for example. It is not the actual asynchronous RL training we are talking about today. In actual asynchronous RL, generation and training run concurrently. So the batches come from policies lagging several steps behind or a single sequence may even be generated by multiple policy versions. Therefore, I think the paper’s framing and title is quite misleading.

(2) I find that the core idea of BYPO closely follows Taylor Expansion Policy Optimization, an ICML 2020 publication, which is not cited. TayPO already shows that the TRPO/PPO surrogate is the first-order Taylor expansion of the RL objective and explains the trade-off between these expansions and full importance sampling. It is even motivated by the same problem of stale behavior policies in asynchronous training. BYPO mainly applies this idea to LLM training and adds gating and scaling heuristics. The authors should cite TayPO and clearly explain what is new in BYPO.

(3) I also find that the experiment settings in this work are quite confusing and unconvincing. Each model is trained on a different dataset with different group sizes and numbers of updates per rollout batch, so the settings cannot be compared with each other. The evaluation protocol (pass@1 or avg@k, sampling temperature, number of seeds) is not reported, and the results appear to come from a single run. The gains of about one point are within typical run-to-run variance.

Questions for authors

Please solve all three weaknesses, which are very critical.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CKj8 (1/3)

Thank you for your constructive feedback! It is not very encouraging to receive an unfavorable evaluation. However, the major raised weaknesses primarily concern the positioning of our method (i.e., its application scenarios and whether it constitutes incremental or redundant work), rather than the method itself! Importantly, these concerns about positioning can be addressed by clarifying and refining how we present our contributions, without requiring changes to the method itself. We therefore hope that our responses below will help clarify these issues and encourage you to reconsider your evaluation of our work.

We have revised the PDF to address the comments regarding “whether asynchronous RL” (in red) and “relations to TayPO” (in blue). More detailed responses are provided below.

Weakness 1: whether asynchronous RL

We agree that our work does not target "the actual asynchronous RL training we are talking about today." Instead, our method addresses rollout staleness, which introduces bias into the objective function because the behavior policy is not updated in a timely manner to match the target policy. This issue also arises in synchronous training with verl, where rollouts for multiple update steps are typically generated in a single batch and subsequently reused for multiple consecutive parameter updates. Consequently, after the first update, the target policy already deviates from the behavior policy. Nevertheless, this is also an important problem that deserves attention, and our baselines, such as TOPR and VESPO, are specifically designed to address this issue.

We acknowledge that the title of our paper and our use of the term "asynchronous RL" may be misleading. In the next revision, we will remove all references to "asynchronous" and replace them with "rollout staleness." We will also change the title to "Debiasing Estimation for Rollout Staleness in Policy Optimization." Given that rollout staleness is an important research problem in its own right, and that BYPO directly addresses this problem, we hope that these revisions will alleviate the reviewer's concerns regarding the applicability and positioning of our work.

Weakness 2: TayPO

Thank you for pointing out this related method,we were indeed previously unaware of this closely related work, as it falls outside the domains of RLVR and LLMs. However, given the differences in the tasks and objective functions, the resulting second-order expansions, our distinct conclusions from variance analysis, and the different ways in which we apply second-order Taylor approximations, as well as the fact that second-order Taylor approximation is only one component of BYPO rather than the entire method, BYPO and TayPO share only partial similarities but are fundamentally different algorithms overall. Therefore, BYPO should not be considered a redundant or duplicate work. Furthermore, BYPO specifically targets LLMs, where additional challenges such as high policy variance and large action spaces must be addressed. Consequently, BYPO still offers substantial novel contributions.

We therefore hope that clarifying the differences between BYPO and TayPO, as well as highlighting our novel contributions, will address the reviewer's concerns regarding this aspect. We will also incorporate the above discussion into a future revision of the paper, explicitly cite TayPO, include it in the related work section, and emphasize the distinctions between our approach and TayPO in the section on second-order Taylor approximation.

Below, we clarify the key differences between BYPO and TayPO.

1. Tasks, Objective Functions and Resulting Second-Order Expansions

BYPO and TayPO target different tasks, and the objective function used in BYPO is a special case of that in TayPO, leading to different forms of the derived Taylor approximation expressions.

TayPO focuses on classical RL, whereas BYPO targets RLVR, a specific setting within RL. RLVR has several distinctive properties: it only provides outcome-based rewards, γ=1\gamma=1, intermediate state transitions are deterministic, and the probabilities of different steps within the same trajectory can be computed in parallel. These properties require BYPO to develop a different objective estimation approach for the Taylor expansion.

AuthorsOpenReview ↗

Response to Reviewer CKj8 (2/3)

The second-order Taylor expansion derived in TayPO is: Ea,s,a′,s′,t(a′)>t(a)[(πθ(a∣s)πold(a∣s)−1)(πθ(a′∣s′)πold(a′∣s′)−1)Qπold(a′,s′)].\mathbb{E}_ {a,s,a',s',t(a')>t(a)} \left[\left( \frac{\pi_\theta(a|s)}{\pi_\text{old}(a|s)} - 1 \right)\left( \frac{\pi_ \theta(a'|s')}{\pi_\text{old}(a'|s')} - 1 \right) Q^{\pi_ \text{old}} (a',s') \right ]. When directly applied to LLMs, each aa corresponds to a token, and each ss corresponds to a prefix, and QQ denotes the token-level action-value function. This makes it infeasible to apply TayPO's results to RLVR: (1) the losses for all steps in a sequence are computed together, making it infeasible to sample state pairs; and (2) the Q-function for each token cannot be estimated in GRPO or DAPO.

Therefore, to accommodate the specific characteristics of RLVR, BYPO derives a different second-order expansion:

\left[ {\textstyle \sum_t {\textstyle \sum_{k \ne t} \left( \omega_t-1 \right) \left( \omega_k-1 \right) r}} \right]$$ This formulation demonstrates how to simultaneously leverage all tokens in a complete reasoning trajectory without requiring Q-function estimation. **We will move the result showing that the surrogate objective in RLVR is a first-order approximation to the preliminaries. In the Taylor approximation subsection of the methodology, we will focus specifically on the distinct second-order Taylor approximation derived and employed by BYPO. We hope these revisions will sufficiently clarify the novel aspects of BYPO's Taylor approximation and distinguish them from TayPO.** ### 2. Variance Analysis Another major theoretical distinction between our work and TayPO is that we provide a **quantitative** variance analysis. The TayPO paper **only qualitatively states** that Taylor approximation has lower variance than importance weighting, whereas our paper provides explicit quantitative results, showing that the variance of the second-order Taylor approximation is **second-order in the policy deviation (Eq. 13)**, while the variance of importance weighting **scales with $T$ (Eq. 10)**. Combined with the error analysis of different approximations, these results guide BYPO in switching between different estimation methods to balance variance and accuracy. ### 3. Ways in Which We Apply Taylor Approximation Our application of Taylor approximation also differs from that in TayPO. TayPO directly uses the second-order approximation **as its objective function**, whereas BYPO estimates **the gradient** of this approximation and integrates it with **PPO-clip** (Eqs. 14–15). This design is motivated by the fact that the RLVR setting targeted by BYPO is considerably more challenging than traditional RL, involving much longer sequences and larger action spaces. Consequently, although the approximation errors are of the same order, **the offline estimation error of the objective function in RLVR is substantially larger** than that in the traditional RL setting studied by TayPO. Therefore, **directly using TayPO's objective function is not feasible**, as it effectively solves the following optimization problem at each step: \$ \max_ {\pi_ \theta} L^\text{taylor}_ {\pi_ \text{old}}(\pi_ \theta) \$ In contrast, **BYPO** combines the second-order approximation with the trust-region idea of TRPO, which essentially solves: \$ \max_ {\mathbb{D}(\pi_\theta, \pi_\text{old}) \le \epsilon} L^ \text{taylor}_ {\pi_ \text{old}}(\pi) \$ First, these are different optimization objectives. Second, BYPO implements this constrained optimization by incorporating the policy gradient into **PPO-clip** (Eqs. 14–15). Specifically, Eq. 15 estimates the policy gradients under different approximation schemes, while Eq. 14 applies PPO-clip to these estimated policy gradients. Note that the purpose of PPO-clip is precisely to optimize the objective function under a constraint on policy deviation. ### 4. Second-order Taylor approximation is only one component of BYPO BYPO is not simply about using Taylor approximation to reduce error. **Rather, it focuses on how to combine different methods to jointly reduce estimation error while controlling variance.** In addition to second-order Taylor approximation, BYPO also includes: 1. **Importance weighting as an alternative objective:** BYPO incorporates importance weighting as another correction approach. 2. **Dynamic selection among different correction schemes:** BYPO introduces a dynamic mechanism to switch between different correction schemes and between correction and no correction, thereby controlling variance. We also provide a corresponding error analysis for this mechanism (Lines 292–293 and 296–297 of Section 5.3). 3. **Integration with PPO-clip:** BYPO integrates different correction terms with PPO-clip to construct the final optimization objective.
AuthorsOpenReview ↗

Response to Reviewer CKj8 (3/3)

Summary

BYPO is neither a redundant reproduction of TayPO nor a straightforward implementation of TayPO in RLVR. First, BYPO differs in how Taylor approximation is implemented, including its gradient estimation method and its integration with PPO-clip. Second, second-order Taylor approximation is only one component of BYPO, BYPO also incorporates other approximation methods, dynamic switching among different approximation methods, adaptive selection between correction and no correction, and the integration of various correction terms with PPO-clip. Finally, BYPO makes several additional theoretical contributions, including a quantitative variance analysis of different correction terms.

We will cite TayPO in the paper, move the results related to the first-order Taylor approximation (Eq. 11) to the Preliminaries section, explicitly clarify the differences between ours (Eq. 12) and TayPO's second-order approximation, and highlight the distinctions between BYPO and TayPO in the Related Work section. We believe these revisions sufficiently distinguish our method from "simply applying TayPO to RLVR", and we hope they will lead the reviewer to reconsider the evaluation of our work.

Weakness 3: Experiments

1. Differences in Experimental Settings

We would like to clarify that we compare BYPO against all baselines separately in three experiments. The datasets and RLVR configurations differ across these experiments to ensure that our evaluation covers as many scenarios as possible. However, within each experiment, BYPO and all baselines use exactly the same datasets, RLVR configurations, and other experimental settings. Readers should therefore compare different algorithms within each experiment, rather than compare results across different experiments.

2. Evaluation

We use the avg@k metric. Specifically, we use avg@50 for benchmarks with fewer than 150 problems, such as AIME and HMMT, and avg@1 for all other datasets. This evaluation strategy ensures that each benchmark has more than 150 evaluation samples to improve score stability, without introducing excessive computational overhead.

During evaluation, we set the temperature to 0.1 and top-p to 0.95, while keeping all other parameters consistent with the training configurations. We will include these additional details in a future revision of the paper.

3. Multiple Runs

Regarding experiments with multiple random seeds, we are unfortunately unable to provide a complete set of results due to the substantial computational cost (several times that of our current experiments). However, we have already conducted multiple runs for several key experiments, with the following results:

MethodQwen7BQwen4BLlama1B
DAPO38.91 ± 0.2460.74 ± 0.1414.77 ± 0.30
TOPR38.77 ± 0.2560.52 ± 0.2515.23 ± 0.19
BYPO40.07 ± 0.1961.61 ± 0.1316.38 ± 0.12

As can be seen, the experimental results are generally stable, with variations across multiple runs not exceeding 0.3 points, indicating that our results are reasonably reliable.

We will also make every effort to provide additional results from multiple runs. For experiments where this is not feasible, we will discuss this limitation in a future revision of the paper.

Final Summary

Our work is positioned as follows:

  • To address rollout staleness caused by the mismatch between behavior and target policies in RLVR
  • We develop correction methods applicable to RLVR, incorporating existing approaches such as Taylor approximation, importance weighting, and no correction, some of which cannot be directly applied to RLVR.
  • Building on our proposed variance and error analyses, we further develop an algorithm that dynamically switches among different correction schemes and integrates them with PPO-clip.
  • Although second-order Taylor approximation, one of the correction schemes we incorporate, was first introduced in TayPO, TayPO's approximation method cannot be directly applied to RLVR.
  • In contrast, BYPO derives a new estimation method and provides a novel quantitative variance analysis. We have also clarified and supplemented our experimental results.

Regarding the reviewer's concerns about the misuse of the term "asynchronous RL" and the similarity between our Taylor approximation and TayPO, we believe these issues can be resolved by adjusting the positioning of BYPO. Given that the BYPO algorithm itself has no fundamental flaws, and that the positioning issues only require refinements to certain descriptions, we sincerely hope that the reviewer will reconsider the initially evaluation.

Thank you for the detailed responses and the additional experiments. I do not think it is enough to give me the reason to raise my rating. i think my original rating is very reasonable for overall quality. I I also check the other reviews and find several of their concerns make sense. I think it is not very enough to be accepted. I highly do not recommend to accept this paper.

Reviewer_wPMYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BYPO to reduce the bias caused by reusing responses from older policies in LLM reinforcement learning. It switches between sequence-level importance sampling, a second-order Taylor correction, and no correction to balance bias and variance. Experiments show improvements on math reasoning and preference alignment.

Critical strengths

The method has clear practical value. BYPO reuses existing rollouts rather than requiring new samples. It also improves several training methods while keeping their advantage estimators unchanged, making it a useful addition to existing training pipelines.

Critical weaknesses

The theory does not fully cover the actual algorithm. The analysis studies full corrections, but the implemented method scales the corrections, selects them based on each sample, and applies clipping. The paper does not clearly establish which bias-reduction guarantees still hold after these changes.

Questions for authors

Could you compare DAPO, full BYPO, and the second-order-only version across multiple training seeds and different levels of rollout staleness?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wPMY (1/2)

Thank you for your detailed feedback! We have responded to the weaknesses and questions.

Weakness: bias-reduction guarantees

Thank you for the feedback. We acknowledge that the bias-reduction guarantees of the practical algorithm were not sufficiently emphasized in the main text (in fact they are only discussed in Lines 278–280, 289–291, 292–293, and 296–297 of Section 5.3).

The final objective function of BYPO (Eqs. 14–17) is derived from the two fully corrected formulations in Eqs.6 and 12, by successively introducing three steps: scaling, selection, and clipping. As we show below, Steps 1 and 3 strictly guarantee bias reduction, while Step 2 may introduce additional error. However, our experimental measurements show that this additional bias is negligible. Therefore, BYPO also guarantees error reduction in practical applications.

We provide a more detailed explanation below, which will be incorporated into the appendix in a future revision.

1. Scaling (Lines 289–291 of Section 5.3)

This step still strictly guarantees bias reduction.

The fully corrected objective (denoted as L2L_2, corresponding to Eq. 7 or Eq. 11) has a smaller error relative to the actual objective (L\mathcal{L} in Eq. 1) than the completely uncorrected objective (denoted as L1L_1, Eq. 2). Therefore, the error of their convex combination is also smaller than that of the completely uncorrected objective:

$| \mathcal{L} - (\alpha L_1 + (1-\alpha) L_2 ) | \le \alpha |\mathcal{L} - L_1 | + (1-\alpha) | \mathcal{L} - L_2 | < \alpha |\mathcal{L} - L_1 | + (1-\alpha) | \mathcal{L} - L_1 | = | \mathcal{L} - L_1 | $

Therefore, scaling also guarantees error reduction.

2. Selection (Lines 292–293 and 296–297 of Section 5.3)

This step introduces additional error. However, our experiments show that the additional error is very small, so the overall error is still reduced.

We select among three objective formulations (importance weighting / Taylor correction / no correction), which means that the actual objective function is like:

$ \mathbb{E} [ (1 - \mathbb{I}_1- \mathbb{I}_2) L _1 + \mathbb{I}_1 L_2 + \mathbb{I}_2 L_3 ] = -\mathbb{E} [L _ 1] + \mathbb{E} [(1 - \mathbb{I}_1) L _ 1 + \mathbb{I} _ 1 L _ 2 ] + \mathbb{E} [(1 - \mathbb{I}_2) L _ 1 + \mathbb{I} _ 2 L _ 3 ], $

where the two I\mathbb{I} cannot simultaneously equal 1.

We observe that both terms have the structure $ \mathbb{E}[\mathbb{I}X + (1 - \mathbb{I}) Y] $, so we first analyze this structure:

$ \mathbb{E}[\mathbb{I}X + (1 - \mathbb{I}) Y] = \mathbb{E} [\mathbb{I}] \mathbb{E}[X] + \mathbb{E}[(1-\mathbb{I})]\mathbb{E}[Y] + \mathrm{Cov}(\mathbb{I}, X-Y) $

This shows that the difference between dynamically selecting between XX and YY and directly taking their linear combination (scaling) is only Cov(I,X−Y)\mathrm{Cov}(\mathbb{I}, X-Y), which represents the correlation between the selection operator and the difference between the two objective functions. In other words, if the selection operator is independent of the correction term, it does not introduce additional bias.

In our experiments, we measured the correlations between the two selection operators (determining whether to apply second-order correction and whether to apply importance-weighting correction, respectively) and their corresponding correction terms. We found that the absolute values of both correlation coefficients were below 0.02, indicating extremely weak correlations. Therefore, the additional error introduced by the selection mechanism is very small.

3. PPO-clip (Lines 278–280 of Section 5.3)

This step does not introduce bias because the actual optimization objective must also account for the confidence-region constraint.

Even after correction, offline estimation error still exists, although it has been reduced. Therefore, as in TRPO, our goal is not simply to solve max⁡L\max {L}, but rather $ \max_{\mathbb{D} \le \epsilon} L $, where LL is the surrogate objective (including the corrected objective), and D\mathbb{D} represents the policy deviation.

PPO-clip provides an unconstrained approach to solving this constrained optimization problem, based on the principle of: $ \max_{\mathbb{D} \le \epsilon} L \rightarrow \max \mathrm{PPOclip}(\nabla L, \theta) $.

This is precisely what BYPO does: it derives the gradient of the correction term (Eqs. 16–17) and then incorporates it into PPO-clip. Therefore, with respect to the objective of optimizing within the confidence region, this step does not introduce additional error.

We will incorporate the above discussion into a future revision of the paper.

AuthorsOpenReview ↗

Response to Reviewer wPMY (2/2)

Questions

We have already reported the results for Llama3.2-1B-Instruct with staleness = 4 in our experiments. The staleness values reported refer to the number of parameter updates performed using a single generation batch (meaning that the target policy and the behavior policy can differ by at most staleness update steps in our experiments). We have conducted the additional experiment on Llama3.2-1B-Instruct, and the results are as follows:

staleness = 8:

MethodAIME24AIME25GSM8kMATH500AMC23OlympiadMinervaHMMTAVG
DAPO1.600.6754.9732.0015.837.584.780.2214.71
Taylor3.600.1353.9033.4015.337.435.880.8315.06
BYPO4.270.2754.5131.6020.178.326.250.9415.79

staleness = 16:

MethodAIME24AIME25GSM8kMATH500AMC23OlympiadMinervaHMMTAVG
DAPO1.470.2753.0731.4014.008.025.510.3314.26
Taylor2.530.0053.1528.8015.837.286.620.5614.35
BYPO2.670.4054.8931.6016.176.094.410.8914.64

As can be seen, the performance of all algorithms degrades as staleness increases. However, BYPO consistently achieves the best performance across all levels of staleness.

Regarding experiments with multiple random seeds, we are unfortunately unable to provide a complete set of results due to the substantial computational cost (several times that of our current experiments). However, we have already conducted multiple runs for several key experiments, with the following results:

MethodQwen7BQwen4BLlama1B
DAPO38.91 ± 0.2460.74 ± 0.1414.77 ± 0.30
TOPR38.77 ± 0.2560.52 ± 0.2515.23 ± 0.19
BYPO40.07 ± 0.1961.61 ± 0.1316.38 ± 0.12

As can be seen, the experimental results are generally stable, with variations across multiple runs not exceeding 0.3 points, indicating that our results are reasonably reliable. We will also make every effort to provide additional results from multiple runs. For experiments where this is not feasible, we will discuss this limitation in a future revision of the paper.

I appreciate the additional experiments, which strengthen the evidence for the reported gains. However, my main concern remains unresolved. The low correlations do not by themselves establish that selection introduces negligible bias, and the discussion of clipping does not show that it preserves the claimed guarantees. These are important gaps given the paper’s focus on bias reduction.

I have also read the other reviews and find several of their concerns reasonable. This is not my main area of expertise, so I am cautious about making firm judgments on the theoretical details.

Comments on the paper

1 comment

Dear Reviewers,

I hope you're doing well.

Please respond to the authors' rebuttal if you have not already done so.

Thank you for your valuable time and engagement.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.