- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the mismatch between the policy that generates rollouts and the policy being updated during LLM reinforcement learning. It proposes BYPO, which corrects the training objective using either sequence-level importance weighting or a second-order Taylor correction. The method switches between these corrections based on their magnitude and falls back to the original objective when both are too large.
The authors evaluate BYPO on mathematical reasoning and preference alignment, compare it with several relevant baselines, and show that it can improve multiple existing optimization methods. In the reported Qwen2.5-7B comparison, BYPO recovers about 84% of the accuracy gap between their synchronous and asynchronous DAPO settings, with little additional training time.
Critical strengths
The correction addresses a relevant problem and can be incorporated into existing training objectives without requiring additional rollouts. I find the second-order correction particularly interesting because it offers a practical way to account for policy mismatch while avoiding the instability of full sequence-level importance weighting. The improvements across several underlying algorithms make this contribution more convincing.
Overall, I find the baseline selection and experimental design adequate for evaluating the proposed correction within the settings studied. The paper includes both reasoning and preference-alignment tasks, comparisons with other methods that address policy mismatch, and useful ablations. BYPO achieves the highest average reasoning score in all three model settings.
Critical weaknesses
My main concern is that the paper conflates asynchronous training with taking several updates on a previously generated batch. Asynchronous training decouples rollout generation from optimization; it does not inherently require reusing the same experience. Rollouts can become stale while being generated or waiting in a queue as the trainer continues updating, even when each rollout is consumed only once. Section 6 appears to describe sequential minibatch updates on a previously generated rollout batch, but does not clearly establish whether generation and training run concurrently. Please clarify whether these operations overlap, how rollout-worker weights are synchronized, and what policy lag is observed. The results support the usefulness of the correction for policy mismatch, but its effectiveness in the asynchronous setting used to motivate the paper remains unclear.
Overall, I appreciate the quality of the work and the effort put into the experiments, and I would be inclined toward acceptance if this concern is resolved. Since asynchronous training is central to the paper’s motivation and claims, the evaluation should demonstrate the method in that setting, or the claims should be revised to match the form of policy mismatch actually studied.
Additionally, the evaluation is limited to Qwen2.5-7B, Qwen3-4B, Llama3.2-1B-Instruct, and Llama3.1-8B-Instruct. Although these experiments provide useful coverage across model families and tasks, they leave applicability to newer or larger models untested. I see this as a limitation of the paper’s broader claims rather than a reason to dismiss the reported improvements.
Questions for authors
- Can you clarify the actual execution of the asynchronous experiments? Do rollout workers generate new samples concurrently with training, how often are their weights synchronized, and are individual responses consumed once or reused? What policy lag do the samples have when they reach the trainer?
- For the synchronous DAPO comparison underlying the 83.70% recovery claim, which quantities are held fixed (training prompts, generated responses, optimizer updates, and minibatch size)? This would help establish how much of the gap can be attributed specifically to rollout staleness.
- Are the improvements over DAPO, and especially the difference between full BYPO and the bounded second-order-only variant, consistent across independent training seeds?
- Which bias and variance conclusions apply to the implemented objective after adaptive selection, correction scaling, and PPO clipping? In particular, can you clarify why the reported small selection-correction correlations are sufficient to conclude that the additional bias is negligible?
3 comments
Response to Reviewer mc63 (2/2)
Question 4: conclusions apply to the implemented objective
In conclusion, the bias-reduction guarantees remain valid for the implemented objective, whereas the variance analysis may no longer strictly hold due to the presence of clipping. However, since clipping is triggered in less than 1% of cases in our experiments, the variance conclusions are likely to remain valid as well. Furthermore, the variance analysis is primarily intended to motivate the switching between different correction terms, rather than to characterize the final objective function. We will clarify and emphasize this distinction in a future revision of the paper.
The final objective function of BYPO (Eqs. 14–17) is derived from the two fully corrected formulations in Eqs.6 and 12, by successively introducing three steps: scaling, selection, and clipping. As we show below, Steps 1 and 3 strictly guarantee bias reduction, while Step 2 may introduce additional error. However, our experimental measurements show that this additional bias is negligible. Therefore, BYPO also guarantees error reduction in practical applications.
We provide a more detailed explanation below, which will be incorporated into the appendix in a future revision.
1. Scaling (Lines 289–291 of Section 5.3)
This step still strictly guarantees bias reduction.
The fully corrected objective (denoted as , corresponding to Eq. 7 or Eq. 11) has a smaller error relative to the actual objective ( in Eq. 1) than the completely uncorrected objective (denoted as , Eq. 2). Therefore, the error of their convex combination is also smaller than that of the completely uncorrected objective:
$| \mathcal{L} - (\alpha L_1 + (1-\alpha) L_2 ) | \le \alpha |\mathcal{L} - L_1 | + (1-\alpha) | \mathcal{L} - L_2 | < \alpha |\mathcal{L} - L_1 | + (1-\alpha) | \mathcal{L} - L_1 | = | \mathcal{L} - L_1 | $
Therefore, scaling also guarantees error reduction.
2. In particular: why the reported small selection-correction correlations are sufficient to conclude that the additional bias is negligible (Lines 292–293 and 296–297 of Section 5.3)
This step introduces additional error. However, our experiments show that the additional error is very small, so the overall error is still reduced.
We select among three objective formulations (importance weighting / Taylor correction / no correction), which means that the actual objective function is:
$ \mathbb{E} [ (1 - \mathbb{I}_1- \mathbb{I}_2) L _1 + \mathbb{I}_1 L_2 + \mathbb{I}_2 L_3 ] = -\mathbb{E} [L _ 1] + \mathbb{E} [(1 - \mathbb{I}_1) L _ 1 + \mathbb{I} _ 1 L _ 2 ] + \mathbb{E} [(1 - \mathbb{I}_2) L _ 1 + \mathbb{I} _ 2 L _ 3 ], $
where the two cannot simultaneously equal 1.
We observe that both terms have the structure $ \mathbb{E}[\mathbb{I}X + (1 - \mathbb{I}) Y] $, so we first analyze this structure:
$ \mathbb{E}[\mathbb{I}X + (1 - \mathbb{I}) Y] = \mathbb{E} [\mathbb{I}] \mathbb{E}[X] + \mathbb{E}[(1-\mathbb{I})]\mathbb{E}[Y] + \mathrm{Cov}(\mathbb{I}, X-Y) $
This shows that the difference between dynamically selecting between and and directly taking their linear combination (scaling) is only , which represents the correlation between the selection operator and the difference between the two objective functions. In other words, if the selection operator is independent of the correction term, it does not introduce additional bias.
In our experiments, we measured the correlations between the two selection operators (determining whether to apply second-order correction and whether to apply importance-weighting correction, respectively) and their corresponding correction terms. We found that the absolute values of both correlation coefficients were below 0.02, indicating extremely weak correlations. Therefore, the additional error introduced by the selection mechanism is very small.
3. PPO-clip (Lines 278–280 of Section 5.3)
This step does not introduce bias because the actual optimization objective must also account for the confidence-region constraint.
Even after correction, offline estimation error still exists, although it has been reduced. Therefore, as in TRPO, our goal is not simply to solve , but rather $ \max_{\mathbb{D} \le \epsilon} L $, where is the surrogate objective (including the corrected objective), and represents the policy deviation.
PPO-clip provides an unconstrained approach to solving this constrained optimization problem, based on the principle of: $ \max \mathrm{PPOclip}(\nabla L, \theta) $.
This is precisely what BYPO does: it derives the gradient of the correction term (Eqs. 16–17) and then incorporates it into PPO-clip. Therefore, with respect to the objective of optimizing within the confidence region, this step does not introduce additional error.
We will incorporate the above discussion into a future revision of the paper.
Thank you for the clarification and additional results. Reframing the paper around rollout staleness addresses my concern about the asynchronous-training claims. However, the response does not sufficiently justify the bias-reduction guarantees for the final objective after adaptive selection and clipping. The additional experiments also show little separation between full BYPO and the simpler second-order-only variant. Overall, I remain unconvinced by the contribution and maintain my weak rejection.
Response to Reviewer mc63 (1/2)
Thank you for your detailed feedback! We have responded to the weaknesses and questions.
Weakness 1: whether asynchronous RL
Thank you for pointing out the need to accurately describe the application scenario. In fact, our work does not target "asynchronous RL"; our previous use of this term was inaccurate. Instead, our method addresses rollout staleness or policy mismatch, which introduces bias into the objective function because the behavior policy is not updated in a timely manner to match the target policy. This issue also arises in synchronous training with verl, where rollouts for multiple update steps are typically generated in a single batch and subsequently reused for multiple consecutive parameter updates. Consequently, after the first update, the target policy already deviates from the behavior policy. Nevertheless, this is also an important problem that deserves attention, and our baselines, such as TOPR and VESPO, are specifically designed to address this issue.
We acknowledge that the title of our paper and our use of the term "asynchronous RL" may be misleading. In the next revision, we will remove all references to "asynchronous" and replace them with "rollout staleness." We will also change the title to "Debiasing Estimation for Rollout Staleness in Policy Optimization." Given that rollout staleness is an important research problem in its own right, and that BYPO directly addresses this problem, we hope that these revisions will alleviate the reviewer's concerns regarding the applicability and positioning of our work.
We have revised these sections, with the changes highlighted in blue in the latest version of the PDF.
Weakness 2: newer or larger models
Thank you for raising the consideration of larger-scale models. We acknowledge that this is a limitation of the broader claims supported by our experiments, and we will explicitly discuss it in the Limitations section of the revised paper.
Question 1: experiment details
As explained in our response to Weakness 1, we previously used the term "asynchronous RL" inaccurately. In fact, BYPO addresses "rollout staleness" and "policy mismatch," which are also important problems. In our experiments, we use a synchronous but stale RL workflow implemented with verl. The rollout workers generate responses to 256 prompts in each rollout batch. In the Qwen7B experiments, these responses are used for 8 consecutive parameter updates, while in the other experiments, they are used for 4 consecutive updates. Therefore, the parameter synchronization intervals are 8 and 4 updates, respectively. We have also reported additional experimental results under more frequent synchronization in our response to reviewer wPMY's question.
Each response is used only once, and policy lag accumulates as the number of parameter updates within each rollout cycle increases. For example, in the Qwen7B experiments, the KL divergence between the target and behavior policies is typically on the order of 10−4 to 10−3 at the final update step.
Question 2: "synchronous DAPO"
The number of samples used for each parameter update (including both the number of prompts and the number of generated responses), the total number of parameter updates, the minibatch size (which, in verl, corresponds to the number of samples used per parameter update), and the microbatch size (which, in verl, corresponds to the number of samples processed per GPU for each gradient computation) are all kept identical. Therefore, both the number of parameter updates and the number of samples used per update are the same across all methods. The only difference is whether the rollout workers generate samples for multiple parameter-update steps togather or regenerate samples after each parameter update.
Thank you for the concern and we will include all these details in a future revision of the paper.
Question 3: independent training seeds
We have added results from multiple seeds for both the bounded second-order-only variant and full BYPO in the Qwen7B experiments. The following results are based on three runs in total:
Moreover, we have already conducted multiple runs for several key experiments, with the following results:
where BYPO consistently outperforms DAPO and other baselines.