acceptodds
Under review as a conference paper at ICLR 2027

LLMs Can Learn To Reason Via Off-Policy RL

Abstract

Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL is significantly more stable and performant than GRPO with importance sampling across several math reasoning benchmarks when using stale data. We also show that OAPL can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL improve test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Language Modeling, LLM Post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JKMCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This work introduces OAPL (Optimal Advantage-based Policy Optimization with Lagged Inference policy), an off-policy RL algorithm for LLMs. In practice, the current training policy is often different from the policy in the inference engine, leading to an off-policy issue. OAPL treats the discrepancy between the trainer and inference policy as a KL-regularized RL problem and formulates a simple least-squares regression objective by estimating the optimal value function under the sampling distribution. Experiments on math and coding tasks show that OAPL is more stable than GRPO under policy lags.

Critical strengths
  1. This work focuses on an important question, and the proposed OAPL has an elegant formulation.
  2. Experimental results demonstrate that OAPL is stable under policy lags.
Critical weaknesses
  1. The authors decouple β\beta into β1\beta_1 and β2\beta_2 in practice, which violates the theoretical optimality. Besides, it could be difficult to find a good β\beta to meet the optimal policy. This work does not provide a deep investigation into parameter selection, which is important for practical use.
  2. This work adopts a sequence-level sum of log-probability ratios, which is in contrast to token-level GRPO. It is unclear how this variant influences performance. Since GSPO also adopts a sequence-level objective, a comparison is suggested.
  3. The model evaluation is limited. The math and coding tasks are each validated on only a single Qwen-based model (Qwen2.5-Math-1.5B for math and DeepSeek-R1-Distill-Qwen-14B for coding). Evaluations on more LLMs and model families are important to validate the generalization of the method.
  4. The results do not fully validate the effectiveness of OAPL on math. Though GRPO performs worse with longer training, it reaches competitive and even slightly better peak performance than OAPL earlier in training (as shown in Figure 7). In practice, it is fairer to choose checkpoints with the best validation results for comparison.
Questions for authors
  1. Are there any ablation results on different choices of β1\beta_1 and β2\beta_2?
  2. Does the method generalize well to other LLM families and scales?
  3. How does OAPL compare to sequence-level baselines like GSPO?
2 comments
AuthorsOpenReview ↗

Review Response

Thank you for the review and helpful suggestions.

Choice of β₁ and β₂: We agree that the exact optimality identity requires a shared β, and using separate values was an empirical choice inherited from A*PO (Section 5). However, later work analyzing OAPL (Sakhi et al., 2026) examines the split β case, and argues that it targets a more conservative update that stabilizes off-policy training. To test the sensitivity of our results to this choice, we have added a 2×2 sweep over β₁ and β₂ (Section 5, Table 1). Across all four settings, OAPL's pass@8 and pass@16 still outperform our baselines, so the performance gains do not depend on a particular choice of hyperparameters.

Comparison to GSPO: We have now added GSPO at a lag of 50 to our results (see the general response). GSPO matches OAPL in average accuracy, but suffers from severe entropy collapse and underperforms OAPL at pass@8 and pass@16 in every seed. Since GSPO and A*PO both use sequence-level objectives and both collapse, OAPL's pass@k gains appear to come from anchoring to the inference policy rather than from the sequence-level formulation itself.

Model families: Our experiments cover two Qwen-based models at different scales and training regimes: a 1.5B math base model and a 14B distilled reasoning model. Additional model families would be beneficial in our setup, but later work has also already shown that OAPL is effective for training even larger models from a different family (GLM-4.5-Air) on more complex agentic search tasks (KARL; Databricks, 2026), and has been built on for off-policy training of a 744B-parameter model (GLM-5.3) on long-horizon, multi-agent tasks (humans&, 2026).

Peak GRPO checkpoints: We agree that GRPO+TIS is competitive early in training. Appendix B.2 reports results using the best checkpoint for each method, and even under this selection, which favors a method that peaks and then degrades, GRPO+TIS is only comparable to OAPL. We report the final checkpoint because it measures how much performance each method retains under lag, and have added a pointer to the best-checkpoint results in the main text.

Thank you for the response! The additional results have well addressed my questions, and I have raised my rating accordingly.

Reviewer_5VLLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This works propose Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL), an off-policy method built on the A*PO objective. OAPL trains the policy to maximize reward while staying close to the old policy that generated the data. It estimates the best value from a group of sampled responses and trains the policy with a simple squared regression loss, which can be applied to stale data without importance sampling. On math reasoning with Qwen2.5-Math-1.5B, OAPL outperforms GRPO with truncated importance sampling on six benchmarks and stays stable with a lag of 50 steps. On code generation, OAPL matches DeepCoder on LiveCodeBench with a lag of about 400 steps and roughly 3× fewer training samples.

Critical strengths

I think the problem studied in this paper is valuable. How to use off-policy RL to train LLMs more efficiently while keeping good performance and stable training is very meaningful. The experimental results on Qwen2.5-Math-1.5B shown in the manuscript are also very encouraging, especially the stable training with a large policy lag.

Critical weaknesses

(1) The loss and most design choices come from A*PO. I think the main change is just using the inference policy as the reference. The manuscript says the method works on stale data because of the KL-regularized optimum, but with two different β\beta values and a very small β2\beta_2, this argument does not really hold.

(2) The manuscript only shows math results with one small model and only one baseline (GRPO+TIS). There is also no comparison with A*PO or other off-policy methods, and no ablation on the key hyperparameters. The code comparison with DeepCoder is also not controlled, and the best checkpoint is selected on the test set. Finally, although the motivation is asynchronous training, there is no wall-clock time or throughput is reported in the experiments, which is not complete.

Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Review Response

Thank you for the detailed review and suggestions.

Novelty over A*PO: We agree that OAPL's loss is adapted from A*PO, and that the key change is using the lagged inference policy as the reference. To isolate the importance of that choice, we have now added A*PO with a lag of 50 to our results (see the general response). Across three seeds, A*PO matches OAPL on average accuracy but finishes with much lower entropy and Pass@16 below the base model. OAPL improves on the base model at Pass@16 (Section 6.1, Figures 3–4). Beyond these performance differences, OAPL also avoids three sources of computational and memory overhead associated with A*PO's fixed-reference design. Specifically, OAPL obtains its reference log-probabilities from the inference engine and estimates V∗V^* from its own rollouts, eliminating the need for a frozen reference model altogether. This avoids (1) APO's upfront sampling from the reference policy, (2) maintaining a full copy of the base model alongside the trainer and inference engine (or offloading and reloading it every step), a cost that grows with model size, and (3) the per-step forward pass through the reference model to compute APO's fixed-reference denominator. We have now clarified these performance and efficiency benefits in Section 3.2 of the submission.

Split β: We agree that the exact optimality identity requires a shared β, and using separate values was an empirical choice inherited from A*PO (Section 5). However, later work analyzing OAPL (Sakhi et al., 2026) examines the split beta case, and argues that it targets a more conservative update that stabilizes off-policy training. To test the sensitivity of our results to this split, we have added a 2×2 sweep over β₁ and β₂ (Section 5, Table 1). Across all four settings, OAPL's pass@8 and pass@16 still outperform our baselines, so the performance gains are not sensitive to a particular choice of hyperparameters.

Baselines and model scale: In addition to A*PO, we have added GSPO at a lag of 50 as a more recent baseline. Like A*PO, GSPO matches OAPL in average accuracy but suffers from entropy collapse and underperforms OAPL at pass@8 and pass@16. For our DeepCoder comparison, we aligned our evaluation with the original DeepCoder report, which also evaluated on their best checkpoint (Luo et al., 2025). While the evaluation is not perfectly controlled (due to resource constraints we did not rerun their on-policy training), it does show that OAPL is sample efficient at large-scales. Later work has also shown that OAPL is effective for training even larger models from a different family (GLM-4.5-Air) on more complex agentic search tasks (KARL; Databricks, 2026), and has been built on for off-policy training of a 744B-parameter model (GLM-5.3) on long-horizon, multi-agent tasks (humans&, 2026).

Wall-clock time: Our math experiments use a small 1.5B model on a single node, so differences in training time due to synchronization are minor in this setting. We instead use a controlled policy lag to test robustness to stale rollouts, which arise naturally in highly asynchronous training with larger models and longer contexts (Noukhovitch et al., 2025; Fu et al., 2025; Piché et al., 2025). We have added discussion of this in Section 1.

Following up on your concern that OAPL's main change over A*PO is using the inference policy as the reference: we have now also run fixed-reference A*PO at L = 1, with the same hyperparameters as OAPL (see our general update and Appendix B.5). A*PO collapses in all three seeds at L = 1, after its training reward rises well above its fixed V^∗\hat V^{\ast} estimate, while OAPL is stable at both L = 1 and L = 50 and retains higher Pass@k than A*PO at L = 50. This indicates that re-anchoring both the reference and V^∗\hat V^{\ast} to the rollout policy, rather than the regression objective itself, is what makes the method stable. We hope this addresses your concern about OAPL's novelty relative to A*PO.

Hi,

Thank you for the detailed response! I agree the problem is important and the stability under large policy lag is encouraging. However, with the very small KL coefficient used in practice, the update behaves essentially like a standard policy gradient, so it is still unclear whether the gains come from using the lagged inference policy as the reference or from how the value baseline is estimated. The code checkpoints are also selected on the test set.

I actually find the direction valuable and encourage the authors to further strengthen the work for a future submission! I will keep my score for now.

Thanks!

Reviewer_YnxAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors present an alternative objective for post-training LLMs that avoids importance sampling to mitigate performance degradation when the token likelihoods under the inference engine policy that generated the tokens drift too far from those of the trainer policy that is being used to compute gradients. This objective directly pushes the trainer policy to remain close to the inference engine policy using a KL loss term, and trains the trainer policy using an off-policy loss function. The authors show that their objective allows RL post training to maintain its performance on math benchmarks even when there is a policy lag of L=50 (and GRPO degrades severely).

Critical strengths
  1. The main strength of the paper is its novelty with respect to tackling a widespread problem of policy lag in a manner that avoids importance sampling, shifting from a variant of on-policy loss function to an off-policy one. Removing πold\pi_{\text{old}} from the objective entirely to control policy lag while minimizing divergence from the inference policy is a fairly novel approach to tackling high-lag settings.
  2. Experimentally, the authors provide strong evidence for their claims that GRPO performance degrades under high policy lag, while OAPL performance does not. In practice, I expect such an approach to improve training stability significantly. Additionally, the coding results demonstrate impressive performance increases even when trained on a limited number of samples, showing that the off-policy approach exhibits better sample efficiency.
Critical weaknesses
  1. I am unsure if the performance improvements reported in the experiments section can fully be attributed to the choice of moving to an off-policy objective by removing πold\pi_{\text{old}}, which I find to be the authors’ most novel proposal. This is because, in order to do so, they also adopt A*PO’s objective, which may indicate that the performance increase is primarily due to the adoption of the optimal advantage in the objective. In order to attribute the performance increase to off-policy sample usage, the authors should compare to A*PO, the on-policy equivalent, and examine if it degrades like GRPO due to policy lag.
  2. The math experiments reported in the paper have been performed on one, much older, smaller model (Qwen2.5-Math-1.5B) alongside one baseline (GRPO), and thus the results, while impressive, are not directly comparable to more recent work on online approaches. Thus, it is not clear whether more recent online algorithms like GSPO, DAPO, etc. would a) exhibit similar degradation and b) have greater performance increases.

References

  1. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS 2025.
  2. Zheng et al. Group Sequence Policy Optimization. arXiv:2507.18071.
Questions for authors
  1. Do the authors have any evidence or results to show that it is primarily the usage of off-policy samples that improves performance and prevents degradation, and not just utilizing optimal advantage with respect to a consistent reference policy?
  2. Did the authors observe any speedups or slowdowns in training time over GRPO due to the usage of off-policy samples in high-lag settings? I think that adding some discussion of the performance-training speed tradeoff can better contextualize whether the infrequent synchronization operations are as expensive as maintaining L=1 for GRPO.
  3. Can the authors describe if they observe similar degradation in GRPO at lower amounts of policy lag? The current set of experimental results seems to suggest that GRPO performance only degrades at high levels of lag (L=50), and I feel that the paper would be greatly improved if the authors can provide some insights into how degradation scales with increase in policy lag.
3 comments
AuthorsOpenReview ↗

Review Response

Thank you for the thoughtful questions and comments.

A*PO vs. the off-policy reference: We have now added A*PO with a lag of 50 to our results. Across three seeds, fixed-reference A*PO reaches similar average accuracy but lower final Pass@8 and Pass@16 than OAPL (Section 6.1, Figures 3–4). Beyond these performance differences, OAPL also avoids three sources of computational and memory overhead associated with A*PO's fixed-reference design. Specifically, OAPL obtains its reference log-probabilities from the inference engine and estimates V∗V^* from its own rollouts, eliminating the need for a frozen reference model altogether. This avoids (1) APO's upfront sampling from the reference policy, (2) maintaining a full copy of the base model alongside the trainer and inference engine (or offloading and reloading it every step), a cost that grows with model size, and (3) the per-step forward pass through the reference model to compute APO's fixed-reference denominator. We have now clarified these performance and efficiency benefits in Section 3.2 of the submission.

Additional Baselines: We have now added GSPO at a lag of 50 to our results. GSPO matches OAPL in Avg@32, but suffers from a similar entropy collapse to A*PO, underperforming OAPL at pass@8 and pass@16. These results are given in Section 6.1 (Figures 3 and 4; exact numbers in Appendix B.4).

Training time and efficiency: Our math experiments use a small 1.5B model and are run on a single node, so the differences in training time due to synchronization are relatively minor here. Because of that, we use our controlled policy lag as a way to test robustness to staleness in highly asynchronous settings. For larger models and longer generations, this kind of lag arises naturally in asynchronous systems that overlap generation and training (Noukhovitch et al., 2025; Piché et al., 2025); AReaL, for example, allows rollouts up to 8 optimizer steps stale for math (Fu et al., 2025). We have added additional discussion and clarification on this in Section 1.

Intermediate lag: We only used lag values of 1 and 50 for our experiments. The truncated importance sampling correction we apply for GRPO is designed to correct for mismatch between the sampling and training policies (Yao et al., 2025), which is effective for smaller lag values but begins to fail at larger values of lag. As one example, AReaL (Fu et al., 2025, Table 2) runs a similar experiment on a 1.5B math model with staleness from 1 to 16 steps: standard PPO degrades as staleness grows, and even their decoupled correction loses accuracy beyond 8 steps.

Thank you for the clarifications, and additional A*PO and GSPO experiments at lag 50. I would urge authors to include 1) A*PO and GSPO results at lag 1 (to better quantify their degradation at lag 50) and 2) increase model diversity if possible and reproduce GRPO vs OAPL degradation for other models. I think the current results show that OAPL achieves modest improvements over A*PO and GSPO at lag 50, but I think my initial concerns about attributing gains to A*PO and generalizability to other models remain due to the missing A*PO lag=1 experiment and lack of model diversity.

Based on the current paper results, I will maintain my current positive score - I would only be open to increasing my score if my generalizability concerns are addressed with more models.

Thank you for engaging with our rebuttal and for the constructive suggestions.

  • Lag-1 baselines: We have added GSPO and fixed-reference A*PO at L = 1 (three seeds each; Section 6.1, with exact numbers in Table 9 and training curves in Figure 10, Appendix B.5). GSPO has higher average accuracy than OAPL at L = 1 (Avg@32: 35.7% vs. 31.3%) but degrades at L = 50 (31.7%), while OAPL is unchanged (31.8%) and keeps higher Pass@8 and Pass@16 at both lags. A*PO shows the opposite pattern: it is stable at L = 50 but collapses in all three seeds at L = 1. We attribute this to its fixed value estimate: as the policy improves, A*PO's advantages become increasingly positive and the policy sharpens until training collapses, whereas OAPL re-estimates V^∗\hat V^{\ast} from each rollout group (analysis in Appendix B.5). This indicates that OAPL's advantage over A*PO comes from anchoring both the reference and V^∗\hat V^{\ast} to the rollout policy, rather than from the optimal-advantage regression alone.

  • Model diversity: We agree that another model family would strengthen the paper, and we will add results for all four methods at both lags on at least one additional model family in the final version; this is not feasible within the remaining discussion period. However, independent follow-up work already applies OAPL beyond Qwen and at much larger scale, to GLM-4.5-Air for agentic search (KARL; Databricks, 2026) and to off-policy training of GLM-5.3 (744B) on long-horizon tasks (humans&, 2026), showing that this approach is generally applicable.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response To All Reviewers

We thank the reviewers for their comments. All three asked how OAPL compares to A*PO, how it compares with newer baselines such as GSPO, and how sensitive it is to the choice of β. We have added these comparisons at policy lag L = 50:

Method (L = 50)Avg@32Pass@8Pass@16Token entropy
Base model19.845.7152.58–
GSPO (3 seeds)31.745.90 ± 0.2349.20 ± 0.020.016
A*PO, fixed reference (3 seeds)32.745.73 ± 0.2549.35 ± 0.550.032
OAPL (3 seeds)31.850.48 ± 0.1955.25 ± 0.131.221
The three methods reach similar average accuracy, but differ substantially in their pass@k performance. Across three seeds, both A*PO and GSPO finish with low entropy and lower Pass@16 than the base model. OAPL retains higher entropy and improves on the base model at Pass@8 and Pass@16 (Figures 3–4). In a 2×2 β sweep over β_1 and β_2, OAPL's Pass@8 and Pass@16 stay above GSPO's in all four settings, showing that both OAPL’s stable training and its improved pass@k are not sensitive to the choice of hyperparameters.
Revisions are marked in red in the updated PDF: the new baseline comparison is in Section 6.1 (Figures 3 and 4; exact numbers in Appendix B.4), the discussion of A*PO in Section 3.2, the β sweep in Section 5 (Table 1), and best-checkpoint results in Appendix B.2.
AuthorsOpenReview ↗

Additional Results

Baselines at L = 1: Following Reviewer YnxA's suggestion, we have also run GSPO and fixed-reference A*PO at L = 1 (three seeds each; Appendix B.5 of the revised PDF).

  • GSPO reaches higher average accuracy than OAPL at L = 1 (Avg@32: 35.7% vs. 31.3%), but degrades under lag (31.7% at L = 50), while OAPL is essentially unchanged (31.8%). OAPL retains higher Pass@8 and Pass@16 at both lags (Pass@16: 55.5% and 55.3%, versus 51.3% and 49.2% for GSPO), and GSPO's Pass@16 falls below the base model (52.6%) even at L = 1.
  • A*PO is stable at L = 50 but collapses at L = 1 in all three seeds (between steps 350 and 465), using the same hyperparameters as OAPL. Because its V^∗\hat V^{\ast} is fixed at the reference policy, its advantages become increasingly positive as the policy improves, and the policy sharpens until training collapses. OAPL re-estimates V^∗\hat V^{\ast} from each rollout group and is stable at both lags. We have added additional discussion of this in Appendix B.5.

Together, these results indicate that OAPL's robustness comes from anchoring both the reference and V^∗\hat V^{\ast} to the rollout policy, rather than from the sequence-level regression objective it shares with A*PO.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.