- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This work introduces OAPL (Optimal Advantage-based Policy Optimization with Lagged Inference policy), an off-policy RL algorithm for LLMs. In practice, the current training policy is often different from the policy in the inference engine, leading to an off-policy issue. OAPL treats the discrepancy between the trainer and inference policy as a KL-regularized RL problem and formulates a simple least-squares regression objective by estimating the optimal value function under the sampling distribution. Experiments on math and coding tasks show that OAPL is more stable than GRPO under policy lags.
Critical strengths
- This work focuses on an important question, and the proposed OAPL has an elegant formulation.
- Experimental results demonstrate that OAPL is stable under policy lags.
Critical weaknesses
- The authors decouple into and in practice, which violates the theoretical optimality. Besides, it could be difficult to find a good to meet the optimal policy. This work does not provide a deep investigation into parameter selection, which is important for practical use.
- This work adopts a sequence-level sum of log-probability ratios, which is in contrast to token-level GRPO. It is unclear how this variant influences performance. Since GSPO also adopts a sequence-level objective, a comparison is suggested.
- The model evaluation is limited. The math and coding tasks are each validated on only a single Qwen-based model (Qwen2.5-Math-1.5B for math and DeepSeek-R1-Distill-Qwen-14B for coding). Evaluations on more LLMs and model families are important to validate the generalization of the method.
- The results do not fully validate the effectiveness of OAPL on math. Though GRPO performs worse with longer training, it reaches competitive and even slightly better peak performance than OAPL earlier in training (as shown in Figure 7). In practice, it is fairer to choose checkpoints with the best validation results for comparison.
Questions for authors
- Are there any ablation results on different choices of and ?
- Does the method generalize well to other LLM families and scales?
- How does OAPL compare to sequence-level baselines like GSPO?
2 comments
Thank you for the response! The additional results have well addressed my questions, and I have raised my rating accordingly.
Review Response
Thank you for the review and helpful suggestions.
Choice of β₁ and β₂: We agree that the exact optimality identity requires a shared β, and using separate values was an empirical choice inherited from A*PO (Section 5). However, later work analyzing OAPL (Sakhi et al., 2026) examines the split β case, and argues that it targets a more conservative update that stabilizes off-policy training. To test the sensitivity of our results to this choice, we have added a 2×2 sweep over β₁ and β₂ (Section 5, Table 1). Across all four settings, OAPL's pass@8 and pass@16 still outperform our baselines, so the performance gains do not depend on a particular choice of hyperparameters.
Comparison to GSPO: We have now added GSPO at a lag of 50 to our results (see the general response). GSPO matches OAPL in average accuracy, but suffers from severe entropy collapse and underperforms OAPL at pass@8 and pass@16 in every seed. Since GSPO and A*PO both use sequence-level objectives and both collapse, OAPL's pass@k gains appear to come from anchoring to the inference policy rather than from the sequence-level formulation itself.
Model families: Our experiments cover two Qwen-based models at different scales and training regimes: a 1.5B math base model and a 14B distilled reasoning model. Additional model families would be beneficial in our setup, but later work has also already shown that OAPL is effective for training even larger models from a different family (GLM-4.5-Air) on more complex agentic search tasks (KARL; Databricks, 2026), and has been built on for off-policy training of a 744B-parameter model (GLM-5.3) on long-horizon, multi-agent tasks (humans&, 2026).
Peak GRPO checkpoints: We agree that GRPO+TIS is competitive early in training. Appendix B.2 reports results using the best checkpoint for each method, and even under this selection, which favors a method that peaks and then degrades, GRPO+TIS is only comparable to OAPL. We report the final checkpoint because it measures how much performance each method retains under lag, and have added a pointer to the best-checkpoint results in the main text.