- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper discusses issues with large scale RL where generation is lagged due to the sheer scale of the training framework. The main issue is that if the generation lags and is created via older policies, the model updates using these policies will not be good (off-policy). This paper talks about construction of ratio-free off-policy objectives that improves training stability.
Critical strengths
The paper is clearly written with clear problem statement. Paper contains detailed proofs of the propositions.
Critical weaknesses
The models and dataset used are not really large scale, also this is single Qwen3 model family, so it is unclear if this method is generalizable. How does this affect the final performance? There seems to be no accuracy result published, does this actually help the final task we are training for? The main contribution seems to focus on OAPL, which obviously has its downsides. But Is this the correct baseline to compare to?
Questions for authors
See weaknesses
3 comments
I still have concerns regarding model family types and size of the models tested. Also, the robustness of the baseline still seems to be an issue, so I will keep my scores.
Thank you for the follow-up. We believe there is a big misunderstanding of the paper’s main contribution.
Model size and scope. Fully fine-tuning 4B and 8B models with RL is already a substantial workload on four H100 GPUs. Our experiments test the theoretical predictions across model sizes, regularization strengths, and policy lags. OAPL already demonstrates effectiveness at scale; our goal is to explain its underlying mechanism.
Baseline robustness is the central question of the paper. OAPL can outperform GRPO but relies on temperature decoupling that its nominal theory does not explain. We explain why this helps and how our advantage shift directly obtains the desired pessimistic regime. The added tuned-OAPL comparison gives rewards close to Lambert, within noise, supporting our explanation and showing comparable performance without an extra advantage temperature to tune.
The baseline’s fragility is therefore exactly the phenomenon we study. We hope this clears up the misunderstanding and helps you assess the paper based on its real contribution.
Thank you for highlighting the clear problem statement, presentation, and detailed proofs. These are exactly the points we want to convey. Our goal is to explain why existing off-policy objectives work and use this understanding to obtain the desired pessimistic regime directly.
Scale and generalization. OAPL already demonstrates that these objectives work at scale. Our contribution is theoretical understanding, supported by experiments that test its predictions. The analysis concerns the learning objective and does not depend on a particular model family. Our experiments on Qwen3-4B and Qwen3-8B test the predicted behavior across model sizes, regularization strengths, and policy lags. Their purpose is to check the proposed explanation in practical training, rather than repeat OAPL’s validation at scale.
Task accuracy is reported. Section 5 explicitly defines “evaluation reward” as held-out accuracy on Minerva, AIME2025, AMC23, and BRUMO. Figure 2 therefore reports task accuracy alongside entropy. It shows that our change can improve both accuracy and stability, including higher final accuracy with lag L=128 than near-on-policy GRPO with L=2.
Choice of baselines. OAPL is the most direct comparison because it uses the same regression loss, allowing us to isolate the effect of advantage normalization. GRPO provides an additional near-on-policy reference.
We have now also tuned OAPL’s advantage temperature β2, keeping β1=10−3; see the table in our response to Reviewer KXAP. Its final rewards are very close to Lambert, with differences within the observed noise. This agrees with Appendix B.1: appropriate temperature decoupling moves OAPL into the pessimistic regime identified by our theory. Our contribution explains this behavior and guarantees the desired regime through a simple advantage shift, without an extra advantage temperature to tune.
We hope this addresses your concerns and makes the paper’s intended scope and the reported accuracy results clearer. We believe these points mainly needed clarification, while our main findings remain intact. If you have no further concerns, we would appreciate it if you reconsidered your score.