- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Overall, I find the paper very didactic, with a theoretical motivation guided by empirical practice. The main research question is: "How do delayed updates to the data-generating policy affect learning in the REINFORCE algorithm?" This connects well with on-policy and off-policy learning. Due to my lack of familiarity with the topic and the constrained time to provide a review, I'm not able to fully judge, for instance, the contribution of the provided theoretical results compared to previous work in the literature. My current impression is that the contribution may be somewhat incremental, with limited empirical analyses. I'll remain open to raising my initial score.
UPDATE: I have raised my initial score.
Critical strengths
Clearly written and easy to follow.
Critical weaknesses
While the use of algorithms such as REINFORCE and its applications to LLM fine-tuning serves as motivation for the work, the authors acknowledge in the limitations section that further empirical analyses would be an extension of the work to bridge the theoretical analyses provided with their application. Therefore, I don't think the lack of practical experiments is a weakness due to the theoretical nature of the work. However, I believe the authors could expand the given empirical analyses to highlight the results; for instance, how the initial probability of the best action impacts the results when keeping K and S fixed. This analysis seems to be particularly important due to assumptions made in Thm 3.6.
Questions for authors
I don't have any questions.
3 comments
Thank you. I have updated my initial score to reflect the new additions to the paper.
Thank you for taking the time to read our response and update your score! We are glad to know that you find the new additions useful, and grateful for your constructive suggestions that have helped us a lot in improving the quality of this work.
Response to Reviewer q3Yq
Thank you for your thoughtful and constructive review! We address your questions and concerns in the following.
Question: Contribution of the provided theoretical results compared to previous work in the literature.
We thank the reviewer for this question and appreciate the opportunity to clarify the contribution of our work. To compare our theoretical results with previous work:
[1]: Štrupl, Miroslav, et al. "Reward-weighted regression converges to a global optimum." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 8. 2022.
[2]: Arnal, Charles, et al. "Asymmetric reinforce for off-policy reinforcement learning: Balancing positive and negative rewards, 2025." arXiv preprint arXiv:2506.20520.
[3]: Mei, Jincheng, et al. "On the global convergence rates of softmax policy gradient methods." International conference on machine learning. PMLR, 2020.
[4]: Liu, Jiacai, Wenye Li, and Ke Wei. "Elementary analysis of policy gradient methods." arXiv preprint arXiv:2404.03372.
Suggestions for additional experiments.
We appreciate the reviewer for your valuable suggestion! We agree that empirically examining the role of initialization can make our work stronger. Following your suggestion, we have conducted additional experiments that investigate the impacts of the initial best-action probability x. The empirical results can be found in Figures 3 and 4 in Appendix D.1 of our revised manuscript. It is shown that as x decreases, on-policy RE(1) and RE(S) with small S plateau at the suboptimal gap for a long period, whereas increasing S progressively lowers the threshold of x needed to avoid the trap and enables steady convergence to the global optimum, confirming Theorem 3.6.
Besides these, we have added other experiments to the revised manuscript, including:
Thank you again for your review, which helps us a lot in improving the current work! We hope that our clarifications on comparisons with prior literature and the newly added empirical experiments have provided satisfactory answers to your questions. In light of these revisions, we would greatly appreciate it if you would consider updating your rating of our work accordingly :)