- Rating
- 1
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper revisits KL regularization in RLHF from a gradient-centric perspective. The authors show that unbiased KL value estimation does not necessarily lead to correct gradient estimation. They analyze several commonly used KL estimators and demonstrate that in reward and as loss produce the correct Reverse KL gradient, while as loss introduces gradient bias relative to the RKL objective. Experiments on Qwen models of different scales are conducted to validate these theoretical findings and examine training stability.
Critical strengths
The paper addresses a practically relevant issue in RLHF by highlighting the difference between KL value estimation and gradient estimation. The gradient-centric framework provides a clear and intuitive way to understand different KL regularization implementations. The theoretical analysis is easy to follow, and the experiments across different model scales help support the main findings.
Critical weaknesses
The problem studied in this paper has already been explored in prior work, and the theoretical findings offer limited new insights. Moreover, the experiments do not demonstrate clear practical advantages over existing KL regularization methods, making the overall contribution rather incremental. I acknowledge the authors' explanation in Appendix Q regarding the timeline of independent development. However, my assessment of novelty has to be based on the existing literature available at the time of review.
Questions for authors
Could the authors provide stronger empirical evidence that leads to more stable training than ? The current evidence mainly relies on KL loss spikes, which do not necessarily indicate actual training instability. A comparison of gradient variance or training performance under matched KL divergence levels would help support this claim.
On novelty relative to Tang & Munos (2025)
We thank the reviewers for raising the relation to Tang & Munos (2025), which is the main basis of the novelty concern. We first clarify the chronology; we will respond to the technical points in our replies to each review.
Tang & Munos (arXiv:2506.09477) was first posted on 11 June 2025. The core analysis of this submission, including the gradients of k2 and k3 as losses and the finding that the k3 gradient is a linearization of the k2 gradient, was publicly released more than three months before that date, partly as code merged into a widely used RLHF framework, as documented by timestamped public records. Because these records identify the authors, we cannot share them here without breaking double-blind review. As Reviewer Xntw suggested and Appendix Q offers, we have provided them in a separate confidential comment, visible to the Program Chairs and Senior Area Chairs, and asked them to confirm the chronology without revealing the authors.
As noted in Appendix Q, overlap with later work does not by itself settle originality; the chronology does. We therefore respectfully ask the reviewers to assess novelty against the full chronology rather than the arXiv date of Tang & Munos alone, and to revisit their assessment once it has been verified.