acceptodds
Under review as a conference paper at ICLR 2027

Rethinking KL Regularization in RLHF: From Wrong Value Estimation to Correct Gradient Optimization

Abstract

Reinforcement Learning from Human Feedback (RLHF) often uses a Kullback–Leibler (KL) penalty to stabilize training and control drift from a reference policy. However, recent implementations sometimes choose KL estimator formulas based on Monte Carlo value-estimation criteria such as bias and variance, and then reuse these formulas with gradient-carrying policy parameters as optimization losses. We show that this transfer from value estimation to loss design is wrong in general: good KL value estimation does not imply a correct KL gradient. We introduce a gradient-centric framework that compares KL implementations by the scalar coefficient they induce on the policy score function, covering both detached k_n in reward terms and directly differentiated k_n as loss terms. Through this framework, we first prove that k_1 as loss, although an unbiased KL value estimator, has zero expected on-policy gradient and provides no reference-directed regularization. We then prove that the PPO-style k_1 in reward implementation and the decoupled k_2 as loss implementation are gradient-equivalent and recover the Reverse KL (RKL) gradient under on-policy sampling. This equivalence, first proven here, identifies both as theoretically sound implementations of the RKL objective. In contrast, the GRPO-style k_3 as loss objective induces a biased first-order approximation to the principled RKL coefficient, with mismatched tail behavior. We further show that off-policy “as loss” KL terms require explicit importance-sampling corrections. Controlled experiments validate these findings, providing a comprehensive, gradient-based rationale for choosing and correctly implementing KL regularization in RLHF.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RLHF, KL Value Estimation, KL Loss, Gradient Analysis, k3

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kthoOpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper revisits KL regularization in RLHF from a gradient-centric perspective. The authors show that unbiased KL value estimation does not necessarily lead to correct gradient estimation. They analyze several commonly used KL estimators and demonstrate that k1k_1 in reward and k2k_2 as loss produce the correct Reverse KL gradient, while k3k_3 as loss introduces gradient bias relative to the RKL objective. Experiments on Qwen models of different scales are conducted to validate these theoretical findings and examine training stability.

Critical strengths

The paper addresses a practically relevant issue in RLHF by highlighting the difference between KL value estimation and gradient estimation. The gradient-centric framework provides a clear and intuitive way to understand different KL regularization implementations. The theoretical analysis is easy to follow, and the experiments across different model scales help support the main findings.

Critical weaknesses

The problem studied in this paper has already been explored in prior work, and the theoretical findings offer limited new insights. Moreover, the experiments do not demonstrate clear practical advantages over existing KL regularization methods, making the overall contribution rather incremental. I acknowledge the authors' explanation in Appendix Q regarding the timeline of independent development. However, my assessment of novelty has to be based on the existing literature available at the time of review.

Questions for authors

Could the authors provide stronger empirical evidence that k2k_2 leads to more stable training than k3k_3? The current evidence mainly relies on KL loss spikes, which do not necessarily indicate actual training instability. A comparison of gradient variance or training performance under matched KL divergence levels would help support this claim.

Reviewer_XntwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines the gradients induced by different KL implementation in RLHF. It focus on the specific design choice that treat the reward that based on KL divergence as differentiable or non-differentiable. The following analysis establishes the on-policy gradient equivalent of k1k_1 in reward and k2k_2 as loss. It also calculates the gradients induced by k1k_1 and k3k_3 as losess and dicusses importance-sampling corrections for off-policy updates.

Critical strengths

The experiments compare KL loss variants across multiple model scales and regularization strengths. These comparisons provide a useful empirical reference for how the different implementations behave during LLM training.

Critical weaknesses
  1. In Section 4.1 the mathematical presentation mixes the objective with its computational implementation. At line 190, the quantity labeled J(θ)J(\theta) denotes a score-function surrogate for the KL term. It does not represent the expected KL term in the original RL objective. Its intended gradient requires holding both the sampling distribution and the coefficient fixed. The distinction between defining an objective and constructing a loss to reproduce its gradient is therefore essential. Although lines 200–201 identify these expressions as surrogates, the notation does not make this distinction sufficiently clear. At line 177, detachment notation appears within an already explicit policy-gradient identity. Its mathematical role at that stage is unclear. These "conventions" repeatedly mix mathematical definitions with instructions for automatic differentiation. The presentation can be improved if derive the target gradient from the original objective. Then define each surrogate and state whether, and under what conditions, its gradient matches that target.
  2. In section 5.1, the k1k_1-as-loss example provides limited new insight. The formulation introduced at line 258 differentiates the log-ratio while holding the sampling distribution fixed. This does not implement the REINFORCE gradient estimator for the original KL objective. It omits the score-function term arising from the policy-dependent sampling distribution. The remaining term has zero expectation by the standard identity Eπθ[∇θlog⁡πθ]=0\mathbb E_{\pi_\theta}[\nabla_\theta\log\pi_\theta]=0. The authors themselves invoke this identity. Tang and Munos [1] also explicitly state this result in their Equation (2). The example provides a useful illustration of an implementation pitfall. However, its significance as a main theoretical contribution needs clearer justification.
  3. The conceptual contribution of Section 5.2 needs clearer positioning. Equation (3) follows from the policy gradient theorem. Under the stated on-policy convention, Theorem 5.1 follows by differentiating the squared log-ratio. This produces the same gradient as the detached score-function surrogate. The equivalence helps connect the two implementations, but its additional theoretical significance remains unclear to me. Section 3 of Tang and Munos [1] also describes these constructions and their gradient equivalence. I appreciate that Appendix Q acknowledges this work and discusses development chronology. I cannot assess priority from the information provided. A precise technical comparison would help clarify the relationship between the results. More broadly, I would appreciate a clearer explanation of what the unified framework adds to our understanding of KL regularization beyond organizing these gradient identities.

[1] Yunhao Tang and Rémi Munos. On a few pitfalls in KL divergence gradient estimation for RL. arXiv:2506.09477, 2025.

Questions for authors
  1. Could you clarify what additional insight the “in reward” versus “as loss” framework provides beyond standard policy-gradient analysis? A specific theoretical result or practical implication would help me assess its contribution. It would also be helpful to explain how this relates to the equivalent gradient constructions in Tang and Munos [1, Section 3].

  2. Could you clarify the basis for the abstract’s statement that the gradient equivalence is “first proven here”? I understand the anonymity concerns discussed in Appendix Q. If the supporting information cannot be shared anonymously, could the relevant timeline and evidence be provided confidentially to the AC, as proposed there?

Reviewer_8JD8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the KL penalty toward a reference policy should be implemented in RLHF. The main idea is to write every KL implementation as a detached scalar coefficient on the score function ∇θlog⁡πθ(y∣x)\nabla_\theta\log\pi_\theta(y|x). This makes two styles comparable at the gradient level: using knk_n as a detached reward term, and differentiating knk_n directly as a loss. Under on-policy sampling, Section 5 shows three things:

  • k1k_1 as loss has zero expected gradient.
  • k1k_1 in reward and k2k_2 as loss both recover the reverse-KL gradient, as stated in Theorem 5.1.
  • GRPO's k3k_3 as loss induces the coefficient 1−πref/πθ1-\pi_{\text{ref}}/\pi_\theta, a first-order approximation with mismatched tails.

Section G further derives importance-sampling and clipping corrections for off-policy KL losses. Experiments train Qwen2.5-Math and Qwen2.5 models from 1.5B to 32B on 7,300 math prompts with fully on-policy updates, comparing no KL with k1k_1, k2k_2, and k3k_3 as loss.

Critical strengths
  • The gradient-centric view makes sense and is clean. Reducing every implementation to its score-function coefficient makes code differences such as detaching a term directly comparable. Figure 1 and Table 2 summarize the consequences clearly. The view also covers practical variants; for example, Appendix M shows that the importance-weighted DeepSeek-V3.2 loss recovers the k1k_1 coefficient.
  • The tail analysis gives a useful explanation for GRPO's KL head. The coefficient induced by k3k_3 as loss saturates at 1 when πθ≫πref\pi_\theta\gg\pi_{\text{ref}}, and its variance equals χ2(πref∣πθ)\chi^2(\pi_{\text{ref}}|\pi_\theta). This explains why the constraint is weak exactly when drift is large, and the k3k_3 KL-loss spike at β = 0.001 in Figure 3 is consistent with it.
Critical weaknesses
  1. Limited novelty relative to existing work. The three findings in Section 5 already appear in Tang & Munos [1]:

    • Differentiating k1k_1 as a loss gives zero expected gradient.
    • Differentiating k2=12(log⁡π/πref)2k_2=\frac12(\log\pi/\pi_{\text{ref}})^2 gives E[log⁡(π/πref)∇log⁡π]\mathbb{E}[\log(\pi/\pi_{\text{ref}})\nabla\log\pi], which is the same as using the detached log-ratio as a reward coefficient.
    • k3k_3 as loss follows the gradient of KL(πref,π)\mathrm{KL}(\pi_{\text{ref}},\pi) rather than the target KL.

    Therefore, the abstract's claim that the equivalence in Theorem 5.1 is "first proven here" is not accurate. Also, Shah et al. [2] compare k1k_1 and k3k_3 both in the reward and as a loss, and show on Qwen2.5-7B and Llama-3.1-8B that only k1k_1 in reward is unbiased on-policy. Zhang et al. [3], published at ICLR 2026 and not cited, correct the off-policy importance weighting of GRPO's KL term, which overlaps with Section G.

    The paper mentions [1] and [2] only in Appendix Q, where it calls them subsequent or contemporaneous, although [1] appeared in June 2025. Both are cited but not seriously engaged. The paper should discuss them in the main text and state what is new beyond them. To me, the remaining contributions are the unified coefficient table, the χ2\chi^2 tail analysis, and the treatment of the DeepSeek-V3.2 and MiniMax-01 variants, which makes the paper incremental.

  2. The theory is sequence-level, but practical KL heads are token-level. Footnote 1 treats the whole response as a single action. However, GRPO applies k3k_3 per token and averages it, and [1] shows that token-level KL losses give at best a partial gradient of the sequence-level KL. As a result, it is unclear whether the recommendation in Table 2 to use k2k_2 as loss holds for the per-token implementations that practitioners actually use. The authors should either analyze the token-level case or state which level their experiments implement.

Questions for authors

See weaknesses

Comments on the paper

1 comment
AuthorsOpenReview ↗

On novelty relative to Tang & Munos (2025)

We thank the reviewers for raising the relation to Tang & Munos (2025), which is the main basis of the novelty concern. We first clarify the chronology; we will respond to the technical points in our replies to each review.

Tang & Munos (arXiv:2506.09477) was first posted on 11 June 2025. The core analysis of this submission, including the gradients of k2k_2 and k3k_3 as losses and the finding that the k3k_3 gradient is a linearization of the k2k_2 gradient, was publicly released more than three months before that date, partly as code merged into a widely used RLHF framework, as documented by timestamped public records. Because these records identify the authors, we cannot share them here without breaking double-blind review. As Reviewer Xntw suggested and Appendix Q offers, we have provided them in a separate confidential comment, visible to the Program Chairs and Senior Area Chairs, and asked them to confirm the chronology without revealing the authors.

As noted in Appendix Q, overlap with later work does not by itself settle originality; the chronology does. We therefore respectfully ask the reviewers to assess novelty against the full chronology rather than the arXiv date of Tang & Munos alone, and to revisit their assessment once it has been verified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.