acceptodds
Under review as a conference paper at ICLR 2027

When KL Regularization Misfires in Group Policy Optimization

Abstract

Why does removing reference-policy KL regularization sometimes improve group policy optimization? We argue that this may stem from misaligned KL regularization and imbalanced aggregation, and analyze seven potential failure modes: residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards. We propose Zero-Sum Calibrated Policy Optimization (ZCPO), which coordinates KL regularization with group-relative reward updates through the construction and joint integration of KL coefficients. Mathematical reasoning experiments and ablations support the effectiveness of ZCPO in our settings.

Primary area:
reinforcement learning
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_LLKLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Motivated by reports that removing reference-KL regularization can improve GRPO-style training, the paper examines seven potential mismatches between reference-KL signals and group-relative reward updates: residual KL gradients at clipped tokens, after reward-gradient cancellation, and in identical-reward groups; accumulation of sequence KL with length; possible increases in the relative KL contribution as more tokens are aggregated; concentration of KL on a few tokens; and sampling noise in k1k_1-based coefficients.

It proposes ZCPO, which computes length-normalized, full-vocabulary conditional KL for each response, centers it within the group, and subtracts it from a max-abs-normalized advantage. The resulting zero-sum coefficients, Ci=Ai/sq−βKiC_i = A_i/s_q - \beta K_i, replace the advantages in the base surrogate and are detached during optimization. Groups with identical rewards are skipped. For fixed inputs, the coefficient vector is the unique solution of a strictly convex within-group calibration problem.

On Qwen2.5-32B trained on DAPO-Math-17K, ZCPO reaches 56.8/48.5 avg@32 on AIME24/25, compared with 47.3/37.1 for DAPO without KL; RPG/GVPO/GOPO score 48.5–52.6 on AIME24. Main results are three-seed means at compute-matched checkpoints. Ablations support within-group centering, length normalization, reward normalization, and conditional KL. Appendix experiments cover Qwen3-4B long-context training on six benchmarks and GLM-4-9B.

Critical strengths
  1. The method is simple and gives substantial gains on Qwen2.5-32B (+9.5/+11.4 points over DAPO without KL), with three-seed means at compute-matched checkpoints. Its reported overhead is about 0.5% relative to standard KL. These results are the main reason I lean toward acceptance.

  2. The ablations are useful, particularly the large drop without within-group KL centering. The k1 coefficient control and F7 diagnostics also support using conditional KL in this construction, while the Qwen3-4B and GLM-4-9B results provide evidence beyond the main setting.

Critical weaknesses
  1. The improvement over the k1 coefficient control needs stronger validation. This variant retains the other ZCPO components and reaches 54.5/47.1, compared with 56.8/48.5 for full ZCPO. The paper reports three-seed means but not their variation. Separate β tuning and per-seed results would help establish whether the 2.3/1.4-point gap is robust. The method-specific hyperparameters and tuning budgets for RPG/GVPO/GOPO also need clarification.

  2. The connection between the seven failure modes and the performance gains remains incomplete. F1 has targeted controls and F3 has a gating ablation, but several other modes are supported mainly by diagnostics. The paper acknowledges this distinction, yet it remains unclear which mechanisms drive most of the gain. Reporting coefficient magnitudes and sign-reversal rates would also help explain how the calibration changes learning.

Questions for authors
  1. How large are KiK_i and βKi\beta K_i at β=4\beta=4 relative to Ai/sqA_i/s_q during training, and how often do CiC_i and AiA_i have opposite signs?

  2. Can you tune β\beta separately for the k1k_1 and conditional-KL variants and report per-seed results? Does the 2.3/1.4-point gap persist?

  3. What method-specific hyperparameters and tuning budgets were used for RPG, GVPO, and GOPO?

  4. What explains the sharp performance drop for independent KL at β=0.04\beta=0.04 compared with β=0.001\beta=0.001 or no KL? Would a broader β\beta sweep and a length-controlled comparison help clarify the cause?

Reviewer_fehJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the problem that KL regularization can lead to performance degradation in RLVR. The authors present seven observations (F1–F7) and provide theoretical explanations for them. The paper proposes ZCPO, which attempts to address this problem by adding a response-level, within-group centered KL penalty to the advantage function. Experiments are conducted on Qwen2.5-32B, where ZCPO indeed demonstrates better performance.

Critical strengths
  1. The research problem is valuable. KL regularization is used to prevent policy collapse, but it can also lead to performance degradation in some RLVR settings. Therefore, studying more effective ways to apply KL regularization is indeed important.
  2. The method is clean and only requires modifying the advantage computation, suggesting good transferability and generality.
  3. The experimental results are good, and ZCPO achieves relatively high scores.
Critical weaknesses
  1. Problems with the motivation. Most of F2–F7 are insufficiently rigorous, inaccurate, or of limited usefulness. For example, F2 suggests that $ \sum_i A_i = 0 $ implies reward-gradient cancellation. However, this requires variables such as m and g to be identical, which is difficult to satisfy. Moreover, the fact that KL gradients do not cancel in F2–F3 is not necessarily a bad thing, and this needs to be demonstrated. F4 argues that $ \sum_t \kappa_{i,t} $ increases with response length, but this does not demonstrate a problem with existing methods. For example, DAPO and GRPO both average token-level KL, which already normalizes for length and is different from the summation analyzed in the paper. F5–F7 are overly empirical, without sufficiently strong evidence to demonstrate that they lead to negative effects of KL regularization.
  2. Missing baselines. There is extensive related work on KL regularization. For example, the authors do not consider the most classic approach of incorporating KL directly into the reward: $ r − \beta \log \tfrac{\pi_\theta}{\pi_{ref}} $. Other similar methods should also be explicitly discussed and compared.
  3. Lack of theoretical explanation and motivation for the method. Why does the method use the same KL penalty for every token within a response, rather than different penalties for different tokens as in traditional methods? Why is Eq. 2 chosen as the objective instead of other alternatives? These questions need to be discussed. The authors should directly explain why the method is effective for RLVR, rather than merely restating their design from another perspective.
  4. The experimental results are rather strange. In particular, for DAPO, although the submission uses the same Qwen2.5-32B model and dapo-math dataset, the reported results are much lower than those in the original paper.
  5. Poor readability. The paper is difficult to read, and the logical connections between sections are not easy to follow.
Questions for authors

None

Reviewer_14hJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why removing reference-policy KL regularization can improve group policy optimization. It identifies seven potential failure modes, including residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards.

The proposed ZCPO uses conditional KL to calibrate group-relative reward coefficients, and characterize coefficient optimality for fixed input. The reported math reasoning, long context, cross-model experiments support the effectiveness of ZCPO.

Critical strengths

The problem is concretely formulated. The paper decomposes the empirical observation that removing KL can help into analyzable gradient interactions, including clipping, cancellation, zero-reward groups, length accumulation, and sampling noise. The proposed method is clearly specified and practical to implement.

Critical weaknesses

There is no comparison of computational costs with the baseline models. Computing conditional KL may introduce additional compute, memory, and training-time. Evaluation on a broader range of task types is needed to establish whether the method generalizes across tasks.

Questions for authors

Results across multiple random seeds would make the experimental findings more reliable.

Reviewer_beJoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper analyzes failure modes of KL regularization in LLM RL and proposes an algorithm that fixes these problems. They propose ZCPO which centers length normalized KL and subtracts that from the normalized group advantages. The main concern with this work is that ZCPO is not a novel method at all. It is the standard "in reward KL" shaping which is already implemented as k1 in libraries like verl. It has just been written in an over-complicated way in the paper.

Critical strengths

None

Critical weaknesses

Regarding the failure modes proposed

I have major concerns with the failure modes they have identified, in that they have already been studied by the community and do not add anything to the field. Here is my explanation:

  • F1 and F2: putting KL inside the reward already clips the KL gradients and cancels the shared prefixes which has already been studied in [2]
  • F3: in LLM-RL identical reward groups are removed using Zero Variance Filtering (ZVF)
  • F4: length normalized KL is a standard practice
  • F5: KL centering is also a standard practice
  • F6: KL suppression on critical tokens is a prior work [1]
  • F7: sampling variance and negative values in k1are also well established

Regarding the method

Their proposed method ZCPO is also a standard practice where you normalize the KL penalty and apply group centering.

Missing Baselines

Furthermore, they miss the DAPO with kl-in-reward baseline and compare against unclipped k3 in loss.

[1] Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning, vassoyan et.al. 2025 [2] A Comedy of Estimators: On KL Regularization in RL Training of LLMs, Shah et.al. 2026

Questions for authors

Please see the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.