- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Motivated by reports that removing reference-KL regularization can improve GRPO-style training, the paper examines seven potential mismatches between reference-KL signals and group-relative reward updates: residual KL gradients at clipped tokens, after reward-gradient cancellation, and in identical-reward groups; accumulation of sequence KL with length; possible increases in the relative KL contribution as more tokens are aggregated; concentration of KL on a few tokens; and sampling noise in -based coefficients.
It proposes ZCPO, which computes length-normalized, full-vocabulary conditional KL for each response, centers it within the group, and subtracts it from a max-abs-normalized advantage. The resulting zero-sum coefficients, , replace the advantages in the base surrogate and are detached during optimization. Groups with identical rewards are skipped. For fixed inputs, the coefficient vector is the unique solution of a strictly convex within-group calibration problem.
On Qwen2.5-32B trained on DAPO-Math-17K, ZCPO reaches 56.8/48.5 avg@32 on AIME24/25, compared with 47.3/37.1 for DAPO without KL; RPG/GVPO/GOPO score 48.5–52.6 on AIME24. Main results are three-seed means at compute-matched checkpoints. Ablations support within-group centering, length normalization, reward normalization, and conditional KL. Appendix experiments cover Qwen3-4B long-context training on six benchmarks and GLM-4-9B.
Critical strengths
-
The method is simple and gives substantial gains on Qwen2.5-32B (+9.5/+11.4 points over DAPO without KL), with three-seed means at compute-matched checkpoints. Its reported overhead is about 0.5% relative to standard KL. These results are the main reason I lean toward acceptance.
-
The ablations are useful, particularly the large drop without within-group KL centering. The k1 coefficient control and F7 diagnostics also support using conditional KL in this construction, while the Qwen3-4B and GLM-4-9B results provide evidence beyond the main setting.
Critical weaknesses
-
The improvement over the k1 coefficient control needs stronger validation. This variant retains the other ZCPO components and reaches 54.5/47.1, compared with 56.8/48.5 for full ZCPO. The paper reports three-seed means but not their variation. Separate β tuning and per-seed results would help establish whether the 2.3/1.4-point gap is robust. The method-specific hyperparameters and tuning budgets for RPG/GVPO/GOPO also need clarification.
-
The connection between the seven failure modes and the performance gains remains incomplete. F1 has targeted controls and F3 has a gating ablation, but several other modes are supported mainly by diagnostics. The paper acknowledges this distinction, yet it remains unclear which mechanisms drive most of the gain. Reporting coefficient magnitudes and sign-reversal rates would also help explain how the calibration changes learning.
Questions for authors
-
How large are and at relative to during training, and how often do and have opposite signs?
-
Can you tune separately for the and conditional-KL variants and report per-seed results? Does the 2.3/1.4-point gap persist?
-
What method-specific hyperparameters and tuning budgets were used for RPG, GVPO, and GOPO?
-
What explains the sharp performance drop for independent KL at compared with or no KL? Would a broader sweep and a length-controlled comparison help clarify the cause?