acceptodds
Under review as a conference paper at ICLR 2027

AdaRLHF :Stabilizing RLHF via PID KL Control and LOO Group Advantages

Abstract

Reinforcement learning from human feedback (RLHF)must balance reward improvement against divergence from the reference policy, yet a static KL penalty βeither invites reward hacking or over-constrains exploration,and group normalized advantages of the kind popularized by GRPO remain imprecisely characterized. We present AdaRLHF,a framework that addresses both issues with an emphasis on verifiable analysis.First,we recast KL coefficient tuning as a discrete-time tracking problem and derive an explicit proportional –integral – derivative (PI+D)law with anti-windup and an annealed divergence budget,for which we prove a non-stationary tracking-error bound under mild Schur condi tions (Prop.1).Second,we replace GRPO-style advantages with a self-free,leave one-out group-normalized estimator (LOO-GNAE)and give an exact variance characterization under exchangeable,intra-group correlated rewards (Thm.1), showing that (i)the full-group baseline introduces a standardization bias propor tional to the reward skewness,(ii)the leave-one-out baseline is immune,and (iii) the variance premium of self-free baselines is exactly (G/(G _1))2 ,independent of the correlation . The bias–variance trade-off admits a closed-form interpola tion λ★ (Cor.1). Our analysis also corrects a variance formula reported in prior adaptive-RLHF work.Experiments on six reasoning benchmarks with five seeds show that AdaRLHF improves average accuracy over GRPO by 6.8 points and PPO-RLHF by 9.1 points,reduces advantage variance by 42%,and eliminates the late-training reward-hacking collapse;component ablations attribute the gains and validate each theoretical prediction empirically.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Reinforcement Learning from Human Feedback, RLHF, KL regularization, reward hacking, group advantage estimation, leave-one-out estimation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_5u9mOpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper combines adaptive KL control with leave-one-out group advantage estimates for RLHF. It also proposes a rule for mixing advantage estimates and reports gains on several tasks. The experimental section includes variance analysis, benchmark tests, and component comparisons, but there are problems with the research method and presentation.

Critical strengths
  • Adaptive KL control addresses policy drift is a importan problem in RLHF.
  • The leave-one-out design gives a concrete way to study how group baselines affect advantage estimates.
  • The reported component tests help separate the effects of the controller and the advantage rule.
Critical weaknesses
  • Appendix B treats exchangeable rewards as independent. This does not hold in general and affects the claim that the covariance is zero.
  • Corollary 1 ignores the covariance between estimates based on the same samples. The authors call this a higher-order term, but this does not prove that it is small enough.
  • The overall quality and presentation of this paper do not meet the standards required for submission. In particular, several figures and tables are unclear and make the paper hard to read.
Questions for authors

Please respond to the concerns raised in Critical Weaknesses.

Reviewer_Zb4YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

AdaRLHF combines an adaptive PI+D controller for the KL penalty with a leave-one-out group-normalized advantage estimator and an interpolation with GRPO-style advantages. The paper presents tracking and estimator analyses and reports results on six language-model benchmarks, together with component ablations and model-size comparisons.

Critical strengths

The paper addresses relevant practical issues in RLHF: controlling departure from the reference policy and constructing group-based advantages. The component ablations attempt to separate the effects of these modifications.

Critical weaknesses
  1. The experimental results contain unexplained inconsistencies. The text claims a 6.5-point advantage over GRPO at 72B, while Table 4 reports 96.0 for both methods. The 7B results also differ between Tables 1 and 4 without an explanation of protocol differences. Actual measurements and fitted predictions are not clearly distinguished, making the main performance and scaling claims difficult to verify.

  2. The two central theoretical guarantees are not established. Appendix B incorrectly infers independence from exchangeability and therefore does not establish the claimed removal of standardization bias under correlated rewards. The closed-loop matrix in Proposition 1 also does not correspond to the integral accumulation in Eq. (3), so the analysis does not establish stability for the stated controller.

  3. The paper has severe formatting problems and unclear figures. Text and equations overlap on pages 3, 11, and 12. Numerous cross-references are unresolved, and several figures have low visual clarity, with broken or overlapping labels.

Questions for authors
  1. Please reconcile the 7B results in Tables 1 and 4 and the claimed 6.5-point improvement at 72B. Which results are measured, and which are fitted predictions?

  2. How does exchangeability justify the independence assumption in Appendix B? How does the closed-loop matrix in Proposition 1 account for the integral accumulation in Eq. (3)?

Reviewer_oemVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to improve online RLHF alignment from the two individual aspects: KL divergence penalty and group-normalized advantage calculation. Accordingly, the authors propose to 1) replace the static KL divergence penalty with an adaptive divergence budget, 2) use a LOO-style group-normalized estimator for advantage calculation. These two components together make up the proposed RLHF algorithm, named as AdaRLHF. To support the advantage of AdaRLHF, the authors first present a theoretical analysis, and then conduct empirical analysis using Qwen3 models trained on a mixture of GSM8K, MATH. The reported results show consistent improvement over the baselines including SFT, offline preference learning, and online RLHF methods.

Critical strengths
  1. This paper targets at an important problem in the LLM alignment community.
  2. The selection of baseline methods is of good coverage.
Critical weaknesses
  1. This paper is having a serious formating issue, concerning the fonts, notations, inserted figures.
  2. The propose method lacks of novelty. It is a combination of LOO-style advanatge calculation plus adaptive KL regularization.
  3. The experimental setup is not solid, and I encourage the authors to demonstrate evidence that these baseline methods are properly tuned in hyperparameters.
  4. The claims made in the paper are not supported by its provided evidence. I would like to see: 1) how can AdaRLHF find a better balance between reward hacking and overly restrictive exploration? 2) Is the debiased advantage estimation leading to better alignment performance? 3) What is the underlying mechanism enabling AdaRLHF to eliminate reward hacking?
Questions for authors
  1. Why the training budget is fixed at 400 steps?
  2. Can you provide more convincing evidence such as pareto analysis comparing AdaRLHF and baseline methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.