acceptodds
Under review as a conference paper at ICLR 2027

ADAPTRLHF: ADAPTIVE KL-CONSTRAINED REIN FORCEMENT LEARNING FROM HUMAN FEEDBACK WITH GROUP-NORMALIZED AD VANTAGE ESTIMATION

Abstract

Reinforcement Learning from Human Feedback (RLHF)has emerged as the dom inant paradigm for aligning large language models (LLMs) with human prefer- ences. However, existing RLHF methods suffer from two critical failure modes: (i) static KL penalty coefficients that cannot adapt to non-stationary training dynamics, causing either reward hacking or excessive conservatism; and (ii) high-variance advantage estimates due to unbounded cross-group reward hetero geneity. We propose AdaptRLHF, a novel RLHF framework that addresses both limitations through: (1) an adaptive KL controller that continuously ad justs the divergence penalty βt based on a proportional-integral (PI) feedback law over the observed KL gap; and (2) Group-Normalized Advantage Estima tion (GNAE) that standardizes rewards within semantically coherent generation groups, substantially reducing gradient variance. We further derive a unified objective that subsumes PPO-style clipping and GRPO-style group rewards un der a single theoretical framework. Extensive experiments on five reasoning benchmarks—GSM8K, ARC-Challenge, HumanEval, MMLU, and BIG-Bench Hard—demonstrate that AdaptRLHF achieves state-of-the-art results, outper forming PPO-RLHF and GRPO by +4 . 5% and +2 . 6% on average accuracy, with a measured 30% reduction in policy-gradient variance (Sec. 5.4) and 1.8 × faster convergence. Code and models are anonymously released at https: //anonymous.4open.science/r/AdaptRLHF.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning from Human Feedback, RLHF, Large Language Model, Adaptive KL penalty, Group‑Normalized Advantage Estimation, Advantage estimation, PPO, Model alignment

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_bzbLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript is still at an early draft stage and shows poor overall quality. The anonymous code link (https: //anonymous.4open.science/r/AdaptRLHF) is inaccessible and returns "Unavailable. This content can’t be shown. The repository was not found on GitHub." The architecture diagram in Fig. 1 is confusingly structured, with unclear connection arrow. Line 198 Eq12 has a bad format, and line 177 Theorem 1 contains an obvious typo ("Eq. equation 9"). Based on these issues, I recommend desk rejection without further review.

Critical strengths

N/A

Critical weaknesses

This manuscript is still at an early draft stage and shows poor overall quality. The anonymous code link (https: //anonymous.4open.science/r/AdaptRLHF) is inaccessible and returns "Unavailable. This content can’t be shown. The repository was not found on GitHub." The architecture diagram in Fig. 1 is confusingly structured, with unclear connection arrow. Line 198 Eq12 has a bad format, and line 177 Theorem 1 contains an obvious typo ("Eq. equation 9"). Based on these issues, I recommend desk rejection without further review.

Questions for authors

N/A

Reviewer_6zTbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AdaptRLHF, which combines an adaptive KL controller with Group-Normalized Advantage Estimation (GNAE) to improve RLHF training stability. The authors claim significant improvements over PPO and GRPO across five reasoning benchmarks, along with reduced gradient variance and better scalability.

Critical strengths

The problems of KL control and advantage variance reduction are relevant to RLHF. The proposed framework is relatively simple, and the paper attempts to provide both theoretical analysis and extensive experimental evaluations.

Critical weaknesses

1. The experimental results are highly questionable.

The paper uses a DeBERTa-v3 reward model trained on HH-RLHF-style conversational preferences. Such a reward model cannot directly verify mathematical correctness or execute code tests. Yet the authors report that their method improves LLaMA-3-8B from 74.2% to 86.7% on GSM8K, from 71.4% to 81.5% on MMLU, and from 63.2% to 74.3% on HumanEval.

How can a relatively small conversational preference model, without task-specific correctness verification, produce such enormous improvements in mathematical reasoning and coding? I find these results extremely difficult to believe without much stronger evidence.

2. The scalability experiments contain suspicious inconsistencies.

Figure 7 reports experiments on 1.3B and 30B models, allegedly from the Llama 2/3 and Qwen2.5 families. However, these are not standard released model sizes in the stated families. The authors do not specify the exact checkpoints corresponding to these points.

Were these custom models, approximations of other model sizes, or something else? Such basic inconsistencies seriously undermine confidence in the reported scalability results.

3. The theoretical analysis contains an obvious mathematical error.

During GPT-6-Astra-assisted checking, followed by manual verification, I identified a direct contradiction in Corollary 1.

The authors claim that AdaptRLHF reduces to standard PPO-RLHF with a value baseline V(x)V(x) when G=1G=1 and βt\beta_t is constant.

However, Equations (7)–(9) contain no value network V(x)V(x). More importantly, when G=1G=1, the group mean becomes

μG=Rs(x,y1),\mu_{\mathcal{G}} = R_s(x,y_1),

and therefore

A1=(Rs(x,y1)−muG)/sigmaG=0A_1 = (R_s(x,y_1) - mu_G) / sigma_G = 0

The advantage is identically zero. How could this possibly reduce to standard PPO-RLHF, which generally has nonzero advantages?

Furthermore, Table 2 even reports an ablation called "w/o Value Baseline", although no such baseline is defined in the proposed method.

This is not a subtle technical gap. It is a direct algebraic contradiction in one of the paper's central theoretical claims.

Combined with the highly implausible experimental results and unexplained model configurations, I have serious concerns about the technical correctness and reliability of this paper.

Questions for authors

1.Experimental credibility: How can a DeBERTa-v3 reward model trained on conversational preferences produce such large improvements on mathematical reasoning and coding benchmarks without correctness verification? Can the authors provide the exact reward model checkpoint, training details, and reproducible evaluation results? Please also clarify the exact model checkpoints used for the 1.3B and 30B experiments in Figure 7.

2.Theoretical correctness: How can Corollary 1 claim that AdaptRLHF reduces to standard PPO-RLHF when G = 1, given that Equations (7)–(9) make the group-normalized advantage identically zero in this case? Furthermore, where is the value baseline V(x) defined, and how was the "w/o Value Baseline" ablation implemented?

Reviewer_x95MOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In RLHF training, static KL regularization struggles to balance exploration and policy stability, and can lead to high-variance advantage estimation. To address these issues, the authors integrate a proportional-integral controller for dynamically adjusting the KL penalty and Group-Normalized Advantage Estimation for stabilizing policy-gradient updates. Experiments show that the proposed approache outperforms competing approaches on reasoning benchmarks.

Critical strengths
  1. The experimental results are impressive, indicating that the proposed method significantly outperforms standard baselines.

  2. The training curve is very smooth and beautiful. I have never observed such a perfect training curve in RL post-training experiments.

Critical weaknesses
  1. The motivation is less justified.

  2. No AI usage statement.

Questions for authors

No Question

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.