acceptodds
Under review as a conference paper at ICLR 2027

STABLE ALIGNMENT: ADAPTIVE KL CONTROL WITH GROUP-NORMALIZED ADVANTAGE ESTIMATION FOR RLHF

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become a standard approach for aligning large language models, yet policy-gradient training remains notoriously unstable: reward scale changes across batches, group-level statis tics are estimated from only a few samples, and KL divergence from the ref erence policy drifts unless the penalty coefficient β is carefully tuned. We in troduce Stable Alignment (SA), a lightweight modification of RLVR/PPO-style training that couples two mechanisms. First, G-NAE (Group-Normalized Ad vantage Estimation) centers advantages within each prompt group and rescales them using a pooled within-group variance estimate. Pooling removes between group difficulty shifts while reducing variance of the scale estimator by O(1/N) in batch size, avoiding both noisy per-group scaling and the between-group at tenuation of global normalization. Second, AKC (Adaptive KL Control) uses an incremental proportional–integral controller on log β so that empirical KL tracks a user-specified budget. Because G-NAE makes the policy update approx imately invariant to reward scale, the KL response to β becomes more station ary, which is precisely the regime in which fixed-gain feedback is reliable. On a controlled verifiable-reward benchmark, SA matches the final accuracy of GRPO and global-normalization baselines while providing penalty-coefficient robustness across three orders of magnitude of βinit and closed-loop KL-budget tracking. Empirically, the pooled scale estimator has about 3 × lower cross-batch coeffi cient of variation than the per-group estimator, supporting the theoretical variance reduction.

Primary area:
reinforcement learning
Keywords:
RLHF、SA、RLVR、KL

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ZM3NOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Stable Alignment, which combines two mechanisms for PPO/GRPO-style training. G-NAE centers rewards within each prompt group and normalizes them using a pooled within-group variance estimate. AKC updates the KL penalty coefficient through an incremental PI controller in log space, aiming to keep policy divergence from the reference model near a specified target.

The authors motivate the approach through variance-estimation analysis and a linear closed-loop model, and evaluate it on a modular-addition task over Z17\mathbb{Z}_{17}. The reported results show final performance comparable to the baselines, reduced variability in the normalization scale estimate, and similar final rewards across different initial penalty coefficients.

Critical strengths

Strength 1: The problem is practically relevant. The interaction between advantage normalization and KL regularization can affect policy optimization stability and hyperparameter tuning costs.

Strength 2: The method is simple and has a reasonable statistical motivation. Under appropriate independence and homoscedasticity assumptions, pooling within-group residuals reduces variance-estimation noise. Combining this approach with feedback control is a direction worth investigating.

Strength 3: The authors appropriately qualify some empirical conclusions. The paper explicitly acknowledges that it does not demonstrate a meaningful final-accuracy advantage and that the synthetic benchmark does not establish benefits for large language models. The ablations separating normalization and control are also useful.

Critical weaknesses

Weaknesses 1: The characteristic polynomial in Proposition 3 is inconsistent with the recurrence being analyzed, invalidating the stated stability guarantee. Equation (10) gives:

Δst=−γKPΔst−1−γKIst−1+Δξt\Delta s_t = -\gamma K_P\Delta s_{t-1} - \gamma K_I s_{t-1} + \Delta\xi_t

Ignoring noise and using Δst=st−st−1\Delta s_t = s_t - s_{t-1} yields:

st=[1−γ(KP+KI)]st−1+γKPst−2s_t = [1-\gamma(K_P+K_I)]s_{t-1} + \gamma K_P s_{t-2}

The corresponding characteristic polynomial is therefore:

z2+[γ(KP+KI)−1]z−γKP=0z^2 + [\gamma(K_P+K_I)-1]z - \gamma K_P = 0

However, Proposition 3 instead uses:

z2+(γKP−1)z+γKI=0z^2 + (\gamma K_P-1)z + \gamma K_I = 0 These polynomials describe different recurrence relations. This discrepancy materially affects the claimed operating range. With the paper’s gains KP=0.7K_P=0.7, KI=0.12K_I=0.12, and γ=2\gamma=2, which lies within the stated range, the correct polynomial becomes: z2+0.64z−1.4=0z^2+0.64z-1.4=0 Its roots are approximately 0.906 and -1.546. Since the latter has magnitude greater than one, the noiseless linear system is not asymptotically stable.

For the positive gains considered here, stability of the correct recurrence requires: γ(2KP+KI)<2\gamma(2K_P+K_I)<2 which gives γ<1.316\gamma<1.316 for the chosen parameters. This does not cover the claimed interval [0.5,2][0.5,2]. This finding does not establish that the implemented algorithm, which includes clipping and nonlinear dynamics, necessarily diverges. It does establish that the current theoretical guarantee is incorrect. The authors need to correct the derivation, revise the operating conditions, and explain how the local response coefficient γ\gamma is estimated or validated in practice.

Weaknesses 2: The reported results do not adequately support the descriptions of KL tracking and controller convergence. The target KL is d=0.15d=0.15, whereas Table 3 reports final-window mean KL values of: 0.193,0.215,0.1610.193,\qquad 0.215,\qquad 0.161 for the three initializations. The default initialization therefore produces a mean KL approximately 43% above the target. These results may indicate some regulatory effect, but they do not establish accurate budget tracking. A finite-horizon discrepancy does not itself refute asymptotic convergence; however, the paper needs to demonstrate control quality within the training horizon it actually evaluates. Section 4.2 also states that the penalty coefficients enter a similar operating region within roughly ten updates and settle near 0.2. Figure 5 does not appear to support this description: the trajectory starting from the smallest coefficient remains below approximately 10−210^{-2} near the end of training, while the trajectory starting from the largest coefficient remains around 10010^0. Furthermore, Table 1 reports a KL standard deviation of 0.043 for SA versus 0.034 for GRPO. Although these numbers alone do not establish statistical significance, they do not directly support lower KL variability for SA. The paper should distinguish three properties: reduced scale-estimation variability, final-performance robustness to initialization, and accurate KL tracking. The evidence supports these properties to substantially different degrees.

Weaknesses 3: The incremental contribution is insufficiently established without comparisons against existing adaptive controllers. Adaptive KL control already appears in the implementation accompanying Ziegler et al. [1], which adjusts the penalty coefficient according to the discrepancy between observed and target KL. Related controller-based multiplier updates have also been studied in constrained reinforcement learning: Stooke et al. [2] introduce PID Lagrangian methods to address oscillation and overshoot. Although these methods are not identical to AKC, they provide relevant precedents and motivate stronger comparisons.

The paper compares fixed penalties with AKC and applies AKC to multiple normalization schemes. However, it does not directly compare AKC against an existing adaptive KL controller or a simpler integral-only update. Consequently, robustness to the initial penalty coefficient may primarily reflect the general benefit of feedback adaptation. The current experiments do not sufficiently establish the additional value of the proposed incremental PI design. Such comparisons should use the same normalization scheme, KL target, and comparable tuning budgets.

The proposed coupling also requires stronger evidence. The claim that G-NAE stabilizes the KL response to the penalty coefficient is central to the motivation, but this response is not directly measured. The existing ablations do not fully establish that mechanism.

References

[1] Daniel M. Ziegler et al. Fine-Tuning Language Models from Human Preferences. arXiv:1909.08593, 2019.

[2] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive Safety in Reinforcement Learning by PID Lagrangian Methods. ICML 2020

Questions for authors

Question 1: Can the authors explain the discrepancy between Equation (10) and the characteristic polynomial in Proposition 3, and provide corrected stability conditions? Is γ\gamma estimated from actual training dynamics?

Question 2: How should the rapid-convergence statement in Section 4.2 be reconciled with Figure 5 and the persistent KL offsets in Table 3? Can the authors provide the corresponding raw trajectories and clarify how the statistics are computed?

Question 3: Under identical normalization and comparable tuning budgets, does AKC consistently outperform existing adaptive KL controllers and simpler integral-only updates?

Question 4: Do the fixed controller gains remain effective when reward scale, prompt heterogeneity, or target KL changes? What direct evidence shows that G-NAE stabilizes the KL response to the penalty coefficient?

1 comment

Since we are near the end of the discussion period and the authors have still not replied to my questions, I will not change my rating. In particular, they have not addressed Equation (10), where the characteristic polynomial in Proposition 3 is inconsistent with the recurrence being analyzed, invalidating the stated stability guarantee.

Reviewer_uMCqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Stable Alignment (SA), which combines two modifications to PPO/GRPO-style training: G-NAE centers rewards within each prompt group and normalizes them using a pooled within-group variance estimate; AKC uses an incremental PI controller in log⁡β\log\beta space to keep empirical KL close to a specified target. The authors analyze the properties of the variance estimator and a simplified closed-loop system, and conduct experiments on a modulo-17 addition task using small character-level models. They report accuracy comparable to the baselines, lower variability in scale estimates, and robustness to the initial KL coefficient.

Critical strengths

The method is simple and has a clear statistical motivation. Within-group centering removes prompt-specific additive reward offsets, while pooling residuals across groups reduces the uncertainty of per-group variance estimation. The core variance formula in Proposition 1 holds under the stated assumptions of independent, homoscedastic Gaussian noise, providing an interpretable starting point for studying how normalization affects optimization.

Critical weaknesses
  1. The core stability analysis contains a derivation error, and the theoretical presentation lacks rigor.

Ignoring noise, Equation (10) implies:

st=[1−γ(KP+KI)]st−1+γKPst−2,s_t=[1-\gamma(K_P+K_I)]s_{t-1}+\gamma K_Ps_{t-2},

so the correct characteristic polynomial is:

z2+[γ(KP+KI)−1]z−γKP,z^2+[\gamma(K_P+K_I)-1]z-\gamma K_P,

rather than the polynomial given in Proposition 3. For positive parameters, the stability condition for the correct recurrence is γ(2KP+KI)<2\gamma(2K_P+K_I)<2. With the paper's choices of KP=0.7,KI=0.12K_P=0.7,K_I=0.12, this requires γ<1.316\gamma<1.316, contradicting the claimed applicability to γ∈[0.5,2]\gamma\in[0.5,2]. At γ=2\gamma=2, one characteristic root is approximately −1.546-1.546, making the system unstable. This error therefore materially affects a core theoretical conclusion.

Moreover, Proposition 1 concerns the statistical properties of the variance estimator; it does not directly imply greater stability of the standard deviation, normalized gradients, or actual training dynamics. The variance decomposition in Proposition 2 also does not establish that global normalization leads to worse optimization performance. The paper has yet to establish a theoretical link between more stable scale estimates and a more stable KL response.

Notation and definition issues further hinder understanding and verification. For example, Equation (2) does not explicitly define ρg,i\rho_{g,i} or specify the granularity at which it is computed; ϵ\epsilon is used for both PPO clipping and denominator stabilization; and μall\mu_{\mathrm{all}}, u′u', and the difference notation are not clearly explained. The stability analysis also directly replaces the clipped error with the unclipped error without adequately specifying the scope of the analysis. The discussion further conflates properties of variance and standard-deviation scales.

  1. The experiments do not sufficiently support the main claims of stability and synergy, and suffer from presentation problems.

The main experiment supports accuracy comparable to the baselines but does not demonstrate a clear stability advantage. In Table 1, SA has a KL standard deviation of 0.0430.043, higher than GRPO's 0.0340.034; its mean KL over the last 10 updates is 0.2150.215, substantially above the target of 0.150.15. Figure 2 likewise does not show lower KL or reward fluctuations for SA than for the baselines. Thus, the reduction in variability of static variance estimates has not translated into clear training-stability benefits.

Figure 4(a) provides some support for the robustness of accuracy to βinit\beta_{\mathrm{init}}, but primarily compares SA with an adaptive coefficient against GRPO with a fixed coefficient. Without results for GRPO+AKC and GlobalNorm+AKC on the same initialization grid, it is unclear whether the gains require G-NAE or reflect synergy between the two modules.

Some textual conclusions are inconsistent with the figures. Lines 322–323 state that βt\beta_t trajectories from different initializations enter a similar operating region within about 10 updates and settle near 0.20.2. However, in Figure 5(a), the trajectory with the low initialization remains below 10−210^{-2} at the end, while the trajectory with the high initialization remains close to 11. Figure 5(b) also does not show tight budget tracking.

The figure presentation also requires correction. The right panel of Figure 6 is visibly truncated and overlaps with the adjacent panel. Figure 3(b) labels both red and green curves, but the green trajectory cannot be independently distinguished. The authors should clarify the purpose of this comparison, the meaning of the shaded regions, and whether the reported CV corresponds to a per-group estimate or a batch-level statistic. These issues hinder verification of the core empirical claims.

Questions for authors
  1. Could you correct the characteristic polynomial and stability condition corresponding to Equation (10), and specify the range over which the selected controller gains remain valid after correction? Have you measured the local sensitivity γ\gamma during actual training to support the simplified model?

  2. How do you explain the discrepancy between Figure 5(a) and the convergence description in lines 322–323? Also, why is the mean KL of the default SA configuration substantially above the target? Please clarify when KL is measured, how the controller is implemented, and the clipping range for β\beta.

  3. How do GRPO+AKC and GlobalNorm+AKC perform in terms of budget error, overshoot, and accuracy on the same βinit\beta_{\mathrm{init}} grid? These comparisons would directly affect the assessment of synergy between the modules.

  4. Could you provide comparative training results for the proposed method and baselines on real-world large language models?

Reviewer_cuDnOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Stable Alignment (SA), a lightweight modification of RLVR/PPO-style training designed to resolve two major instability sources in RLHF: high-variance advantage normalization and brittle KL penalty tuning. SA integrates two complementary techniques: Group-Normalized Advantage Estimation (G-NAE) and Adaptive KL Control (AKC). G-NAE centers advantages within each prompt group while scaling them by a pooled within-group variance estimate across the batch, reducing scale-estimator variance by a factor of O(1/N) without suffering from between-group attenuation AKC utilizes a velocity-form proportional-integral (PI) controller on log β to dynamically track a user-specified empirical KL target. On a controlled modular-addition benchmark, SA matches the final accuracy of GRPO and global normalization baselines, achieves a roughly 3x reduction in scale-estimator variability, and ensures robust KL regulation across three orders of magnitude of β_init.

Critical strengths
  1. The method integrates seamlessly into standard critic-free PPO/GRPO pipelines without extra reward models or complex architectures. Replacing per-group denominators and updating penalty coefficients introduces minimal computational overhead.
  2. G-NAE addresses high-variance noise in per-group scaling using classical pooled within-group variance estimation. It preserves within-group rankings while theoretically reducing scale variance by a factor of ‭O(1/N)O(1/N)‬‭‬‭‬‭‬‭‬.
  3. AKC effectively eliminates the need for manual ‭β\beta‬ tuning by automatically regulating the KL penalty. Performance remains stable across three orders of magnitude of ‭βinit\beta_{init}‬, providing valuable engineering utility.
Critical weaknesses
  1. Validation is restricted entirely to a toy 1.5M-parameter character-level model on a synthetic modular-addition task. There is no evaluation on realistic LLMs or practical RLVR benchmarks involving code, mathematics, or long responses.
  2. The proposed method merely matches GRPO's final reward (‭0.9570.957‬) and falls behind GlobalNorm (‭0.9680.968‬). Furthermore, its tracking error reaches up to ‭0.0650.065‬ relative to a ‭0.150.15‬ budget, providing no clear downstream quality gain.
  3. Proposition 1 relies on equal reward variances across prompts. In realistic binary-reward tasks (‭p(1−p)p(1-p)‬‭‬‭‬‭‬‭‬), variances diverge significantly across easy, medium, and hard prompts, violating homoscedasticity and potentially distorting relative gradient weights across groups.
  4. The claimed synergy between G-NAE and AKC lacks sufficient empirical proof. Adding AKC to GRPO yields comparable final performance, and the paper omits direct comparisons of controller oscillation, convergence speed, or gain sensitivity across normalizations.
Questions for authors
  1. Can the authors provide results on realistic open-source LLMs evaluated on standard RLVR benchmarks (e.g., GSM8K, MATH, HumanEval) to justify relevance to modern alignment pipelines?
  2. How robust is G-NAE when prompt difficulties vary drastically and reward variances are heteroscedastic, as is standard in verifiable binary rewards?
  3. Could you provide comparative ablation metrics (e.g., KL tracking error, controller overshoot, gain sensitivity) demonstrating that G-NAE specifically stabilizes AKC better than standard per-group normalization?
  4. Beyond variance reduction of the scale estimator, what concrete downstream benefits justify adopting Stable Alignment over GRPO or GlobalNorm given the identical final accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.