- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Stable Alignment, which combines two mechanisms for PPO/GRPO-style training. G-NAE centers rewards within each prompt group and normalizes them using a pooled within-group variance estimate. AKC updates the KL penalty coefficient through an incremental PI controller in log space, aiming to keep policy divergence from the reference model near a specified target.
The authors motivate the approach through variance-estimation analysis and a linear closed-loop model, and evaluate it on a modular-addition task over . The reported results show final performance comparable to the baselines, reduced variability in the normalization scale estimate, and similar final rewards across different initial penalty coefficients.
Critical strengths
Strength 1: The problem is practically relevant. The interaction between advantage normalization and KL regularization can affect policy optimization stability and hyperparameter tuning costs.
Strength 2: The method is simple and has a reasonable statistical motivation. Under appropriate independence and homoscedasticity assumptions, pooling within-group residuals reduces variance-estimation noise. Combining this approach with feedback control is a direction worth investigating.
Strength 3: The authors appropriately qualify some empirical conclusions. The paper explicitly acknowledges that it does not demonstrate a meaningful final-accuracy advantage and that the synthetic benchmark does not establish benefits for large language models. The ablations separating normalization and control are also useful.
Critical weaknesses
Weaknesses 1: The characteristic polynomial in Proposition 3 is inconsistent with the recurrence being analyzed, invalidating the stated stability guarantee. Equation (10) gives:
Ignoring noise and using yields:
The corresponding characteristic polynomial is therefore:
However, Proposition 3 instead uses:
These polynomials describe different recurrence relations. This discrepancy materially affects the claimed operating range. With the paper’s gains , , and , which lies within the stated range, the correct polynomial becomes: Its roots are approximately 0.906 and -1.546. Since the latter has magnitude greater than one, the noiseless linear system is not asymptotically stable.
For the positive gains considered here, stability of the correct recurrence requires: which gives for the chosen parameters. This does not cover the claimed interval . This finding does not establish that the implemented algorithm, which includes clipping and nonlinear dynamics, necessarily diverges. It does establish that the current theoretical guarantee is incorrect. The authors need to correct the derivation, revise the operating conditions, and explain how the local response coefficient is estimated or validated in practice.
Weaknesses 2: The reported results do not adequately support the descriptions of KL tracking and controller convergence. The target KL is , whereas Table 3 reports final-window mean KL values of: for the three initializations. The default initialization therefore produces a mean KL approximately 43% above the target. These results may indicate some regulatory effect, but they do not establish accurate budget tracking. A finite-horizon discrepancy does not itself refute asymptotic convergence; however, the paper needs to demonstrate control quality within the training horizon it actually evaluates. Section 4.2 also states that the penalty coefficients enter a similar operating region within roughly ten updates and settle near 0.2. Figure 5 does not appear to support this description: the trajectory starting from the smallest coefficient remains below approximately near the end of training, while the trajectory starting from the largest coefficient remains around . Furthermore, Table 1 reports a KL standard deviation of 0.043 for SA versus 0.034 for GRPO. Although these numbers alone do not establish statistical significance, they do not directly support lower KL variability for SA. The paper should distinguish three properties: reduced scale-estimation variability, final-performance robustness to initialization, and accurate KL tracking. The evidence supports these properties to substantially different degrees.
Weaknesses 3: The incremental contribution is insufficiently established without comparisons against existing adaptive controllers. Adaptive KL control already appears in the implementation accompanying Ziegler et al. [1], which adjusts the penalty coefficient according to the discrepancy between observed and target KL. Related controller-based multiplier updates have also been studied in constrained reinforcement learning: Stooke et al. [2] introduce PID Lagrangian methods to address oscillation and overshoot. Although these methods are not identical to AKC, they provide relevant precedents and motivate stronger comparisons.
The paper compares fixed penalties with AKC and applies AKC to multiple normalization schemes. However, it does not directly compare AKC against an existing adaptive KL controller or a simpler integral-only update. Consequently, robustness to the initial penalty coefficient may primarily reflect the general benefit of feedback adaptation. The current experiments do not sufficiently establish the additional value of the proposed incremental PI design. Such comparisons should use the same normalization scheme, KL target, and comparable tuning budgets.
The proposed coupling also requires stronger evidence. The claim that G-NAE stabilizes the KL response to the penalty coefficient is central to the motivation, but this response is not directly measured. The existing ablations do not fully establish that mechanism.
References
[1] Daniel M. Ziegler et al. Fine-Tuning Language Models from Human Preferences. arXiv:1909.08593, 2019.
[2] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive Safety in Reinforcement Learning by PID Lagrangian Methods. ICML 2020
Questions for authors
Question 1: Can the authors explain the discrepancy between Equation (10) and the characteristic polynomial in Proposition 3, and provide corrected stability conditions? Is estimated from actual training dynamics?
Question 2: How should the rapid-convergence statement in Section 4.2 be reconciled with Figure 5 and the persistent KL offsets in Table 3? Can the authors provide the corresponding raw trajectories and clarify how the statistics are computed?
Question 3: Under identical normalization and comparable tuning budgets, does AKC consistently outperform existing adaptive KL controllers and simpler integral-only updates?
Question 4: Do the fixed controller gains remain effective when reward scale, prompt heterogeneity, or target KL changes? What direct evidence shows that G-NAE stabilizes the KL response to the penalty coefficient?
Since we are near the end of the discussion period and the authors have still not replied to my questions, I will not change my rating. In particular, they have not addressed Equation (10), where the characteristic polynomial in Proposition 3 is inconsistent with the recurrence being analyzed, invalidating the stated stability guarantee.