- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies instability in constrained RLHF, where a policy maximizes helpfulness while satisfying expected safety constraints. It proposes optimistic predictor–corrector updates for both the policy and constraint multipliers. The main theoretical result establishes geometric last-iterate convergence for exact updates in distribution space, with a conditional extension to approximate parameterized policies. Experiments using TinyLlama-1.1B on PKU-SafeRLHF-30K show reduced oscillations and better final reward–constraint scores than the implemented primal–dual baseline.
Critical strengths
- Geometric last-iterate convergence is stronger than an average-iterate guarantee. Handling a unique optimal policy alongside potentially non-unique optimal multipliers is also useful.
- A technically interesting part is that KL regularization supplies curvature in the policy variable, while the Lagrangian remains linear in the multipliers. Their analysis handles this asymmetry and allows multiple optimal multipliers.
- The paper explicitly considers approximate policy updates. Its extension identifies how implementation errors affect final accuracy rather than assuming every neural policy update exactly solves the ideal subproblem.
- The experiments show reduced oscillations and better performance compared to the primal dual baseline.
Critical weaknesses
- The theory analyzes KL-mirror updates, with a natural-gradient formulation for parameterized policies. The implementation uses clipped PPO, evaluates the corrector gradient at the predicted policy which creates a gap. Although the paper is of theoretical nature, however a more NPG style algorithm would have validated the theory better.
- The empirical scope and safety evaluation are limited. Results use one 1.1B model and one dataset, primarily evaluated through learned reward scores. Larger models, multiple simultaneous constraints, independent safety evaluations, and explicit violation rates would provide stronger evidence of general usefulness.
- The strongest contribution appears to be the convergence analysis for this particular KL-regularized constrained RLHF structure. A sharper comparison with existing optimistic constrained-RL results would clarify what new technical obstacle is resolved.
Questions for authors
- Under the assumptions of Theorem 3.4, can standard primal–dual updates also achieve last-iterate convergence with sufficiently small step sizes? A comparison of their convergence rates and admissible step sizes would clarify precisely what optimism improves.
- Assumption 3.6 requires both proximity to the ideal mirror update and control of the increase in KL distance from a comparator, including the unknown optimal policy. Remark B.11 provides sufficient log-ratio and coverage conditions, but can these conditions be established for a concrete parameterized update?
- Although the analysis avoids an explicit inverse minimum response probability, the rate depends on the reference-weighted reward covariance and its associated Hoffman constant. Can the authors characterize this dependence? For example, if two constraints differ only on responses that are extremely unlikely under the reference policy, how does the convergence rate deteriorate? A discussion around the same will be very helpful.
3 comments
The strongest contribution appears to be the convergence analysis for this particular KL-regularized constrained RLHF structure. A sharper comparison with existing optimistic constrained-RL results would clarify what new technical obstacle is resolved.
Thank you for the suggestion. We compare with related work and expanded Appendix A.1 to explain the technical differences. The related works can be divided into two groups. One kind of works studies convergence of the last iterate in constrained RL, but its guarantees do not directly cover our optimistic KL updates for constrained RLHF. The other studies constrained RLHF and provides guarantees for the best iterate or for performance averaged over the iterates, and does not imply convergence of the last iterate.
Ding et al. [1] proves geometric convergence for exact Euclidean optimistic updates in discounted CMDPs. They use the dual objective gap to bound the multiplier distance. With a fixed KL term, that gap can be quadratic near an interior optimum, so the same bound can fail. We instead directly bound the squared distance from the updated multiplier to the optimal multiplier set. We describe this set using policy optimality and complementary slackness. We use a Hoffman bound controls the distance to this set by the residuals of these two conditions, and we use the exact policy and dual updates to bound both residuals by policy error and the gaps between predicted and corrected iterates. The relative density bound allows us to control the log ratios in the policy optimality condition. Also, we prove a conditional approximation guarantee for the optimistic KL updates in Corollary 3.8 .
Zhang et al. [2] alternates Lagrangian maximization over policy parameters with dual descent. They only give guarantees for the best dual iterate with approximate policy optimization and stochastic dual feedback. Du et al. [3] combines DPO with multiplier updates and returns a policy sampled from the training history. The suboptimality and constraint satisfaction guarantees in this work apply only to performance averaged over the iterates. We instead prove the convergence at the last optimistic iterate by showing that the contribution of previous iterate errors geometrically decreases over iterations.
[1] Dongsheng Ding, Chen-Yu Wei, Kaiqing Zhang, and Alejandro Ribeiro. Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs. NeurIPS, 2023.
[2] Botong Zhang, Shuo Li, Ignacio Hounie, Osbert Bastani, Dongsheng Ding, and Alejandro Ribeiro. Alignment of large language models with constrained learning. NeurIPS, 2025.
[3] Yihan Du, Seo Taek Kong, and R. Srikant. Primal-Dual Direct Preference Optimization for Constrained LLM Alignment. arXiv:2510.05703v1, 2025.
Under the assumptions of Theorem 3.4, can standard primal–dual updates also achieve last-iterate convergence with sufficiently small step sizes? A comparison of their convergence rates and admissible step sizes would clarify precisely what optimism improves.
Our current analysis does not establish a convergence guarantee for standard PD under the assumptions of Theorem 3.4 or provide a comparison of convergence rates and admissible step sizes. Instead, our contribution focuses on the geometric last iterate convergence guarantee for OPD in distribution space, without policy averaging or an added dual penalty. Corollary 3.8 further quantifies the effect of inexact policy updates under the stated approximation and relative density assumptions, with exact dual updates.
Assumption 3.6 requires both proximity to the ideal mirror update and control of the increase in KL distance from a comparator, including the unknown optimal policy. Remark B.11 provides sufficient log-ratio and coverage conditions, but can these conditions be established for a concrete parameterized update?
We add a sampled SGD example after Remark B.11. We assume that the feature matrix has full row rank on the finite prompt and response domain, so it can represent every target there. Every pair must have positive sampling probability. The proof first shows that the expected squared fitting error decreases geometrically with the number of SGD steps. It then bounds the change in the log normalizer to obtain a uniform log ratio bound. After inner steps, with probability at least , both oracle errors are at most The uniform bound holds for every comparator, including the unknown optimal policy and the later corrector policy. Allocating failure probabilities across calls gives simultaneous validity of Assumption 3.6.
Although the analysis avoids an explicit inverse minimum response probability, the rate depends on the reference-weighted reward covariance and its associated Hoffman constant. Can the authors characterize this dependence? For example, if two constraints differ only on responses that are extremely unlikely under the reference policy, how does the convergence rate deteriorate? A discussion around the same will be very helpful.
When constraint scores are difficult to distinguish under the reference policy, the Hoffman constant can be large and the certified convergence rate can be slower. With the other constants held fixed, the iteration bound for a fixed error reduction grows quadratically as the Hoffman constant becomes large, as follows from Equations (39) and (40) in Appendix C.3. We want to note that unnder our stated assumptions, the theorem guarantees geometric convergence for each fixed problem even when the Hoffman constant is large.
Moreover, the Hoffman constant depends on the optimization problem, including the constraint scores, reference policy, prompt distribution, and multiplier optimality conditions. When defining the task, careful choices of constraints and prompts can help improve conditioning in two ways.
-
Constraint selection. We can choose constraints that capture different requirements and remove redundancies. For example, safety and efficiency assess different aspects of a response. A response can be safe without being efficient, and efficient without being safe. Such objectives can provide complementary information when their scores distinguish responses in different ways under the reference policy.
-
Prompt selection. We can choose diverse and challenging prompts that reveal meaningful differences in constraint scores. For example, we can include prompts on which the reference policy produces responses with different safety scores.
Carefully choosing constraints and prompts can make the constraint signals easier to distinguish under the reference policy and help reduce the Hoffman constant.
We implemented the exact KL-mirror update with Euclidean multiplier projection, including zero multipliers, and its full-Fisher tabular NPG equivalent. Across the finite experiment configurations, the maximum absolute difference in policy probabilities and multipliers over 60 steps is 2.01×10−13.
For the step size comparison, we use 5 synthetic problems generated with seeds 11 to 15. Each instance has 8 discrete contexts x∈{1,…,8}, each with probability 1/8, and 16 available actions $y\in\lbrace 1,\ldots,16\rbrace $ per context. The policy π(y∣x) is an 8×16 probability table, represented by softmax logits. The contexts and actions are abstract indices without associated text. We maximize reward with a KL penalty weighted by 0.05 relative to a fixed reference policy, subject to three expected reward constraints with threshold zero. The reward tables are constructed to give a known optimal policy and optimal multipliers. Two constraints equal zero at the optimum, while the third equals 0.25 for every policy.
Both methods start from the same reference policy and zero multipliers. We compute all expectations and gradients exactly over the 128 context and action pairs and give each method 10000 policy oracle calls. We measure error Et as the average KL divergence from the known optimal policy to the current policy, plus the squared Euclidean distance between the current and optimal multipliers. The table reports how many of the five instances reach Et≤10−6 at the common oracle budget.
Both methods reach the target at 0.05 and 0.1, and OPD also succeeds at larger steps. At 0.03 neither reaches the target, and PD has smaller error. This validates the finite KL-mirror/NPG dynamics; neural PPO remains an approximate implementation.
We extended the evaluation to Qwen2.5-3B-Instruct using frozen reward and constrained-reward scoring models of the same size. PD and OPD share the same initial policy, PKU-SafeRLHF training split, random seed (101), rank-16 LoRA configuration, and training budget of 400 batches with 32 responses each. Both use policy/critic learning rates of 5e−5/1e−4, a dual step size of 0.005, and a KL weight of 0.05. We evaluate reward and constrained reward on 798 PKU test prompts, harmful responses on 240 HarmBench prompts using its independent Llama-2-13B classifier, and refusal-phrase frequency on 200 benign requests.
Under the same training budget, OPD achieves a higher mean reward (1.246 vs. 0.689) and a lower HarmBench harmful-response rate (12.92% vs. 18.33%). The refusal-phrase rate on benign requests is also slightly lower (15.5% vs. 16.5%).
For multiple simultaneous constraints, we refer to our response to Weakness 1, which details five synthetic tabular instances with two active constraints and one slack constraint. Under the same setting, OPD succeeds at a larger stepsize range.