acceptodds
Under review as a conference paper at ICLR 2027

Last-Iterate Convergence of Optimistic Primal–Dual Methods for KL-Regularized Constrained RLHF

Abstract

Reinforcement Learning from Human Feedback (RLHF) plays a significant role in aligning Large Language Models (LLMs) with human preferences. RLHF with expected reward constraints can be formulated as a KL-regularized saddle-point problem, in which simultaneous primal–dual updates can exhibit oscillatory dynamics. We study an optimistic predictor–corrector method that applies optimism to both the policy and the constraint multipliers. In the distributional policy space, we prove geometric last-iterate convergence to the regularized saddle-point set. For parameterized policies, we give an inexact-oracle extension whose residual separates primal solver error and comparator coverage under an additional uniform relative-density bound. Experiments with a PPO-style surrogate on PKU-SafeRLHF-30K illustrate reduced oscillation and improved final-iterate robustness over the implemented PPO-Lagrangian baseline.

Primary area:
optimization
Keywords:
reinforcement learning, constrained optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_s5xEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies instability in constrained RLHF, where a policy maximizes helpfulness while satisfying expected safety constraints. It proposes optimistic predictor–corrector updates for both the policy and constraint multipliers. The main theoretical result establishes geometric last-iterate convergence for exact updates in distribution space, with a conditional extension to approximate parameterized policies. Experiments using TinyLlama-1.1B on PKU-SafeRLHF-30K show reduced oscillations and better final reward–constraint scores than the implemented primal–dual baseline.

Critical strengths
  1. Geometric last-iterate convergence is stronger than an average-iterate guarantee. Handling a unique optimal policy alongside potentially non-unique optimal multipliers is also useful.
  2. A technically interesting part is that KL regularization supplies curvature in the policy variable, while the Lagrangian remains linear in the multipliers. Their analysis handles this asymmetry and allows multiple optimal multipliers.
  3. The paper explicitly considers approximate policy updates. Its extension identifies how implementation errors affect final accuracy rather than assuming every neural policy update exactly solves the ideal subproblem.
  4. The experiments show reduced oscillations and better performance compared to the primal dual baseline.
Critical weaknesses
  1. The theory analyzes KL-mirror updates, with a natural-gradient formulation for parameterized policies. The implementation uses clipped PPO, evaluates the corrector gradient at the predicted policy which creates a gap. Although the paper is of theoretical nature, however a more NPG style algorithm would have validated the theory better.
  2. The empirical scope and safety evaluation are limited. Results use one 1.1B model and one dataset, primarily evaluated through learned reward scores. Larger models, multiple simultaneous constraints, independent safety evaluations, and explicit violation rates would provide stronger evidence of general usefulness.
  3. The strongest contribution appears to be the convergence analysis for this particular KL-regularized constrained RLHF structure. A sharper comparison with existing optimistic constrained-RL results would clarify what new technical obstacle is resolved.
Questions for authors
  1. Under the assumptions of Theorem 3.4, can standard primal–dual updates also achieve last-iterate convergence with sufficiently small step sizes? A comparison of their convergence rates and admissible step sizes would clarify precisely what optimism improves.
  2. Assumption 3.6 requires both proximity to the ideal mirror update and control of the increase in KL distance from a comparator, including the unknown optimal policy. Remark B.11 provides sufficient log-ratio and coverage conditions, but can these conditions be established for a concrete parameterized update?
  3. Although the analysis avoids an explicit inverse minimum response probability, the rate depends on the reference-weighted reward covariance and its associated Hoffman constant. Can the authors characterize this dependence? For example, if two constraints differ only on responses that are extremely unlikely under the reference policy, how does the convergence rate deteriorate? A discussion around the same will be very helpful.
3 comments

The theory analyzes KL-mirror updates, with a natural-gradient formulation for parameterized policies. The implementation uses clipped PPO, evaluates the corrector gradient at the predicted policy which creates a gap. Although the paper is of theoretical nature, however a more NPG style algorithm would have validated the theory better.

We implemented the exact KL-mirror update with Euclidean multiplier projection, including zero multipliers, and its full-Fisher tabular NPG equivalent. Across the finite experiment configurations, the maximum absolute difference in policy probabilities and multipliers over 60 steps is 2.01×10−132.01\times10^{-13}.

For the step size comparison, we use 5 synthetic problems generated with seeds 11 to 15. Each instance has 8 discrete contexts x∈{1,…,8}x\in\lbrace 1,\ldots,8\rbrace, each with probability 1/8, and 16 available actions $y\in\lbrace 1,\ldots,16\rbrace $ per context. The policy π(y∣x)\pi(y\mid x) is an 8×16 probability table, represented by softmax logits. The contexts and actions are abstract indices without associated text. We maximize reward with a KL penalty weighted by 0.05 relative to a fixed reference policy, subject to three expected reward constraints with threshold zero. The reward tables are constructed to give a known optimal policy and optimal multipliers. Two constraints equal zero at the optimum, while the third equals 0.25 for every policy.

Both methods start from the same reference policy and zero multipliers. We compute all expectations and gradients exactly over the 128 context and action pairs and give each method 10000 policy oracle calls. We measure error EtE_t as the average KL divergence from the known optimal policy to the current policy, plus the squared Euclidean distance between the current and optimal multipliers. The table reports how many of the five instances reach Et≤10−6E_t\leq 10^{-6} at the common oracle budget.

Effective step size0.030.050.10.20.30.6
PD0/55/55/50/50/50/5
OPD / exact NPG0/55/55/55/55/55/5

Both methods reach the target at 0.05 and 0.1, and OPD also succeeds at larger steps. At 0.03 neither reaches the target, and PD has smaller error. This validates the finite KL-mirror/NPG dynamics; neural PPO remains an approximate implementation.

The empirical scope and safety evaluation are limited. Results use one 1.1B model and one dataset, primarily evaluated through learned reward scores. Larger models, multiple simultaneous constraints, independent safety evaluations, and explicit violation rates would provide stronger evidence of general usefulness.

We extended the evaluation to Qwen2.5-3B-Instruct using frozen reward and constrained-reward scoring models of the same size. PD and OPD share the same initial policy, PKU-SafeRLHF training split, random seed (101), rank-16 LoRA configuration, and training budget of 400 batches with 32 responses each. Both use policy/critic learning rates of 5e−5/1e−4, a dual step size of 0.005, and a KL weight of 0.05. We evaluate reward and constrained reward on 798 PKU test prompts, harmful responses on 240 HarmBench prompts using its independent Llama-2-13B classifier, and refusal-phrase frequency on 200 benign requests.

MethodReward ↑Constrained Reward ↑Harmful (%) ↓Benign refusal strings (%)
PD0.6891.65318.3316.5
OPD1.2461.30312.9215.5

Under the same training budget, OPD achieves a higher mean reward (1.246 vs. 0.689) and a lower HarmBench harmful-response rate (12.92% vs. 18.33%). The refusal-phrase rate on benign requests is also slightly lower (15.5% vs. 16.5%).

For multiple simultaneous constraints, we refer to our response to Weakness 1, which details five synthetic tabular instances with two active constraints and one slack constraint. Under the same setting, OPD succeeds at a larger stepsize range.

The strongest contribution appears to be the convergence analysis for this particular KL-regularized constrained RLHF structure. A sharper comparison with existing optimistic constrained-RL results would clarify what new technical obstacle is resolved.

Thank you for the suggestion. We compare with related work and expanded Appendix A.1 to explain the technical differences. The related works can be divided into two groups. One kind of works studies convergence of the last iterate in constrained RL, but its guarantees do not directly cover our optimistic KL updates for constrained RLHF. The other studies constrained RLHF and provides guarantees for the best iterate or for performance averaged over the iterates, and does not imply convergence of the last iterate.

Ding et al. [1] proves geometric convergence for exact Euclidean optimistic updates in discounted CMDPs. They use the dual objective gap to bound the multiplier distance. With a fixed KL term, that gap can be quadratic near an interior optimum, so the same bound can fail. We instead directly bound the squared distance from the updated multiplier to the optimal multiplier set. We describe this set using policy optimality and complementary slackness. We use a Hoffman bound controls the distance to this set by the residuals of these two conditions, and we use the exact policy and dual updates to bound both residuals by policy error and the gaps between predicted and corrected iterates. The relative density bound allows us to control the log ratios in the policy optimality condition. Also, we prove a conditional approximation guarantee for the optimistic KL updates in Corollary 3.8 .

Zhang et al. [2] alternates Lagrangian maximization over policy parameters with dual descent. They only give guarantees for the best dual iterate with approximate policy optimization and stochastic dual feedback. Du et al. [3] combines DPO with multiplier updates and returns a policy sampled from the training history. The suboptimality and constraint satisfaction guarantees in this work apply only to performance averaged over the iterates. We instead prove the convergence at the last optimistic iterate by showing that the contribution of previous iterate errors geometrically decreases over iterations.

[1] Dongsheng Ding, Chen-Yu Wei, Kaiqing Zhang, and Alejandro Ribeiro. Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs. NeurIPS, 2023.

[2] Botong Zhang, Shuo Li, Ignacio Hounie, Osbert Bastani, Dongsheng Ding, and Alejandro Ribeiro. Alignment of large language models with constrained learning. NeurIPS, 2025.

[3] Yihan Du, Seo Taek Kong, and R. Srikant. Primal-Dual Direct Preference Optimization for Constrained LLM Alignment. arXiv:2510.05703v1, 2025.

Under the assumptions of Theorem 3.4, can standard primal–dual updates also achieve last-iterate convergence with sufficiently small step sizes? A comparison of their convergence rates and admissible step sizes would clarify precisely what optimism improves.

Our current analysis does not establish a convergence guarantee for standard PD under the assumptions of Theorem 3.4 or provide a comparison of convergence rates and admissible step sizes. Instead, our contribution focuses on the geometric last iterate convergence guarantee for OPD in distribution space, without policy averaging or an added dual penalty. Corollary 3.8 further quantifies the effect of inexact policy updates under the stated approximation and relative density assumptions, with exact dual updates.

Assumption 3.6 requires both proximity to the ideal mirror update and control of the increase in KL distance from a comparator, including the unknown optimal policy. Remark B.11 provides sufficient log-ratio and coverage conditions, but can these conditions be established for a concrete parameterized update?

We add a sampled SGD example after Remark B.11. We assume that the feature matrix has full row rank on the finite prompt and response domain, so it can represent every target there. Every pair must have positive sampling probability. The proof first shows that the expected squared fitting error decreases geometrically with the number of SGD steps. It then bounds the change in the log normalizer to obtain a uniform log ratio bound. After NN inner steps, with probability at least 1−δ1-\delta, both oracle errors are at most CrN/2δ,0<r<1.\frac{C r^{N/2}}{\sqrt\delta}, 0<r<1. The uniform bound holds for every comparator, including the unknown optimal policy and the later corrector policy. Allocating failure probabilities across calls gives simultaneous validity of Assumption 3.6.

Although the analysis avoids an explicit inverse minimum response probability, the rate depends on the reference-weighted reward covariance and its associated Hoffman constant. Can the authors characterize this dependence? For example, if two constraints differ only on responses that are extremely unlikely under the reference policy, how does the convergence rate deteriorate? A discussion around the same will be very helpful.

When constraint scores are difficult to distinguish under the reference policy, the Hoffman constant can be large and the certified convergence rate can be slower. With the other constants held fixed, the iteration bound for a fixed error reduction grows quadratically as the Hoffman constant becomes large, as follows from Equations (39) and (40) in Appendix C.3. We want to note that unnder our stated assumptions, the theorem guarantees geometric convergence for each fixed problem even when the Hoffman constant is large.

Moreover, the Hoffman constant depends on the optimization problem, including the constraint scores, reference policy, prompt distribution, and multiplier optimality conditions. When defining the task, careful choices of constraints and prompts can help improve conditioning in two ways.

  1. Constraint selection. We can choose constraints that capture different requirements and remove redundancies. For example, safety and efficiency assess different aspects of a response. A response can be safe without being efficient, and efficient without being safe. Such objectives can provide complementary information when their scores distinguish responses in different ways under the reference policy.

  2. Prompt selection. We can choose diverse and challenging prompts that reveal meaningful differences in constraint scores. For example, we can include prompts on which the reference policy produces responses with different safety scores.

Carefully choosing constraints and prompts can make the constraint signals easier to distinguish under the reference policy and help reduce the Hoffman constant.

Reviewer_1dmfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies last-iterate stability in KL-regularized constrained RLHF. The authors formulate helpfulness maximization under expected safety-reward constraints as a primal–dual saddle-point problem, and study an optimistic method with predictor and corrector updates for both the policy and the constraint multipliers. Theorem 3.4 proves geometric last-iterate convergence for exact KL-mirror updates in a finite distributional policy space, while an inexact-oracle extension gives conditional error bounds for parameterized policies. Experiments with a PPO-based implementation on TinyLlama-1.1B and PKU-SafeRLHF-30K report smaller reward oscillations and better final reward–constraint scores than a standard primal–dual baseline.

Critical strengths
  1. The focus on the final iterate is well motivated. In constrained RLHF, the policy at the end of training is often the one actually deployed, so stability of that iterate matters more than a guarantee for an average of past policies. The KL-specific distribution-space analysis addresses a meaningful optimization question.

  2. The authors connect the formal analysis to a neural-policy setting with detailed experiments. The experiments include three seeds, a learning-rate sweep, and a report of computational overhead, which makes the empirical picture more informative.

Critical weaknesses
  1. The incremental contribution over prior optimistic primal–dual work is not sufficiently clear. Ding et al. [1] already study optimistic policy-gradient primal–dual updates for constrained MDPs and establish linear last-iterate convergence. The fixed KL-to-reference objective and the inexact-oracle analysis here may require additional arguments, but I found it difficult to tell precisely what is new relative to those results. A more direct comparison of the algorithms, assumptions, and convergence statements would help establish the theoretical contribution.

  2. There is a gap between the algorithms analyzed and the one used in the LLM experiments. Theorem 3.4 concerns exact population KL-mirror updates, and Algorithm 3 uses Fisher natural policy gradients. Section 4 instead describes clipped PPO and says the corrector gradient is evaluated at the predicted policy; Algorithm 3 evaluates both primal gradients at the current corrected policy. The practical dual update also uses log-multiplier updates, smoothed batch costs, and a multiplier range of [0.5, 10], rather than the projected Euclidean update used in the analysis. These choices may be reasonable engineering decisions, but they change the update dynamics. I do not see evidence that the stated convergence assumptions hold for this particular implementation; the positive multiplier lower bound could also exclude an optimal multiplier equal to zero.

  3. The empirical evidence for a general practical advantage is still limited. The main LLM evaluation uses one 1.1B model and one preference dataset, and safety is assessed primarily through learned constrained-reward scores. I would like to see a more direct measure of safety constraint satisfaction or harmful behavior. There is also a meaningful computational cost: the reported step times are 991 ms for OPD versus 714 ms for PD, with higher peak memory. Comparing the methods only by training step makes it hard to judge the benefit at matched wall-clock time or compute, especially without a more extensively tuned PD baseline.

[1] Ding, Dongsheng, et al. "Last-iterate convergent policy gradient primal-dual methods for constrained MDPs." Advances in Neural Information Processing Systems 36 (2023): 66138-66200.

Questions for authors
  1. Section 2.1 states that the reward models are trained using the Bradley–Terry pairwise preference objective, while Section 4 enforces an expected constrained reward of at least zero. As I understand it, the Bradley–Terry loss only depends on relative reward differences. Adding the same offset to the reward scores of two responses for a given prompt does not change their predicted preference probability, but it does change the absolute expected reward. How is the zero threshold determined? What does an expected constrained reward of zero represent in terms of actual safety or constraint satisfaction?

  2. Could the authors show λtλ_t for OPD and PD, and report an explicit constraint-violation metric? This would help assess whether the reduced training oscillation is accompanied by more stable multiplier dynamics and better constraint satisfaction.

5 comments

The incremental contribution over prior optimistic primal–dual work is not sufficiently clear. Ding et al. [1] already study optimistic policy-gradient primal–dual updates for constrained MDPs and establish linear last-iterate convergence. The fixed KL-to-reference objective and the inexact-oracle analysis here may require additional arguments, but I found it difficult to tell precisely what is new relative to those results. A more direct comparison of the algorithms, assumptions, and convergence statements would help establish the theoretical contribution.

Our contribution is a convergence analysis of optimistic KL updates for the fixed objective in finite constrained RLHF formulation. We establish geometric convergence for exact updates and quantify how approximation errors affect the policy and multipliers at the last iterate.

1. Geometric convergence for the fixed KL objective. Ding et al.'s Theorem 6 analyzes exact Euclidean optimistic updates. In their proof of Lemma 21, Equation (39a) bounds the distance to the optimal multiplier set by a constant times the dual objective gap. This bound need not hold after adding the fixed KL term, since the gap can be quadratic in the multiplier distance near an interior optimum. Under the stated assumptions, initialization, and step sizes, Theorem 3.4 proves that the last policy converges geometrically to the constrained optimum and the distance to the optimal multiplier set converges geometrically to zero. Appendix C.3 instead relates the distance to the optimal multiplier set to the policy error and the gaps between predicted and corrected iterates. These estimates allow the decrease in our error measure at each step to control both the remaining policy error and the distance to the optimal multiplier set, completing the geometric convergence proof.

2. Approximation guarantees for optimistic updates. Ding et al.'s Theorem 4 extends their regularized method in Theorem 2 to function approximation. Their paper does not give a corresponding approximation guarantee for their optimistic method. On the other hand, our Corollary 3.8 provides a conditional approximation guarantee for optimistic KL updates in Corollary 3.8.

There is a gap between the algorithms analyzed and the one used in the LLM experiments. Theorem 3.4 concerns exact population KL-mirror updates, and Algorithm 3 uses Fisher natural policy gradients. Section 4 instead describes clipped PPO and says the corrector gradient is evaluated at the predicted policy; Algorithm 3 evaluates both primal gradients at the current corrected policy. The practical dual update also uses log-multiplier updates, smoothed batch costs, and a multiplier range of [0.5, 10], rather than the projected Euclidean update used in the analysis. These choices may be reasonable engineering decisions, but they change the update dynamics. I do not see evidence that the stated convergence assumptions hold for this particular implementation; the positive multiplier lower bound could also exclude an optimal multiplier equal to zero.

We agree that there is a gap between the algorithm analyzed and the neural PPO implementation used in our experiments, but we want to emphasize this is inevitable as the real algorithms with PPO and LLM is hard to get the exact guarantees. Instead we focused on the analysis of a simplier setting. Our theory establishes convergence in a finite constrained RLHF setting with specified KL mirror updates. Within this setting, Theorem 3.4 proves geometric convergence of the last policy to the constrained optimum and of the multipliers to their optimal set, without policy averaging or an added dual penalty. Remark 3.5 gives an equivalent tabular NPG implementation under the stated conditions. Corollary 3.8 allows approximate policy updates under Assumptions 3.6 and 3.7, with exact dual updates.

To validate the theory in the setting it covers, we implemented the exact KL-mirror update with Euclidean multiplier projection (zero multipliers allowed) and its full-Fisher tabular NPG counterpart. Over 60 steps the two agree to within 2.01×10−132.01\times10^{-13} in policy probabilities and multipliers. On five synthetic instances (8 contexts, 16 actions, KL weight 0.050.05, three constraints, known optimum), with 10,000 policy-oracle calls per method and target error 10−610^{-6}:

Effective step size0.030.050.10.20.30.6
PD0/55/55/50/50/50/5
OPD0/55/55/55/55/55/5

PD converges at small step sizes, while OPD converges over a wider range. Details are in our response to Reviewer s5xE (Weakness 1).

The empirical evidence for a general practical advantage is still limited. The main LLM evaluation uses one 1.1B model and one preference dataset, and safety is assessed primarily through learned constrained-reward scores. I would like to see a more direct measure of safety constraint satisfaction or harmful behavior.

We trained PD and OPD on Qwen2.5-3B-Instruct with frozen reward/cost models of matching size. Each pair uses the same initial weights, PKU-SafeRLHF training split, seed 101, rank-16 LoRA adapters and 400 batches of 32 responses. Policy/critic learning rates are 10−510^{−5}/10−410^{−4}, dual step size is 0.005 and KL weight is 0.05. Evaluation uses 798 PKU test prompts, 240 HarmBench prompts with its independent Llama-2-13B classifier, and 200 benign requests checked for refusal phrases.

MethodReward ↑Constrained Reward ↑Harmful (%) ↓Benign refusal strings (%)
PD0.6891.65318.3316.5
OPD1.2461.30312.9215.5

In this setting, OPD achieves a higher reward (1.246 vs. 0.689) and a lower harmful response rate (12.92% vs. 18.33%) than PD.

There is also a meaningful computational cost: the reported step times are 991 ms for OPD versus 714 ms for PD, with higher peak memory. Comparing the methods only by training step makes it hard to judge the benefit at matched wall-clock time or compute, especially without a more extensively tuned PD baseline.

Both PD and OPD use TinyLlama-1.1B, frozen 1.1B scoring models, PKU-SafeRLHF training prompts, batch size 32, policy/critic learning rates 10−510^{−5}/2×10−62\times 10^{−6}, dual step size 0.05 and KL weight 0.05.Evaluation uses 798 PKU test prompts. The 60-minute budget includes loading, training, and saving, and we evaluate the latest fully saved policy within that limit. Values are averaged across paired training seeds.

BudgetPaired seedsPD reward ↑OPD reward ↑PD constrained reward ↑OPD constrained reward ↑
60 minutes312.08414.3537.71610.142
400 rollout batches213.61314.75010.08812.147

OPD achieves higher mean reward and higher mean constrained-reward under both budgets, with reward gains of 2.269 at 60 minutes and 1.138 at 400 rollout batches. These results support a better reward–cost trade-off for OPD at matched time and rollout budgets.

Section 2.1 states that the reward models are trained using the Bradley–Terry pairwise preference objective, while Section 4 enforces an expected constrained reward of at least zero. As I understand it, the Bradley–Terry loss only depends on relative reward differences. Adding the same offset to the reward scores of two responses for a given prompt does not change their predicted preference probability, but it does change the absolute expected reward. How is the zero threshold determined? What does an expected constrained reward of zero represent in terms of actual safety or constraint satisfaction?

The reviewer is correct that a pure Bradley--Terry loss identifies scores only up to a per-prompt offset, so a threshold on its expectation would have no absolute meaning. The description in Section~2.1 was incomplete. The helpfulness reward model is trained with the Bradley-Terry loss alone. For constrained-reward model, we follow approach in Safe RLHF [1]. In addition to the pairwise loss, each response has a binary safety label s∈{+1,−1}s\in\lbrace +1,-1\rbrace (+1+1 harmless) and the loss includes the classification term −log⁡σ(sR1(x,y))-\log\sigma(sR_1(x,y)). This anchors the scale: R1(x,y)>0R_1(x,y)>0 corresponds to a response the model classifies as harmless and R1(x,y)<0R_1(x,y)<0 to one it classifies as harmful. We have added the safety labels, the additional classification term, and this interpretation to Section 2.1.

[1] Josef Dai et al. Safe RLHF: Safe Reinforcement Learning from Human Feedback. ICLR, 2024.

Could the authors show λt\lambda_t for OPD and PD, and report an explicit constraint-violation metric? This would help assess whether the reduced training oscillation is accompanied by more stable multiplier dynamics and better constraint satisfaction.

We include this figure showing λt\lambda_t and validation constraint violation for PD and OPD on TinyLlama-1.1B, seeds 201-203 (left to right). Both methods use policy/critic learning rates 2×10−5/2×10−62\times10^{-5}/2\times10^{-6}, dual step size 0.050.05, KL weight 0.050.05, cost-smoothing decay 0.90.9, and multiplier range [0.5,10][0.5,10], and each run consumes 400 batches of 32 responses. Violation is vt=max⁡(−R1+3.5,0)v_t=\max(-R_1+3.5,0), where R1R_1 is the mean constrained reward on 500 fixed validation prompts at each checkpoint, evaluated every 50 batches.

After an initial rise, OPD's multiplier returns to the lower bound of 0.50.5 by batch 64 and remains there. PD's multiplier shows later rebounds, the largest in seed 203. OPD has zero violation at every check from batch 50 onward. PD violates again at batch 100 in seed 201 and first reaches zero violation at batch 150 in seed 203. It shows that OPD reaches and holds feasibility earlier and without the rebounds seen in PD.

Reviewer_JvzXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes an algorithm basing on optimistic predictory-corrector method, and theoretically prove later-iterate convergence in both distributional and parameterized policy spaces. in the experiment section, the proposed method obtains more stable training dynamics and a better helpfulness/safety trade-off than its baseline.

Critical strengths

1,This work theoretically and empirically proves that the proposed method could improve the training stability of its iterative counterparts.

2, The problem is well-described and motivated.

Critical weaknesses

1, While the proposed method outperforms its PD and PPO counterparts, those baselines are weak. Furthermore, there are one-shot methods, which usually have better stability during training. To my knowledge, none of those methods is included in the baselines. This makes it hard to evaluate how much the proposed algorithm improves upon current SOTA methods in this field.

2, While I do appreciate the theoretical analysis, the mathematical part is bit hard to follow. For example, in line 220, the notation BλB_{\lambda} is introduced without any further explanation in the main script; same to the L0L_0 notation in line 230. While the theoretical parts is developed with AI tools, I would appreciate if the manuscript could be phrased so that broader audiences could interpret.

Questions for authors

Can you explain why the first claimed contribution, aka a unified primal-dual framework, is significant? From my understanding, it is just a typical high-level primal-dual update pseudo-code.

1 comment

While the proposed method outperforms its PD and PPO counterparts, those baselines are weak. Furthermore, there are one-shot methods, which usually have better stability during training.

We added a one-shot baseline, adapting MoCAN [1] to the same TinyLlama-1.1B starting policy and frozen 1.1B reward and constrained-reward models used by PD and OPD. Both PD and OPD were tuned over the same four (policy learning rate, critic learning rate, dual step size) configurations: (10−6,5×10−7,0.001)(10^{-6},5\times10^{-7},0.001) and (5×10−6,2×10−6,d)(5\times10^{-6},2\times10^{-6},d) for d∈{0.001,0.005,0.05}d\in\lbrace 0.001,0.005,0.05\rbrace. All reported runs use policy learning rate 5×10−65\times10^{-6}. PD and OPD each consume 400 batches of 32 responses, with KL weight 0.05 and critic learning rate 2×10−62\times10^{-6}. The dual step sizes 0.050.05 and 0.0010.001 are selected for PD and OPD separately. MoCAN solves an offline dual using the same scores, forms synthetic preferences from reward plus λ\lambda times constrained reward, and trains the policy with DPO.

We evaluate greedy responses on all 798 distinct PKU-SafeRLHF-30K test prompts.

MethodReward ↑Constrained reward ↑Training responsesTime (h)
PD10.9202.95612,8001.291
OPD10.4354.53612,8001.351
MoCAN6.9324.17012,8000.681
MoCAN7.8437.442204,8009.353

At 12,800 responses, both OPD and bounded-dual MoCAN meet the mean constraint, but OPD has higher reward and constrained reward. The expanded MoCAN run achieves higher constrained reward using 16 times as many responses.

[1] Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, and Dongsheng Ding. One-Shot Safety Alignment for Large Language Models via Optimal Dualization. Advances in Neural Information Processing Systems, 37:84350–84383, 2024.

The mathematical part is bit hard to follow. For example, in line 220, the notation BλB_\lambda is introduced without any further explanation in the main script; same to the L0L_0 notation in line 230.

We revised Section 3 so that every symbol in the theorem statement is explained where it first appears, and we added proof outlines.

  1. BλB_\lambda bounds the ℓ1\ell_1 norm of every optimal multiplier. It is the gap between the unconstrained optimal objective value and the value of a strictly feasible policy, divided by that policy's constraint margin. Projecting onto {λ≥0:∥λ∥1≤Bλ}\left\lbrace \lambda \ge 0 : \lVert \lambda \rVert_1 \le B_\lambda \right\rbrace keeps the dual iterates bounded without excluding any optimal multiplier.

  2. L0=exp⁡(2(1+Bλ)Rmax⁡/β)L_0=\exp(2(1+B_\lambda)R_{\max}/\beta) bounds every exact iterate relative to the reference policy: L0−1≤π/πref≤L0L_0^{-1}\le\pi/\pi_{\rm ref}\le L_0. This follows from bounded rewards and initialization at πref\pi_{\rm ref}. We note plainly that L0L_0 is exponential in 1/β1/\beta, so the certified contraction factor ρ\rho can be very close to 11 when β\beta is small.

Theorem 3.4 (exact OPD, Appendix C). We define a potential Φt\Phi_t as a weighted sum of (i) the KL divergence from the optimal policy to the current policy, (ii) the squared distance from the current multiplier to the optimal multiplier set, and (iii) the gaps between the previous predictors and the current corrected iterates. We show Φt+1≤ρΦt\Phi_{t+1}\le\rho\Phi_t with 0<ρ<10<\rho<1. The argument does not require a unique optimal multiplier, policy averaging, or an added dual regularizer.

Corollary 3.8 (approximate policy updates, Appendix E). The dual updates remain exact and the policy updates may be approximate. Under the two approximation bounds of Assumption 3.6 and the relative-density lower bound of Assumption 3.7, the same argument yields the contraction with an additive error at each step. Errors from earlier steps are multiplied by subsequent contraction factors, so the bound is a geometrically decaying initial term plus a geometrically weighted sum of the per-step errors. If both approximation errors tend to zero, the policy error and the distance to the optimal multiplier set.

Can you explain why the first claimed contribution, aka a unified primal-dual framework, is significant? From my understanding, it is just a typical high-level primal-dual update pseudo-code.

We agree. Algorithm 1 is a summary of how existing Lagrangian methods for constrained RLHF coordinate policy and multiplier updates, and it serves to place OPD among them. It is better presented as background than as a separate contribution. We removed it from the contribution list and from the conclusion, and kept it in Section 2.2.

The contribution statement now focuses on the specified optimistic KL updates, their convergence guarantees, and the empirical evaluation. With the stated assumptions and step sizes, our main result establishes geometric convergence of the last-iterate policy to the constrained optimum and of the multipliers to their optimal set. We also evaluates how approximate policy updates affect the optimality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.