acceptodds
Under review as a conference paper at ICLR 2027

ZAPD: Zero-Shot Adaptation via Primal-Dual Updates for Constrained Reinforcement Learning

Abstract

In constrained reinforcement learning (CRL), agents maximize expected reward while ensuring the expected cost does not exceed a given threshold. Most existing CRL methods, including the widely used primal-dual approaches, assume this threshold is fixed, but in real-world applications it can shift abruptly. Standard CRL methods then need substantial re-learning, and existing threshold-adaptive methods require access to a range of thresholds or a library of pre-trained policies. To address this problem, we propose Zero-shot Adaptation via Primal-Dual updates (ZAPD), which adapts the primal-dual solution to a new threshold without any additional environment interaction. ZAPD treats the shift as a perturbation of the converged solution and employs a first-order sensitivity analysis to derive closed-form primal and dual updates. Both updates are computed from data the underlying method already collects, so ZAPD plugs into a broad range of primal-dual CRL algorithms. Experiments show that ZAPD achieves the lowest immediate approximation error on all four Bullet Safety Gym tasks in the single-threshold setting, and that this improved initialization translates into faster post-shift convergence than threshold-adaptive baselines.

Primary area:
reinforcement learning
Keywords:
Constrained Reinforcement Learning, Primal-Dual

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HdvAOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies abrupt changes in constraint thresholds in constrained reinforcement learning (CRL). The authors propose ZAPD, a zero-shot adaptation method that treats a threshold change as a perturbation of a previously converged constrained optimization problem. Using first-order analysis, ZAPD constructs closed-form corrections for the policy parameters and Lagrange multipliers using only pre-shift information, without additional environment interaction.

The practical algorithm estimates the constraint Jacobian from policy-gradient information and approximates the curvature needed for the dual correction using off-policy AlgaeDICE gradients and a diagonal secant approximation. Empirically, ZAPD is evaluated on Bullet Safety Gym, CostLQR, and Safety-Gymnasium Swimmer, across multiple base CRL algorithms and both linear and neural policies. The reported results show improved immediate post-shift approximation and generally faster reconvergence than CCPO and CAPS.

Critical strengths
  1. The paper studies an important and underexplored adaptation setting with a simple and useful perspective. The setting in which a CRL agent has converged under one constraint threshold and must immediately adapt to an unseen threshold without further environment interaction is practically meaningful and differs from methods that require training across a range of thresholds. Treating the threshold change as a perturbation of an already solved optimization problem is a clean idea and gives the method a clear motivation.
  2. The empirical evaluation is relatively broad and shows a consistent positive signal. The paper evaluates ZAPD across multiple environments, base CRL algorithms, and both linear and neural policy parameterizations, with 20 independent runs in the main experiments. The results generally show lower immediate approximation error and faster post-shift convergence, suggesting that the proposed correction can provide a useful initialization in practice.
Critical weaknesses
  1. The theoretical interpretation of the primal update as a first-order sensitivity of the optimal policy is not sufficiently justified. Theorem 1 derives the primal correction by solving the minimum-Euclidean-norm step satisfying the linearized constraint, Gdθ=dbG d\theta = db, which gives dθ=G⊤(GG⊤)−1dbd\theta = G^\top(GG^\top)^{-1}db. However, differentiating the full KKT conditions for the constrained optimum would generally give Hdθ−G⊤dλ=0H d\theta - G^\top d\lambda = 0 and Gdθ=dbG d\theta = db, and therefore dθ=H−1G⊤(GH−1G⊤)−1dbd\theta = H^{-1}G^\top(GH^{-1}G^\top)^{-1}db. Thus, the proposed primal correction appears to be a first-order feasibility/minimum-norm correction rather than the general sensitivity of the optimal primal solution, unless additional assumptions are imposed. This matters because the paper presents the primal and dual updates as arising from a unified sensitivity analysis, while the dual update explicitly uses the curvature information that is absent from the primal update. Relatedly, the proof of the dual result uses negative definiteness of the Hessian based on the solution being a maximizer, whereas the stated assumptions mainly impose stationarity and invertibility. I believe the theoretical claims and assumptions should be tightened substantially.

  2. The main experiments lack the most direct baseline for evaluating the claimed benefit of ZAPD. For a method whose main purpose is to improve initialization after a threshold shift, the key comparison should be against the exact same underlying CRL method starting from the same pre-shift solution but performing no ZAPD correction and simply continuing optimization under the new threshold. The main experiments instead focus on CCPO and CAPS, which operate under different adaptation mechanisms and information assumptions. A no-ZAPD continuation baseline is therefore necessary to isolate how much of the faster reconvergence is actually caused by the proposed correction.

Questions for authors
  1. Can the authors justify why the update in Theorem 1 should approximate the sensitivity of the optimal policy rather than only the minimum-norm solution of the linearized feasibility constraint?
  2. How does ZAPD handle active-set changes? A constraint that is slack before the shift may become active after tightening, while an active constraint may become slack after relaxation. Since the post-shift activity assumption cannot be verified from pre-shift data, how frequently does this occur in the experiments, and what happens to ZAPD in such cases?
  3. It would be important to include, for all main experiments, a baseline using the same underlying C-PG/PPO-L/CVPO policy and the same pre-shift solution but with no ZAPD correction, followed by standard optimization under the new threshold. This seems necessary to isolate the actual contribution of the proposed update. Could the authors provide this comparison?
  4. The evaluation would be more convincing with direct safety-oriented post-shift metrics, especially immediate and cumulative constraint violation after tightening, rather than only error relative to the reference solution. The current approximation-error metric penalizes both costs above and below the reference value and therefore does not directly establish that ZAPD reduces the period or magnitude of constraint violation. Can the authors provide such results?
4 comments

We thank the reviewer for recognizing the importance of our work and the empirical results. We appreciate your comprehensive review and would like to address each of your weaknesses and questions.

W1 (primal update as a first-order sensitivity of the optimal policy): Reviewer is right to point out our Theorem 1 is different from standard sensitivity analysis. This is a feature, not a bug. Standard analysis involves a Hessian term, which is hard to estimate precisely during runtime. Our Theorem 1 avoids this term while preserving near-feasibility. In particular, combining Theorem 1 with Eqs. (5)-(6) leads to the following corollary, which we will include in the update paper:

J(θ~)=J(θ∗)+∇θJ(θ∗)⊤(θ~−θ∗)+O(∥θ~−θ∗∥2)=b^+O(∥θ~−θ∗∥2).J(\tilde{\theta}) = J(\theta^{\ast}) + \nabla_{\theta} J(\theta^{\ast})^\top (\tilde{\theta}-\theta^{\ast}) + O(\lVert\tilde{\theta}-\theta^{\ast}\rVert^2) = \hat{b} + O(\lVert\tilde{\theta}-\theta^{\ast}\rVert^2).

Regarding the dual-update proof, [Paternain et al., Constrained Reinforcement Learning Has Zero Duality Gap, NeurIPS, 2019] establish a bound on the duality gap for parameterized CRL policies. So, θ∗\theta^{\ast} is a maximizer.

W2 (show no-ZAPD): The no-ZAPD policy is equivalent to CAPS in the single-threshold case, which is considerably worse than ZAPD in all cases. (P.S. CAPS trains one policy for each pre-shift threshold and then chooses the appropriate policy based on the actual threshold.)

Q1 (Thm1 ≈\approx optimal policy?): Please see the response to W1.

Q2 (active-set): We search our data of Bullet Safety Gym tasks to find all instances of the situations: a constraint goes from active to inactive and vice versa. Below are the immediate approximation average error of both cases. Active-set changes occur in 36.336.3\\% of cases (27.427.4\\% active-to-inactive, 8.88.8\\% inactive-to-active).

MethodActive → InactiveInactive → Active
ZAPD44.0740.15
CCPO64.6579.63
CAPS47.5949.01

In both of them, ZAPD remains the best.

Q3 (Show No-ZAPD): Please see the response to W2.

Q4 (Immediate and Cumulative constraint violation after tightening): We thank the reviewer for highlighting the gap between approximation error and safety violations. We report immediate and cumulative constraint violation after tightening (single-threshold). Immediate violation averages max⁡(0,episode cost−b^)\max(0, \text{episode cost} - \hat{b}) over 50 frozen-policy episodes. Cumulative violation sums max⁡(0,iteration cost−b^)\max(0, \text{iteration cost} - \hat{b}) until the first feasible iteration, over which cumulative reward error is averaged. The results show that ZAPD has a much smaller violation than CAPS. The violation of ZAPD is slightly larger than CCPO in some cases but that is because CCPO has a much worse reward performance. In particular, ZAPD achieves the lowest cumulative violation in most cases.

Table 1: Immediate constraint violation and reward error for the Bullet Safety Gym tasks.

EnvironmentZAPD ViolationZAPD Reward error (%)CCPO ViolationCCPO Reward error (%)CAPS ViolationCAPS Reward error (%)
Ball-Circle2.76 ± 2.4916.60 ± 2.770.01 ± 0.0385.03 ± 6.4318.95 ± 1.657.33 ± 0.87
Car-Circle0.88 ± 0.4637.05 ± 5.464.08 ± 7.3563.36 ± 8.1223.24 ± 3.953.52 ± 1.04
Drone-Circle3.46 ± 1.5534.58 ± 7.750.12 ± 0.1275.40 ± 6.4718.90 ± 1.898.18 ± 2.75
Drone-Run11.19 ± 15.1818.55 ± 15.635.03 ± 12.1032.48 ± 13.8118.91 ± 20.598.87 ± 11.52

Table 2: Cumulative constraint violation and reward error for the Bullet Safety Gym tasks.

EnvironmentZAPD ViolationZAPD Reward error (%)CCPO ViolationCCPO Reward error (%)CAPS ViolationCAPS Reward error (%)
Ball-Circle7.4 ± 4.812.27 ± 3.9156.9 ± 48.795.80 ± 3.8615.7 ± 8.761.77 ± 4.28
Car-Circle5.1 ± 2.722.49 ± 3.763.5 ± 7.298.70 ± 0.83117.0 ± 76.450.65 ± 11.25
Drone-Circle5.8 ± 4.926.75 ± 8.2712.9 ± 3.699.67 ± 0.345.2 ± 2.797.09 ± 2.96
Drone-Run17.4 ± 62.116.38 ± 15.0125.3 ± 30.458.25 ± 6.9620.6 ± 21.235.81 ± 4.07

Table 3: Immediate constraint violation and reward error for CostLQR and Swimmer.

SettingZAPD ViolationZAPD Reward error (%)CAPS ViolationCAPS Reward error (%)
CostLQR, Linear C-PG0.43 ± 0.204.8 ± 4.31.09 ± 0.375.6 ± 2.6
CostLQR, Linear PPO-L0.62 ± 0.286.1 ± 4.11.40 ± 0.716.2 ± 4.0
CostLQR, NN C-PG0.55 ± 0.4211.7 ± 13.51.13 ± 0.687.0 ± 3.5
CostLQR, NN PPO-L0.73 ± 0.5011.4 ± 9.01.37 ± 0.835.4 ± 3.2
Swimmer, Linear C-PG0.15 ± 0.2416.7 ± 13.91.00 ± 0.638.8 ± 4.9
Swimmer, Linear PPO-L0.12 ± 0.094.9 ± 2.30.99 ± 0.1910.5 ± 2.3
Swimmer, NN C-PG0.63 ± 0.9519.0 ± 21.81.45 ± 1.349.5 ± 5.2
Swimmer, NN PPO-L0.12 ± 0.118.0 ± 4.91.01 ± 0.209.5 ± 2.0

Table 4: Cumulative constraint violation and reward error for CostLQR and Swimmer.

SettingZAPD ViolationZAPD Reward error (%)CAPS ViolationCAPS Reward error (%)
CostLQR, Linear C-PG6.7 ± 15.14.6 ± 1.595.6 ± 53.14.4 ± 0.6
CostLQR, Linear PPO-L5.7 ± 7.14.1 ± 2.035.1 ± 15.34.5 ± 0.8
CostLQR, NN C-PG16.9 ± 24.37.3 ± 4.771.6 ± 41.35.7 ± 1.3
CostLQR, NN PPO-L9.2 ± 10.46.1 ± 2.826.7 ± 11.14.7 ± 0.8
Swimmer, Linear C-PG41.0 ± 104.89.3 ± 5.8334.6 ± 166.96.5 ± 2.4
Swimmer, Linear PPO-L21.6 ± 18.34.2 ± 1.178.4 ± 11.45.2 ± 0.4
Swimmer, NN C-PG49.2 ± 82.58.6 ± 3.3298.7 ± 271.97.7 ± 3.8
Swimmer, NN PPO-L11.2 ± 13.34.2 ± 2.777.1 ± 13.64.7 ± 0.5

Thank you for the clarifications and additional experiments. My concerns have been largely addressed, so I am raising my rating to 3.

Thank you very much for raising your score. We greatly appreciate it.

The Authors

Reviewer_eGS1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ZAPD, a method that adapts the primal-dual solution to a new constraint threshold without any additional environment interaction. ZAPD is based on closed-form primal and dual updates derived from a first-order sensitivity analysis. It is validated by numerical experiments on benchmarks such as Bullet Safety Gym and Safety-Gymnasium.

Critical strengths
  1. ZAPD can adapt zero-shot to a new constraint threshold using only the data the agent has already collected before the shift. As a result, it converges faster and does not need to be re-trained from scratch.

  2. ZAPD is a plug-in that works across different base methods (CVPO, C-PG, PPO-L) and policy parameterizations.

Critical weaknesses
  1. ZAPD relies heavily on a first-order Taylor expansion, which is accurate only when Δb\Delta b is very small. Thus, ZAPD is naturally hard to apply to tasks, where the threshold shift Δb\Delta b is not necessarily small.

  2. The approximations in the practical dual update is not justified, so it is unclear how close the practical algorithm is to Theorem 2.

Questions for authors
  1. Is the right-hand equation in Eq. (3) incorrect? I think λk\lambda_k should take θk\theta_k as input, not θk−1\theta_{k−1}.

  2. If (θ⋆,λ⋆)(\theta^\star, \lambda^\star) has not been reached at the iteration kck_c where the shift occurs, what should one do?

  3. Per Lines 203–204, did you try applying ZAPD to non-Lagrangian methods, such as CPO and IPO? Does it work?

2 comments

We thank the reviewer for recognizing the data efficiency and compatibility of our work. We appreciate your constructive review and would like to address each of your weaknesses and questions.

W1 (large ∣Δb∣|\Delta b|): We acknowledge that first-order approximations may become less accurate for large shifts (line 485). However, we evaluate ZAPD on the same tasks as CCPO, with a large threshold range (55 to 7575, a 1515\\% difference). The results show that ZAPD is superior than others.

W2 (λ\lambda gap between theorem and practice): We agree that we do not provide a theoretical error bound between Theorem 2 and the practical dual update. Nevertheless, our motivation is to accelerate reconvergence after a threshold shift, and Table 4 demonstrates that the practical dual correction improves reconvergence speed over the primal-only update.

Q1 (Eq.(3) incorrect?): We introduce how three works (line 117) conduct update:

  • Ding et al., 2020:

    In Eq. (7),

    λ(t+1)=PΛ(λ(t)−η2(Vgθ(t)(ρ)−b))\lambda^{(t+1)} = \mathcal{P}_{\Lambda}\left(\lambda^{(t)} - \eta_2\left(V_g^{\theta^{(t)}}(\rho) - b\right)\right)

    where θ,λ\theta,\lambda are primal and dual variable, tt is iteration. Thus, in our notation, λk\lambda_k takes θk−1\theta_{k-1}.

  • Paternain et al., 2019:

    In Eq. (20),

    λk+1=[λk−η∂^dk]+=[λk−η(V(θk+1)−s)]+\lambda_{k+1} = \left[\lambda_k - \eta \hat{\partial} d_k\right]_+ = \left[\lambda_k - \eta\left(V(\theta_{k+1}) - s\right)\right]_+

    where θ,λ\theta,\lambda are primal and dual variable, kk is iteration. Thus, in our notation, λk\lambda_k takes θk\theta_k.

  • Montenegro et al., 2024:

    In Algorithm 1,

    λk←λk−1+ζλ,k−1∇^λL~A,ω(θk−1,λk−1,ηk−1)\lambda_k \leftarrow \lambda_{k-1} + \zeta_{\lambda,k-1} \widehat{\nabla}_{\lambda}\widetilde{\mathcal{L}}_{A,\omega}(\theta_{k-1},\lambda_{k-1},\eta_{k-1})

    where θ,λ\theta,\lambda are primal and dual variable, kk is iteration. Thus, in our notation, λk\lambda_k takes θk−1\theta_{k-1}.

Thus, Eq. (3) follows a standard update convention.

Q2 (What if not converged?): ZAPD assumes a converged pre-shift solution and is therefore naturally suited to algorithms with last-iterate convergence guarantees, such as C-PG [Montenegro et al., Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning, NeurIPS, 2024]. If convergence has not been reached, our theoretical guarantees do not apply. In practice, Algorithm 1 (line 10) checks Assumption 4 and 5 before applying ZAPD.

Q3 (applying ZAPD to non-Lagrangian): Yes we did evaluate non-Lagrangian method. CVPO in Section 5.2 is a non-Lagrangian method. ZAPD performs very well with CVPO.

Thank you. However, I don't think my W2 has been adequately addressed. Also, ZAPD is limited to small shifts and applies only after convergence. I will keep my score.

Reviewer_H6rZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free method to adapt to perturbations in the constraints of a CMDP. The idea is to provide a zero-shot adaptation of a pre-trained CRL policy to a new environment in which only the constraint specification is changed. The approach is based on a first-order Taylor expansion of the Lagrangian around the pre-trained saddle-point. Several approximations are proposed to estimate the corrections and render them tractable (e.g., stochastic approximation of expectations or diagonal approximation of Hessians). The final approach is tested on typical RL benchmarks.

Critical strengths

The problem tackled is practically relevant. In particular, the fact that no information on the threshold is needed at training time is a key strength of the formulation. Other threshold-adaptive baselines must be trained on specific thresholds that will be potentially faced during deployment. The problem setting of ZAPD only needs one converged solution and the new threshold once it is observed. Finding a closed-form correction (even if parts of it must be estimated) is naturally attractive. This is especially the case in RL, where environment interactions are costly. The possibility of reusing samples is therefore always welcome. Despite the weaknesses discussed next, which in my opinion are critical and render the proposed solution ineffective, I therefore strongly encourage the authors to continue pursuing this problem.

Critical weaknesses

The paper scientific contributions appear to be minor. Using Taylor expansions to produce local policy improvements is typical in optimization (e.g., gradient descent) and RL (e.g., natural policy gradient). In fact, (6) is reminescent of linear minimization oracles (LMOs) and conditional gradient methods, such as Frank-Wolfe. What is more, there is a substantial literature perturbation of optimization problems that already provide results such Theorems 1 and 2 under considerably more general settings. See, e.g., [Bonnans and Shapiro, Perturbation Analysis of Optimization Problems, 2000] and [Fiacco, 1976] (cited, but not discussed) or the dual function curvature and perturbation results from [Guigues, Inexact stochastic mirror descent for two-stage nonlinear stochastic programs, Mathematical Programming, 2020; Elenter et al., Near-optimal solutions of constrained learning problems, ICLR, 2024].

That is not to say that their application is not valid, but they are certainly not novel. As such, the manuscript should point out and discuss their relationship (similarity and differences) to these well-established optimization and RL methods.

Many of these references provide results under less stringent settings than those covered by Assumptions 1--5. In particular, Assumption 1 can be replaced by milder assumptions from [Paternain, 2019] that ensure strong duality. THis is especially the case in view of Assumptions 5 (essentially LICQ), which in turn could be relaxed to consider only the active constraints (as noted around line 170). Assumption 3 is claimed to be standard in line 165. I don't believe any of the policy gradient analyses cited in the paper require such a regularity of optimal solutions (including [Ding et al., 2020; Paternain et al., 2019; Montenegro et al., 2024]). What is more, it is not immediate that the Lagrangian minimizers θ(λ)\theta(\lambda) are even a continuous function of λ\lambda around λ⋆\lambda^\star if, for instance, λi⋆=0\lambda^\star_i = 0. This assumption requires more justifications.

Assumption 4 is particularly critical, especially since (i) it cannot be checked J(θ^⋆)=b^J(\hat{\theta}^\star) = \hat{b} without solving the problem. The manuscript acknowledges this point in line 175, but does not discuss its consequence. Particularly critical to their proof and results appears to be the case where an inactive constraint J(θ⋆)<bJ(\theta^\star) < b becomes active J(θ^⋆)=b^J(\hat{\theta}^\star) = \hat{b}. The correction in Theorem 1, for instance, will ignore this constraint and likely lead to an infeasible or at the very least a deterioration of its performance (cf. results on Car-Circle in the single-threshold setting). T

Without these assumptions, the corrections in Theorems 1 and 2 are inappropriate to the problem posed in the manuscript. What is more, even if we granted these assumptions, no guarantees are provided that the corrections will keep the solution (near-)feasible and/or (near-)optimal. In fact, Assumption 1 is not sufficient to ensure the feasibility of the policy θ⋆\theta^\star [Calvo-Fullana et al., State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards, IEEE TAC, 2024]. The training procedure in this work does ensure feasibility and can accomodate threshold adaptations.

The submission also provides an anonymized code package, but no main script to reproduce the results. In particular, the reviewer could not run the CostLQR/Swimmer part on a standard OmniSafe install.

Questions for authors
  • Are there specific works that would justify the fact that Assumption 3 is standard?
  • How critical is Assumption 4? What are the exact implications of J(θ^⋆)<b^J(\hat{\theta}^\star) < \hat{b}?
  • CCPO is trained with thresholds from {40}, {30, 50} or {20, 40, 60}, and is evaluated for b^∈5,…,75\hat{b} \in {5, \dots, 75}. Hence, the evaluation is mostly extrapolation (in the single and double threshold settings). What happens when the deployment thresholds are within the training threshold?
  • In general, we do not have access to a θ⋆\theta^\star that satisfies Assumption 1. What happens if it is satisfied approximately? How is the pre-shift policy θkc\theta_{k_c} chosen? I understand that the code suggest to take the last iterate. How can you verify that it (approximately) satisfies Assumption 1? What about problems in which primal-dual iterated do not converge (see, e.g., [Calvo-Fullana et al., State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards, IEEE TAC, 2024])?
  • The use of EMA smoothing will add latency to the policy tracking, even though the goal is to react as quickly as possible. Is this hyperparameter hard to tune? What if the constraint changes continuously?
2 comments

We thank the reviewer for recognizing the importance of our study and encouraging further research. We appreciate your comprehensive review and would like to address each of your weaknesses and questions.

W1 (P1&2, minor contributions): Our contribution is to build on established sensitivity analysis for fast constraint-threshold adaptation in widely used primal-dual CRL. While the four references cited by the reviewer provide related results, none explicitly derives the closed-form for Theorems 1 and 2. In particular, we use a different approach to obtain Theorem 1. Existing sensitivity analysis will require a Hessian term in the derivation, which is difficult to estimate in runtime. Our Theorem 1 has a different derivation that avoids using the Hessian term while preserving near-feasibility. (See the response to W4). We will clarify the connections to existing literature.

W2 (P3, strong assumptions): We agree some assumptions can be relaxed. We state them this way to make them easier to relate to in the context of CMDP and CRL. For Assumption 3, [Bonnans et al., 2000] and [Fiacco, 1976] assume that θ(b)\theta(b) and λ(b)\lambda(b) are continuously differentiable with bb, which implies our Assumption 3 since λ(b)\lambda(b) is monotone.

W3 (P4, Assumption 4): We report weighted averages of constraint violation, reward and immediate approximation average error across four Bullet Safety Gym tasks for inactive-to-active shifts in the table below. It can be shown that our ZAPD is much better than CCPO, and is slightly better than CAPS. For Car-Circle, J(θ~)≈0J(\tilde{\theta})\approx 0 at b^∈{5,10,15}\hat{b} \in \lbrace 5,10,15 \rbrace (Table 8), indicating feasibility.

MethodViolationRewardError (%)
ZAPD13.67563.8744.17
CCPO4.25270.2774.08
CAPS14.80568.1550.40

W4 (P5, feasibility): We would like to point out that Theorem 1 guarantees near-feasibility. Under Assumption 4, Theorem 1 and Eqs. (5)-(6) imply:

J(θ~)=J(θ∗)+∇θJ(θ∗)⊤(θ~−θ∗)+O(∣θ~−θ∗∣2)=b^+O(∣θ~−θ∗∣2).J(\tilde{\theta}) = J(\theta^{\ast}) + \nabla_{\theta} J(\theta^{\ast})^\top (\tilde{\theta}-\theta^{\ast}) + O(\\|\tilde{\theta}-\theta^{\ast}\\|^2) = \hat{b} + O(\\|\tilde{\theta}-\theta^{\ast}\\|^2).

We will state this as a corollary.

W5 (P6, code): We updated and provide a script. To run the CostLQR/Swimmer, please see costlqr_swimmer folder.

Q1 (Is Assumption 3 standard?): Please see the response to W2.

Q2 (How critical is Assumption 4?): Assumption 4 is critical to both Theorems. However, we may encounter the case when this does not hold in practice. We report immediate approximation average error across four Bullet Safety Gym tasks of single-threshold setting when J(θ^∗)<b^J(\hat{\theta}^{\ast}) < \hat{b} in the table below. When J(θ^∗)<b^J(\hat{\theta}^{\ast}) < \hat{b}, ZAPD has the smallest error.

MethodError (%)
ZAPD44.17
CCPO74.08
CAPS50.40

Q3 (CCPO with interpolation): We report immediate approximation average error across four Bullet Safety Gym tasks of double, triple threshold setting where the deployment thresholds are within the training threshold range. ZAPD remains better than CCPO.

SettingZAPDCCPO
Double18.3319.68
Triple21.6425.03

Q4 (Assumption 1 and Convergence): Assumption 1 can be approximately assessed at the pre-shift using the EMA reward and cost gradients (Eq. (13) and line 307) with the current dual variables. Line 10 of Algorithm 1 checks the convergence condition, which is further explained in details in Appendix C.4. We use C-PG because it provides last-iterate convergence guarantees [Montenegro et al., Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning, NeurIPS, 2024]. We further test CVPO and PPO-L empirically, demonstrating its convergence although we do not have convergence guarantees for these methods.

Q5 (EMA): EMA only smooths pre-shift gradient estimates, so it does not delay adaptation to a threshold shift. Given an approximate number of pre-shift iterations, β\beta can be selected in advance. For reference, we set β=0.95\beta=0.95 for 8080–120120 pre-shift iterations and β=0.99\beta=0.99 for 2,000 pre-shift iterations. Continuously changing thresholds may violate our assumptions and are therefore beyond our scope.

Thank you for the responses. I address each of them below one-by-one. In view of these, I find that substantial theoretical and novelty issues remain with the current version of the manuscript and therefore cannot suggest its acceptance in its current form.

W1

The works I meantioned do not require second-order information to provide perturbation results. In particular, the Hölder stability results in Ch. 4 of [Bonnans and Shapiro, 2000] (e.g., Prop. 4.41) do not require it. Neither does the Lipschitz continuity guarantee in Prop. 4.51 (notice that the second-order guarantee can be replaced by the Magasarian-Fromovitz condition or LICQ). The proofs follow the same first-order Taylor expansions used in this work. In fact, though the results from [Guigues, 2020] rely on the existence of the Hessian, it is not necessary to obtain first-order perturbation results in their settings.

W2

Unfortunately, λ(b)\lambda(b) need not be monotone, especially for non-convex optimization problem such as those in RL. Even in convex problems, as soon as there are multiple constraints, this is no longer the case. For instance,

min⁡x,y x2−xy+y2−3x−3ysubject tox≤u, y≤v\min_{x,y}\ x^2 -xy + y^2 - 3x - 3y \quad \text{subject to} \quad x \leq u,\ y \leq v

For (u,v)=(0,0)(u,v) = (0,0), λ⋆=(3,3)\lambda^\star = (3,3). But for (u,v)=(1,0)(u,v) = (1,0), λ⋆=(1,4)\lambda^\star = (1,4). For non-convex problems, monotonicity need not hold even for single-constraint problems.

W3 / Q2

It is not clear to me how these experiments validate the results from Assumption 4, given that all solutions violate the constraints. In particular, the ZAPD and CAPS violate the constraints considerably more than CCPO, which explain why they achieve higher values of the reward. Without feasibility, improving the objective does not address problem (1).

W4

My remark went to the point that a stationary point need not be feasible (particularly in RL). Requires feasibility in the first place, which is not easy to achieve in CRL using dual methods (as per [Calvo-Fullanae et al., 2024]).

Q3

The authors' point is well-taken, but the distinction is less considerable. Also, it is fundamental to report the feasibility of the solutions, seen as it is easy to improve performance by violating the constraints.

Comments on the paper

1 comment

Dear reviewers,

The authors have now submitted their responses. Please review them and share your thoughts on whether your concerns have been addressed. I encourage you to engage with one another’s assessments, discuss any remaining disagreements, and update your reviews and scores as appropriate.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.