- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies abrupt changes in constraint thresholds in constrained reinforcement learning (CRL). The authors propose ZAPD, a zero-shot adaptation method that treats a threshold change as a perturbation of a previously converged constrained optimization problem. Using first-order analysis, ZAPD constructs closed-form corrections for the policy parameters and Lagrange multipliers using only pre-shift information, without additional environment interaction.
The practical algorithm estimates the constraint Jacobian from policy-gradient information and approximates the curvature needed for the dual correction using off-policy AlgaeDICE gradients and a diagonal secant approximation. Empirically, ZAPD is evaluated on Bullet Safety Gym, CostLQR, and Safety-Gymnasium Swimmer, across multiple base CRL algorithms and both linear and neural policies. The reported results show improved immediate post-shift approximation and generally faster reconvergence than CCPO and CAPS.
Critical strengths
- The paper studies an important and underexplored adaptation setting with a simple and useful perspective. The setting in which a CRL agent has converged under one constraint threshold and must immediately adapt to an unseen threshold without further environment interaction is practically meaningful and differs from methods that require training across a range of thresholds. Treating the threshold change as a perturbation of an already solved optimization problem is a clean idea and gives the method a clear motivation.
- The empirical evaluation is relatively broad and shows a consistent positive signal. The paper evaluates ZAPD across multiple environments, base CRL algorithms, and both linear and neural policy parameterizations, with 20 independent runs in the main experiments. The results generally show lower immediate approximation error and faster post-shift convergence, suggesting that the proposed correction can provide a useful initialization in practice.
Critical weaknesses
-
The theoretical interpretation of the primal update as a first-order sensitivity of the optimal policy is not sufficiently justified. Theorem 1 derives the primal correction by solving the minimum-Euclidean-norm step satisfying the linearized constraint, , which gives . However, differentiating the full KKT conditions for the constrained optimum would generally give and , and therefore . Thus, the proposed primal correction appears to be a first-order feasibility/minimum-norm correction rather than the general sensitivity of the optimal primal solution, unless additional assumptions are imposed. This matters because the paper presents the primal and dual updates as arising from a unified sensitivity analysis, while the dual update explicitly uses the curvature information that is absent from the primal update. Relatedly, the proof of the dual result uses negative definiteness of the Hessian based on the solution being a maximizer, whereas the stated assumptions mainly impose stationarity and invertibility. I believe the theoretical claims and assumptions should be tightened substantially.
-
The main experiments lack the most direct baseline for evaluating the claimed benefit of ZAPD. For a method whose main purpose is to improve initialization after a threshold shift, the key comparison should be against the exact same underlying CRL method starting from the same pre-shift solution but performing no ZAPD correction and simply continuing optimization under the new threshold. The main experiments instead focus on CCPO and CAPS, which operate under different adaptation mechanisms and information assumptions. A no-ZAPD continuation baseline is therefore necessary to isolate how much of the faster reconvergence is actually caused by the proposed correction.
Questions for authors
- Can the authors justify why the update in Theorem 1 should approximate the sensitivity of the optimal policy rather than only the minimum-norm solution of the linearized feasibility constraint?
- How does ZAPD handle active-set changes? A constraint that is slack before the shift may become active after tightening, while an active constraint may become slack after relaxation. Since the post-shift activity assumption cannot be verified from pre-shift data, how frequently does this occur in the experiments, and what happens to ZAPD in such cases?
- It would be important to include, for all main experiments, a baseline using the same underlying C-PG/PPO-L/CVPO policy and the same pre-shift solution but with no ZAPD correction, followed by standard optimization under the new threshold. This seems necessary to isolate the actual contribution of the proposed update. Could the authors provide this comparison?
- The evaluation would be more convincing with direct safety-oriented post-shift metrics, especially immediate and cumulative constraint violation after tightening, rather than only error relative to the reference solution. The current approximation-error metric penalizes both costs above and below the reference value and therefore does not directly establish that ZAPD reduces the period or magnitude of constraint violation. Can the authors provide such results?
4 comments
Table 3: Immediate constraint violation and reward error for CostLQR and Swimmer.
| Setting | ZAPD Violation | ZAPD Reward error (%) | CAPS Violation | CAPS Reward error (%) |
|---|---|---|---|---|
| CostLQR, Linear C-PG | 0.43 ± 0.20 | 4.8 ± 4.3 | 1.09 ± 0.37 | 5.6 ± 2.6 |
| CostLQR, Linear PPO-L | 0.62 ± 0.28 | 6.1 ± 4.1 | 1.40 ± 0.71 | 6.2 ± 4.0 |
| CostLQR, NN C-PG | 0.55 ± 0.42 | 11.7 ± 13.5 | 1.13 ± 0.68 | 7.0 ± 3.5 |
| CostLQR, NN PPO-L | 0.73 ± 0.50 | 11.4 ± 9.0 | 1.37 ± 0.83 | 5.4 ± 3.2 |
| Swimmer, Linear C-PG | 0.15 ± 0.24 | 16.7 ± 13.9 | 1.00 ± 0.63 | 8.8 ± 4.9 |
| Swimmer, Linear PPO-L | 0.12 ± 0.09 | 4.9 ± 2.3 | 0.99 ± 0.19 | 10.5 ± 2.3 |
| Swimmer, NN C-PG | 0.63 ± 0.95 | 19.0 ± 21.8 | 1.45 ± 1.34 | 9.5 ± 5.2 |
| Swimmer, NN PPO-L | 0.12 ± 0.11 | 8.0 ± 4.9 | 1.01 ± 0.20 | 9.5 ± 2.0 |
Table 4: Cumulative constraint violation and reward error for CostLQR and Swimmer.
| Setting | ZAPD Violation | ZAPD Reward error (%) | CAPS Violation | CAPS Reward error (%) |
|---|---|---|---|---|
| CostLQR, Linear C-PG | 6.7 ± 15.1 | 4.6 ± 1.5 | 95.6 ± 53.1 | 4.4 ± 0.6 |
| CostLQR, Linear PPO-L | 5.7 ± 7.1 | 4.1 ± 2.0 | 35.1 ± 15.3 | 4.5 ± 0.8 |
| CostLQR, NN C-PG | 16.9 ± 24.3 | 7.3 ± 4.7 | 71.6 ± 41.3 | 5.7 ± 1.3 |
| CostLQR, NN PPO-L | 9.2 ± 10.4 | 6.1 ± 2.8 | 26.7 ± 11.1 | 4.7 ± 0.8 |
| Swimmer, Linear C-PG | 41.0 ± 104.8 | 9.3 ± 5.8 | 334.6 ± 166.9 | 6.5 ± 2.4 |
| Swimmer, Linear PPO-L | 21.6 ± 18.3 | 4.2 ± 1.1 | 78.4 ± 11.4 | 5.2 ± 0.4 |
| Swimmer, NN C-PG | 49.2 ± 82.5 | 8.6 ± 3.3 | 298.7 ± 271.9 | 7.7 ± 3.8 |
| Swimmer, NN PPO-L | 11.2 ± 13.3 | 4.2 ± 2.7 | 77.1 ± 13.6 | 4.7 ± 0.5 |
Thank you for the clarifications and additional experiments. My concerns have been largely addressed, so I am raising my rating to 3.
Thank you very much for raising your score. We greatly appreciate it.
The Authors
We thank the reviewer for recognizing the importance of our work and the empirical results. We appreciate your comprehensive review and would like to address each of your weaknesses and questions.
W1 (primal update as a first-order sensitivity of the optimal policy): Reviewer is right to point out our Theorem 1 is different from standard sensitivity analysis. This is a feature, not a bug. Standard analysis involves a Hessian term, which is hard to estimate precisely during runtime. Our Theorem 1 avoids this term while preserving near-feasibility. In particular, combining Theorem 1 with Eqs. (5)-(6) leads to the following corollary, which we will include in the update paper:
J(θ~)=J(θ∗)+∇θJ(θ∗)⊤(θ~−θ∗)+O(∥θ~−θ∗∥2)=b^+O(∥θ~−θ∗∥2).
Regarding the dual-update proof, [Paternain et al., Constrained Reinforcement Learning Has Zero Duality Gap, NeurIPS, 2019] establish a bound on the duality gap for parameterized CRL policies. So, θ∗ is a maximizer.
W2 (show no-ZAPD): The no-ZAPD policy is equivalent to CAPS in the single-threshold case, which is considerably worse than ZAPD in all cases. (P.S. CAPS trains one policy for each pre-shift threshold and then chooses the appropriate policy based on the actual threshold.)
Q1 (Thm1 ≈ optimal policy?): Please see the response to W1.
Q2 (active-set): We search our data of Bullet Safety Gym tasks to find all instances of the situations: a constraint goes from active to inactive and vice versa. Below are the immediate approximation average error of both cases. Active-set changes occur in 36.3 of cases (27.4 active-to-inactive, 8.8 inactive-to-active).
In both of them, ZAPD remains the best.
Q3 (Show No-ZAPD): Please see the response to W2.
Q4 (Immediate and Cumulative constraint violation after tightening): We thank the reviewer for highlighting the gap between approximation error and safety violations. We report immediate and cumulative constraint violation after tightening (single-threshold). Immediate violation averages max(0,episode cost−b^) over 50 frozen-policy episodes. Cumulative violation sums max(0,iteration cost−b^) until the first feasible iteration, over which cumulative reward error is averaged. The results show that ZAPD has a much smaller violation than CAPS. The violation of ZAPD is slightly larger than CCPO in some cases but that is because CCPO has a much worse reward performance. In particular, ZAPD achieves the lowest cumulative violation in most cases.
Table 1: Immediate constraint violation and reward error for the Bullet Safety Gym tasks.
Table 2: Cumulative constraint violation and reward error for the Bullet Safety Gym tasks.