acceptodds
Withdrawn from ICLR 2027

Beyond Lagrangian Methods for Robust Peak-Cost Constrained Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study robust peak-cost constrained reinforcement learning, where the goal is to maximize reward while bounding the largest cost incurred along a trajectory – unlike the constrained MDP (CMDP), which only keeps the expected cumulative cost below a threshold. We first show that the problem is exactly equivalent to a CMDP on a state augmented with the running maximum cost, which restores zero duality gap but requires history-dependent policies, yields a cost signal that is active only at a new peak, and makes worst-case evaluation conservative. We therefore keep the reachability-constrained (RCRL) formulation on the original state space, for which strong duality can fail, and optimize a surrogate rather than a Lagrangian, estimating worst-case values through an integral probability metric. Our main contribution is the first convergence analysis for this surrogate. We prove that every iterate stays -safe from a safe start which the primal-dual-based approach can not guarantee, and that a stationary point of the exact surrogate is reached. Under a nondegeneracy condition, we further obtain a globally -optimal and -safe policy in iterations from a safe start, and otherwise. Experiments on continuous-control benchmarks, including RCRL, augmented CMDP and robust CMDP baselines, show that the method sustains reward while enforcing peak-cost safety under dynamics perturbations.

Primary area:
reinforcement learning
Keywords:
Robust Reinforcement Learning, Constrained MDP

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fhwwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how reinforcement learning agents can earn high rewards while keeping the largest cost during a task below a safety limit, even when the environment changes. A surrogate objective is used instead of Lagrangian optimisation and provides conditions for safety during training and convergence to a near-optimal policy. The paper adopts a reachability-based formulation on the original state space and analyses a surrogate that takes the maximum of scaled negative reward and constraint violation. For exact proximal updates in finite MDPs, the paper establishes approximate safety throughout optimisation and a stationarity guarantee, with global near-optimality under additional nondegeneracy and coverage assumptions. Experiments on continuous-control tasks suggest that the method balances reward and safety under changes in the environment’s dynamics.

Critical strengths
  1. The paper addresses an important safety objective: controlling large individual costs under dynamics mismatch. This complements cumulative-cost formulations, which can obscure isolated severe violations.

  2. The distinction between expected trajectory peak cost and the reachability-based value is useful. The augmented CMDP construction clarifies when additive constrained optimization applies, while Remark 1 acknowledges that the original-state formulation generally represents a different constraint.

  3. The convergence analysis provides a meaningful contribution beyond the surrogate’s solution-quality guarantee. Representing the objective as a maximum of smooth branch values enables weak-convexity analysis and an approximate-safety bound for intermediate proximal iterates.

  4. The experiments examine several relevant aspects, including multiple constraints, feasible initialization, sensitivity to the surrogate parameter, and dynamics perturbations. Evaluating observed trajectory peaks rather than relying exclusively on critic predictions is also appropriate.

  5. The appendix explicitly discusses limitations of the theory, including the nondegeneracy condition and the unresolved analysis at points where reward and constraint branches meet. This helps readers assess the scope of the guarantees.

  6. The method handles multiple safety constraints without learning a separate Lagrange multiplier for each one, offering a simpler way to manage competing constraints.

Critical weaknesses
  1. Theorem 2 establishes an approximate-safety bound from a feasible initialization. However, the contributions and conclusion claim that violation never rises above its initial level. Since the theorem permits positive violation up to J(π0)/βJ(\pi_0)/\beta even when initial violation is zero, this stronger claim is not established by the provided proof.

  2. The global optimality guarantee in Theorem 3 requires matching gradients across all active branches and a finite coverage coefficient. As acknowledged in Remark 8, surrogate minimisers may lie where the reward and constraint branches meet, where the gradient condition can fail. Since the analysis does not establish that the selected iterate satisfies this condition, the global convergence rates remain conditional and may not cover important solutions.

  3. SO and RP-CRL are trained under different conditions: RP-CRL also sees changes in the environment during training. Therefore, it is unclear whether its better performance comes from the IPM critic or from having trained with these changes.

  4. The experiments include several relevant baselines and studies of the surrogate parameter and policy initialization. However, the number of seeds and matched training budgets are unclear, and computational overhead is not reported. The robustness comparison also changes the training environment alongside the IPM critic, making it difficult to identify which component produces the improvement.

Questions for authors
  1. Can the nondegeneracy condition be verified for the returned policy? What guarantee remains when the selected iterate lies at a reward–constraint kink with different active gradients?

  2. How does the one-step peak TD advantage, sampled under ordinary trajectories, account for the killed-chain occupancy appearing in the theoretical gradient?

  3. Were SO and RP-CRL trained with identical dynamics perturbations? If not, how should their comparison be interpreted?

  4. Please clarify the implemented branch-selection rule, the number of independent seeds, training budgets, and the definition of the reported violation rate.

Also see the weaknesses section. The paper offers a useful conditional convergence analysis for a relevant surrogate, but its strongest safety and global optimality claims need narrower wording. My main reservations concern the scope of the guarantees, their connection to the practical actor update, and the interpretation of the experimental comparisons.

Reviewer_979xOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The tackles a RL problem involving a peak cost constraints. This criterion is not linear in the occupancy measure unless the CMDP is augmented with a new state responsible for tracking this peak cost. The work considers instead a surrogate problem.

Main theoretical result: convergence analysis for the policies optimizing this surrogate problem in an exact setting (no function approximation, exact proximal steps). The method allows to learn safely (with probability epsilon) if started from a feasible policy.

Main practical results: a robust PPO-based algorithm evaluated on continuous benchmarks.

Critical strengths

The paper tackles an interesting problem an is transparent about the limitation so its approach and theoretical results. The manuscript also explores solutions beyond Lagrangian methods, that are are mainstream in the field.

Critical weaknesses
  • The theoretical results hold for the tabular setting using direct parametrization. This setting considerably simplifies the problem since it makes RL problems "essentially convex" and policy gradient is well-known to converge (see, e.g., [Bhandari et al., Global optimality guarantees for policy gradient methods, 2024] or [Mei et al., On the global convergence rates of softmax policy gradient methods, ICML, 2020]).
  • The additional nondegeneracy requirement on the "active branches of FβF_\beta" essentially renders the maximum operation differentiable. This assumption is not mild and is, in fact, unlikely to hold in practice given that optimizing peak and average rewards are not typically consistent (if that were the case, then the problem tackled in this paper would not be relevant).
  • The manuscript is missing a discussion on the complexity of computing the proximal step in (9). In fact, the experiments do not appear to use it, so that the theoretical guarantees do not apply. Though theoretically useful, proximal methods can only really be applied when the prox operator is easy to compute (which in general stands for "exists in closed form"). This does not appear to be the case in for FβF_\beta.
  • It is difficult to fully interpret the experimental results in view of many missing details. What is the "PD" algorithm (learning rates? critics?)? What is the exact criterion optimized by each method during training? Are the same training hyperparameters used across methods? How many seeds are compared (line 360: "Results are averaged over N seeds")? These typos and missing information make it difficult to evaluate the results and contributions of the paper.
  • This issue is even more pronounced in Section 4, the main section of the paper. Its organization makes it difficult to go through and clearly distinguish which results are new and which already exist in, e.g., [Yu el al., 2022; Mukhopadhyay et al., 2026; Ganguly et al., 2026]. Also, a discussion on the relation (similarities and differences) between new and existing results and/or proof techniques would make it easier to pinpoint the exact contribution of the paper. A substantial copy-editing is necessary to clarify notation and fix typos that severely affect readability. For instance, notations are often used before clear definition, including ΠS\Pi_\mathcal{S} (line 202) or ρ\rho which has different meanings in line 259 and 312; line 303-304 has the partial sentence "Fβ(π) πF_\beta(\pi)\ \pi, but it is a pointwise maximum"...
  • Experiments should also compare to other constrained methods such as [Sootla et al., 2022] and [Paternain et al., 2022] (unless they are not applicable this scenario). In particular, the latter derives probabilistic guarantees from value function constraints that appear to apply to this problem as well.
  • The manuscript goes through a series of towering transformations of the original problem in (1) to the point that it is no longer clear whether it actually tackles it or not. More precisely, (2) is written in terms of max⁡{c(s,a),Es′[Vc,peak(s′)]}\max \{ c(s,a), \mathbb{E}_{s^\prime}[V_{c,peak}(s^\prime)] \}. Since the critic is fitted by least squares using the target in (29) that considers a single sample st+1s_{t+1}, it instead converges to Es′[max⁡{c(s,a),Vc,peak(s′)}]\mathbb{E}_{s^\prime}[ \max \{ c(s,a), V_{c,peak}(s^\prime) \}]. Both quantities coincide under deterministic transitions, but the second is larger for stochastic transitions (by Jensen's inequality). This appears to affect the results of Section 4 (except the example in Theorem 1, which is deterministic).
Questions for authors
  • What are the relations between the convergence results and/or proof techniques from Section 4 and those in, e.g., [Bhandari et al., Global optimality guarantees for policy gradient methods, 2024] on the fact that RL objectives satisfy the PL property?
  • How stringent is the nondegeneracy requirement on the "active branches of FβF_\beta" from Theorem 3? How critical is it to the proof of the results?
  • What is the "PD" algorithm (learning rates? critics?)? What is the exact criterion optimized by each method during training? Are the same training hyperparameters used across methods? How many seeds are compared (line 360: "Results are averaged over N seeds")?
  • Could the authors clarify which results from Section 4 are new and how they relate (similarity and differences) to previous results?
  • Could the probabilistic safety guarantees from [Paternain et al., 2022] be used in this setting (letting safety be defined in terms of a peak cost restriction)? If so, how would those probabilistic requirements compare to the average requirement in (1)? In this case, could the surrogate results from [Ganguly et al., 2026] be applied directly?
  • How hard is it to tune β\beta in practice? This hyperparameter has to absorb the scale difference between JJ and W−bW-b, which is problem-dependent (cf. experiments). How was it done in the experiments?
  • With respect to comment around (2) and (29): which of the two quantities is the critic meant to estimate? If it is Es′[max⁡{c(s,a),Vc,peak(s′)}]\mathbb{E}_{s^\prime}[ \max \{ c(s,a), V_{c,peak}(s^\prime) \}], do Lemma 2 and the results of Section 4 carry over to the corresponding operator?
Reviewer_Vv5ROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper concludes further new convergence analysis under tabulated settings to a previous safe RL framework (RP-CRL) [1] under peak-cost setting, in which the maximum single step cost of constraint violations is focused, and further extend this method's benchmarks by including more environments. The results show that RP-CRL can perform competitively against Lagrangian based safe RL methods in the tested tasks.

[1] Shilpa Mukhopadhyay, Sourav Ganguly, Santosh Mohan Rajkumar, Honghao Wei, Debdipta Goswami, and Arnob Ghosh. Robust peak cost constrained reinforcement learning. arXiv preprint arXiv:2607.15457, 2026.

Critical strengths
  1. The theoretical analysis is a nice addition. Writing the peak-cost value as a maximum of smooth branch values makes the surrogate weakly convex. This gives the first convergence guarantees for this surrogate. The authors are also showing the limitations in their derivations.
  2. The motivation for peak-cost constraints over cumulative ones is clear. The discussion of why max state augmentation is unattractive is informative and interesting.
  3. The experiments cover a reasonable range of environments, including Humanoid, a multi-constraint task and a quadrotor, against several baselines. Hyperparameters are reported in detail.
Critical weaknesses
  1. Visualization: The figures appear to be low-resolution PNGs, and the space between figures and text is too tight. In Fig. 3, the HalfCheetah cost text is covered by the figure above it. Plot ranges are odd: curves are squeezed together while space is wasted (e.g., Swimmer cost in Fig. 2). Fig. 1, meant as motivation, conveys little, and its caption doesn’t help.
  2. Clarity of Writing: The writing is often hard to follow. Terms are used before being defined or cited (IPM at line 95, PPO at line 97), and RP-CRL is never spelled out. Some statements are ambiguous: “the problem” in the abstract is said to be exactly an augmented CMDP, but this only holds for the non-robust expected-peak problem.
  3. Limited contribution. The surrogate, IPM critics, duality-gap result and RP-CRL itself all come from Mukhopadhyay et al. (2026). The new part is mainly the convergence analysis, which covers an exact tabular proximal-point method rather than the evaluated algorithm.
  4. Overclaim. Robustness to the worst case is claimed, but evaluation averages over random perturbations, and ρ is tiny and never ablated. It motivates with robustness under uncertainty, but the tasks evaluated are not really demonstrating that.
Questions for authors
  1. Beyond the convergence analysis and additional experiments, what is new relative to Mukhopadhyay et al. (2026)? Is RP-CRL the same algorithm?
  2. Most costs (Ant, HalfCheetah, Swimmer, Humanoid, quadrotor) depend only on the action. How does RP-CRL perform on tasks where the peak cost depends on the state and the dynamics?
  3. Only circle task within navigation suite is evaluated, it would be nice to test also other navigation tasks such as Goal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.