acceptodds
Under review as a conference paper at ICLR 2027

Taming Phantom Costs: Incremental Lower Bounds for Constrained Reinforcement Learning

Abstract

Constrained Reinforcement Learning (CRL) promises to improve the safety and efficiency of Reinforcement Learning under real world assumptions by replacing the need to tune uniterpretable reward weights with interpretable constraints. However, CRL comes with its own exploration challenges: Cost estimators suffer from random initialization errors and bad generalization which produces spuriously high cost estimates. Since high costs states are, by design, underexplored, those errors never get corrected and cause hallucinated “phantom costs” that needlessly constrain the CRL policy. Existing EM-style surrogate methods address this via monotone safety improvement, but their convergence speed is entangled with the scale of the Q-function through a manually tuned hyperparameter, leading to slow and brittle feasibility recovery as the optimization over- and under- estimates the local uncertainty. We propose ILB-CRL (Incremental Lower Bound CRL), which replaces the hyperparameter with an automatically computed trust coefficient derived from a lower confidence bound (LCB) on the optimal safety improvement . This formulation bridges the gap between incremental improvements, hard projections, and statistical reliability of the cost estimates. We prove geometric convergence under very limited assumptions and demonstrate either above average or state of the art result on a large set of safety-gymnasium environments.

Primary area:
reinforcement learning
Keywords:
Constrained Reinforcement Learning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WVWXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an off-policy constrained RL algorithm based on EM-style surrogate optimisation. Its main innovation is replacing a brittle hyperparameter with a more principled requirement for surrogate cost improvement, enforced through a lower confidence bound (LCB). The approach is supported by two theoretical results. Experiments on Safety Gymnasium tasks show promising performance and good adherence to cost limits.

Critical strengths

The empirical results are reasonably promising, with competitive performance across several tasks and good adherence to the cost limit.

Critical weaknesses
  1. The connection between phantom costs and the proposed method is insufficiently established. The paper is motivated by phantom costs, but it provides neither convincing theoretical justification nor direct empirical evidence that the proposed mechanism addresses this phenomenon. The experiments show that the method can perform reasonably well, but do not establish that its improvements result from correcting phantom costs rather than from other changes to the safety objective or policy-update procedure.

  2. The change to the safety likelihood is insufficiently motivated and evaluated. The authors claim that their safety likelihood better aligns with the notion of safety in CMDPs, but do not adequately explain or demonstrate what this change buys. I find the discussion in Sec. 5.2 unclear and not convincing.

  3. The theoretical results do not clearly support the claims made in the paper. In particular, it is unclear what the theorems imply for the original reward and cost performance of the algorithm, as opposed to the surrogate quantities analysed in the proofs. The abstract's claim of geometric convergence therefore seems insufficiently qualified.

  4. The paper is difficult to follow and insufficiently self-contained. The method is not explained clearly enough for a reader to understand its motivation and design choices. The background on SafeMPO is limited, and important notation is introduced without adequate explanation or motivation. For example, the roles of qq in Eq. (2b) and β\beta in Eq. (3) should be made explicit. The paper would benefit from a clearer exposition of the underlying problem, the proposed mechanism, and how each component addresses the stated motivation.

  5. Experimental reproducibility is insufficiently supported. Appendix C does not provide enough implementation detail to assess whether the comparisons are fair or to reproduce the results. The paper should include a consolidated table of hyperparameters, batch sizes, network architectures, and other relevant implementation choices for both ILB-CRL and its baselines. Computational cost is also not discussed adequately; the authors should report the additional cost of the ensemble-based uncertainty estimation and the overall overhead relative to the baselines.

Questions for authors
  1. What does the claimed geometric convergence guarantee apply to, and what implications does it have for the actual reward and cost performance of the implemented algorithm?

  2. Why should Δ\Delta follow a truncated normal distribution merely because negative improvements are excluded? Can this assumption be validated empirically?

  3. Why is the mean-plus-standard-deviation benchmark of Mattick et al. (2026) used instead of the full bootstrapping estimation of Agarwal et al. (2021)? The paper states that the latter may not apply to the SafeMPO likelihood because it is closer to a per-trajectory than an aggregate formulation. Could the authors clarify this concern and its implications for the reliability of their uncertainty estimates?

11 comments

We first want to thank the reviewer for their time reading our paper. We believe we can address all of your concerns:

The connection between phantom costs and the proposed method is insufficiently established. The paper is motivated by phantom costs, but it provides neither convincing theoretical justification nor direct empirical evidence that the proposed mechanism addresses this phenomenon. The experiments show that the method can perform reasonably well, but do not establish that its improvements result from correcting phantom costs rather than from other changes to the safety objective or policy-update procedure.

We can answer this question in two parts: First, regarding theoretical motivation. Our method relies on general estimator theory, notably that ensembles can be used to measure a model’s epistemic uncertainty (i.e. the prediction uncertainty introduced by a lack of data). This epistemic uncertainty has long been used to guide exploration and prevent getting stuck in local minima in RL (e.g. Osband et al. “Deep Exploration via Bootstrapped DQN”). This work studies a related problem, namely getting stuck in constrained RL, not due to limited exploration, but rather due to an estimator predicting costs in areas where there is none (i.e. “hallucinations” or “erroneous generalizations”). Our algorithm then uses the fact that deep ensembles can estimate epistemic uncertainty, to prevent the policy optimization from overfitting to the cost critic, performing small (perhaps even zero) steps under high uncertainty, and performing full greedy steps under high certainty.

Where we would agree with you is that we did not establish sufficient empirical evidence for reduced stepsize yielding better performance under uncertainty: In practice, supplying this evidence for Deep RL is quite challenging, since, while estimation errors happen almost surely in real RL problems, those RL problems will also be so complex that it becomes hard to nail down why any specific error occurred. Nevertheless, we have designed a simplified Bandit and tabular test (see appendix) where we can isolate the impact of cost critic hallucinations from reward or policy training noise. In these simple environments all methods will eventually recover the true policy (the environments are small enough to enumerate all options), but overtrusting the cost critic will lead to ocscillations and slower convergence. This benchmark demonstrates that even in a simple bandit setting, performing greedy steps α=1\alpha=1 generally leads to slower convergence than performing conservative steps.

Do you think this is sufficient? Alternatively: Is there an experiment you have in mind that can show the difference better?

The change to the safety likelihood is insufficiently motivated and evaluated. The authors claim that their safety likelihood better aligns with the notion of safety in CMDPs, but do not adequately explain or demonstrate what this change buys. I find the discussion in Sec. 5.2 unclear and not convincing. […]Why is the mean-plus-standard-deviation benchmark of Mattick et al. (2026) used instead of the full bootstrapping estimation of Agarwal et al. (2021)? The paper states that the latter may not apply to the SafeMPO likelihood because it is closer to a per-trajectory than an aggregate formulation. Could the authors clarify this concern and its implications for the reliability of their uncertainty estimates?

To ease your concerns, we have added a full cross-validation of all combinations of SafeMPO likelihoods/algorithms and ILB-CRL likelihoods/algorithms. This experiment reveals that ILB-CRL generally outperforms SafeMPO regardless of which likelihood is chosen, and the ILB-CRL likelihood outperforms SafeMPO’s for ILB-CRL. From a theoretical point of view, I think an example might help: Consider the following batch used for estimating the safety likelihood with a budget of 25, where states and actions have equal weights (for the sake of argument)

  • State 1: action 1 cost=30 reward=1, action 2 cost = 30 reward=1
  • State 2: action 1 cost=0 reward=0, action 2 cost=25 reward=1

SafeMPO’s likelihood looks at every state in isolation, so

  • State 1: everything is unsafe, best solution has cost 30
  • State 2: both actions are safe, but action 2 has higher reward, so we chose action 2 with cost 25

Notice that after the update, the expected cost has increased!

This is because State 1 has an irreducible error (so no reduction by default) and state 2 has all actions as safe (isolated to this state!).

ILB-CRL does the following:

  • Compute the expected error over the initial config: 1230+302+120+252\frac{1}{2}\frac{30+30}{2}+\frac{1}{2}\frac{0+25}{2}, so unsafe
  • We cannot do anything in state 1
  • However, we can downweight action 2 in state 1, if 1230+302+12(a0+(1−a)∗25≤25)\frac{1}{2}\frac{30+30}{2}+\frac{1}{2}(a0 + (1-a)*25 \leq 25) works for a=0.2a=0.2

So, ILB-CRL immediately constructs a feasible solution. This is because its likelihood compensates an unsafe state 1 by being more safe in state 2. This way you end up not breaking the expected value constraint. SafeMPO’s likelihood effectively “disables” state compensation, which is why it is not appropriate for expected value constraints.

The theoretical results do not clearly support the claims made in the paper. In particular, it is unclear what the theorems imply for the original reward and cost performance of the algorithm, as opposed to the surrogate quantities analysed in the proofs. The abstract's claim of geometric convergence therefore seems insufficiently qualified.

The proofs analyze the GG function derived directly from the cost function (see method’s section). The reason we abstract away from the direct cost is because our proofs work for either choice of GG. In particular, our new GG is essentially just a reparameterization of the original cost constraint, just written with a negative sign to match the “safety maximization” framework. The geometric convergence is directly in terms of cost, with the basis being the stepsize α\alpha.

The paper is difficult to follow and insufficiently self-contained. The method is not explained clearly enough for a reader to understand its motivation and design choices. The background on SafeMPO is limited, and important notation is introduced without adequate explanation or motivation. For example, the roles of in Eq. (2b) and in Eq. (3) should be made explicit. The paper would benefit from a clearer exposition of the underlying problem, the proposed mechanism, and how each component addresses the stated motivation.

We have used the additional page we have available in the rebuttal to better explain the existing SafeMPO work. Specifically, SafeMPO uses an Expectation-Maximization framework, which iterates between a surrogate qq (represented nonparametrically, i.e. from samples), and the neural network policy. The surrogate qq is optimized to improve the performance of the current policy over the current batch. This can be done with standard techniques of mathematical progamming, since no estimator is used. The optimal qq is then used as a target to optimize the neural network against, i.e. we minimize KL(q∥π)KL(q\|\pi), which is a supervised learning behavioral cloning problem. For better clarity, we have added the constrained problem used by SafeMPO into the preliminaries section.

Please check whether the additional details help! Do you have additional questions or need for clarification?

Experimental reproducibility is insufficiently supported. Appendix C does not provide enough implementation detail to assess whether the comparisons are fair or to reproduce the results. The paper should include a consolidated table of hyperparameters, batch sizes, network architectures, and other relevant implementation choices for both ILB-CRL and its baselines. Computational cost is also not discussed adequately; the authors should report the additional cost of the ensemble-based uncertainty estimation and the overall overhead relative to the baselines.

This is a fair point. We have added the full hyperparameter list into the appendix. We further note that we plan to release the code upon acceptance to further aid reproducibility. Regarding computational cost: This is somewhat hard to analyze. In our benchmarks compute was never the bottleneck, only environment interactions, which is a common property of RL. This is because training a multi-headed RL model is still rather cheap on any modern (or near modern) GPU simply because RL models are not that large in the grand scheme of things. Further, for running the model in the environment, only the policy is executed, which means the ensemble does not change inference compute at all. In wall-clock terms, our method is faster than the original SafeMPO and can be close to or faster than the omnisafe algorithms we benchmarked due to the ability to asynchronously run the environment and training. This is particularly true in slow simulation environments like SafetyAntGoal, where training is completely “free” in wall clock terms since the simulation takes so much time that by the time a new rollout chunk is finished, training has long been completed. In short, while our method is computationally more expensive than others, it’s not in a way that’s realistically measurable since model evaluations are not critical in the first place. As far as wall-clock time is concerned, our method is faster than SafeMPO, though this is just as much a question of superior implementation as it is of genuine algorithmic performance.

What does the claimed geometric convergence guarantee apply to, and what implications does it have for the actual reward and cost performance of the implemented algorithm?

Geometric convergence is proven here in terms of cost first and foremost. Convergence in reward follows directly from the EM-algorithm used (For intuition: the exact MPO algorithm is essentially a Natural policy gradient method with dynamically chosen temperature. You essentially inherit all properties from NPG, which is globally convergent, see e.g., Agarwal et al. “Optimality and Approximation with Policy Gradient Methods in Markov Decision Processes”).

Of course, our method makes simplifying assumptions, specifically that the cost function is realizable in the parameters (see also the discussion with the other reviewers) and that policy is equally representable in the parameters. These are only approximately satisfied in Deep Learning, though empirically, as can be seen in our benchmarks, we appear to be at least sufficiently close to the regime where this holds. We also have to stress that these assumptions are the same as for, e.g. policy gradient methods where one equally has to assume the optimal policy is parameterizable.

In e.g. tabular settings we can guarantee convergence even for naïve fixed α\alpha configurations (for additional empirical verification: see our new toy setup in the appendix). In short, we prove convergence under idealized estimator capacity assumptions, very similar to the settings of contemporary policy gradient methods. We then validate that for realistic scenarios, our method still performs well and is able to learn safe policies.

Why should follow a truncated normal distribution merely because negative improvements are excluded? Can this assumption be validated empirically?

This is indeed an assumption, though a theoretically well motivated one: It is known that the distribution over ensembles (i.e. posterior distribution after observing data) converges to a Normal distribution (the Bernstein-von Mises Theorem). We also know from Mattick et al., that Δ>0\Delta>0. The truncated normal distribution arises from conditioning a normal distribution on the knowledge that values live in a certain band, i.e. if Δ∼N(μ,σ)\Delta\sim N(\mu,\sigma) and we add in the fact that Δ≥0\Delta\geq 0 we obtain Δ∣(Δ≥0)∼TN[0,∞)(μ,σ)\Delta | (\Delta \geq 0) \sim TN_{[0,\infty)}(\mu,\sigma)

Which is the setup we use in the paper. Empirical validation is challenging: We simply do not know what the true Δ\Delta does, we only know its estimate. You can characterize our truncated Normal model as a direct theoretical conclusion from the assumption that the posterior is normally distribution and adding the SafeMPO result that improvements are positive.

Reviewer_iYJpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem of "phantom costs" in constrained reinforcement learning (CRL). Building on SafeMPO, the authors propose three modifications: a new safety likelihood, an E-step solved in two stages that replaces the log barrier, and an automatic choice of α\alpha. On the theoretical side, they argue three points:

  • α\alpha is bounded away from zero.
  • The feasibility residual converges at a geometric rate.
  • α\alpha is the largest step size supported with posterior probability 1−δ1 - \delta. ILB-CRL is compared with seven on-policy baselines implemented in OmniSafe on six Safety-Gymnasium tasks. It performs best on CarPush1 and CarButton1, is competitive on CarGoal1, PointGoal1 and AntGoal1, and falls in the middle of the ranking on HalfCheetahVelocity.
Critical strengths
  1. The problem is well motivated. In constrained RL, costs in rarely visited regions can be overestimated, and this overestimation tends to reinforce itself. The paper also explains why the common remedy, adding an exploration bonus, conflicts with the goal of remaining safe during training.
  2. The reformulation of the incremental safety E-step keeps the optimization problem convex, reduces its dual to only two variables, and yields qq in closed form. It also removes the log barrier used in SafeMPO.
  3. The method achieves strong results on the harder interactive tasks (CarPush1 and CarButton1), and the results are aggregated with a statistically sound protocol.
Critical weaknesses
  1. The choice of baselines is not entirely appropriate. All methods in Table 1 are on-policy baselines implemented in OmniSafe. ILB-CRL, in contrast, is an off-policy method that also relies on RETRACE, PopArt, a critic ensemble, and state augmentation with the consumed budget. It is therefore difficult to tell how much of the improvement comes from the core mechanism proposed in the paper.
  2. The ablation study does not really support the need for an adaptive α\alpha. With a fixed α=0.25\alpha = 0.25, the method still stays within the cost budget on three of the four tasks. Its only clear failure is CarButton1, where the cost reaches 38.9. This is not sufficient to support the claim that fixing α\alpha breaks the method.
  3. The convergence results do not cover the algorithm as it is actually run. Assumption 2 requires the posterior to be uniformly consistent over the entire domain Ω\Omega. This would require data from exactly the regions that the policy actively avoids, so the assumption effectively assumes away the very problem the paper sets out to address.
Questions for authors
  1. Were the SafeMPO results in Table 2 rerun under the experimental protocol of this paper, or taken directly from Mattick et al. (2026)? Have the authors compared the full ILB-CRL method against off-policy constrained RL baselines (e.g., CVPO, CAL or SafeMPO) under the same experimental setup?
  2. How are G^⋆\hat{G}^\star and its values for individual ensemble members computed? If qq in Eq. (7) is fitted to the ensemble mean and then evaluated for each member on the same batch, how is the resulting selection bias handled?
  3. How is the conclusion of Theorem 1 (αm→1\alpha_m \to 1) consistent with the results in Fig. 5? Does Corollary 1 still hold when Δm→0\Delta_m \to 0 along the actual iterates?
  4. Please also address the concerns raised in the Weaknesses section.
2 comments

We want to thank the reviewers for their time reading our work. We particularly want to thank the reviewers for highlighting the strengths of our work, specifically the structurally self-reinforcing estimation errors in constrained RL. We especially want to thank the reviewer for also looking at our appendix, something which is not required by the reviewing guidelines!

Regarding the weaknesses you note, we do think we can clear up all of them:

The choice of baselines is not entirely appropriate. All methods in Table 1 are on-policy baselines implemented in OmniSafe. ILB-CRL, in contrast, is an off-policy method that also relies on RETRACE, PopArt, a critic ensemble, and state augmentation with the consumed budget. It is therefore difficult to tell how much of the improvement comes from the core mechanism proposed in the paper. […] Were the SafeMPO results in Table 2 rerun under the experimental protocol of this paper, or taken directly from Mattick et al. (2026)? Have the authors compared the full ILB-CRL method against off-policy constrained RL baselines (e.g., CVPO, CAL or SafeMPO) under the same experimental setup?

I think these are fair criticisms: The reason we focus mostly on On-policy methods is because, in our experience, they end up being better than the off-policy literature at this moment, mostly due to the complexity of estimating costs/lagrange multipliers off-policy. Our comparison was restricted to SafeMPO since, as a successor to the original CVPO, it removes core limitations of the original CPVO algorithm (To give a specific example: CVPO’s E-step is undefined when a batch randomly has no feasible solution, SafeMPO remains well defined even for infeasible policies) To ease your concerns, we have expanded the original table 2 results for ILB-CRL vs SafeMPO across all possible mixtures of SafeMPO and ILB-CRL likelihoods. We have also replaced the SafeMPO evaluation criterion with our more thorough bootstrapping CI (the original results under SafeMPO’s evaluation scheme have been moved into the appendix) This shows that our method generally outperforms SafeMPO, even when SafeMPO’s likelihood gets replaced by our improved likelihood function. Regarding Architecture: We orient ourselves on SafeMPO’s implementation, which already used RETRACE and PopArt.

The ablation study does not really support the need for an adaptive . With a fixed , the method still stays within the cost budget on three of the four tasks. Its only clear failure is CarButton1, where the cost reaches 38.9. This is not sufficient to support the claim that fixing breaks the method.

Maybe “break” was a little too strong of a word, though moving from a safe policy to a signifcantly unsafe one is a notable issue. In general, for a fixed coefficient in a safety constrained environment, one should not think about the best case configuration for an environment, but rather the worst case: In a real setup, where one does not have the benefit of e.g. crashing cars to estimate a good α\alpha, one has to assume that the found α\alpha will not be the ideal one from the column, but rather a potentially really bad one. Would you agree with our statement if we reduced our claim from “completely breaks” down to “significantly reduces performance”?

The convergence results do not cover the algorithm as it is actually run. Assumption 2 requires the posterior to be uniformly consistent over the entire domain . This would require data from exactly the regions that the policy actively avoids, so the assumption effectively assumes away the very problem the paper sets out to address.

I think this is another good point: However, notice that we only need posterior consistency as a limiting criterion. Basically, to ensure that an estimator has the theoretical capability of learning the true cost function, we need to assume something like posterior consistency in the limit. This way we can then use the fact that as we converge to the true estimator, we will have a strictly positive α\alpha arbitrarily often. This does not mean our updates want (or need) α=1\alpha=1 at any finite point in time. To ease your concern more, we have provided an addition proposition (see appendix), which proves that our estimator does not even need posterior consistency over the entire space: We only need posterior consistency in such a way that we can distinguish good from bad candidate policies within the ε\varepsilon-KL ball. In words, this basically means our assumption gets reduced down to “The estimator has to become good enough to eventually be able to distinguish between candidate policies”, which is probably at least close to a minimal assumption. Does this reduce your concerns regarding assumptions in theorem 1?

Were the SafeMPO results in Table 2 rerun under the experimental protocol of this paper, or taken directly from Mattick et al. (2026)? Have the authors compared the full ILB-CRL method against off-policy constrained RL baselines (e.g., CVPO, CAL or SafeMPO) under the same experimental setup?

We have re-run the experiments in Table 2 under this papers experimental protocol and moved the original results into the appendix. Our experiments show that our method remains superior under equal protocols.

How are G⋆G^\star and its values for individual ensemble members computed? If in Eq. (7) is fitted to the ensemble mean and then evaluated for each member on the same batch, how is the resulting selection bias handled?

This is a good point and probably immediately connected with BwtS’ question regarding Bayesian vs Frequentist viewpoints: Our method takes the Bayesian viewpoint, where the “optimizer’s curse” gets resolved not by using independent draws, but rather by choosing an appropriate prior. We would argue that our choice of using truncated and clipped normals (which is what a clipped Student-T interval corresponds to) are both justifiable assumptions. The question is more whether the chosen prior is sensible and whether a Bayesian credibility interval or a frequentist confidence interval is estimated. For resources regarding this distinction, have a look at “Empirical Bayes and Selective Inference” by Rasines et. al. Alternatively, if one wants to apply a frequentist confidence interval point of view to this optimization scheme, you could easily resolve the selection bias by training an ensemble of size 2N2N and chose half to select Δ\Delta, and half to select α\alpha. However, from the bayesian point of view of this paper this is not necessary.

How is the conclusion of Theorem 1 () consistent with the results in Fig. 5?

Convergence of α\alpha is generally not monotonic (which is completely fine from Theorem 1's point of view). That also makes sense: At the beginning of training the cost decreases within an ε\varepsilon-Ball are quite simple (“don’t walk straight into the wall”) which means α\alpha is high. Later on, where the residual costs are complex to reduce, confidence naturally drops. In fact, this fits quite nicely with our extended proof that show that one only needs to be eventually consistent inside the set of feasible candidates: At the beginning this set is quite obvious, while later on it becomes harder to model.

Does Corollary 1 still hold when along the actual iterates?

Surprisingly yes! This is due to Mattick et al.’s result that demonstrates that for all infeasible policies there will always be a policy that strictly improves within the feasible set by a bounded-from-below fraction, under the minimal assumption that GG is not constant over the entire sampled set. For our work, this has an even nicer mental image: Imagine this like having a KL ball move towards a boundary, either

  • The KL ball overlaps with the feasible set, in which case Δ\Delta may drop, but only because G⋆G^\star is already feasible
  • The KL ball does not overlap the feasible set, in which case G⋆G^\star will be at the boundary, and whatever Δ\Delta is, its density will be a KL distance of ε\varepsilon away This comes directly from the fact that CRL is convex. The actual α\alpha bound does not care, assuming that we eventually recover the true GG within the ε\varepsilon ball, we will eventually “snap” into the feasible set. Before that, we will slowly walk towards the feasible boundary at the rate defined by α\alpha.

We hope we were able to address all of your concerns. If not, please let us know what you still need clarified!

Reviewer_BwtSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ILB-CRL, an uncertainty-aware extension of SafeMPO for handling overestimated costs in CRL. The method uses a more suitable objective and adapts the size of each safety update according to uncertainty in the cost critic. Experiments show competitive results on several benchmarks, but the evidence does not fully support the paper's stronger claims about calibration, phantom-cost correction, or geometric convergence.

Critical strengths
  1. This paper is well-organized and easy-to-follow.

  2. Clear motivation. The paper gives the safety update an interpretable uncertainty-based step size, which is a useful and reasonabe extension of SafeMPO.

  3. The empirical study is fairly broad. Results across several environments and random seeds show that the method is competitive.

  4. The revised aggregate safety likelihood is conceptually better aligned with the expected-cost constraint in a CMDP than a purely per-trajectory criterion. This makes the method easier to interpret and potentially useful beyond the particular benchmark suite.

Critical weaknesses
  1. The theory is stronger than what the implementation establishes. Its convergence analysis assumes posterior consistency and idealized optimization, and the claimed optimality of α relies on a truncated-normal model that is not empirically validated.

  2. The paper does not directly test whether phantom costs are corrected. Its experiments measure final performance, but do not introduce controlled critic errors or track recovery of affected states. The SafeMPO comparison is also difficult to interpret, since the baseline is not fully retuned with the proposed likelihood.

  3. From an algorithmic perspective, the novelty is somewhat limited. ILB-CRL stays within the SafeMPO/MPO surrogate framework, so the paper should more clearly separate its contribution from an improved implementation of an existing approach.

  4. Robustness is not fully established by the experiments. Choices like the KL radius and critic architecture receive little sensitivity analysis. It is therefore difficult to tell how dependent the results are on the selected configuration.

  5. (minor) The presentation needs polishing. I find that many notation choices and terms are inconsistent, and the distinction between frequentist confidence and Bayesian credibility is not maintained throughout the theoretical discussion.

Questions for authors
  1. Is LCB intended as a frequentist confidence bound or a Bayesian credible bound? Can the authors report empirical coverage as a function of the nominal for the ensemble used in the experiments?

  2. Can the authors provide a phantom-cost experiment, in which selected states receive deliberately inflated cost estimates, and report state visitation, critic error, and recovery behavior over training?

  3. Can the authors provide a ablation that separates the effects of the aggregate safety likelihood, removal of the log barrier, and adaptive α, including careful retuning of SafeMPO with the proposed likelihood?

2 comments

First and foremost, we want to thank the reviewer for his informed analysis of our work. We also thank the reviewer for highlighting the strength of our empirical evaluation and clear motivation of our work. Regarding the weaknesses:

The theory is stronger than what the implementation establishes. Its convergence analysis assumes posterior consistency and idealized optimization, and the claimed optimality of α relies on a truncated-normal model that is not empirically validated.

We agree that the assumption of posterior consistency is stronger than necessary: This was also flagged by reviewer iYJp. In general, the reason we rely on a posterior consistency assumption is not because it is completely necessary, but because we need a formal guarantee that whatever estimation/learning scheme is chosen has to theoretical capacity to capture the true cost function. This does not mean we need to ever reach a consistent estimator after finite numbers of steps: Posterior consistency allows us to prove that as data increases, α>α0\alpha>\alpha_0 arbitrarily often, which is what we actually care about for convergence: The α\alpha has to be nonzero often enough. This does not mean that during training the estimator has to be consistent, just that it has the capacity to be consistent eventually To remove your concerns further, we can make the argument in theorem 1 stricter: Notice that we need not know the true cost for any possible state-action, we only need to know it in so far that it affects the change in policy within the ε\varepsilon-KL Ball (and even within that ball: We only need to know for relevant candidates, which further cuts down the knowledge needed). We have added an additional proposition in the appendix which shows exactly this: This amounts to reducing the assumption from “We need to eventually recover the true cost function” to “We need to eventually be able to distinguish different-cost policy candidates”. Does this ease your concern?

Can the authors provide a phantom-cost experiment, in which selected states receive deliberately inflated cost estimates, and report state visitation, critic error, and recovery behavior over training?

Once again, a fair point: In general, setting up experiments for proving this specific issue is quite challenging since it relies on the complex estimator training dynamics especially present in DL. In short: while estimation errors are almost surely going to happen, a benchmark challenging enough to force such estimation errors will naturally be hard to interpret. We nevertheless tried to build a tabular toy benchmark for this challenge, though be aware that this is naturally limited since we cannot simulate the complex training dynamics of DL models. The most notable difference in the dynamics of tabular toy-benchmarks is that, by design, exhaustive exploration is possible, so a phantom cost will not cause a lack of convergence (since π>0\pi>0), but instead will cause slower convergence. This is different to true estimator based systems, where persistent bias (e.g. due to coupling or just very large state-spaces) can cause divergence (or at least a lack of convergence). The results can be found in the appendix: In short, even for a simple bandit-like setup, we can observe that a partial/adaptive step helps in removing estimation errors. In tabular multi-state approaches the same advantage is visible. To really isolate the impact of cost estimation, we give the algorithm white box access to the transition dynamics, reward function, and therefore also Q-function: The only concept the estimator has to learn is the cost per state. Even under these very favorable assumptions, adaptive α\alpha is close to the best fixed coefficient in every setting, while α=1\alpha=1 only performs well in our “control” setups where no phantom costs are present. This supports our conclusions that penalizing overconfidence is a contributor to performance in scenarios where estimation errors are present. While these experiments are admittedly simplifications of the true complexity of deep RL, we do think that it is at least a strong argument for why estimation errors can cause slowing convergence. Do you think this is sufficient to show that hallucinated costs can be reduced via reduced steps? Do you have a different experiment in mind?

Is LCB intended as a frequentist confidence bound or a Bayesian credible bound? Can the authors report empirical coverage as a function of the nominal for the ensemble used in the experiments?

The work should be seen as a Bayesian bound: This was a language barrier issue as the name in the author’s native language are synonyms (just with “Bayesian” or “frequentist” as prefixes). We have revised the work to consistently use “credibility” instead of “confidence”. Specifically, we assume a distributional family (clipped normal for the truncated student-T, truncated Normal for the truncated Normal model), and estimate the credible interval under an uninformative (pseudo)-prior (though at least for the Student-T the difference is academic since frequentist and Bayesian bounds happen to coincide). Regarding coverage: The issue with proving coverage is that we cannot measure what the true Δ\Delta given data would be if we had white box access to it. The reason we opted for a truncated Normal model is due to a mixture of computational and estimator theory (von Mises-Bernstein), combined with the assumption \Delta_\min\geq0, which was proven in Mattick et al. We are sorry for the confusion caused by the mixing of credibility vs confidence and hope this has cleared up any problems you might have had.

Can the authors provide a ablation that separates the effects of the aggregate safety likelihood, removal of the log barrier, and adaptive α, including careful retuning of SafeMPO with the proposed likelihood?

We already provide an ablation of adaptive vs fixed α\alpha in the appendix. For the other alterations: removing the log⁡\log-barrier without adding anything to force the model towards the feasible set leads to a model that never improves beyond initial costs (see SafeMPO, theorem 1). In our case this “push” is performed by forcing the policy into a strictly improving area (since G⋆G^\star is improving if the policy is not safe already, and α>0\alpha>0 eventually). Regarding the SafeMPO with our new likelihood function: We have added a full comparison (ours with SafeMPO likelihood, Ours with our likelihood, SafeMPO with our likelihood, SafeMPO with SafeMPO likelihood) under our stringent evaluation standards. This replaces the existing table 2 (which has been moved to the appendix) with a more thorough evaluation. The benchmark demonstrates that our likelihood generally performs better in our algorithm, but not necessarily in SafeMPO. In general, ILB-CRL performs better than SafeMPO regardless of likelihood function chosen.

Regarding our contribution: This work mostly contributes three core aspects to the existing literature

  • A better safety likelihood function
  • A mathematically maximal greedy improvement step
  • A discount system to prevent overfitting to erroneous cost estimates

Steps 2 and 3 specifically follow the basic intuition of modern reinforcement learning: Derive a greedy step to know where to go, then introduce a conservatism regularization to account for estimation errors.

Comments on the paper

1 comment

As the discussion period is approaching its end, please take a moment to review the authors’ responses and share any remaining thoughts or updates to your assessment. Thanks for your thoughtful reviews and discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.