- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes an off-policy constrained RL algorithm based on EM-style surrogate optimisation. Its main innovation is replacing a brittle hyperparameter with a more principled requirement for surrogate cost improvement, enforced through a lower confidence bound (LCB). The approach is supported by two theoretical results. Experiments on Safety Gymnasium tasks show promising performance and good adherence to cost limits.
Critical strengths
The empirical results are reasonably promising, with competitive performance across several tasks and good adherence to the cost limit.
Critical weaknesses
-
The connection between phantom costs and the proposed method is insufficiently established. The paper is motivated by phantom costs, but it provides neither convincing theoretical justification nor direct empirical evidence that the proposed mechanism addresses this phenomenon. The experiments show that the method can perform reasonably well, but do not establish that its improvements result from correcting phantom costs rather than from other changes to the safety objective or policy-update procedure.
-
The change to the safety likelihood is insufficiently motivated and evaluated. The authors claim that their safety likelihood better aligns with the notion of safety in CMDPs, but do not adequately explain or demonstrate what this change buys. I find the discussion in Sec. 5.2 unclear and not convincing.
-
The theoretical results do not clearly support the claims made in the paper. In particular, it is unclear what the theorems imply for the original reward and cost performance of the algorithm, as opposed to the surrogate quantities analysed in the proofs. The abstract's claim of geometric convergence therefore seems insufficiently qualified.
-
The paper is difficult to follow and insufficiently self-contained. The method is not explained clearly enough for a reader to understand its motivation and design choices. The background on SafeMPO is limited, and important notation is introduced without adequate explanation or motivation. For example, the roles of in Eq. (2b) and in Eq. (3) should be made explicit. The paper would benefit from a clearer exposition of the underlying problem, the proposed mechanism, and how each component addresses the stated motivation.
-
Experimental reproducibility is insufficiently supported. Appendix C does not provide enough implementation detail to assess whether the comparisons are fair or to reproduce the results. The paper should include a consolidated table of hyperparameters, batch sizes, network architectures, and other relevant implementation choices for both ILB-CRL and its baselines. Computational cost is also not discussed adequately; the authors should report the additional cost of the ensemble-based uncertainty estimation and the overall overhead relative to the baselines.
Questions for authors
-
What does the claimed geometric convergence guarantee apply to, and what implications does it have for the actual reward and cost performance of the implemented algorithm?
-
Why should follow a truncated normal distribution merely because negative improvements are excluded? Can this assumption be validated empirically?
-
Why is the mean-plus-standard-deviation benchmark of Mattick et al. (2026) used instead of the full bootstrapping estimation of Agarwal et al. (2021)? The paper states that the latter may not apply to the SafeMPO likelihood because it is closer to a per-trajectory than an aggregate formulation. Could the authors clarify this concern and its implications for the reliability of their uncertainty estimates?
11 comments
The change to the safety likelihood is insufficiently motivated and evaluated. The authors claim that their safety likelihood better aligns with the notion of safety in CMDPs, but do not adequately explain or demonstrate what this change buys. I find the discussion in Sec. 5.2 unclear and not convincing. […]Why is the mean-plus-standard-deviation benchmark of Mattick et al. (2026) used instead of the full bootstrapping estimation of Agarwal et al. (2021)? The paper states that the latter may not apply to the SafeMPO likelihood because it is closer to a per-trajectory than an aggregate formulation. Could the authors clarify this concern and its implications for the reliability of their uncertainty estimates?
To ease your concerns, we have added a full cross-validation of all combinations of SafeMPO likelihoods/algorithms and ILB-CRL likelihoods/algorithms. This experiment reveals that ILB-CRL generally outperforms SafeMPO regardless of which likelihood is chosen, and the ILB-CRL likelihood outperforms SafeMPO’s for ILB-CRL. From a theoretical point of view, I think an example might help: Consider the following batch used for estimating the safety likelihood with a budget of 25, where states and actions have equal weights (for the sake of argument)
- State 1: action 1 cost=30 reward=1, action 2 cost = 30 reward=1
- State 2: action 1 cost=0 reward=0, action 2 cost=25 reward=1
SafeMPO’s likelihood looks at every state in isolation, so
- State 1: everything is unsafe, best solution has cost 30
- State 2: both actions are safe, but action 2 has higher reward, so we chose action 2 with cost 25
Notice that after the update, the expected cost has increased!
This is because State 1 has an irreducible error (so no reduction by default) and state 2 has all actions as safe (isolated to this state!).
ILB-CRL does the following:
- Compute the expected error over the initial config: , so unsafe
- We cannot do anything in state 1
- However, we can downweight action 2 in state 1, if works for
So, ILB-CRL immediately constructs a feasible solution. This is because its likelihood compensates an unsafe state 1 by being more safe in state 2. This way you end up not breaking the expected value constraint. SafeMPO’s likelihood effectively “disables” state compensation, which is why it is not appropriate for expected value constraints.
The theoretical results do not clearly support the claims made in the paper. In particular, it is unclear what the theorems imply for the original reward and cost performance of the algorithm, as opposed to the surrogate quantities analysed in the proofs. The abstract's claim of geometric convergence therefore seems insufficiently qualified.
The proofs analyze the function derived directly from the cost function (see method’s section). The reason we abstract away from the direct cost is because our proofs work for either choice of . In particular, our new is essentially just a reparameterization of the original cost constraint, just written with a negative sign to match the “safety maximization” framework. The geometric convergence is directly in terms of cost, with the basis being the stepsize .
The paper is difficult to follow and insufficiently self-contained. The method is not explained clearly enough for a reader to understand its motivation and design choices. The background on SafeMPO is limited, and important notation is introduced without adequate explanation or motivation. For example, the roles of in Eq. (2b) and in Eq. (3) should be made explicit. The paper would benefit from a clearer exposition of the underlying problem, the proposed mechanism, and how each component addresses the stated motivation.
We have used the additional page we have available in the rebuttal to better explain the existing SafeMPO work. Specifically, SafeMPO uses an Expectation-Maximization framework, which iterates between a surrogate (represented nonparametrically, i.e. from samples), and the neural network policy. The surrogate is optimized to improve the performance of the current policy over the current batch. This can be done with standard techniques of mathematical progamming, since no estimator is used. The optimal is then used as a target to optimize the neural network against, i.e. we minimize , which is a supervised learning behavioral cloning problem. For better clarity, we have added the constrained problem used by SafeMPO into the preliminaries section.
Please check whether the additional details help! Do you have additional questions or need for clarification?
Experimental reproducibility is insufficiently supported. Appendix C does not provide enough implementation detail to assess whether the comparisons are fair or to reproduce the results. The paper should include a consolidated table of hyperparameters, batch sizes, network architectures, and other relevant implementation choices for both ILB-CRL and its baselines. Computational cost is also not discussed adequately; the authors should report the additional cost of the ensemble-based uncertainty estimation and the overall overhead relative to the baselines.
This is a fair point. We have added the full hyperparameter list into the appendix. We further note that we plan to release the code upon acceptance to further aid reproducibility. Regarding computational cost: This is somewhat hard to analyze. In our benchmarks compute was never the bottleneck, only environment interactions, which is a common property of RL. This is because training a multi-headed RL model is still rather cheap on any modern (or near modern) GPU simply because RL models are not that large in the grand scheme of things. Further, for running the model in the environment, only the policy is executed, which means the ensemble does not change inference compute at all. In wall-clock terms, our method is faster than the original SafeMPO and can be close to or faster than the omnisafe algorithms we benchmarked due to the ability to asynchronously run the environment and training. This is particularly true in slow simulation environments like SafetyAntGoal, where training is completely “free” in wall clock terms since the simulation takes so much time that by the time a new rollout chunk is finished, training has long been completed. In short, while our method is computationally more expensive than others, it’s not in a way that’s realistically measurable since model evaluations are not critical in the first place. As far as wall-clock time is concerned, our method is faster than SafeMPO, though this is just as much a question of superior implementation as it is of genuine algorithmic performance.
What does the claimed geometric convergence guarantee apply to, and what implications does it have for the actual reward and cost performance of the implemented algorithm?
Geometric convergence is proven here in terms of cost first and foremost. Convergence in reward follows directly from the EM-algorithm used (For intuition: the exact MPO algorithm is essentially a Natural policy gradient method with dynamically chosen temperature. You essentially inherit all properties from NPG, which is globally convergent, see e.g., Agarwal et al. “Optimality and Approximation with Policy Gradient Methods in Markov Decision Processes”).
Of course, our method makes simplifying assumptions, specifically that the cost function is realizable in the parameters (see also the discussion with the other reviewers) and that policy is equally representable in the parameters. These are only approximately satisfied in Deep Learning, though empirically, as can be seen in our benchmarks, we appear to be at least sufficiently close to the regime where this holds. We also have to stress that these assumptions are the same as for, e.g. policy gradient methods where one equally has to assume the optimal policy is parameterizable.
In e.g. tabular settings we can guarantee convergence even for naïve fixed configurations (for additional empirical verification: see our new toy setup in the appendix). In short, we prove convergence under idealized estimator capacity assumptions, very similar to the settings of contemporary policy gradient methods. We then validate that for realistic scenarios, our method still performs well and is able to learn safe policies.
Why should follow a truncated normal distribution merely because negative improvements are excluded? Can this assumption be validated empirically?
This is indeed an assumption, though a theoretically well motivated one: It is known that the distribution over ensembles (i.e. posterior distribution after observing data) converges to a Normal distribution (the Bernstein-von Mises Theorem). We also know from Mattick et al., that . The truncated normal distribution arises from conditioning a normal distribution on the knowledge that values live in a certain band, i.e. if and we add in the fact that we obtain
Which is the setup we use in the paper. Empirical validation is challenging: We simply do not know what the true does, we only know its estimate. You can characterize our truncated Normal model as a direct theoretical conclusion from the assumption that the posterior is normally distribution and adding the SafeMPO result that improvements are positive.
We first want to thank the reviewer for their time reading our paper. We believe we can address all of your concerns:
We can answer this question in two parts: First, regarding theoretical motivation. Our method relies on general estimator theory, notably that ensembles can be used to measure a model’s epistemic uncertainty (i.e. the prediction uncertainty introduced by a lack of data). This epistemic uncertainty has long been used to guide exploration and prevent getting stuck in local minima in RL (e.g. Osband et al. “Deep Exploration via Bootstrapped DQN”). This work studies a related problem, namely getting stuck in constrained RL, not due to limited exploration, but rather due to an estimator predicting costs in areas where there is none (i.e. “hallucinations” or “erroneous generalizations”). Our algorithm then uses the fact that deep ensembles can estimate epistemic uncertainty, to prevent the policy optimization from overfitting to the cost critic, performing small (perhaps even zero) steps under high uncertainty, and performing full greedy steps under high certainty.
Where we would agree with you is that we did not establish sufficient empirical evidence for reduced stepsize yielding better performance under uncertainty: In practice, supplying this evidence for Deep RL is quite challenging, since, while estimation errors happen almost surely in real RL problems, those RL problems will also be so complex that it becomes hard to nail down why any specific error occurred. Nevertheless, we have designed a simplified Bandit and tabular test (see appendix) where we can isolate the impact of cost critic hallucinations from reward or policy training noise. In these simple environments all methods will eventually recover the true policy (the environments are small enough to enumerate all options), but overtrusting the cost critic will lead to ocscillations and slower convergence. This benchmark demonstrates that even in a simple bandit setting, performing greedy steps α=1 generally leads to slower convergence than performing conservative steps.
Do you think this is sufficient? Alternatively: Is there an experiment you have in mind that can show the difference better?