acceptodds
Withdrawn from ICLR 2027

Safe on Average, Unsafe in the Tail: When Is the Episodic-Cost Tail Controllable?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safe reinforcement learning seeks policies that maximize return while satisfying constraints on cumulative cost. Most methods impose these constraints on ex- pected episodic cost. Consequently, standard evaluations report mean episodic cost without characterizing how cost is distributed across episodes. A policy that satisfies the mean-cost criterion may therefore remain unsafe in its worst episodes. Mean-cost reporting neither identifies this tail violation nor shows whether it can be brought within budget while preserving return. In this work, We measure the episodic-cost tail using , the average cost of the worst 10% of episodes. We classify a policy as tail-safe when is within the safety budget. This allows us first to identify policies that are safe on average but unsafe in the tail and then to study whether their tail violations can be controlled while preserving return. To identify tail-unsafe policies, we evaluate five standard algorithms on three Safety-Gymnasium navigation tasks. We then examine four constraint fami- lies on dense-hazard navigation and assess tail control across four navigation and four locomotion tasks.

Primary area:
reinforcement learning
Keywords:
safe reinforcement learning, risk-sensitive RL, CVaR / tail risk, constrained MDP, Safety-Gymnasium.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hYPwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the limitations of the expected-cost constraints formulation in safe RL. The analysis and experiments use CVaR as the main metric to measure the risk of high-cost episodes and assess whether distributional safe RL approaches can control this tail risk. The OQ-SAC in this paper is an ORAC-inspired SAC variant (ORAC is a pre-printed distributional safe RL method). Experiments on safety-gymnasium tasks show that tail-risk in episodes is controllable for some locomotion tasks, but remains difficult in safe navigation tasks. The following analysis shows that tail-risk controllability can be influenced by the occurrence of high-return episodes.

Critical strengths
  1. The chosen CVaR metric and the evaluation protocol are reasonable for assessing the tail risk in episodes for safe RL algorithms.

  2. This paper presents an interesting explanation of why tail-risk control for navigation tasks is more difficult.

Critical weaknesses
  1. This paper does not have much added value or novelty. It does not show any new observed phenomena, nor does it identify any new questions or provide any new safe RL algorithms. Episode-level tail-risk control is a well-known issue in safe RL. This paper seems to conduct experiments to analyze this phenomenon again, and the conclusions are mostly already reported in quite some existing papers.

  2. The benchmark and experimental design are problematic. Distributional safe RL is a widely used method to mitigate tail risk. However, the authors only benchmark and evaluate conventional, point-value safe RL methods in the analysis, and treat OQ-SAC, a distributional safe RL variant, as a solution. This looks like the paper propose an "existing" question and using an "existing" solution to fix it.

Questions for authors

Q1. What is actually novel in this paper? As the second weakness says, the tail-risk issue and the distributional safe RL solutions have already been proposed and studied in the literature (including ORAC cited by this paper). This paper seems analysis-only.

Q2. In Table 1, only point-value safe RL methods are used to show the tail-risk issue. Why not include distributional safe RL methods to examine if they also have the same issue?

Q3. In Table 2, as OQ-SAC is a variant of ORAC, why is it not compared with ORAC, but with two point-value safe RL methods again? According to the ORAC paper, it can already effectively mitigate tail risk.

Q4. There is a severe result issue in Section 8: the authors write, "The worst-decile cost reaches 10.2 times the mean...". How could this be possible? Mathematically, CVaR 0.1 can be maximum 10 times the mean, right?

Q5. For the result of Figure 3, the authors say that the separate clusters of episode cost probably make tail-risk control harder. However, this result cannot provide any causal explanation. Theoretically, quantile-based critics can handle this case effectively. I think the authors should avoid this over-claiming or add controlled experiments or ablations to clarify the real causes of the failure of navigation tasks.

Reviewer_xtCUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that expected episodic-cost constraints conceal tail risk, and introduces a shared post-training protocol reporting both mean episodic cost and CVaR_0.1, the average cost of the worst 10% of episodes. On three Safety Gymnasium navigation tasks, five standard algorithms exhibit worst-decile costs 4.4 to 18.2 times the budget, and none is tail-safe. Four constraint families on dense-hazard navigation (feasibility gating, budget-as-state, tail-driven Lagrangian, quantile-CVaR critic) all fail: return collapses before the tail enters the budget. The authors then propose OQ-SAC, an ORAC-style controller combining optimistic exploration with a quantile-CVaR critic penalty, which is seed-level tail-safe on three of four locomotion tasks but no navigation task; ablations show neither component is necessary. Episode-level analysis suggests tail control is possible when safe, high-return episodes occur frequently enough. The main contribution is evaluative: a protocol plus an empirical account of when episodic tails are controllable.

Critical strengths
  1. The paper delivers an important evaluation message with a shared protocol: mean episodic cost systematically hides worst-case behavior, and the measured 4.4-18.2x tail violations across five standard algorithms are a clear, consequential warning for how safe RL results are currently reported and benchmarked.
  2. Evaluating four heterogeneous constraint families under one protocol and documenting a consistent failure mode on dense-hazard navigation is unusually comprehensive negative evidence. Standardized post-training operating points in Figure 2 make the tightening behavior comparable across mechanisms rather than anecdotal.
  3. The episode-level analysis, including return-cost scatter plots, safe-episode frequency, and within-episode cost trajectories, offers a plausible empirical characterization of tail controllability and is reported candidly, with seed-level variation and explicit limitations on domain confounds and estimator noise summarized in Table 3.
Critical weaknesses
  1. Each per-seed CVaR_0.1 averages only ten tail episodes with no confidence intervals or significance tests; under heavy-tailed costs this estimator is extremely noisy, so safe-on-every-seed claims are statistically weak. Moreover, all methods train value functions on discounted costs while the protocol evaluates undiscounted totals, an uncontrolled mismatch that could itself drive tail violations.
  2. OQ-SAC's positive results depend on task-specific fixed multipliers spanning two orders of magnitude (0.06 to 30), and ablations show neither the quantile-CVaR penalty nor optimistic exploration is necessary; the method thus lacks an identifiable mechanism, the reactive-multiplier rejection rests on a single seed, and the algorithmic contribution over ORAC is unclear.
  3. The main failure case, navigation, is never compared against a modern tail-aware baseline, so tails-are-uncontrollable cannot be attributed to the tested mechanisms; the hazard-count study is single-seed with an unexplained non-monotonic two-hazard result, and no analysis diagnoses why constraint tightening collapses return, leaving the proposed explanation post-hoc.
Questions for authors

See weaknesses.

Reviewer_g5kAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that current safe RL algorithms only evaluate average episodic cost which fails to describe how the worst-case episode rollouts perform. The paper proposed to use tail risk or tail safeness as a metric which is essentially a CVAR(0.1) or the average of worst 10% of the episodes. The paper also analyze whether tightening the constraint drives return to zero before the tail reaches the budget. The paper runs analysis finding that for safe velocity environments the tail safe behavior can be achieved without hurting the episodic return significantly, but for safe navigation environments that's not the case.

Critical strengths
  1. The paper focuses on studying the important problem of constraint satisfaction in the tail. It makes sense for papers to propose methods and report results reporting tail safe behavior.
Critical weaknesses
  1. The evaluation protocol is pretty well highlighted in literature. CVar is standard object in risk-constrained RL [1, 2, 3].

  2. The paper analyzes methods that optimize mean episodic cost (standard CMDP objective) on a metric that they don't optimize and the paper doesn't propose a method or algorithm to address the "tail safety".

  3. The proposed explanation of controllability is largely definitional and confounded. Because VaR ≤ CVaR, CVaR(0.1) ≤ d already implies that at least 90% of episodes are within budget. So "safe episodes must be frequent" follows from the definition of tail safety, and "high-return" is the paper's own definition of controllability at held return (for which no tolerance is specified). The indicator is computed from the trained policy and cannot predict controllability before training.

[1] Y. Chow, M. Ghavamzadeh, L. Janson, M. Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167):1–51, 2018.

[2] Q. Yang, T. D. Simão, S. H. Tindemans, M. T. J. Spaan. WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning. AAAI Conference on Artificial Intelligence, 35(12):10639–10646, 2021.

[3] C. Ying, X. Zhou, H. Su, D. Yan, N. Chen, J. Zhu. Towards safe reinforcement learning via constraining conditional value-at-risk. International Joint Conference on Artificial Intelligence (IJCAI), pp. 3673–3680, 2022.

Questions for authors
  1. Since the expected-cost baselines are not designed to control the tail why the paper analyzes those methods?

  2. What does the "frequent safe, high-return episodes" indicator add beyond the definition of CVaR? Is there a task-level property that is measurable before constrained training and predicts controllability on held-out tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.