- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies the limitations of the expected-cost constraints formulation in safe RL. The analysis and experiments use CVaR as the main metric to measure the risk of high-cost episodes and assess whether distributional safe RL approaches can control this tail risk. The OQ-SAC in this paper is an ORAC-inspired SAC variant (ORAC is a pre-printed distributional safe RL method). Experiments on safety-gymnasium tasks show that tail-risk in episodes is controllable for some locomotion tasks, but remains difficult in safe navigation tasks. The following analysis shows that tail-risk controllability can be influenced by the occurrence of high-return episodes.
Critical strengths
-
The chosen CVaR metric and the evaluation protocol are reasonable for assessing the tail risk in episodes for safe RL algorithms.
-
This paper presents an interesting explanation of why tail-risk control for navigation tasks is more difficult.
Critical weaknesses
-
This paper does not have much added value or novelty. It does not show any new observed phenomena, nor does it identify any new questions or provide any new safe RL algorithms. Episode-level tail-risk control is a well-known issue in safe RL. This paper seems to conduct experiments to analyze this phenomenon again, and the conclusions are mostly already reported in quite some existing papers.
-
The benchmark and experimental design are problematic. Distributional safe RL is a widely used method to mitigate tail risk. However, the authors only benchmark and evaluate conventional, point-value safe RL methods in the analysis, and treat OQ-SAC, a distributional safe RL variant, as a solution. This looks like the paper propose an "existing" question and using an "existing" solution to fix it.
Questions for authors
Q1. What is actually novel in this paper? As the second weakness says, the tail-risk issue and the distributional safe RL solutions have already been proposed and studied in the literature (including ORAC cited by this paper). This paper seems analysis-only.
Q2. In Table 1, only point-value safe RL methods are used to show the tail-risk issue. Why not include distributional safe RL methods to examine if they also have the same issue?
Q3. In Table 2, as OQ-SAC is a variant of ORAC, why is it not compared with ORAC, but with two point-value safe RL methods again? According to the ORAC paper, it can already effectively mitigate tail risk.
Q4. There is a severe result issue in Section 8: the authors write, "The worst-decile cost reaches 10.2 times the mean...". How could this be possible? Mathematically, CVaR 0.1 can be maximum 10 times the mean, right?
Q5. For the result of Figure 3, the authors say that the separate clusters of episode cost probably make tail-risk control harder. However, this result cannot provide any causal explanation. Theoretically, quantile-based critics can handle this case effectively. I think the authors should avoid this over-claiming or add controlled experiments or ablations to clarify the real causes of the failure of navigation tasks.