- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a Bellman operator that unifies task value and safety value. In the tabular setting, the formulation is proven to converge to the optimal safety-constrained task value, given a sufficiently large margin scale. The framework is novel to my knowledge, yet the validation does not specifically back the optimality claim in continuous control environments (or at least its robustness under practical assumption violations).
Critical strengths
-
The core idea is novel and conceptually clean, with substantial theoretical analysis.
-
Substantial experimental results are included, including optimality-related tests in the tabular setting and practical tests with function approximations in MuJoCo environments.
Critical weaknesses
-
The theory and the practical implementation have a notable mismatch. While the theoretical motivation is optimality, for various continuous tasks where the optimum lies near the constraint set boundary, optimality can require an intractably large . The paper thus does not discuss optimality in the practical continuous experiments, instead focusing on stable convergence, which is a distinct property compared to optimality and is not really justified by the theoretical framework. This appears to be unaddressable. However, I do not regard this as a major problem, since strict guarantees generally do not carry over with function approximation without additional system knowledge. I suggest improving the presentation of this mismatch by more explicitly stating it early in the paper.
-
The practical is hand-tuned per environment, choosing ‘the maximum value of such that there are no safety violations / algorithmic instabilities’. The exact selection mechanism is not clear. Even with this tuning, convergence is not very stable for some environments (e.g., SafetyWalker2dVel-v1).
-
No source code is provided with the submission. This does limit reproducibility at the time of review and affects my confidence score.
Questions for authors
-
Figure 4, where the instability of large should be analyzed, does not show the standard deviation from different runs. I believe it is very important to see how much numerical instability large actually introduces, as optimality hinges on large .
-
The authors could use the smallest among states in training that are certified safe to estimate a (lower bound on ). If even this is way larger than the used in the experiments, the optimality claim might not be practically attainable.
2 comments
I thank the authors for the added experiments, which provide further insights into the practical continuous control experiments. I maintain my original recommendation.
Response to Reviewer 4Cj2
We thank the reviewer for their insightful comments and suggestions. The main concerns are (1) explicitly acknowledging the theory-practical mismatch early on in the paper, (2) adding variance bands in Fig. 4, (3) obtaining/estimating K~∗ based on the formula for g(s) provided, and (4) non-availability of source code. Together with the clarifications below, does this address the reviewer's concern?
Thank you for the suggestion. We have clarified in the end of Section 1 that the theoretical analysis holds only for finite and tabular settings; with continuous control experiments, we intend to show the scalability and stable learning of our joint formulation for high-dimensional state-action spaces.
Thank you for the suggestion. We have added the variance bands (±1 std) to Figure 4 in Appendix C.4. We can generally see that, as K increases, the variance band either stays the same or tends to slightly increase. We ran the experiments up to K=2000 as each of the K values possesses convergent training performance; beyond this, we observed that not all environments possessed stable training.
We would like to clarify that the calculation of K∗ only holds for finite tabular settings; for continuous spaces, this doesn't directly carry over. This is because there could exist an arbitrary number of states closer to the constraint boundary, and thus, from these states, Vπϕ≤g can be arbitrarily small and thus, δΩ=0. Thus, we empirically study how the safety value truncation varies as K increases in Appendix C.6. We collect transition data from sampling a batch every 2000 steps and measure the number of active safety truncations in that batch. Over 5M training steps, we collect a total of 2,500 data points, and average across the training seeds. We illustrate the results in Figure 7. It can be generally seen that the overall active safety value truncations for transitions during training reduces as K increases (except SafetyHopperVel-v0, where the K-increase converge close to similar values).
We understand the reviewer's concern around the source code availability; we will release the complete codebase upon acceptance. The currently revised appendix, to the best of our knowledge, supplies most of the required details and information pertaining to the empirical study and implementation. We'd be happy to answer further questions that the reviewer may have during the discussion period.