- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VETO, a post-training safety filtering mechanism for RL policies under distribution shift. VETO uses an ensemble-based upper confidence bound on immediate costs to determine action admissibility, with gradient-based projection and fallback mechanisms for rejected actions. To address outdated safety certificates, the method rechecks actions against the current admissibility condition before executing them. The authors verify the soundness of this filtering logic using Rocq and derive a statistical coverage bound under bounded distribution shift, supported by periodic recalibration of the cost estimates. Experiments on two Safety-Gymnasium environments with three constrained RL algorithms show reductions in per-step constraint violations under several types of distribution shift, although these improvements come at the cost of high refusal rates and reward degradation.
Critical strengths
-
The paper addresses an important problem in safe RL, the reliability of safety filters under distribution shift. The distinction between soundness of the action-filtering mechanism and statistical coverage of the learned cost bound is useful, and the counterexamples showing the problems with outdated safety certificates are well motivated.
-
VETO filtering can be applied to pretrained RL policies without requiring an explicit dynamics model or changes to the policy-training procedure. The experiments consider multiple types of distribution shift, and the reporting of reward degradation, refusals, and negative results helps understand the practical limitations of the method.
Critical weaknesses
-
Practicality of the proposed algorithm is not sufficiently supported. VETO as described is a very costly algorithm; not only is the algorithm itself computationally expensive (e.g. Appendix E shows 19.47 ms per decision on PointGoal which is significantly larger than the 100 Hz control frequency common in robotics), high refusal rates also mean a lot of these computations are essentially wasted. Furthermore the empirical results seem unconvincing. Although VETO is implemented on top of an existing controller the base algorithms are very old, it should also be tested with more relevant state-of-the-art algorithms, especially on the filtering side. And quantitative results seem statistically insignificant at times, e.g. 2.14% vs 1.77% violation rate on PID-Lag CarGoal could potentially be attributed to seed noise.
-
Applicability of the theoretical coverage guarantee seems also limited. Theorem 6 relies on the assumption that the observations for calibration are independent. However in general, RL will have temporally dependent observations. For example in the Goal task once the robot goes inside the hazard it will receive a cost for multiple timesteps until it gets outside. Therefore, it is unclear whether the statistical coverage guarantee applies to the actual evaluation setting. Although the authors acknowledge this assumption, they do not provide an analysis accounting for this temporal dependence.
Questions for authors
-
Isn’t inner loop projection with gradient descent through an ensemble rather costly? Have you considered replacing gradient-based projection with a sampling-based action selection approach? Sampled fallback mentioned in Appendix E seems relevant, but empirical comparisons would be useful. Furthermore, under distribution shift, could projection exploit errors in the learned cost model and produce actions that appear admissible but are actually unsafe? Does your method handle that somewhere?
-
The soundness property appears to follow directly from applying an admissibility check before every action. However, I don’t understand why the VETO algorithm warrants the use of a formal verification tool beyond a conventional proof; since it doesn’t cover the actual experimental implementation, it seems rather unnecessary.
-
VETO only considers immediate costs, which may be insufficient when avoiding a violation requires taking actions several steps in advance. For example, in Safety-Gymnasium navigation tasks a robot moving toward a hazard at high speed might need to start braking or turning well before any immediate cost is incurred. Although the paper acknowledges the lack of recursive feasibility, could you elaborate on how severely this limitation affects the intended applications? More specifically, I believe empirically evaluating scenarios requiring such anticipatory safety decisions would be useful here to determine the impact of this drawback.