acceptodds
Under review as a conference paper at ICLR 2027

VETO: A Sound Safety Envelope for Reinforcement Learning under Distribution Shift

Abstract

Safety envelopes filter the actions of a reinforcement learning policy through a predicted cost bound, and their appeal is that the bound is checked at every step. However, we find that the natural envelope fails in two ways under distribution shift. First, its fallback path reuses actions certified under an older calibration or at a different state, so the envelope is unsound and can emit an action its own bound rejects. Second, its calibration is fitted once, so the coverage the bound promises drifts with the deployment distribution. We introduce VETO, which combines a re-check of every emitted action at the current monitor state, a single temperature refit online from a scalar drift statistic, and a margin on the ensemble spread. Soundness then holds for any cost model and is machine-checked in the Rocq prover, and coverage transfer reduces to a Kolmogorov–Smirnov bound on one scalar, because coverage is a threshold event, which is why one temperature replaces per-step quantile tracking. In a pilot on two Safety-Gymnasium tasks with three constrained-optimization base policies, VETO lowers the per-step violation rate in 58 of 60 evaluation cells. The naive fallback emits inadmissible actions on up to 75.2% of steps, and a conformal filter with the same ensemble and no re-check has a violation rate at or above VETO’s in 54 of 60 cells. The price is an intervention rate that we report cell by cell. A sound envelope with online calibration is therefore what lets a per-step guarantee survive shift.

Primary area:
reinforcement learning
Keywords:
safe reinforcement learning, constrained MDPs, runtime enforcement, shielding, distribution shift, conformal calibration, formal verification, mechanized proof

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4JRTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VETO, a post-training safety filtering mechanism for RL policies under distribution shift. VETO uses an ensemble-based upper confidence bound on immediate costs to determine action admissibility, with gradient-based projection and fallback mechanisms for rejected actions. To address outdated safety certificates, the method rechecks actions against the current admissibility condition before executing them. The authors verify the soundness of this filtering logic using Rocq and derive a statistical coverage bound under bounded distribution shift, supported by periodic recalibration of the cost estimates. Experiments on two Safety-Gymnasium environments with three constrained RL algorithms show reductions in per-step constraint violations under several types of distribution shift, although these improvements come at the cost of high refusal rates and reward degradation.

Critical strengths
  1. The paper addresses an important problem in safe RL, the reliability of safety filters under distribution shift. The distinction between soundness of the action-filtering mechanism and statistical coverage of the learned cost bound is useful, and the counterexamples showing the problems with outdated safety certificates are well motivated.

  2. VETO filtering can be applied to pretrained RL policies without requiring an explicit dynamics model or changes to the policy-training procedure. The experiments consider multiple types of distribution shift, and the reporting of reward degradation, refusals, and negative results helps understand the practical limitations of the method.

Critical weaknesses
  1. Practicality of the proposed algorithm is not sufficiently supported. VETO as described is a very costly algorithm; not only is the algorithm itself computationally expensive (e.g. Appendix E shows 19.47 ms per decision on PointGoal which is significantly larger than the 100 Hz control frequency common in robotics), high refusal rates also mean a lot of these computations are essentially wasted. Furthermore the empirical results seem unconvincing. Although VETO is implemented on top of an existing controller the base algorithms are very old, it should also be tested with more relevant state-of-the-art algorithms, especially on the filtering side. And quantitative results seem statistically insignificant at times, e.g. 2.14% vs 1.77% violation rate on PID-Lag CarGoal could potentially be attributed to seed noise.

  2. Applicability of the theoretical coverage guarantee seems also limited. Theorem 6 relies on the assumption that the observations for calibration are independent. However in general, RL will have temporally dependent observations. For example in the Goal task once the robot goes inside the hazard it will receive a cost for multiple timesteps until it gets outside. Therefore, it is unclear whether the statistical coverage guarantee applies to the actual evaluation setting. Although the authors acknowledge this assumption, they do not provide an analysis accounting for this temporal dependence.

Questions for authors
  1. Isn’t inner loop projection with gradient descent through an ensemble rather costly? Have you considered replacing gradient-based projection with a sampling-based action selection approach? Sampled fallback mentioned in Appendix E seems relevant, but empirical comparisons would be useful. Furthermore, under distribution shift, could projection exploit errors in the learned cost model and produce actions that appear admissible but are actually unsafe? Does your method handle that somewhere?

  2. The soundness property appears to follow directly from applying an admissibility check before every action. However, I don’t understand why the VETO algorithm warrants the use of a formal verification tool beyond a conventional proof; since it doesn’t cover the actual experimental implementation, it seems rather unnecessary.

  3. VETO only considers immediate costs, which may be insufficient when avoiding a violation requires taking actions several steps in advance. For example, in Safety-Gymnasium navigation tasks a robot moving toward a hazard at high speed might need to start braking or turning well before any immediate cost is incurred. Although the paper acknowledges the lack of recursive feasibility, could you elaborate on how severely this limitation affects the intended applications? More specifically, I believe empirically evaluating scenarios requiring such anticipatory safety decisions would be useful here to determine the impact of this drawback.

Reviewer_tELVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies VETO, a safety filter between a trained reinforcement learning (RL) policy and the environment. At each step, an ensemble of small networks predicts the cost of the proposed action. If the predicted upper bound is too high, VETO tries a nearby action, then a fallback list (the last accepted action, the zero action, recent accepted actions), and refuses if nothing passes. There are two changes over a naive filter: (1) every action is checked again right before it is executed, and (2) one scaling factor (a temperature) is refitting every 20 episodes from recent true costs, with some rules depending on its prediction errors. The paper proves that every action VETO outputs passes its own check and gives a coverage bound under drift. Their experiments show that the proposed algorithm lowers the violation rate in 58 of 60 settings.

Critical strengths
  1. The framing is clean because it keeps two guarantees: the filter never outputs an action that fails its own check, which is machine-checked in Rocq, and a separate statistical guarantee that the check covers the true cost under drift.

  2. I like the writing, in the age of AI, this paper seems to be written by human mostly, or at least they proofread.. so writing is clean, and understandable, even though some figures do not look good for presentation.

  3. Clean experimental results and clean conclusion of showing violation drop on most environments.

Critical weaknesses
  1. Novelty is quite limited (because of this, it's really hard to change my mind, even though I appreciate the author's work). The main new idea, re-checking an action right before it runs to avoid stale approvals, is a one-line fix (Sec. 1). The rest combines standard tools: ensemble cost bounds, safety fallback, a temperature refit, and a Kolmogorov–Smirnov (KS) distance check. The paper also never shows that existing filters actually suffer from stale approvals, and it does not compare against existing safety filters.
  2. In experiments, VETO does run actions that failed its own check. When no action passes, it refuses and hands control to the zero action. But the zero action (they kinda consider this is fallback action, but it's not safe fallback in many cases..) was already in the candidate set, so a refusal only happens when the zero action failed too.
  3. The safety guarantee is weaker than using no filter. By the paper's own numbers (Sec. 6), the coverage bound allows about 9% violating steps, whereas every base policy without the filter already violates less (below 6.0%, Table 1).
  4. The main ablation does not isolate the re-check. The conformal filter does not "lack only the re-check". It also uses an adaptive conformal quantile instead of VETO's temperature refit and margin (App. E). VETO's lower violations may come from this margin (which causes near-total refusal on PointGoal), not the re-check. There is no comparison at matched refusal rate or reward.
  5. some figures are hard to read, for example, figure 1 and 2. Both are too small to read, and figure 2 has overlapping characters.

Because of these several weaknesses, it's really hard to change my mind on my decision, even though I agree this engineering works has some contributions. Please keep up the good work, but i personally think the work is not good enough for ICLR venue.

Questions for authors
  1. On refusal steps, the zero action is executed even though it failed the check. If refusal steps are counted, how many executed actions fail the check?
  2. Can you compare VETO and the conformal filter with the same calibration and margin, differing only in the re-check, or at a matched refusal rate or reward? A comparison with an existing safety filter (e.g., [1]) would also help.
  3. Is there any setting where the Theorem 6 bound is below the violation rate of the base policy alone?
  4. Can you show standard errors or confidence intervals for Table 1 and the shift results, as Sec. 7 says?

[1] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757, 2018.

Reviewer_EQZHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

VETO is a post-training safety filter for RL policies under distribution shift. It rechecks proposed actions against a learned cost bound, adjusts the bound through online recalibration, and refuses actions when no candidate passes. The paper proves that emitted actions satisfy the filter's admissibility criterion and derives a conditional statistical coverage bound. Experiments on two Safety-Gymnasium tasks report fewer per-step violations, often with substantial refusal rates and reward losses.

Critical strengths
  • The paper distinguishes compliance with the learned admissibility criterion from coverage of the true cost, avoiding an unconditional claim of environmental safety.

  • The authors provide counterexamples showing that cached actions can become inadmissible after calibration changes or when reused at different states. The formal analysis establishes that rechecking prevents this specific failure.

Critical weaknesses
  • On PointGoal, VETO refuses 97–99% of actions for two base policies, reducing rewards almost to zero. Under severe rare-event shifts, it refuses every action yet still incurs safety violations. These results raise serious concerns about whether the method provides useful safe control rather than simply preventing the agent from acting.

  • The experiments primarily use base policies trained under expected cumulative-cost constraints, without comparisons to methods explicitly targeting state-wise safety, such as ASCPO (https://arxiv.org/abs/2410.01212) or the adaptive shielding approach identified as closest related work. A filter with the same final check but without online recalibration is also missing, leaving the contribution of recalibration unclear.

  • When no action passes the admissibility check, VETO refuses and the experiments execute a zero action, which is not necessarily safe. The observed violations despite complete refusal under severe shifts demonstrate this limitation. The paper does not establish how the method can maintain safety or recover when admissible actions cannot be found.

  • Several figures appear to have rendering or formatting errors, with overlapping or misplaced text, mathematical symbols, and numerical labels. These issues make parts of the figures unreadable and suggest insufficient quality checks of the final manuscript.

  • The related-work discussion could also briefly mention works like EVO (https://arxiv.org/abs/2601.12008) and the more recent SteinGate (https://arxiv.org/abs/2607.13175), which address rare or severe safety failures that may be obscured by expected cumulative-cost constraints. Although their formulations differ from VETO's per-step filtering approach, they provide useful context for positioning the work.

Questions for authors

Please check weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.