- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a shield for fine-tuning a pretrained policy (PPO-Lag) under a shifting environment. When the policy lands somewhere new, its sense of risk was learned in the old world and goes stale, so it starts breaking the cost budget. The key idea here is that they built on an existing online conformal abstention (OCA) idea to keep the average damage of executed (after filtering) actions under the picked safety level, and run it as the policy's action gate, so the level holds even when the world changes, without knowing dynamics. The gate: each step the policy proposes an action, then a learned cost score compares it against a bar (the threshold): (1) under the bar, run it (2) over the bar, hand it to a supervisor (INT) or wait (HAL), fallback actions.
The bar starts from a rough guess and then moves on its own, using only the costs of actions that actually ran, so it re-fits itself when the world shifts. Blocking a robot action is the same as the abstention algorithm saying "skip". Under assumptions (the world can shift however it wants, but only based on the past, and the gate learns only from actions it let run), they prove the executed-action cost (SSR) stays under the safety level with high probability, up to a small margin. But most of the theory comes from an earlier Paper. The new parts is a smarter starting guess for the bar, so the gate does not block everything at the start. They show that on several shifted robot tasks their algorithm achieves the safety level asymptotically, while the baselines fail.
Critical strengths
-
This work uses an existing abstention algorithm (answer or skip) as the policy's action gate, and blocking actions is the just the abstention algorithm saying 'skip' so the existing safety guarantee from the abstention algorithm holds. Good thing about this is that the gate needs no dynamics model, no calibrated cost score, and no back up controller. Because the bar refits itself online from the costs it observes, the safety guarantee holds even with the shift by design.
-
The experiments are controlled. All methods train the same way and only the shield differs. And comprehensive ablation results for different safety level, and experiments on diverse domain including vision based environments
Critical weaknesses
-
Most of the core mechanism borrows technique from existing method OCA including theoretical guarantee, so regarding novelty sense, not sure even though engineering sense, I agree it has contribution. For example, Assumption 1 and proofs essentially follow the previous OCA paper. Also, in the theorem they do not mention what value is, if = 1, the equation becomes very trivial, they need to define the range clearly.
-
"Safe" literally excludes the steps where the shield acts. Then use fallback actions such as INT/HAL from a supervisor or assuming some existing safe fallback action. But the guarantee covers only executed task actions under those assumptions. So intuitively theoretical result is kinda trivial: if you know the action is safe, then execute so it will be safe, otherwise, hand it over to supervisor or take the safe existing fallback action, even though the paper say it's their limitation, still this is weakness.
-
Each environment's different behavior is described, but not explained why. The paper gives one generic story "the bar starts conservative and relaxes as it sees costs and for the steps that end above it says "still decreasing, consistent with the vanishing slack." No env diagnosis: which property of each shift sets how fast the bar can relax, why recovery rates differ, and what slope "vanishing slack" should even look like is never checked.
-
One base learner. Theory allows any history-dependent policy and scorer updates; every experiment trains PPO-Lag. No other safe-RL algorithms (CPO, SAC-Lag, ...) is tested.
Questions for authors
-
The theory allows any base learner, you only ran PPO-Lag. What happens with a different safe-RL optimizer (SAC-Lag, CPO, RCPO, PCPO.. extra)? Does the guarantee hold the same, and does efficiency still hold?
-
Environments respond differently: SSR falls fast in some, efficiency dips and comes back at different rates. Which property of each shift sets that response? Is there a mechanism or theory that predicts, per environment, where SSR vs efficiency/success should land before running it?
-
You show only the average of 3 seeds. Please show the number from each seed (or an error range) for the closest gaps - 0.024 vs 0.031 (MultiCarrot), 0.026 vs 0.030 (VisionWhole). At gap this small, one seed can flip "OCS under the level, baselines above," so I need the spread, not the mean.
-
Your guarantee is not "cost under safety level" - it is "under safety plus extra margin" (B_OCA), and the margin is never computed at the budgets you test. Could you compute this, for example, at 8k steps and at 10^4 steps - and report which delta you use?
6 comments
W3 & Q2. Environment-Dependent Behavior Is Described but Not Explained
Different distribution shifts can alter state–action visitation, predicted risk scores, and their relationship with realized safety costs, leading to different threshold adaptation and intervention patterns. Theorem 1 makes part of this dependence explicit. In (4), the slack is , so it vanishes faster when more proposed actions are executed. On ManiSkill, OCS executes most actions, so the bound tightens quickly. On SafetyPointGoal2, the unshielded policy incurs about five times the target risk, so OCS intervenes more often, and the lower efficiency makes the bound about 2.5 times larger than on Goal1 at the same budget. Since efficiency is itself an outcome of adaptation, (4) explains this pattern but does not predict final efficiency or success before running.
These differences also depend on the benchmark. ManiSkill uses a 7B VLA policy under geometric and visual shifts, where geometric shifts make task completion more difficult even without shielding (Table 6). Safety-Gymnasium uses a state-based control policy, and SafetyPointGoal2 introduces additional hazards, requiring more interventions to approach the target risk. The vanishing theoretical slack bounds the risk violation but does not prescribe the convergence rate of empirical SSR.
W4 & Q1. Only One Base Learner Is Evaluated
The theoretical safety guarantee of OCS is independent of the specific base learner, provided its updates satisfy the non-anticipating interaction setting of Assumption 1 (Theorem 1). Thus, replacing PPO-Lag with another optimizer does not change the guarantee, although execution efficiency and task performance may differ depending on the learned policy.
To empirically examine this generality, we additionally evaluate OCS with CPO as the base learner on Safety-Gymnasium, using the same settings as PPO-Lag and the prior centered at 0.9 (Figure 12, Appendix E.4; three seeds). OCS with CPO keeps the final SSR below the target in both environments, while CPO alone exceeds it. As expected, the more hazardous SafetyPointGoal2 yields lower efficiency and success.
Question 3,4
Q3. The Spread Over Seeds Is Not Shown for the Closest Gaps
We expanded our experiments from three to five seeds across all five ManiSkill environments, and the revised paper reports SSR as mean ± standard deviation for every environment (Table 7). OCS achieves lower mean SSR than Fixed threshold-Calibration in both highlighted environments. With paired confidence intervals over seeds, the gap is not distinguishable from seed variation on MultiCarrot, the in-distribution environment where a threshold calibrated on the pretraining distribution can reasonably perform well ([−0.014, 0.021]), but excludes zero on VisionWhole ([0.001, 0.014]) (Table 8, Appendix E.2).
The guarantee does not require every seed to fall below : Theorem 1 bounds SSR by plus a finite-sample slack (See Q4), so a run slightly above is consistent with it. The relevant comparison is how close SSR stays to across seeds, not which side of each seed lands on.
Q4. The Finite-Sample Margin Is Not Computed at the Tested Budgets
Theorem 1 guarantees with probability at least , rather than at finite budgets. Using Eq. (6) with , , , and , we obtain the slack in Table 2 (Appendix B.1). With full task-action execution (), it is 0.146 at 8,000 and 0.131 at 10,000 active steps (0.143 at 8,000 for ), and 0.019 at the Safety-Gymnasium budget. At the observed efficiency, it is 0.151–0.153 at 8,000 and 0.131–0.134 at 10,000 active steps, while the observed SSR is 0.020–0.026.
The slack decreases as and is already of the same order as at the Safety-Gymnasium budget (Figure 5). With full execution, it falls below at about active steps. At the ManiSkill budgets, the slack exceeds , so the near-target SSR observed there is an empirical result rather than a finite-budget certification at ; in every run, the observed SSR lies well below plus the slack. Tightening the constant of the bound is future work.
Weakness 1,2
We thank the reviewer for the constructive feedback and for recognizing the soundness of our framework, the applicability of OCA-based shielding without requiring a dynamics model or calibrated risk scorer, and the robustness of its safety guarantee under distribution shifts. We also appreciate the positive assessment of our controlled experimental setup, comprehensive ablations across safety levels, and evaluations in diverse domains, including vision-based manipulation tasks. We address the concerns below.
W1. The Contribution Beyond OCA Is Unclear, and the Range of α Is Not Defined
(1) The core mechanism and the theoretical guarantee are borrowed from OCA. We state the OCA guarantee [1] as Assumption 1 and do not claim it as our contribution. Our contribution is twofold.
First, we show that the guarantee carries over to shielding during online RL adaptation, where it is not directly applicable. The OCA evaluation allows the scoring function to change over time but updates no policy; in OCS, the shield's own decisions determine the training data of the task policy and the online-trained scorer, and therefore the future scores, in addition to environment shift. Intervened costs are never observed, and episodes contain halting, early termination, and variable horizons. Theorem 1 handles these by mapping active steps to OCA rounds, excluding waiting steps from the updates, and tying the active-step budget to the OCA horizon; Appendix C.2 verifies step by step that the resulting interaction is non-anticipating.
Second, we introduce a prior-aware variant of ExAUL and extend its analysis to non-uniform initialization (Theorem 2), because early threshold selection matters in safety-critical adaptation. The prior replaces the term lnK in the bound with ln(1/w1(ξ⋆)), which is smaller when the prior puts more than 1/K mass on the best threshold ξ⋆, in line with prior-dependent regret bounds in the bandit literature [2, 3, 4]. The prior-versus-uniform comparison is on SafetyPointGoal1 (Appendix E.3, Figure 11).
(2) The range of α is not defined, and α=1 makes the theorem trivial. Assumption 1 and Theorems 1 and 2 are stated for α,δ∈(0,1), so α=1 is excluded; as the reviewer notes, it would make Eq. (4) trivial, since costs lie in [0,1]. More generally, Eq. (4) is informative only when α+BOCA(Nact,δ)/Ntask<1 (See Q4), and the revised paper states this condition right after Theorem 1.
[1] M. Lee et al. Online Conformal Abstention Under Adversarial Bandit Feedback. In NeurIPS, 2026.
[2] N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
[3] D. Russo and B. Van Roy. An Information-Theoretic Analysis of Thompson Sampling. JMLR, 2016.
[4] B. Kveton et al. Meta-Thompson Sampling. In ICML, 2021.
W2. The Guarantee Covers Only Executed Task Actions, Which Makes It Seem Trivial
The main challenge addressed by OCS is controlling execution-time safety risk under unknown dynamics and distribution shifts, where the safety of proposed actions cannot be determined in advance due to uncertainty.
Existing shielding methods often rely on strong assumptions, such as accurate dynamics models, formal safety specifications, or certified safe sets. Under these assumptions, it is trivial to choose a safe action, as the review pointed out, but they are difficult to satisfy during online adaptation, where environment shifts and continuous policy/scorer updates can invalidate prior safety assessments. OCS adaptively updates its pre-execution shielding threshold using only safety feedback from executed actions, without observing the counterfactual costs of intervened actions. Under non-anticipating interaction, Theorem 1 guarantees selective risk control up to finite-sample slack, even under coupled distribution shifts.
We acknowledge that the safety of fallback interventions is assumed rather than guaranteed. Nevertheless, our contribution is adaptive execution-time risk control during online learning under partial feedback and unknown dynamics, rather than simply executing actions already known to be safe.