acceptodds
Under review as a conference paper at ICLR 2027

Online Conformal Shielding: Safe Adaptation under Coupled Distribution Shifts

Abstract

Safe online adaptation requires balancing useful exploration with execution-time risk control. We identify two challenges: a safety–exploration imbalance between overly conservative and overly permissive control, and safety staleness, where risk control becomes outdated under coupled exogenous environment shifts and endogenous policy/scorer updates. We propose Online Conformal Shielding (OCS), a pre-execution safety framework for online adaptation. OCS selectively executes proposed actions or intervenes, updating its threshold from partial safety feedback while the task policy and scorer continue to adapt. Our generic reduction to online conformal abstention (OCA) transfers suitable selective-risk guarantees to shielding without explicitly modeling the source of distribution shift. We further develop a prior-aware OCA variant that uses pre-adaptation information to guide early threshold selection, with a well-placed prior improving the resulting safety guarantee. Under non-anticipating interaction, we establish a high-probability adaptive safety guarantee: at a prescribed interaction budget, the realized average safety cost among executed task actions is bounded by the target level plus an explicit finite-sample slack. On VLA manipulation in ManiSkill and continuous control in Safety-Gymnasium, OCS keeps the selective safety risk near the target level under coupled shifts, where policy-level safe RL and fixed shields exceed it, while executing over 96% of the steps on ManiSkill.

Primary area:
applications to robotics, autonomy, planning
Keywords:
safe online adaptation, distribution shift, safety shielding

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ohf7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a shield for fine-tuning a pretrained policy (PPO-Lag) under a shifting environment. When the policy lands somewhere new, its sense of risk was learned in the old world and goes stale, so it starts breaking the cost budget. The key idea here is that they built on an existing online conformal abstention (OCA) idea to keep the average damage of executed (after filtering) actions under the picked safety level, and run it as the policy's action gate, so the level holds even when the world changes, without knowing dynamics. The gate: each step the policy proposes an action, then a learned cost score compares it against a bar (the threshold): (1) under the bar, run it (2) over the bar, hand it to a supervisor (INT) or wait (HAL), fallback actions.

The bar starts from a rough guess and then moves on its own, using only the costs of actions that actually ran, so it re-fits itself when the world shifts. Blocking a robot action is the same as the abstention algorithm saying "skip". Under assumptions (the world can shift however it wants, but only based on the past, and the gate learns only from actions it let run), they prove the executed-action cost (SSR) stays under the safety level with high probability, up to a small margin. But most of the theory comes from an earlier Paper. The new parts is a smarter starting guess for the bar, so the gate does not block everything at the start. They show that on several shifted robot tasks their algorithm achieves the safety level asymptotically, while the baselines fail.

Critical strengths
  1. This work uses an existing abstention algorithm (answer or skip) as the policy's action gate, and blocking actions is the just the abstention algorithm saying 'skip' so the existing safety guarantee from the abstention algorithm holds. Good thing about this is that the gate needs no dynamics model, no calibrated cost score, and no back up controller. Because the bar refits itself online from the costs it observes, the safety guarantee holds even with the shift by design.

  2. The experiments are controlled. All methods train the same way and only the shield differs. And comprehensive ablation results for different safety level, and experiments on diverse domain including vision based environments

Critical weaknesses
  1. Most of the core mechanism borrows technique from existing method OCA including theoretical guarantee, so regarding novelty sense, not sure even though engineering sense, I agree it has contribution. For example, Assumption 1 and proofs essentially follow the previous OCA paper. Also, in the theorem they do not mention what α\alpha value is, if α\alpha = 1, the equation becomes very trivial, they need to define the range clearly.

  2. "Safe" literally excludes the steps where the shield acts. Then use fallback actions such as INT/HAL from a supervisor or assuming some existing safe fallback action. But the guarantee covers only executed task actions under those assumptions. So intuitively theoretical result is kinda trivial: if you know the action is safe, then execute so it will be safe, otherwise, hand it over to supervisor or take the safe existing fallback action, even though the paper say it's their limitation, still this is weakness.

  3. Each environment's different behavior is described, but not explained why. The paper gives one generic story "the bar starts conservative and relaxes as it sees costs and for the steps that end above α\alpha it says "still decreasing, consistent with the vanishing slack." No env diagnosis: which property of each shift sets how fast the bar can relax, why recovery rates differ, and what slope "vanishing slack" should even look like is never checked.

  4. One base learner. Theory allows any history-dependent policy and scorer updates; every experiment trains PPO-Lag. No other safe-RL algorithms (CPO, SAC-Lag, ...) is tested.

Questions for authors
  1. The theory allows any base learner, you only ran PPO-Lag. What happens with a different safe-RL optimizer (SAC-Lag, CPO, RCPO, PCPO.. extra)? Does the guarantee hold the same, and does efficiency still hold?

  2. Environments respond differently: SSR falls fast in some, efficiency dips and comes back at different rates. Which property of each shift sets that response? Is there a mechanism or theory that predicts, per environment, where SSR vs efficiency/success should land before running it?

  3. You show only the average of 3 seeds. Please show the number from each seed (or an error range) for the closest gaps - 0.024 vs 0.031 (MultiCarrot), 0.026 vs 0.030 (VisionWhole). At gap this small, one seed can flip "OCS under the level, baselines above," so I need the spread, not the mean.

  4. Your guarantee is not "cost under safety level" - it is "under safety plus extra margin" (B_OCA), and the margin is never computed at the budgets you test. Could you compute this, for example, at 8k steps and at 10^4 steps - and report which delta you use?

6 comments
AuthorsOpenReview ↗

Weakness 1,2

We thank the reviewer for the constructive feedback and for recognizing the soundness of our framework, the applicability of OCA-based shielding without requiring a dynamics model or calibrated risk scorer, and the robustness of its safety guarantee under distribution shifts. We also appreciate the positive assessment of our controlled experimental setup, comprehensive ablations across safety levels, and evaluations in diverse domains, including vision-based manipulation tasks. We address the concerns below.

W1. The Contribution Beyond OCA Is Unclear, and the Range of α\alpha Is Not Defined

(1) The core mechanism and the theoretical guarantee are borrowed from OCA. We state the OCA guarantee [1] as Assumption 1 and do not claim it as our contribution. Our contribution is twofold.

First, we show that the guarantee carries over to shielding during online RL adaptation, where it is not directly applicable. The OCA evaluation allows the scoring function to change over time but updates no policy; in OCS, the shield's own decisions determine the training data of the task policy and the online-trained scorer, and therefore the future scores, in addition to environment shift. Intervened costs are never observed, and episodes contain halting, early termination, and variable horizons. Theorem 1 handles these by mapping active steps to OCA rounds, excluding waiting steps from the updates, and tying the active-step budget to the OCA horizon; Appendix C.2 verifies step by step that the resulting interaction is non-anticipating.

Second, we introduce a prior-aware variant of ExAUL and extend its analysis to non-uniform initialization (Theorem 2), because early threshold selection matters in safety-critical adaptation. The prior replaces the term ln⁡K\ln K in the bound with ln⁡(1/w1(ξ⋆))\ln(1/w_1(\xi^\star)), which is smaller when the prior puts more than 1/K1/K mass on the best threshold ξ⋆\xi^\star, in line with prior-dependent regret bounds in the bandit literature [2, 3, 4]. The prior-versus-uniform comparison is on SafetyPointGoal1 (Appendix E.3, Figure 11).

(2) The range of α\alpha is not defined, and α=1\alpha = 1 makes the theorem trivial. Assumption 1 and Theorems 1 and 2 are stated for α,δ∈(0,1)\alpha, \delta \in (0, 1), so α=1\alpha = 1 is excluded; as the reviewer notes, it would make Eq. (4) trivial, since costs lie in [0,1][0, 1]. More generally, Eq. (4) is informative only when α+BOCA(Nact,δ)/Ntask<1\alpha + B_\text{OCA}(N^\text{act}, \delta)/N^\text{task} < 1 (See Q4), and the revised paper states this condition right after Theorem 1.

[1] M. Lee et al. Online Conformal Abstention Under Adversarial Bandit Feedback. In NeurIPS, 2026.

[2] N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.

[3] D. Russo and B. Van Roy. An Information-Theoretic Analysis of Thompson Sampling. JMLR, 2016.

[4] B. Kveton et al. Meta-Thompson Sampling. In ICML, 2021.

W2. The Guarantee Covers Only Executed Task Actions, Which Makes It Seem Trivial

The main challenge addressed by OCS is controlling execution-time safety risk under unknown dynamics and distribution shifts, where the safety of proposed actions cannot be determined in advance due to uncertainty.

Existing shielding methods often rely on strong assumptions, such as accurate dynamics models, formal safety specifications, or certified safe sets. Under these assumptions, it is trivial to choose a safe action, as the review pointed out, but they are difficult to satisfy during online adaptation, where environment shifts and continuous policy/scorer updates can invalidate prior safety assessments. OCS adaptively updates its pre-execution shielding threshold using only safety feedback from executed actions, without observing the counterfactual costs of intervened actions. Under non-anticipating interaction, Theorem 1 guarantees selective risk control up to finite-sample slack, even under coupled distribution shifts.

We acknowledge that the safety of fallback interventions is assumed rather than guaranteed. Nevertheless, our contribution is adaptive execution-time risk control during online learning under partial feedback and unknown dynamics, rather than simply executing actions already known to be safe.

W3 & Q2. Environment-Dependent Behavior Is Described but Not Explained

Different distribution shifts can alter state–action visitation, predicted risk scores, and their relationship with realized safety costs, leading to different threshold adaptation and intervention patterns. Theorem 1 makes part of this dependence explicit. In (4), the slack is BOCA(Nact,δ)/(Nact Eff)B_{\mathrm{OCA}}(N^{\mathrm{act}},\delta)/(N^{\mathrm{act}}\,\mathrm{Eff}), so it vanishes faster when more proposed actions are executed. On ManiSkill, OCS executes most actions, so the bound tightens quickly. On SafetyPointGoal2, the unshielded policy incurs about five times the target risk, so OCS intervenes more often, and the lower efficiency makes the bound about 2.5 times larger than on Goal1 at the same budget. Since efficiency is itself an outcome of adaptation, (4) explains this pattern but does not predict final efficiency or success before running.

These differences also depend on the benchmark. ManiSkill uses a 7B VLA policy under geometric and visual shifts, where geometric shifts make task completion more difficult even without shielding (Table 6). Safety-Gymnasium uses a state-based control policy, and SafetyPointGoal2 introduces additional hazards, requiring more interventions to approach the target risk. The vanishing theoretical slack bounds the risk violation but does not prescribe the convergence rate of empirical SSR.

W4 & Q1. Only One Base Learner Is Evaluated

The theoretical safety guarantee of OCS is independent of the specific base learner, provided its updates satisfy the non-anticipating interaction setting of Assumption 1 (Theorem 1). Thus, replacing PPO-Lag with another optimizer does not change the guarantee, although execution efficiency and task performance may differ depending on the learned policy.

To empirically examine this generality, we additionally evaluate OCS with CPO as the base learner on Safety-Gymnasium, using the same settings as PPO-Lag and the prior centered at 0.9 (Figure 12, Appendix E.4; three seeds). OCS with CPO keeps the final SSR below the target in both environments, while CPO alone exceeds it. As expected, the more hazardous SafetyPointGoal2 yields lower efficiency and success.

AuthorsOpenReview ↗

Question 3,4

Q3. The Spread Over Seeds Is Not Shown for the Closest Gaps

We expanded our experiments from three to five seeds across all five ManiSkill environments, and the revised paper reports SSR as mean ± standard deviation for every environment (Table 7). OCS achieves lower mean SSR than Fixed threshold-Calibration in both highlighted environments. With paired confidence intervals over seeds, the gap is not distinguishable from seed variation on MultiCarrot, the in-distribution environment where a threshold calibrated on the pretraining distribution can reasonably perform well ([−0.014, 0.021]), but excludes zero on VisionWhole ([0.001, 0.014]) (Table 8, Appendix E.2).

The guarantee does not require every seed to fall below α\alpha: Theorem 1 bounds SSR by α\alpha plus a finite-sample slack (See Q4), so a run slightly above α\alpha is consistent with it. The relevant comparison is how close SSR stays to α\alpha across seeds, not which side of α\alpha each seed lands on.

Q4. The Finite-Sample Margin Is Not Computed at the Tested Budgets

Theorem 1 guarantees SSR≤α+finite-sample slack\text{SSR} \le \alpha + \text{finite-sample slack} with probability at least 1−δ1-\delta, rather than SSR≤α\text{SSR} \le \alpha at finite budgets. Using Eq. (6) with K=1002K = 1002, λ=N\lambda = \sqrt{N}, ν=2γ=ln⁡K/N\nu = 2\gamma = \sqrt{\ln K / N}, and δ=0.05\delta = 0.05, we obtain the slack BOCA/NtaskB_\text{OCA}/N^\text{task} in Table 2 (Appendix B.1). With full task-action execution (Ntask=NactN^\text{task} = N^\text{act}), it is 0.146 at 8,000 and 0.131 at 10,000 active steps (0.143 at 8,000 for δ=0.1\delta = 0.1), and 0.019 at the Safety-Gymnasium budget. At the observed efficiency, it is 0.151–0.153 at 8,000 and 0.131–0.134 at 10,000 active steps, while the observed SSR is 0.020–0.026.

The slack decreases as 1/N1/\sqrt{N} and is already of the same order as α\alpha at the Safety-Gymnasium budget (Figure 5). With full execution, it falls below α=0.025\alpha = 0.025 at about 2.7×1052.7 \times 10^5 active steps. At the ManiSkill budgets, the slack exceeds α\alpha, so the near-target SSR observed there is an empirical result rather than a finite-budget certification at α\alpha; in every run, the observed SSR lies well below α\alpha plus the slack. Tightening the constant of the bound is future work.

Reviewer_XUNjOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a technique to select and reject actions proposed by a policy using a cost critic and a continuously tuning online threshold that either permits the action proposed by a policy or moves into a HALT state. If the risk scorer allows multiple actions that results in no violations, it increases the accepting threshold, allowing more actions. If actions result in violations, the accepting threshold goes down. The method maintains a set of candidate thresholds, where the weights get updated after each action execution for selection of threshold. Experiments with ManiSkill and safetygymnasium show improved selective safety risk.

Critical strengths

The paper's proposal is quite interesting. Instead of relying on a fixed threshold gate to permit/halt actions produced by a policy, it uses an adaptive online threshold that changes as per the policy's executions in real time. The algorithm itself appears sound, it provides a nice reduction from Online conformal shielding to online conformal abstention. The experiments also show that the algorithm maintains SSR better than fixed threshold shielding.

Critical weaknesses
  1. I don't think this paper's proposition is about shielding. I consider shielding as a task where we always execute safe actions and there is a planner that provides safe actions. Instead, this paper proposes an algorithm about when to intervene. After intervening, the policy basically quits, and no alternative controller takes over. I assume the idea is that someone else will develop a safe controller, but then it confounds the original proposal. If a shield takes over and never switches back to the original policy, it might be possible that the shield just halts or stops the execution after bringing it back to the safe state, but does not have knowledge about maximising reward/return. Usual simplex/MPS approaches either switch between policy and safe planner, or ask the safe planner to convert the policy action to a safer action. The proposed method provides no way of switching back to the policy.

  2. I did not understand the logic of using a Q function as a safety risk scorer and clipping its values from 0 to 1. A softmax would have made more sense here.

  3. The metrics chosen here support maniskill experiments but weaken the paper's position on the safetypointgoal experiments. A more complex environment like Goal2 show that it reduces SSR, but at the cost of efficiency and task success. This means that the algorithm selected a very low threshold and maintained it, thus getting to the desired SST, but not allowing any action to execute. I don't think simply achieving a target SSR should be the goal. It should be somewhere along a pareto front, meauring reward and violations.

  4. I would have expected to see some experiments with algorithms like DMPS, VELM (https://herowanzhu.github.io/CAV_2024.pdf), SPICE (https://arxiv.org/abs/2209.14148), SPARKD (https://dl.acm.org/doi/10.1007/978-3-032-28079-4_15) or RAMPS (https://openreview.net/forum?id=2bbqHOWFTU). It would be useful to see the SSR, efficiency and task success of existing shielding algorithms compared to OCS. Given that most of these algorithms achieve very high reward with near 0 violations after they learn to navigate the environment, I assume they would score much higher than OCS.

Minor: 3 seeds is very low to establish significant dominance. I would expect at minimum 5 seeds.


Minor presentation fixes - L375 has a full stop (.) in environments, L376 has an ellipsis.

Questions for authors

Why were 1002 values chosen? What happens when you choose 102 values, or 12?

Why was clipping used over softmax?

7 comments
AuthorsOpenReview ↗

Weakness1,2, Question2

We thank the reviewer for the careful review. We appreciate the recognition that replacing a fixed threshold gate with an online threshold that adapts to the policy's executions is an interesting proposal, that the algorithm is sound and provides a clean reduction from online conformal shielding to online conformal abstention, and that the experiments show OCS maintains SSR better than fixed-threshold shielding. We address each concern below.

W1. Whether OCS Counts as Shielding When It Does Not Return Control to the Policy

(1) The proposal is about when to intervene rather than about shielding. In safe RL, shielding broadly refers to runtime mechanisms that monitor a policy and block or replace actions that may violate a safety specification [1, 2]. Within this family, some methods compute a safe replacement action, as in model predictive shielding, while others block the action and hand control to an expert, a supervisor, or a recovery policy [3, 4]. OCS belongs to the second group: it addresses when to intervene, with a guarantee on the executed actions, rather than how to construct the replacement action.

(2) A shield should always execute safe actions provided by a planner. Always executing a provably safe action requires strong assumptions, such as a known or learned dynamics model, a specified safe set, and a safe planner [5, 6]. OCS abstracts the replacement as the intervention action INT, so any safe planner or backup can take over, including a model-predictive one, and the guarantee does not depend on which backup is used.

(3) After an intervention, control never returns to the policy. An intervention ends task execution only for the current episode, and the next adaptation episode starts again with the task policy, which keeps being updated. Resuming the policy within an episode requires a backup that brings the system to a recoverable state, which existing methods obtain through additional assumptions, such as a safe expert [3], a recovery policy pretrained on offline violation data [4], or a dynamics model with a specified safe set [5, 6]. With such a backup, OCS can hand control back without changing its decision rule or guarantee, but designing that backup is orthogonal to the question OCS addresses.

[1] R. Bloem et al. Shield Synthesis: Runtime Enforcement for Reactive Systems. In TACAS, 2015.

[2] M. Alshiekh et al. Safe Reinforcement Learning via Shielding. In AAAI, 2018.

[3] J. Zhang and K. Cho. Query-Efficient Imitation Learning for End-to-End Simulated Driving. In AAAI, 2017.

[4] B. Thananjeyan et al. Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones. IEEE Robotics and Automation Letters, 2021.

[5] O. Bastani. Safe Reinforcement Learning with Nonlinear Dynamics via Model Predictive Shielding. In ACC, 2021.

[6] A. Banerjee et al. Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning. In NeurIPS, 2024.

W2 & Q2. The Use of a Q-Function Scorer With Clipping Instead of Softmax Is Not Justified

Softmax does not apply here, because the scorer produces a single scalar for each state–action pair rather than a set of values to normalize. The shield requires an action-conditioned risk estimate to evaluate each proposed action before execution, so we use a cost-Q head that shares the task policy's backbone and is trained with a horizon-normalized TD target (Appendix B.2). We use clipping to bound the predicted scores to [0, 1] while preserving their original values within this range, so each threshold can be read as an admissible average cost.

Note that clipping, or an alternative such as a sigmoid, is a design choice: our safety guarantee does not require calibrated risk scores or a specific score transformation, provided Assumption 1 holds. The transformation, like the scorer's predictive quality, affects only the efficiency of selective execution.

AuthorsOpenReview ↗

Weakness3,4

W3. On Goal2, Risk Control Comes at a Large Cost in Utility

(1) On Goal2, SSR is reduced at the cost of efficiency and success, as if the threshold stopped execution. The low utility on Goal2 comes from the risk of the environment, not from a threshold that stops execution. Without a shield, the risk there is about five times the target, so meeting the target requires executing fewer actions; over five seeds, OCS reaches SSR 0.030 with efficiency 0.21 and success 0.04. An intervention halts the episode, and its remaining steps count as waiting in the denominator of efficiency, so the fraction of executed decisions on active steps is higher than the efficiency.

(2) Achieving a target SSR should not be the goal. Our primary goal is to control execution-time risk during adaptation; preserving success is the secondary goal. We keep the base learner fixed to a standard method so that the effect of the shield is isolated. Improving success under a tight risk target is a question of the base learner, which is orthogonal to how the shield controls risk.

(3) The trade-off should be shown along a Pareto front of reward and violations. We agree, and Figure 13 (Appendix E.8) shows that α\alpha moves OCS along a risk–utility front, and that at matched risk OCS attains higher utility than the fixed thresholds on Goal1 and comparable utility on Goal2. We ran an α\alpha sweep of OCS under the same active-step budget (10610^6), together with fixed thresholds calibrated on source rollouts to α\alpha and α/2\alpha/2 and an intermediate threshold η=0.5\eta = 0.5, and measure utility by efficiency and task success. Tighter targets take longer to reach within a fixed budget; at α=0.01\alpha = 0.01, SSR remains above the target in both environments.

The risk of a fixed threshold depends on the environment: η=0.5\eta = 0.5 meets the target on Goal1 but nearly doubles it on Goal2, and every fixed threshold in the sweep exceeds α\alpha on Goal2, whereas α\alpha specifies the risk level directly. For example, on Goal2, OCS with α=0.05\alpha = 0.05 has slightly lower SSR than η=0.5\eta = 0.5 but higher efficiency and success.

W4. Existing Shielding Methods Are Not Compared

The methods listed rely on assumptions that OCS does not make, such as a learned or symbolic dynamics model used to compute safe sets or plan recovery, a backup policy, and a safe region specified in the state space; Table 1 in Appendix A compares these assumptions for representative methods, including DMPS and RAMPS. OCS uses only observed costs, which allows it to be applied where such models and specifications are difficult to obtain.

To provide a direct comparison, we evaluated RAMPS [1], the most recent of the listed methods, on SafetyPointGoal1 with the authors' released code, the same pretrained policy, adaptation loop, 10610^6-active-step budget, and five seeds (Table 9, Appendix E.5). We report two settings: (i) system identification, with the observations available to OCS (robot sensors and lidar) and an initial system-identification dataset not counted in the budget or the risk; and (ii) a semi-oracle setting with hazard coordinates taken from the simulator. For RAMPS, QP-corrected actions count as executed.

In the system-identification setting, RAMPS avoids violations only by modifying nearly every action, because its learned dynamics model is inaccurate there. The observation dynamics are strongly nonlinear, since rotation shifts hazards across lidar bins, and the 99th-percentile error in the predicted distance to the nearest hazard is about 60% of the hazard radius; the robust QP must keep this margin. Without the system-identification data, the results are similar, so the limitation comes from the representation rather than from the amount of data. In the semi-oracle setting, the error falls below 10% of the hazard radius, and RAMPS has lower risk and higher utility than OCS; this, however, requires hazard positions that are not observable in deployment.

A model-based shield is therefore the stronger choice when the safe set can be written as a constraint on a state whose dynamics the model captures; OCS needs neither, and uses only the costs observed after execution. Since RAMPS was not evaluated on SafetyPointGoal in its paper, we adapted its safe region, backup, and semi-oracle constraints to this environment, as described in Appendix E.5.

[1] T. Ambadkar et al. Robust Adaptive Multi-Step Predictive Shielding. In ICLR, 2026.

AuthorsOpenReview ↗

Question1, Minor Comments

Q1. The Choice of 1002 Threshold Candidates Is Not Justified

K = 102 and K = 1002 perform similarly in both risk and utility, whereas the coarsest grid K = 12 (spacing about 0.09) has higher risk, exceeding α\alpha on Goal1, and lower efficiency and success (Table 12, Appendix E.7). The candidate set consists of 0.001-spaced values on [0, 1] plus the always-intervening threshold −0.001 (Appendix D.4), which Theorem 2 requires. The bound depends on K only through ln K, which grows from 2.5 at K = 12 to 6.9 at K = 1002, so a finer grid costs little in the guarantee, while a coarse grid restricts how closely the selected threshold can match the target risk; we use K = 1002 for this reason. On Goal2, all grid sizes end slightly above α\alpha within this budget.

Minor Comments

(1) Three seeds are too few to establish significant dominance. We expanded the main comparisons to five seeds, and the revised paper reports mean ± standard deviation and confidence intervals (Tables 7 and 8).

(2) There are punctuation errors at L375–376. We have corrected them in the revised paper.

Reviewer_dp2HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Online Conformal Shielding (OCS), a framework that selectively executes proposed actions or intervenes during online policy adaptation. It addresses safety staleness under coupled distribution shifts, including exogenous environment changes and endogenous shifts induced by policy and risk-scorer updates. OCS uses Online Conformal Abstention (OCA) to adapt the execution threshold from partial safety feedback while the policy and scorer continue to learn. The theoretical analysis establishes a high-probability bound on cumulative selective safety risk at a prescribed interaction budget, up to a finite-sample slack. Experiments on ManiSkill and Safety-Gymnasium show that OCS can maintain risk near the target while preserving useful task performance and execution efficiency.

Critical strengths
  • The paper addresses a relevant problem in online policy adaptation: maintaining execution-level risk control while both the task policy and the safety-risk scorer continue to change. The proposed framework is principled and shows empirical reductions in cumulative selective safety risk.

  • The problem formulation, selective-risk metric, and execution protocol are clearly specified. The separation between action proposal, risk scoring, and threshold adaptation makes the framework easy to follow.

Critical weaknesses
  • The core threshold-learning mechanism and risk guarantee are largely inherited from OCA, making the overall technical contribution appear incremental. The paper could more clearly articulate the technical advances introduced by the RL reduction and distinguish OCS from existing adaptive shielding methods.

  • Comparisons at different realized risk levels do not establish superior utility at comparable safety. In particular, on SafetyPointGoal1, the fixed-threshold shields appear to achieve risk close to the target while maintaining higher execution efficiency.

Questions for authors
  • Could the authors distinguish scorer degradation, changes in the policy-dependent cost target, and failure of threshold calibration under changing visitation distributions? Frozen-policy/scorer ablations would help identify the mechanism behind “safety staleness.”

  • On ManiSkill, why does Fixed threshold-prior yield higher SSR despite its lower threshold? Although SSR need not be monotonic in the threshold, this result warrants a more specific explanation, particularly regarding risk ranking, the long-term/immediate-cost mismatch, and adaptation dynamics.

5 comments

We thank the reviewer for the constructive feedback and for recognizing the importance of execution level risk control during online adaptation, the principled formulation, and the clarity of our framework. We address the concerns below.

W1. The Technical Contribution Beyond OCA and the Distinction From Adaptive Shielding Are Unclear

(1) The threshold-learning mechanism and the risk guarantee are largely inherited from OCA, and the advances of the RL reduction are not articulated. We state the OCA guarantee [1] as Assumption 1 and do not claim it as our contribution. OCA's guarantee holds for arbitrary data sequences, and in the OCA evaluation the scoring function may change over time, but no policy is updated and abstention decisions do not change the subsequent data. Robot adaptation adds this closed loop: the task policy must adapt after a shift, the scorer is trained online alongside it, and the shield's decisions determine the training data of both, which changes future proposals and scores (endogenous shift) on top of the environment shift.

Our contribution is twofold. First, we show that the OCA guarantee carries over to this closed loop. By Theorem 1, with probability at least 1−δ1-\delta, the selective safety risk stays below α\alpha plus a slack that vanishes as the budget grows, provided a nonvanishing fraction of active steps is executed. It handles unobserved intervened costs, halting, early termination, and variable horizons by mapping active steps to OCA rounds, excluding waiting steps from the updates, and tying the active-step budget to the OCA horizon (Appendix C.2). Scorer quality is not a condition of the guarantee, but it determines how quickly the risk approaches α\alpha within a budget (See Q1). Second, because early threshold selection matters in safety-critical adaptation, we introduce a prior-aware variant of ExAUL (Theorem 2), whose bound replaces ln⁡K\ln K with ln⁡(1/w1(ξ⋆))\ln(1/w_1(\xi^\star)); the prior-versus-uniform comparison is in Appendix E.3 (Figure 11).

To our knowledge, OCS is the first to use online conformal abstention as a shield during RL adaptation, where its own interventions change the training data of the policy and the scorer.

(2) OCS is not distinguished from existing adaptive shielding methods. Existing shields either fix a threshold calibrated before deployment or rely on a dynamics model and a specified safe set; OCS differs from both. A fixed calibrated threshold relies on exchangeability between calibration and deployment data, which the closed loop above breaks, so its realized risk cannot be set in advance; OCS instead re-selects the threshold online (See Q1). Among model-based shields, we compared with RAMPS [2] on SafetyPointGoal1 (Table 9, Appendix E.5): with the observations available to OCS, it has no violations but executes few actions, and it outperforms OCS only with hazard coordinates that are not observable in deployment. Table 1 in Appendix A compares the assumptions of these methods.

[1] M. Lee et al. Online Conformal Abstention Under Adversarial Bandit Feedback. In NeurIPS, 2026.

[2] T. Ambadkar et al. Robust Adaptive Multi-Step Predictive Shielding. In ICLR, 2026.

AuthorsOpenReview ↗

Weakness2, Question1,2

W2. Comparisons at Different Realized Risks Do Not Establish Utility at Comparable Safety

(1) Comparisons at different realized risk levels do not establish superior utility at comparable safety. We agree, and we do not claim utility superiority at matched risk; our primary objective is to control execution-time risk during adaptation. On ManiSkill, no tested fixed threshold keeps risk near the target with high success across the five environments (Figure 10, Appendix E.2): η=0.722\eta = 0.722, the threshold of Fixed-Calib, and 0.990.99 exceed α\alpha everywhere, and η=0.3\eta = 0.3 meets α\alpha in only three environments, with lower success than OCS in every environment. OCS lies below the line joining the fixed thresholds in all five environments.

(2) On SafetyPointGoal1, the fixed-threshold shields appear to achieve risk close to the target with higher efficiency. A threshold can be tuned to hold the target in one environment, but the same choice does not carry over once the environment shifts (Figure 13, Appendix E.8). On SafetyPointGoal1, the environment does not change between pretraining and adaptation, but the policy and scorer keep changing. There, the threshold calibrated on source rollouts for α\alpha already exceeds α\alpha, and the stricter ones (calibrated for α/2\alpha/2, and η=0.5\eta = 0.5) hold the target, the former with efficiency and success close to OCS, but both exceed it on Goal2. OCS at α=0.025\alpha = 0.025 meets the target on Goal1 and has the SSR closest to it on Goal2, slightly above.

OCS also sets the operating point through α\alpha directly: raising α\alpha raises both SSR and efficiency in both environments. Compared at the same SSR, OCS attains higher efficiency and success than the fixed thresholds on Goal1 and comparable utility on Goal2.

Q1. The Sources of Safety Staleness Are Not Disentangled

We distinguish three potential mechanisms: (1) Scorer degradation: environment shifts may weaken the scorer's risk ranking. (2) Policy-dependent cost target changes: policy updates alter the expected future safety costs estimated by the cost-Q scorer. (3) Threshold miscalibration: changes in state–action visitation and the score–cost relationship can invalidate a previously calibrated threshold. Our ablation shows that all three adapted components are needed: only the fully adapted configuration keeps SSR below α\alpha at high efficiency, and a fixed threshold becomes stale even without scorer degradation.

To disentangle these effects, we freeze each adapted component of OCS on Position: the task policy, the scorer, and the threshold learner, where a frozen threshold learner uses the fixed threshold of Fixed-Calib (η=0.722\eta = 0.722) (Table 10, Appendix E.6). The frozen-scorer runs were extended to check whether SSR approaches α\alpha with more interaction.

  • Threshold miscalibration. With an adapted scorer, the fixed threshold exceeds α\alpha whether the policy is adapted or frozen, while OCS meets it.
  • Scorer degradation. With a frozen scorer, the fixed threshold blocks more, and success collapses to about 1%, so its realized risk moves above or below α\alpha depending on whether the scorer is updated. OCS keeps its efficiency, but its SSR stays slightly above α\alpha even after 32,000 active steps; scorer learning mainly determines how fast OCS reaches the target.
  • Policy-dependent cost target changes. With an adapted scorer, freezing the policy lowers success from 0.22 to 0.135 under OCS and from 0.44 to 0.279 under the fixed threshold; the policy mainly drives task success rather than risk control.

We do not claim that OCS outperforms the fixed threshold in every configuration; with a frozen scorer, the fixed threshold has lower SSR, at near-zero success.

Q2. Fixed Threshold-Prior Has Higher SSR Despite Its Lower Threshold

A lower threshold reduces action acceptance but does not necessarily reduce SSR, which measures the average realized safety cost conditional on execution. On ManiSkill, Fixed threshold-prior uses a lower threshold than Fixed threshold-calibration (approximately 0.5 vs. 0.722), resulting in stronger shielding.

Several mechanisms may contribute to this behavior. First, the scorer estimates horizon-normalized cost-to-go, whereas SSR measures immediate safety costs, so predicted risk rankings may not align with realized costs. Second, different shielding thresholds change the executed transitions used for policy and scorer updates, leading to different state–action visitation and score distributions during adaptation. OCS instead continuously adapts its threshold using observed immediate costs, enabling selective-risk control under these evolving distributions without requiring perfectly calibrated risk scores.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

Thank you again for your time and thoughtful review. As the discussion period is ending soon, we would like to kindly check whether our responses have addressed your concerns. We have expanded the main comparisons to five seeds and added the requested analyses and experiments to the revised paper. If any questions remain, we would be happy to clarify them before the discussion closes

Reviewer_63dGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses safety during online adaptation of a pretrained policy to a shifted environment, where exploration is needed to adapt but carried-out actions can be costly. It casts the problem as a selective decision made before execution: the task policy proposes an action, a learned safety-risk scorer scores it, and an online-adapted threshold decides whether the action is executed or replaced by an intervention, while the policy and the scorer keep updating from the interaction that results. The threshold is adapted by reducing the problem to online conformal abstention, so a selective-risk guarantee for that subroutine transfers to the shield, including under feedback that only arrives for executed actions and under a closed loop in which the policy and scorer are themselves changing. The authors add a variant that initializes the threshold distribution from pre-adaptation information, and prove that the realized average safety cost among executed task actions is bounded by the target level plus a finite-sample slack that vanishes as interaction grows, provided the executed-task fraction stays away from zero. Experiments use VLA manipulation in ManiSkill under five scene, pose and visual shifts and continuous control in Safety-Gymnasium with added hazards, comparing against unshielded constrained RL, two fixed thresholds, and a conformal policy-level baseline.

Critical strengths
  1. The reduction is the right abstraction and it is done properly. Mapping each active step to an abstention round, identifying an intervention with abstention, and keeping the feedback partial — an intervened action's cost is never revealed — is exactly the structure that a shielding method has, and it lets the guarantee survive the closed loop in which the policy and the scorer change the very distribution they are being evaluated on. The imported guarantee is stated as an assumption with its source rather than re-derived, the correspondence is written out step by step, and the active-step budget is tied to the conformal horizon so that the fixed-horizon argument applies. That last detail, that waiting steps do not consume the budget, is the kind of thing that is usually glossed over.

  2. The empirical setting is the part of the paper that convinces me the problem is real. Five ManiSkill shifts with a VLA policy and continuous control with added hazards is a fair spread of coupled shifts, and the paper reports the trade-off honestly in places where it costs them: on EEPose the shield reaches 0.35 success against 0.45 and 0.56 for the fixed shields, and the paper says so, adding that the baselines operate at roughly twice the target risk. The α sweep showing the risk level tracking the target, and the threshold-prior ablation showing that the prior moves utility rather than safety, are the two results that most convincingly demonstrate the mechanism works as described rather than the numbers being tuned.

Critical weaknesses
  1. The controlled quantity is a conditional average over the shield's own executed actions, and the claims about not paying for it are not supported by the tables. The risk measure divides realized cost by the number of executed task actions, intervention cost is assigned a unit value that never enters the risk measure or the conformal feedback, and the theorem's slack is divided by the executed-action count, so a shield that intervenes more reports lower risk by construction and efficiency and success have to carry the argument. On the hardest environment the shield intervenes on about two thirds of steps, with efficiency near 0.33. The claim that risk control is achieved without sacrificing efficiency and success also fails against the strongest baselines: the five-setting mean success is 0.664 against 0.682 for the fixed prior threshold and 0.700 for the fixed calibrated one, and on Position the method reaches 0.22 against 0.34 for the unshielded policy. Three things make that average hard to read as a claim about the method. The settings are not comparable: success ranges from 0.22 to 0.91 with three near ceiling, so an unweighted mean moves by several points with any change in task mix. The comparison is not at the same operating point: the paper notes that on EEPose the fixed shields run at roughly twice the target risk, so their higher success is bought with the risk budget this method is holding. And the headline compares against the unshielded policy alone, whose success sits near its floor or its ceiling. A fixed shield tuned to realize the same realized risk would settle the question; without it, neither 11 points over the unshielded policy nor falling below the fixed shields can be read as the cost or the benefit of meeting the target.

  2. The comparisons are weakly powered, and at least one is dominated by variation that the paper does not report. Every curve and table averages three seeds with no confidence intervals or paired tests, and two configurations that both execute every proposed action — the unshielded method and the fixed calibrated threshold both report efficiency of exactly 1.00 — differ by 38 points of success on the same EEPose shift (0.18 against 0.56) at essentially the same risk (0.046 against 0.044). That is the scale of sampling variation here, and it exceeds the 11-point improvement the abstract emphasizes. Separately, the calibrated threshold is chosen on pretraining rollouts by taking the candidate with the highest success and efficiency among those meeting the target, which selects the most permissive admissible threshold; its efficiency of 1.00 in all five shifted environments means it never intervenes, so the failure attributed to staleness is partly the failure of a shield that is switched off. The baseline set also stops short of the control that would isolate the contribution: the method is the online conformal abstention subroutine of Lee et al. instantiated with one algorithm and a pre-adaptation prior, yet the subroutine is never a row, the only conformal baseline is policy-level and given twelve times the interaction budget, and the closest ablation changes the prior rather than removing the adaptation. Why this is consequential: both headline questions rest on comparisons that cannot be separated from noise, from baseline construction, or from the untested gap between the method and the subroutine it builds on.

Questions for authors
  1. Can you evaluate the finite-sample slack numerically at the reported operating point, with the active-step budget of 8,000 and a target of 0.025? If the bound implied by the theorem is of the same order as the target, the theory does not constrain the reported regime and the empirical curves carry the claim alone. A single number would settle this.

  2. What is the fixed calibrated shield's intervention rate in the pretraining environment, where its threshold was selected? The reported efficiency of 1.00 across all shifted environments says it never fires there; if it also never fires in-distribution, the staleness comparison is between an adaptive shield and a disabled one, and a calibrated threshold that fires in-distribution but stops firing after the scorer drifts would be a much stronger baseline.

  3. How should the per-environment success numbers, and the 66% average, be read? The method is below the unshielded baseline on Position (0.22 against 0.34) and below both fixed thresholds on EEPose (0.35 against 0.45 and 0.56), and the five settings span 0.22 to 0.91, so an unweighted mean over them mixes near-ceiling tasks with difficult ones. What success rate do these tasks admit for a well-tuned policy — a published reference on the same environments, or an unshielded upper-bound run — and what is the per-environment variance over seeds and episodes? Is 66% meant as a level achieved on these benchmarks, or only as an average relative to the baselines you ran?

  4. Why is the online conformal abstention subroutine itself never a baseline? The method is that subroutine instantiated with one algorithm and a pre-adaptation prior, so the row that isolates the contribution is the same shield running plain OCA with a uniform prior, on the ManiSkill table rather than on a single Safety-Gym environment. Relatedly, were OCA algorithms other than ExAUL tried, and how much of the reported behaviour depends on that choice?

  5. Which other conformal-selection or online-adaptation baselines were considered, and why is CPC the only conformal method in the comparison? CPC is policy-level, requires a safe reference policy, and consumes twelve times the interaction budget; the nearest competitors for a threshold-adaptation method are other online calibrations — adaptive-conformal-style updates, alternative abstention or selection rules — and in their absence it is hard to tell whether the advantage comes from adapting the threshold online or from the specific instantiation chosen.

5 comments

We thank the reviewer for the careful and detailed review. We appreciate the recognition that the OCS-to-OCA reduction is the right abstraction for shielding, including its treatment of partial feedback and the active-step budget, and that the ManiSkill and continuous-control settings make the coupled-shift problem concrete. We also appreciate the note that the α\alpha sweep and the threshold-prior ablation demonstrate the mechanism. We address each concern below.

W1. Risk Control Is Not Shown to Come Without a Utility Cost

We do not claim that risk control comes at no cost. Our primary objective is to control the risk of executed task actions during adaptation. "Without sacrificing" appears in the statement of our research question (Q2), and Section 5.3 reports the utility cost on EEPose and SafetyPointGoal2. We rephrased Q2 to ask how much utility is retained while the risk is controlled, and discuss utility-aware threshold selection, for example with contextual bandits, as future work.

The reviewer raises five points, which we group into three parts.

W1-(a). The risk measure may be lowered simply by intervening more

(1) The risk measure is a conditional average over the shield's own executed actions. Intervening more does not lower SSR by construction: SSR decreases only if the removed actions are riskier than those that remain, which depends on the scorer's ranking. At a fixed active-step budget, executing fewer task actions also enlarges the slack BOCA/NtaskB_{\mathrm{OCA}}/N^{\mathrm{task}} in Theorem 1, so the guarantee becomes weaker, not stronger, under over-intervention. SSR must still be read together with how much is executed; this is why we report efficiency, which counts both interventions and subsequent waiting as lost task execution.

(2) On the hardest environment, the shield appears to intervene on about two thirds of steps. On SafetyPointGoal2, OCS reaches SSR 0.030 with efficiency 0.21 over five seeds. The unshielded risk there is about five times the target, so meeting the target requires executing fewer task actions (Section 5.3). This efficiency does not imply interventions on most steps: an intervention halts the episode, and its remaining steps count as waiting in the denominator, so the intervention rate on active steps is much lower.

W1-(b). The utility claims are not supported against the strongest baselines

(3) The claim that risk control does not sacrifice efficiency and success is not supported against the strongest baselines. As stated above, we do not make this claim. The results should be read by first examining deviation from the target risk, and then comparing efficiency among the methods that control risk. The per-environment comparison, including Position, where OCS success is below the unshielded policy, is in Q3.

(4) The unweighted success mean over non-comparable settings, and the headline against the unshielded policy alone, are hard to read. We replaced the 66% average and the 11-point headline with per-environment results against all baselines (See W2 and Q3).

W1-(c). The comparisons are not at the same operating point

(5) The comparisons are not at the same operating point; a fixed shield tuned to the same realized risk would settle the question. No single fixed threshold keeps risk near the target with high success in all five environments, whereas OCS does with a single α\alpha (Figure 10, Appendix E.2). We evaluated the calibrated threshold η=0.722\eta=0.722 (Fixed-Calib, Appendix D.4), stricter thresholds (η=0.3\eta=0.3 and 0.10.1), and a nearly permissive one (η=0.99\eta=0.99). The permissive thresholds exceed α\alpha everywhere, and their higher success in some environments comes with this higher risk. The stricter thresholds lower the risk but give up task success: η=0.3\eta=0.3 meets α\alpha in only three environments, with lower success than OCS in every environment, and η=0.1\eta=0.1 almost never completes the task. In Figure 10, OCS lies below the line joining the fixed thresholds in all five environments. A threshold matched to OCS's risk also cannot be chosen before adaptation, since finding it requires feedback from the new environment.

W2. Comparisons Are Weakly Powered, and the Calibrated Baseline May Be Inactive

(1) Every result averages three seeds, without confidence intervals or paired tests. With five seeds, OCS keeps SSR near α\alpha in every environment with a small spread across seeds, while PPO-Lag stays well above it. We report mean ± standard deviation (Table 7) and 95% paired confidence intervals of the SSR difference, pairing runs by seed (Table 8, Appendix E.2). The interval excludes zero against PPO-Lag in all five environments and against Fixed-Calib in four; on MultiCarrot, the pretraining environment, the gap to Fixed-Calib is not distinguishable from seed variation. These intervals are not corrected for multiple comparisons.

(2) On EEPose, two configurations at the same risk differ by 38 points of success, more than the 11-point improvement. The 38-point EEPose gap reflects that environment's unusually high variance rather than the scale of variation across the benchmark: EEPose success has standard deviation 0.17–0.24 across methods, compared with at most 0.10 in the other four environments. With five seeds, the gap between Fixed-Calib and PPO-Lag is 30 points, less than two standard deviations of either method. We agree that this gap should not serve as a headline, and we removed the aggregate 11-point claim.

(3) The calibrated threshold is the most permissive admissible one and appears never to intervene. Fixed-Calib is not a disabled shield. The displayed efficiency of 1.00 was a rounded value; its five-seed EEPose efficiency is 0.986 ± 0.012, and it also intervenes in the pretraining environment (See Q2).

(4) The baseline set does not isolate the contribution. The online conformal abstention subroutine itself is already a row: the ManiSkill OCS configuration is OCA with a uniform prior (See Q4). To isolate what controls SSR, we freeze each of the three adapted components of OCS on Position: the task policy, the scorer, and the threshold learner, where a frozen threshold learner uses the fixed threshold of Fixed-Calib (Table 10, Appendix E.6).

  • Threshold learner: calibrates the shield to α\alpha. Without it, the fixed threshold misses α\alpha in either direction: with an adapted scorer, it exceeds α\alpha (0.0291 and 0.0278), and with a frozen scorer, it blocks too much and success collapses to about 1%.
  • Scorer: determines how much risk the shield lets through. With the threshold learner, freezing the scorer leaves SSR slightly above α\alpha even after 32,000 active steps (0.0252 and 0.0265).
  • Task policy: determines task success. With an adapted scorer, freezing the policy lowers success from 0.22 to 0.135 with the threshold learner and from 0.44 to 0.279 with the fixed threshold.

Only the configuration with all three components adapted keeps SSR below α\alpha at high efficiency. In the revision, CPC is evaluated under the same active-step budget by counting every environment step of its generated trajectories, and it remains above α\alpha in every ManiSkill environment.

Among online-adaptation shields, we compare with RAMPS [1], a recent model-predictive shield that also updates its dynamics model during adaptation (Table 9, Appendix E.5). RAMPS is given an initial system-identification dataset, not counted in the budget, and we also report a semi-oracle setting with hazard coordinates from the simulator, which OCS does not use. With lidar observations, the learned model's 99th-percentile error in the distance to the nearest hazard is about 60% of the hazard radius. The robust QP must keep this margin, so it modifies nearly every action: RAMPS has no violations but low success, rather than controlling the risk at α\alpha. In the semi-oracle setting, the error falls below 10% of the hazard radius, and RAMPS has lower risk and higher utility than OCS, but this requires hazard positions that are not observable in deployment.

[1] T. Ambadkar et al. Robust Adaptive Multi-Step Predictive Shielding. In ICLR, 2026.

Q1. The Finite-Sample Slack Is Not Evaluated at the Tested Budgets

The slack decreases as 1/N1/\sqrt{N}: it exceeds α\alpha at the ManiSkill budget, and it is of the same order as α\alpha at the Safety-Gymnasium budget. Using Eq. (6) with K=1002K = 1002, λ=N\lambda = \sqrt{N}, ν=ln⁡K/N\nu = \sqrt{\ln K/N}, and δ=0.05\delta = 0.05, Theorem 1 gives the slack BOCA/NtaskB_{\mathrm{OCA}}/N^{\mathrm{task}} in Table 2 (Appendix B.1). With full execution (Ntask=NactN^{\mathrm{task}} = N^{\mathrm{act}}, the smallest possible value at a given budget), it is 0.146 at 8,000 active steps and 0.019 at the Safety-Gymnasium budget of 10610^6; at the efficiency observed in our runs, it is 0.151–0.153 at 8,000 active steps, while the observed SSR is 0.020–0.026.

The near-target SSR on ManiSkill is therefore an empirical result rather than a finite-budget certification at α\alpha. With full execution, the slack falls below α\alpha at about 2.7×1052.7 \times 10^5 active steps (Figure 5). The theorem's role is to establish that a cumulative bound remains valid under coupled adaptive shifts, with slack vanishing under the stated condition; tightening its constant is future work.

Q2. The Calibrated Shield May Not Intervene Even in the Pretraining Environment

Fixed-Calib is an active and strong baseline in the pretraining environment. At η=0.722\eta=0.722, selected on pretraining rollouts as the threshold with the highest success and efficiency among those satisfying α\alpha (Appendix D.4), the fixed calibrated shield has an inefficiency (1 − Eff) of 1.5% in the pretraining environment, counting interventions and the subsequent waiting, while achieving SSR 0.025.

Under shift, it keeps intervening at a similar rate, yet its SSR exceeds the target in every shifted environment. The failure is not that the shield stops firing, but that a threshold selected before adaptation does not retain the target risk once the policy, scorer, and environment change.

Q3. The 66% Average Success Mixes Environments of Very Different Difficulty

The 66% was a descriptive, unweighted average over the five settings, not an estimate of an attainable level, and we removed it together with the 11-point headline. Our claim is instead that OCS keeps the risk near the target in every environment with a single α\alpha and no per-environment tuning.

We are not aware of a published reference for our shifted settings. The source benchmark [1] studies reward-only fine-tuning with a much larger interaction budget (RL overtakes its best supervised baseline only after about 0.4M environment steps, compared with our 8,000 active steps), so its numbers do not provide a reference for constrained adaptation at our budget. Within our experiments, the closest reference is unshielded PPO-Lag, although it is not an upper bound on attainable success.

On Position and EEPose, OCS has lower success than Fixed-Calib, and on Position also lower than PPO-Lag, while both baselines operate above the target risk (Table 7). Higher baseline success at higher risk does not isolate the utility cost of meeting the target, and lower OCS risk does not establish that its utility is optimal. The standard deviations in Table 7 are over five seeds; we do not treat episodes of an adapting policy as independent repetitions, so we do not report an episode-level estimate.

[1] J. Liu, F. Gao, B. Wei, X. Chen, Q. Liao, Y. Wu, C. Yu, and Y. Wang. What Can RL Bring to VLA Generalization? An Empirical Study. In NeurIPS, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.