acceptodds
Under review as a conference paper at ICLR 2027

Sanctuary: Safe Correlated Exploration for Online Vision-Language-Action Models Adaptation

Abstract

Online reinforcement learning (RL) can adapt vision–language–action (VLA) models to new tasks, but exploration during RL exposes robots to physical risks. Execution safeguards can prevent hazardous actions from being carried out, but the policy still needs to learn to avoid proposing them. We introduce SANCTU- ARY, a framework that preserves this risk information to guide exploration under real-time execution protection. SANCTUARY evaluates each complete action chunk before intervention and combines its risk score with the subsequent execu- tion outcome to learn long-term risk. Task value and long-term risk then jointly guide the action mean and exploration correlations across time and action dimen- sions, while marginal noise variances remain fixed. This allows blocked hazardous proposals to inform policy improvement even when protection prevents physical violations. On LIBERO-Safety and 12 custom simulation scenarios, SANCTU- ARY improves success rates by 6.3–9.0 percentage points over the strongest base- line in each evaluation, with no observed physical violations. Ablations support the complementary roles of risk learning and execution protection

Primary area:
reinforcement learning
Keywords:
Safe Reinforcement Learning, Safe Exploration, Vision-Language-Action Models, Constrained Robot Learning, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hWuHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SANCTUARY, a safety-aware online RL fine-tuning framework for vision-language-action models. The method distinguishes between an original action-chunk proposal and its protected execution: a task critic and a safety critic learn from the original proposal, while hard constraints intervene during execution to prevent unsafe commands. SANCTUARY further combines a Risk Potential Field, a Latent Risk Monitor, and learned correlations for action-chunk exploration. The paper evaluates the method on LIBERO-Safety, twelve custom simulation scenarios, and one real-robot pick-and-place task, reporting improved task success and lower violation rates relative to several VLA and safe RL baselines.

Critical strengths

The paper addresses an important problem: reducing risk during online RL fine-tuning of VLA models. Its central design distinction between the original action proposal and the protected executed action is well motivated. By retaining risk feedback for proposals that are blocked by the execution-time safety mechanism, the method aims to avoid treating a prevented hazardous action as evidence that the proposal itself was safe.

Critical weaknesses
  1. The empirical evidence is currently incomplete and insufficient to fully support the paper’s main effectiveness and safety claims. The paper explicitly states that simulation sample counts, training seeds, and uncertainty estimates remain to be reported. It also acknowledges that the correspondence between the baseline protocols used in the main comparison and ablation tables remains to be verified, and therefore differences across these tables should not be interpreted. Without these details, it is difficult to assess the reliability of the reported performance margins, safety improvements, and zero-violation results.

  2. Figure 3 is presented as a trajectory analysis, but the paper explicitly describes it as a schematic illustration rather than measured training trajectories or collision statistics. It therefore helps explain the intended mechanism, but does not provide empirical evidence that SANCTUARY produces safer exploration during training.

Questions for authors
  1. Could the authors provide the simulation sample counts, number of training seeds, and uncertainty estimates for Tables 1 and 2, and clarify whether the main-comparison and ablation results use comparable evaluation protocols?

  2. Could the authors provide measured training-time safety evidence, such as violation counts, protective-intervention rates, or clearance statistics across multiple seeds, rather than only the schematic illustration in Figure 3?

Reviewer_D7yZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors present an online safe reinforcement learning algorithm for improving pretrained VLAs without violating safety constraints. The key idea is to add negative supervision from potential safety violations, even if the safety filters prevent the policy from actually executing them. They build a system with several safety filters for soft and hard constraints for their tasks which they use to supervise the policy and safety critic. They present results showing their method achieving a higher success rate and lower constraint violation rate than their comparisons.

Critical strengths

Recent work has achieved great success finetuning VLAs with online RL and the authors consider safe RL, which could make this process more scalable

Critical weaknesses
  • The experimental section is very difficult to follow and lacking key details and results. The paper concerns online RL but the experimental section actually makes no reference to online RL training. It is unclear to me which results arise from offline training vs online finetuning. Furthermore, the entire point of safe RL is to minimize online constraint violations but the authors do not report any results for this metric.
  • The authors seem to give substantial additional information to their own policy. Are the reported benefits simply arising from their policy making use of infrastructure like the hard coded Hard Constraints function or demonstrator support that the authors provide?

The main simulation comparisons retain each method’s own safety mechanisms rather than adding SANCTUARY’s Hard Constraints to every baseline; these comparisons therefore evaluate complete systems

  • The pipeline seems to fundamentally depend on several hard-coded functions that the authors provide. Is this scalable? To what extent can these be learned while maintaining the same safety guarantees?
  • The authors actually repeatedly mention, in the main text of their paper, that details are missing from the paper. For example line 254:

Training-time safety statistics, simulation sample counts, and uncertainty estimates remain to be provided.

Questions for authors
  • Can you please provide details necessary to understand the online RL results?
  • Can you clarify whether the baselines have access to the same information that your policies do? In particular functions you wrote such as the hard constraints and the familiar episode support.
  • Can you comment on the scalability and whether these functions can in principle be learned?
Reviewer_h3pZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The work proposed a new framework for safe reinforcement learning using vision-language-action models. Specifically, it addresses an important problem of safe exploration through shielding unsafe actions and defining latent risk monitoring mechanism. The latent risk monitoring mechanism estimates the long-term risks and hence, helps in safe exploration. The proposed framewok combines blocked actions and structured action-chunk exploration to learn safe policy in VLA models. The proposed safe VLA based RL approach is demonstrated on LIBERO-Safety tasks and real-life experiment (pick-and-place with robot arm) and compared with several SOTAs.

Critical strengths
  1. The structured action-chunk exploration method with fixed noise variance is a good approach for safe exploration in RL setting.
  2. The joint learning of task value and long-term risk leads to safer and better rewarding actions during exploration.
  3. The real-life demonstration of the proposed approach is definitely plus.
Critical weaknesses
  1. Several important details including notations arer provided in appendices. Hence, it is difficult to read the paper without often referring to the appendix, particularly, Appendix C.
  2. The experimental evaluation primarily focuses on simulation (LIBERO-Safety and real-life application), and hence, generalizability to other tasks, for example, tasks in Safety-MuJoCo.
  3. The results on LIBERO-safety and the ablation studies are described mostly qualitatively. It is difficult to understand performance on different tasks and the results on each task are needed to assess the individual contribution of each component.
  4. The LRM and structured action-chunk exploration may introduce additional computational burden. However, no discussion on the computational burden.
Questions for authors
  1. The choice of a fixed noise variance σ\sigma in the structured action-chunk exploration is not clearly explained. How the σ\sigma is chosen in this work? Is it possible to improve the performance by adaptively adjusting the noise variance based on the estimated risk or the uncertainty in the policy?
  2. What is the rationale behind this specific value of retention probability=0.5? How sensitive are the results in this work to changes in this probability?
  3. It is not clear how the Hard constraints are included in the proposed approach. Can you provide how hard constraints are handled in your approach during training and inference?
  4. What will be the performance of the proposed approach on HazardArena or Safety-MuJoCo?
  5. Is it possible to provide details on the sample sizes, uncertainty estimates, details of the metrics used to evaluate the real-experiments of pick-and-place task for robotic arm?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.