acceptodds
Under review as a conference paper at ICLR 2027

SafeVLA-Bench: A Benchmark for the Success–Safety Gap in Vision-Language-Action Models

10 views

Abstract

Vision-language-action (VLA) benchmarks measure whether a policy completes a requested manipulation task, but binary success can hide safety violations along the trajectory: a policy may reach the goal while applying excessive contact, disturbing bystander objects, destabilizing a held object, or entering robot self-contact. We present SafeVLA-Bench, a post-hoc safety-evaluation framework for existing simulator-based VLA benchmarks that reveals violations missed by success-only evaluation. It encodes task-aware safety requirements as Signal Temporal Logic (STL) invariants with quantitative robustness semantics. Alongside native success, it reports the safety rate and the success-but-unsafe rate (SBU) used in prior safety evaluations, and introduces the Violation Severity Index (VSI), a bounded worst-violation depth score. We instantiate SafeVLA-Bench on LIBERO and RoboCasa-365, evaluating twenty-seven policy–benchmark entries across tabletop and kitchen manipulation tasks. High task success does not imply safe execution: the fifteen tabletop policies above 90% mean success still have 18–28% unsafe-episode rates, and 38–56% of successful RoboCasa-365 rollouts violate at least one active safety clause. A post-training case study further shows that SafeVLA-Bench can be used to improve policy safety.

Primary area:
datasets and benchmarks
Keywords:
vision-language-action models, robot safety, benchmark, signal temporal logic, robot manipulation, safety evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mWEKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SafeVLA-Bench, a post-hoc safety evaluation framework designed to assess safety violations in vision-language-action (VLA) policies beyond binary task success. The framework augments existing LIBERO and RoboCasa-365 evaluations with trajectory-level safety monitoring while preserving their original tasks, success predicates, and rollout protocols. It defines eight task-aware safety constraint families spanning scene interaction, object handling, and robot execution, using an applicability registry and quantitative Signal Temporal Logic (STL) robustness. Alongside native success rate (SR), the benchmark reports Safety and success-but-unsafe rate (SBU), and introduces the Violation Severity Index (VSI), which measures the average worst normalized violation depth across episodes.

The authors evaluate 24 distinct policies across 27 policy–benchmark entries, demonstrating that high task success does not necessarily imply safe execution. The paper further investigates discrepancies in policy rankings, task-structural safety violations, threshold sensitivity, and temporal violation patterns. Finally, a post-training case study on LIBERO-Spatial constructs same-initial-state preference pairs using SafeVLA-Bench safety labels and applies Diffusion-DPO to π0.5, improving its measured safety performance.

Critical strengths
  • A practical and well-designed post-hoc safety evaluation framework that preserves comparability with existing benchmarks. Unlike safety benchmarks that require constructing new tasks or benchmark-specific policy fine-tuning, SafeVLA-Bench integrates safety evaluation into existing VLA benchmarks without modifying their original tasks, success predicates, or rollout protocols.

  • Additional hardware validation and safety post-training experiments strengthen the practical relevance of the benchmark. Beyond evaluating VLA safety in simulation, the paper includes a small-scale real-world robot experiment and a safety-oriented post-training case study, making the experimental evaluation more comprehensive. In particular, the post-training experiments demonstrate that SafeVLA-Bench can serve not only as a safety evaluation tool but also as a source of safety preference labels for further VLA optimization.

Critical weaknesses
  • Limited methodological novelty and insufficient evidence for the incremental utility of VSI. SafeVLA-Bench shares substantial similarities with existing safety evaluation benchmarks, particularly SafeManip, which also performs task-aware temporal safety monitoring on existing RoboCasa-365 tasks using LTLf specifications. Although SafeVLA-Bench extends this direction by introducing quantitative STL robustness and the Violation Severity Index (VSI), the paper lacks a direct comparison or discussion with SafeManip. Moreover, while the authors analyze policy-ranking discrepancies and violation severity distributions, the paper provides limited insights into the underlying failure modes that explain VSI differences across policies and how such insights can guide safety improvements. This concern is further reflected in the post-training experiments: the successful DPO approach relies primarily on binary safe/unsafe preference labels rather than directly leveraging VSI, whereas the VSI-informed Advantage-conditioned SFT performs worse than the base policy in terms of safety. These results leave the incremental contribution of VSI to downstream safety optimization insufficiently demonstrated.
  • Limited effective safety coverage despite the eight defined constraint families. Although SafeVLA-Bench defines eight safety constraint families, Tables 7–9 reveal that several constraints are either structurally unavailable or rarely violated in the evaluated benchmarks. For instance, grasp-slip and joint-torque violations are consistently absent. Consequently, the reported safety violations appear to be dominated by a relatively small subset of constraints, particularly contact force and bystander displacement in LIBERO, and force-related constraints and transport tilt in RoboCasa-365. This raises concerns about whether the benchmark provides sufficiently comprehensive safety evaluation and whether its reported safety rankings are disproportionately influenced by a few simulator- or task-specific signals.
Questions for authors
  • Novelty relative to SafeManip. Could the authors clarify the essential methodological contributions of SafeVLA-Bench beyond SafeManip.
  • Validity and coverage of safety constraints. Given that several safety constraints are rarely or never violated (e.g., grasp slip and joint torque), while a few others dominate the reported violations (Tables 7–9), could the authors further justify the validity and applicability of these constraints and analyze their individual contributions to the overall safety evaluation?
  • Analysis of High-SI Tasks. Could the authors provide further analysis of why PickPlaceDrawerToCounter has an SI of 100%? Is this due to inherent task requirements, limitations of current VLA policies in executing the task safely, or potentially inappropriate safety constraints?
  • Evaluation of force-aware VLAs. Could the authors extend the evaluation to force-aware VLA models and examine whether the existing force-related safety constraints adequately capture their safety performance?
  • Task-aware Safety Constraint Applicability. How are task templates and semantic tags assigned to individual tasks? Is this process manually annotated or automated, and how is the correctness of the resulting safety-constraint applicability validated?
1 comment

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts in addressing my concerns and will maintain my positive rating.

Reviewer_nMVyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

SafeVLA-Bench adds post-hoc safety evaluation to existing LIBERO and RoboCasa-365 rollouts without requiring additional policy training or changing native task-success predicates. It does this by combining task-dependent applicability rules, Signal Temporal Logic specifications, and a Violation Severity Index (VSI). The evaluation covers 24 distinct policies and shows that high task success can coexist with frequent safety-specification violations. Additionally, a post-training study demonstrates that these safety labels can also improve policy behavior.

Critical strengths

Useful evaluation objective and substantial coverage: The paper examines behavior that binary task-success metrics overlook. Evaluating 27 policy-benchmark entries across two environments provides substantial evidence that policies with high success rates can also violate defined safety specifications within the evaluated settings. Practical integration with existing benchmarks: Preserving native success predicates allows researchers to examine safety alongside familiar performance measures. The explicit signal interface, specification library, and applicability registry make the framework reasonably concrete and reusable. Detailed analysis and demonstrated utility: The appendices provide per-constraint breakdowns, threshold and duration sensitivity analyses, and implementation details. The post-training study includes alternative training approaches and three DPO seeds, demonstrating a practical use for the labels beyond policy ranking. The authors also clearly acknowledge several limitations of their measurements.

Critical weaknesses

The physical interpretation of the safety proxies needs stronger validation. Selecting thresholds independently of the evaluated policies is desirable, but external provenance alone does not establish their suitability for the measured interactions. For example, the contact-force proxy draws on human-robot contact guidance while the evaluated scenes involve objects and furniture. The displacement and tilt boundaries likewise encode very specific assumptions about undesirable outcomes. The authors appropriately distinguish specification compliance from certified physical safety. However, the former has to capture some extent of the latter to be useful in real deployment. Threshold sensitivity demonstrates dependence on specification choices rather than validating their physical meaning. The hardware pilot is interesting in that it shows the existence of unsafe successes under video-based criteria, but does not validate quantitative robustness or VSI. Scalability of semantic specification and preservation of safety coverage remain uncertain. The seven reusable templates reduce the need to author rules independently for every task. However, extending to new mechanisms, objects, and contexts still requires semantic annotation and validation. Whole-episode exclusions illustrate this challenge as well: a constraint invalidated by one component is disabled throughout the rollout. Wine-rack insertion requires tilting during insertion, but this does not establish that unrestricted tilt is appropriate during every preceding phase. Such exclusions avoid false positives while potentially missing relevant violations. For these reasons, open-world coverage, with its explosion in task diversity and composition, is beyond the demonstrated scope; it could be useful if the authors could consider and address the question of how annotation effort and missed hazards grow as tasks become more diverse and compositional. Ranking-stability claims warrant qualification: the sensitivity analysis is useful, including its treatment of perturbations that actually change scores. However, Table 12 shows that relaxing the force threshold from 200 N to 500 N changes the top policy by Safety in all five suites. Broad ranking agreement should therefore be distinguished from reliable identification of the safest policy.

Questions for authors
  1. Beyond threshold sensitivity, what evidence supports the relevance of the selected safety boundaries and severe anchors to meaningful undesirable outcomes?
  2. What annotation and validation effort is required for a previously unseen task mechanism? Could phase-specific or object-specific applicability preserve coverage currently lost through whole-episode exclusions?
  3. Which conclusions about policy rankings remain reliable across plausible force thresholds? (Presentation: A worked example in the main text tracing task tags through applicable constraints, measured signals, and final scores would improve accessibility. This is a presentation suggestion rather than a central reason for the recommendation.)
Reviewer_pwggOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SafeVLA-Bench proposes a benchmark that encodes task-aware safety requirements as an add-on to existing simulator VLA benchmarks (LIBERO and RoboCasa-365). The benchmark reports success-but-unsafe (SBU) rate and introduces the Violation Severity Index (VSI), which measures a normalized depth of the worst safety violation in each rollout. The authors evaluate 27 policy-benchmark entries across 58 manipulation tasks and show that high task success is not necessarily indicative of safe execution. Namely, they find that across policies, 38-56% of successes in RoboCasa-365 have at least one active safety clause violation, and that tabletop policies with above 90% mean success have 18-28% unsafe episode rates. The authors also demonstrate that SafeVLA-Bench can be used to finetune π0.5\pi_{0.5} on LIBERO-Spatial to improve safety performance.

Critical strengths

The safety clauses that are proposed are reasonable and cover a wide range of physical safety concerns, including contact force, object displacement, tilt, joint-torque, and collisions. The authors also evaluate a large suite of policies and show that high success does not necessarily correlate with safety, motivating the need for their benchmark.

Critical weaknesses

My primary concern is that the paper’s contributions beyond their evaluation are not well supported. A set of the claimed contributions (post-hoc safety monitoring, STL robustness, and continuous severity scoring) have each appeared in prior works (table 1). Thus, the paper’s novelty rests largely on VSI as well as its demonstration that the benchmark can be applied practically (the post-training study), but neither feel convincingly demonstrated.

The contribution of the paper’s novel metric VSI is not quite clear. The paper claims that VSI scores differentiate from, e.g., SBU. However, SBU counts unsafe successes, in contrast to VSI averaging over all rollouts (notably including failures). The examples where VSI and SBU are not in agreement are in policies with lower success rates (OpenVLA-7B in table 3, π0\pi_0 in table 4). Therefore, it is likely that these differences are coming from failed rollouts. The paper also shows that including VSI during training (Advantage-conditioned SFT) does worse in all safety metrics than base π0.5\pi_{0.5}.

Further, while the authors were comprehensive in their evaluation of policies on the benchmark, they only evaluate π0.5\pi_{0.5} on LIBERO-spatial on the same set of tasks that it was trained on, using only different initial states, according to Appendix H. Even in this favorable setting, the average safety improvements (85% to 90.8%) seem modest. It would be beneficial to see safety improvements over a wide cast of policies and to evaluate on a held-out suite of tasks as the post-training study is the paper’s main evidence that the benchmark is practically useful.

Questions for authors

The following are clarification questions for the author.

  1. Does post-training improvement hold for other policies or when evaluating on held-out tasks/suites? Is the improvement over the base policy statistically significant?
  2. Are there settings where VSI is more useful as a training signal than binary safety labels?
  3. What do VSI averages look like when averaged over unsafe rollouts only? Does VSI still rank policies differently than safety?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.