- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents SafeVLA-Bench, a post-hoc safety evaluation framework designed to assess safety violations in vision-language-action (VLA) policies beyond binary task success. The framework augments existing LIBERO and RoboCasa-365 evaluations with trajectory-level safety monitoring while preserving their original tasks, success predicates, and rollout protocols. It defines eight task-aware safety constraint families spanning scene interaction, object handling, and robot execution, using an applicability registry and quantitative Signal Temporal Logic (STL) robustness. Alongside native success rate (SR), the benchmark reports Safety and success-but-unsafe rate (SBU), and introduces the Violation Severity Index (VSI), which measures the average worst normalized violation depth across episodes.
The authors evaluate 24 distinct policies across 27 policy–benchmark entries, demonstrating that high task success does not necessarily imply safe execution. The paper further investigates discrepancies in policy rankings, task-structural safety violations, threshold sensitivity, and temporal violation patterns. Finally, a post-training case study on LIBERO-Spatial constructs same-initial-state preference pairs using SafeVLA-Bench safety labels and applies Diffusion-DPO to π0.5, improving its measured safety performance.
Critical strengths
-
A practical and well-designed post-hoc safety evaluation framework that preserves comparability with existing benchmarks. Unlike safety benchmarks that require constructing new tasks or benchmark-specific policy fine-tuning, SafeVLA-Bench integrates safety evaluation into existing VLA benchmarks without modifying their original tasks, success predicates, or rollout protocols.
-
Additional hardware validation and safety post-training experiments strengthen the practical relevance of the benchmark. Beyond evaluating VLA safety in simulation, the paper includes a small-scale real-world robot experiment and a safety-oriented post-training case study, making the experimental evaluation more comprehensive. In particular, the post-training experiments demonstrate that SafeVLA-Bench can serve not only as a safety evaluation tool but also as a source of safety preference labels for further VLA optimization.
Critical weaknesses
- Limited methodological novelty and insufficient evidence for the incremental utility of VSI. SafeVLA-Bench shares substantial similarities with existing safety evaluation benchmarks, particularly SafeManip, which also performs task-aware temporal safety monitoring on existing RoboCasa-365 tasks using LTLf specifications. Although SafeVLA-Bench extends this direction by introducing quantitative STL robustness and the Violation Severity Index (VSI), the paper lacks a direct comparison or discussion with SafeManip. Moreover, while the authors analyze policy-ranking discrepancies and violation severity distributions, the paper provides limited insights into the underlying failure modes that explain VSI differences across policies and how such insights can guide safety improvements. This concern is further reflected in the post-training experiments: the successful DPO approach relies primarily on binary safe/unsafe preference labels rather than directly leveraging VSI, whereas the VSI-informed Advantage-conditioned SFT performs worse than the base policy in terms of safety. These results leave the incremental contribution of VSI to downstream safety optimization insufficiently demonstrated.
- Limited effective safety coverage despite the eight defined constraint families. Although SafeVLA-Bench defines eight safety constraint families, Tables 7–9 reveal that several constraints are either structurally unavailable or rarely violated in the evaluated benchmarks. For instance, grasp-slip and joint-torque violations are consistently absent. Consequently, the reported safety violations appear to be dominated by a relatively small subset of constraints, particularly contact force and bystander displacement in LIBERO, and force-related constraints and transport tilt in RoboCasa-365. This raises concerns about whether the benchmark provides sufficiently comprehensive safety evaluation and whether its reported safety rankings are disproportionately influenced by a few simulator- or task-specific signals.
Questions for authors
- Novelty relative to SafeManip. Could the authors clarify the essential methodological contributions of SafeVLA-Bench beyond SafeManip.
- Validity and coverage of safety constraints. Given that several safety constraints are rarely or never violated (e.g., grasp slip and joint torque), while a few others dominate the reported violations (Tables 7–9), could the authors further justify the validity and applicability of these constraints and analyze their individual contributions to the overall safety evaluation?
- Analysis of High-SI Tasks. Could the authors provide further analysis of why
PickPlaceDrawerToCounterhas an SI of 100%? Is this due to inherent task requirements, limitations of current VLA policies in executing the task safely, or potentially inappropriate safety constraints? - Evaluation of force-aware VLAs. Could the authors extend the evaluation to force-aware VLA models and examine whether the existing force-related safety constraints adequately capture their safety performance?
- Task-aware Safety Constraint Applicability. How are task templates and semantic tags assigned to individual tasks? Is this process manually annotated or automated, and how is the correctness of the resulting safety-constraint applicability validated?
Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts in addressing my concerns and will maintain my positive rating.