- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RASBench, a system-level safety benchmark for embodied agents operating under abnormal functional-module conditions. The benchmark consists of 200 task instances across eight simulated scenes, 14 types of physical safety constraints, and controlled interventions targeting six functional interfaces, including instructions, visual observations, LiDAR, proprioceptive feedback, and control commands. The authors introduce paired clean and abnormal executions to evaluate task-performance degradation and execution safety. The paper addresses a meaningful and practically relevant problem. However, I have significant concerns regarding the statistical reliability of the evaluation, the limited diversity of effectively evaluated agents, and the extent to which the benchmark demonstrates new insights beyond existing safety and robustness evaluations.
Critical strengths
-
Important research problem. Evaluating embodied agents under abnormal operating conditions is relevant to real-world robotic deployment. The distinction between task completion and execution safety is particularly meaningful.
-
Systematic benchmark design. The benchmark covers multiple functional interfaces and introduces task-specific physical safety constraints. The paired evaluation protocol provides a useful way to examine changes in task performance and safety relative to normal operation.
-
Useful diagnostic perspective. The experiments demonstrate that task-performance retention and execution safety do not necessarily coincide. Reporting both absolute safety violations and intervention-induced changes provides complementary information.
Critical weaknesses
-
Insufficient statistical transparency and validation. Although the benchmark contains 200 tasks, the main results do not clearly report the number of registered interventions, successfully activated interventions, valid evaluation pairs, or clean-safe pairs for each interface. These quantities are essential for interpreting metrics such as the safety loss rate. Moreover, the paper lacks confidence intervals, repeated-run analysis, and intervention-severity studies. Since only successfully activated and valid interventions enter the paired analysis, the results may also be affected by selection bias. The current evidence is insufficient to assess the statistical stability and representativeness of the reported safety differences.
-
Limited diversity of effectively evaluated agents. The experiments primarily rely on two configurations of the same fine-tuned GR00T system, with and without a defense module. Although a π0.5 variant is also included, its clean task score is extremely low, and four of the six intervention interfaces cannot be evaluated because the required activation conditions are not reached. Consequently, the experiments do not sufficiently establish the applicability and discriminative ability of RASBench across different capable embodied agents.
-
Insufficient demonstration of the benchmark's unique diagnostic value. Existing benchmarks, such as RobustVLA, SafeVLA-Bench, and ForesightSafety-VLA, already investigate multimodal robustness and physical execution safety from complementary perspectives. While RASBench extends intervention coverage to proprioceptive and control interfaces, the paper does not sufficiently demonstrate what additional safety failure modes or generalizable insights these interfaces reveal. Furthermore, the results mainly report aggregate violation rates without detailed analysis of violation severity, intervention intensity, or the relationships between specific abnormalities and safety-constraint failures. The reproducibility and completeness of the planned benchmark release also require clarification.
Questions for authors
-
Can the authors provide the number of registered, activated, and valid interventions for each interface and system configuration, along with the clean-safe denominators and confidence intervals for the reported safety metrics? How sensitive are the results to intervention severity and activation conditions?
-
Can the authors evaluate additional embodied agents that achieve meaningful task performance under normal conditions? How does the benchmark distinguish systems with different safety and robustness capabilities when some agents cannot reach the intervention triggers?
-
What specific diagnostic insights does RASBench provide beyond existing multimodal robustness and execution-safety benchmarks? Can the authors provide a more detailed analysis linking intervention types to physical safety violations, and clarify whether all benchmark tasks, intervention configurations, safety thresholds, and evaluation scripts will be released?
Response to Reviewer TLem
Thank you for your careful review and valuable feedback.
RASBench is part of an embodied safety competition (https://tianchi.aliyun.com/competition/entrance/532507/information). At submission, the competition was at an early stage. We withheld task details to protect participants' privacy and competition fairness. Materials that do not affect the ongoing competition, including scenes, tasks, and our evaluated agents, are now public: https://anonymous.4open.science/r/RASbench-7EDB
Computational and other costs limited each episode to one execution per configuration. Single-run results may be affected by system randomness. Intervention registration information and execution configurations are public: https://anonymous.4open.science/r/RASbench-7EDB/benchmark
RASBench links injection points, system responses, task performance, and physical safety events within the same execution. This reveals both task progress and dangerous behavior. Benchmark information, execution configurations, and intervention parameters are public: https://anonymous.4open.science/r/RASbench-7EDB/benchmark
Safety criteria record whether violation conditions are triggered. Numerical thresholds and region definitions depend on each scene and task. Although collision severity can be described by force, different violation types are difficult to compare on a common scale. We therefore did not separately report violation severity. Configurations: https://anonymous.4open.science/r/RASbench-7EDB/benchmark/safety_constraints
RASBench examines how abnormalities in the diverse signals supporting robot operation affect tasks and safety. Severity has different physical or semantic meanings across interfaces: RGB changes and IMU deviations cannot simply be ranked on one scale. We report task and safety outcomes by abnormality type and configuration rather than use a unified cross-interface severity scale. Benchmark and agent information, configurations, intervention parameters, and safety thresholds are public: https://anonymous.4open.science/r/RASbench-7EDB
Our tasks require long-distance locomotion and object manipulation, which M20 Pro + Piper supports. Stationary arms cannot complete this sequence, and common wheeled dual-arm configurations we examined do not meet this task set's efficiency and traversability requirements in large scenes.
We found no publicly available, reproducible agent that could adequately operate this configuration. We built our agent and fine-tuned GR00T to establish the basic task capabilities needed for evaluation.
By design, RASBench supports more agents: scenes and tasks are not tied to a robot model. An agent can participate if it controls a robot meeting the scene's size, traversability, and mobile manipulation requirements and integrates the simulation and evaluation interfaces. M20 Pro + Piper is our chosen platform; tasks and safety criteria concern actual robot behavior. Long-horizon tasks in large scenes demand task execution capabilities and platform adaptation. Among the publicly available, reproducible agents examined, we found none that could be directly applied with good task performance.
Competition participants' agents have also been evaluated. Their results cannot currently be disclosed; we will consult participants after the competition before deciding on release.