acceptodds
Under review as a conference paper at ICLR 2027

RASBench: A Safety Benchmark for Embodied Agents

Abstract

General-purpose embodied agents are designed to understand user intent and com- plete tasks. These agents rely on functional modules to receive user instructions, observe the environment, sense their proprioceptive state, and execute physical actions. However, in practice, these modules may operate abnormally during task execution, causing robots to deviate from their intended tasks or behave unsafely. Existing evaluations have not adequately characterized how such abnormalities affect agent operation. We introduce RASBench (Robotic Agent Safety Bench- mark), a system-level benchmark for evaluating embodied agents under abnormal operating conditions across diverse application settings, including warehouses, kitchens, and retail environments. RASBench comprises 200 task instances across eight simulation scenes and defines 14 types of safety constraints on robots’ phys- ical actions. It applies controlled interventions to these functional modules to induce abnormal operating conditions and uses execution traces from paired sim- ulation runs with and without intervention to quantify changes in task performance and execution safety. With RASBench, we evaluate several embodied agents on a simulated quadrupedal mobile manipulation platform. Our experiments show that some agents exhibit severe degradation in task performance and execution safety under the tested interventions. These findings highlight the need to account for ex- ecution safety under complex operating conditions when designing and evaluating embodied agents.

Primary area:
datasets and benchmarks
Keywords:
Embodied AI, Robot Safety, Safety Evaluation, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TLemOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RASBench, a system-level safety benchmark for embodied agents operating under abnormal functional-module conditions. The benchmark consists of 200 task instances across eight simulated scenes, 14 types of physical safety constraints, and controlled interventions targeting six functional interfaces, including instructions, visual observations, LiDAR, proprioceptive feedback, and control commands. The authors introduce paired clean and abnormal executions to evaluate task-performance degradation and execution safety. The paper addresses a meaningful and practically relevant problem. However, I have significant concerns regarding the statistical reliability of the evaluation, the limited diversity of effectively evaluated agents, and the extent to which the benchmark demonstrates new insights beyond existing safety and robustness evaluations.

Critical strengths
  1. Important research problem. Evaluating embodied agents under abnormal operating conditions is relevant to real-world robotic deployment. The distinction between task completion and execution safety is particularly meaningful.

  2. Systematic benchmark design. The benchmark covers multiple functional interfaces and introduces task-specific physical safety constraints. The paired evaluation protocol provides a useful way to examine changes in task performance and safety relative to normal operation.

  3. Useful diagnostic perspective. The experiments demonstrate that task-performance retention and execution safety do not necessarily coincide. Reporting both absolute safety violations and intervention-induced changes provides complementary information.

Critical weaknesses
  1. Insufficient statistical transparency and validation. Although the benchmark contains 200 tasks, the main results do not clearly report the number of registered interventions, successfully activated interventions, valid evaluation pairs, or clean-safe pairs for each interface. These quantities are essential for interpreting metrics such as the safety loss rate. Moreover, the paper lacks confidence intervals, repeated-run analysis, and intervention-severity studies. Since only successfully activated and valid interventions enter the paired analysis, the results may also be affected by selection bias. The current evidence is insufficient to assess the statistical stability and representativeness of the reported safety differences.

  2. Limited diversity of effectively evaluated agents. The experiments primarily rely on two configurations of the same fine-tuned GR00T system, with and without a defense module. Although a π0.5 variant is also included, its clean task score is extremely low, and four of the six intervention interfaces cannot be evaluated because the required activation conditions are not reached. Consequently, the experiments do not sufficiently establish the applicability and discriminative ability of RASBench across different capable embodied agents.

  3. Insufficient demonstration of the benchmark's unique diagnostic value. Existing benchmarks, such as RobustVLA, SafeVLA-Bench, and ForesightSafety-VLA, already investigate multimodal robustness and physical execution safety from complementary perspectives. While RASBench extends intervention coverage to proprioceptive and control interfaces, the paper does not sufficiently demonstrate what additional safety failure modes or generalizable insights these interfaces reveal. Furthermore, the results mainly report aggregate violation rates without detailed analysis of violation severity, intervention intensity, or the relationships between specific abnormalities and safety-constraint failures. The reproducibility and completeness of the planned benchmark release also require clarification.

Questions for authors
  1. Can the authors provide the number of registered, activated, and valid interventions for each interface and system configuration, along with the clean-safe denominators and confidence intervals for the reported safety metrics? How sensitive are the results to intervention severity and activation conditions?

  2. Can the authors evaluate additional embodied agents that achieve meaningful task performance under normal conditions? How does the benchmark distinguish systems with different safety and robustness capabilities when some agents cannot reach the intervention triggers?

  3. What specific diagnostic insights does RASBench provide beyond existing multimodal robustness and execution-safety benchmarks? Can the authors provide a more detailed analysis linking intervention types to physical safety violations, and clarify whether all benchmark tasks, intervention configurations, safety thresholds, and evaluation scripts will be released?

1 comment
AuthorsOpenReview ↗

Response to Reviewer TLem

Thank you for your careful review and valuable feedback.

RASBench is part of an embodied safety competition (https://tianchi.aliyun.com/competition/entrance/532507/information). At submission, the competition was at an early stage. We withheld task details to protect participants' privacy and competition fairness. Materials that do not affect the ongoing competition, including scenes, tasks, and our evaluated agents, are now public: https://anonymous.4open.science/r/RASbench-7EDB

W1, Q1: Valid pairs and statistical reliability

Computational and other costs limited each episode to one execution per configuration. Single-run results may be affected by system randomness. Intervention registration information and execution configurations are public: https://anonymous.4open.science/r/RASbench-7EDB/benchmark

W3, Q3: Diagnostic value and evaluation materials

RASBench links injection points, system responses, task performance, and physical safety events within the same execution. This reveals both task progress and dangerous behavior. Benchmark information, execution configurations, and intervention parameters are public: https://anonymous.4open.science/r/RASbench-7EDB/benchmark

W3, Q3: Safety violation severity

Safety criteria record whether violation conditions are triggered. Numerical thresholds and region definitions depend on each scene and task. Although collision severity can be described by force, different violation types are difficult to compare on a common scale. We therefore did not separately report violation severity. Configurations: https://anonymous.4open.science/r/RASbench-7EDB/benchmark/safety_constraints

W1, W3, Q1: Intervention severity

RASBench examines how abnormalities in the diverse signals supporting robot operation affect tasks and safety. Severity has different physical or semantic meanings across interfaces: RGB changes and IMU deviations cannot simply be ranked on one scale. We report task and safety outcomes by abnormality type and configuration rather than use a unified cross-interface severity scale. Benchmark and agent information, configurations, intervention parameters, and safety thresholds are public: https://anonymous.4open.science/r/RASbench-7EDB

W2, Q2: Agents, platform, and applicability

Our tasks require long-distance locomotion and object manipulation, which M20 Pro + Piper supports. Stationary arms cannot complete this sequence, and common wheeled dual-arm configurations we examined do not meet this task set's efficiency and traversability requirements in large scenes.

We found no publicly available, reproducible agent that could adequately operate this configuration. We built our agent and fine-tuned GR00T to establish the basic task capabilities needed for evaluation.

By design, RASBench supports more agents: scenes and tasks are not tied to a robot model. An agent can participate if it controls a robot meeting the scene's size, traversability, and mobile manipulation requirements and integrates the simulation and evaluation interfaces. M20 Pro + Piper is our chosen platform; tasks and safety criteria concern actual robot behavior. Long-horizon tasks in large scenes demand task execution capabilities and platform adaptation. Among the publicly available, reproducible agents examined, we found none that could be directly applied with good task performance.

Competition participants' agents have also been evaluated. Their results cannot currently be disclosed; we will consult participants after the competition before deciding on release.

Reviewer_uLYnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RASBench, a benchmark for evaluating embodied agents' task performance and execution safety under functional module abnormalities. The benchmark has 200 tasks across 8 Isaac Sim scenes, with 14 categories of physical safety constraints. It evaluates the effects of interventions on six system interfaces, including task instructions, RGB, proprioceptive feedback, and control commands. The experiment is conducted through paired executions under normal and abnormal operating conditions. The authors evaluate three agent configurations built around GR00T and π0.5.

Critical strengths

System-level safety evaluation

The benchmark extends robustness and safety evaluation beyond language and visual inputs to lower-level robotic interfaces, including LiDAR, joint-state feedback, IMU, and control commands. This broader coverage is potentially valuable for understanding how failures in different components of an embodied system affect physical execution safety.

Critical weaknesses

Lack of clear task coverage

Although the authors claim 200 task instances across 8 scenes, the paper provides very limited information about the actual task instances. Appendix A reports only the task distribution across environments and a few illustrative examples (picking up and placing a tea box, lifting a can, and pressing a button corresponding to a red light). The exact task setup, task instructions, completion criteria, and task-specific safety constraints are not provided. That being said, it is extremely difficult to assess the diversity, complexity, and coverage of the proposed benchmark or independently reproduce its evaluation. Furthermore, the authors did not provide any implementation detail in the paper and the appendix (the anonymous repo only contains a one-sentence README).

The usefulness and representativeness of the injected abnormalities are unvalidated

First, the paper fails to explain how the functional abnormalities injection are implemented. The only reference in the paper is Table 8 which briefly describes the different families of abnormalities. Without the implementation details, it is quite hard to validate the claims made by the paper in its experiment section. Furthermore, the paper does not establish whether its selected synthetic interventions and their severity levels meaningfully represent the reality.

Limited evidence for generalization

Although the benchmark is described as covering diverse application settings, the experiments involve:

  • One shared agent architecture, with different configurations.
  • 8 scenes (unclear how diverse the tasks are at all)
  • No real world experiment This brings out the more important concern - a benchmark intended for evaluating general embodied agents should demonstrate that its tasks, intervention mechanisms, and safety monitors can meaningfully evaluate agents with different architectures. But only the authors' own system are tested in the current version of the paper.

Limited novelty compared to existing benchmarks

And finally, potentially also the biggest issue with the current submission is regarding its novalt. The novelty of the proposed benchmark is somewhat limited compared to existing works. Prior benchmarks such as RobustVLA and NavTrust have already investigated the robustness of embodied agents under multimodal perturbations, while SafeVLA-Bench and ManiGuard have explored execution-level safety evaluation extensively. The current submission primarily extends these existing evaluation settings by introducing abnormalities in additional system interfaces, including LiDAR, joint-state feedback, IMU, and control commands. While such extensions are potentially useful, the underlying methodology of injecting perturbations and evaluating task performance and safety is not fundamentally new. More importantly, the paper does not provide sufficient experimental analysis to demonstrate what new insights these additional intervention categories offer beyond existing benchmarks. As a result, the significance of the incremental contribution remains unclear.

Questions for authors
  1. In the reproducibility statement it is said that the baseline implementation will be released through the project repository specified in Appendix F. Yet, so far the repo only contains a one line README. When is this going to be released?
  2. For the motion/manipulation-command condition without defense, Table 4 reports a 100% full-run violation rate, yet all three paired safety-change metrics are zero. Could the authors provide the a breakdown of violations occurring before and after the intervention-reference time? Without these details, it is difficult to interpret whether the zero paired changes reflect the same or different unsafe behavior.
  3. How is Gr00t finetuned in the experiment? What was the data used for finetuning? Why was π0.5\pi_{0.5} not finetuned? Without knowing how Gr00t was fine-tuned, it is difficult to reproduce the reported performance or evaluate whether the learned policies generalize beyond the benchmark tasks.
2 comments
AuthorsOpenReview ↗

Response to Reviewer uLYn

Thank you for your careful review and valuable feedback.

RASBench encourages agents to account for environmental perception, robot state, and execution beyond a VLA model's prescribed inputs. LiDAR, IMU, other sensors, and even battery information can affect task performance and physical safety.

RASBench is part of an embodied safety competition (https://tianchi.aliyun.com/competition/entrance/532507/information). At submission, the competition was at an early stage. We withheld task details to protect participants' privacy and competition fairness. Materials that do not affect the ongoing competition, including scenes, tasks, and our evaluated agents, are now public: https://anonymous.4open.science/r/RASbench-7EDB

W4: Contribution and practical motivation

RASBench combines functional-interface abnormalities, task-level safety constraints, and paired executions to examine effects throughout task execution. LiDAR, IMU, joint feedback, and control interfaces extend coverage of operational failures.

During pre-competition testing at the 2026 World Humanoid Robot Games, we observed:

  • Erroneous IMU gravity feedback prevented a robot from recognizing that it had fallen; it continued full-speed sprinting motions.
  • A visual recognition error caused a robot to mistake a circular advertising logo for a ball and attempt to kick it.
  • LiDAR failure interrupted point-cloud output, causing obstacle-avoidance failures, collisions, and falls.
  • Erroneous joint feedback caused an arm to repeatedly release its grasp and drop an object.

Privacy prevents disclosure of case details or images. These failures show harmful consequences of abnormalities that the agents failed to handle, even by stopping. Existing benchmarks give these abnormalities insufficient attention. RASBench covers a wider range through controlled interventions: https://anonymous.4open.science/r/RASbench-7EDB/benchmark/interventions

Our aim is to encourage comprehensive control of robots beyond a single model's input scope. The included interfaces turn this system-level concern into executable task and safety evaluations with observable outcomes.

W3: Platform, agents, and experiments

Large scenes such as factories require long-distance travel followed by manipulation. Stationary arms cannot provide locomotion, and common wheeled dual-arm configurations do not meet this task set's efficiency and traversability requirements. M20 Pro + Piper supports both locomotion and manipulation.

We found no publicly available, reproducible agent that could adequately operate this configuration, so we built one and fine-tuned GR00T to establish basic task capabilities.

Competition participants' agents have also been evaluated. Their results cannot currently be disclosed; we will consult participants after the competition before deciding on release.

Simulation controls initial states and interventions for paired clean/intervention analysis. We have corresponding physical scenes and real robots, but have not conducted physical experiments because abnormal behavior could endanger personnel. Agent details: https://anonymous.4open.science/r/RASbench-7EDB/our_agent

W2: Abnormalities and robot operation

Distorted sensory feedback, abnormal state feedback, and altered control commands can change a robot's assessment of its environment and state, affecting actions and task outcomes. Types and configurations: https://anonymous.4open.science/r/RASbench-7EDB/benchmark/interventions

Q1, Q2: Materials and execution configurations

RASBench details and execution configurations are public: https://anonymous.4open.science/r/RASbench-7EDB

Q3: Model fine-tuning

Our M20 Pro + Piper platform is custom-built; existing models were not pretrained on this robot type. Without fine-tuning, π0.5 and GR00T lack basic task capabilities on it. Fine-tuning enables robot operation but is not RASBench's focus. Some GR00T N1.7 fine-tuning information and a Docker image containing our fine-tuned model are public: https://anonymous.4open.science/r/RASbench-7EDB/our_agent

Our fine-tuning data can be obtained within this image:

docker pull ccic-dmp-eval-images-registry.cn-hangzhou.cr.aliyuncs.com/cieval/embodied-competition-runner:260915_baseline_with_datasets

Echoing what reviewer NxcE mentioned regarding the link to external websites, and the updated anonymous repo link still does not seem to contain a comprehensible codebase for understanding and reproducing. There does not seem to be much task description in the listed repo as well. Given that, my main concerns in the original review have not been addressed and I will maintain my score at this time.

Reviewer_NxcEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RASBench, a benchmark evaluating embodied-agent task performance and physical safety under abnormalities affecting user instructions, observation of the environment, proprioception state, and action control.

The benchmark consists of 200 tasks across eight Isaac Sim scenes, with 14 safety-constraint types and interventions applied through the modeled functional interfaces. It utilizes a paired clean/intervention trace protocol to help distinguish task-performance degradation and infer changes in physical safety.

Experiments evaluate three system configurations on a simulated mobile manipulation platform and illustrate in high-level the benchmark goals.

Critical strengths

The paper is motivated by an important gap in current benchmarks: successful task completion is not evidence of safe physical execution. Moreover, it tries to identify how different abnormalities affect the performance of a given system.

The authors also observe that task progress and safety can favour different systems. Defensive behaviour may reduce collisions or unsafe actions while decreasing task completion, and the benchmark explicitly looks into this trade-off.

The library of physical safety constraints and the intervention specifications could be a potentially useful foundation for standardizing system-level evaluation.

Critical weaknesses

W1. The empirical evaluation is too narrow for the proposed scope

The benchmark is intended for general embodied agents, but the evaluation is conducted on just one simulated robot platform and one agent architecture (even if in three configurations). Two configurations use the same fine-tuned GR00T model and the third uses an unadapted π0.5 variant that achieves very low scores even under clean conditions. This setup doesn't properly support the claims of the benchmark. The third configuration contributes very little information about robustness as it generally cannot complete underlying tasks or reach intervention triggers. The authors do acknowledge this limitation, but it substantially weakens the evidence for the benchmark's capabilities and its general applicability.

W2. Statistical reliability is unclear

The paper defines pair eligibility and its metrics as a key contribution, but the main results assume one run is fully deterministic and can be used as to represent correct behaviour and do not report the number of eligible pairs for each condition. Also, Table 4 contains many percentages such as 33.33%, 50%, 66.67%, and 75%, suggesting that comparisons may involve small subsets.

W3. The impact of the defense mechanism is not sufficiently explained and no characterization of violation severity/magnitude or agent recovery behaviour

The defended and undefended configurations perform differently, but the experiments cannot provide insights into why. For example, lower safety violation rates could result from rejecting dangerous actions, early termination, improved observation validation, or avoiding task execution altogether. The benchmark also does not capture the severity or circumstances of unsafe behavior. For example, two agents may each collide only once, but the collisions could differ in force or physical consequences. And lastly, the benchmark does not account for an agent taking recovery actions to still accomplish tasks or mitigate/fix safety issues.

W4. Lack of details of benchmark cases and no release information

The paper doesn't provide any samples of the benchmark cases for review and even the linked supplemental material in Appendix F is empty. It would also be important do include details on the release of the benchmarks and its license.

Questions for authors
  • In what form would be the benchmark be released and under which license?

  • Why not report zero-shot Gr00t results as a better baseline than pi0.5? Is there any pi0.5-specific insight?

  • How many clean/intervention pairs contribute to each metric in Table 4?

  • How much of the apparent safety improvement from the defense module can be attributed to stopping early or avoiding actions?

  • How does task-performance degradation correlate with intervention severity?

  • Do you think the conclusions generalize to agents with substantially different navigation, manipulation, and recovery architectures? How?

1 comment
AuthorsOpenReview ↗

Response to Reviewer NxcE

Thank you for your careful review and valuable feedback.

RASBench is part of an embodied safety competition (https://tianchi.aliyun.com/competition/entrance/532507/information). At submission, the competition was at an early stage. We withheld task details to protect participants' privacy and competition fairness. Materials that do not affect the ongoing competition, including scenes, tasks, and our evaluated agents, are now public: https://anonymous.4open.science/r/RASbench-7EDB

W1, Q6: Evaluated systems and applicability

RASBench requires long-distance locomotion and object manipulation. Stationary arms cannot complete this sequence; wheeled dual-arm and bipedal robots do not meet this task set's efficiency and traversability requirements in large scenes. We therefore use an M20 Pro quadruped base with a Piper arm.

We found no publicly available, reproducible agent that could adequately operate this configuration. We built an agent and fine-tuned GR00T N1.7 to establish basic task capabilities for safety evaluation, with results we can disclose.

Our scenes and tasks are not tied to a robot model. Agents with different navigation, manipulation, and recovery architectures can participate if their robots meet the scene's size, traversability, and manipulation requirements and integrate the simulation and evaluation interfaces. Tasks and safety criteria concern actual robot behavior.

Intervention applicability depends on the signals and control pathways a system uses. If neither the agent nor its underlying navigation/control modules use a signal such as LiDAR or IMU feedback, its interventions are marked inapplicable, with reasons recorded. Clean and other applicable evaluations remain available. Recovery from interventions affecting used interfaces is reflected in task progress, scores, and safety events. Applicability, intervention validity, and task/safety outcomes help explain architectural differences.

Competition participants' agents have also been evaluated. Their results cannot currently be disclosed; we will consult participants after the competition before deciding on release.

Benchmark and agent details: https://anonymous.4open.science/r/RASbench-7EDB/benchmark https://anonymous.4open.science/r/RASbench-7EDB/our_agent

Q2: GR00T and π0.5 baseline selection

We evaluate system task performance and physical safety under interface abnormalities; the models are components of those systems. In preliminary small-scale fine-tuning evaluations, GR00T performed better than π0.5, motivating our choice of GR00T N1.7: https://anonymous.4open.science/r/RASbench-7EDB/our_agent/experiments

Without fine-tuning, both models perform similarly on our platform and lack basic task capabilities. Fine-tuning enables robot operation; measuring gains from fine-tuning is not our focus. We chose π0.5 as a zero-shot reference with a different model architecture from GR00T.

Q5: Intervention severity and task performance

Severity has different physical or semantic meanings across interfaces: RGB changes and IMU deviations cannot simply be ranked on one scale. We report task and safety outcomes by abnormality type and configuration rather than use a unified cross-interface severity scale. Parameters: https://anonymous.4open.science/r/RASbench-7EDB/benchmark

W2, Q3: Valid pairs and statistical reliability

Tasks and execution configurations are public: https://anonymous.4open.science/r/RASbench-7EDB/benchmark

Computational and other constraints limited each episode to one execution per configuration. Single-run results may be affected by system randomness.

W3, Q4: Defense and early termination

Agent architecture, implementation, and execution behavior are public: https://anonymous.4open.science/r/RASbench-7EDB/our_agent

The defense module can terminate execution early, for example after a retry limit is exceeded. We evaluate its overall effect through task and safety outcomes; defense-triggered early termination has not been separately quantified.

W4: Benchmark release

Benchmark information is public: https://anonymous.4open.science/r/RASbench-7EDB/benchmark

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.