- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors present CHARM, a benchmarking task suite for measuring capability-to-harm generalization. CHARM curates 50 scenarios in a simulation platform for evaluation. The benchmark is evaluated on 17 robot policies, spanning modular policies to end-to-end policies. This work demonstrates that policies that are more capable in performing the task also tend to be more harmful.
Critical strengths
- The paper finds and curates 50 benchmark tasks in OmniGibson. The paper evaluates 17 generalist robot policies spanning modular architectures, VLAs, and WAMs. So the findings hold across different types of policies.
- The paper shows a clear correlation between improved task success and higher harm scores. This can be useful for the community to consider a co-optimization of robot performance and safety.
Critical weaknesses
- The evaluation benchmark contains just 50 tasks on a simulation platform that already exists.
- The findings in this work are not very surprising. It is not very surprising that with greater capacity to execute tasks (however benign), the policies tend to follow these instructions without concern for safety. Indeed, these policies were trained to optimize for task success, rather than co-optimizing with safety. If the concern is on both task success and harmful outputs, why not also evaluate policies optimized for both or policies along with a runtime safety filter?
- Paper presentation:
- Some important details are left in the appendix. For instance, the definition of Hazard Progress Score (HPS) is central to this work, and it is hard to follow what q_c is and why there are two conditions in Equation 1.
- It is also unclear what B+, B-, H+, H- are.
Questions for authors
- Can you contextualize the contribution of evaluating on explicit harmful tasks? I would like to better understand this contribution in comparison with prior work on jailbreaking or red teaming robot models. For instance, Jailbreaking LLM-Controlled Robots, or RoboPAIR (https://arxiv.org/abs/2410.13691) shows that LLM-controlled robots can be susceptible to harmful language instructions.
- Can you make it clearer in the main text how the HPS parameters are defined? For instance, it is not clear what q_c is and why there are two conditions in Equation 1. Given how central this is to the paper, I would want to see these definitions and how they are computed in the main text rather than in the Appendix.
- How do you select the tasks? I see that “expert curation” was involved in filtering the tasks. Could you please clarify?
- I don’t see B+, B-, H+, and H- defined anywhere in the paper. Can you please define these?
- Have you considered any sort of safety filter or safety-aligned policy? Why only present results on policies trained to improve task success and evaluate safety at the same time?
10 comments
Response to Reviewer Ho71 (Part2)
W2 Reaffirm the motivation behind our work: The findings in this work are not very surprising. It is not very surprising that with greater capacity to execute tasks (however benign), the policies tend to follow these instructions without concern for safety. Indeed, these policies were trained to optimize for task success, rather than co-optimizing with safety. If the concern is on both task success and harmful outputs, why not also evaluate policies optimized for both or policies along with a runtime safety filter?
We thank the reviewer for this important comment. We agree that most existing generalist robot policies are primarily optimized for task success rather than semantic safety. However, we would like to clarify two important points: (1) capability-to-harm generalization is not an automatic consequence of task-success optimization in current robot foundation models; and (2) existing robot safety mechanisms do not yet comprehensively address the goal-level semantic hazards evaluated by CHARM. We provide additional experiments and discussion to substantiate both points.
(1) Capability-to-harm generalization is non-trivial for current robot foundation models. Unlike LLMs/VLMs, which exhibit relatively strong generalization across language and visual tasks, current generalist robot policies, particularly VLAs/WAMs, still face substantial challenges in generalizing to unseen manipulation tasks, objects, and scenes. Robotic manipulation requires not only semantic understanding but also accurate visual grounding and successful closed-loop physical execution. Consequently, strong performance on familiar benign tasks does not necessarily imply the ability to execute novel hazardous objectives, even when they involve similar manipulation primitives.
To further substantiate this point, we conducted additional experiments using VLA checkpoints fine-tuned on LIBERO, LIBERO-Safety, and VLA-Arena, evaluating their generalization to newly constructed benign–hazardous task pairs.
| Model / Checkpoint | Original Benign TSR | New Benign TSR | Hazardous TSR |
|---|---|---|---|
| π₀.₅ | 98.8% | 0.0% | 0.4% |
| π₀-FAST | 96.4% | 0.0% | 0.0% |
| GR00T N1.7 | 86.8% | 0.0% | 0.0% |
| MolmoACT2 | 98.8% | 0.0% | 0.0% |
| π₀.₅ (LIBERO-Safety fine-tuned) | 52.0% | 14.8% | 0.4% |
| π₀.₅ (VLA-Arena fine-tuned) | 56.0% | 4.4% | 0.4% |
Despite achieving 86.8%–98.8% TSR on their original benign tasks, the first four models fail on nearly all newly constructed benign and hazardous tasks. Importantly, their failure on hazardous tasks cannot be attributed to safety awareness, as they also fail on the matched benign tasks. Even the LIBERO-Safety- and VLA-Arena-fine-tuned checkpoints exhibit limited generalization to new benign tasks. In contrast, several generalist policies evaluated in CHARM successfully execute novel hazardous objectives without CHARM-specific fine-tuning. These contrasting results demonstrate that capability-to-harm generalization is neither automatic nor universal, but a non-trivial phenomenon that cannot be inferred from conventional task-success performance alone.
(2) Generalizable semantic safety alignment remains largely underexplored in current robot foundation models. We agree that jointly optimizing task success and safety is an important research direction. However, much of the existing embodied safety research focuses on physical safety constraints, such as collision avoidance and safe action execution. For example, SafeVLA incorporates predefined safety constraints through constrained reinforcement learning, while AEGIS employs Control Barrier Functions (CBFs) for runtime action correction. Semantic safety alignment that enables generalist VLAs/WAMs to recognize and avoid diverse unseen hazardous task goals remains insufficiently established. Importantly, a robot may execute physically feasible, collision-free actions while still accomplishing a semantically hazardous objective. Therefore, physical safety mechanisms alone cannot be assumed to ensure goal-level semantic safety.
Response to Reviewer Ho71 (Part3)
Our additional runtime safety evaluation further illustrates this distinction. To directly address the reviewer's suggestion, we integrated AEGIS with Cosmos3-Nano-Policy on CHARM. While AEGIS reduces physical disturbances (e.g., CDR: 35.3% → 22.0% in benign cluttered scenes), hazardous progress remains nearly unchanged (HPS: 0.353 → 0.348). Moreover, the reduction in hazardous TSR (20.0% → 16.0%) is accompanied by a decrease in benign TSR (20.7% → 18.0%), suggesting that physical action filtering does not necessarily provide selective protection against semantically hazardous goals. We also conducted a preliminary single-task safety post-training experiment using CHARM-annotated data, exploring whether hazardous execution can be suppressed while retaining benign-task performance. Detailed experimental results and discussion of both safety interventions are provided in our response to Q5.
In summary, CHARM does not merely demonstrate that task-oriented policies may follow harmful instructions. Instead, it systematically investigates whether and to what extent pre-existing manipulation capabilities generalize to semantically hazardous objectives, a phenomenon that is neither inevitable nor adequately captured by conventional task-success evaluations. Our additional safety-filter experiments further suggest that improvements in physical safety do not necessarily address goal-level semantic risks, highlighting the need for complementary semantic safety evaluation.
W3 & Q2 & Q3 Paper presentation: Can you make it clearer in the main text how the HPS parameters are defined? For instance, it is not clear what q_c is and why there are two conditions in Equation 1. Given how central this is to the paper, I would want to see these definitions and how they are computed in the main text rather than in the Appendix.
I don’t see B+, B-, H+, and H- defined anywhere in the paper. Can you please define these?
We thank the reviewer for identifying these ambiguities. We agree that the definition and computation of HPS, as a central metric of CHARM, should be clearly explained in the main text. We have revised Section 3.3 to clarify the physical meaning of its parameters, the rationale behind the two branches in Eq. (1), and their concrete instantiations. We have also explicitly defined the task-pair outcome labels used in Fig. 4(b).
(1) Definition of HPS parameters and the two branches in Eq. (1). Here, (q_c(t)) denotes a hazard-relevant physical state variable instantiated for each safety constitution (c), while (q_{c,0}) and (q_{c,H}) denote its initial value and hazardous boundary, respectively. The specific physical quantity depends on the underlying hazard mechanism. For example:
- Proximity-based hazards: For a sharp tool approaching a human hand, (q_c(t)) is instantiated as the distance between their axis-aligned bounding boxes (AABBs). A decreasing distance ((q_c(t) \downarrow 0)) indicates increasing progress toward hazardous contact.
- Energy-based hazards: For impact or breakage caused by dropping objects, (q_c(t)) can be instantiated as relative impact energy. An increasing energy ((q_c(t) \uparrow E_H), where (E_H = 0.30) J) indicates increasing progress toward the hazardous threshold.
Therefore, the two branches in Eq. (1) account for the opposite directions in which different physical quantities approach their hazardous boundaries. Both are normalized to ([0,1]), ensuring that higher values consistently indicate greater progress toward a hazardous state, regardless of the underlying physical quantity.
(2) Definition of B+, B−, H+, and H− in Fig. 4(b). These labels characterize the outcomes of matched benign–hazardous task pairs. Specifically, B+/B− indicate successful/unsuccessful execution of the benign task, while H+/H− indicate successful/unsuccessful execution of its hazardous counterpart. Their combinations represent four possible outcomes:
- B+H+: Both tasks succeed, indicating that benign manipulation capability also supports hazardous execution.
- B+H−: Only the benign task succeeds.
- B−H+: Only the hazardous task succeeds.
- B−H−: Neither task succeeds.
This decomposition allows us to analyze capability-to-harm generalization at the matched-task level, beyond aggregate task success rates. We have added these definitions to the caption of Fig. 4(b) to improve clarity.
Response to Reviewer Ho71 (Part1)
We greatly appreciate the reviewer's insightful comments and positive assessment, particularly your recognition of the comprehensive cross-architecture evaluation and the importance of revealing how improved task capabilities can be accompanied by increased hazardous outcomes. We are encouraged that you find these observations valuable for motivating future research on the co-optimization of robotic performance and safety. We have carefully addressed your concerns through additional experiments and detailed analyses, as presented below. Besides, we highlighted the corresponding revisions in purple in the revised manuscript. We further conduct real-world evaluation, and videos of the real-world experiments are available on our anonymized project page
We thank the reviewer for raising this concern and would like to clarify the scale and construction of CHARM.
First, the benchmark scale is comparable to existing embodied AI benchmarks, including LIBERO-Safety (75 tasks), RoboTwin 2.0 (50 tasks), and the safety subset of VLA-Arena (75 tasks). More importantly, CHARM comprises 50 matched benign–hazardous task pairs, rather than 50 isolated tasks. By further considering two instruction regimes (explicit/implicit) and two scene complexities (clean/cluttered), CHARM provides 400 distinct evaluation conditions (50 × 2 × 2 × 2). These controlled settings enable a systematic investigation of how manipulation capabilities generalize toward hazardous objectives across different instruction modalities and scene complexities.
Second, CHARM introduces an extensible framework for systematically generating and validating executable semantic hazard scenarios. Although CHARM builds upon OmniGibson, we introduce a constitution-guided Generator–Grounder–Critic pipeline that systematically constructs and validates executable semantic hazard scenarios. This framework supports diverse hazard categories and manipulation primitives while enabling controlled benign–hazardous comparisons. Therefore, the curated 50 task pairs represent a validated benchmark collection built upon an extensible safety-oriented task-generation framework, rather than the limit of scenarios that the framework can potentially support.