- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In the paper, the authors propose a new benchmark called SafeManip, which tries to evaluate the safety of robotic manipulation for more than just task completion. The main motivation is that a robot might complete a task successfully but still will perform an unsafe actions during execution, and the existing task success metrics may not capture since they only care about task completion. To address this, the authors introduce a set of temporal safety rules using Linear Temporal Logic (LTLf), that includes eight different safety categories, such as collisions, grasp stability, contamination, and object release. Then they further propose metrics that measure the safety violations relative to the number of opportunities where a safety rule becomes relevant. The benchmark is evaluated on several robot foundation models and the main findings are that task success does not necessarily mean safe execution, different models have different types of safety failures, and longer tasks can lead to more safety violations.
Critical strengths
- The paper addresses a relevant problem in robotic manipulation safety. I also appreciate that the authors evaluate several existing robotic foundation models rather than limiting the study to a single model.
Critical weaknesses
-
I think the novelty of the paper is quite limited, becase there are several benchmarks already exist for evaluating safety in robotic manipulation around the ideas of temploral logic and trajectory level planning. I am not convinced that the paper clearly establishes what is different in this benchmark compared to the existing ones. I would have liked to see a more direct comparison showing what safety failures SafeManip can identify that these benchmarks cannot.
-
I feel the central claim of the paper was new safety metrics, but I find these metrics rather straightforward. They just calculate the number of safety violations divided by the number of opportunities where a violation could occur. Similarly, the unsafe exposure metric measures the duration of violations normalized by these opportunities. These seem like very standard normalization techniques rather than novel metrics. Therefore I do not agree with the claim and do not like to call it as a major contribution.
-
Since the paper is primarily a benchmark paper, I would expect the experimental findings to provide significant new insights. But, I see that most of the findings seem relatively expected. For example, the observation that a robot can complete a task while performing unsafe actions is already well understood and has motivated several previous safety benchmarks. Similarly, the finding that longer and more complex manipulation tasks can result in more safety violations is not surprising at all. Therefore, I do not think the experiments provide sufficiently strong evidence for a general conclusion about the relationship between model capability and safety.
Questions for authors
-
Several benchmarks already evaluate safety in robotic manipulation using temporal logic specifications or trajectory-level planning. What safety failures can SafeManip detect that these benchmarks cannot?
-
That robots can complete tasks while acting unsafely, and that longer, more complex tasks produce more violations, are both expected findings. What result in the paper would not have been predicted from prior safety benchmarks?
9 comments
Response to Reviewer Djrc (Part 2)
Table R1. Coverage of our ten safety templates: concrete failures and reported detectors.
| Safety category / template | Concrete unsafe execution | SENTINEL [1] | IS-Bench [6] | SafeVLA-Bench [7] | SafeManip |
|---|---|---|---|---|---|
| Collision and Contact Safety () | A carried pan strikes nearby glassware | ✓* | ✗ | ✓* | ✓ |
| Grasp Stability () | A pan slips during transport and drops | ✗ | ✗ | ✓* | ✓ |
| Release Stability () | A released bottle keeps rolling and falls | ✗ | ✗ | ✗ | ✓ |
| Cross-Contamination Safety () | Touching a clean bowl after touching raw food can lead to food poisoning | ✗ | ✗ | ✗ | ✓ |
| Action-Onset Safety () | Placing an object onto an already occupied surface may cause objects to fall over | ✓* | ✓* | ✗ | ✓ |
| Mechanism Safety () | Trying to open an obstructed drawer can break the drawer or damage the robot arm | ✗ | ✗ | ✗ | ✓ |
| Containment Safety () | Pouring liquid without a proper container underneath leads to spills | ✗ | ✗ | ✗ | ✓ |
| Enclosure and Access Safety () | Placing another plate into an occupied microwave may lead to collision | ✗ | ✗ | ✗ | ✓ |
| Enclosure and Access Safety () | Reaching through a partly opened drawer can lead to unintended damage to the drawer or robot arm | ✗ | ✗ | ✗ | ✓ |
| Enclosure and Access Safety () | Releasing a jar before there is internal support can lead to the jar shattering | ✗ | ✗ | ✗ | ✓ |
Legend: ✓ reported; ✓* related or narrower coverage; ✗ not reported. SENTINEL analyzes execution collisions [1]. SafeVLA-Bench checks contact and uses a downward-slip proxy [7].
Table R2. The combination of interface, requirements, and measurement matters.
| Reported feature | SENTINEL [1] | IS-Bench [6] | SafeVLA-Bench [7] | SafeManip |
|---|---|---|---|---|
| VLA evaluation support | ✗ | ✗ | ✓ | ✓ |
| Semantic/procedural safety | ✓ | ✓ | ✗ | ✓ |
| Temporal safety beyond instantaneous states | ✓ | ✓ | ✗ | ✓ |
| Opportunity-normalized metrics | ✗ | ✓ | ✗ | ✓ |
| Repeated violations and exposure | ✗ | ✗ | ✗ | ✓ |
Response to Reviewer Djrc (Part 3)
R2. The main contribution of the paper is the proposed safety metrics, and they are straightforward and standard techniques.
We respectfully disagree that the safety metrics are our main contribution or the paper's central claim. SafeManip's primary contribution is a benchmark for evaluating physical and semantic temporal safety directly from continuous-control manipulation rollouts, as presented in the Introduction and Figure 1. This includes ManipVerse's reusable object-property bindings (Section 3.2), manipulation-specific predicate grounding and temporal specifications (Section 3.3), and recovery-based monitoring that identifies repeated violations and their duration within a rollout (Section 3.4). These components enable concrete checks such as distinguishing uncontrolled grasp loss from intentional release and tracking contamination caused by earlier interactions. The opportunity-conditioned metrics in Section 4.2 summarize the events produced by this monitoring framework; their contribution lies in what is consistently measured and compared, not in the mathematical novelty of division. Evaluating the paper primarily through the simplicity of these ratios therefore overlooks the specification, grounding, and monitoring capabilities that constitute the benchmark.
However, even for the safety metrics, the task of identifying and tracking safety opportunities during execution is nontrivial. Most compared benchmarks neither report nor support opportunity-normalized metrics (Table R5). SafeLIBERO [2] measures obstacle avoidance, while SafeVLA [3] aggregates unsafe-state costs. These state-based scores alone do not identify when a grasp or release obligation starts, recovers, and becomes active again. Normalizing requires these grounded events, not simply dividing an existing violation count.
SafeManip makes low-level safety opportunities measurable throughout continuous-control execution. Our predicates identify when an object is grasped, when it slips, when it is released, and whether it settles safely. The temporal monitor tracks each obligation through activation, violation, and recovery, then re-arms to capture subsequent failures. This enables violations per triggered event and unsafe-state exposure per triggered event, distinguishing repeated slips from a single lasting unsafe state. This capability requires the predicate design and temporal monitoring in Tables R1–R2; the normalization formula alone cannot provide it.
Skill-level triggers do not resolve the low-level execution hazards that SafeManip measures. IS-Bench [6] scores triggered procedural requirements in VLM plans, such as turning off a stove after use or clearing flammable objects before activating a burner. Its published predicates do not detect grasp slip, unstable release, or failed retraction after obstructed fixture motion. A correctly ordered skill plan can still incur these hazards during execution. SafeManip evaluates whether the physical execution is safe, beyond whether the procedural steps are correct. Table R5 makes this measurement distinction explicit.
Table R5. Evaluation features provided by SafeManip.
| Work | Episode safety / violation rate | Trigger-conditioned score | Repeated-event violations + exposure per trigger |
|---|---|---|---|
| VLSA / SafeLIBERO [2] | ✓ | ✗ | ✗ |
| SafeVLA / Safety-CHORES [3] | ✗ | ✗ | ✗ |
| VLA-Arena [4] | ✗ | ✗ | ✗ |
| RedVLA [5] | ✓ | ✗ | ✗ |
| SENTINEL [1] | ✓ | ✗ | ✗ |
| IS-Bench [6] | ✓ | ✓ | ✗ |
| SafeVLA-Bench [7] | ✓ | ✗ | ✗ |
| SafeManip | ✓ | ✓ | ✓ |
Response to Reviewer Djrc (Part 1)
Thank you for highlighting the evaluation across several robot foundation models. We address the concerns below.
R1. Novelty relative to existing temporal-based work?
SafeManip adds manipulation-specific physical predicates and recurring-obligation measurement to temporal safety evaluation. The distinction has two parts:
Secs. 3.3–3.4 and Appendix F.1 of the main paper detail the predicate design, temporal monitoring, and grounding across RoboCasa365 and LIBERO. Tables R1–R2 retain the broader benchmark comparisons.