acceptodds
Under review as a conference paper at ICLR 2027

SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation

Abstract

Robotic manipulation is typically evaluated by task success, but reaching the correct final state does not guarantee safe execution. A robot may succeed despite contamination, premature release, or incorrect sequencing. These failures are poorly captured by task-completion metrics or isolated state checks because safety often depends on how behavior unfolds over time. We introduce SafeManip, a benchmark for evaluating temporal safety directly from low-level manipulation rollouts across eight physical and semantic categories. SafeManip grounds symbolic manipulation predicates into reusable temporal properties over finite executions, formalized with LTLf. We further introduce ManipVerse, a property ontology enabling semantic safety rules generalize across objects, tasks, and environments. Finally, SafeManip introduces per-trigger safety metrics that normalize violations by the safety-relevant opportunities that activate each obligation, accounting for differences in how often those opportunities arise across policies and tasks. We instantiate SafeManip on RoboCasa365 and LIBERO to audit seven state-of-the-art robot foundation-model policies. Our results reveal a gap between task capability and temporal safety: higher task success does not necessarily imply safer execution, successful rollouts can still contain violations, and failure patterns vary systematically across safety categories, task horizons, and manipulation suites. SafeManip provides a reusable layer for evaluating how safely manipulation tasks are executed, not only whether they are completed.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic Manipulation, Safety, Vision-Language-Action Models, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DjrcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In the paper, the authors propose a new benchmark called SafeManip, which tries to evaluate the safety of robotic manipulation for more than just task completion. The main motivation is that a robot might complete a task successfully but still will perform an unsafe actions during execution, and the existing task success metrics may not capture since they only care about task completion. To address this, the authors introduce a set of temporal safety rules using Linear Temporal Logic (LTLf), that includes eight different safety categories, such as collisions, grasp stability, contamination, and object release. Then they further propose metrics that measure the safety violations relative to the number of opportunities where a safety rule becomes relevant. The benchmark is evaluated on several robot foundation models and the main findings are that task success does not necessarily mean safe execution, different models have different types of safety failures, and longer tasks can lead to more safety violations.

Critical strengths
  • The paper addresses a relevant problem in robotic manipulation safety. I also appreciate that the authors evaluate several existing robotic foundation models rather than limiting the study to a single model.
Critical weaknesses
  • I think the novelty of the paper is quite limited, becase there are several benchmarks already exist for evaluating safety in robotic manipulation around the ideas of temploral logic and trajectory level planning. I am not convinced that the paper clearly establishes what is different in this benchmark compared to the existing ones. I would have liked to see a more direct comparison showing what safety failures SafeManip can identify that these benchmarks cannot.

  • I feel the central claim of the paper was new safety metrics, but I find these metrics rather straightforward. They just calculate the number of safety violations divided by the number of opportunities where a violation could occur. Similarly, the unsafe exposure metric measures the duration of violations normalized by these opportunities. These seem like very standard normalization techniques rather than novel metrics. Therefore I do not agree with the claim and do not like to call it as a major contribution.

  • Since the paper is primarily a benchmark paper, I would expect the experimental findings to provide significant new insights. But, I see that most of the findings seem relatively expected. For example, the observation that a robot can complete a task while performing unsafe actions is already well understood and has motivated several previous safety benchmarks. Similarly, the finding that longer and more complex manipulation tasks can result in more safety violations is not surprising at all. Therefore, I do not think the experiments provide sufficiently strong evidence for a general conclusion about the relationship between model capability and safety.

Questions for authors
  1. Several benchmarks already evaluate safety in robotic manipulation using temporal logic specifications or trajectory-level planning. What safety failures can SafeManip detect that these benchmarks cannot?

  2. That robots can complete tasks while acting unsafely, and that longer, more complex tasks produce more violations, are both expected findings. What result in the paper would not have been predicted from prior safety benchmarks?

9 comments
AuthorsOpenReview ↗

Response to Reviewer Djrc (Part 1)

Thank you for highlighting the evaluation across several robot foundation models. We address the concerns below.

R1. Novelty relative to existing temporal-based work?

SafeManip adds manipulation-specific physical predicates and recurring-obligation measurement to temporal safety evaluation. The distinction has two parts:

  • SafeManip audits continuous grasp and release dynamics that existing temporal-based works do not measure. SENTINEL [1] is the closest related work that also uses temporal logic. However, it cannot fully evaluate end-to-end VLA policies because its safety predicates are built around AI2-THOR's high-level semantic actions and states, rather than low-level manipulation dynamics. In other words, SENTINEL's capability stopped at evaluating safety at semantic level of logical planning rather than both dynamics level (required low level information which AI2-THOR cannot support) and semantics level. These predicates cannot be directly transferred to MuJoCo-based benchmarks such as RoboCasa365 and LIBERO, even though both frameworks use temporal logic. For example, AI2-THOR uses magic grasp, which automatically attaches the object to the gripper. This means SENTINEL can only evaluate the procedural reasoning of a policy but not the raw execution. SafeManip instead builds its LTL$_f$ specifications from lower-level manipulation predicates grounded in contact, gripper state, object motion, and object semantics. Our predicate design enables direct evaluation of continuous-control VLA policies. Table R1 demonstrates that SENTINEL can only partially cover two of the ten safety templates we cover in SafeManip. IS-Bench [6] evaluates VLM-generated plans executed through predefined skills, with triggered requirements such as turning off a stove after use. Its skill-level checks likewise do not audit grasp slip, post-release settling, or recovery after obstructed fixture motion.
  • SafeManip makes recurring low-level safety opportunities measurable throughout continuous-control execution. Identifying and tracking these opportunities is nontrivial. Most compared benchmarks do not report opportunity-normalized metrics (Table R5): SafeLIBERO [6] measures obstacle avoidance, while SafeVLA [7] aggregates unsafe-state costs. These scores alone do not identify when a grasp or release obligation starts, recovers, and becomes active again. Our predicates identify grasps, slips, releases, and settling; the temporal monitor tracks activation, violation, and recovery, then re-arms to capture subsequent failures. This enables violations per triggered event and unsafe-state exposure per triggered event, distinguishing repeated slips from a single lasting unsafe state. Normalizing requires these grounded events, not simply dividing an existing violation count. Tables R1–R2 show the predicate and monitoring capabilities that make this measurement possible.

Secs. 3.3–3.4 and Appendix F.1 of the main paper detail the predicate design, temporal monitoring, and grounding across RoboCasa365 and LIBERO. Tables R1–R2 retain the broader benchmark comparisons.

AuthorsOpenReview ↗

Response to Reviewer Djrc (Part 2)

Table R1. Coverage of our ten safety templates: concrete failures and reported detectors.

Safety category / templateConcrete unsafe executionSENTINEL [1]IS-Bench [6]SafeVLA-Bench [7]SafeManip
Collision and Contact Safety (φ1\varphi_1)A carried pan strikes nearby glassware✓*✗✓*✓
Grasp Stability (φ2\varphi_2)A pan slips during transport and drops✗✗✓*✓
Release Stability (φ3\varphi_3)A released bottle keeps rolling and falls✗✗✗✓
Cross-Contamination Safety (φ4\varphi_4)Touching a clean bowl after touching raw food can lead to food poisoning✗✗✗✓
Action-Onset Safety (φ5\varphi_5)Placing an object onto an already occupied surface may cause objects to fall over✓*✓*✗✓
Mechanism Safety (φ6\varphi_6)Trying to open an obstructed drawer can break the drawer or damage the robot arm✗✗✗✓
Containment Safety (φ7\varphi_7)Pouring liquid without a proper container underneath leads to spills✗✗✗✓
Enclosure and Access Safety (φ8\varphi_8)Placing another plate into an occupied microwave may lead to collision✗✗✗✓
Enclosure and Access Safety (φ9\varphi_9)Reaching through a partly opened drawer can lead to unintended damage to the drawer or robot arm✗✗✗✓
Enclosure and Access Safety (φ10\varphi_{10})Releasing a jar before there is internal support can lead to the jar shattering✗✗✗✓

Legend: ✓ reported; ✓* related or narrower coverage; ✗ not reported. SENTINEL analyzes execution collisions [1]. SafeVLA-Bench checks contact and uses a downward-slip proxy [7].

Table R2. The combination of interface, requirements, and measurement matters.

Reported featureSENTINEL [1]IS-Bench [6]SafeVLA-Bench [7]SafeManip
VLA evaluation support✗✗✓✓
Semantic/procedural safety✓✓✗✓
Temporal safety beyond instantaneous states✓✓✗✓
Opportunity-normalized metrics✗✓✗✓
Repeated violations and exposure✗✗✗✓
AuthorsOpenReview ↗

Response to Reviewer Djrc (Part 3)

R2. The main contribution of the paper is the proposed safety metrics, and they are straightforward and standard techniques.

We respectfully disagree that the safety metrics are our main contribution or the paper's central claim. SafeManip's primary contribution is a benchmark for evaluating physical and semantic temporal safety directly from continuous-control manipulation rollouts, as presented in the Introduction and Figure 1. This includes ManipVerse's reusable object-property bindings (Section 3.2), manipulation-specific predicate grounding and temporal specifications (Section 3.3), and recovery-based monitoring that identifies repeated violations and their duration within a rollout (Section 3.4). These components enable concrete checks such as distinguishing uncontrolled grasp loss from intentional release and tracking contamination caused by earlier interactions. The opportunity-conditioned metrics in Section 4.2 summarize the events produced by this monitoring framework; their contribution lies in what is consistently measured and compared, not in the mathematical novelty of division. Evaluating the paper primarily through the simplicity of these ratios therefore overlooks the specification, grounding, and monitoring capabilities that constitute the benchmark.

However, even for the safety metrics, the task of identifying and tracking safety opportunities during execution is nontrivial. Most compared benchmarks neither report nor support opportunity-normalized metrics (Table R5). SafeLIBERO [2] measures obstacle avoidance, while SafeVLA [3] aggregates unsafe-state costs. These state-based scores alone do not identify when a grasp or release obligation starts, recovers, and becomes active again. Normalizing requires these grounded events, not simply dividing an existing violation count.

SafeManip makes low-level safety opportunities measurable throughout continuous-control execution. Our predicates identify when an object is grasped, when it slips, when it is released, and whether it settles safely. The temporal monitor tracks each obligation through activation, violation, and recovery, then re-arms to capture subsequent failures. This enables violations per triggered event and unsafe-state exposure per triggered event, distinguishing repeated slips from a single lasting unsafe state. This capability requires the predicate design and temporal monitoring in Tables R1–R2; the normalization formula alone cannot provide it.

Skill-level triggers do not resolve the low-level execution hazards that SafeManip measures. IS-Bench [6] scores triggered procedural requirements in VLM plans, such as turning off a stove after use or clearing flammable objects before activating a burner. Its published predicates do not detect grasp slip, unstable release, or failed retraction after obstructed fixture motion. A correctly ordered skill plan can still incur these hazards during execution. SafeManip evaluates whether the physical execution is safe, beyond whether the procedural steps are correct. Table R5 makes this measurement distinction explicit.

Table R5. Evaluation features provided by SafeManip.

WorkEpisode safety / violation rateTrigger-conditioned scoreRepeated-event violations + exposure per trigger
VLSA / SafeLIBERO [2]✓✗✗
SafeVLA / Safety-CHORES [3]✗✗✗
VLA-Arena [4]✗✗✗
RedVLA [5]✓✗✗
SENTINEL [1]✓✗✗
IS-Bench [6]✓✓✗
SafeVLA-Bench [7]✓✗✗
SafeManip✓✓✓
Reviewer_4tJaOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SafeManip, a temporal-logic-based safety evaluation framework for robot manipulation that checks for safety rule violations in execution trajectories rather than focusing solely on final task success rates. The authors designed evaluation metrics covering eight categories of safety issues and evaluated robotic models on RoboCasa365 and LIBERO. Experiments found that task success rates and execution safety are not fully consistent, and that violation patterns differ across safety categories and task settings.

Critical strengths
  • This paper is well organized, and easy to follow.
  • The authors conduct extensive experiments to support their conclusions.
Critical weaknesses
  • The distinction between safety risks and general execution quality is not sufficiently clear.

    Many of the safety domains defined in the paper broadly reflect basic operational capabilities and trajectory execution quality. The paper does not adequately explain under what conditions violations of these properties constitute safety risks, or how to distinguish harmless execution errors from behaviors that could potentially cause harm. While temporal logic provides a precise way to specify and check these properties, it does not, by itself, establish their validity as safety evaluation criteria. Further validation is needed to determine whether the proposed metrics primarily measure safety risks rather than general execution quality.

  • The interpretation of the utility–safety gap is confounded, and the statistical evidence is limited.

    The overall task success rates on RoboCasa365 are relatively low across models, suggesting that the observed violations may partly stem from insufficient basic execution capabilities and task difficulty, rather than reflecting an independent separation between task competence and safety. Moreover, the analysis is based on only five model-level data points, and the reported correlation (r=0.66) is not statistically significant, providing insufficient evidence to support a robust reversal of the relationship. The results in Figure 10 also indicate that very few successful rollouts are classified as unsafe.

  • Typographical and citation errors.

    Table 1 contains an unresolved citation, “FailureBench (?)”, and Line 95 also includes an incorrectly rendered citation marker.

Questions for authors
  • How do the authors distinguish general execution errors from genuine safety risks?

  • Could the authors provide independent annotations of hazardous behaviors to validate whether the proposed metrics identify safety risks, instead of merely reflecting poor execution quality?

  • Have the authors evaluated the utility-safety gap across different safety categories? Such an analysis could help identify which categories are more indicative of safety risks rather than general model capabilities.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4tJa (Part 1)

Thank you for highlighting the paper's organization and the breadth of the experiments. We address the concerns below.

R1. Distinguishing safety from execution quality.

Our specifications detect concrete safety risks, not execution quality in general. We define safety in terms of risks of injury, object or equipment damage, and food contamination. We believe that "operational capabilities and trajectory execution quality" is a much broader umbrella that covers multiple facets of robot manipulation, including operational costs, success realization, safety, etc. In this paper, we have "safety" as our main focus and branch out our experiments to expose the non-orthogonal relationship between different facets.

Examples include: Failing to pick up an object is an execution failure; losing control of an already-held object creates a falling-object hazard. Failing to open a drawer is an execution failure; continuing to force an obstructed drawer creates crushing and trapping risks. These conditions are explicitly encoded in our predicates.

All ten templates target identifiable hazards (Table R4), informed by OSHA's robot-hazard guidance and the FDA Food Code, as stated in Sec. 3 of the main paper. The cited clauses provide the hazard-control principles; our predicates and temporal rules adapt them to robot manipulation [1-5].

Table R4. The safety risk checked by each template and its OSHA/FDA inspiration.

Safety category / templateWhat constitutes a violationConcrete safety riskOSHA/FDA inspiration
Collision and Contact Safety (φ1\varphi_1)Sustained unintended contact with surrounding objects or fixtures.Impact, crushing, breakage.OSHA OTM IV:4, Sec. V.1 [1].
Grasp Stability (φ2\varphi_2)A held object slips relative to the gripper or separates without deliberate release.Falling objects; fragile cups breaking.OSHA OTM IV:4, Sec. V.1: gripper failure and parts release [1].
Release Stability (φ3\varphi_3)A released object does not reach stable, compatible support with gripper clearance within the settling window.Falling, rolling, breakage, unsuitable food contact.OSHA 1910.176(b) [2]; FDA 3-304.11 [4].
Cross-Contamination Safety (φ4\varphi_4)Contaminated-to-clean contact without sanitization.Foodborne contamination.FDA 2-301.14(F–G), 3-302.11(A)(1), 3-304.11 [4].
Action-Onset Safety (φ5\varphi_5)Starting an action with a blocked path, unstable target/support, unsuitable placement geometry, excessive fragile-object clutter, or unready fixture.Collision, falling objects, breakage, unguarded operation.OSHA 1910.176(a) [2]; 1910.212(a)(1) [3].
Mechanism Safety (φ6\varphi_6)Continuing to drive into an obstruction without timely stopping and clear retraction.Crushing, trapping, equipment damage.OSHA OTM IV:4, Sec. V.1 [1].
Containment Safety (φ7\varphi_7)Transferred contents fail to settle in an appropriate receiver; dispensing lacks an aligned, upright vessel.Spills, slips, burns from hot contents.OSHA OTM IV:4, Sec. V.1 [1]; 1910.22(a)(3) [5].
Enclosure and Access Safety (φ8\varphi_8)Inserting into a microwave occupied by other food.Collision with contents; unintended food contact.OSHA 1910.176(a), clearance principle [2].
Enclosure and Access Safety (φ9\varphi_9)Reaching inside before the fixture is sufficiently open.Pinching or trapping at the door or drawer.OSHA 1910.176(a) [2]; OTM IV:4, Sec. V.1 [1].
Enclosure and Access Safety (φ10\varphi_{10})Releasing at the doorway before the object is inside.Falling objects and breakage.OSHA 1910.176(b) [2]; OTM IV:4, Sec. V.1 [1].
AuthorsOpenReview ↗

Response to Reviewer 4tJa (Part 2)

R2. Annotations of hazardous behaviors.

Please refer to Appendix E, which already reports 100 human-confirmed safety-property instances: all 81 flagged safety risks and 19 safe spot-checks. Concrete examples include:

  • Collision and Contact Safety (25): the gripper pressed against the refrigerator, toaster oven or other fixtures while performing tasks that do not require fixture contact.
  • Grasp Stability (10): a baguette or ice cube slipped inside the gripper; bread lost its grasp without deliberate release. Nine instances involved slipping; one involved unintended grasp loss.
  • Release Stability (1): a released straw did not land on the support.
  • Cross-Contamination Safety (34): the gripper touched raw meat, then a clean vegetable without sanitization.
  • Action-Onset Safety (11): three picks started on moving objects; one reached for a colander through fruit blocking the path; seven placements started onto unstable supports, including a moving glass or bowl.

The annotators inspected video, state playback, and predicate timelines to confirm these physical and semantic safety conditions. All 81 violations were human-confirmed, with 100% agreement across the 100 reviewed instances. The audit therefore supports human alignment of the observed safety-condition violations, including contact, loss of object control, unsafe release, contamination, and unsafe action initiation.

R3. Confounding utility–safety gap.

We agree that execution competence can influence the frequency of safety violations. However, our per-trigger metric (Sec. 4.2 in the main paper) is specifically designed to mitigate this confounding factor. This was due to the need of avoiding unfairly penalizing more successful policies that could perform much better in extremely long-horizon tasks. Instead of aggregating violations across entire rollouts which could make better policies appear less safe, we evaluate safety conditional on the relevant interaction occurring, distinguishing how frequently a policy encounters safety-relevant situations from how safely it handles them. For example, for grasp stability,

  • Execution capability. Consider a task requiring three grasps. A weaker policy attempts only one grasp and experiences one slip, while a stronger policy attempts all three grasps and experiences two slips. Although the weaker policy produces fewer violations in absolute terms, its per-trigger violation rate is higher (100% vs. 67%). Thus, our metric prevents weaker policies from appearing safer simply because they fail early and encounter fewer safety-relevant interactions.
  • Task difficulty. Consider an easier task requiring three grasps and a harder task requiring ten grasps. A policy experiences one slip in the easier task and two slips in the harder task. Although the harder task produces more violations in absolute terms, its per-trigger violation rate is lower (20% vs. 33.3%). Thus, our metric accounts for the greater number of safety-relevant interactions in more complex tasks, rather than automatically penalizing them for having more opportunities for violations.

These examples demonstrate that our metric mitigates confounding from differences in execution capability and task complexity by normalizing for actual safety-relevant opportunities.

AuthorsOpenReview ↗

Response to Reviewer 4tJa (Part 3)

R4. Category-specific analysis.

Sec. 5.2 and Fig. 6 of the main paper compare safety categories, then identify policy-specific weaknesses:

  • Category patterns on RoboCasa365. Collision/contact occurs most often, but Release Stability and Action-Onset Safety have comparable per-trigger violation rates despite fewer encounters. Cross-Contamination Safety has the greatest unsafe-state exposure (0.48% to 1.01% per trigger).
  • Category patterns on LIBERO. Collision/contact has the highest violation rate and exposure per trigger across all seven policies. Release Stability also varies substantially across policies (0.004 to 0.069 violations per trigger), while Grasp Stability violations are more frequent per trigger than Action-Onset Safety violations.
  • Policy-specific weaknesses. On RoboCasa365, π0\pi_0 and π0.5\pi_{0.5} have the highest Containment Safety violation rates (0.111 and 0.081). On LIBERO, OpenVLA has the highest violation rates for Collision and Contact Safety (0.128), Release Stability (0.069), and Enclosure and Access Safety (0.054).

All violation rates above are per triggered event. Figs. 11–12 in Appendix D.4 of the main paper additionally report trigger counts and repeated violations.

R5. The reversal relationship between safety and success is not statistically significant on RoboCasa365. Results in Figure 10 show that very few successful rollouts are classified as unsafe.

  • The differing relationships between task success and safety highlight the complementary value of our safety metrics beyond task success rate. On LIBERO, task success is strongly negatively correlated with per-trigger safety violations (r=−0.99r=-0.99, p<0.001p<0.001, n=7n=7), whereas RoboCasa365 exhibits an opposite but statistically insignificant trend (r=+0.66r=+0.66, n=5n=5; Fig. 5, Sec. 5.1). Although the RoboCasa365 result does not establish a significant reversal, these differing relationships demonstrate that safety cannot be reliably inferred from task success alone. Therefore, our supported conclusion is that success alone does not fully characterize compliance. Appendix D.7, Results paragraph (pp. 21–22), supplies complementary within-architecture evidence: substantially higher success without a corresponding observed improvement in aggregate per-trigger safety.
  • Unsafe successful rollouts are not negligible. We respectfully disagree with this characterization of Fig. 10 in Appendix D.3 of the main paper. On RoboCasa365, 332 of 2,805 successful rollouts (11.84%) violate safety requirements, reaching 22.18% for GR00T N1.5. On LIBERO, the corresponding figure is 461 of 12,806 (3.60%). Among completed tasks, these violations remain invisible to task success alone.

R6. Citation errors. We have corrected the FailureBench citation and the malformed citation marker in the updated manuscript.

References

[1] OSHA. Technical Manual, Section IV, Chapter 4: Industrial Robot Systems and Industrial Robot System Safety. Section V.1, Robot Application Hazards.

[2] OSHA. 29 CFR 1910.176, Handling materials—general. Paragraph (a), safe clearances and unobstructed passage; paragraph (b), stable and secure storage.

[3] OSHA. 29 CFR 1910.212, General requirements for all machines. Paragraph (a)(1), guarding against point-of-operation and related machinery hazards.

[4] FDA. Food Code (2022). Sections 2-301.14(F–G), cleaning when changing tasks and switching from raw to ready-to-eat food; 3-302.11(A)(1), raw/ready-to-eat separation; 3-304.11, cleaned and sanitized food-contact surfaces.

[5] OSHA. 29 CFR 1910.22, General requirements for walking-working surfaces. Paragraph (a)(3), surfaces free from hazards including leaks and spills.

Reviewer_1e9wOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
1(−1)
Soundness
2
Read review
Paper summary

The paper think that final state does not guarantee safe execution, because safety often depends on how behavior unfolds over time. This paper proposes a benchmark called SafeManip: In the two simulation environments of RoboCasa365 and LIBERO, a temporal logic monitor checks each step of the robot's strategy's underlying actions, and counts safety violations on each occurrence of an event. The result is that higher task success does not necessarily imply safer execution and that successful rollouts can still contain violations.

Critical strengths
  1. An important problem that most benchmarks do not consider. The paper focuses on safety during execution. Most existing benchmarks only evaluate whether the final state is reached.
  2. Evaluation across VLA and WAM policies. The experiments cover VLAs and WAM) policy. The results suggest that current models may only be imitating the demonstrations rather than learning how to do tasks safely.
  3. A fairly comprehensive framework. The property templates cover both persistent constraints and multi-stage temporal requirements, from collision and grasp stability to cross-contamination and enclosure and access . This covers many kinds of safety problems in household and kitchen manipulation.
Critical weaknesses
  1. Limited novelty of the logic-based monitoring framework. Logic-based safety monitoring has already been studied in many prior works. The paper itself notes that SENTINEL is particularly close to its formal perspective in its use of temporal logic, but the difference is only briefly discussed. The authors should compare SafeManip with these works more explicitly and clarify itself.

  2. Benchmarks are too simple to support the conclusions. Some models, such as Cosmos Policy, are evaluated only on LIBERO, which is now too simple for the latest models. The paper itself notes that LIBERO has no long-horizon tasks and invokes only a subset of the properties. Results on newer and harder benchmarks, such as LIBERO-Plus (Fei et al., 2025) or RoboDojo (arXiv:2607.04434), would better support the effectiveness of SafeManip.

Questions for authors
  1. What is the main difference between SafeManip and prior logic-based safety monitoring frameworks, in particular SENTINEL? Which parts of the framework do the authors consider new?
  2. Can the authors evaluate Cosmos Policy, and ideally the other policies, on newer and harder benchmarks such as LIBERO-Plus or RoboDojo?
  3. Appendix D.7 shows that changing the training paradigm raises task success but do not raise per-opportunity violation rate. Does this mean that more demonstrations alone do not make policies safer? Have the authors tried feeding the per-trigger safety signals back into training as shaped rewards or constraints, as suggested in Sec. 6?
12 comments
AuthorsOpenReview ↗

Response to Reviewer 1e9w (Part 1)

Thank you for highlighting the comprehensive framework and the evaluation of both VLA and world action models. We address the concerns below.

R1. What is new relative to SENTINEL?

SENTINEL cannot fully evaluate end-to-end VLA policies because its safety predicates are built around AI2-THOR's high-level semantic actions and states, rather than low-level manipulation dynamics. In other words, SENTINEL's capability stopped at evaluating safety at semantic level of logical planning rather than both dynamics level (required low level information which AI2-THOR cannot support) and semantics level. These predicates cannot be directly transferred to MuJoCo-based benchmarks such as RoboCasa365 and LIBERO, even though both frameworks use temporal logic. For example, AI2-THOR uses magic grasp, which automatically attaches the object to the gripper. This means SENTINEL can only evaluate the procedural reasoning of a policy but not the raw execution. SafeManip instead builds its LTL$_f$ specifications from lower-level manipulation predicates grounded in contact, gripper state, object motion, and object semantics. Our predicate design enables direct evaluation of continuous-control VLA policies. Table R1 demonstrates that SENTINEL can only partially cover two of the ten safety templates we cover in SafeManip. Table R2 further highlights how SafeManip differs from existing benchmarks by combining continuous-control VLA evaluation, semantic and temporal safety specifications, and opportunity-conditioned measurement of repeated safety violations and unsafe-state exposure.

Table R1. Coverage of our ten safety templates: concrete failures and reported detectors.

Safety category / templateConcrete unsafe executionSENTINEL [1]IS-Bench [2]SafeVLA-Bench [3]SafeManip
Collision and Contact Safety (φ1\varphi_1)A carried pan strikes nearby glassware✓*✗✓*✓
Grasp Stability (φ2\varphi_2)A pan slips during transport and drops✗✗✓*✓
Release Stability (φ3\varphi_3)A released bottle keeps rolling and falls✗✗✗✓
Cross-Contamination Safety (φ4\varphi_4)Touching a clean bowl after touching raw food can lead to food poisoning✗✗✗✓
Action-Onset Safety (φ5\varphi_5)Placing an object onto an already occupied surface may cause objects to fall over✓*✓*✗✓
Mechanism Safety (φ6\varphi_6)Trying to open an obstructed drawer can break the drawer or damage the robot arm✗✗✗✓
Containment Safety (φ7\varphi_7)Pouring liquid without a proper container underneath leads to spills✗✗✗✓
Enclosure and Access Safety (φ8\varphi_8)Placing another plate into an occupied microwave may lead to collision✗✗✗✓
Enclosure and Access Safety (φ9\varphi_9)Reaching through a partly opened drawer can lead to unintended damage to the drawer or robot arm✗✗✗✓
Enclosure and Access Safety (φ10\varphi_{10})Releasing a jar before there is internal support can lead to the jar shattering✗✗✗✓

Legend: ✓ reported; ✓* related or narrower coverage; ✗ not reported. SENTINEL analyzes execution collisions [1]. SafeVLA-Bench checks contact and uses a downward-slip proxy [3].

Table R2. The combination of interface, requirements, and measurement matters.

Reported featureSENTINEL [1]IS-Bench [2]SafeVLA-Bench [3]SafeManip
VLA evaluation support✗✗✓✓
Semantic/procedural safety✓✓✗✓
Temporal safety beyond instantaneous states✓✓✗✓
Opportunity-normalized metrics✗✓✗✓
Repeated violations and exposure✗✗✗✓
AuthorsOpenReview ↗

Response to Reviewer 1e9w (Part 2)

R2. Additional evaluation on a more challenging benchmark.

We extended the evaluation to 70,000 LIBERO-PRO rollouts across all seven policies, including Cosmos Policy. LIBERO-PRO [4] covers 200 perturbed tasks, with 50 episodes per task per policy. Table R3 reports task success, episode-level violations, and both per-trigger safety metrics. Following the main paper's RQs:

  • RQ1: LIBERO-PRO retains a significant negative success–safety relationship similar to LIBERO, and successful execution still includes violations. Across seven policies, success correlates negatively with violations per trigger (r=−0.84r=-0.84, p=0.018p=0.018). 1,147 successful rollouts are unsafe (3.77%), close to LIBERO's 3.60%. The correlation follows LIBERO's negative pattern.
  • RQ2: LIBERO-PRO exposes more collision/contact and release violations per interaction. Both rates increase for all seven policies relative to LIBERO. OpenVLA's collision/contact rate rises from 0.128 to 0.146 and its Release Stability rate from 0.069 to 0.086.
  • RQ3: LIBERO-PRO reveals high risk even in atomic tasks. Atomic tasks average 0.099 violations per trigger, versus 0.032 on LIBERO; short and medium tasks average 0.056 and 0.057. LIBERO-PRO follows LIBERO's pattern: atomic tasks have the highest per-trigger risk, with substantially higher rates under perturbations. This differs from RoboCasa365's increase toward long-horizon tasks.

New finding: perturbation types expose different safety risks. LIBERO-PRO's environment-perturbation variant averages 0.084 violations per trigger, compared with 0.022 for the language-perturbation variant, a 3.9-fold difference across the seven policy-level rates.

Table R3. LIBERO-PRO: task success and per-trigger safety (10,000 episodes per policy).

PolicyTask success rateSafety violation rate (episodes)Triggered events NNSafety violations per triggered eventUnsafe-state exposure per triggered event (frames)
π0\pi_044.40%26.83%69,7040.062992.42
π0.5\pi_{0.5}56.20%16.12%62,4490.039331.47
RLDX-146.59%21.23%71,9560.044872.15
GR00T N1.539.55%22.01%62,6200.051983.44
GR00T N1.643.76%27.72%75,3450.063642.37
OpenVLA28.02%32.87%64,5780.083064.21
Cosmos Policy46.00%17.34%65,1920.039621.67

We pool applicable properties across rollouts. Violations per triggered event count violation events; unsafe-state exposure per triggered event measures their duration in monitor frames. Episode-level rates use all monitor verdicts and native task-success flags.

AuthorsOpenReview ↗

Response to Reviewer 1e9w (Part 3)

R3. Why more demonstrations alone do not make policies safer?

One plausible explanation is that demonstrations can themselves contain safety-property violations. In Appendix E of the paper, we already reported that our human audit confirmed 81 safety-property violations within successful training demonstrations: 0.73% of monitored demonstration property instances on RoboCasa365 and 0.10% on LIBERO. Human teleoperation and successful task completion do not guarantee safety-compliant demonstrations. For example, teleoperation in clutter can introduce unnecessary sustained contact with fixtures, such as cabinets or refrigerators. An object can also slip from the gripper yet land at the intended destination, satisfying the task goal despite loss of stable object control. These examples illustrate how collection based on task success can retain unsafe execution. Our audit confirms that such data contain safety-property violations; adding more success-filtered demonstrations does not by itself ensure better safety. The comparison shows that gains in task performance do not necessarily translate into gains in temporal safety.

References

[1] Zhan et al. (2025). SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of Foundation Model-based Embodied Agents. arXiv:2510.12985.

[2] Lu et al. (2025). IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks. arXiv:2506.16402.

[3] Fan et al. (2026). SafeVLA-Bench: A Benchmark for the Success–Safety Gap in Vision-Language-Action Models. arXiv:2606.00773.

[4] Zhou et al. (2025). LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization. arXiv:2510.03827.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall response

We thank the reviewers for recognizing the importance of evaluating safety during execution and the comprehensive framework (1e9w), the evaluation across multiple policies (1e9w, Djrc), and the clear organization and extensive experiments (4tJa). Our responses address the following points:

  • The contribution extends beyond temporal logic and normalization (1e9w, Djrc). Our grounded predicates enable checks across a significantly larger range of safety risks, informed by published industrial safety guidance and regulations, compared with prior work. This includes grasp slip, unstable release, contamination across contacts, and obstructed-mechanism recovery during continuous-control execution. Tables R1–R2 compare the resulting coverage and capabilities.
  • Our predicates identify concrete safety hazards, not execution failure in general (4tJa). Table R4 connects each template to its hazard and OSHA/FDA inspiration. Appendix E reports human confirmation of the monitored physical and semantic conditions across 100 reviewed instances.
  • Opportunity-normalized evaluation requires nontrivial event grounding, which we provide (4tJa, Djrc). We identify each obligation's activation, violation, and recovery. Per-trigger metrics separate interaction frequency from safety compliance and distinguish repeated failures from persistent unsafe states (Table R5).
  • We added a new evaluation for this rebuttal on LIBERO-PRO across all seven policies (1e9w). The new LIBERO-PRO evaluation includes Cosmos Policy and 70,000 rollouts, with success, violations, and unsafe-state exposure reported in Table R3. LIBERO-PRO confirms that task success does not guarantee safety, while revealing higher collision/contact and release violation rates than LIBERO for all seven policies and 3.9-fold higher per-trigger risk under environment versus language perturbations.
  • The findings reveal safety differences that success alone misses (4tJa, Djrc). We report benchmark-dependent success–safety relationships, policy-specific category weaknesses, and different rankings by violation frequency and persistence. On RoboCasa365, 11.84% of successful rollouts still violate safety requirements.
  • More demonstrations do not guarantee safer behavior (1e9w). The training-data audit identifies safety-property violations in human-collected demonstrations. Increasing success-filtered data alone does not ensure that unsafe execution is removed.

We have also corrected the citation errors identified by 4tJa.

AuthorsOpenReview ↗

Additional Rebuttal Updates

Following our initial response, we have provided further evidence addressing the remaining concerns:

  • Further clarification of novelty and diagnostic value (1e9w, Djrc). We provided a detailed comparison with Farrell et al. (2022), highlighting concrete differences in grasp and release safety, broader manipulation-specific coverage, recovery-aware monitoring and per-trigger diagnostics, and general-purpose evaluation through ManipVerse. We further compared SafeManip with SafeLIBERO, VLA-Arena, and RedVLA, providing observed failure trajectories and human-confirmed examples that demonstrate additional diagnostic capabilities beyond their published safety checks.

  • Practical safety improvement through agentic steering (1e9w). We integrated SafeManip's safety signals into HarnessVLA as runtime execution constraints. Across 50 tasks, safe task success improved from 42% to 62%, while episodes with safety violations decreased from 54% to 0%, demonstrating that the benchmark's signals can actively improve safe execution.

  • Real-robot safety evaluation through real-to-sim reconstruction (1e9w). We added Section 7 and Figure 10, demonstrating a workflow that reconstructs a real Franka robot execution into a physically simulated digital twin using multi-view RGB recordings and logged robot states. SafeManip evaluates the reconstructed trajectory using its existing safety predicates and temporal specifications, identifying violations that can be inspected against the original real-world observations. This provides an initial demonstration of how SafeManip can evaluate real-robot behavior without assuming direct access to privileged simulator states on physical hardware.

We appreciate the reviewers' continued engagement and hope these additional results and clarifications will be considered alongside our initial response in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.