acceptodds
Under review as a conference paper at ICLR 2027

ManiGuard: A Specification-Grounded Framework for Safety Evaluation and Demonstration Generation in Robotic Manipulation

21 views

Abstract

Foundation-model policies for robotic manipulation are rapidly improving in task success, yet whether they complete tasks safely remains poorly characterized. We introduce ManiGuard, a framework that provides safety evaluation for manipulation policies and safety-annotated demonstrations for downstream policy learning, both grounded in the same formal specifications, expressed in linear temporal logic over finite traces (LTLf). Its benchmark, ManiGuard, contains 200 contact-rich household tasks across six families, with safety defined independently of task success; one in-distribution condition and four single-axis out-of-distribution perturbations per task yield 1,000 fixed scenarios in physics simulation, each checked at every step of a rollout by automaton monitors compiled from the LTLf specifications over physics-grounded predicates rather than learned judges. Its demonstration pipeline uses the same monitors to verify and annotate trajectories, yielding 8,000 safe, successful demonstrations (40 per task) for supervised fine-tuning (SFT). Across more than 23,000 rollouts of vision-language-action policies, task success remains an unreliable proxy for safety: 6–21% of successful SFT rollouts incur a counted safety violation. Standard SFT on the generated demonstrations raises safe-success rates to 7.5–29.8% and improves engagement-conditioned safety relative to available zero-shot counterparts, yet 21–42% of engaged SFT rollouts still violate the constraints, and failures persist under distribution shift and on a physical Franka running matched tasks. ManiGuard thus links specification-grounded data collection, standard policy learning, and safety measurement in one workflow, and exposes the safety gaps that remain.

Primary area:
datasets and benchmarks
Keywords:
robotic manipulation, robot safety, vision-language-action models, safety benchmarking, temporal logic

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wf62OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

As I understand it, this is mainly about a safety benchmark for robot manipulation. It checks predefined rules (human-defined) using simulator states, and uses the same checks to filter training demonstrations. Then provide 200 tasks and 8,000 demonstrations, evaluate and fine-tune several VLAs, and include a small real-robot comparison.

Critical strengths
  1. Separating task success from safety is useful and make sense. A robot can finish the task while breaking safety rules along the way, and the bench can provide more diverse dataset (safe label) for training.

2.The overall setup is fairly complete. The 200 tasks, 8,000 demonstrations, and shared rules for data filtering and evaluation give others a useful starting point for further work.

Critical weaknesses
  1. I’m not sure how much the simulation safety scores tell us about real-world safety(sim 2 real gap which is important for bench about safety). Just as Clutter tests in paper. There is no real-robot comparison before and after fine-tuning to show whether the filtered data actually makes the VLA safer. I'm concerned about this(this make the paper much more useful but unclear now).

  2. It’s unclear whether the gains come from safety filtering or just task-specific training. A matched comparison without safety filtering is needed.

  3. The hand-crafted rules may be too rigid and flag reasonable alternatives as violations. Filtering data this way could LIMIT the VLA’s flexibility. but this risk is not tested.

Questions for authors

See Critical Weaknesses Please.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wf62 (1/2): real-world safety; safety filtering versus task-specific training

We thank Reviewer wf62 for the careful review. We answer each point below and list the resulting revisions at the end of comment 2/2.

W1. What simulated safety scores say about real-world safety.

Scope and physical evidence. The physical study (Q5) provides a small-scale case study on approximately matched tasks, examining safety outcomes and illustrative failure cases across simulation and hardware. The submission already states that these experiments “do not establish calibrated real-world safety prediction” (Sec. 5, lines 482–483) and that “correlation measures association, not agreement in absolute rates” (Sec. 4.2, lines 458–459).

The physical results show that the safety problems studied by ManiGuard also occur on hardware: policies can complete tasks while violating their safety requirements. In Cabinet, for example, the robot may open the drawer while knocking over the obstacle instead of moving it aside; Fig. 5c illustrates this behavior in both domains. The model families also show the same ordering in safe task completion across the matched settings. However, the Clutter discrepancy you identify demonstrates the limit of this agreement: the simulation recorded no violations, while the physical runs did. ManiGuard therefore supports controlled comparisons and failure diagnosis under its simulated specifications, with initial evidence of real-world relevance; a low simulated violation rate does not establish hardware safety.

Before and after fine-tuning. A before-and-after hardware comparison would help assess whether task-specific adaptation improves physical safety. Establishing the additional effect of safety filtering would require a matched success-only training condition. Our current physical study includes neither comparison. Its policies were trained on safely completed teleoperated demonstrations, separately from their simulated counterparts, and physical outcomes were scored by the operator against the written task specifications.

These policies still exhibited unsafe behavior, showing that safe demonstrations alone did not prevent violations in this setting. The matched Jar experiment in W2 examines the separate contribution of filtering under controlled simulated conditions; it does not establish a hardware safety benefit. The revised PDF states this distinction in App. I.

W2. Do the gains come from safety filtering or from task-specific training?

ManiGuard contributes a safety benchmark built around safety constraints in contact-rich and fluid manipulation, with the same formal monitors used to evaluate policies and to label demonstrations. Our SFT experiments demonstrate that the generated data supports downstream learning. Safe, successful demonstrations are a straightforward baseline choice, while the pipeline also supports retaining violating trajectories with their safety labels for alternative training recipes.

We agree that improvements over zero-shot policies alone cannot isolate the contribution of safety filtering. The submission acknowledges this in Secs. 4.2 and 5, and we have now conducted the suggested matched comparison. We fine-tuned two π0.5 policies on Jar using equally sized datasets of 1,040 successful demonstrations. They share 780 safe demonstrations and differ in 260 slots containing either safe or violating successes from the same tasks and generation pipeline. Training settings and evaluation conditions were held fixed.

The table reports in-distribution results averaged over three evaluation seeds. Safe success means completing the task without a violation; safety among engaged rollouts measures the fraction without violations once the robot interacts with scene objects.

Metric (%)Safety-filteredSuccess-only
Safe success20.535.9
Safety among engaged rollouts55.459.0
Task success24.439.7

Both training sets improve safe success over the zero-shot policy. Success-only training achieves higher task success and safe success, and the two policies violate the specification about equally often (31 vs. 32 of 78). In this setting, the gain therefore comes from in-domain fine-tuning rather than filtering, which matches your concern. It is also consistent with the paper's main finding that task success and safety diverge, since training on safe demonstrations alone did not make the policy safer. App. I reports the comparison.

The pipeline enables these data choices to be studied under consistent safety criteria. We now release the Jar and Stack rerun output, including safe and unsafe trajectories with per-step annotations, to support further studies of data composition and monitor-derived supervision (App. H).

AuthorsOpenReview ↗

Response to Reviewer wf62 (2/2): rigidity of the rules and flexibility; revisions

W3. Are the rules too rigid, and does filtering limit the policy's flexibility?

Safety requirements allow multiple solutions. ManiGuard’s specifications constrain safety-relevant object states and prerequisites, while leaving the motion strategy open. For example, Jar requires closure before lifting but does not prescribe a particular grasp, approach, or path. Different strategies can satisfy the same specification. When we reran Jar data generation with the filter recording instead of rejecting, each of the 424 successful trajectories it flagged either tilts the jar beyond 30° (246) or lifts it before it is closed (178). These are the unsafe events themselves, not alternative ways of doing the task. These requirements define the benchmark’s safety standard; thresholds are explicit parameters that can be adjusted for other settings (App. E.1).

Filtering may affect training coverage. Retaining the same number of demonstrations does not guarantee the same behavioral diversity. In W2, the policy trained with some violating demonstrations completed more tasks, suggesting that these trajectories can contain useful task information. The comparison does not isolate whether the difference comes from broader behavioral coverage, so it does not rule out the concern you raise.

Data selection remains flexible. ManiGuard separates safety annotation from the decision to discard a trajectory. Safe-only SFT is our baseline recipe; the framework also supports retaining violating trajectories with their per-step labels. The released Jar and Stack rerun data enable alternative data-selection and learning strategies under the same evaluation criteria. This flexibility is central to the framework: it provides the generation, annotation, and evaluation infrastructure for studying how to use safety-labeled data effectively, including the data-composition questions discussed in App. H.

Revisions in response to this review.

Changes are marked in blue in the revised PDF, with margin tags naming the review.

  1. A new appendix section (App. I) with the filter-off rerun and the matched comparison (design, per-level outcome counts and rates including task success, limitations); the limitation sentences in Sec. 4.2 (Q1) and Sec. 5 are replaced by its result, including the in-distribution difference in task success; the labeled rerun output is released.
  2. App. E.1 states that the thresholds are specification parameters and gives the rationale for each value.
  3. Sec. 4.2 (Q5) and Table 28 state that the physical-robot policies are single-task fine-tunes trained separately in each domain (60 teleoperated or simulated demonstrations), and what the physical study supports: agreement in how policies compare and fail, not in violation rates.
AuthorsOpenReview ↗

Follow-up to Reviewer wf62

Thank you again for the review. We ran the matched comparison you asked for (App. I), and a short summary is posted in a separate comment to all reviewers. On rigidity, each of the 424 successful Jar trajectories the filter flagged either tilted the jar beyond 30° or lifted it before closure, so the flagged cases are unsafe events rather than alternative ways to do the task. We do not yet have a before-and-after comparison on hardware, and the paper states this. We would be grateful to know whether the new results address your concerns.

Reviewer_EaNyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ManiGuard presents a framework that uses LTLf safety specifications for manipulation-task design, runtime evaluation, demonstration filtering, and downstream policy training. It includes 200 tasks, 1,000 ID/OOD scenarios, 8,000 safety-annotated demonstrations, and more than 23,000 policy rollouts—and the paper convincingly demonstrates that task completion is a poor proxy for safe manipulation. The strongest aspect is the clean separation of safety from task success and the use of deterministic physics-grounded monitors rather than learned safety judges; however, safety remains only as complete as the manually authored specification and predicate library, and the SFT experiments do not isolate the effect of safety filtering from ordinary in-domain fine-tuning. Overall, I view this as a potentially valuable benchmark and evaluation infrastructure contribution, with the current sim-to-real evidence being preliminary.

Critical strengths

Unification of the safety pipeline. The same formal specification serves as task requirement, trajectory monitor, demonstration filter, and downstream evaluation criterion, avoiding inconsistent definitions of safety at different stages.

Scale and breadth. The benchmark contains 200 contact-rich household tasks, 1,000 evaluation scenarios, 8,000 generated demonstrations and more than 23,000 evaluated rollouts, including OOD conditions and some physical-robot validation.

The paper recognizes that a policy can appear “safe” simply because it does nothing, and jointly measures completion, engagement, and violations. It also demonstrates empirically that 21–42% of engaged fine-tuned rollouts can remain unsafe.

Critical weaknesses

Safety is only relative to the hand-designed specification and predicates. The current library omits important physical safety dimensions including force limits, human proximity, and some environmental collisions. A policy may therefore be labeled safe while violating an unmodeled hazard. It may be interesting to consider (or at least discuss) ideas such as this one, where safety is extracted from language: Semantically safe robot manipulation: From semantic scene understanding to motion safeguards (RAL 2025)

The learning experiment does not cleanly establish the value of safety-filtered demonstrations. The authors explicitly acknowledge that they do not compare safety-filtered demonstrations against a matched success-only dataset, so improvements may partly reflect ordinary in-domain fine-tuning rather than the safety filtering itself. I would recommend looking at and possibly benchmarking against works such as: From demonstrations to safe deployment: Path-consistent safety filtering for diffusion policies (ICRA2026).

The real-world validation is weak relative to the scale of the claims. Physical scenes are only approximately matched to simulation and safety verdicts are manually scored by the operator rather than by the formal monitor, so this does not establish calibrated sim-to-real safety prediction.

Questions for authors

My main questions are already addressed in the Weaknesses part.

Additionally: How do you see this scale in the long term, and what results would you hope this benchmark will enable?

4 comments
AuthorsOpenReview ↗

Response to Reviewer EaNy (1/3): specification coverage and the matched comparison

We thank Reviewer EaNy for the careful review. We answer each point below and list the resulting revisions at the end of comment 3/3.

W1. Safety is relative to the hand-designed specifications and predicates.

Safety in ManiGuard is relative to its instantiated specifications, as stated in Sec. 5 and App. H. Fine-grained physical safety dimensions, including force limits, environmental collisions, and human proximity, are important extensions, but evaluating them reliably requires sufficiently accurate simulation of the relevant interactions and scene elements. Our framework supports incorporating additional predicates and temporal constraints; extending and validating this coverage as simulation fidelity and modeling capabilities improve is an important direction for future work.

As discussed in the revised Discussion (App. H), another promising extension is the automatic discovery of safety specifications from scene context and task instructions. Brunke et al. [1] illustrate this direction by inferring unsafe conditions from semantic scene representations, potentially reducing manual specification effort and adapting constraints to new environments. Inferred constraints that can be grounded in ManiGuard’s predicates could be incorporated into its monitoring pipeline, enabling studies of their coverage and reliability alongside the current authored specifications.

W2. The value of safety-filtered demonstrations.

ManiGuard contributes a safety benchmark whose tasks are built around safety constraints in contact-rich and fluid manipulation, and uses the same formal monitors to evaluate policies and to generate and label training demonstrations. The pipeline supports different data-selection choices: it can retain safe, successful demonstrations or preserve violating trajectories together with their safety annotations. For the released SFT baseline, we chose safe, successful demonstrations as straightforward imitation targets; identifying the best training distribution is a research question this infrastructure enables but is left as future work, as discussed in App. H.

Your suggested comparison is a useful example of such a study. As proposed in Sec. 5 of the submission, we have now compared π0.5 policies trained on two matched Jar datasets of 1,040 successful demonstrations each. The datasets share 780 safe demonstrations and differ in 260 slots containing either safe or violating successes. Both policies use the same training protocol and are evaluated on all 26 ID tasks with three evaluation seeds (78 rollouts per policy). Rates are percentages averaged over these seeds:

Safety-filteredSuccess-only
ID SSR20.535.9
ID Safe | Eng.55.459.0
ID task success24.439.7

In this comparison, success-only training achieves higher task success and SSR, and the two policies violate the specification about equally often (31 and 32 of 78). This supports your concern that gains over zero-shot policies should not be attributed to filtering alone, and the revised PDF states this in Secs. 4.2 and 5. The result is also consistent with the paper's main finding that task success and safety diverge: in this setting, training on safe demonstrations alone did not make the policy safer.

The broader value of the pipeline is that these data choices can be investigated under a shared safety definition. We now release the Jar and Stack rerun output, including safe and unsafe trajectories with per-step annotations, to support studies of data composition and monitor-derived supervision discussed in App. H. App. I documents the comparison and its limitations.

References.

[1] Lukas Brunke et al. Semantically safe robot manipulation: From semantic scene understanding to motion safeguards. IEEE Robotics and Automation Letters, 10(5):4810–4817, 2025.

AuthorsOpenReview ↗

Response to Reviewer EaNy (2/3): PACS and real-world validation

W2 (continued). PACS.

Thank you for pointing us to PACS [2], which provides deployment-time safety filtering for learned policies through reachability analysis. Our SFT experiments demonstrate that the generated demonstrations support downstream learning, rather than establish a new safety-learning algorithm or compare deployment-time safety interventions. PACS is therefore relevant as a complementary approach that motivates further uses of the benchmark, including the runtime-shielding direction discussed in App. H. The revised PDF cites and discusses it in the extended related work (App. A).

W3. Real-world validation.

The physical study provides a small-scale case study on approximately matched tasks, examining safety outcomes and illustrative failure cases on hardware alongside their simulated counterparts. The submission already states that these experiments “do not establish calibrated real-world safety prediction” (Sec. 5, lines 482–483) and that “correlation measures association, not agreement in absolute rates” (Sec. 4.2, lines 458–459).

The case study shows why the safety questions examined by ManiGuard matter on hardware: policies can complete a physical task while violating its safety requirements. We also observe corresponding failure cases across domains. For example, in Cabinet, the robot may open the drawer while knocking over the obstacle in front of it instead of moving it aside; Fig. 5c illustrates this behavior in both simulation and reality. The evaluated model families also follow the same ordering in successful, safe task completion across the matched settings. These observations provide initial evidence of real-world relevance, while the differences in violation rates show that simulated safety scores cannot be directly interpreted as hardware risk.

We agree that the protocol needs clearer explanation. The simulated and physical policies were fine-tuned separately on demonstrations from their respective domains. Physical rollouts were scored by the operator against the written task specifications, rather than by an automated monitor using measured physical states. The requirements are therefore shared, but the measurement procedures differ. The revised PDF makes these distinctions explicit in Sec. 4.2 and Table 28.

The physical case study provides supplementary evidence that the problems ManiGuard studies also occur on real robots. As emphasized in simulation-based evaluation work such as SIMPLER [3], useful evaluation signals need not reproduce real-world behavior exactly, and simulation does not replace hardware evaluation. Our case study supports this limited role, rather than establishing reliable prediction of physical safety.

References.

[2] Ralf Römer et al. From demonstrations to safe deployment: Path-consistent safety filtering for diffusion policies. In IEEE International Conference on Robotics and Automation (ICRA), 2026.

[3] Xuanlin Li et al. Evaluating Real-World Robot Manipulation Policies in Simulation. Conference on Robot Learning, PMLR 270:3705–3728, 2025.

AuthorsOpenReview ↗

Response to Reviewer EaNy (3/3): long-term outlook and revisions

Q. Long-term scale, and the results we hope the benchmark enables.

We see six priorities for scaling ManiGuard and the research it can support:

  1. Expand task coverage. New task families can reuse the generation, annotation, and evaluation infrastructure, with family-specific scenes, specifications, motion templates, and grasp annotations (Apps. C–D). This allows the benchmark to grow without rebuilding the full pipeline.

  2. Extend physical safety coverage. Force limits, environmental collisions, and human proximity are important directions. Reliable evaluation requires appropriate simulation fidelity and scene modeling; we aim to extend and validate these dimensions as the underlying capabilities improve.

  3. Explore automatic specification discovery. Inferring safety constraints from scene semantics and task instructions, as illustrated by Brunke et al. [1], could reduce manual specification effort. ManiGuard could support evaluating the coverage and reliability of inferred constraints alongside authored specifications.

  4. Make safety and completion jointly evaluated. We hope the benchmark encourages reporting whether policies complete tasks safely, while accounting for apparent safety caused by inaction. This makes safety failures visible even when task-success scores improve.

  5. Support different approaches to safer policies. The shared tasks, specifications, and metrics enable controlled studies of how safety information is used: through demonstration selection, monitor-derived training objectives, or deployment-time interventions. App. H discusses direct optimization, targeted collection, and runtime shielding as future directions.

  6. Study data composition beyond volume. As discussed in App. H, an important question is which demonstrations are most useful—for example, those covering safety-critical interactions, prerequisite transitions, or recovery behaviors. The generation pipeline and per-step annotations provide tools for investigating these choices under consistent evaluation criteria.

Revisions in response to this review.

Changes are marked in blue in the revised PDF, with margin tags naming the review.

  1. A new appendix section (App. I) with the filter-off rerun and the matched comparison (design, per-level outcome counts and rates, limitations); the limitation sentences in Sec. 4.2 (Q1) and Sec. 5 are replaced by its result; the labeled rerun output is released.
  2. App. A discusses semantic safety filters (Brunke et al. [1], comment 1/3) and deployment-time safety filters (PACS [2], comment 2/3); App. H states how a new hazard enters the specification library, how either kind of filter can be evaluated with ManiGuard's monitor, and how inferred specifications could be monitored alongside the authored ones.
  3. Sec. 4.2 (Q5) and Table 28 state that the physical-robot policies are single-task fine-tunes trained separately in each domain (60 teleoperated or simulated demonstrations) and that the operator scored every physical rollout live against the written specification; Table 1 marks the real-robot entry as a small-scale, operator-scored evaluation.

References.

[1] Lukas Brunke et al. Semantically safe robot manipulation: From semantic scene understanding to motion safeguards. IEEE Robotics and Automation Letters, 10(5):4810–4817, 2025.

[2] Ralf Römer et al. From demonstrations to safe deployment: Path-consistent safety filtering for diffusion policies. In IEEE International Conference on Robotics and Automation (ICRA), 2026.

AuthorsOpenReview ↗

Follow-up to Reviewer EaNy

Thank you again for the review. Following your suggestion, we ran the matched success-only comparison (App. I). It supports your concern that the SFT gains should not be attributed to filtering alone, and we revised Secs. 4.2 and 5 accordingly. As you note among the strengths, the same formal specification serves as task requirement, monitor, demonstration filter, and evaluation criterion, which is what made this comparison possible. A short summary for all reviewers is posted in a separate comment, and we would be glad to clarify anything further.

Reviewer_JqpMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether manipulation policies that complete tasks also complete them safely. Its key idea is to write each task’s safety requirement once as a formal specification and reuse it as the safety definition, the evaluation monitor, and a filter for generated demonstrations. The evidence suggests task success and safety diverge: policies can succeed unsafely, and weak policies look safe only because they rarely act. Fine-tuning on the filtered demonstrations helps, but substantial safety gaps remain.

Critical strengths
  • Per-step monitoring enables the time-to-violation analysis. This analysis reveals weak policies fail at first contact, while stronger ones fail mid-manipulation.

  • The released demonstrations, grasp database, and shared planning/teleoperation infra have some practical value. Contact-rich and fluid physics plus temporal ordering constraints go beyond the state-only checks of rigid-body safety benchmarks.

Critical weaknesses
  • The safety-annotated dataset contains only safe trajectories. Its per-step labels are therefore constant and provide no negative examples for constraint or cost learning. Collision-aware scripted planning may also rarely violate these constraints, in which case the safety filter adds little beyond success filtering.

  • Most of novelties were proposed in SafeManip, which already applies LTLf monitors and a joint success–safety view to manipulation. The improvements are physics fidelity, engagement-aware metrics, and the data pipeline, but the comparison table is drawn around the authors’ own feature set.

Questions for authors
  • Since the released data contains only safe trajectories, what information do the per-step safety labels carry, and how do you envision downstream methods using these data?

  • Is the absence of violations on clutter a property of the policies or of the predicate thresholds? How were the thresholds chosen, and were they checked against human judgments of unsafe outcomes?

  • Are the sim2real policies the mainline family checkpoints or single-task fine-tunes? Given that violation rates did not track across domains, what should users conclude about real-world safety from simulated violation rates?

6 comments
AuthorsOpenReview ↗

Response to Reviewer JqpM (1/3): per-step labels and negative examples

We thank Reviewer JqpM for the careful review. We answer each point below and list the resulting revisions at the end of comment 3/3.

W1 / Q1. What the per-step labels carry when every released trajectory passed the safety gate.

At each step the monitor records every proposition's value and the state of the specification's automaton. On a safe trajectory the violation flag is constant, but the automaton state is not: in Jar, it changes when the jar is closed, which discharges the clause on(jar, table) W closed(jar), so the trace marks the step from which lifting is permitted.

App. H names three uses of per-step violations and constraint identities: costs or rewards for policy optimization, prioritizing counterexamples in data collection, and runtime shielding of unsafe actions. They build on two lines of work discussed in App. A: temporal specifications as structured reward or progress signals in reinforcement learning (e.g., [1, 2]), and learning safety constraints from demonstrations, often treating demonstrations as evidence of the desired behavior (e.g., [3, 4]). The monitor also labels any new trajectory in the simulator, so methods that optimize monitor-derived costs or rewards can query it online rather than rely on stored labels.

Every released demonstration passed the monitor gate (App. D.2), so its safety verdict is constant and it contains no negative examples. The files released with the submission store this trajectory-level verdict, not the per-step trace. No reported result depends on the trace: standard SFT imitates the retained actions and does not use it (Sec. 3.2).

However, our trajectory collection pipeline can provide fine-grained per-step labeling. We reran our pipeline on Jar and Stack with the gate recording instead of rejecting (same planner, scenes and cameras as the release) and release its full output (Jar, Stack), including both safe and unsafe trajectories, each with its per-step trace. For 51 of the 54 Jar and Stack tasks, the pool contains successful trajectories with and without violations, produced by the same planner in the same scene for the same goal; each violating trajectory is labeled with the violated clause and the step at which the violation occurs.

AuthorsOpenReview ↗

Response to Reviewer JqpM (2/3): the safety filter and SafeManip

W1. Does the safety filter add anything beyond success filtering?

ManiGuard is a safety benchmark with an accompanying trajectory-generation and annotation pipeline. Its contribution is to make task success and safety separately measurable and to provide data and evaluation infrastructure for studying them under consistent specifications. For the released SFT baseline, we retain safe, successful demonstrations as a straightforward choice of imitation targets. This choice is not a claim that safe-only demonstrations are the optimal training distribution, or that filtering alone is sufficient to improve policy safety.

What filtering adds beyond success checking. The filter enforces requirements that task completion and collision-aware motion planning do not by themselves guarantee. With safety monitoring enabled but rejection disabled, 40.8% of successful Jar trajectories and 24.4% of successful Stack trajectories violate the specifications. These include tilting the jar beyond its limit and lifting it before closure. A success-only filter would retain these executions. Thus, the safety gate materially changes which demonstrations are released; its role is to enforce the specified data-selection criterion.

What the pipeline enables beyond safe-only SFT. The same pipeline can retain violating trajectories with per-step monitor traces. As one controlled comparison, we trained two π0.5 policies on matched Jar datasets of 1,040 demonstrations, sharing 780 safe demonstrations and differing in 260 slots containing either safe or violating successes. On the 78 ID evaluation rollouts per policy, success-only training achieved higher task success and SSR, while the policies incurred similar numbers of violations:

Safety-filteredSuccess-only
ID task success (%)24.439.7
ID SSR (%)20.535.9
ID violating rollouts31/7832/78

In this setting, filtering alone therefore does not make the policy safer. This is consistent with the paper’s main finding that task success and safety diverge. Standard imitation picks up task completion from the demonstrations more readily than the safety constraints, and this shows up because safety is measured separately from success. It also motivates the labeled violating trajectories we now release (comment 1/3) for safety-aware learning. The revised PDF reports the comparison in App. I (one task family, one training run) and clarifies the role of safe-only SFT in Secs. 4.2 and 5.

W2. Novelty relative to SafeManip.

We agree that LTLf monitoring is not new, and we use it as a tool rather than claim it as a contribution. Temporal-logic specifications and runtime monitors are established in robotics [5, 6], and SENTINEL [7] and SafeManip [8] apply them to embodied safety evaluation. We adopt SafeManip's joint success–safety view with attribution (Sec. 3.3), and SafeManip covers more safety property categories than ManiGuard.

The two benchmarks are complementary and do not overlap in tasks or simulator. SafeManip adds LTLf monitoring to existing RoboCasa tasks in MuJoCo. ManiGuard builds its own tasks around safety constraints in Isaac Sim, including tasks with liquids, where spills and tilt during transport matter. The same policies can therefore be evaluated on both. ManiGuard also connects its monitors to demonstration generation and training, which made the matched comparison above and the labeled data release possible. It also includes a small physical-robot study on two approximately matched tasks, whereas SafeManip leaves real-robot evaluation to future work.

The revised Table 1 caption states the axes it compares, and App. A defines each column and notes the dimensions on which other benchmarks are broader.

AuthorsOpenReview ↗

Response to Reviewer JqpM (3/3): Clutter, thresholds, sim-to-real, revisions

Q2. Clutter's zero violations, the thresholds, and human judgment.

We do not read Clutter's zero as evidence that the policies avoid violations. It is what the simulation monitor recorded for that family, and the physical runs on matched Clutter tasks did record violations: under OOD conditions, 14 of 30 π0.5-SFT rollouts and 10 of 30 π0-SFT rollouts were unsafe successes (Table 29).

Thresholds and human judgment. The thresholds ground the atomic propositions in simulator state, which is common practice in the literature [7, 9]. In ManiGuard the values are part of each task's safety specification, not of the monitor: a user who wants a stricter or looser standard changes them, and the monitor checks the new specification exactly (App. E.1). The benchmark ships one such standard, set per object type to match everyday judgment: a dry object counts as toppled beyond 45° (Clutter, Cabinet), a stacked board, plate or jar beyond 30° (Stack, Jar), and a liquid-carrying container beyond 15°. Human judgment enters when these values are set. During development we reviewed the generated demonstrations and the evaluation videos: outcomes a viewer would call unsafe, such as an object knocked onto its side, lie well beyond the limit. Within a few degrees of a limit, a viewer cannot tell which side a tilt falls on, and the fixed value gives every policy the same verdict. Results are relative to this standard, as Sec. 5 states for the specifications as a whole.

Q3. Sim-to-real.

The physical-robot policies are single-task fine-tunes, not the mainline family checkpoints, and each domain has its own: for each of the three model families, one policy was fine-tuned on 60 teleoperated demonstrations of the physical task and one on 60 simulated demonstrations of the matched task (Clutter task_0048, Cabinet task_0019; Table 28). "The three strongest SFT policies" in Sec. 4.2 refers to the model families. Q5 therefore compares evaluations of the same model families across domains; it does not transfer one checkpoint.

The physical study (Q5) provides a small-scale case study on two approximately matched tasks, examining safety outcomes and illustrative failure cases on hardware alongside their simulated counterparts. It was not designed to calibrate simulated violation rates against physical ones (Sec. 5; App. H). Unsafe outcomes occur on the robot for every model family on both tasks, in 123 of the 360 physical rollouts (Table 29). Across the four task–condition combinations (Clutter and Cabinet, each under ID and OOD conditions), the three model families have the same observed safe-success ordering in simulation and on hardware. The failure modes match too: on Cabinet, every simulated violation is an object knocked over, and on the robot the violations were mainly of the same kind, the obstacle in front of the drawer knocked over instead of being moved aside; Fig. 5c shows such a case in each domain.

What users can conclude. From simulated safety results, users can conclude which policies complete tasks more safely and which failures to expect. They cannot conclude real-world violation rates, or that a policy with few simulated violations is safe on hardware. This is similar to current literature such as SIMPLER [10], which also distinguishes useful signals about relative policy performance from one-to-one reproduction of real-world behavior, while emphasizing that simulation does not replace hardware evaluation.

Revisions in response to this review.

Changes are marked in blue in the revised PDF, with margin tags naming the review.

  1. A new appendix section (App. I) with the filter-off rerun and the matched comparison (design, per-level outcome counts and rates, limitations); the limitation sentences in Sec. 4.2 (Q1) and Sec. 5 are replaced by its result.
  2. Sec. 3.2, Fig. 1, App. D.3 and the conclusion state precisely what each released dataset contains and which labels it carries; the labeled rerun output is released.
  3. Table 1's caption states the axes it compares, and App. A defines its columns and discusses the dimensions it does not cover (comment 2/3).
  4. Sec. 4.2 (Q5) and Table 28 state that the physical-robot policies are single-task fine-tunes trained separately in each domain (60 teleoperated or simulated demonstrations), and what the physical study supports: agreement in how policies compare and fail, not in violation rates.
  5. App. E.1 states that the thresholds are specification parameters and gives the rationale for each value.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for all reviewers

We thank the reviewers again. In short, ManiGuard contributes a safety benchmark built around safety constraints in contact-rich and fluid manipulation, with the same formal monitors used to evaluate policies and to label demonstrations. All three reviews asked whether safety-filtered demonstrations add value beyond success-only data, and how far the simulated results carry over to hardware. Below is a short summary of the new results. Details are in the individual responses and the revised PDF (changes in blue).

  1. Filtering changes the data. With rejection disabled, 40.8% of successful Jar trajectories and 24.4% of successful Stack trajectories violate their specifications, for example by tilting the jar beyond 30° or lifting it before it is closed. A success-only filter would keep these trajectories.
  2. In our matched Jar comparison, filtering alone did not make the policy safer. The two π0.5 policies were trained on 1,040 demonstrations each, which differ only in 260 slots holding safe or violating successes. They violate the specification about equally often (31 vs. 32 of 78 ID rollouts), and the success-only policy completes the task more often. Under distribution shift, the safety-filtered policy is slightly safer once engaged (App. I, Table 31). This is consistent with the paper’s main finding that task success and safety diverge (one task family, one training run).
  3. The released data now includes negative examples. The Jar and Stack rerun output keeps both safe and violating trajectories with per-step traces. For 51 of the 54 Jar and Stack tasks, it contains successful trajectories with and without violations, and each violation is labeled with its clause and step.
  4. The physical study is a small case study, not a calibration of violation rates. Unsafe outcomes occur for every model family on both tasks (123 of 360 rollouts), and the three families keep the same safe-success ordering as in simulation in all four task–condition settings (Sec. 4.2; Tables 28–29).
  5. The specifications constrain safety-relevant object states and event order, not the motion strategy. Each of the 424 successful Jar trajectories the filter flagged either tilts the jar beyond 30° (246) or lifts it before closure (178). Thresholds are explicit parameters of the specification (App. E.1).

We would appreciate any further feedback before the discussion period closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.