- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
As I understand it, this is mainly about a safety benchmark for robot manipulation. It checks predefined rules (human-defined) using simulator states, and uses the same checks to filter training demonstrations. Then provide 200 tasks and 8,000 demonstrations, evaluate and fine-tune several VLAs, and include a small real-robot comparison.
Critical strengths
- Separating task success from safety is useful and make sense. A robot can finish the task while breaking safety rules along the way, and the bench can provide more diverse dataset (safe label) for training.
2.The overall setup is fairly complete. The 200 tasks, 8,000 demonstrations, and shared rules for data filtering and evaluation give others a useful starting point for further work.
Critical weaknesses
-
I’m not sure how much the simulation safety scores tell us about real-world safety(sim 2 real gap which is important for bench about safety). Just as Clutter tests in paper. There is no real-robot comparison before and after fine-tuning to show whether the filtered data actually makes the VLA safer. I'm concerned about this(this make the paper much more useful but unclear now).
-
It’s unclear whether the gains come from safety filtering or just task-specific training. A matched comparison without safety filtering is needed.
-
The hand-crafted rules may be too rigid and flag reasonable alternatives as violations. Filtering data this way could LIMIT the VLA’s flexibility. but this risk is not tested.
Questions for authors
See Critical Weaknesses Please.
3 comments
Response to Reviewer wf62 (2/2): rigidity of the rules and flexibility; revisions
W3. Are the rules too rigid, and does filtering limit the policy's flexibility?
Safety requirements allow multiple solutions. ManiGuard’s specifications constrain safety-relevant object states and prerequisites, while leaving the motion strategy open. For example, Jar requires closure before lifting but does not prescribe a particular grasp, approach, or path. Different strategies can satisfy the same specification. When we reran Jar data generation with the filter recording instead of rejecting, each of the 424 successful trajectories it flagged either tilts the jar beyond 30° (246) or lifts it before it is closed (178). These are the unsafe events themselves, not alternative ways of doing the task. These requirements define the benchmark’s safety standard; thresholds are explicit parameters that can be adjusted for other settings (App. E.1).
Filtering may affect training coverage. Retaining the same number of demonstrations does not guarantee the same behavioral diversity. In W2, the policy trained with some violating demonstrations completed more tasks, suggesting that these trajectories can contain useful task information. The comparison does not isolate whether the difference comes from broader behavioral coverage, so it does not rule out the concern you raise.
Data selection remains flexible. ManiGuard separates safety annotation from the decision to discard a trajectory. Safe-only SFT is our baseline recipe; the framework also supports retaining violating trajectories with their per-step labels. The released Jar and Stack rerun data enable alternative data-selection and learning strategies under the same evaluation criteria. This flexibility is central to the framework: it provides the generation, annotation, and evaluation infrastructure for studying how to use safety-labeled data effectively, including the data-composition questions discussed in App. H.
Revisions in response to this review.
Changes are marked in blue in the revised PDF, with margin tags naming the review.
- A new appendix section (App. I) with the filter-off rerun and the matched comparison (design, per-level outcome counts and rates including task success, limitations); the limitation sentences in Sec. 4.2 (Q1) and Sec. 5 are replaced by its result, including the in-distribution difference in task success; the labeled rerun output is released.
- App. E.1 states that the thresholds are specification parameters and gives the rationale for each value.
- Sec. 4.2 (Q5) and Table 28 state that the physical-robot policies are single-task fine-tunes trained separately in each domain (60 teleoperated or simulated demonstrations), and what the physical study supports: agreement in how policies compare and fail, not in violation rates.
Follow-up to Reviewer wf62
Thank you again for the review. We ran the matched comparison you asked for (App. I), and a short summary is posted in a separate comment to all reviewers. On rigidity, each of the 424 successful Jar trajectories the filter flagged either tilted the jar beyond 30° or lifted it before closure, so the flagged cases are unsafe events rather than alternative ways to do the task. We do not yet have a before-and-after comparison on hardware, and the paper states this. We would be grateful to know whether the new results address your concerns.
Response to Reviewer wf62 (1/2): real-world safety; safety filtering versus task-specific training
We thank Reviewer wf62 for the careful review. We answer each point below and list the resulting revisions at the end of comment 2/2.
W1. What simulated safety scores say about real-world safety.
W2. Do the gains come from safety filtering or from task-specific training?