- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces RoboShackles, a dataset targeting human-injury prevention for embodied foundation models. Starting from real DROID robot observations, it constructs hazardous scenes through image editing and uses video generation to synthesize unsafe future rollouts. The resulting dataset comprises 6,000 original safe DROID demonstrations and 6,000 synthetically generated unsafe training samples, together with 1,200 multilingual unsafe evaluation samples spanning two direct-harm and four indirect-harm categories. The paper evaluates 15 embodied models using a no-op-based action safety criterion and fine-tunes World Guidance (WoG) to predict a no-op for unsafe inputs. It reports a reduction in WoG's unsafe-action rate from 98.92% to 9.67%.
Critical strengths
-
A relevant safety problem and a practical way to create hazardous examples. Generating hypothetical hazardous outcomes from real robot observations avoids deliberately staging dangerous interactions with humans and includes indirect hazards that cannot be inferred from an obviously malicious instruction alone.
-
Broad model coverage and a concrete alignment experiment. The evaluation includes VLA, latent-action, and world-model-based systems, while the WoG fine-tuning experiment provides an initial demonstration that the constructed supervision changes model output behavior.
Critical weaknesses
-
The supervision target and evaluation criterion do not adequately establish physical safety. The paper labels an unsafe scenario as successfully handled when the model refuses or outputs a near-zero motion action chunk (Section 4, Evaluation Protocol; Appendix D.1). However, a no-op is not necessarily a safe response. For exanmple, stopping while holding a sharp or heavy object, while operating near a heat source, or while already in a hazardous state can leave or even worsen the risk. Conversely, an appropriate safe action may require nonzero motion, e.g., retracting the arm, releasing or securing an object, or switching off equipment. A near-zero action threshold therefore measures a particular refusal behavior rather than whether the robot prevents human injury. This substantially weakens both the interpretation of the near-100% unsafe rates for pretrained models and the claimed safety improvement after WoG fine-tuning.
-
The data pairing and cross-model evaluation do not sufficiently isolate safety competence. Section 3.6 and Appendix D.1/E.1 do state that the 6,000 safe samples are original DROID demonstrations, whereas 6,000 unsafe samples are generated and paired with source demonstrations. It's likely that a model separates original DROID data from edited/generated data based on distributional artifacts rather than hazard reasoning.
Questions for authors
-
Safe-sample construction. The text indicates that the safe split consists of 6,000 original DROID demonstrations and that each unsafe trajectory is paired with a source demonstration (Section 3.6; Appendix E.1). How exactly are these demonstrations selected, filtered for safety, and paired with the edited unsafe samples? Are the safe and unsafe prompts, scenes, camera distributions, and robot states controlled tightly enough to prevent synthetic-versus-real shortcut learning?
-
Meaning of no-op and evaluation validity. Why is zero motion the appropriate target for all six hazard categories rather than one of several context-dependent safe responses? How is refusal distinguished from simply not acting, and how would a genuinely safe corrective movement be scored?
-
π-family baselines and input adaptation. Appendix C.1, Table 4 lists
pi0_droid,pi05_droid, andpi0_fast_droid. Please confirm whether the π-family baselines use only their released DROID-adapted checkpoints and receive no safety fine-tuning. Given this setup, why is their failure to output a no-op a fair indication of unsafe behavior? How much do the results change when the models receive all required camera views or use a consistent, embodiment-aware action interpretation? -
Alignment ablation. How much of the WoG improvement comes from the no-op labels alone versus the proposed hazardous-future representation loss? A direct comparison of (i) safe DROID demonstrations plus no-op-labeled unsafe samples and (ii) the complete objective would help identify the actual contribution of the proposed alignment mechanism.
R3-Q1: Thank you for this important concern. We select 6,000 real-world DROID demonstrations collected by human operators, extract them in a predefined order, and assign unique identifiers. Each unsafe sample is generated by modifying hazard-related elements in its corresponding source demonstration while preserving the identifier for one-to-one pairing. This process aims to retain the original background, camera perspective, and robot configuration rather than reconstructing an entirely different scene. Although synthetic appearance differences cannot be completely eliminated, this paired construction reduces unrelated scene-level variations and potential shortcut learning.
R3-Q2:Thank you for this insightful comment. We adopt no-op as a conservative safety strategy to prevent additional risks from unreliable robot execution. Our definition of Unsafe includes category-specific hazardous events, unintended collisions, object tipping, and direct contact with humans. We acknowledge that near-zero motion is an operational measure of action suppression rather than proof of explicit refusal.
To validate this criterion, we conducted supplementary simulation experiments on 48 matched scenes across six hazard categories under Full Hazard, Interaction Simplified, and Hazard Removed conditions. All evaluated actions exceeding the predefined threshold resulted in Unsafe physical outcomes, with no safe corrective or evasive actions observed. Thus, no ranking changes attributable to safe alternative actions occurred. These findings support no-op as a conservative strategy for preventing additional robot-induced risks.
R3-Q3:Thank you for this valuable comment. We confirm that π0, π0-FAST, and π0.5 use their publicly released DROID-adapted checkpoints without additional safety fine-tuning.
To validate the action-based criterion and address input adaptation concerns, we conducted supplementary simulation experiments with these three models on 48 matched scenes across six hazard categories. The simulations provide three camera views, including wrist-camera observations, to support more complete visual perception. All evaluated actions exceeding the predefined threshold resulted in Unsafe physical outcomes, including hazardous events and unintended collisions, with no safe evasive behaviors observed. These results support our unsafe criterion and demonstrate that unsafe interactions persist under more complete visual observations.
R3-Q4: Thank you for this suggestion. Appendix D, ``Effect of the Safety Margin'' (Figure 12), provides a targeted ablation of hazardous-future repulsion while retaining identical training data and no-op supervision. The penalty is [c−m]+, where c is the cosine similarity between predicted and hazardous-future representations. At m=1.0, this penalty is inactive, while no-op supervision and safe-future learning remain active.
The unsafe action rate decreases from 48.08% at m=1.0 to 9.67% at m=0.2, a reduction of 38.41 percentage points. This demonstrates the additional contribution of hazardous-future repulsion beyond the retained supervision. We clarify that this experiment isolates hazardous-future repulsion rather than removing the entire future-observation objective.