acceptodds
Under review as a conference paper at ICLR 2027

RoboShackles: A Multilingual Safety Dataset for Human-Injury Prevention in Embodied Foundation Models

Abstract

Embodied Foundation Models (EFMs) integrate multimodal perception, future-state reasoning, and executable robot control. However, their safety alignment for preventing human injury remains largely underexplored, primarily because real-world data involving robots harming humans or creating hazardous household situations cannot be collected safely or ethically. To address this challenge, we propose a controllable safety-data construction pipeline, that transforms real-world DROID observations into safety-critical robotic rollouts. Specifically, we generate scenario images containing potential hazards, create hazard-triggering action prompts, and synthesize videos depicting hazardous events. Using this pipeline, a DROID-based robotic video dataset, RoboShackles, is constructed comprising 12,000 training clips and 1,200 hazardous multilingual evaluation clips across six safety categories. Based on this dataset, we conduct a comprehensive evaluation of fifteen representative EFMs and further align WoG with motion-stopping supervision for safety. Experimental results reveal that existing EFMs are highly prone to generating actions that could cause human harm. Moreover, fine-tuning with RoboShackles reduces the unsafe action rate from 98.92% to 9.67%, substantially improving the practical safety of EFMs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Safety, Dataset

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZFEdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RoboShackles, a dataset targeting human-injury prevention for embodied foundation models. Starting from real DROID robot observations, it constructs hazardous scenes through image editing and uses video generation to synthesize unsafe future rollouts. The resulting dataset comprises 6,000 original safe DROID demonstrations and 6,000 synthetically generated unsafe training samples, together with 1,200 multilingual unsafe evaluation samples spanning two direct-harm and four indirect-harm categories. The paper evaluates 15 embodied models using a no-op-based action safety criterion and fine-tunes World Guidance (WoG) to predict a no-op for unsafe inputs. It reports a reduction in WoG's unsafe-action rate from 98.92% to 9.67%.

Critical strengths
  1. A relevant safety problem and a practical way to create hazardous examples. Generating hypothetical hazardous outcomes from real robot observations avoids deliberately staging dangerous interactions with humans and includes indirect hazards that cannot be inferred from an obviously malicious instruction alone.

  2. Broad model coverage and a concrete alignment experiment. The evaluation includes VLA, latent-action, and world-model-based systems, while the WoG fine-tuning experiment provides an initial demonstration that the constructed supervision changes model output behavior.

Critical weaknesses
  1. The supervision target and evaluation criterion do not adequately establish physical safety. The paper labels an unsafe scenario as successfully handled when the model refuses or outputs a near-zero motion action chunk (Section 4, Evaluation Protocol; Appendix D.1). However, a no-op is not necessarily a safe response. For exanmple, stopping while holding a sharp or heavy object, while operating near a heat source, or while already in a hazardous state can leave or even worsen the risk. Conversely, an appropriate safe action may require nonzero motion, e.g., retracting the arm, releasing or securing an object, or switching off equipment. A near-zero action threshold therefore measures a particular refusal behavior rather than whether the robot prevents human injury. This substantially weakens both the interpretation of the near-100% unsafe rates for pretrained models and the claimed safety improvement after WoG fine-tuning.

  2. The data pairing and cross-model evaluation do not sufficiently isolate safety competence. Section 3.6 and Appendix D.1/E.1 do state that the 6,000 safe samples are original DROID demonstrations, whereas 6,000 unsafe samples are generated and paired with source demonstrations. It's likely that a model separates original DROID data from edited/generated data based on distributional artifacts rather than hazard reasoning.

Questions for authors
  1. Safe-sample construction. The text indicates that the safe split consists of 6,000 original DROID demonstrations and that each unsafe trajectory is paired with a source demonstration (Section 3.6; Appendix E.1). How exactly are these demonstrations selected, filtered for safety, and paired with the edited unsafe samples? Are the safe and unsafe prompts, scenes, camera distributions, and robot states controlled tightly enough to prevent synthetic-versus-real shortcut learning?

  2. Meaning of no-op and evaluation validity. Why is zero motion the appropriate target for all six hazard categories rather than one of several context-dependent safe responses? How is refusal distinguished from simply not acting, and how would a genuinely safe corrective movement be scored?

  3. π-family baselines and input adaptation. Appendix C.1, Table 4 lists pi0_droid, pi05_droid, and pi0_fast_droid. Please confirm whether the π-family baselines use only their released DROID-adapted checkpoints and receive no safety fine-tuning. Given this setup, why is their failure to output a no-op a fair indication of unsafe behavior? How much do the results change when the models receive all required camera views or use a consistent, embodiment-aware action interpretation?

  4. Alignment ablation. How much of the WoG improvement comes from the no-op labels alone versus the proposed hazardous-future representation loss? A direct comparison of (i) safe DROID demonstrations plus no-op-labeled unsafe samples and (ii) the complete objective would help identify the actual contribution of the proposed alignment mechanism.

1 comment

R3-Q1: Thank you for this important concern. We select 6,000 real-world DROID demonstrations collected by human operators, extract them in a predefined order, and assign unique identifiers. Each unsafe sample is generated by modifying hazard-related elements in its corresponding source demonstration while preserving the identifier for one-to-one pairing. This process aims to retain the original background, camera perspective, and robot configuration rather than reconstructing an entirely different scene. Although synthetic appearance differences cannot be completely eliminated, this paired construction reduces unrelated scene-level variations and potential shortcut learning.

R3-Q2:Thank you for this insightful comment. We adopt no-op as a conservative safety strategy to prevent additional risks from unreliable robot execution. Our definition of Unsafe includes category-specific hazardous events, unintended collisions, object tipping, and direct contact with humans. We acknowledge that near-zero motion is an operational measure of action suppression rather than proof of explicit refusal.

To validate this criterion, we conducted supplementary simulation experiments on 48 matched scenes across six hazard categories under Full Hazard, Interaction Simplified, and Hazard Removed conditions. All evaluated actions exceeding the predefined threshold resulted in Unsafe physical outcomes, with no safe corrective or evasive actions observed. Thus, no ranking changes attributable to safe alternative actions occurred. These findings support no-op as a conservative strategy for preventing additional robot-induced risks.

R3-Q3:Thank you for this valuable comment. We confirm that π0\pi_0, π0\pi_0-FAST, and π0.5\pi_{0.5} use their publicly released DROID-adapted checkpoints without additional safety fine-tuning.

To validate the action-based criterion and address input adaptation concerns, we conducted supplementary simulation experiments with these three models on 48 matched scenes across six hazard categories. The simulations provide three camera views, including wrist-camera observations, to support more complete visual perception. All evaluated actions exceeding the predefined threshold resulted in Unsafe physical outcomes, including hazardous events and unintended collisions, with no safe evasive behaviors observed. These results support our unsafe criterion and demonstrate that unsafe interactions persist under more complete visual observations.

R3-Q4: Thank you for this suggestion. Appendix D, ``Effect of the Safety Margin'' (Figure 12), provides a targeted ablation of hazardous-future repulsion while retaining identical training data and no-op supervision. The penalty is [c−m]+[c-m]_{+}, where cc is the cosine similarity between predicted and hazardous-future representations. At m=1.0m=1.0, this penalty is inactive, while no-op supervision and safe-future learning remain active.

The unsafe action rate decreases from 48.08% at m=1.0m=1.0 to 9.67% at m=0.2m=0.2, a reduction of 38.41 percentage points. This demonstrates the additional contribution of hazardous-future repulsion beyond the retained supervision. We clarify that this experiment isolates hazardous-future repulsion rather than removing the entire future-observation objective.

Reviewer_39buOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a controllable pipeline that converts real DROID observations into synthetic safety-critical robot videos, and uses it to build RoboShackles: 12,000 training clips (6,000 safe DROID demonstrations plus 6,000 generated unsafe trajectories) and 1,200 multilingual evaluation clips across six categories (direct hand/human; indirect fire/electrical/water/falling). The pipeline uses Qwen3-VL for captions and edit instructions, Qwen-Image for edited initial frames, and Wan2.7 for future rollouts. The paper evaluates fifteen VLA, latent-action, and world-action models using a no-op-based "unsafe action rate" and normalized AUC, then fine-tunes WoG with a mixed safe/unsafe objective that replaces unsafe actions with no-ops and adds a representation-repulsion loss. It reports reducing the unsafe action rate from 98.92% to 9.67%, with utility measured on SimplerEnv.

Critical strengths
  • The resource is substantial and well motivated. Transforming real DROID observations into paired safe/hazardous rollouts is a sensible way to study human-injury scenarios that cannot be collected ethically. The scale (13,200 clips), the six-category taxonomy, the multilingual prompts, and the planned release of videos, annotations, split manifests, and generation prompts make this a potentially valuable community resource. The safety-margin ablation and the utility evaluation also show that the authors are thinking beyond a single headline number.
  • The alignment experiment is executed with unusual care. The paper reports three seeds, a Wilson confidence interval, an exact McNemar test, an explicit no-op target, a representation-repulsion loss, and SimplerEnv utility results. It also reports residual failures (9.67% unsafe, falling risk hardest) and acknowledges utility loss rather than hiding them. This level of reporting is commendable and makes the alignment result worth taking seriously.
Critical weaknesses
  • The central metric is not a validated measure of harm, and the evaluation is one-class with no positive control.

  • Dataset quality and alignment evidence are circular and confounded.

  • The multilingual, novelty, and provenance claims exceed the evidence.

Questions for authors
  • For each of the fifteen models, what is task success or action plausibility on matched benign DROID scenes under the exact Table 4 adaptation? If these models cannot perform the benign task, how should the 100% unsafe rates be interpreted?
  • Why does the evaluation set contain only unsafe clips? Can you add matched held-out safe DROID clips and generated-safe clips, and report safe acceptance or false refusal alongside unsafe action rate?
  • What prior work or empirical validation supports treating no-op as safe, and how do you establish that "non-no-op" equals "unsafe"? Please report human harmfulness judgments or trajectory matching on a sample, and show what happens to the ranking when a model emits a safe alternative action (retreat, close a valve, move an object away).
  • Was m = 0.2 selected on the same 1,200 evaluation clips used for the headline result, and how was the step-500 checkpoint chosen? Are the evaluation source DROID videos disjoint from the 6,000 training sources? Please provide a held-out validation split and a source-level split manifest, and re-report the headline numbers.
  • What are the pre-filter PSP, TAC, RSS, and TC distributions and the per-category rejection rates? How many annotators were used, with what qualifications, blinding, and inter-annotator agreement?
  • Who translated the prompts, and were the translations validated by native speakers? Are the per-language clips the same scenes? Please report per-language confidence intervals or significance tests and a direct test of the "linguistic distance" hypothesis. Why does VITA appear in Table 3 but not in the fifteen evaluated models? Which rows are your runs and which are borrowed, and under what protocol and seeds?
  • Can you ablate the no-op target and the representation loss, compare against safe-data-only fine-tuning and a SafeVLA-style constrained baseline, and report the safety-utility trade-off for the margin sweep?
  • What access controls, content warnings, watermarking or provenance metadata, and misuse monitoring will accompany the CC BY 4.0 release of realistic violent synthetic videos and the generation pipeline? How will you handle any real people in DROID-derived scenes?
1 comment

R2-Q1: Thank you for this important concern. All fifteen models have demonstrated manipulation capabilities in their original benchmarks, with several trained or adapted on real-world Franka demonstrations. Thus, the high unsafe action rates primarily reflect the occurrence of unsafe behaviors rather than a lack of basic manipulation ability.

R2-Q2:Thank you for this valuable suggestion. To address this concern, we conducted supplementary simulation experiments with DeFi, FastWAM, π0\pi_0, π0\pi_0-FAST, and π0.5\pi_{0.5} on 48 matched scenes across six hazard categories, using three camera views. We evaluated three conditions: Full Hazard, Interaction Simplified, and Hazard Removed. In particular, Hazard Removed serves as a matched benign control by removing designated hazards while preserving the original tasks and scene context. Compared with independently selected safe DROID clips, this setting better isolates the effects of hazardous conditions and enables a more direct assessment of task execution and physical risks beyond the original unsafe action rate.

R2-Q3:Thank you for this insightful comment. We adopt no-op as a conservative safety strategy to prevent additional risks from unintended robot interactions. Our definition of Unsafe includes both category-specific hazardous events and unintended collisions with surrounding objects, object tipping, and direct contact with humans. To validate this criterion, we conducted supplementary simulation experiments with DeFi, FastWAM, π0\pi_0, π0\pi_0-FAST, and π0.5\pi_{0.5} on 48 matched scenes across six hazard categories under Full Hazard, Interaction Simplified, and Hazard Removed conditions. All evaluated actions exceeding the predefined threshold resulted in Unsafe outcomes, with no safe alternative actions observed. Thus, no ranking changes attributable to safe alternatives occurred. These findings support no-op as a conservative strategy for preventing additional risks arising from unreliable robot execution. R2-Q4:Thank you for raising these concerns. Both m=0.2m=0.2 and the step-500 checkpoint were selected through empirical comparisons, with the former achieving the lowest unsafe action rate in Appendix D and the latter performing best across evaluated training stages. We acknowledge that the margin selection used the original 1,200 evaluation clips and will clarify this limitation. Each DROID source demonstration is assigned a unique identifier, preserved throughout unsafe sample generation. This enables source-level overlap verification, and we will provide the corresponding split manifest to improve reproducibility.

R2-Q5:Thank you for this important question. All authors participated in sample screening and quality evaluation, with the first author conducting the final review to ensure consistent application of the predefined criteria. Samples failing the PSP, TAC, RSS, or TC requirements were excluded

R2-Q6:Thank you for these comments. All prompts were manually translated by the authors and reviewed using AI models. DROID scenes were divided by unique identifiers into four non-overlapping language subsets, each containing 300 samples. We computed 95% Wilson confidence intervals for each language and will include them in the revised results. Since languages correspond to different scenes, we will avoid causal claims regarding linguistic distance.

VITA's results were taken from the original WoG paper. As no suitable Franka-compatible checkpoint was available, we excluded VITA from our evaluation and removed its entry from Table 3 to ensure consistency.

R2-Q7: Appendix D provides a targeted ablation of hazardous-future repulsion while retaining no-op supervision. Disabling this penalty at m=1.0m=1.0 increases the unsafe action rate from 9.67% to 48.08%, demonstrating its additional contribution. The margin sweep also reports the corresponding safety--utility trade-off.

Regarding SafeVLA, a direct comparison is challenging due to fundamental differences in training paradigms. SafeVLA employs online constrained reinforcement learning in AI2-THOR with discrete actions, whereas WoG uses offline diffusion training with continuous actions. Adapting SafeVLA would require substantial modifications to the environment, safety costs, and training pipeline, potentially introducing confounding factors and compromising comparison fairness. We will clarify this limitation without claiming superiority over SafeVLA.

R2-Q8: All DROID samples containing real people were excluded during dataset construction, and human-related hazardous scenarios are entirely synthetic.

To mitigate misuse, we will publicly release only the test set and safety-aligned WoG weights, while the training set and generation pipeline will require application-based approval. We will also provide content warnings and provenance metadata to identify synthetic content. These measures aim to support reproducible safety research while limiting potential misuse.

Reviewer_9EDcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces RoboShackles, a multilingual safety dataset and safety alignment framework designed to prevent human injury caused by Embodied Foundation Models (EFMs). Building upon real-world robotic manipulation data from DROID, the authors develop an automated data generation pipeline using Qwen3-VL, Qwen-Image, and Wan2.7 to synthesize safety-critical robotic scenarios and future video rollouts across two types of harm, direct and indirect, covering six safety categories. The dataset comprises 12,000 training samples and 1,200 multilingual evaluation samples. The authors evaluate 15 representative EFMs and further perform safety fine-tuning on World Guidance (WoG). Experimental results show that fine-tuning reduces WoG's unsafe action generation rate from 98.92% to 9.67%, while preserving a certain level of general robotic manipulation capability.

Critical strengths
  1. A well-structured safety data generation pipeline. The authors develop a comprehensive pipeline that combines multimodal scene understanding, image editing, and video generation to construct safety-critical scenarios from real DROID robotic observations, without physically reproducing hazardous events. The dataset covers six safety categories and four languages, with both human verification and automated quality metrics used to screen generated samples. This provides a scalable approach to constructing safety-critical datasets for embodied AI.

  2. Extensive experimental evaluation with consideration of the safety–utility trade-off. The paper evaluates 15 EFMs with diverse architectures and demonstrates the potential value of the proposed dataset through safety fine-tuning of WoG. Furthermore, the authors assess general manipulation performance on the Google Robot and WidowX Robot tasks in SimplerEnv, examining whether safety alignment compromises the models' original task capabilities. These experiments provide a more comprehensive assessment of the proposed approach.

Critical weaknesses
  1. The definition of unsafe action rate is overly simplified and may not accurately reflect actual safety risks. According to the evaluation protocol in Section 4, the authors primarily determine whether an action is unsafe based on its deviation from a no-op action. For continuous-action models, an action is classified as unsafe if the maximum absolute value of its first six motion dimensions exceeds a predefined threshold. However, non-zero actions are not necessarily unsafe, and no-op actions are not necessarily safe. For example, a robot may move its arm away from a hazardous object or perform other evasive actions to prevent harm, yet such actions could still be classified as unsafe under the current metric. Therefore, the nearly 100% unsafe action rates reported for most models in Table 2 may partly reflect their tendency to generate non-zero control actions rather than their actual likelihood of causing harm. The authors are encouraged to incorporate additional safety metrics based on action semantics, environmental state changes, and actual hazardous consequences to provide a more reliable assessment of embodied safety.

  2. The visual plausibility of synthesized videos does not establish the physical causal validity of hazardous events. The paper uses generative video models to synthesize hazardous outcomes following robotic actions and filters the generated samples through human inspection and visual quality metrics. However, visually plausible hazardous videos do not necessarily follow real-world physical laws. Events such as fires, electrical accidents, or falling objects may reflect the generative model's semantic priors rather than the actual physical consequences of the corresponding initial states and robotic actions. Although Table 1 reports scores of 1.000 for Physical-Semantic Plausibility, Task-Adherence Consistency, and Robot-Subject Stability across all retained samples, these metrics primarily assess visual and semantic plausibility and do not establish a physically valid causal relationship between actions and hazardous outcomes. The authors are encouraged to validate representative synthesized scenarios using physics-based simulation or controlled experiments with verifiable causal mechanisms.

Questions for authors

Q1. Could the authors clarify the rationale for using non-zero actions as the primary criterion for unsafe behavior? The current evaluation protocol classifies robotic motions exceeding a predefined threshold as unsafe, although certain non-zero actions may represent safe evasive behaviors. Could the authors provide additional evaluations based on action semantics, environmental state changes, or actual hazardous consequences to demonstrate that the reported unsafe action rates reliably reflect the real safety risks of embodied agents?

Q2. Could the authors further disentangle the contributions of no-op supervision and hazardous-future representation learning? The proposed training method directly replaces the target actions of unsafe samples with no-op actions, while the evaluation also relies primarily on a no-op criterion. Consequently, the substantial safety improvement may result from learning to suppress actions rather than understanding hazardous consequences. Could the authors provide separate ablation studies for these components and evaluate the model under identical task instructions but different safety conditions to verify whether it has acquired context-sensitive safety decision-making capabilities?

1 comment

R1-Q1: We thank the reviewer for this insightful comment. Our unsafe action criterion is specifically designed for hazardous tasks. Importantly, our definition of Unsafe encompasses both category-specific hazardous events and unintended physical interactions, including collisions with tables, walls, other non-target objects, and direct contact between the robotic arm and humans. Under this evaluation setting, non-zero actions may introduce physical risks, even when designated hazards are removed.

To validate our original criterion, we conducted three complementary simulation-based experiments: (1) Full Hazard, which preserves the original hazardous scenarios; (2) Interaction Simplified, which retains hazardous elements while reducing manipulation difficulty; and (3) Hazard Removed, which removes designated hazards while preserving ordinary manipulation tasks. We evaluated DeFi, FastWAM, π0\pi_0, π0\pi_0-FAST, and π0.5\pi_{0.5} on 48 matched scenes, with eight scenes per hazard category.

Our supplementary experiments demonstrate that all evaluated actions exceeding the predefined threshold resulted in Unsafe physical outcomes, with no safe evasive behaviors observed. These findings provide empirical support for our original action-based unsafe criterion, demonstrating its consistency with actual physical risks in the evaluated hazardous scenarios and reinforcing the validity of the reported unsafe action rates.

R1-Q2. Thank you for this valuable comment. We agree that distinguishing the contribution of hazardous-future representation learning from no-op supervision is important. Appendix D, ``Effect of the Safety Margin'' (Figure 12), provides a targeted ablation in which we vary the safety margin mm while keeping no-op supervision and other training settings unchanged. The hazardous-future repulsion term is defined as [c−m]+[c-m]_{+}, where cc denotes the cosine similarity between the predicted and hazardous-future representations. At m=1.0m=1.0, this penalty becomes inactive, while no-op supervision and safe-future learning remain active. Under the same evaluation protocol, the unsafe action rate decreases from 48.08% at m=1.0m=1.0 to 9.67% at m=0.2m=0.2, a reduction of 38.41 percentage points. This result demonstrates the additional contribution of hazardous-future repulsion beyond no-op supervision.

To further examine whether the model responds appropriately to different safety conditions, we conducted supplementary SAGE-assisted simulation experiments on 48 matched scenes across six hazard categories. We evaluated three conditions: Full Hazard, Interaction Simplified, and Hazard Removed, with the task instructions preserved across conditions. The evaluation considers both task execution and actual physical outcomes, including category-specific hazardous events and unintended collisions. The results show that all evaluated actions exceeding the predefined threshold resulted in Unsafe outcomes, with no safe evasive behaviors observed. These findings provide additional support for our action-based safety criterion and enable a more comprehensive assessment of execution reliability and physical risk beyond no-op supervision alone.

Comments on the paper

1 comment

We sincerely thank the Area Chair and reviewers for their valuable feedback and constructive suggestions. We have carefully addressed all reviewer comments, conducted additional experiments where necessary, and revised the manuscript accordingly. Detailed point-by-point responses and corresponding revisions are provided to clarify the concerns raised and strengthen the rigor and reproducibility of our work. We greatly appreciate the time and effort devoted to evaluating our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.