acceptodds
Withdrawn from ICLR 2027

SafeHousehold: Benchmarking Vision-Language-Action Models for Household Manipulation with Implicit Risks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Household robots must perform everyday tasks safely in visually rich environments, yet task instructions often omit risk conditions that determine the appropriate action sequence. We define an implicit risk as an instruction-omitted, visually observable condition whose presence requires a corresponding mitigation. For example, given the instruction to put an apple on the plate, a policy should clean a dirty plate first but place the apple directly when it is clean. Existing benchmarks do not jointly evaluate whether closed-loop VLA policies infer such risks, execute mitigations in the required temporal order, and avoid unnecessary responses when risks are absent. We introduce SafeHousehold, a household manipulation benchmark in which policies receive only the task instruction, RGB observations, and a proprioceptive state, without risk labels, mitigation descriptions, or safety specifications. SafeHousehold contains 35 task templates across seven risk categories and 1,750 manually reviewed human-teleoperated demonstrations. We evaluate four representative VLA models under single-risk, two-risk composition, and no-risk scenarios. The protocol distinguishes task success from safe task success with process-level metrics, using automatically computed scores for reproducible model comparison and human-reviewed diagnostics for failure analysis. In the single-risk scenario, policies often initiate a risk response without completing the mitigation; even the best-performing model achieves 39.0% task success but only 27.3% safe task success across seven risk categories. Safe task success is nearly absent in the two-risk composition scenario, while unnecessary risk responses remain frequent in the no-risk scenario. These results identify mitigation execution, responses to multiple risks, and risk-conditioned behavior as key bottlenecks in current VLA policies.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language-Action Models, Embodied Safety, Robot Manipulation, Implicit Risks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rRomOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a practically relevant problem, but its distinct contribution over existing safety benchmarks is insufficiently established. While the experiments show low safe task success rates, the analysis needs to better distinguish failures in risk recognition, temporal planning, and basic manipulation to support specific conclusions about model safety. The paper also omits discussion of RationalVLA, leaving its relationship to relevant prior work unclear. Overall, the justification of novelty and the diagnostic evidence are insufficient to support acceptance.

Critical strengths
  1. The paper addresses a practically relevant problem by evaluating whether VLA policies can identify and mitigate implicit risks that are not explicitly mentioned in task instructions.
  2. The benchmark provides a structured evaluation protocol covering seven risk categories and three complementary scenarios: single-risk, two-risk composition, and no-risk.
  3. The paper distinguishes task success from safe task success and introduces process-level metrics to evaluate mitigation execution, temporal-order compliance, and unnecessary risk responses.
  4. The evaluation covers four representative VLA models and reveals substantial performance degradation in safety-critical scenarios, particularly when multiple risks must be handled simultaneously.
Critical weaknesses
  1. Limited novelty over existing safety benchmarks. The paper shares substantial similarities with existing benchmarks such as IS-Bench, SafeVLA-Bench, and LIBERO-Safety, which also investigate risk awareness, safety constraints, and process-level safety evaluation. Although SafeHousehold introduces controlled risk-present and risk-absent scenarios, the authors should more clearly establish what fundamentally new capabilities or evaluation insights are enabled by their benchmark.
  2. Insufficient disentanglement of safety reasoning and manipulation capabilities. The evaluated models exhibit relatively low task success rates even without considering safety requirements. For example, the strongest model achieves only 39.0% task success in the single-risk scenario. Consequently, low safe task success rates may partly reflect limitations in basic manipulation rather than failures in safety reasoning. Although the paper provides failure-mode statistics, further controlled experiments are needed to isolate these factors.
  3. Limited evaluation of compositional generalization. The two-risk evaluation includes only four risk combinations, with 20 episodes per combination. Moreover, two combinations introduce additional scene shifts, potentially confounding multi-risk reasoning with out-of-distribution generalization. A broader and more controlled evaluation would strengthen the conclusions regarding multi-risk composition.
  4. Limited coverage of risk-absent scenarios. The no-risk evaluation covers only ten task templates from three of the seven risk categories. As a result, it remains unclear whether the observed unnecessary risk responses generalize across the full risk taxonomy.
Questions for authors

see weaknesses.

Reviewer_d2ZoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a safety benchmark for VLA models in household scenarios. The benchmark covers 35 task templates across seven risk categories. The scenarios cover single-risk, two-risk composition, and no-risk cases. The benchmark was used to fine-tune and evaluate four representative VLA models, measured by both task success and safety metrics. Experimental results show that fine-tuned VLAs often initiate risk responses without completing the required mitigations, rarely achieve safe task success in two-risk compositions, and often exhibit unnecessary risk responses in the no-risk scenario.

Critical strengths
  • Coverage of the benchmark. The proposed benchmark covers a broad range of risk categories, mostly grounded in existing regulations.
Critical weaknesses
  • The underlying scientific question is unclear. The paper proposes the benchmark without clearly explaining which capabilities of VLAs are evaluated. For example, safety awareness, the capability to reject when risks are identified, or disentangling the composition of risks? I don't think the benchmark design follows clear construct validity.

  • The evaluation protocol is not fully convincing. The proposed benchmark is used to fine-tune VLA models, and the results show their deficiencies. However, it is unclear whether the training configurations are optimized by jointly considering the VLA and the benchmark characteristics.

  • The main takeaway is weak. The proposed benchmark was not tested on SOTA models. The deficiencies of the fine-tuned models may be due to the limited model size.

  • Lack of OOD evaluation. While an OOD evaluation is necessary if the models perform well, the experiment should at least cover this aspect for comparison with learning on the full taxonomy.

Questions for authors
  • Position of the proposed benchmark. In the related work section, several benchmarks for embodied agents have been mentioned. Would the risks be mitigated if there is a planning module based on a strong model, e.g., the SOTA GPT model?
Reviewer_m3GfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces “SafeHousehold”, a benchmark for evaluating whether VLA policies are capable of identifying and preventing visually observable risks that are not present in task instruction. The benchmark has 35 task templates across multiple risk categories and 1750 human teleop demonstrations. The paper evaluates on four VLA policies using evaluation metrics that distinguish between task completion and safe completion. The results show that policies often fail to mitigate the risk and have near-zero task success when the risks are stacked.

Critical strengths
  • This paper focuses on an interesting and relevant problem setting. The benchmark addresses visually observable risks, thus demonstrating that current policies are often unable to handle such risks.
  • This paper provides multiple evaluation metrics beyond task success. The protocol distinguishes between task completion, risk mitigation success, and ordering.
Critical weaknesses
  • The paper is missing a critical baseline (as currently presented). The authors should consider what happens when the risk is provided in the task instruction. For instance, in the dirty plate example, the task instruction would be something like “Put the apple on the plate, but first clean the plate.” or “Put the apple on the plate. The plate is dirty.” These variants would test whether making the risk more explicit or the mitigation more explicit would improve both the task performance and risk mitigation.
  • The conclusions on unnecessary response are a bit confounded. In particular, the training data seems to be heavily skewed to demonstrations containing risks. So, it would make sense that the fine-tuned policy seems to attempt risks even when risks are not present.
  • The tasks themselves have limited diversity. Many of the tasks involve cutting different food items and then placing the knife in the sink. The diversity of the tasks is a bit overstated in this paper.
  • Some of the safety specifications require a particular solution, rather than a general safety-related issue. For instance, cutting tasks require placing the knife in the sink and chemical hazards seem to require specific locations for placing objects. What would happen if there were alternative, but safe outcomes?
Questions for authors
  • Can you evaluate the baselines with the baseline variants discussed in the “Weaknesses” section? It is natural to compare how well the policies perform when they are explicitly provided explicit safety instructions to separate how much of the issue should be “obvious” to a robot vs. what is possible if the safety specification is made explicit.
  • Can you provide evidence of unnecessary response when the policy is finetuned, but not on such a skewed dataset in terms of risk-related behavior?
  • Can you respond to the last two weaknesses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.