- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses a practically relevant problem, but its distinct contribution over existing safety benchmarks is insufficiently established. While the experiments show low safe task success rates, the analysis needs to better distinguish failures in risk recognition, temporal planning, and basic manipulation to support specific conclusions about model safety. The paper also omits discussion of RationalVLA, leaving its relationship to relevant prior work unclear. Overall, the justification of novelty and the diagnostic evidence are insufficient to support acceptance.
Critical strengths
- The paper addresses a practically relevant problem by evaluating whether VLA policies can identify and mitigate implicit risks that are not explicitly mentioned in task instructions.
- The benchmark provides a structured evaluation protocol covering seven risk categories and three complementary scenarios: single-risk, two-risk composition, and no-risk.
- The paper distinguishes task success from safe task success and introduces process-level metrics to evaluate mitigation execution, temporal-order compliance, and unnecessary risk responses.
- The evaluation covers four representative VLA models and reveals substantial performance degradation in safety-critical scenarios, particularly when multiple risks must be handled simultaneously.
Critical weaknesses
- Limited novelty over existing safety benchmarks. The paper shares substantial similarities with existing benchmarks such as IS-Bench, SafeVLA-Bench, and LIBERO-Safety, which also investigate risk awareness, safety constraints, and process-level safety evaluation. Although SafeHousehold introduces controlled risk-present and risk-absent scenarios, the authors should more clearly establish what fundamentally new capabilities or evaluation insights are enabled by their benchmark.
- Insufficient disentanglement of safety reasoning and manipulation capabilities. The evaluated models exhibit relatively low task success rates even without considering safety requirements. For example, the strongest model achieves only 39.0% task success in the single-risk scenario. Consequently, low safe task success rates may partly reflect limitations in basic manipulation rather than failures in safety reasoning. Although the paper provides failure-mode statistics, further controlled experiments are needed to isolate these factors.
- Limited evaluation of compositional generalization. The two-risk evaluation includes only four risk combinations, with 20 episodes per combination. Moreover, two combinations introduce additional scene shifts, potentially confounding multi-risk reasoning with out-of-distribution generalization. A broader and more controlled evaluation would strengthen the conclusions regarding multi-risk composition.
- Limited coverage of risk-absent scenarios. The no-risk evaluation covers only ten task templates from three of the seven risk categories. As a result, it remains unclear whether the observed unnecessary risk responses generalize across the full risk taxonomy.
Questions for authors
see weaknesses.