- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LIBERO-Risk is proposed as an automated construction pipeline that injects safety constraints into existing manipulation tasks. Compared with existing safety-related benchmarks, the pipeline is fully automated, achieves a construction success rate of roughly 75%, and can be applied to existing tasks while preserving the original task semantics. Its evaluation goes beyond task success to include a process-level safety score.
Critical strengths
- Compared with existing safety-related benchmarks, this pipeline is automated, achieves a success rate of approximately 75%, and can be applied to existing tasks while preserving the original task intent. The mutation mechanisms include forbidden-object layout (FOL), physical blocking (PB), and trajectory-bound hazard layout (TBHL).
- The evaluation incorporates both task success rate and process-level safety score, deriving five process-safety measurements that characterize contact, spatial proximity, and temporal risk.
Critical weaknesses
- The risk types are limited and far removed from real-world usage. Whether such an automated safety pipeline is even needed also remains open to question.
- The experiments cover only basic models and do not include recent state-of-the-art models, such as π0.5.
- The evaluation is restricted to LIBERO, which is relatively outdated, and thus cannot sufficiently support the robustness of the automated pipeline.
Questions for authors
- Consider more useful risk types that reflects real world application.
- Evaluation on other dataset except for LIBERO series.
- Add more recent models into the experiments.
I appreciate the authors' detailed response. I understand this takes quite a bit of effort. Unfortunately, I am going to keep my score as is. Two hurdles are stopping me from raising: 1. If pure collision avoidance is the primary concern, there are quite strong classical methods to accomplish this, e.g. a control barrier function as a safety layer modifying the VLA output (operating on a mono-to-depth model from the image). If it is just collision avoidance, there is no need for the semantic nature of the benchmark. 2. The engineer deep inside me says this is not why robot's fail. They fail due to cascading system level errors and interactions that cannot be replicated in a simulation benchmark with a closed object set. It just not rich enough to find to real system faults. I applaud the research direction and wish the author team the best of luck!