- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article introduces HazardArena, a benchmark for evaluating the semantic safety of visual-language-action (VLA) models, aiming to study whether the models can adjust their action behaviors based on the risk information in the scenes and instructions. This benchmark covers seven types of safety risks and 51 dangerous scenarios. By constructing safety and danger contrast tasks that match the motion requirements and designing three-stage indicators of Attempt, Commit, and Success, it distinguishes the models' safety awareness from their action execution capabilities. Experiments were conducted on models such as OpenVLA-OFT, VLA-Adapter, NORA, π₀, and π₀.₅, revealing that although fine-tuning with only safety task data improved the success rate of normal tasks, it also increased the execution rate of corresponding dangerous tasks. The authors further evaluated two mitigation strategies: external VLM safety filtering and trajectory-enhanced rejection fine-tuning. They verified the diagnostic value of the benchmark and the aforementioned phenomenon through ablation analysis, multiple random seeds evaluation, and real robot experiments with the Franka Panda.
Critical strengths
-
The article constructs safety and danger scenarios that match the action requirements as much as possible to minimize the influence on the robot's operational capabilities, thereby enabling a more accurate assessment of whether the VLA model can adjust its behavior based on semantic risks. Moreover, the three-stage indicators of Attempt, Commit, and Success can reveal dangerous behaviors that might be overlooked when relying solely on the final task success rate.
-
The experiments cover a wide range and the authors observed consistent phenomena on multiple representative VLA models: using only safety task data for fine-tuning can improve the model's ability to perform normal tasks while also potentially enhancing its ability to perform dangerous tasks. The article further validates this phenomenon through multiple random seeds evaluation, additional tasks, two safety mitigation strategies, and real robot experiments, providing strong empirical support for the main findings.
Critical weaknesses
-
This article presents a benchmark for the security assessment of VLA, but the revelations it brings are actually not new. An un-safely aligned VLA will definitely accept all instructions. Their output is actions that have no reason to refuse or deliberately fail. The two proposed defense strategies are also relatively simple. So I think what the author is claiming, "By doing SFT on safe tasks, the success rate in malicious tasks has also increased", is not something interesting or novel. In fact, it is somewhat misleading. The VLA model has no idea what is safe and what is malicious at all. Whether he can accept this task and complete it depends solely on whether he was familiar with this trajectory during his training stage. If a model that has undergone safety alignment still exhibits such a phenomenon, then I would consider this claim to be interesting and valid. For instance, in a previous community study, an aligned LLM was fine-tuned on some security or personalized datasets, and as a result, its probability of rejecting malicious content also decreased.
-
For instance, the tasks were somewhat forced: putting a capsule into a wine glass, throwing something into the trash can. These "hazards" only make sense when considering the context clearly (For instance, the capsule might be toxic, or the object might not be discarded but is actually very valuable, because the ring could also be something that the user no longer wants and thus needs to be disposed of); otherwise, I don't think putting a capsule into a wine glass is a malicious act that needs to be rejected.
-
I noticed that the anonymous repository seems to be incomplete; some codes are not provided. For example, "unseen tasks" are not truly implemented in the code: the get_unseen_task_config and seen functions return the same configuration, and the relevant generalization statements need to be verified.
-
The text was leaked, and what was measured might be the key words rather than the actual understanding of security. "Gently, safely, for a safe demo" only appear in the safe execution scenarios, and never in dangerous instructions. Therefore, there is a possibility that the model can learn shortcuts: execute based on seeing these words without understanding the context. I speculate that if you add malicious instructions to these "safely" keywords, such as safely directing the knife towards that baby, the damage rates of all aligned models will increase. This is the effect brought about by the shortcut. I think for the sake of rigor, we can uniformly remove all of them.
-
Regarding the reliability of the determination and tasks, COMMIT is a geometric determination based on the distance radius. However, in the oven task, the script never triggered COMMIT even once (0/50) when performing dangerous actions. This should be a bug, and the microwave oven task is basically unusable. The safety scenario 1 of the socket task uses the placeholder sentence "insert the object into the target". These flaws will significantly reduce the value of the benchmark for the community.
minor: 1.The resolution of Figure 3 is not clear.
Questions for authors
-
If we were to re-examine the argument of the article on the aligned VLA model, "By making minor adjustments in safe tasks, his success rate in malicious tasks has increased" would still hold true?
-
If the task of placing the capsule into the wine glass is also classified as a malicious task, it is likely to cause the model to overly reject some benign tasks. How can this trade-off be balanced?
-
What is the purpose of adding "Gently, safely, for a safe demonstration" to some security tasks? Is it because these tasks (such as those involving sensitive objects) are more likely to be rejected? So that's why it was done as a relief measure?