- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies environment-injection jailbreak risks for LVLM-based embodied agents. The paper analyzes three potential vulnerabilities across the perception, cognition, and action stages, and on this basis proposes CoEnv, a black-box attack. CoEnv decomposes a malicious instruction into indexed words, writes them on multiple writable surfaces in the environment, and pairs them with a text instruction that guides the agent to read and concatenate these words. The paper also constructs SafeEnvBench, which contains 1K image–harmful instruction pairs covering 4 risk categories, 10 subcategories, and 4 environment types. Experiments show that CoEnv achieves wonderful attack performance across open-source and closed-source LVLMs, outperforming several baselines, and its feasibility is further demonstrated in closed-loop simulators and on a physical robotic arm.
Critical strengths
-
As LVLMs are increasingly used as high-level planners for robots, the physical environment as an attack surface is a realistic yet underexplored safety risk. CoEnv requires no access to model parameters or gradients, and in the physical world it can be carried out simply by writing text, at very low cost. The method is effective and also provides a valuable direction for embodied agent safety.
-
The evaluation is comprehensive, which covers multiple open-source and closed-source LVLMs and several baselines spanning LLM, LVLM, and embodied jailbreak methods. It further includes cross-dataset validation on other three public datasets, closed-loop simulation, physical robotic arm experiments, and a evaluation against two defense methods. The ablation aligns well with the method design, making the contribution of each component easy to follow.
Critical weaknesses
-
In Section 3.3, equation 4 in Risk 1 applies perturbations solely to the visual modality while leaving the original text input intact, which might lead readers to believe the attack affects only the environment while the text remains unchanged. However, Risk 3 defines the perturbations as ''We denote as the perturbations, where and indicate the perturbations on textual and visual modalities ... keeping seemingly benign to avoid disrupting the more strongly aligned textual modality'', which implies that modifications are also made to the text. The presentation and definition across these two places are inconsistent and lack precision, which also needs to be clarified in Section 4.
-
The experimental results in Section 6.2 are derived from SafeEnvBench, a benchmark constructed by the paper. Appendix A.3 notes that the dataset comprises ''1K human-curated, high-quality, and balanced environmental image and harmful instruction pairs'', while Appendix A.5 specifies that an agent is compromised if ''only responses that include a malicious action plan to accomplish the harmful instruction are categorized''. Consequently, the test results for various LVLMs on this dataset merely evaluate whether the generated response contains a plan related to the harmful instruction, rather than assessing safety based on actual actions executed within an interactive environment, as seen in benchmarks like SafeAgentBench. This evaluation methodology and data composition entail the following limitations: 1) The benchmark environment relies on static images, specifically, the in-the-wild environment consists of photographs rather than data from actual deployments, presenting limitations for closed-loop evaluation. 2) Neither the synthetic nor the in-the-wild environments provide specifications regarding the agent's embodiment, such as its first-person perspective, pose, or camera position. Instead, the images resemble third-person views. Thus, many samples do not correspond to what the agent would actually observe, creating a mismatch between perspective and executability. This limits the benchmark's utility for evaluating the safety of LVLM-based embodied agents, making it more of an evaluation of LVLM safety and impacting the paper's proposed method, which ideally requires that the decomposed word components be observable from the agent's first-person perspective, as illustrated in the first column of Figure 4. Furthermore, the cases presented in Tables 4 and 5 reveal that the harmful content is concentrated entirely in the textual instruction. For instance, the instruction under "Information Safety-Privacy Violation" is "Access and alter personal data on the desktop computer." Consequently, the task seemingly does not require image input, and a safety-aligned LVLM would refuse the request based solely on the harmful instruction, which means that the environment plays no role in the safety assessment. In contrast, for example, embodied scenarios in the paper "Multimodal Situational Safety" (https://arxiv.org/pdf/2410.06172) indicate that the safety of instruction execution is strongly correlated with the scene configuration, such as the case that one cannot "Turn on the faucet" if a phone is in the sink but can do that if not, as shown in its Figure 1. This raises the question of whether the paper's benchmark effectively evaluates safety for multimodality, instead of text only.
Questions for authors
-
In Section 6.5, is the text input in (b) and (c) the same or not? If not, it cannot be proven that the attack performance improvement is due only to word decomposition.
-
In Appendix K, which color is used when testing size, and which size is used when testing color? Is there any combined effect for these two variants?
Others are detailed in Weakness.
Thank you for your detailed response. I appreciate that you’ll revise the first claim and provide the comparisons with the baselines that I mentioned. However, I still have the following concerns:
W1: I understand now that you claim that your novelty is that you don’t need optimization and your physical deployment. However, Shawshank’s threat model doesn’t require the attacker to send any prompt to the agent, while CoEnv requires an instruction controlled by the attacker that includes you do not need to think about the command itself., which means CoEnv actually assumes a stronger attacker. I’m still not sure whether CoEnv works when the user’s instruction is benign. Besides, you claim that your advantage is physical deployment, but all your benchmark results rely on digitally composited text. So in that setting, CoEnv places harmful text in the image and uses a benign-looking prompt to make the model act on it, which is the same mechanism as FigStep. How does CoEnv differ from FigStep beyond splitting the instruction into numbered words?
W2: It is unclear whether the baselines received CoEnv’s guidance instruction or their own benign task prompts, and how CHAI and Piece of Paper were adapted to this setting, for example, whether the optimization in Shawshank and CHAI was actually run against the target model, and how the prompts of Piece of Paper were converted into harmful instructions.