- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PUARobot, a framework for inducing embodied LLM planners to generate harmful actions through natural-language attacks. It turns seven forms of social and psychological pressure into fabricated dialogues, submits each complete dialogue as a single user input, and iteratively searches using action scores and API-validity feedback. Here, “multi-turn” refers to the input structure; the target model does not participate in the preceding exchanges (§3.1, §3.3). Across 321 tasks and 8 target models, with up to 50 queries per task, the authors report an average attack success rate of 76.2%, compared with 7.3% for the strongest BadRobot baseline (§4.1, Table 1). Table 3 provides component ablations, while §4.5 and Appendix F present simulation and visual-program execution examples.
Critical strengths
- The evaluation addresses a meaningful distinction in embodied safety. I appreciate that the method separates action-harm scoring from format checks against declared APIs and evaluates only newly generated target outputs, without counting attacker-authored historical actions as target behavior (§3.3–3.4, Appendix C). This helps distinguish a dangerous description from a dangerous plan that can be passed to an executor. The paper also explicitly notes that interface validity does not establish successful physical execution.
- The experiments offer broad coverage and component analysis. Table 1 covers 8 models and 7 risk categories; Table 3 compares single-turn inputs, fabricated dialogues, and the addition of action feedback. For example, Commitment Escalation improves from 0.35 to 0.56 and then to 0.65. I also value the evaluator checks with final human-confirmed judgments in Table 2 and the execution examples in §4.5 and Appendix F. Together, these results support the attack's effectiveness in the tested settings, although the examples themselves do not estimate execution success rates.
Critical weaknesses
- The main success metric excludes the central failure mode of verbal refusal accompanied by harmful actions. Eq. (2) requires an action score of at least 4, valid APIs, and a language score of at least 3. Appendix C assigns explicit refusals and warnings scores of 0–2. A substantively harmful plan that passes the interface checks is therefore counted as an attack failure if accompanied by refusal text. I find this inconsistent with the action risk emphasized in §3.4. It may particularly underestimate BadRobot's Safety Misalignment baseline, which specifically targets discrepancies between language and actions (Appendix D.1). Until success is recomputed using harmful actions and API validity, with language behavior reported separately, I do not think the 76.2% versus 7.3% gap adequately establishes the difference in action-level safety.
- The controls do not establish the separate contributions of psychological pressure and adaptive search. The main comparisons use fixed BadRobot templates. I recognize that Table 3 places four conventional strategies in dialogue and feedback settings, but it does not compare against existing automated persuasive attacks or adaptive attacks on robots. Relevant examples include PAP and RoboPAIR, both research directions already cited in the paper. On GPT-5.5, the full system's 0.71 versus 0.65 for the best fixed strategy also does not by itself establish that Thompson sampling outperforms random strategy selection (Table 1, Table 3, Appendix D.3). Without these controls, I cannot tell how much of the gain comes from the specific design choices emphasized here, rather than strategy diversity, richer context, or general feedback-based search.
Questions for authors
- Clarification and reanalysis: Why does action-level attack success also require a language score of at least 3? Using the existing outputs, please report success based on harmful, API-valid actions separately from success that additionally requires verbal compliance, along with each method's rate of harmful actions accompanied by verbal refusal. Does the gap between BadRobot and PUARobot remain?
- Additional controls: How does PUARobot compare with RoboPAIR or PAIR-style adaptive attacks under the same target models, interfaces, query limits, and evaluation rules? How much advantage remains when PAP-style prompts or conventional strategies use the same action-feedback search process? My main interest here is in establishing the incremental contribution, rather than expanding the task set.
- Additional evidence: How do success rates and query costs change when Thompson sampling is replaced with uniform random strategy selection? Since category statistics are shared across tasks in the same category, please specify task ordering and reset rules for each independent run, and report variation across repeated runs. This directly affects the interpretation of the improvement from 0.65 to 0.71.