- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper discusses the problem of AI safety in multi-turn agentic setups where the model interacts with tools/environments, user inputs, and makes decisions. The authors here argue that the unsafe behavior can creep into the run at either stage, be it tool call output or native reasoning of the model. So, the authors attempt to study and exactly pinpoint the turn at which the unsafe behavior starts and builds up, instead of ascertaining it at the outcome level. To study this, they have built their benchmark MTRiskTrace in two languages (English and Chinese), in which they evaluate multiple models across various multi-turn scenarios and tools to identify exact points of safety vulnerabilities. The unsafe behavior identification mainly works by identifying if an agent made an unsafe move in the run (Safety Judgement), then zooming into the exact first unsafe step (Risk Localization), and characterizing/classifying the failure in 4 categories (Capability Diagnosis) for further analysis.
Critical strengths
- The paper is well-written, presents a clean flow, along with coherent and cogent arguments.
- I appreciate the problem chosen related to unsafe behavior identification in their execution itself, rather than outcome based identifcation. It's well motivated in the introduction, and it's a real problem to be studied.
- I also appreciate the effort of the authors for the very deep analysis across multiple models, scenarios, and tools.
- Given that the authors plan to open-source their work, I believe it will be a quick test for new and upcoming models to assess and identify exact failure points in their interactions, for builders to further iterate on them.
Critical weaknesses
- I don't see any glaring errors in the paper here for now, which would push me to reject this work. I am not sure if I can call this a weakness, but this is standard benchmark and analysis work. I don't condemn it, but it would have been good to discuss some sort of mitigation strategies for this harmful behavior. Like any simple techniques that would have controlled this behavior? But it also feels like the takeaways from this work are not clear. The paper feels "overcooked" with multiple dimensions, scenarios, cases, etc., but fails to do justice to them overall/individually. I would have much better appreciated it if the cases were lower, but the authors showed a deeper analysis along each and gave me takeaways as a user/developer of agents, and hence I find the takeaways quite squished/compressed. the. Also, paper claims to deal with both
EnandZh, but I don't see any major takeaways/comparisons across languages, and to be honest, while reading the paper, I almost forgot that it deals with another language other than English.
Questions for authors
- Clarification: What was the need to work with 3 harnesses? I felt the comparison across them didn't really make sense. Like, is there something in
OpenClaw, thatClaude Code, orCodexmissed and that gave it an advantage? or vice-versa? - Clarification: How's this study/benchmark different from Indirect-Prompt-Injection (IPI) benchmarks?
- I felt the following analysis was missing, or if present, not motivated well: Once a model begins an unsafe behavior, how long does it continue like that? Is there any break from it? If so, what was causing it?
- It would have been good to present some important takeaways for some scenarios and tool usages that user/developers should be aware of (like a head-up). For example, if I am using databases, what's the most common form of unsafe behavior my model can be exposed to, and how, and what happens in those? I know this can be a very long analysis, but a small table about the why, where, and how of some important scenarios/tools can help. I guess this can help as a quick check for people who plan to deploy their agents.
8 comments
Response 2 for Reviewer vqSX
2. Takeaways
To make the implications of our results clearer, we added a dedicated subsection of practical takeaways (Appendix G.8) and analyses of high-risk tools and scenarios (Appendix G.10), summarized in Section 4.2. The main takeaways are:
-
Check actions, not only answers. Behavioral risks reach 72.56% ASR, compared with 28.71% for content risks. Moreover, 86.2% of behavioral failures first emerge at the Act stage (Figure 4). Agent safety evaluation should therefore examine tool calls and their execution throughout the interaction, rather than relying only on final responses.
-
Test full dialogues. Among five-turn cases, 77.8% of first violations occur at Turns 4–5 (Figure 5a). Safety evaluations should preserve multi-turn context and examine whether agents maintain constraints as requests evolve, rather than testing only isolated prompts.
-
Match safeguards to failure patterns. The dominant capability weakness differs across models: 87.2% of Claude Sonnet 4.6's failures are C1, whereas 69.5% of Llama 3.3-70B's failures are C2 (Figure 5b). This suggests that model-specific safeguards should address the observed failure patterns rather than rely on a single general-purpose reminder.
-
Account for cross-lingual differences. Although C2 is the largest failure category in both Chinese (54.9%) and English (56.4%) cases, model-level performance differs across languages. Eight of the eleven models have higher ASR on Chinese cases, with gaps reaching 10.58 percentage points, and the safest model differs between the two languages (Appendix G.4). Therefore, bilingual safety evaluation should examine both shared failure patterns and language-dependent differences, rather than assume that safety performance transfers directly across languages.
High-risk tools and scenarios. To provide more concrete guidance for agent developers, Table R1-1 summarizes the three tool categories and three scenario categories with the highest ASR.
Table R1-1: High-risk tool and scenario categories (11 OpenClaw models, GPT-5 judge).
| Type | Category | ASR (%) | Why: Dominant Failure | Where: Stage / Peak Onset Turn | How: Most Effective Attack |
|---|---|---|---|---|---|
| Tool | Database | 52.33 | C2 (62.8%) | Act (70.9%) / T3 | A1 (60.19%) |
| Tool | File | 50.60 | C2 (50.6%); C1 (31.1%) | Act (51.6%); Answer (48.4%) / T3 | A5 (52.28%) |
| Tool | Code | 48.67 | C2 (58.2%) | Act (64.3%) / T3 | A3 (50.91%) |
| Scenario | Customer service and marketing | 54.81 | C2 (51.88%) | Act (57.35%) / T3 | A3 (61.17%) |
| Scenario | Facilities and security | 52.97 | C2 (50.25%) | Act (53.19%) / T4 | A1 (58.72%) |
| Scenario | IT and cloud operations | 51.23 | C2 (55.88%) | Act (63.20%) / T3 | A3 (57.58%) |
C2 is the dominant failure category across all six selected groups, and first violations occur mainly at the Act stage. However, the most effective attack strategy varies by category: A1 for Database and Facilities, A3 for Code, Customer Service, and IT, and A5 for File. These findings suggest several concrete deployment considerations. Database agents should validate requested fields, record-level permissions, and result limits before each query. File-handling agents should inspect generated answers as well as tool calls, since 48.4% of first violations in this category occur after the Act stage. Code-related agents should validate commands and their target resources before execution. More generally, defenses should account for the dominant failure type, onset stage, and attack strategy associated with each tool and task setting. Representative cases and results for all five tool categories and eleven scenario categories are provided in Appendix G.10.
Related Revised Sections: Section 4.1 (Cross-lingual comparison); Section 4.2 (Category-level analysis); Appendix G.4 (Cross-lingual analysis); Appendix G.8 (Practical Takeaways); Appendix G.9 (Mitigation directions); Appendix G.10 (Results for all tool and scenario categories, with representative cases).
[1] Wang et al., 2026. AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents. ICSE 2026.
[2] Shi et al., 2025. Progent: Programmable Privilege Control for LLM Agents. arXiv:2504.11703.
[3] Xie et al., 2023. Defending ChatGPT against Jailbreak Attack via Self-Reminders. Nature Machine Intelligence.
[4] Li et al., 2024. Measuring and Controlling Instruction (In)Stability in Language Model Dialogs. COLM 2024.
[5] Chennabasappa et al., 2025. LlamaFirewall: An Open Source Guardrail System for Building Secure AI Agents. arXiv:2505.03574.
[6] Hines et al., 2024. Defending Against Indirect Prompt Injection Attacks With Spotlighting. arXiv:2403.14720.
Response 3 for Reviewer vqSX
Q1: What was the need to work with 3 harnesses? I felt the comparison across them didn't really make sense. Like, is there something in OpenClaw, that Claude Code, or Codex missed and that gave it an advantage? or vice-versa?
R2: Thank you for this question. The three harnesses serve two complementary evaluation purposes rather than forming a direct three-way comparison: OpenClaw provides a common setting for cross-model evaluation, while paired comparisons with Codex and Claude Code examine harness effects under fixed models.
1. Two Comparison Axes: Separating Model and Harness Comparisons
These two comparison axes aim to distinguish model-level safety differences from harness-related effects by holding one factor fixed in each comparison.
-
Cross-model comparison. OpenClaw is selected as the common evaluation harness because of its broad model compatibility, supporting all 11 evaluated models. This allows us to compare model safety under the same harness and execution environment, reducing harness-related variation in cross-model analysis. All cross-model conclusions in our main experiments are therefore based on OpenClaw.
-
Cross-harness comparison. Harnesses determine how models receive instructions, access tools, and execute multi-turn interactions. To examine their influence while keeping the model fixed, we conduct two paired comparisons: GPT-5.5 under OpenClaw and Codex, and Claude Sonnet 4.6 under OpenClaw and Claude Code. Codex and Claude Code are paired with their respective providers' models to evaluate native deployment settings without introducing additional model-routing or adaptation layers. We do not directly compare Codex with Claude Code, as this would change both the model and the harness simultaneously.
2. Shared Environment and Harness-specific Configurations
To clarify what differs across harnesses, we summarize their configurations in Table R2-1.
Table R2-1: Agent configurations and execution settings across the three harnesses.
| Component | OpenClaw | Codex | Claude Code |
|---|---|---|---|
| Constraint placement | Initial User Message | Developer Instructions | Appended System Prompt |
| Tool interface | Plugins | MCP | MCP |
| Native tool and permission settings | Only experimental plugin tools enabled | MCP approval configuration | Native tools disabled, strict MCP configuration, dontAsk permission mode, and MCP allowlist |
| Agent execution | OpenClaw-native execution loop | Codex-native execution loop | Claude Code-native execution loop |
All three harnesses connect to the same custom-built stateful simulator and use the tools declared in each evaluation case. Thus, the underlying task environments and tool definitions remain consistent, while instruction placement, tool interfaces, permission settings, and execution mechanisms differ. These configurations provide the basis for examining how harness implementations relate to agent safety under the paired comparisons.
3. Interpretation and Scope
The comparisons examine the overall influence of harness configurations rather than establish a universal safety ranking or attribute differences to individual components. By fixing the model in each paired comparison, we can examine how the same model behaves under different harness configurations. Conversely, the common OpenClaw setting enables controlled cross-model evaluation. Together, these complementary comparisons separate model-level analysis from harness-level analysis without conflating the two.
Related Revised Sections: Section 4 (Models and Harnesses); Appendix G.13 (Harness settings and execution outcomes of unsafe tool-call attempts).
Response 1 for Reviewer vqSX
We thank the reviewer for the insightful and useful feedback. We appreciate the reviewer's in-depth examination of our method, experimental design, and research contributions. The comments have helped us identify several aspects that require further clarification, stronger empirical evidence, and more thorough analysis. Our point-by-point responses are provided below and we also revised our main paper and appendix accordingly, with changes marked in red.
R1: We connect our findings to practical agent safety through diagnosis-informed mitigation, including model training and runtime enforcement, and focused analyses of high-risk tools, scenarios, and cross-lingual differences.
1. Mitigation Methods and Techniques
Our findings inform supervised fine-tuning (SFT), reinforcement learning (RL), and runtime enforcement. Risk Localization identifies when violations emerge, while Capability Diagnosis identifies failure types; together, they guide model optimization and runtime safeguards.
Risk-localization-guided supervised fine-tuning. Annotated trajectories can provide stage-specific training examples based on the first risk step. For Planning violations, train agents to identify task constraints, verify authorization, and formulate safe plans. For Act violations, focus on selecting permitted tools, resources, recipients, and arguments while rejecting unauthorized operations. For Answer violations, prevent unauthorized disclosure and unsupported claims of execution or approval. Multi-turn examples should preserve interaction histories and teach agents to maintain established constraints when later requests reframe or expand authorized scope.
Turn- and stage-aware reinforcement learning. Since 77.8% of first violations in five-turn cases occur at Turns 4–5 (Figure 5a), training should reinforce constraint persistence. A process-level reward model can reward constraint recognition during Planning, authorized tool selection during Act, and safe responses during Answer, while penalizing the earliest observable violation. Rewards can target C1–C4: risk recognition and safe planning (C1), tool-boundary compliance (C2), external-information verification (C3), and consistent enforcement of established constraints (C4). Later requests challenging earlier authorization should be evaluated explicitly. Rewarding legitimate completion within the permitted scope can also discourage excessive refusal.
Runtime authorization enforcement (C2). C2 accounts for 76.4% of behavioral failures (Figure 4). A lightweight pre-execution mechanism can compare each tool call's name, target resource, recipient, and arguments against session permissions, blocking or escalating out-of-scope operations. AgentSpec prevents over 90% of unsafe code executions [1], while Progent reduces AgentDojo ASR from 39.9% to 1.0% using LLM-generated policies [2].
Cross-turn constraint tracking (C4). C4 accounts for 7.5%–32.3% of failures across models (Figure 5b). Agents can maintain explicit authorization state and revalidate it before each action. State changes should require valid authorization evidence, not repeated requests, rephrasing, or unsupported authority claims. Self-Reminder reduces jailbreak ASR from 67.21% to 19.34% by repeating safety constraints [3]. Prior work reports that instructions provided only at dialogue onset can drift within eight rounds [4].
Risk and provenance verification (C1 and C3). Before acting or responding, agents should check planned operations against task constraints and distinguish trusted instructions from untrusted tool outputs. For C1, review planned operations and authorization prerequisites; for C3, treat external information as untrusted until its source and reliability are verified. LlamaFirewall checks agent reasoning against the user's goal, reducing AgentDojo ASR from 17.6% to 1.75% [5]. Spotlighting marks untrusted input and reduces injection ASR from over 50% to below 2% [6].
These approaches provide complementary safeguards. SFT and RL improve learned safety behavior, while runtime enforcement checks actions against authorization constraints. Risk Localization evaluates where safeguards prevent violations, and Capability Diagnosis identifies remaining failure patterns.