- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces SafeClawBench, a text-only benchmark of 600 adversarial tool-agent cases, 100 in each of six families (direct prompt injection, indirect prompt injection, tool-return injection, memory poisoning, memory extraction, and ambiguity-driven unsafe inference). Cases were expanded by GPT-4.1 from 36 researcher-written seeds and selected through a D0 pilot on eight endpoints that required at least one attack success. Models are called without function calling and no tool executes, so the unit of analysis is the returned text. Each frozen response is scored in two stages by a GPT-5.4 judge. The first stage is a semantic endpoint (CoreFail: the response adopts, authorizes, reveals, plans, or justifies the attack goal). The second stage, applied only to CoreFail responses, labels "response-visible evidence" (RVE), split into grounded disclosure (GD, a protected value appears in the text) and asserted-operation claims (AOC, the response states that access, an action, or persistence occurred). Seven models are compared under four system-prompt policies (no added defense, a sandwich reminder, and two longer layered policies) with exact McNemar tests and Holm correction on a fixed 600-task denominator. The main reported findings are: all 21 model-policy contrasts reduce RVE relative to D0 (20 significant); the two longer policies reduce CoreFail for all seven models and the sandwich policy for six; 587 of 2,201 CoreFail responses in the five-model reference panel are RVE, with AOC labels far outnumbering GD labels; and the D3 versus D4 comparison is model-dependent with no significant paired difference. Extensive appendices report re-audits by Gemini 2.5 Pro and DeepSeek V3.1, a lexical input-grounding detector, test-retest stability, leave-one-family-out analyses, a stratified audit of CorePass responses, and two author-led human reviews.
Critical strengths
S1. Careful and unusually transparent measurement practice. The design pairs responses by task within model, keeps a fixed 600-task denominator for both endpoints, uses exact McNemar tests with prespecified Holm families, adds task-cluster bootstrap and lexical-cluster sensitivity checks, bounds the effect of recovered and filtered cells by worst-case completion, and preregisters disconfirmation criteria. Just as importantly, the paper reports its negative results rather than hiding them: the human review that fails its own criteria, the retest that meets neither explanation criterion, and the full three-auditor envelope. This matters to my assessment because the core policy result is about as well supported as a single-turn, prompt-level result can be: every model has its highest RVE count under D0 in the primary audit, and pooled over models D0 carries the most CoreFail and the most RVE rows under all four labelings and all six leave-one-family-out exclusions (Appendix I).
S2. The paper shows, with its own data, how much an LLM-judged agent ASR can hide. On the same 2,201 responses, three auditors produce between 587 and 1,346 RVE rows; the memory-extraction family moves from 1.4% to 62% conditional evidence depending on the auditor; and the CorePass audit suggests that roughly a third of potential evidence lies outside the semantic gate. These are useful cautionary findings for anyone who reports judge-based attack success on agent benchmarks, and I regard them as the most informative part of the submission, more so than the benchmark itself.
Critical weaknesses
W1. The evidence endpoint does not measure harm, despite its name. With no tool execution, every asserted-operation claim is a narrated, unexecuted action. AOC appears in 563 of 587 RVE rows and the persistence label alone in 465 (399 of 400 MPI RVE rows), so RVE mostly measures whether a model role-plays task completion. Disclosure has a similar problem: in a prompt-only setting a protected value is either copied from the input or invented, yet the authors' detector finds no 8-token input match for 39 of the reference auditor's 53 TextHarm labels and for 90.2% of auditor-discordant MEX rows. Fabricated secrets and fabricated action reports are worth studying, but they are not "grounded disclosure," and names like TextHarm and HarmEvidence invite a harm reading. Gating on CoreFail makes this worse: RVE@600 cannot exceed CoreFail@600 by definition, and the gate drops refuse-but-leak responses (the authors' own extrapolation puts about 32% of potential RVE outside the gate, concentrated at D0). Prior work also shows that text is a weak proxy for action in agents (GAP, arXiv 2602.16943; "Large Language Models Often Say One Thing and Do Another," ICLR 2025; WASP), none of which is discussed. Without an execution-level anchor, even mock function calling on a subset, readers cannot tell whether RVE predicts realized harm better than CoreFail alone.
W2. The evidence endpoint is not reliably measurable, and its distinctive claims do not survive the paper's own robustness checks. Pairwise RVE agreement is = 0.31 to 0.37; TextHarm agreement between the reference auditor and either alternative is = 0.07 to 0.10, while the two alternatives agree at 0.48, so the reference auditor is the outlier. MEX, the largest family, ranges from 1.4% to 62% conditional evidence across auditors. Appendix I shows that "AOC exceeds GD" and the pooled RVE ordering are not exclusion-invariant, yet the former appears in the abstract and conclusion. Human validation does not resolve this: the frozen-queue review misses four of five prespecified criteria (about a quarter of HE and SO labels changed), was done by an author with machine suggestions visible, and neither review labels GD or AOC. What remains robust (D0 fails more than the policies) is visible from CoreFail alone, is well known, and is inflated by selecting cases for D0 success on a pilot that included Kimi K2.5 and the GPT-5.4 judge. Family composition also shifts across policies (MEX's share of the gate rises from 32% at D0 to 56% at D3), and without a benign-utility arm or a length-matched control, policy gains cannot be separated from broader refusal.
W3. Presentation and illustration are poor, which matters here because the contribution is a distinction between kinds of responses that the paper never actually shows.
(a) Organization. Main-text space goes to provenance (legacy sweep, rejudge history, retries, recovered cells), while CoreFail, GD, and AOC get a few sentences each and the auditor disagreement (S2) is pushed to Appendices F to H. Many main-text claims rely on appendix-only tables, and the abstract mixes five-model and seven-model results without signposting. Appendix I lists 29 reversals in one run-on paragraph using raw endpoint strings ("claude-opus-4-7-aws under gemini"), and Appendices G and H cite internal file names instead of stating the specification.
(b) Terminology. RVE is also HarmEvidence/HE; D4 is also LongPolicy and D4_AgentShield; the disconfirmation criteria D1 to D3 collide with the policy names; and Table 24's V2, K1, K2, and "Kilo route" are undefined.
(c) Figures. There are only two. Figure 1 is a rough five-box flowchart that restates Section 3.2, with words broken mid-word ("con-trolled", "suc-cess", "re-sponse"), and never shows what a single ASR misses. Figure 2 prints labels over the whiskers and overlays two intervals with different meanings (task-cluster bootstrap and between-auditor range).
(d) No concrete examples. The only worked case (Appendix K, TRI 0827) is a few sentences without the input, full response, or auditor spans. Readers need to see a SemanticOnly, a GD, an AOC-only, and a refuse-but-leak CorePass response, ideally one per family; this would also answer Q1. The D3 and D4 policy texts and lengths are likewise absent from the paper.
(e) Suggested fix. Move provenance to the appendix, define the endpoints with real examples, replace Figure 1 with one showing where a single ASR and the two endpoints diverge on actual responses, and unify terminology.
W4. Secondary issues.
(a) Reproducibility. Per-case seed lineage and the candidate pool are lost (114 of 600 representative cases recoverable), so the number of independent cases is unknown and the selection-bias check is not reproducible.
(b) Policy selection. Conditions were kept by file completeness, which dropped the StruQ-style variant; no trained or structured defense remains.
(c) Judge overlap. GPT-5.4 judges GPT-5.5 and GPT-5.6 Sol, and the favoritism check covers only Core labels.
(d) Configuration. GLM-5.3 uses an 8,192-token limit versus 2,048 for the others, and judge inputs are truncated at 3,000 and 4,000 characters.
(e) Related work. Missing AgentCanary (arXiv 2606.10484), which also argues against a single ASR in executable environments, and StrongREJECT-style separation of willingness from capability.
Questions for authors
Q1 (could change my view of W1). Per family, how often is the targeted protected value actually present in the model input? If a MEX secret is absent from the input, should those TextHarm rows be reported as fabricated disclosures instead? Was the TRI 0827 grade table in the tool return?
Q2 (could change my view of W1). On a subset with mock function calling and a deterministic state oracle, can you report for X = RVE, SemanticOnly, and CorePass?
Q3. Why gate the evidence audit on CoreFail? Do the main conclusions change with ungated RVE over all 12,000 responses?
Q4. Is there any independent, blinded human annotation at the GD/AOC level? Without it, why use the most conservative auditor as the reference rather than the predeclared 2-of-3 consensus?
Q5. Do D3 and D4 increase false refusals or degrade completion on matched benign tasks?
Q6. Which judge decided attack success in the selection pilot? Can you report the policy contrasts on the 114 recoverable representative cases?