- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces Warden-Bench, a 23,601-example benchmark covering 12 prompt-injection categories, and WardenGuard, a defense for agent-style contexts. It set inputs into six channels with trust labels and WardenGuard combines decoding, permission checks, and memory protection. Easy cases are tackled by local model and harder cases are routed to another judge. The evaluation includes five benchmarks and attack-success measurements on five target LLMs. The authors report improved detection and lower residual attack success, with advantages kept with open-source judge model.
Critical strengths
None
Critical weaknesses
-
The construction and agent-level validity of Warden-Bench are not well presented. Limited detail on scenario generation and validation, together with predominantly text-based evaluation, makes its added value over existing benchmarks unclear.
-
The full guard pipeline is not clearly specified. An overview is only provided in the appendix, but Algorithm 1 only covers decoding and a single judge call, omitting memory checks, attestation, and the two-stage decision logic. Unified pseudocode specifying execution order, escalation conditions, and decision thresholds would make the method easier to understand and reproduce.
Questions for authors
-
Could the authors provide representative examples showing how source payloads are converted into coherent agent tasks, tool/memory contexts, and attacker goals?
-
How is scenario validity of Warden-Bench checked after generation and mutation?
-
Which agent-specific failure modes does Warden-Bench reveal that existing benchmarks miss?
-
Could the authors compare all guards using the same complete input context, while preserving each method’s own processing pipeline, to assess whether WardenGuard’s performance advantage persists?
Pipeline, benchmark validity and same-input comparison
Thank you for the careful review and questions.
W1 (construction, validity). Sec. 6.1 covers construction, validity and scope; detail in G.6 (generators, seed sources, regeneration checks), G.9 (twenty mutation transforms, applied after labelling) and G.8 (audit of every row: 91.9% agreement on 23,252 parsed rows, κ 0.789). Released generators rebuild T5 and T11 row for row. Rows are static agent contexts, not agent runs: Warden-Bench tests whether a detector separates an override from channel content; residual ASR (Sec. 5.4; one tool-free response per target model) is a proxy for harm.
W2 (full pipeline). Algorithm 1 (Sec. 6.1), per case, in order:
Band and threshold are set on the development half; App. F details the decoder.
Q1 (examples). On T6 to T9 a generator joins an agent role and request, a seed payload independent of the request, a placement and one to three transforms. Sec. 6.1 traces two rows; G.6 adds memory and history rows.
Q2 (validity). Labels come from the generator branch; mutated rows are not each re-validated (G.9). The audit agrees on 92.4% of mutated and 95.3% of unmutated attacks. Two judges re-judged 176 of the 709 mutated T6 to T10 attacks the auditor disputes: an estimated 95.4% of the 9,465 are valid (G.17.2). WardenGuard's residual ASR stays significantly lower in 20/20 cells on the 7,837 unmutated and 16,251 audit-confirmed attacks (G.17.1). On T9, 612 attacks use seeds their source labels benign and hold 363 of WardenGuard's misses (App. C).
Q3 (new failure modes). External benchmarks fill at most two channels, with no memory writes, principal claims or prior turns. On Warden-Bench, PromptArmor falls from 94.1% (T1) to 59.8% (T6); on T11's 208 in-channel base85/base32/atbash/ROT47 attacks, PromptArmor catches 32.7%, WardenGuard 97.6%; 1,010 attacks fall only to a refused memory write; cross-turn T9 is WardenGuard's weakest tier (69.9%; G.7).
Q4 (same complete input). Each guard keeps its own pipeline (inputs in G.1.2). On the 2,645 attacks whose payload sits in a channel every guard reads, plus all 5,899 benign rows (G.17.3, Table A21):
Given all six channels, DataSentinel scores F1 0.3597 and ProtectAI 0.6606 (FPR 36.51%, all rows); WardenGuard's lead persists.
Revisions in the updated paper:
We hope this resolves your concerns. Accordingly, we respectfully ask the reviewer to reconsider the rating.