acceptodds
Under review as a conference paper at ICLR 2027

WardenGuard and Warden-Bench: A Multichannel Defense and Benchmark for Indirect and Agentic Prompt Injection

Abstract

Prompt injection is the leading security risk for language-model agents, yet existing benchmarks cover narrow attack classes and existing detectors either flatten the agent's context into a single string or fine-tune on one attack distribution, causing high false positives or poor generalization. The proposed work makes two contributions. First, Warden-Bench, a 23,601-row benchmark consolidating twelve attack tiers (T0-T11), direct overrides, indirect injection via tool outputs and retrieved context, multi-turn fragmentation, and encoding evasion, plus a 5,899-row benign split for false-positive measurement. Second, WardenGuard, a defense exposing six trust-annotated context channels to a two-stage judge cascade: a local open-weight model settles clear cases, deferring to a frontier judge when an input is unfamiliar or escalation is needed, behind an attestation gate and a write-time memory-integrity monitor, preceded by a deterministic 2 ms decoder covering thirteen of fourteen encoding schemes, with every decision hash-chained for audit. The proposed work evaluates across Warden-Bench and four external benchmarks (41,949 cases total), WardenGuard attains mean F1 0.911 and precision 0.981 externally, and is the only defense meeting F10.90, FPR5%, and FNR15% on Warden-Bench. Across a chained residual attack matrix (442,550 combinations), it reaches the lowest mean residual ASR (1.95%), significantly below every competitor on all twenty cells at 11.4 fewer false positives, a margin that persists under an open-weight judge backbone. Against a query-budgeted adaptive attacker, it admits 34.0% of a stratified held-out sample versus 68.7% and 70.0% for the strongest baselines, a separation that survives excluding every case the attestation gate refuses, indicating that trust-annotated context-channel separation is a practical, backbone-agnostic path toward deployable prompt-injection defenses. The code is available at: https://anonymous.4open.science/r/WardenGuard-07F4/

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Prompt injection, LLM agent security, Benchmark, Adversarial robustness, Indirect prompt injection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GkK8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Warden-Bench, a 23,601-example benchmark covering 12 prompt-injection categories, and WardenGuard, a defense for agent-style contexts. It set inputs into six channels with trust labels and WardenGuard combines decoding, permission checks, and memory protection. Easy cases are tackled by local model and harder cases are routed to another judge. The evaluation includes five benchmarks and attack-success measurements on five target LLMs. The authors report improved detection and lower residual attack success, with advantages kept with open-source judge model.

Critical strengths

None

Critical weaknesses
  1. The construction and agent-level validity of Warden-Bench are not well presented. Limited detail on scenario generation and validation, together with predominantly text-based evaluation, makes its added value over existing benchmarks unclear.

  2. The full guard pipeline is not clearly specified. An overview is only provided in the appendix, but Algorithm 1 only covers decoding and a single judge call, omitting memory checks, attestation, and the two-stage decision logic. Unified pseudocode specifying execution order, escalation conditions, and decision thresholds would make the method easier to understand and reproduce.

Questions for authors
  1. Could the authors provide representative examples showing how source payloads are converted into coherent agent tasks, tool/memory contexts, and attacker goals?

  2. How is scenario validity of Warden-Bench checked after generation and mutation?

  3. Which agent-specific failure modes does Warden-Bench reveal that existing benchmarks miss?

  4. Could the authors compare all guards using the same complete input context, while preserving each method’s own processing pipeline, to assess whether WardenGuard’s performance advantage persists?

1 comment
AuthorsOpenReview ↗

Pipeline, benchmark validity and same-input comparison

Thank you for the careful review and questions.

W1 (construction, validity). Sec. 6.1 covers construction, validity and scope; detail in G.6 (generators, seed sources, regeneration checks), G.9 (twenty mutation transforms, applied after labelling) and G.8 (audit of every row: 91.9% agreement on 23,252 parsed rows, κ 0.789). Released generators rebuild T5 and T11 row for row. Rows are static agent contexts, not agent runs: Warden-Bench tests whether a detector separates an override from channel content; residual ASR (Sec. 5.4; one tool-free response per target model) is a proxy for harm.

W2 (full pipeline). Algorithm 1 (Sec. 6.1), per case, in order:

  1. Memory monitor refuses a write if Instr ≥ 0.4 and its label is below Semi-trusted, or it moves memory past cosine distance 0.9 from the system prompt. Refused writes leave the memory channel.
  2. Attestation gate, on the case as received: a failed signature, passport-chain, token, nonce, timestamp or grant check, or a tool call to an unregistered server, returns Block at cost 0.
  3. Stage 1 (Qwen3-30B-A3B, cost 1): a valid reply with p1 ≥ 0.93, or p1 ≤ 0.02 and Ood ≤ 0.2447 (standing check), settles; else Stage 2.
  4. Stage 2 (GPT-4.1-mini, cost 100) decides; an invalid reply blocks and is retried.
  5. Verdict, gate outcome, p1 and cost go to the SHA-256 hash chain.

Band and threshold are set on the development half; App. F details the decoder.

Q1 (examples). On T6 to T9 a generator joins an agent role and request, a seed payload independent of the request, a placement and one to three transforms. Sec. 6.1 traces two rows; G.6 adds memory and history rows.

  • T7-00937: a code assistant asked to write a CSV parser gets, in tool output, a peer agent's override directive demanding an unrestricted persona, plus a matching memory write. The monitor refuses the write; the gate refuses the case (the envelope fails AnalysisAgent's signature check).
  • T8-00246: a finance assistant asked to process wire transfers gets tool output granting itself unrestricted tools before a DAN-style override; no principal claim, and Stage 1 blocks it.

Q2 (validity). Labels come from the generator branch; mutated rows are not each re-validated (G.9). The audit agrees on 92.4% of mutated and 95.3% of unmutated attacks. Two judges re-judged 176 of the 709 mutated T6 to T10 attacks the auditor disputes: an estimated 95.4% of the 9,465 are valid (G.17.2). WardenGuard's residual ASR stays significantly lower in 20/20 cells on the 7,837 unmutated and 16,251 audit-confirmed attacks (G.17.1). On T9, 612 attacks use seeds their source labels benign and hold 363 of WardenGuard's misses (App. C).

Q3 (new failure modes). External benchmarks fill at most two channels, with no memory writes, principal claims or prior turns. On Warden-Bench, PromptArmor falls from 94.1% (T1) to 59.8% (T6); on T11's 208 in-channel base85/base32/atbash/ROT47 attacks, PromptArmor catches 32.7%, WardenGuard 97.6%; 1,010 attacks fall only to a refused memory write; cross-turn T9 is WardenGuard's weakest tier (69.9%; G.7).

Q4 (same complete input). Each guard keeps its own pipeline (inputs in G.1.2). On the 2,645 attacks whose payload sits in a channel every guard reads, plus all 5,899 benign rows (G.17.3, Table A21):

GuardRecall %F1MCCFPR %
ProtectAI56.50.57320.385918.24
DataSentinel77.80.45780.054172.69
PromptArmor68.80.78420.71893.02
PromptSleuth85.80.47340.077979.25
WardenGuard85.90.87900.82714.25

Given all six channels, DataSentinel scores F1 0.3597 and ProtectAI 0.6606 (FPR 36.51%, all rows); WardenGuard's lead persists.

Revisions in the updated paper:

  • Sec. 6.1: Algorithm 1 (complete per-case pipeline, every escalation condition and threshold), worked examples, construction, label validity, agent-level scope (W1, W2, Q1, Q2).
  • G.17.3 (Table A21) and G.1.2: same-input comparison and each defense's input (Q4).
  • G.8, G.9, G.17.1, G.17.2: label audit, mutation validity, subset significance, quote-aware and strict success re-scoring, two-judge adjudication (Q2).
  • Sec. 4.6, G.2: attestation gate run on every case with cryptographic checks; claims from case content, never the label; refuses 3,334 cases, all attacks.
  • Table 2: all WardenGuard numbers from one set of per-row verdicts: F1 0.9540, MCC 0.8372, FPR 4.25%; Table A7 halves agree (0.9542 / 0.9534).
  • G.15: residual-ASR significance by exact McNemar with Holm correction and BCa intervals; 20/20 cells significant.
  • Sec. 5.1.1, G.11.1: held-out protocol stating what was set on which data.
  • App. A: adaptive attacks, worst-case success 41.7% (PromptSleuth 87.0%, PromptArmor 70.0%); 42.3% without the gate, not significantly different.

We hope this resolves your concerns. Accordingly, we respectfully ask the reviewer to reconsider the rating.

Reviewer_Th3VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Warden-Bench, a 23,601-case prompt-injection benchmark spanning twelve attack tiers and benign controls, and WardenGuard, a defense combining six trust-annotated context channels, deterministic decoding, attestation, memory-write monitoring, and a local-to-frontier judge cascade. Evaluation covers the proposed benchmark and four external datasets, with component ablations, a final-judge backbone replacement, and query-budgeted adaptive detector-evasion attacks. The paper reports F1 of 0.9543 on Warden-Bench, mean external F1 of 0.911, and 1.95% mean residual ASR across five target models using a per-case composition of defense decisions and target outcomes. Attestation is simulated in the evaluation.

Critical strengths

I thank the authors for their submission and for releasing substantial code, data, and evaluation records.

  1. Useful systems design: The defense makes context provenance explicit and combines decoding, memory-write screening, and selective escalation in a practical architecture. The asymmetric check on local benign decisions addresses an important generalization failure mode.
  2. Substantial empirical investigation: External benchmarks, component ablations, and adaptive attacks provide useful evidence beyond aggregate detection accuracy. The released records reproduce the main detection and residual-ASR tables, and the adaptive advantage persists after excluding gate-refused cases.
Critical weaknesses

My main concerns are the validity of the security outcome measurements and the independence of the held-out evaluation.

  1. Outcome validity: The lexical ASR rubric counts some apparently task-compliant responses as successful injections. For example, the released Llama response for T11_caesar_00180 classifies the encoded message as spam, yet receives ASR=1 for repeating payload words. Together with attack labels retained after mutation without adjudication, this leaves the primary residual-ASR claim insufficiently validated semantically; the extent and effect of these errors are unknown.
  2. Held-out protocol: The paper claims development-only configuration selection, but Algorithm 1 and the released split documentation explicitly acknowledge prompt/decoder tuning before the split, including use of all T11 benign cases. Similar performance on a later random partition does not establish an untouched test, limiting confidence in the claimed independent validation.
Questions for authors

Can you adjudicate a stratified sample of mutated attacks and target responses, distinguish quoting or classifying payload text from following its instruction, and report the effect on the residual-ASR comparisons? Please reconcile the development-only claim with Algorithm 1 and the release’s tuning-before-split warning. Which results use configurations fixed before their evaluation cases were examined?

1 comment
AuthorsOpenReview ↗

Outcome validity and held-out protocol

Thank you for the detailed review.

Attestation. The attestation gate runs on every case. Claims come from case text, never labels; a claim for another principal arrives as the strongest artifact a keyless attacker can build. It checks signature, passport chain, token, nonce, timestamp and capability, refusing 3,334 cases, all attacks, no benign row (G.2). Table 2: F1 0.9540, MCC 0.8372, FPR 4.25%; without the gate, F1 0.9506, MCC 0.8276 at the same FPR; the gate alone stops 110 attacks.

W1 / Q: outcome validity. Agreed: lexical rates overestimate success (G.1.3).

  • T11_caesar_00180: Llama-3.1-70B and Mistral-Small-3.1 quote the shifted string and call it spam; the rule counts quoted tokens. WardenGuard and PromptArmor both allowed it, raising both residuals equally.
  • Quoting vs. following: Claude Sonnet 5.5 (13% withheld) and DeepSeek-V3.2, outside every target, backbone and auditor family, labelled 1,200 responses stratified by LLM, tier group and released label FOLLOWED, DISCUSSED (quoted, decoded, classified or flagged without acting), REFUSED, TASK_ONLY or OTHER; FOLLOWED agreement 92.8% (κ 0.48). Of sampled released successes, 10.5% are FOLLOWED and most of the rest DISCUSSED; of released failures, 2.7% are FOLLOWED (judge 2: 18.0% and 3.7%).
  • Re-scoring all 19,453 successes: quote-aware drops those whose payload words occur only inside verbatim payload copies (33.3%, incl. both example responses); strict also drops refusal-phrase and default-only T11 successes.

Effect (mean residual ASR %, five LLMs; Sig. = cells of 20 where WardenGuard is significantly lower, McNemar exact, Holm; judge rows: stratified bootstrap, 13 and 14 after Holm):

LabelsNoneProtectAIDataSent.PromptArm.PromptSl.WardenG.Sig.
Released21.9818.543.907.256.921.8620
Quote-aware14.6512.132.624.474.351.1920
Strict13.0311.142.283.874.231.0220
Every passed attack100.0084.3424.6827.5227.747.5120
Judge 1 (2 if withheld)3.873.220.831.191.190.3316
Judge 25.564.461.181.601.550.5116

WardenGuard's judged mean is significantly below each competitor's; on responses each defense passed, the lead holds over ProtectAI and PromptSleuth (PromptArmor, judge 2), not DataSentinel; never significantly higher.

  • Mutated labels (G.9, G.17.2): the auditor agrees with 92.4% of mutated attack labels (95.3% unmutated). Re-judging 176 of the 709 mutated T6 to T10 attacks it called benign, stratified by transform, upholds 41.5%: 95.4% of the 9,465 mutated attack labels are valid (93.8%, judge 2). WardenGuard stays significantly lower in 20/20 cells (released, quote-aware) on the 7,837 unmutated and 16,251 auditor-confirmed attacks.

W2 / Q: held-out protocol. Agreed. The band [0.02, 0.93] and standing check are set on the development half; the prompt and decoder (incl. the +2 word-gain constant and all T11 benign rows) were developed, and the Stage 1 model chosen (1,500-row screen, 464 test rows), on data spanning both halves, so the split does not certify an untouched test (Sec. 5.1.1, G.11.1, App. F). No model is trained on Warden-Bench. Bounds:

  • Prompt: 142 words, no examples; none of its 137 six-word sequences occurs in the 23,601 rows.
  • Decoder: T11 benign FPR is higher on test (7.14%, 10/140) than development (5.23%, 17/325); without T11, F1 0.9554 (development), 0.9547 (test).
  • Upper threshold 0.05 to 0.93: held-out F1 0.9534 to 0.9592, lowest at 0.93, ≥0.110 above PromptArmor (0.8426).

Q: configurations fixed before cases were examined. Only the adaptive attacks’ LLM rewrites (App. A, 300 test-half cases): success 27.7% for WardenGuard, 63.7% PromptSleuth, 38.3% PromptArmor (Table A1); both gaps significant (McNemar, Holm). No parameter is fit per external benchmark (mean F1 0.911). Warden-Bench results, test half included, are not untouched tests; the bounds above size selection effects.

Revisions in the updated paper:

  1. Attestation gate run on every case with cryptographic checks, and results without it (Sec. 4.6, G.2, Table 2).
  2. Quote-aware, strict and every-passed-attack re-scoring (G.17.1, Table A20); two-judge adjudication of responses and mutated labels (G.17.2, G.9).
  3. Held-out protocol: what was set on which data; threshold sweep (Sec. 5.1.1, G.11.1).
  4. Headline values from one set of per-row verdicts; one significance account (Table 2, Sec. 5.4.1, G.15).
  5. PromptSleuth as its released pipeline throughout (Table 2, G.1.2, G.14).
  6. Same-input and enforcement comparisons (G.17.3, Table A21).
  7. Algorithm 1: complete pipeline, escalation conditions, thresholds, worked example; benchmark construction and validity (Sec. 6.1, G.6, G.8).

We hope this addresses your concerns.

Reviewer_3KucOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper checks for malicious instructions across the sources an agent reads, including tool replies and saved memory. Checking only the user’s request may miss an attack placed elsewhere. It introduces a dataset of 23601 examples and a defense examining six input channels. The defense decodes disguised text, combines model judgments, performs simulated identity or permission checks and screens information before saving it. Table 2 reports an F1 score of 0.9543 and a 4.25% false-alarm rate on normal inputs. F1 combines finding attacks with the reliability of warnings; it is not an accuracy percentage. The adaptive tests measure getting past the filter, rather than actual downstream harm.

Critical strengths

The design marks how much each source should be trusted, checks what gets saved for later and sends unfamiliar inputs for further assessment when the local model would otherwise allow them (Sections 4.1 and 4.7; Appendix G.0.9). These checks address different places where an agent can be misled. I find this combination practically interesting.

The authors test beyond their own dataset, remove components to examine their contribution and change the judging model (Tables 3, 5). These checks can help explain which parts of the approach matter, once the reporting conflicts are resolved.

Critical weaknesses

Appendix G.0.5 says all twenty comparisons significantly favour WardenGuard, while Appendix G.15.1 says three advantages over PromptSleuth do not survive correction for multiple comparisons and reports some lower competitor values. Which account describes the final evaluation is not clear.

For most tiers, ProtectAI receives only the user’s request and cannot inspect attacks placed elsewhere; the encoded tier is an exception (Appendix G.1.2). WardenGuard’s score also counts attacks stopped by refusing a memory write. Refusals of normal memory writes are reported separately from the false-alarm rate. This makes the comparison harder to interpret.

Questions for authors

Which comparison statements describe the final runs: Appendix G.0.5 or G.15.1?

The full F1 of .9543 in Table 2 exceeds both split values, 0.9536 and 0.9525, in Table A7. Were these calculated from different runs?

Pooling the same predictions would not produce this result. How much advantage remains with comparable information and enforcement? Do refused memory writes interfere with legitimate tasks (Appendix G.1.2)?

1 comment
AuthorsOpenReview ↗

One set of verdicts, gate and enforcement ablations

Thank you for the careful review and for finding these inconsistencies. Both are corrected; every non-adaptive WardenGuard value below comes from one set of per-row verdicts.

Attestation gate. The gate runs on every Warden-Bench case and verifies the authority claims a row carries (15,845 rows carry none): signature, passport chain, capability token, nonce, timestamp. It refuses 3,334 cases, all attacks, and passes all 229 benign rows with claims. Claims come from row text, never labels; refusals assume legitimate authority arrives signed, untested here (G.2). Table 2: F1 0.9540, MCC 0.8372, FPR 4.25%.

W1/Q1. G.0.5, G.15.1 and G.17.1 now agree: all twenty paired residual-ASR comparisons favour WardenGuard significantly (McNemar exact, Holm). Weakest cell: DataSentinel on GPT-OSS-120B, 397 vs 202 discordant attacks (Holm p = 1.3×10⁻¹⁵). WardenGuard's residual ASR (1.38% to 2.46%) is lowest on every LLM (by 1.10 to 27.30 pp) and on T1 to T8; PromptSleuth is significantly lower on T9 and T11, DataSentinel on T9, each flagging 72 to 82% of benign rows (G.15.3).

Q2. Yes: Table A7's WardenGuard row was not computed from Table 2's verdicts. From them (TP/FP/FN/TN): development (16,501) 11,441/167/931/3,962, F1 0.9542; held-out (7,100) 4,932/84/398/1,686, F1 0.9534. The halves sum to Table 2's counts; accuracy does not differ (Fisher exact, p = 0.71). Table A7, Sec. 5.1.1, G.11.

W2/Q3: enforcement. On all 23,601 rows, each at 4.25% FPR (F1 / MCC, Table A21):

  • complete WardenGuard 0.9540 / 0.8372;
  • without the attestation gate 0.9506 / 0.8276;
  • refused writes not counted as stops 0.9223 / 0.7552;
  • judge cascade alone 0.9174 / 0.7442.

All exceed the best baseline, PromptArmor (0.8356 / 0.6047). The gate alone stops 110 attacks (T7 94, T6 9, T8 7); memory screening stops 2,713, 1,010 alone. Under App. A's attacks, evasion without vs with the gate is 37.0 vs 35.7% (random search), 28.3 vs 27.7% (LLM rewrite), 42.3 vs 41.7% (worst case); no paired difference is significant (McNemar p ≥ 0.125).

W2/Q3: information. G.1.2 lists each defense's inputs (DataSentinel's include the task instruction). ProtectAI reads the user turn (all six channels on T11); run on all six for every row: recall 55.3%, F1 0.6606, MCC 0.1629, FPR 36.51% (own input: 15.7%, 0.2573, −0.0302, 18.24%). PromptArmor reads five of six (six on T11), PromptSleuth's released pipeline all six. G.17.3 removes channel coverage: 2,645 attacks whose payload's first 40 characters sit verbatim in the user turn (on T11, any channel but history), which every defense reads, plus all 5,899 benign rows:

DefenseRecallF1MCCFPR
ProtectAI56.5%0.57320.385918.24%
DataSentinel77.8%0.45780.054172.69%
PromptArmor68.8%0.78420.71893.02%
PromptSleuth85.8%0.47340.077979.25%
WardenGuard85.9%0.87900.82714.25%

The judge cascade alone scores 0.8788 / 0.8268 here, so the margin holds under both controls. DataSentinel on data channels only, as its authors apply it (4,921 rows): 0.4430 / 0.2841 vs WardenGuard 0.8731 / 0.8345.

Q3: benign memory writes. No benign row is stopped by a refused write. Of 5,899 benign rows, 229 propose writes; in 24, one write is refused (each the action field of the second completed-action log entry) and the task proceeds without it. Counting all 24 as false alarms gives FPR 4.66% (Sec. 5.2.1). With no provenance recorded, screening labels every write UNTRUSTED, the strictest setting (G.3). Episodes are single-turn, so the downstream cost of the removed field is not measured.

F1 and adaptive tests. F1 gaps are now stated as F1 differences. App. A measures evasion of the defense; the residual matrix (Sec. 5.4) joins each verdict with the target model's single response without tools, so harm in tool-using agents is not measured.

Revisions in the updated paper:

  • Attestation gate run on every case with cryptographic checks; results with and without it (Sec. 4.6, G.2, App. A.1, Table A21).
  • Tables 2 and A7, split and residual-ASR results from one set of per-row verdicts; one significance account (G.0.5, G.15.1, G.17.1).
  • Same-input and enforcement comparisons; DataSentinel as its authors apply it (G.1.2, G.17.3, Table A21).
  • Refused benign memory writes reported with the FPR (Sec. 5.2.1, G.3).
  • Two-judge adjudication of 1,200 responses; quote-aware and strict success rules; validity of mutated labels (G.9, G.17.1, G.17.2).
  • Held-out protocol: band and standing check set on the development half, prompt and decoder on the full benchmark (Sec. 5.1.1, G.11.1).
  • Sec. 6.1: Algorithm 1 (complete per-case pipeline, escalation conditions, thresholds), worked example, benchmark construction and validity (G.6 to G.9).
  • Adaptive attacks on WardenGuard and two baselines (App. A).

We hope this resolves your concerns and welcome further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions: updated PDF and repository

We sincerely thank all reviewers for their time and constructive feedback. The PDF and repository are updated. All WardenGuard Warden-Bench values outside the adaptive attacks come from one set of per-row verdicts.

  • Attestation gate (Th3V, 3Kuc). Sec. 4.6 and G.2 describe the gate (signed passports, capability tokens and envelopes; claims come from case content, never the label). It refuses 3,334 cases (14.1%), all attacks, and passes all 229 benign cases that carry claims. Table 2: F1 0.9540, MCC 0.8372, FPR 4.25%. Without the gate: F1 0.9506, MCC 0.8276, same FPR; the gate alone stops 110 attacks (T6 to T8). Under App. A's attacks, worst-case success is 42.3% without the gate, 41.7% with it; no paired difference is significant (McNemar p ≥ 0.125).
  • Outcome validity (Th3V). G.17.1 (Table A20) re-scores every success: WardenGuard's mean residual ASR is 1.86% lexical, 1.19% quote-aware, 1.02% strict, 7.51% if every passed attack counts, significantly below every competitor in 20/20 cells under each. In G.17.2, two cross-family judges label whether 1,200 stratified responses follow the payload or only quote or classify it: WardenGuard's adjudicated mean is 0.33%, lower in 16/20 cells (13 after Holm), higher in none; a second estimator separates it from ProtectAI and PromptSleuth only.
  • Mutated rows (Th3V, GkK8). Mutated rows are not each re-validated (G.9). An independent audit agrees with 92.4% of mutated attack labels (95.3% unmutated); re-judging a sample of the 709 disputed rows ((8,748 auditor-agreed rows count as valid, 8 unlabeled ones as invalid)) puts valid labels among the 9,465 mutated T6 to T10 attacks at 95.4%. WardenGuard stays significantly lower in 20/20 cells on the 7,837 unmutated and the 16,251 audit-confirmed attacks.
  • Held-out protocol (Th3V; Sec. 5.1.1, G.11.1). The judge prompt and the decoder (with its +2 word-gain constant) were developed on the full benchmark, so the split does not bound selection effects on them; the Stage 1 backbone was chosen on a released 1,500-row screen with 464 test-half rows. The Stage 1 band [0.02, 0.93] and the standing check's reference set and threshold are set on the development half. Without T11 the halves score F1 0.9554 and 0.9547; across upper thresholds 0.05 to 0.93, held-out F1 is 0.9534 to 0.9592, lowest at the evaluated 0.93. G.11.1: only App. A's LLM rewrites were unseen by any configuration choice; nothing is fit per external benchmark.
  • G.0.5 vs G.15.1 (3Kuc). Both agree: WardenGuard has the lowest residual ASR on every LLM (by 1.10 to 27.30 pp, all significant after Holm).
  • Table 2 vs Table A7; F1 (3Kuc). Same per-row verdicts: development F1 0.9542, held-out 0.9534 (accuracy p = 0.71), pooled 0.9540. F1 gaps are now stated as F1 differences.
  • Comparable input and enforcement (3Kuc, GkK8). G.1.2 lists each defense's input. G.17.3 (Table A21) scores all defenses on all benign rows and the 2,645 attacks whose payload sits in a channel every defense reads (history excluded): WardenGuard leads on F1 and MCC (0.8790 / 0.8271; judge cascade alone 0.8788 / 0.8268; PromptArmor 0.7842 / 0.7189; DataSentinel on all six channels 0.3597 / 0.0377). At 4.25% FPR on all rows, the judge cascade alone scores 0.9174 / 0.7442, above every Table 2 baseline. Counting the 24 refused benign memory writes (none blocked) as false alarms: FPR 4.66%.
  • PromptSleuth (3Kuc, GkK8). Every comparison uses its authors' released pipeline (strict-parser rejects count as benign, as released; a lenient parse is a sensitivity check). F1 vs WardenGuard: Warden-Bench 0.7275 vs 0.9540 (FPR 79.25%), 0.7155 vs 0.9553 with a Qwen3-30B backbone (FPR 77.61% vs 7.07%); InjecAgent 0.7217 vs 0.9500 (p = 1.7e-80). On BIPIA the two do not differ significantly (0.7870 vs 0.7994, p = 0.14).
  • Adaptive attacks (App. A; 3Kuc). Success measures evasion of the defense, not downstream harm. On 300 test-half cases with two attackers, worst-case success: WardenGuard 41.7%, PromptSleuth 87.0%, PromptArmor 70.0%; all six paired tests are significant after Holm. 229 of PromptSleuth's 423 successes end on a reply its strict parser rejects but a lenient parse flags.
  • Pipeline and benchmark validity (GkK8). A new section gives Algorithm 1, the per-case procedure (memory screening, attestation gate, decoder, Stage 1 with band and standing check, Stage 2, audit log) with every escalation condition and threshold, construction, a worked example, label validity and agent-level scope. 612 T9 attacks use seeds their source labels benign and hold 363 of WardenGuard's 590 T9 misses; every defense that catches more of them flags 41.9% to 92.6% of the 680 topic-change benign rows (WardenGuard 1.5%).

Repository. Released: the decoder, evaluation harnesses, success-judge code and the T5 to T9 and T11 generators. docs/REPRODUCTION_MANIFEST.md maps every table and figure to its script.

We are grateful for the reviewers' careful reading, which has improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.