- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes WARD, a small guard model (Qwen-3.5-0.8B and 2B) that runs in parallel with a web agent and checks the user task, page HTML, and screenshot for prompt injection. It is trained in three stages: fine-tuning on WARD-Base (about 177K samples from 709 real websites and 10 simulated platforms, with paired benign and malicious variants), fine-tuning on WARD-PIG (attacks that also try to manipulate the guard), and A3T, where an LLM attacker with memory creates new attacks against the current guard, which is then updated with GRPO. WARD reports 98.47%–100% recall on four out-of-distribution benchmarks, a 0.20%–0.35% step-level false positive rate on WebArena, a 3%–6% sample success rate for adaptive attacks (vs. 74%–79% for WebAgentGuard), and about 2.3 seconds per step.
Critical strengths
The work is practical and thorough. Detection, utility, robustness, and efficiency are measured together, and the results are strong on all four, with about 3× fewer output tokens and 4× lower latency than WebAgentGuard from a much smaller model (Table 12) and stable results over three training seeds (Table 5). WARD would be a strong baseline for future work.
Attacks on the guard itself are treated as a real threat, with useful controls. WebAgentGuard’s recall drops as low as 2.5% when the page contains text aimed at the guard (Table 2), which motivates WARD-PIG and A3T. The matched-size study (Table 14) shows that the same backbone trained on the same number of samples, but from WARD-Base, raises WARD-Test F1 from 87.01 to 96.64, so the gains do not come only from scale.
Critical weaknesses
It is hard to tell how much headroom remains, because most robustness results are recall-only or come from the authors’ own pipeline. EIA, VPI, WASP, MUZZLE, and the guard-targeted tests report recall only, and PIArena reports attack success without a false positive rate; a guard that leans toward flagging would also look strong on these. The WARD-Test benign samples come from the same generator as training, and the hard benign cases that motivate the paper, such as tutorials and support pages (line 065), are not tested as an independent set. The in-domain adaptive test reuses A3T with the same attacker and validator as training, and strong LLMs already reach 100% recall on EIA and WASP. So 100% recall on every guard-targeted setting and on all 817 MUZZLE observations does not show where WARD starts to fail.
The novelty lies mainly in data and training scale. The parallel multimodal guard setting follows WebAgentGuard, and training on guard-targeted prompts and attacker–defender co-evolution are known ideas from adversarial training and red teaming. The paper should state clearly what A3T adds beyond these ideas and compare it with a simpler adversarial training loop.
Questions for authors
What is WARD’s false positive rate on PIArena’s clean inputs and on an independently collected set of benign pages with instruction-like text (e.g., tutorials, help pages)? This would show whether the low attack success comes with over-flagging. Minor (no response needed): Appendix C.3.1 (line 1347) says the utility study uses Claude 3.5 Sonnet, while Tables 11–12 and Appendix B.2 say Claude 3.7 Sonnet. Line 274: “The training SFT prompt are” should be “is”.
8 comments
Response to the Reviewer j6Wc (Part 2)
W1. Concern about recall-only evaluation, false positives, and hard benign cases (continued)
5. Additional evaluation on benign–malicious pairs with similar instruction wording
We also conduct a controlled evaluation on benign–malicious pairs with similar instruction wording, where benign examples can contain legitimate instructions.
Evaluation construction. We use six platforms: GitHub, Hugging Face, Custom Mail, Zalo (a messaging platform), Amazon, and Instagram. Benign content is first generated based on webpage screenshots, topics, and writing-style hints, covering README files, emails, messages, product content, and social posts. These examples may contain legitimate requests, recommendations, or instructions without attempting to redirect the agent.
Each malicious counterpart is then generated using its benign example as a reference, preserving the instruction phrasing, channel style, structure, approximate length, and opening wording while introducing an attack objective. Both examples share the same platform context and user task, and are injected into the webpage using the same procedure, following the corresponding injection channel and its required format. We use Gemini-3.8-Flash to generate both benign and malicious samples.
To ensure similarity in instruction wording, we require either a token-level Dice similarity ≥ 0.5 or at least 8 consecutive shared words in the instructions. We continue generating and filtering until we obtain 100 qualifying pairs per platform, yielding 600 pairs (1,200 examples).
Results. We evaluate both final WARD checkpoints (0.8B and 2B) on all 1,200 examples.
Table: Per-platform and overall sample-level classification performance (%).
| Model | Metric | GitHub | Hugging Face | Custom Mail | Zalo | Amazon | All | |
|---|---|---|---|---|---|---|---|---|
| WARD-0.8B | Accuracy | 96.50 | 96.50 | 99.50 | 99.00 | 100.00 | 100.00 | 98.58 |
| Recall | 96.00 | 97.00 | 99.00 | 100.00 | 100.00 | 100.00 | 98.67 | |
| Precision | 96.97 | 96.04 | 100.00 | 98.04 | 100.00 | 100.00 | 98.50 | |
| F1 | 96.48 | 96.52 | 99.50 | 99.01 | 100.00 | 100.00 | 98.58 | |
| WARD-2B | Accuracy | 98.00 | 98.00 | 99.50 | 99.00 | 100.00 | 100.00 | 99.08 |
| Recall | 98.00 | 99.00 | 99.00 | 99.00 | 100.00 | 100.00 | 99.17 | |
| Precision | 98.00 | 97.06 | 100.00 | 99.00 | 100.00 | 100.00 | 99.00 | |
| F1 | 98.00 | 98.02 | 99.50 | 99.00 | 100.00 | 100.00 | 99.08 |
We additionally report the stricter pair-level accuracy, where a pair is counted as correct only when both its benign and malicious examples are classified correctly.
Table: Pair-level accuracy across six platforms.
| Platform | Pairs | WARD-0.8B | WARD-2B |
|---|---|---|---|
| GitHub | 100 | 93/100 | 96/100 |
| Hugging Face | 100 | 93/100 | 96/100 |
| Custom Mail | 100 | 99/100 | 99/100 |
| Zalo | 100 | 98/100 | 98/100 |
| Amazon | 100 | 100/100 | 100/100 |
| 100 | 100/100 | 100/100 | |
| Total | 600 | 583/600 (97.17%) | 589/600 (98.17%) |
WARD-0.8B produces 9 false positives and 8 false negatives, while WARD-2B produces 6 false positives and 5 false negatives. All examples, including failures, are included in the results. Despite substantial overlap in instruction wording, both checkpoints maintain high classification performance, achieving pair-level accuracies of 97.17% and 98.17%. These results provide additional evidence that WARD can distinguish legitimate instructions from malicious redirections even when their wording is similar, reducing the concern that its performance primarily relies on superficial classification shortcuts. We have added this experiment to the Appendix I of the revised version.
Response to the Reviewer j6Wc (Part 3)
W2: Concern about where WARD starts to fail.
Our adaptive evaluations also reveal WARD's limitations. At a trial budget of 10, WARD-2B and WARD-0.8B achieve SSRs of 3.12% and 5.62%, respectively (Figure 3). When replacing Gemini-3-Flash with Claude Sonnet 5, SSR increases to 6.25% and 7.50%, respectively (Table 11). These results show that WARD is not perfectly robust, and that stronger attacks and larger attack budgets can expose further vulnerabilities. We view WARD as a step forward in defending against existing attacks, with A3T further challenging and improving its robustness, while acknowledging that prompt injection defense is an ongoing arms race, where each advance in defense motivates new attack strategies.
W3. Methodological novelty of A3T and comparison with simpler adversarial training
Relationship to prior work. We agree that adversarial training and attacker–defender co-evolution are established ideas, as explored in MART (Ge et al., 2024), R2D2 (Mazeika et al., 2024), MAGIC (Wen et al., 2026), and CoRL (Zhang et al., 2026). However, to the best of our knowledge, no prior work has specifically proposed adaptive adversarial training for multimodal web-agent guards.
Directly applying existing adversarial training methods to this setting is nontrivial. For a web guard, simply bypassing the detector does not guarantee a meaningful attack. The payload must also preserve its malicious objective and remain valid within the webpage context, injection channel, and modality.
To address these challenges, A3T introduces a validity-constrained adversarial training framework across diverse attack goals, injection channels, and modalities, where the attacker must both bypass the current guard and pass a multimodal validator that assesses whether the attack remains valid and plausible when embedded into the target webpage. This prevents the attacker from exploiting the guard through invalid or meaningless payloads. To support this process, A3T maintains attack memory that incorporates feedback from both the guard and the validator, allowing the attacker to iteratively refine its strategies while preserving attack validity.
Additionally, another distinction is that WebAgentGuard does not explicitly localize prompt injections, whereas WARD is designed to identify their locations within the input. On 1,500 attack samples in WARD-Test, WARD-0.8B correctly localizes 1,486 attacks (99.07%), while WARD-2B correctly localizes 1,490 attacks (99.33%). This capability provides more fine-grained information about malicious content beyond binary detection.
Comparison with simpler adversarial training. To assess the benefits of these components, we conduct an additional experiment using the first A3T training cycle.
We generate 180 initial malicious samples from WARD-Seed using the WARD-Base generation procedure, matching the number used in A3T Cycle 1. Each candidate is evaluated against the current WARD-2B checkpoint. If the attack fails to bypass the guard, the attacker receives the guard's feedback and revises the payload once, followed by a final evaluation. This provides a simpler feedback-based adversarial training loop without A3T's persistent attack memory or multimodal validator.
We retain the same benign balancing strategy and GRPO configuration. Both methods start from the same WARD-PIG checkpoint and are evaluated on the same 160 held-out samples with a maximum budget of 10 attack attempts per sample (A3T framework). We additionally measure FPR on 1,500 benign WARD-Test samples.
Table: Comparison of adversarial training methods on WARD-2B after Cycle 1.
| Training method | SSR ↓ | WARD-Test FPR ↓ |
|---|---|---|
| Before A3T | 21.88% | 0.20% |
| Simple adversarial training | 19.38% | 3.93% |
| A3T | 13.75% | 0.33% |
Results and discussion. The simpler feedback-based adversarial training loop achieves only a modest reduction in SSR from 21.88% to 19.38%, while substantially increasing FPR from 0.20% to 3.93%. In contrast, A3T reduces SSR to 13.75% after the same training cycle, while maintaining a low FPR of 0.33%.
These results highlight the importance of validity-constrained adversarial training in multimodal web environments. Without a multimodal validator, the simpler baseline may collect attacks that bypass the guard but fail to preserve their malicious objectives or remain valid in the webpage context, potentially introducing noisy supervision. By combining multimodal validity checking with memory-guided refinement, A3T improves robustness while largely preserving benign-content detection. We have added this experiment to the Appendix F of the revised version.
Response to the Reviewer j6Wc (Part 1)
We thank the reviewer for the constructive feedback with a positive score and for acknowledging WARD's strong detection performance, low false positive rates, practical efficiency, and comprehensive evaluation. We address each concern below.
W1. Concern about recall-only evaluation, false positives, and hard benign cases
We address the limitations of recall-only evaluations through the following analyses:
1. Why EIA, VPI, and WASP originally report recall only
EIA, VPI, and WASP primarily provide malicious samples constructed from a much smaller set of benign source contexts. Specifically, EIA contains 1,414 malicious samples derived from 177 benign webpage forms; VPI contains 306 malicious samples based on 5 benign contexts (Amazon, Booking, BBC, Email, and Message Sender); and WASP contains 84 malicious samples built from 2 contexts (GitLab issue and Reddit post). Given this substantial class imbalance, we originally reported recall on malicious samples to avoid potentially misleading aggregate classification metrics.
2. Existing benign evaluation on WebArena
Beyond these static datasets, Section 6.3 and Tables 5 and 13 evaluate WARD on benign webpage states encountered during WebArena agent execution. We evaluate 802 tasks across six domains (GitLab, Map, Reddit, Shopping, Admin, and Wikipedia) using three agent backbones (GPT-4o, Claude 3.7 Sonnet, and Gemini-3-Flash), covering a total of 22,341 agent steps. Across all three backbones, WARD achieves consistently low false positive rates (0.20–0.35%) and negligible task performance degradation (0.25–0.38%). These results demonstrate that WARD rarely misclassifies benign content during realistic agent interactions.
WebArena also includes content that may contain legitimate instructions, such as GitLab repositories and issues, Reddit posts, and shopping or administrative interfaces. However, these evaluations do not separately isolate tutorials or support pages, which we further investigate through additional experiments below.
3. Additional benign evaluation on independent benchmarks
To further investigate whether WARD tends to overpredict malicious labels, we evaluate both final checkpoints on benign examples associated with EIA, VPI, WASP, MUZZLE, and PIArena. For EIA, VPI, WASP, and MUZZLE we evaluate the original benign source cases underlying their malicious examples. For PIArena, we evaluate 1,700 benign examples from the original general text-based tasks, rather than web-agent tasks. Since PIArena provides text-only inputs, we follow the same text-only preprocessing and input format used for its malicious samples in our paper, without screenshots.
Table: False positive rates on benign examples from five benchmarks.
These results further demonstrate that both WARD checkpoints maintain consistently low false positive rates across independent benign evaluation sets.
4. Additional evaluation on real-world documentation pages
To directly address the reviewer's concern about hard benign cases containing legitimate instructions, such as tutorials and support pages, we collect 300 publicly available documentation pages from four sources: 100 GitHub repository READMEs, 113 MDN Web Docs pages, 49 Django documentation pages, and 38 Docker documentation pages.
The pages are downloaded directly from their original sources, filtered by instruction-related keywords and deduplicated by URL and normalized text. We manually inspect the selected samples to ensure they contain legitimate instructional content, including setup instructions, step-by-step tutorials, usage guides, and troubleshooting procedures, rather than merely descriptive text.
For each page, we construct a user task based on its title, asking the agent to summarize the content or extract relevant instructions. The page content is stored as HTML and rendered into a screenshot offline, allowing evaluation using WARD's multimodal input format.
Table: False positive rates on real-world instructional documentation.
Despite the presence of legitimate instructions that could superficially resemble prompt injections, both checkpoints maintain near-zero false positive rates. These results directly address the reviewer's concern about WARD's ability to distinguish ordinary instructional content from malicious attempts to redirect the agent. We have added this experiment to the Appendix G of the revised version.