acceptodds
Under review as a conference paper at ICLR 2027

WARD: Adversarially Robust Defense of Web Agents Against Prompt Injections

Abstract

Web agents can autonomously complete online tasks by interacting with websites, but their exposure to open web environments makes them vulnerable to prompt injection attacks embedded in HTML content or visual interfaces. Existing guard models still suffer from limited generalization to unseen domains and attack patterns, high false positive rates on benign content, reduced deployment efficiency due to added latency at each step, and vulnerability to adversarial attacks that evolve over time or directly target the guard itself. To address these limitations, we propose WARD (Web Agent Robust Defense against Prompt Injection), a practical guard model for secure and efficient web agents. WARD is built on WARD-Base, a large-scale dataset with around 177K samples collected from 719 high-traffic URLs and platforms, and WARD-PIG, a dedicated dataset designed for prompt injection attacks targeting the guard model. We further introduce A3T, an adaptive adversarial attack training framework that iteratively strengthens WARD through a memory-based attacker and guard co-evolution process. Extensive experiments show that WARD achieves 98.47–100% recall on out-of-distribution benchmarks, maintains low false positive rates to preserve agent utility, remains robust against guard-targeted and adaptive attacks under substantial distribution shifts, and runs efficiently in parallel with the agent.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Computer-Use Agent, Web Agent, Guard, Prompt Injection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_j6WcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WARD, a small guard model (Qwen-3.5-0.8B and 2B) that runs in parallel with a web agent and checks the user task, page HTML, and screenshot for prompt injection. It is trained in three stages: fine-tuning on WARD-Base (about 177K samples from 709 real websites and 10 simulated platforms, with paired benign and malicious variants), fine-tuning on WARD-PIG (attacks that also try to manipulate the guard), and A3T, where an LLM attacker with memory creates new attacks against the current guard, which is then updated with GRPO. WARD reports 98.47%–100% recall on four out-of-distribution benchmarks, a 0.20%–0.35% step-level false positive rate on WebArena, a 3%–6% sample success rate for adaptive attacks (vs. 74%–79% for WebAgentGuard), and about 2.3 seconds per step.

Critical strengths

The work is practical and thorough. Detection, utility, robustness, and efficiency are measured together, and the results are strong on all four, with about 3× fewer output tokens and 4× lower latency than WebAgentGuard from a much smaller model (Table 12) and stable results over three training seeds (Table 5). WARD would be a strong baseline for future work.

Attacks on the guard itself are treated as a real threat, with useful controls. WebAgentGuard’s recall drops as low as 2.5% when the page contains text aimed at the guard (Table 2), which motivates WARD-PIG and A3T. The matched-size study (Table 14) shows that the same backbone trained on the same number of samples, but from WARD-Base, raises WARD-Test F1 from 87.01 to 96.64, so the gains do not come only from scale.

Critical weaknesses

It is hard to tell how much headroom remains, because most robustness results are recall-only or come from the authors’ own pipeline. EIA, VPI, WASP, MUZZLE, and the guard-targeted tests report recall only, and PIArena reports attack success without a false positive rate; a guard that leans toward flagging would also look strong on these. The WARD-Test benign samples come from the same generator as training, and the hard benign cases that motivate the paper, such as tutorials and support pages (line 065), are not tested as an independent set. The in-domain adaptive test reuses A3T with the same attacker and validator as training, and strong LLMs already reach 100% recall on EIA and WASP. So 100% recall on every guard-targeted setting and on all 817 MUZZLE observations does not show where WARD starts to fail.

The novelty lies mainly in data and training scale. The parallel multimodal guard setting follows WebAgentGuard, and training on guard-targeted prompts and attacker–defender co-evolution are known ideas from adversarial training and red teaming. The paper should state clearly what A3T adds beyond these ideas and compare it with a simpler adversarial training loop.

Questions for authors

What is WARD’s false positive rate on PIArena’s clean inputs and on an independently collected set of benign pages with instruction-like text (e.g., tutorials, help pages)? This would show whether the low attack success comes with over-flagging. Minor (no response needed): Appendix C.3.1 (line 1347) says the utility study uses Claude 3.5 Sonnet, while Tables 11–12 and Appendix B.2 say Claude 3.7 Sonnet. Line 274: “The training SFT prompt are” should be “is”.

8 comments
AuthorsOpenReview ↗

Response to the Reviewer j6Wc (Part 1)

We thank the reviewer for the constructive feedback with a positive score and for acknowledging WARD's strong detection performance, low false positive rates, practical efficiency, and comprehensive evaluation. We address each concern below.

W1. Concern about recall-only evaluation, false positives, and hard benign cases

We address the limitations of recall-only evaluations through the following analyses:

1. Why EIA, VPI, and WASP originally report recall only

EIA, VPI, and WASP primarily provide malicious samples constructed from a much smaller set of benign source contexts. Specifically, EIA contains 1,414 malicious samples derived from 177 benign webpage forms; VPI contains 306 malicious samples based on 5 benign contexts (Amazon, Booking, BBC, Email, and Message Sender); and WASP contains 84 malicious samples built from 2 contexts (GitLab issue and Reddit post). Given this substantial class imbalance, we originally reported recall on malicious samples to avoid potentially misleading aggregate classification metrics.

2. Existing benign evaluation on WebArena

Beyond these static datasets, Section 6.3 and Tables 5 and 13 evaluate WARD on benign webpage states encountered during WebArena agent execution. We evaluate 802 tasks across six domains (GitLab, Map, Reddit, Shopping, Admin, and Wikipedia) using three agent backbones (GPT-4o, Claude 3.7 Sonnet, and Gemini-3-Flash), covering a total of 22,341 agent steps. Across all three backbones, WARD achieves consistently low false positive rates (0.20–0.35%) and negligible task performance degradation (0.25–0.38%). These results demonstrate that WARD rarely misclassifies benign content during realistic agent interactions.

WebArena also includes content that may contain legitimate instructions, such as GitLab repositories and issues, Reddit posts, and shopping or administrative interfaces. However, these evaluations do not separately isolate tutorials or support pages, which we further investigate through additional experiments below.

3. Additional benign evaluation on independent benchmarks

To further investigate whether WARD tends to overpredict malicious labels, we evaluate both final checkpoints on benign examples associated with EIA, VPI, WASP, MUZZLE, and PIArena. For EIA, VPI, WASP, and MUZZLE we evaluate the original benign source cases underlying their malicious examples. For PIArena, we evaluate 1,700 benign examples from the original general text-based tasks, rather than web-agent tasks. Since PIArena provides text-only inputs, we follow the same text-only preprocessing and input format used for its malicious samples in our paper, without screenshots.

Table: False positive rates on benign examples from five benchmarks.

BenchmarkBenign casesWARD-0.8B FPRWARD-2B FPR
EIA1770.00% (0/177)0.00% (0/177)
VPI50.00% (0/5)0.00% (0/5)
WASP20.00% (0/2)0.00% (0/2)
MUZZLE1091.83% (2/109)1.83% (2/109)
PIArena (general tasks)1,7000.47% (8/1,700)0.53% (9/1,700)

These results further demonstrate that both WARD checkpoints maintain consistently low false positive rates across independent benign evaluation sets.

4. Additional evaluation on real-world documentation pages

To directly address the reviewer's concern about hard benign cases containing legitimate instructions, such as tutorials and support pages, we collect 300 publicly available documentation pages from four sources: 100 GitHub repository READMEs, 113 MDN Web Docs pages, 49 Django documentation pages, and 38 Docker documentation pages.

The pages are downloaded directly from their original sources, filtered by instruction-related keywords and deduplicated by URL and normalized text. We manually inspect the selected samples to ensure they contain legitimate instructional content, including setup instructions, step-by-step tutorials, usage guides, and troubleshooting procedures, rather than merely descriptive text.

For each page, we construct a user task based on its title, asking the agent to summarize the content or extract relevant instructions. The page content is stored as HTML and rendered into a screenshot offline, allowing evaluation using WARD's multimodal input format.

Table: False positive rates on real-world instructional documentation.

ModelFalse positivesFPR (%)
WARD-0.8B1/3000.33
WARD-2B0/3000.00

Despite the presence of legitimate instructions that could superficially resemble prompt injections, both checkpoints maintain near-zero false positive rates. These results directly address the reviewer's concern about WARD's ability to distinguish ordinary instructional content from malicious attempts to redirect the agent. We have added this experiment to the Appendix G of the revised version.

AuthorsOpenReview ↗

Response to the Reviewer j6Wc (Part 2)

W1. Concern about recall-only evaluation, false positives, and hard benign cases (continued)

5. Additional evaluation on benign–malicious pairs with similar instruction wording

We also conduct a controlled evaluation on benign–malicious pairs with similar instruction wording, where benign examples can contain legitimate instructions.

Evaluation construction. We use six platforms: GitHub, Hugging Face, Custom Mail, Zalo (a messaging platform), Amazon, and Instagram. Benign content is first generated based on webpage screenshots, topics, and writing-style hints, covering README files, emails, messages, product content, and social posts. These examples may contain legitimate requests, recommendations, or instructions without attempting to redirect the agent.

Each malicious counterpart is then generated using its benign example as a reference, preserving the instruction phrasing, channel style, structure, approximate length, and opening wording while introducing an attack objective. Both examples share the same platform context and user task, and are injected into the webpage using the same procedure, following the corresponding injection channel and its required format. We use Gemini-3.8-Flash to generate both benign and malicious samples.

To ensure similarity in instruction wording, we require either a token-level Dice similarity ≥ 0.5 or at least 8 consecutive shared words in the instructions. We continue generating and filtering until we obtain 100 qualifying pairs per platform, yielding 600 pairs (1,200 examples).

Results. We evaluate both final WARD checkpoints (0.8B and 2B) on all 1,200 examples.

Table: Per-platform and overall sample-level classification performance (%).

ModelMetricGitHubHugging FaceCustom MailZaloAmazonInstagramAll
WARD-0.8BAccuracy96.5096.5099.5099.00100.00100.0098.58
Recall96.0097.0099.00100.00100.00100.0098.67
Precision96.9796.04100.0098.04100.00100.0098.50
F196.4896.5299.5099.01100.00100.0098.58
WARD-2BAccuracy98.0098.0099.5099.00100.00100.0099.08
Recall98.0099.0099.0099.00100.00100.0099.17
Precision98.0097.06100.0099.00100.00100.0099.00
F198.0098.0299.5099.00100.00100.0099.08

We additionally report the stricter pair-level accuracy, where a pair is counted as correct only when both its benign and malicious examples are classified correctly.

Table: Pair-level accuracy across six platforms.

PlatformPairsWARD-0.8BWARD-2B
GitHub10093/10096/100
Hugging Face10093/10096/100
Custom Mail10099/10099/100
Zalo10098/10098/100
Amazon100100/100100/100
Instagram100100/100100/100
Total600583/600 (97.17%)589/600 (98.17%)

WARD-0.8B produces 9 false positives and 8 false negatives, while WARD-2B produces 6 false positives and 5 false negatives. All examples, including failures, are included in the results. Despite substantial overlap in instruction wording, both checkpoints maintain high classification performance, achieving pair-level accuracies of 97.17% and 98.17%. These results provide additional evidence that WARD can distinguish legitimate instructions from malicious redirections even when their wording is similar, reducing the concern that its performance primarily relies on superficial classification shortcuts. We have added this experiment to the Appendix I of the revised version.

AuthorsOpenReview ↗

Response to the Reviewer j6Wc (Part 3)

W2: Concern about where WARD starts to fail.

Our adaptive evaluations also reveal WARD's limitations. At a trial budget of 10, WARD-2B and WARD-0.8B achieve SSRs of 3.12% and 5.62%, respectively (Figure 3). When replacing Gemini-3-Flash with Claude Sonnet 5, SSR increases to 6.25% and 7.50%, respectively (Table 11). These results show that WARD is not perfectly robust, and that stronger attacks and larger attack budgets can expose further vulnerabilities. We view WARD as a step forward in defending against existing attacks, with A3T further challenging and improving its robustness, while acknowledging that prompt injection defense is an ongoing arms race, where each advance in defense motivates new attack strategies.

W3. Methodological novelty of A3T and comparison with simpler adversarial training

Relationship to prior work. We agree that adversarial training and attacker–defender co-evolution are established ideas, as explored in MART (Ge et al., 2024), R2D2 (Mazeika et al., 2024), MAGIC (Wen et al., 2026), and CoRL (Zhang et al., 2026). However, to the best of our knowledge, no prior work has specifically proposed adaptive adversarial training for multimodal web-agent guards.

Directly applying existing adversarial training methods to this setting is nontrivial. For a web guard, simply bypassing the detector does not guarantee a meaningful attack. The payload must also preserve its malicious objective and remain valid within the webpage context, injection channel, and modality.

To address these challenges, A3T introduces a validity-constrained adversarial training framework across diverse attack goals, injection channels, and modalities, where the attacker must both bypass the current guard and pass a multimodal validator that assesses whether the attack remains valid and plausible when embedded into the target webpage. This prevents the attacker from exploiting the guard through invalid or meaningless payloads. To support this process, A3T maintains attack memory that incorporates feedback from both the guard and the validator, allowing the attacker to iteratively refine its strategies while preserving attack validity.

Additionally, another distinction is that WebAgentGuard does not explicitly localize prompt injections, whereas WARD is designed to identify their locations within the input. On 1,500 attack samples in WARD-Test, WARD-0.8B correctly localizes 1,486 attacks (99.07%), while WARD-2B correctly localizes 1,490 attacks (99.33%). This capability provides more fine-grained information about malicious content beyond binary detection.

Comparison with simpler adversarial training. To assess the benefits of these components, we conduct an additional experiment using the first A3T training cycle.

We generate 180 initial malicious samples from WARD-Seed using the WARD-Base generation procedure, matching the number used in A3T Cycle 1. Each candidate is evaluated against the current WARD-2B checkpoint. If the attack fails to bypass the guard, the attacker receives the guard's feedback and revises the payload once, followed by a final evaluation. This provides a simpler feedback-based adversarial training loop without A3T's persistent attack memory or multimodal validator.

We retain the same benign balancing strategy and GRPO configuration. Both methods start from the same WARD-PIG checkpoint and are evaluated on the same 160 held-out samples with a maximum budget of 10 attack attempts per sample (A3T framework). We additionally measure FPR on 1,500 benign WARD-Test samples.

Table: Comparison of adversarial training methods on WARD-2B after Cycle 1.

Training methodSSR ↓WARD-Test FPR ↓
Before A3T21.88%0.20%
Simple adversarial training19.38%3.93%
A3T13.75%0.33%

Results and discussion. The simpler feedback-based adversarial training loop achieves only a modest reduction in SSR from 21.88% to 19.38%, while substantially increasing FPR from 0.20% to 3.93%. In contrast, A3T reduces SSR to 13.75% after the same training cycle, while maintaining a low FPR of 0.33%.

These results highlight the importance of validity-constrained adversarial training in multimodal web environments. Without a multimodal validator, the simpler baseline may collect attacks that bypass the guard but fail to preserve their malicious objectives or remain valid in the webpage context, potentially introducing noisy supervision. By combining multimodal validity checking with memory-guided refinement, A3T improves robustness while largely preserving benign-content detection. We have added this experiment to the Appendix F of the revised version.

Reviewer_K6RcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WARD, a multimodal guard model for defending web agents against prompt injection attacks. The framework combines a large-scale dataset (WARD-Base) for general detection, the guard-targeted adversarial training (WARD-PIG), and the adaptive adversarial attack training (A3T) using RL to improve robustness. Lots of experiments across multiple benchmarks demonstrate strong detection performance, robustness against adaptive attacks, low FPRs (low utility losses), and improved inference efficiency compared to existing guard models.

Critical strengths
  1. The paper provides a comprehensive evaluation. The paper evaluates WARD across diverse benchmarks, including guard-targeted, adaptive, cross-attacker, cross-framework, and cross-domain attacks.

  2. WARD detector outperforms existing guard models while maintaining low false positive rates and minimal impact on agent utility.

  3. WARD achieves lower inference latency and output token usage compared to existing guard models, making it practical for deployment.

Critical weaknesses
  1. Insufficient justification for using GRPO instead of SFT. The paper employs GRPO in A3T to improve the guard's robustness against adaptive attacks. However, it remains unclear whether the improvements come from GRPO itself or from the additional adversarial training samples. Since these samples have known ground-truth labels and injection locations, continued SFT would be a natural baseline.

  2. The adaptive robustness evaluation primarily measures whether generated attacks can bypass the guard, rather than whether they successfully manipulate the protected web agent. So the paper lack an end-to-end evaluation. Although a validator assesses attack validity and plausibility, this does not guarantee actual attack success. The paper includes agent-level experiments on VPI, but lacks end-to-end evaluation using A3T-generated adaptive attacks.

  3. There might be some overlap between adaptive training and evaluation frameworks. The paper uses the A3T framework for both adversarial training and part of the adaptive robustness evaluation. Although the test websites and attack configurations are unseen, the underlying attack-generation process remains similar, potentially limiting the strength of the generalization claims.

  4. The evaluation of computational efficiency might be incomplete. The paper reports inference latency and output token usage for the efficiency measurement. However, it does not provide sufficient information about the computational cost of constructing WARD-Base, generating reasoning annotations, or performing iterative A3T training. Given the extensive use of closed-source models for data generation and adversarial attacks, reporting total API costs and training time would help assess the reproducibility and practical efficiency of the framework.

Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to the Reviewer K6Rc (Part 1)

We thank the reviewer for the positive score and constructive feedback, as well as for recognizing the strengths of our comprehensive evaluation, detection performance, and inference efficiency. We address each concern below.

W1. Insufficient justification for using GRPO instead of SFT

We note that applying SFT to A3T-generated samples is not straightforward, as these samples contain ground-truth labels and injection locations but lack the reasoning annotations required by WARD's original SFT formulation. Simply training on labels would change the output format, making the comparison less controlled.

To disentangle the contributions of adaptive adversarial data collection and GRPO optimization, we conduct two complementary experiments using the first A3T training cycle:

  1. SFT vs. GRPO on identical A3T data. Both methods start from the same WARD-PIG checkpoint and use identical adversarial and benign training samples. For SFT, we generate reasoning annotations using the generator–evaluator pipeline described in Section 4.2, preserving WARD's structured output format. We perform full-parameter SFT for one epoch with a batch size of 16, while GRPO follows the original Cycle-1 configuration.

  2. Standard vs. adaptive adversarial data with GRPO. We generate 180 malicious samples from WARD-Seed using the WARD-Base generation procedure, matching the number used in A3T Cycle 1, while keeping the same benign balancing strategy and GRPO configuration.

All methods are evaluated on the same held-out configurations (160 samples) with a maximum budget of 10 attack attempts per sample.

Ablation of adversarial data generation and optimization in A3T Cycle 1.

Training methodWARD-0.8B SSR ↓WARD-2B SSR ↓
Before A3T28.75%21.88%
Standard data + GRPO26.88%22.50%
A3T data + SFT23.13%16.88%
A3T data + GRPO20.62%13.75%

Contribution of adaptive adversarial data. Standard data generation with GRPO yields limited improvement, whereas A3T-generated data reduces SSR to 20.62% and 13.75% for WARD-0.8B and WARD-2B, respectively. This highlights the importance of adaptive hard-example collection, as standard generation does not specifically target the guard's weaknesses.

Contribution of GRPO. SFT on the same A3T data also improves robustness, achieving 23.13% and 16.88% SSR. GRPO provides further gains and is more straightforward to apply, as it does not require additional reasoning annotations.

Overall, both adaptive data collection and GRPO contribute to A3T's effectiveness, with adaptive data generation playing a particularly important role. We have added this experiment to the Appendix E of the revised version.

W2. End-to-end evaluation of A3T-generated adaptive attacks

To address this concern, we extend A3T to 102 VPI-Bench samples across Amazon (79) and Messenger (23), using GPT-4o as the victim agent and Gemini-3-Flash as both the adaptive attacker and validator. All attacks target the final WARD-2B.

For each sample, we allow up to 10 adaptive attack attempts. Every validator-approved attack is evaluated against victim agent (GPT-4o), with and without WARD-2B. We stop early once a valid attack bypasses WARD-2B.

Evaluation metrics. Due to the high cost of multi-step agent execution with API-based LLMs, we measure Attempted Rate (AR) rather than completed Attack Success Rate. Following the VPI-Bench [1] protocol (Section 3.3), we use GPT-4o to examine the agent's execution logs and determine whether the agent initiates actions toward the attacker's objective, even if the objective is not completed. Merely discussing or repeating the injected instructions does not count. AR is calculated as the proportion of validator-approved attacks that induce such behavior.

Agent-level evaluation of WARD-2B on Amazon and Messenger (102 samples).

PlatformWithout WARD (AR)With WARD-2B (AR) ↓
Amazon (79 samples)556/691 (80.46%)5/691 (0.72%)
Messenger (23 samples)164/199 (82.41%)2/199 (1.01%)
Overall (102 samples)720/890 (80.90%)7/890 (0.79%)

Across both platforms, 890 attacks pass the validator. Among these, 720 (80.90%) induce malicious attempts in the unprotected GPT-4o agent, compared with only 7 (0.79%) when WARD-2B is deployed.

These results demonstrate that A3T-generated adaptive attacks can induce malicious behavior in web agents, while WARD-2B substantially reduces such attempts during agent execution. We have added this experiment to Appendix H of the revised version.

[1] VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents. Tri Cao et al. ICLR 2026.

AuthorsOpenReview ↗

Response to the Reviewer K6Rc (Part 2)

W3. The concern about overlap between adaptive training and evaluation

We acknowledge that using A3T for both training and evaluation may introduce methodological overlap. Our paper already includes several evaluations designed to assess robustness beyond the training framework:

  1. Cross-attacker evaluation (Table 11): We replace the training attacker, Gemini-3-Flash, with Claude Sonnet 5 while keeping the evaluation settings unchanged, testing robustness against a stronger, unseen attacker model.
  2. Cross-framework evaluation with MUZZLE (Table 3): We evaluate on 817 injected browser-state observations generated by an independent adaptive web-agent attack framework. Both WARD checkpoints achieve 100% recall.
  3. Cross-domain evaluation with PIArena (Table 4): We evaluate against PIArena's independent Strategy Search algorithm on 1,700 samples across 13 datasets covering QA, RAG, summarization, long-context reasoning, and code generation. WARD-0.8B and WARD-2B achieve average ASRs of 3.23% and 3.08%, respectively.

Additional adaptive evaluation in web-agent settings. To further strengthen our robustness claim, we extend PIArena [2]'s Strategy Search algorithm to 84 WASP web-agent scenarios. Notably, our original PIArena evaluation reported in the paper was conducted on general text-based tasks rather than web-agent environments. This additional experiment extends the evaluation to realistic web-agent interactions.

We follow PIArena's original algorithm and adapt it to jointly target WARD and the protected web agent, since bypassing the guard alone does not guarantee successful manipulation of the victim agent. For each scenario, GPT-5 acts as the attacker, generating 5 initial payloads for each of PIArena's 10 predefined strategies (50 candidates in total). Each candidate is evaluated against WARD and the Gemini-3-Flash victim agent. Following PIArena's evaluation procedure, GPT-4o serves as a judge to determine whether the victim agent follows the malicious instructions.

Based on WARD's predictions and the judge's assessment of agent behavior, PIArena filters and ranks the initial 50 candidates, prioritizing those that successfully manipulate the victim agent and show potential to bypass WARD. The PIArena attacker (GPT-5) then uses this feedback to refine the selected attacks for up to 50 additional attempts, stopping early if an attack both bypasses WARD and successfully manipulates the victim agent.

If no attack satisfies both conditions, we select an agent-successful candidate if available; otherwise, the scenario is counted as unsuccessful against the agent. We do not select attacks that bypass WARD but fail to manipulate the victim agent, as such bypasses do not constitute meaningful attack success and may result from modifications that render the payload harmless. This 50-initial + 50-refinement setup follows PIArena's default configuration. Further details are available in the PIArena repository and paper.

Adaptive attack evaluation using PIArena on 84 WASP scenarios.

Target guard---Success in Tricking the Agent---Success in Tricking the Agent and Bypassing WARD
WARD-0.8B42/84 (50.00%)1/84 (1.19%)
WARD-2B50/84 (59.52%)1/84 (1.19%)

This additional experiment provides further evidence of WARD's robustness against an independent adaptive attacker. We have added this experiment to Section 6.2 of the revised version.

W4. Response to the concern about computational training cost

API cost breakdown. We provide the actual API expenditures for the main stages of our framework:

StageAPI cost (USD)
Web exploration~$359
WARD-Base and WARD-PIG prompt generation and reasoning annotation~$1,878
A3T adaptive attack generation~$486
Total~$2,723

Training compute. As described in Appendix C.1, we train both WARD checkpoints using two NVIDIA H200 GPUs (140 GB each). Each model undergoes SFT on WARD-Base and WARD-PIG, followed by three A3T training cycles using GRPO.

The approximate training costs are:

ModelSFT timeA3T time (3 × 50 min)Total timeGPU-hours
WARD-0.8B6.5 h2.5 h9 h18
WARD-2B4.5 h2.5 h7 h14
Total11 h5 h16 h32

These measurements cover model optimization only and exclude data-generation time, particularly the API-based attack generation and validation required during A3T. The overall data-generation process is estimated to take approximately one week (excluding the time required to develop custom websites for native-platform data), depending on API throughput, concurrency, and web exploration speed. We have added these computational training cost to Appendix C.7 of the revised version.

[2] Piarena: A platform for prompt injection evaluation. Runpeng Geng et al.. ACL 2026

Dear Reviewer K6Rc,

Thank you again for your thoughtful review and encouraging feedback! As the discussion phase is approaching its end, we kindly request to let us know if the above clarifications and the previously added experiments have addressed your concerns. We would be happy to address any additional points the reviewer may have during the remaining time of the discussion phase. We thank the reviewer for engaging with us in the discussion.

Best regards, Authors of Paper 27819

Reviewer_fxEhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WARD, a guard model that helps protect web agents from malicious instructions. At each step, WARD checks the user task, the processed HTML, and a screenshot of the webpage. It then provides a short explanation, identifies the possible attack goal and where the attack appears, and labels the content as safe or malicious. WARD is trained in three stages: WARD-Base learns from safe and malicious examples, WARD-PIG adds attacks targeting the safety model, and A3T improves WARD through three rounds of testing and training on successful attacks. WARD achieves 98.47–100% attack recall, with only 0.20–0.35% false positives.

Critical strengths
  1. WARD is affordable and performs well on benchmarks. It achieves high recall on evaluated attacks with low false positives. The time cost is also reasonable.
  2. The paper also shows that WARD's success comes from its training data. Using the same model and sample count as WebAgentGuard, WARD's data raises defense performance noticeably.
Critical weaknesses
  1. Limited novelty. WARD builds on existing multimodal guard models and improves them through better training data, PIG samples, and A3T. These improvements are useful, but the underlying approach is similar to prior work such as WebAgentGuard. The paper could more clearly explain what is methodologically new, beyond improvements to data generation and training.
  2. The benefits of GRPO are not fully isolated. Table 13 shows that A3T substantially improves robustness. However, A3T introduces both new adversarial examples and GRPO training. It is unclear how much of the improvement comes from each. An SFT baseline using the same adversarial examples would help clarify the contribution of GRPO.
  3. The adaptive evaluation needs further investigation. The main adaptive evaluation uses the same attack framework as A3T training. The paper addresses this concern through additional experiments with Claude, MUZZLE, and PIArena, which provide useful evidence of generalization. One further experiment with an attacker adapting directly to the final WARD model in a web agent setting would strengthen the robustness claim.
  4. The generated data may contain classification shortcuts. Appendix H.3 asks malicious samples to redirect the agent, while benign samples must avoid instructions and references to the user's task. This may introduce wording patterns that make the two classes easier to distinguish. WARD's strong results on other benchmarks make this concern less serious, but do not fully address it. Testing WARD on benign and malicious examples that use similar instruction wording would help show whether it can detect attacks without relying on these patterns.
  5. Section 6.2 suggests that 224 samples were used for evaluation ((50 + 6) × 4), but the SSR values in Table 8 seem to be based on 160 samples. Please clarify how many samples were actually used. Also, does the evaluation stop once the first attack successfully bypasses the guard, as in A3T training? For the ASR results in Figure 3, are attacks rejected by the validator included in the total number of attempts?
  6. Appendix C.3.1 states that Claude 3.5 Sonnet was evaluated on 802 WebArena tasks, but Tables 11–12 report results for Claude 3.7 Sonnet instead. Were the results for Claude 3.5 Sonnet omitted, or is this a reporting error?
Questions for authors
  1. What are the main methodological differences between WARD and WebAgentGuard, beyond the data generation and training setup?
  2. Table 13 shows that A3T improves robustness, but how much of this improvement comes from GRPO rather than the additional adversarial training data? Could the authors compare GRPO with SFT using the same adversarial samples?
  3. Could the authors evaluate an adaptive attacker that specifically targets the final WARD model in a multimodal web agent setting?
  4. Could the authors test whether WARD still performs well when both classes contain similar instruction wording in Appendix H.3? For example, benign pages with legitimate instructions and malicious pages without explicit commands.
8 comments
AuthorsOpenReview ↗

Response to the Reviewer fxEh (Part 1)

We thank the reviewer for the constructive feedback and for recognizing WARD's strong detection performance, low false positive rates, efficiency, and the effectiveness of our training data. We address each concern below.

W1. Methodological differences from WebAgentGuard beyond data generation and training.

While WARD builds upon the parallel multimodal guard architecture of WebAgentGuard, its main methodological distinctions lie in the security problem formulation and adaptive defense framework.

  1. WARD explicitly considers the guard itself as an adversarial target. While WebAgentGuard focuses on detecting prompt injections targeting web agents, WARD extends the threat model to include attacks that directly manipulate the guard's predictions and reasoning. This requires the guard to recognize malicious instructions without being influenced by them.

  2. We agree that the concept of adversarial training is not novel, but to the best of our knowledge, no prior work has specifically proposed adaptive adversarial training for dedicated multimodal web-agent guards. Directly applying existing adversarial training methods to this setting is nontrivial. For a web guard, simply bypassing the detector does not guarantee a meaningful attack. The payload must also preserve its malicious objective and remain valid within the webpage context, injection channel, and modality. To address these challenges, A3T introduces a validity-constrained adversarial training framework across diverse attack goals, injection channels, and modalities, where the attacker must both bypass the current guard and pass a multimodal validator that assesses whether the attack remains valid and plausible when embedded into the target webpage. This prevents the attacker from exploiting the guard through invalid or meaningless payloads. To support this process, A3T maintains attack memory that incorporates feedback from both the guard and the validator, allowing the attacker to iteratively refine its strategies while preserving attack validity.

Thus, the resulting datasets are outcomes of our methodologies, while the broader contributions lie in the threat formulation and adaptive defense framework, which can be used in the future to generate more data. Additionally, another distinction is that WebAgentGuard does not explicitly localize prompt injections, whereas WARD is designed to identify their locations within the input. On 1,500 attack samples in WARD-Test, WARD-0.8B correctly localizes 1,486 attacks (99.07%), while WARD-2B correctly localizes 1,490 attacks (99.33%). This capability provides more fine-grained information about malicious content beyond binary detection.

We have added this clarification to the Appendix J of the revised version.

AuthorsOpenReview ↗

Response to the Reviewer fxEh (Part 2)

W2. Isolating the contribution of GRPO from additional adversarial data

We note that applying SFT to A3T-generated samples is not straightforward, as these samples contain ground-truth labels and injection locations but lack the reasoning annotations required by WARD's original SFT formulation. Simply training on labels would change the output format, making the comparison less controlled.

To disentangle the contributions of adaptive adversarial data collection and GRPO optimization, we conduct two complementary experiments using the first A3T training cycle:

  1. SFT vs. GRPO on identical A3T data. Both methods start from the same WARD-PIG checkpoint and use the same adversarial and benign training samples as those used in A3T Cycle 1. For SFT, we generate reasoning annotations using the generator–evaluator pipeline described in Section 4.2, preserving WARD's structured output format. We perform full-parameter SFT for one epoch with a batch size of 16, while GRPO follows the original Cycle-1 configuration.

  2. Standard vs. adaptive adversarial data with GRPO. We generate 180 malicious samples from WARD-Seed using the WARD-Base generation procedure, matching the number used in A3T Cycle 1, while keeping the same benign balancing strategy and GRPO configuration.

All methods are evaluated on the same held-out configurations on WARD-Test (160 samples) with a maximum budget of 10 attack attempts per sample.

SSR: Sample Success Rate

Training methodWARD-0.8B SSR ↓WARD-2B SSR ↓
WARD-PIG checkpoint28.75%21.88%
+Standard data + GRPO26.88%22.50%
+A3T data + SFT23.13%16.88%
+A3T data + GRPO20.62%13.75%

Contribution of adaptive adversarial data. Standard data generation with GRPO yields limited improvement, whereas A3T-generated data reduces SSR to 20.62% and 13.75% for WARD-0.8B and WARD-2B, respectively. This highlights the importance of adaptive hard-example collection.

Contribution of GRPO. SFT on the same A3T data also improves robustness, achieving 23.13% and 16.88% SSR. GRPO provides further gains and is more straightforward to apply, as it does not require additional reasoning annotations.

Overall, both adaptive data collection and GRPO contribute to A3T's effectiveness, with adaptive data generation playing a particularly important role. We have added this clarification to the Appendix E of the revised version.

W3. Independent adaptive evaluation in a web-agent setting

To further strengthen our robustness claim, we extend PIArena [1]'s Strategy Search algorithm to 84 WASP web-agent scenarios. Notably, our original PIArena evaluation reported in the paper was conducted on general text-based tasks rather than web-agent environments. This additional experiment extends the evaluation to realistic web-agent interactions.

We follow PIArena's original algorithm and adapt it to jointly target WARD and the protected web agent, since bypassing the guard alone does not guarantee successful manipulation of the victim agent. For each scenario, GPT-5 acts as the attacker, generating 5 initial payloads for each of PIArena's 10 predefined strategies (50 candidates in total). Each candidate is evaluated against WARD and the Gemini-3-Flash victim agent. Following PIArena's evaluation procedure, GPT-4o serves as a judge to determine whether the victim agent follows the malicious instructions.

Based on WARD's predictions and the judge's assessment of agent behavior, PIArena filters and ranks the initial 50 candidates, prioritizing those that successfully manipulate the victim agent and show potential to bypass WARD. The PIArena attacker (GPT-5) then uses this feedback to refine the selected attacks for up to 50 additional attempts, stopping early if an attack both bypasses WARD and successfully manipulates the victim agent.

If no attack satisfies both conditions, we select an agent-successful candidate if available; otherwise, the scenario is counted as unsuccessful against the agent. We do not select attacks that bypass WARD but fail to manipulate the victim agent, as such bypasses do not constitute meaningful attack success and may result from modifications that render the payload harmless. This 50-initial + 50-refinement setup follows PIArena's default configuration. Further details are available in the PIArena repository and paper. Adaptive attack evaluation using PIArena on 84 WASP scenarios.

Target guard---Success in Tricking the Agent---Success in Tricking the Agent and Bypassing WARD
WARD-0.8B42/84 (50.00%)1/84 (1.19%)
WARD-2B50/84 (59.52%)1/84 (1.19%)
This additional experiment provides further evidence of WARD's robustness against an independent adaptive attacker.

We have added this experiment to Section 6.2 of the revised version.

[1] Piarena: A platform for prompt injection evaluation. Runpeng Geng et al.. ACL 2026

AuthorsOpenReview ↗

Response to the Reviewer fxEh (Part 3)

W4. Response to the concern about classification shortcuts

To address this concern, we conducted an additional evaluation on benign–malicious pairs with similar instruction wording, where benign examples also contain legitimate instructions.

Evaluation construction. We use six platforms: GitHub, Hugging Face, Custom Mail, Zalo (a messaging platform), Amazon, and Instagram. Benign content is first generated based on webpage screenshots, topics, and writing-style hints, covering README files, emails, messages, product content, and social posts. These examples may contain legitimate requests, recommendations, or instructions without attempting to redirect the agent.

Each malicious counterpart is then generated using its benign example as a reference, preserving the instruction phrasing, channel style, structure, approximate length, and opening wording while introducing an attack objective. Both examples share the same platform context and user task, and are injected into the webpage using the same procedure, following the corresponding injection channel and its required format. We use Gemini-3.8-Flash to generate both benign and malicious samples.

To ensure similarity in instruction wording, we require either a token-level Dice similarity ≥ 0.5 or at least 8 consecutive shared words in the instructions. We continue generating and filtering until we obtain 100 qualifying pairs per platform, yielding 600 pairs (1,200 examples).

Results. We evaluate both final WARD checkpoints (0.8B and 2B) on all 1,200 examples.

Per-platform and overall sample-level classification performance (%).

ModelMetricGitHubHugging FaceCustom MailZaloAmazonInstagramAll
WARD-0.8BAccuracy96.5096.5099.5099.00100.00100.0098.58
Recall96.0097.0099.00100.00100.00100.0098.67
Precision96.9796.04100.0098.04100.00100.0098.50
F196.4896.5299.5099.01100.00100.0098.58
WARD-2BAccuracy98.0098.0099.5099.00100.00100.0099.08
Recall98.0099.0099.0099.00100.00100.0099.17
Precision98.0097.06100.0099.00100.00100.0099.00
F198.0098.0299.5099.00100.00100.0099.08

We additionally report the stricter pair-level accuracy, where a pair is counted as correct only when both its benign and malicious examples are classified correctly.

Pair-level accuracy across six platforms.

PlatformPairsWARD-0.8BWARD-2B
GitHub10093/10096/100
Hugging Face10093/10096/100
Custom Mail10099/10099/100
Zalo10098/10098/100
Amazon100100/100100/100
Instagram100100/100100/100
Total600583/600 (97.17%)589/600 (98.17%)

WARD-0.8B produces 9 false positives and 8 false negatives, while WARD-2B produces 6 false positives and 5 false negatives.

Discussion. Despite substantial overlap in instruction wording, both checkpoints maintain high classification performance, achieving pair-level accuracies of 97.17% and 98.17%. These results provide additional evidence that WARD can distinguish legitimate instructions from malicious redirections even when their wording is similar, reducing the concern that its performance primarily relies on superficial classification shortcuts. We have added this experiment to the Appendix I of the revised version.

Comments on the paper

1 comment

Dear Reviewers,

Authors’ kindly responded to address your concerns. Please check rebuttals and share your opinions with authors. Thanks for your contributions!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.