acceptodds
Under review as a conference paper at ICLR 2027

DefenseFlywheel: Securing Agents Through Harness-Level Attack–Defense Co-Evolution

Abstract

Existing defenses for LLM agents are often static: designed against known attacks, evaluated on fixed benchmarks, and manually updated after failures. This paradigm struggles to keep pace with the evolving attack surfaces, model behaviors, and environments of long-running agents. We introduce DefenseFlywheel, an attack–defense co-evolution framework that optimizes executable agent harnesses while keeping the underlying model frozen. DefenseFlywheel treats evolving attacks as an adversarial curriculum: an attacker iteratively discovers hard failures against the current defense stack, while a defender proposes harness-level patches spanning prompts, validators, and tool-control logic. Across five victim models on AgentDojo’s Banking suite, DefenseFlywheel reduces attack success from 36.3% to 3.2%, with negligible decrease in benign task completion and improved completion under attack. The evolved harnesses transfer across models and task suites without further optimization and generalize to unseen attacks, reducing attack success from 77.5% to 20.0% against an adaptive attacker. A WASP study supports applicability to browser agents. Ablations show that harness-level evolution achieves stronger security and utility than prompt optimization, highlighting executable harnesses as an effective optimization surface for adaptive agent defense. Our code and data are available at: https://anonymous.4open.science/r/DefenseFlyWheel-68DC.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agent, Security, Agent Harness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Z7inOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DefenseFlywheel, a framework that alternates between generating prompt-injection attacks and improving an agent's executable defense harness while keeping the underlying model fixed. The attacker searches for effective strategies and task-specific payloads, while the defender updates prompts, validators, and tool-control logic using execution feedback. The paper evaluates security and task utility on AgentDojo, including cross-attack, cross-model, and cross-scenario tests, and provides an additional browser-agent study on WASP.

Critical strengths

Overall, I think this is a well-polished paper. The writing, figures, framework design, and experimental presentation are all strong, and I appreciate the experimental contribution. Within the established direction of synthesizing attacks and using them to improve defenses, the paper brings the components together thoughtfully and evaluates them carefully.

Critical weaknesses

My main concern is the novelty and coverage of the attack-generation approach. Automated attack generation, evolutionary red-teaming and co-evolve attack & defense have a substantial history, including AutoDAN [1], SuperRed [2], and DARWIN [3]. I appreciate the extension to executable harnesses, but, in my experience, LLM-driven loops mainly produce textual variations or recombinations of familiar attacks (i.e., see paper of Adversarial Déjà Vu). These may help train guardrails, while the more consequential challenge for agent security is uncovering previously unrecognized failure mechanisms. The cross-attack experiments do not yet convince me that the generator addresses this challenge. I would be more interested in diverse external findings, such as those indexed in Promptfoo’s LM Security Database [4], and faithful reproductions of newly reported attacks. I would raise my score if the authors can show why their generator offers a compelling way to discover these blind spots.

BTW, I'm not sure if Gray Swan has released their prompt injection (PI) dataset, which contains a lot of real-world PI attacks. It might be worth testing the proposed defense.

[1] Xiaogeng Liu et al. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. ICLR 2024. Paper.

[2] SuperRed: Open-source AI Red-teaming and AI System Security Framework. Project.

[3] Weiwei Qi et al. DARWIN: Evolving Jailbreak Adversary and Guardrail for LLM Safety Evaluation and Protection. arXiv:2607.19829, 2026. Paper.

[4] Promptfoo. LM Security Database. Research catalog and API.

Questions for authors

See above, i only have one concern.

5 comments

We sincerely thank the reviewer for their comments and suggestions, and provide a detailed discussion below:

W1: The novelty and mechanism coverage of the attack generator remain insufficiently demonstrated, particularly its ability to discover previously unrecognized failure mechanisms beyond variations or recombinations of familiar attacks.

We address this concern through methodological clarification and additional empirical evidence on mechanism coverage.

Three complementary design choices underpin our approach. First, co-evolution makes the attacker probe the updated harness, helping reduce specialization to a fixed target or attack bank. Second, harness-level evolution expands the search beyond prompts to validation and tool-control logic, exposing more implementation decisions to attack. Third, strategy–payload separation decouples reusable strategy search from task-specific textual refinement, helping prevent local rewrites from dominating exploration. Defense evolution prioritizes general principles before residual patterns. Our ablations support these choices (Section 4.5).

Our cross-attack experiments and qualitative analysis in Figure 3 provide initial evidence of mechanism coverage beyond the seed attacks. Figure 3 illustrates role-token forgery targeting role boundaries in chat templates, a mechanism conceptually equal to ChatInject (ICLR'26). In our cross-attack evaluation, the frozen evolved defense achieves 0.0% ASR against ChatInject, which was excluded from the initial attack set.

But we agree that the manuscript would benefit from a more comprehensive analysis using Promptfoo’s LM Security Database. Following your suggestion, we retrospectively analyzed all generated attack strategies retained across the five rounds of co-evolution and compared their mechanisms with recent studies indexed in the database. These strategies were generated using Claude Sonnet 4.6, with a reported reliable knowledge cutoff of August 2025. We disabled WebSearch and WebFetch during generation to limit contamination from external attack materials. Representative mechanism correspondences are summarized below.

RoundStrategyPromptfoo entryPaperPublish date
2Delimiter escape + User-source ForgeryStructural Template Agent HijackAutomating Agent Hijacking via Structural Template Injection2026-02-18
3Tool-result SpoofingTool Stream Injection HijackVIGIL: Defending LLM Agents Against Tool Stream Injection via Verify-Before-Commit2026-01-09
4Role-token ForgeryN/AChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents2025-09-26
5Structural Data SpoofingTrusted-Metadata SpoofingAgent Data Injection Attacks are Realistic Threats to AI Agents2026-07-06

These findings suggest that DefenseFlywheel can generate attacks corresponding to recently reported failure mechanisms, offering a promising approach to uncovering blind spots in evolving agent defenses. We have revised the manuscript to incorporate this analysis into Appendix E.2.

W2: I'm not sure if Gray Swan has released their prompt injection (PI) dataset, which contains a lot of real-world PI attacks. It might be worth testing the proposed defense.

Thank you for this suggestion. Gray Swan has released a public subset of 95 prompt-injection attacks covering 28 behaviors [1]. We transferred a frozen defense evolved on AgentDojo to Gray Swan IPI Arena without further defense optimization. Each attack was run three times for both the baseline and DefenseFlywheel.

We used the same models for the baseline and DefenseFlywheel: Qwen3-VL as the victim model, Gemini 3.5 Flash for WorldSim, and GPT-6.1 Terra as the judge. We report attack success rate (ASR) under the benchmark’s official success rule and attack action rate (AAR), an auxiliary metric measuring the fraction of rollouts satisfying at least one harmful-action criterion.

Results are summarized below.

Metricw/o defenseDefenseFlywheel
ASR ↓34.3%11.3%
AAR ↓41.4%13.8%

Both metrics decrease substantially, indicating improvements in attack resistance and suppression of the targeted harmful behaviors. These findings extend our evidence from transfer across domains within a benchmark to transfer across benchmarks, showing that the evolved defense retains security benefits on external attack scenarios without further optimization. We have revised the manuscript to incorporate these cross-benchmark evaluation results into Appendix I.

[1] https://huggingface.co/datasets/sureheremarv/ipi_arena_attacks

Thanks! I've increased my score. Good luck!

Just a few more recommendations. AI safety, frontier LLMs, and coding agents are evolving rapidly, and I think it's worth looking beyond attacks that rely purely on malicious prompts.

For example, there is some interesting research on reward hacking, autoresearch, and safety failures that emerge during long-horizon tasks (e.g., Internal Safety Collapse, NeurIPS; ImpossibleBench, ICLR). These directions could offer useful insights into the next generation of defense.

Frontier models are already becoming much more robust against direct jailbreaks, especially with safeguards such as Constitutional AI and Prompt Guard. As these defenses improve, I expect attacks to become less obvious and more subtle, exploiting the environment, task objectives, or agent behavior rather than relying on explicit malicious instructions.

I think this is an important direction worth considering in future work.

Thank you for these suggestions. We agree that agent safety extends beyond explicit malicious instructions to failures arising from the environment, task objectives, and the agent’s own behavior.

Our DTap case study partially addresses this concern by examining indirect attacks embedded in tool descriptions, skill instructions, and application state, including emails, documents, and database records. This extends our analysis to environment-mediated attacks encountered during task execution. However, it remains focused on externally induced attacks and does not yet assess reward hacking, specification gaming, or unsafe behavior arising from the agent’s own long-horizon execution.

We have revised the Conclusion to identify these remaining issues as future directions, informed by the studies you highlighted. We also emphasize the need to evolve challenges and evaluation criteria to distinguish genuine task success from evaluator exploitation.

Reviewer_girNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DefenseFlywheel, a framework for improving agent security through the coevolution of attacks and executable defenses. With the backbone model frozen, the framework iteratively discovers attacks and updates the surrounding harness, including prompts, validators, and tool control logic. The method combines incremental patching, refinement from general security principles to specific attack patterns, and attack replay and augmentation. Experiments on AgentDojo and a small WASP case study show reduced attack success while largely preserving legitimate task completion.

Critical strengths
  1. The paper formulates agent security improvement as an iterative optimization problem over executable harness components. Using adversarial feedback to generate and select defense patches is a useful research direction, particularly because the approach keeps the underlying model fixed and can improve safeguards without requiring model retraining.

  2. The evaluation considers attack success, benign task completion, and legitimate task completion under attack. In the main Banking experiment, average attack success decreases from 36.3% to 3.2%, while benign task completion changes from 86.5% to 85.5%. These results suggest that the method can achieve substantial security gains while retaining useful agent functionality.

  3. The ablations examine harness optimization, staged refinement, attack replay and augmentation, and incremental patching. In particular, the comparison with prompt optimization supports the value of modifying executable harness components. The additional evaluations across attacks, task suites, and models also provide useful initial evidence of transferability.

Critical weaknesses
  1. Practical applicability to broader agent harnesses remains exploratory. The main experiments focus on AgentDojo, while the WASP study uses only six Reddit tasks and performs a separate round of defense optimization in that environment. This leaves open how effectively the algorithm can improve an agent framework intended for broad use, such as OpenClaw, with its tools, skills, memory, and persistent workflows. Evaluating such a harness would make the practical contribution more convincing.

  2. The evaluated threat model is narrow, focusing on indirect prompt injection while leaving other realistic safety risks unaddressed, including direct prompt injection and unintended unsafe actions during benign tasks, such as transferring an incorrect amount of money.

  3. The paper should therefore clarify the task identities and separation between optimization, validation, and final testing. Although the existing transfer experiments are encouraging, I would be more convinced by evaluation on richer workflows, such as those provided by the DecodingTrust-Agent Platform (DTap), which includes simulated applications and attacks through prompts, tools, skills, and environments. This should clearly distinguish transfer without further optimization from improvements obtained by running additional evolution on the target benchmark. Such evidence would strengthen the claim that the method learns reusable safeguards beyond its development setting.

Chen, Z., Liu, X., Tong, H., Guo, C., Nie, Y., Zhang, J., Kang, M., Xu, C., Liu, Q., Liu, X., Shi, T., Xiao, C., Koyejo, S., Liang, P., Guo, W., Song, D., & Li, B. (2026). DecodingTrust-Agent Platform (DTap): A controllable and interactive red-teaming platform for AI agents. arXiv. https://doi.org/10.48550/arXiv.2605.04808

Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Response-Part 1

Thank you for these constructive suggestions. To address your concerns about practical applicability, broader safety risks, and evaluation protocols, we conducted additional experiments on OpenClaw v2026.5.7 with GPT-5.6 Sol using DTap [1]. We evaluate co-evolution in Workflow, safety under adversarial and benign conditions, and transfer of the frozen defense to new tasks and domains. We further test cross-benchmark transfer from AgentDojo to Gray Swan IPI Arena. We have revised the manuscript to incorporate the OpenClaw/DTap results into Section 4.6.1. Detailed clarifications, experimental results, and analyses are provided below.

W1: Practical applicability to general-purpose agent frameworks remains insufficiently evaluated, as the experiments mainly focus on AgentDojo and a small WASP study.

We apply DefenseFlywheel to OpenClaw and perform five rounds of attack–defense co-evolution in DTap’s Workflow domain. At each round, attack rollouts provide execution feedback for proposing and evaluating updates to the agent harness.

The table summarizes changes in attack success rate (ASR), benign utility rate (BUR), and adversarial utility rate (AUR) within each round. Arrows indicate Attack Evolution Phase → Defense Evolution Phase.

Co-evolution StepASR ↓BUR ↑AUR ↑
135.7% → 12.6%82% → 82%51.2% → 58.6%
213.2% → 9.9%82% → 80%60.5% → 66.0%
314.3% → 8.8%80% → 86%56.8% → 64.8%
412.6% → 7.1%84% → 88%56.2% → 58.0%
510.8% → 5.4%86% → 88%52.7% → 59.4%

The results show an overall decline in ASR across co-evolution rounds, while BUR and AUR remain broadly stable and improve in later rounds. This indicates that the evolving defense can address the newly discovered attacks while preserving task utility, supporting DefenseFlywheel’s practical applicability to more complex, general-purpose agent harnesses.

W2: The evaluated threat model is limited to indirect prompt injection and does not cover other realistic agent safety risks, such as direct attacks and unintended unsafe actions during benign tasks.

Thank you for this suggestion. We broaden our evaluation to cover diverse adversarial threats and execution safety during benign interactions. Our evaluation spans ten DTap domains and includes direct attacks through user prompts, alongside indirect attacks through tool descriptions, skill instructions, and application state, including emails, documents, and database records. The tasks include combinations of these injection surfaces and cover safety objectives such as data exfiltration, financial fraud, destructive actions, harmful communications, deception, and discrimination.

Threat modelOriginal OpenClaw ASR ↓Evolved defense ASR ↓
Direct attacks40.5%29.5%
Indirect prompt injection30.0%10.5%

The results show that DefenseFlywheel reduces ASR under both direct and indirect attacks while maintaining aggregate benign task completion. These improvements across diverse attack surfaces and application domains support its effectiveness against a broader range of adversarial risks beyond indirect prompt injection.

Furthermore, we examine execution safety during benign payment tasks. Following your recommendation, we evaluate correctly specified requests and requests containing accidental input errors, with 20 rollouts per configuration in each setting. We measure task completion in the former and the execution of unintended actions in the latter.

MetricOriginal OpenClawEvolved defense
Task completion ↑6/20 (30%)16/20 (80%)
Unintended actions ↓17/20 (85%)0/20 (0%)

The payment results show improved completion of valid requests while reducing unintended actions arising from user input errors, supporting safer execution during benign interactions.

AuthorsOpenReview ↗

Response-Part 2

W3: Task identities and the separation between optimization, validation, and testing need clarification, together with evaluation on richer workflows that distinguishes frozen transfer from further adaptation.

We would like to clarify that our main AgentDojo experiments share the same 20 attacked Banking instances and 16 benign tasks across optimization, candidate validation, and final evaluation. The attack protocols used at these stages are summarized below.

StageAttack set
OptimizationCurrent attacks + replay + augmentation
Candidate validationCurrent attacks + 5 historical + 5 shadow attacks
Final evaluation5 native + 5 evolved attacks (best per round)

To evaluate generalization to richer workflows, we further evolve a defense in DTap’s Workflow domain and evaluate it on additional Workflow tasks excluded from optimization, together with tasks from nine other domains. No further defense optimization or checkpoint selection is performed on these evaluation domains. Arrows indicate original OpenClaw → frozen evolved defense.

DomainDirect ASR ↓Indirect ASR ↓BUR ↑AUR ↑
Workflow40.0% → 34.5%15.0% → 2.5%70.0% → 77.5%60.0% → 62.5%
CRM65.0% → 48.5%35.0% → 15.0%80.0% → 77.5%70.0% → 72.5%
Finance20.0% → 9.0%20.0% → 7.5%95.0% → 97.5%67.5% → 80.0%
Legal37.5% → 29.5%27.5% → 5.0%90.0% → 85.0%75.0% → 70.0%
Travel20.0% → 13.5%15.0% → 5.0%82.5% → 75.0%45.0% → 55.0%
Telecom45.0% → 31.5%35.0% → 0.0%65.0% → 62.5%90.0% → 90.0%
Research50.0% → 38.5%12.5% → 10.0%100.0% → 100.0%55.0% → 55.0%
Code27.0% → 18.0%82.5% → 57.5%92.5% → 100.0%92.5% → 95.0%
Browser37.5% → 31.5%10.0% → 0.0%100.0% → 100.0%60.0% → 77.5%
Customer service64.0% → 39.5%46.5% → 3.5%87.5% → 87.5%100.0% → 100.0%
Average40.5% → 29.5%30.0% → 10.5%86.5% → 86.5%71.5% → 76.0%

The results show that the frozen defense reduces ASR while preserving both benign utility and adversarial utility. These results support the transferability of the evolved safety policy across different applications and task structures.

Beyond transfer within DTap, we further evaluate cross-benchmark generalization from AgentDojo to Gray Swan IPI Arena [2]. We transfer defense evolved on AgentDojo to IPI Arena without further defense optimization. We report ASR and attack action rate (AAR), an auxiliary metric measuring the fraction of rollouts satisfying at least one of the benchmark’s harmful-action criteria.

Metricw/o defenseDefenseFlywheel
ASR ↓34.3%11.3%
AAR ↓41.4%13.8%

The reductions in both metrics show that the evolved defense remains effective against external attacks under a total different benchmark. This cross-benchmark evaluation verifies the generalization across task instances, application domains, and evaluation environments. We have revised the manuscript to incorporate these cross-benchmark evaluation results into Appendix I.

[1] Chen, Z., Liu, X., Tong, H., Guo, C., Nie, Y., Zhang, J., ... & Li, B. (2026). DecodingTrust-Agent platform (DTap): A controllable and interactive red-teaming platform for AI agents. arXiv preprint arXiv:2605.04808. [2] Dziemian, M., Lin, M., Fu, X., Nowak, M., Winter, N., Jones, E., ... & Kolter, Z. (2026). How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition. arXiv preprint arXiv:2603.15714.

My previous concerns have been resolved. However, I read Reviewer Z7in's comments and find the concerns reasonable. In particular, I share the concern about the novelty of the attack evolution approach. I would defer the assessment of this novelty issue to the other reviewers and AC. If they find it satisfactory, I would be willing to raise my score.

Thank you for acknowledging that our response addresses your previous concerns. Reviewer Z7in has also positively assessed the additional evidence and increased their score. We would like to briefly summarize the methodological rationale and empirical findings supporting our approach.

DefenseFlywheel couples an evolving adversarial curriculum with an executable harness search space: the attacker probes weaknesses in the updated implementation, while the defender can modify validation and tool-control logic alongside prompts. Separating reusable strategy search from task-specific payload refinement further encourages exploration beyond local textual rewrites. Our matched-budget ablation supports this interaction, with 7.7% ASR on unseen attacks versus 38.1% for fixed-bank optimization. The Promptfoo database analysis additionally identifies correspondences between generated attacks and recently reported failure mechanisms, supporting the approach’s potential to expose blind spots in evolving defenses.

If our responses have addressed your reservations, we would be grateful if you could consider updating your scores accordingly.

Reviewer_Hc8jOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

DefenseFlywheel automatically improves the executable harness around a frozen agent model through alternating attack and defense search. The attacker develops reusable strategies and task-specific payloads; the defender turns observed failures into patches to prompts, validators, and tool-control logic. A branching patch history, attack replay, and augmentation support the search, with candidate selection constrained by benign and adversarial utility. The experiments cover AgentDojo, transfer across models and task suites, unseen and adaptive attacks, and a smaller browser-agent study on WASP.

Critical strengths
  • The paper makes a persuasive case for treating executable harnesses as an optimization surface for agent security. This is a useful contribution even though the individual search components are familiar. The prompt-only comparison in Figure 4 is particularly relevant: harness evolution reaches roughly 10% ASR versus 27% for prompt optimization, while preserving better benign utility.
  • The separate ablations on staged refinement, replay, and patch preservation help explain why the complete system works. The result goes beyond simply showing that an LLM can write a guardrail. The strongest evidence is that the resulting defenses remain useful after freezing. A Banking harness transfers to three unseen suites without further optimization, reducing average ASR from 16.7% to 0.2% while increasing benign completion from 77.9% to 84.6%.
  • The additional evaluations on excluded attack methods, an adaptive attacker, and other victim models make this more convincing than an evaluation against the search population alone. Reporting adversarial utility also matters: several baselines obtain low ASR largely by preventing useful work, whereas the evolved harness generally retains it.
Critical weaknesses
  1. The separation between development performance and held-out evaluation needs to be clearer. Appendix A describes 20 attacked instances and 16 benign tasks for evolution and repeated evaluation, without identifying a disjoint Banking test set. The six-task WASP study has the same ambiguity. I therefore would not interpret their final scores as generalization to new tasks within those environments. The frozen cross-suite experiment provides stronger evidence and is an important reason I remain positive. Appendix E also needs the adaptive attacker's query budget and evaluation procedure so that the 20.0% residual ASR has a reproducible interpretation.
  2. The ablations establish the value of executable patches more directly than the value of co-evolution itself. There is no comparison with harness optimization against a fixed, strong attack bank under the same total search budget. Such a control would distinguish benefits from the moving adversarial curriculum from benefits of additional code-search effort. This limits the attribution of the gains to attack–defense co-evolution, although it does not negate the demonstrated usefulness of the final harnesses.
Questions for authors
  1. Were the Banking instances and WASP tasks used for the final comparisons also used during evolution or candidate selection? Please state the overlap explicitly and identify any evaluation on held-out task instances, separately from the cross-suite results.
  2. How does co-evolution compare with optimizing the harness against a fixed attack bank when proposer calls, victim evaluations, and defense-search opportunities are matched? This would clarify which benefit comes specifically from updating the attacker.
  3. What query and iteration budgets were used for AGENTVIGIL against each defense? Were selected attacks re-evaluated with fresh rollouts, and does the reported ASR measure the best attack found per instance or an average over generated candidates?
3 comments
AuthorsOpenReview ↗

Response-Part 1

We sincerely thank the reviewer for their comments and suggestions, and provide a detailed discussion below:

W1 & Q1: The overlap between development and final evaluation on Banking and WASP is unclear, leaving generalization to held-out tasks within these environments insufficiently established.

We apologize for not making the task overlap sufficiently clear. We would like to clarify that the main Banking and WASP comparisons reuse task instances from evolution for the final evaluation. The cross-attack and adaptive-attack experiments assess robustness to different attacks on these instances. Our preliminary experiments suggested that instances within a suite often share similar tool interfaces and workflow structures, motivating our emphasis on cross-suite transfer to assess larger distributional shifts. Specifically, we freeze the defense evolved on Banking and evaluate it on Slack, Travel, and Workspace without further optimization, as reported in the cross-scenario experiments in Section 4.4 (Table 3).

Furthermore, during this rebuttal period, we conducted additional experiments on DTap and Gray Swan IPI Arena. On DTap, we evolved an OpenClaw defense in Workflow, froze it, and evaluated it on additional Workflow tasks excluded from optimization and validation, together with tasks from nine other domains. On Gray Swan IPI Arena, we transferred a frozen defense evolved on AgentDojo without further optimization, reducing ASR from 34.3% to 11.3%. These evaluations provide complementary evidence of generalization to new task instances, different domains, and an external benchmark. Detailed results are provided in our responses to Reviewer girN, W3, and Reviewer Z7in, W2.

[1] Chen, Z., Liu, X., Tong, H., Guo, C., Nie, Y., Zhang, J., ... & Li, B. (2026). DecodingTrust-Agent platform (DTap): A controllable and interactive red-teaming platform for AI agents. arXiv preprint arXiv:2605.04808. [2] Dziemian, M., Lin, M., Fu, X., Nowak, M., Winter, N., Jones, E., ... & Kolter, Z. (2026). How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition. arXiv preprint arXiv:2603.15714.

W2 & Q2: The specific benefit of attack–defense co-evolution remains unclear without comparison to harness optimization against a fixed, strong attack bank under matched search budgets.

Thank you for this suggestion. We conducted an additional ablation that fixes the initial attack bank while continuing to optimize the harness. The search budget and other settings are kept consistent with full co-evolution, allowing us to assess the contribution of updating the attacker.

We evaluate both variants on three attack sets: seed attacks are the five native AgentDojo attacks; evolved attacks contain the strongest attack from each of the five co-evolution rounds; and unseen attacks comprise the four static methods in our cross-attack evaluation—ChatInject, TopicAttack, FalseError, and Multilingual Attack. The table reports ASR on these shared evaluation sets.

Evaluation attack setNo DefenseFixed-Bank OptimizationCo-Evolution
Seed attacks51.3%2.9%5.6%
Evolved attacks55.5%32.8%6.2%
Unseen attacks51.6%38.1%7.7%

Fixed-bank optimization achieves lower ASR on the seed attacks, but its effectiveness decreases substantially on evolved and unseen attacks. Co-evolution maintains low ASR across all three sets, indicating that continually updating the attacker helps the defense generalize beyond the initial attack bank. Under matched search budgets, these results support the contribution of the evolving adversarial curriculum to the observed robustness gains. We have revised the manuscript to incorporate these ablation results into Section 4.5 (Figure 4).

AuthorsOpenReview ↗

Response-Part 2

Q3: The AGENTVIGIL evaluation requires clearer reporting of query and iteration budgets, attack re-evaluation, and ASR aggregation to make the reported results reproducible.

Thank you for requesting these details. Following AGENTVIGIL’s mutation schedule, we independently optimize attacks against each frozen defense for 10 search iterations, generating three mutated attack candidates per iteration. Each candidate is evaluated on the same 20 Banking cases, with one victim-agent rollout per case. The mutation phase therefore comprises 30 attack candidates and 600 candidate-evaluation rollouts per defense. Here, a query denotes a complete agent rollout, which may involve multiple underlying LLM calls.

For each defense, we select one attack with the highest mean search-stage ASR across the 20 cases, rather than a different best attack for each case. We then freeze the selected attack and re-evaluate it with eight fresh rollouts per case, yielding 160 evaluation rollouts per defense. The reported ASR averages attack success over these 160 fresh evaluation rollouts, rather than over all generated candidates. These outcomes are not used for attack selection or further refinement. Search and final evaluation use the same case identities; the fresh rollouts provide a separate estimate of the selected attack’s repeatability. We have clarified the search budget, attack selection, re-evaluation, and ASR aggregation protocol in Appendix F of the revised manuscript.

We thank the reviewer for their thoughtful feedback. In our responses and revised manuscript, we have clarified the task overlap and frozen-transfer evaluations, added a matched-budget comparison with fixed-bank optimization, and detailed AGENTVIGIL’s search budget, attack selection, and fresh-rollout evaluation protocol. We would appreciate it if you could take another look and let us know if any concerns remain. If these additions have addressed your reservations, we would be grateful if you could consider updating your assessment and score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.