- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents DefenseFlywheel, a framework that alternates between generating prompt-injection attacks and improving an agent's executable defense harness while keeping the underlying model fixed. The attacker searches for effective strategies and task-specific payloads, while the defender updates prompts, validators, and tool-control logic using execution feedback. The paper evaluates security and task utility on AgentDojo, including cross-attack, cross-model, and cross-scenario tests, and provides an additional browser-agent study on WASP.
Critical strengths
Overall, I think this is a well-polished paper. The writing, figures, framework design, and experimental presentation are all strong, and I appreciate the experimental contribution. Within the established direction of synthesizing attacks and using them to improve defenses, the paper brings the components together thoughtfully and evaluates them carefully.
Critical weaknesses
My main concern is the novelty and coverage of the attack-generation approach. Automated attack generation, evolutionary red-teaming and co-evolve attack & defense have a substantial history, including AutoDAN [1], SuperRed [2], and DARWIN [3]. I appreciate the extension to executable harnesses, but, in my experience, LLM-driven loops mainly produce textual variations or recombinations of familiar attacks (i.e., see paper of Adversarial Déjà Vu). These may help train guardrails, while the more consequential challenge for agent security is uncovering previously unrecognized failure mechanisms. The cross-attack experiments do not yet convince me that the generator addresses this challenge. I would be more interested in diverse external findings, such as those indexed in Promptfoo’s LM Security Database [4], and faithful reproductions of newly reported attacks. I would raise my score if the authors can show why their generator offers a compelling way to discover these blind spots.
BTW, I'm not sure if Gray Swan has released their prompt injection (PI) dataset, which contains a lot of real-world PI attacks. It might be worth testing the proposed defense.
[1] Xiaogeng Liu et al. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. ICLR 2024. Paper.
[2] SuperRed: Open-source AI Red-teaming and AI System Security Framework. Project.
[3] Weiwei Qi et al. DARWIN: Evolving Jailbreak Adversary and Guardrail for LLM Safety Evaluation and Protection. arXiv:2607.19829, 2026. Paper.
[4] Promptfoo. LM Security Database. Research catalog and API.
Questions for authors
See above, i only have one concern.
5 comments
Thanks! I've increased my score. Good luck!
Just a few more recommendations. AI safety, frontier LLMs, and coding agents are evolving rapidly, and I think it's worth looking beyond attacks that rely purely on malicious prompts.
For example, there is some interesting research on reward hacking, autoresearch, and safety failures that emerge during long-horizon tasks (e.g., Internal Safety Collapse, NeurIPS; ImpossibleBench, ICLR). These directions could offer useful insights into the next generation of defense.
Frontier models are already becoming much more robust against direct jailbreaks, especially with safeguards such as Constitutional AI and Prompt Guard. As these defenses improve, I expect attacks to become less obvious and more subtle, exploiting the environment, task objectives, or agent behavior rather than relying on explicit malicious instructions.
I think this is an important direction worth considering in future work.
Thank you for these suggestions. We agree that agent safety extends beyond explicit malicious instructions to failures arising from the environment, task objectives, and the agent’s own behavior.
Our DTap case study partially addresses this concern by examining indirect attacks embedded in tool descriptions, skill instructions, and application state, including emails, documents, and database records. This extends our analysis to environment-mediated attacks encountered during task execution. However, it remains focused on externally induced attacks and does not yet assess reward hacking, specification gaming, or unsafe behavior arising from the agent’s own long-horizon execution.
We have revised the Conclusion to identify these remaining issues as future directions, informed by the studies you highlighted. We also emphasize the need to evolve challenges and evaluation criteria to distinguish genuine task success from evaluator exploitation.
We sincerely thank the reviewer for their comments and suggestions, and provide a detailed discussion below:
We address this concern through methodological clarification and additional empirical evidence on mechanism coverage.
Three complementary design choices underpin our approach. First, co-evolution makes the attacker probe the updated harness, helping reduce specialization to a fixed target or attack bank. Second, harness-level evolution expands the search beyond prompts to validation and tool-control logic, exposing more implementation decisions to attack. Third, strategy–payload separation decouples reusable strategy search from task-specific textual refinement, helping prevent local rewrites from dominating exploration. Defense evolution prioritizes general principles before residual patterns. Our ablations support these choices (Section 4.5).
Our cross-attack experiments and qualitative analysis in Figure 3 provide initial evidence of mechanism coverage beyond the seed attacks. Figure 3 illustrates role-token forgery targeting role boundaries in chat templates, a mechanism conceptually equal to ChatInject (ICLR'26). In our cross-attack evaluation, the frozen evolved defense achieves 0.0% ASR against ChatInject, which was excluded from the initial attack set.
But we agree that the manuscript would benefit from a more comprehensive analysis using Promptfoo’s LM Security Database. Following your suggestion, we retrospectively analyzed all generated attack strategies retained across the five rounds of co-evolution and compared their mechanisms with recent studies indexed in the database. These strategies were generated using Claude Sonnet 4.6, with a reported reliable knowledge cutoff of August 2025. We disabled WebSearch and WebFetch during generation to limit contamination from external attack materials. Representative mechanism correspondences are summarized below.
These findings suggest that DefenseFlywheel can generate attacks corresponding to recently reported failure mechanisms, offering a promising approach to uncovering blind spots in evolving agent defenses. We have revised the manuscript to incorporate this analysis into Appendix E.2.
Thank you for this suggestion. Gray Swan has released a public subset of 95 prompt-injection attacks covering 28 behaviors [1]. We transferred a frozen defense evolved on AgentDojo to Gray Swan IPI Arena without further defense optimization. Each attack was run three times for both the baseline and DefenseFlywheel.
We used the same models for the baseline and DefenseFlywheel: Qwen3-VL as the victim model, Gemini 3.5 Flash for WorldSim, and GPT-6.1 Terra as the judge. We report attack success rate (ASR) under the benchmark’s official success rule and attack action rate (AAR), an auxiliary metric measuring the fraction of rollouts satisfying at least one harmful-action criterion.
Results are summarized below.
Both metrics decrease substantially, indicating improvements in attack resistance and suppression of the targeted harmful behaviors. These findings extend our evidence from transfer across domains within a benchmark to transfer across benchmarks, showing that the evolved defense retains security benefits on external attack scenarios without further optimization. We have revised the manuscript to incorporate these cross-benchmark evaluation results into Appendix I.
[1] https://huggingface.co/datasets/sureheremarv/ipi_arena_attacks