1. The contribution of the adaptive feedback loop and the meaning of the evaluation metric are not sufficiently validated.
AutoDojo’s central idea is to iteratively improve injections using attack-success feedback obtained from AgentDojo. However, the paper lacks a feedback-free best-of-K baseline that uses the same optimizer LLM, seeds, number of candidates, and target rollout budget, but generates candidates independently without access to previous outcomes. Consequently, it is unclear whether the improvement comes from genuine feedback-guided adaptation or simply from generating multiple candidates with a strong frontier LLM and selecting the best one. Varying the number of iterations does not resolve this issue because it simultaneously increases both the number of candidates and the amount of accumulated feedback.
Moreover, AutoDojo is weaker than the static attack in several settings in Figure 3. For example, on Gemini-2.5-Flash, ASR decreases from 50.4% to 34.7% without a defense and from 54.5% to 36.2% under Spotlighting. The authors attribute this to requiring AutoDojo to return a newly generated injection. However, a realistic attacker could retain a seed whenever newly generated candidates are weaker. The reported AutoDojo ASR therefore represents neither the strongest attack available to a realistic adaptive adversary nor the isolated contribution of feedback. The paper should separately report the best-seed ASR, feedback-free best-of-K ASR, best-generated ASR, and their maximum as the effective adaptive-attacker ASR.
2. AutoDojo discovers more successful attacks than AgentDojo, but provides limited new generalizable security insight from these discoveries.
AutoDojo identifies vulnerabilities missed by AgentDojo’s fixed attack, particularly showing that content-filtering defenses such as PIGuard and DataFilter can be bypassed when malicious instructions are expressed as legitimate-looking data. The observation that provenance- and information-flow-based defenses such as CaMeL and ROPE are comparatively more robust is also useful. However, these findings largely align with the established expectation that defenses relying on surface-level attack patterns are vulnerable to adaptive attacks. The paper provides limited analysis of generalizable failure patterns beyond reporting higher ASR.
In particular, the paper does not report how many cases transition from static-attack failure to AutoDojo success, how many fail under both attacks, or how many succeed under the static attack but fail under AutoDojo. It also does not systematically analyze which strategies successful injections use or at which stage attacks succeed or fail, such as filter evasion, action selection, tool invocation, argument generation, or policy enforcement. The billing example in Figure 13 is informative but remains a single case study.
The under-specification analysis also has important limitations. AgentDojo contains only four action-open user tasks, and task category is confounded with suite, tool usage, and execution complexity. The absence of the proposed ordering in AgentDyn’s DailyLife suite further weakens the structural conclusion. Thus, while the paper establishes that AutoDojo finds more successful attacks than AgentDojo, it provides limited failure taxonomy and actionable security insight of the kind expected from a new benchmark. A case-level transition matrix based on existing logs, systematic analysis of attack strategies and failure stages, quantitative analysis of defense-ranking changes, and controlled comparisons that vary only task-specification precision would substantially strengthen the contribution.
We thank the reviewer for the review. We address each weakness below.
W1: Methodological novelty. AutoDojo is not proposing a new attack method; its contribution is a generative benchmark that does not go stale as LLMs evolve. Our motivation is that such a benchmark should evaluate a defense (as well as the underlying model) against the adaptive attacker it will realistically face, and existing adaptive attacks are costly to run for that purpose: White-box attacks need the target model’s gradients, which a deployed agent does not expose. Human red-teaming is neither scalable nor reproducible. RL-trained attackers need a separate training run for each target, where data collection and reward design requires substantial efforts and resources. These attacks can show that a defense is breakable in principle, but none gives a cheap, repeatable score for every (agent, defense) pair. AutoDojo instead uses what any realistic attacker already has: an off-the-shelf LLM, a few queries, and the simplest observable feedback. Moreover, existing attacks can be used as seeds, so the benchmark becomes stronger as attacks improve.
W2: Frontier models. We did not run the full evaluation on the newest frontier models, both because of cost and because these models likely have seen these benchmark-style injections during training, so they can appear very robust to the attacks in these formats. However, on DeepSeek-v4-Pro (Appendix C.1), PIGuard brings the static ASR to 0.0%, yet AutoDojo discovers injections that reach 8.2%.
W3: Repeated runs. It is relatively common in agent benchmarks and evaluations to run each experiment once because of cost. With this limitation in mind, we do not draw conclusions from small differences in results, because they can genuinely be due to the underlying randomness of the models. Because static and AutoDojo injections are evaluated on the same 389 cases, we can still test each comparison with a paired test. The table below uses the exact McNemar test, which counts the cases that flip in each direction.
These tests only cover the sampling of cases, not the randomness of re-running the optimizer or the evaluated agent. In Appendix C.3 (Figure 9), we replace the optimizer LLM and find that it changes ASR by at most 3.1 points (28.0%, 26.2% and 24.9%), much smaller than the main gaps above (for example, 28.0 points for PIGuard on GPT-4o-mini).