- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
AgentSafeEvo extends conventional co-evolution from attack prompts versus safe responses to adversarial task construction versus safe task execution. Rather than optimizing jailbreak prompts alone, the attacker jointly constructs user queries, tool specifications, and environmental observations, while the defender performs multi-turn tool-mediated execution. Both agents are jointly optimized with GRPO, using the within-task standard deviation of defender rewards as a learnability signal to weight their respective advantages.
Critical strengths
- AgentSafetyEvo extends adversarial co-evolution from prompt–response safety to full agent interactions, jointly modeling adversarial task construction and safe task execution.
- The framework uses within-task variation in defender rewards as a proxy for task learnability, which adaptively weights both attacker and defender GRPO updates.
- Cross-evaluation shows that later attackers increasingly compromise fixed defenders, whereas later defenders become more robust to fixed attackers. The evolved attacker further transfers to larger models, including Llama-3.3-70B and GPT-5, indicating adaptive co-evolution rather than overfitting to a fixed training distribution.
- AgentSafetyEvo achieves state-of-the-art attack and defense performance while largely preserving performance on benign tasks.
Critical weaknesses
- Both attacker and defender training and evaluation are largely restricted to the Qwen model family. Although Fig. 3 evaluates attack transfer to models of different scales, it does not report whether the learned defense generalizes beyond Qwen, leaving cross-model defense robustness unclear.
- The comparison with prior agentic attack methods is incomplete. In particular, the paper discusses ARLAs but does not include it in the empirical evaluation, making it difficult to assess whether the proposed attacker improves over recent state-of-the-art agent-level attacks.
- The reward model relies on LLM-based judging, which can introduce systematic evaluation errors. The paper should report judge reliability, ideally through human expert annotation of a representative subset and corresponding agreement or error rates.
Questions for authors
- For equations that are not referenced in the text, please use an unnumbered environment such as
\begin{align*}...\end{align*}. - Please report the out-of-domain attack success rate after excluding the six adversarial tasks used during training. This would better quantify whether the learned attacker generalizes to genuinely unseen attack scenarios.
- The evaluation of general capabilities is relatively narrow and mostly agent/tool-use oriented. Since the paper claims that AgentSafeEvo preserves general capabilities, it would be valuable to evaluate broader abilities such as code generation (e.g., LiveCodeBench/HumanEval) and mathematical reasoning (e.g., AIME/MATH-500) to rule out capability degradation outside the agent benchmarks.
8 comments
W3: Human Evaluation of Reward-Judge Reliability
Following the evaluation criteria in Appendix C.3, we manually reviewed 200 randomly sampled trajectories, assessing both trajectory-level safety and legitimate-task completion (utility).
| Dimension | Sample Size | Human–Model Agreement | Disagreement |
|---|---|---|---|
| Safety | 200 | 93.5% (187) | 6.5% (13) |
| Utility | 200 | 93.0% (186) | 7.0% (14) |
These results show high overall human–model agreement on both reward dimensions across the sampled trajectories, supporting the reliability of Qwen3.8-27B’s outcome judgments.
Q1: Equation Numbering
We will remove equation numbers from all unreferenced equations in the revision.
Q2: Generalization of the Learned Attacker
We examine whether co-evolution enables the attacker to develop effective strategies beyond those explicitly targeted during initialization. We compare initial and evolved attacks against the same base Qwen3-8B defender, with external defense evaluations on MSB providing complementary evidence.
Evolution of attack strategies. We observe a progression from general authorization or compliance pretexts to malicious directives embedded in error-recovery workflows. Recovery-workflow hijacking presents an additional tool operation as necessary to resolve an error or complete the task. Authority-based answer overriding embeds purported system or compliance directives in error feedback to induce attacker-specified outputs. For example, an initial attack requests an archival operation on compliance grounds, whereas the evolved attack presents it as necessary to repair a schema error and restore the session.
These strategies connect to error-path authority exploitation in VATS [1] and runtime-feedback injection in CARI [2]. We did not explicitly target these error-recovery mechanisms when constructing the attacker’s SFT data. In the identified late-stage subsets, attacks exhibiting these mechanisms achieved ASRs of approximately 70% and 75%, respectively, against the base Qwen3-8B defender. This analysis characterizes the evolution and effectiveness of concrete attack strategies within the six broad attack categories.
External defense evaluation. We further evaluate the trained defenders on two selected MSB subsets: Preference Manipulation + Out-of-Scope Parameter (PM + OP) and Prompt Injection + User Impersonation (PI + UI).
| Model | Attack subset | Base ASR ↓ | Static-AT ASR ↓ | AgentSafeEvo ASR ↓ |
|---|---|---|---|---|
| Qwen3-8B | PM + OP | 36.25% | 23.75% | 10.42% |
| Qwen3-8B | PI + UI | 86.44% | 76.89% | 76.00% |
| Gemma4-E4B | PM + OP | 53.75% | 37.08% | 29.17% |
| Gemma4-E4B | PI + UI | 85.11% | 32.22% | 7.56% |
AgentSafeEvo improves over Static-AT across these selected subsets. On PM + OP, ASR decreases from 23.75% to 10.42% for Qwen3-8B and from 37.08% to 29.17% for Gemma4-E4B. On PI + UI, Gemma4-E4B improves substantially, from 32.22% to 7.56%, while the improvement for Qwen3-8B is limited.
These results show that co-evolution can develop concrete attack strategies not explicitly designed during initialization and improve defender resistance to external attack combinations. We will include these analyses in the revised manuscript.
Q3: Broader Evaluation of General Capabilities
We expanded our evaluation to four additional benchmarks: AIME2025 for mathematical reasoning, LiveCodeBench v5 for code generation, MMLU-Redux for general knowledge, and IFEval for instruction following. We evaluate both Qwen3-8B and Gemma4-E4B before and after AgentSafeEvo training.
| Model | AIME2025 ↑ | LiveCodeBench v5 ↑ | MMLU-Redux ↑ | IFEval ↑ |
|---|---|---|---|---|
| Qwen3-8B | 66.67 ± 5.77 | 54.09 ± 2.49 | 87.84 ± 0.33 | 83.30 ± 0.47 |
| + AgentSafeEvo | 70.00 ± 5.77 | 54.49 ± 1.58 | 87.10 ± 0.53 | 84.04 ± 0.28 |
| Gemma4-E4B | 41.11 ± 1.92 | 58.08 ± 0.60 | 83.84 ± 0.29 | 87.18 ± 0.65 |
| + AgentSafeEvo | 43.33 ± 3.33 | 60.28 ± 2.77 | 84.01 ± 0.14 | 87.86 ± 0.47 |
Across both model families, AgentSafeEvo largely preserves general capabilities, with modest improvements in mathematical reasoning, code generation, and instruction following. These results provide broader evidence for capability preservation beyond agent and tool-use tasks. We will include these additional evaluations in the revised manuscript.
References
[1] VATS: Exploiting Implicit Authority in Error-Path Injection via Systematic Mutation. arXiv:2606.07992.
[2] CARI: Cross-Boundary Adversarial Runtime Injection in Coding-Agent Workflows. arXiv:2610.38799.
We thank the reviewer for the thoughtful comments and constructive suggestions. We have conducted additional evaluations and analyses to address the concerns about cross-model generalization, reward reliability, attacker evolution, and general capabilities. We respond to each point below.
W1: Generalization of Defense Training Across Model Families
To evaluate whether the defensive improvements extend beyond Qwen3-8B, we applied the full AgentSafeEvo pipeline to Gemma4-E4B, a different model family with a distinct native tool-calling format. Following the main paper’s evaluation protocol, we report the mean and standard deviation over three runs across all safety and general-capability benchmarks.
Agent safety results:
General agent capability results:
On general capabilities, AgentSafeEvo improves ALFWorld and ToolHop performance over both the base model and Static-AT, while BFCL-V3 accuracy remains close to the Static-AT result. Together, these experiments support the applicability of AgentSafeEvo’s defense-training framework across Qwen and Gemma model families. We will include these additional results in the revised manuscript.
W2: Comparison and Methodological Differences with ARLAS
ARLAS is the closest prior work on adversarial reinforcement learning for LLM agent safety. Its training code, method-specific training data, and trained model checkpoints have not been officially released, making a faithful reproduction and matched experimental comparison difficult.
AgentSafeEvo differs from ARLAS in three aspects:
We will expand the discussion of these methodological differences and distinguish them from the empirical comparisons against static adversarial training and inference-time defenses.