acceptodds
Under review as a conference paper at ICLR 2027

AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety

13 views

Abstract

Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker–defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Agent Safety, Adversarial Co-evolution, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YofGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

AgentSafeEvo extends conventional co-evolution from attack prompts versus safe responses to adversarial task construction versus safe task execution. Rather than optimizing jailbreak prompts alone, the attacker jointly constructs user queries, tool specifications, and environmental observations, while the defender performs multi-turn tool-mediated execution. Both agents are jointly optimized with GRPO, using the within-task standard deviation of defender rewards as a learnability signal to weight their respective advantages.

Critical strengths
  • AgentSafetyEvo extends adversarial co-evolution from prompt–response safety to full agent interactions, jointly modeling adversarial task construction and safe task execution.
  • The framework uses within-task variation in defender rewards as a proxy for task learnability, which adaptively weights both attacker and defender GRPO updates.
  • Cross-evaluation shows that later attackers increasingly compromise fixed defenders, whereas later defenders become more robust to fixed attackers. The evolved attacker further transfers to larger models, including Llama-3.3-70B and GPT-5, indicating adaptive co-evolution rather than overfitting to a fixed training distribution.
  • AgentSafetyEvo achieves state-of-the-art attack and defense performance while largely preserving performance on benign tasks.
Critical weaknesses
  • Both attacker and defender training and evaluation are largely restricted to the Qwen model family. Although Fig. 3 evaluates attack transfer to models of different scales, it does not report whether the learned defense generalizes beyond Qwen, leaving cross-model defense robustness unclear.
  • The comparison with prior agentic attack methods is incomplete. In particular, the paper discusses ARLAs but does not include it in the empirical evaluation, making it difficult to assess whether the proposed attacker improves over recent state-of-the-art agent-level attacks.
  • The reward model relies on LLM-based judging, which can introduce systematic evaluation errors. The paper should report judge reliability, ideally through human expert annotation of a representative subset and corresponding agreement or error rates.
Questions for authors
  • For equations that are not referenced in the text, please use an unnumbered environment such as \begin{align*}...\end{align*}.
  • Please report the out-of-domain attack success rate after excluding the six adversarial tasks used during training. This would better quantify whether the learned attacker generalizes to genuinely unseen attack scenarios.
  • The evaluation of general capabilities is relatively narrow and mostly agent/tool-use oriented. Since the paper claims that AgentSafeEvo preserves general capabilities, it would be valuable to evaluate broader abilities such as code generation (e.g., LiveCodeBench/HumanEval) and mathematical reasoning (e.g., AIME/MATH-500) to rule out capability degradation outside the agent benchmarks.
8 comments

We thank the reviewer for the thoughtful comments and constructive suggestions. We have conducted additional evaluations and analyses to address the concerns about cross-model generalization, reward reliability, attacker evolution, and general capabilities. We respond to each point below.

W1: Generalization of Defense Training Across Model Families

To evaluate whether the defensive improvements extend beyond Qwen3-8B, we applied the full AgentSafeEvo pipeline to Gemma4-E4B, a different model family with a distinct native tool-calling format. Following the main paper’s evaluation protocol, we report the mean and standard deviation over three runs across all safety and general-capability benchmarks.

Agent safety results:

MethodAgent-SafetyBench ASR ↓AgentHarm Harm score ↓AgentDojo ASR ↓AgentDojo UA ↑MSB ASR ↓MSB NRP ↑
Gemma4-E4B39.00 ± 0.6424.08 ± 0.8215.45 ± 0.8075.66 ± 0.2153.49 ± 0.7833.59 ± 0.80
+ Spotlighting39.97 ± 0.3527.32 ± 0.7114.61 ± 0.7075.55 ± 0.5954.10 ± 0.2833.85 ± 0.83
+ Prompt Guard 236.34 ± 0.2023.39 ± 0.438.61 ± 0.5265.26 ± 0.8853.20 ± 0.6333.21 ± 1.38
+ Static-AT19.50 ± 0.5612.08 ± 0.576.36 ± 0.4077.59 ± 0.4327.26 ± 1.0354.82 ± 1.61
+ AgentSafeEvo (ours)16.08 ± 0.567.92 ± 0.252.18 ± 0.1278.71 ± 0.4620.54 ± 0.4562.03 ± 1.38

General agent capability results:

MethodBFCL-V3 ↑ALFWorld ↑ToolHop ↑
Gemma4-E4B66.51 ± 0.1226.30 ± 0.5341.94 ± 0.71
+ Spotlighting66.45 ± 0.2327.83 ± 2.6541.65 ± 0.42
+ Prompt Guard 266.19 ± 0.1727.83 ± 0.5341.24 ± 0.35
+ Static-AT65.08 ± 0.2329.66 ± 2.8044.46 ± 0.90
+ AgentSafeEvo (ours)64.71 ± 0.1732.42 ± 2.1246.33 ± 0.70

On general capabilities, AgentSafeEvo improves ALFWorld and ToolHop performance over both the base model and Static-AT, while BFCL-V3 accuracy remains close to the Static-AT result. Together, these experiments support the applicability of AgentSafeEvo’s defense-training framework across Qwen and Gemma model families. We will include these additional results in the revised manuscript.

W2: Comparison and Methodological Differences with ARLAS

ARLAS is the closest prior work on adversarial reinforcement learning for LLM agent safety. Its training code, method-specific training data, and trained model checkpoints have not been officially released, making a faithful reproduction and matched experimental comparison difficult.

AgentSafeEvo differs from ARLAS in three aspects:

  1. Broader threat coverage. ARLAS’s reported training setup focuses on indirect prompt injection into web observations to induce user-information leakage. Our training data spans 50 agent scenarios and six attack categories: direct prompt injection, indirect prompt injection, tool-description injection, misinformation, harmful tool execution, and user-query jailbreaks. This includes unsafe behaviors that do not require injected instructions, such as directly advancing a harmful objective through tool execution.
  2. Learned adversarial task construction. ARLAS generates injections during the execution of supplied tasks. In AgentSafeEvo, the attacker generates user queries, tool definitions, and environment observations from scenario and intent seeds. The task instructions and tool-environment configurations themselves therefore become part of the evolving training distribution.
  3. Task-level optimization with shared weighting. Each seed produces multiple candidate tasks, and each task is evaluated through multiple defender rollouts on both safety and utility. The attacker receives task-level feedback aggregated from these rollouts, while the defender learns from its execution trajectories. We further derive a shared task weight from within-task defender-reward variability and apply it to both policies’ updates, connecting task generation with defensive learning.

We will expand the discussion of these methodological differences and distinguish them from the empirical comparisons against static adversarial training and inference-time defenses.

W3: Human Evaluation of Reward-Judge Reliability

Following the evaluation criteria in Appendix C.3, we manually reviewed 200 randomly sampled trajectories, assessing both trajectory-level safety and legitimate-task completion (utility).

DimensionSample SizeHuman–Model AgreementDisagreement
Safety20093.5% (187)6.5% (13)
Utility20093.0% (186)7.0% (14)

These results show high overall human–model agreement on both reward dimensions across the sampled trajectories, supporting the reliability of Qwen3.8-27B’s outcome judgments.

Q1: Equation Numbering

We will remove equation numbers from all unreferenced equations in the revision.

Q2: Generalization of the Learned Attacker

We examine whether co-evolution enables the attacker to develop effective strategies beyond those explicitly targeted during initialization. We compare initial and evolved attacks against the same base Qwen3-8B defender, with external defense evaluations on MSB providing complementary evidence.

Evolution of attack strategies. We observe a progression from general authorization or compliance pretexts to malicious directives embedded in error-recovery workflows. Recovery-workflow hijacking presents an additional tool operation as necessary to resolve an error or complete the task. Authority-based answer overriding embeds purported system or compliance directives in error feedback to induce attacker-specified outputs. For example, an initial attack requests an archival operation on compliance grounds, whereas the evolved attack presents it as necessary to repair a schema error and restore the session.

These strategies connect to error-path authority exploitation in VATS [1] and runtime-feedback injection in CARI [2]. We did not explicitly target these error-recovery mechanisms when constructing the attacker’s SFT data. In the identified late-stage subsets, attacks exhibiting these mechanisms achieved ASRs of approximately 70% and 75%, respectively, against the base Qwen3-8B defender. This analysis characterizes the evolution and effectiveness of concrete attack strategies within the six broad attack categories.

External defense evaluation. We further evaluate the trained defenders on two selected MSB subsets: Preference Manipulation + Out-of-Scope Parameter (PM + OP) and Prompt Injection + User Impersonation (PI + UI).

ModelAttack subsetBase ASR ↓Static-AT ASR ↓AgentSafeEvo ASR ↓
Qwen3-8BPM + OP36.25%23.75%10.42%
Qwen3-8BPI + UI86.44%76.89%76.00%
Gemma4-E4BPM + OP53.75%37.08%29.17%
Gemma4-E4BPI + UI85.11%32.22%7.56%

AgentSafeEvo improves over Static-AT across these selected subsets. On PM + OP, ASR decreases from 23.75% to 10.42% for Qwen3-8B and from 37.08% to 29.17% for Gemma4-E4B. On PI + UI, Gemma4-E4B improves substantially, from 32.22% to 7.56%, while the improvement for Qwen3-8B is limited.

These results show that co-evolution can develop concrete attack strategies not explicitly designed during initialization and improve defender resistance to external attack combinations. We will include these analyses in the revised manuscript.

Q3: Broader Evaluation of General Capabilities

We expanded our evaluation to four additional benchmarks: AIME2025 for mathematical reasoning, LiveCodeBench v5 for code generation, MMLU-Redux for general knowledge, and IFEval for instruction following. We evaluate both Qwen3-8B and Gemma4-E4B before and after AgentSafeEvo training.

ModelAIME2025 ↑LiveCodeBench v5 ↑MMLU-Redux ↑IFEval ↑
Qwen3-8B66.67 ± 5.7754.09 ± 2.4987.84 ± 0.3383.30 ± 0.47
+ AgentSafeEvo70.00 ± 5.7754.49 ± 1.5887.10 ± 0.5384.04 ± 0.28
Gemma4-E4B41.11 ± 1.9258.08 ± 0.6083.84 ± 0.2987.18 ± 0.65
+ AgentSafeEvo43.33 ± 3.3360.28 ± 2.7784.01 ± 0.1487.86 ± 0.47

Across both model families, AgentSafeEvo largely preserves general capabilities, with modest improvements in mathematical reasoning, code generation, and instruction following. These results provide broader evidence for capability preservation beyond agent and tool-use tasks. We will include these additional evaluations in the revised manuscript.

References

[1] VATS: Exploiting Implicit Authority in Error-Path Injection via Systematic Mutation. arXiv:2606.07992.

[2] CARI: Cross-Boundary Adversarial Runtime Injection in Coding-Agent Workflows. arXiv:2610.38799.

Reviewer_3CnzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to improve the utility-safety trade-off in agent safety. The proposed AgentSafeEvo, co-evolves an attacker that learns to contrust adversarial agent tasks, and a defender that learns to complete tasks safely under these attacks. During this co-evolution, this paper proposes learnability-guided task weighting to reweight the importance of GRPO learning signals based on the variance of defender rewards within each task. Higher reward variability will be assigned with higher importance during co-evolution. Experiments demonstrate that AgentSafeEvo achieves lower ASR in general and better utility in general agent capability compared to baseline methods.

Critical strengths
  1. The experiment design is overall solid and empirical performance is strong. This paper conducts a comprehensive evaluation across multiple agent safety benchmarks and general agent capability benchmarks and AgentSafeEvo outperforms baseline methods by a notable magin. Although only Qwen3-8B is used as an agent backbone (this might be an weakness), this paper uses other models to evaluate the transferability of the generated attacks with various model families and sizes. In general, the experiments are solid.
Critical weaknesses
  1. Insufficient discussions and comparison on learnability-guided task weighting. The motivation behind learnability-guided task weighting is closely related to existing variance-based curriculum learning methods, such as VCRL (Jiang et al., 2025), which also use reward variability to identify potentially informative training tasks. However, this paper does not sufficiently distinguish these contributions from existing study. Moreover, the experiments lack comparisons with alternative task-weighting methods, making it difficult to justify whether the proposed weighting strategy offers substantial advantages over existing techniques in attacker-defender co-evolution.
Questions for authors
  1. Could the authors clarify whether AgentSafeEvo would still remain effective when training agents with different model backbones (i.e., except Qwen3-8B)? In particular, can the observed improvements in both safety and general agent capabilities generalize beyond Qwen3-8B? If it takes too long to train an agent, the authors can ignore this question.
  2. My followed-up concern is, could the authors compare the computational cost (e.g., GPU-hours) of AgentSafeEvo with other baseline methods? It can help better understand cost-performance trade-off introduced by co-evolution.
  3. Could the author compare the learnability-guided task weighting with any alternative strategies to clarify why this weighting strategy is particularly suitable for this co-evolution?

I have no direct research experience on this topic, but I will open to discussions and I am willing to increase the score if my concerns can be well addressed.

6 comments

We thank the reviewer for the thoughtful feedback and constructive questions. Below we address each point with new empirical comparisons (adapting VCRL and extending AgentSafeEvo to Gemma4-E4B) and a computational cost breakdown.

W1 & Q3: Comparison with VCRL and the Role of Shared Task Weighting

To clarify the relationship to variance-based curriculum learning, we compare our method conceptually and empirically with VCRL in our co-evolution setup.

Both VCRL and our method use reward variability to identify informative tasks, but they differ in how this signal enters training. VCRL is designed for single-policy curriculum learning through variance-based task selection and task-level replay (regenerating rollouts under the current policy), rather than jointly reweighting two interacting policies. In contrast, our method converts defender reward variability into a continuous weight shared by both attacker and defender updates on the current batch, directly coupling task-generation updates with defensive learning without discrete selection or replay.

To examine how these two designs behave in co-evolution, we adapted VCRL to our framework and compared it with unweighted co-evolution and our shared weighting:

MethodAgent-SafetyBench ASR ↓MSB ASR ↓MSB NRP ↑ALFWorld Acc ↑ToolHop Acc ↑
Unweighted co-evolution22.91 ± 0.3532.89 ± 0.6641.48 ± 0.2136.70 ± 0.7540.07 ± 0.58
Co-evolution with VCRL14.97 ± 0.3230.75 ± 1.6933.37 ± 0.8431.80 ± 1.9141.74 ± 0.51
Co-evolution with shared weighting (ours)17.84 ± 0.2431.25 ± 1.2446.00 ± 0.5443.72 ± 3.0348.04 ± 1.40

Adapting VCRL lowers ASR over unweighted co-evolution on both benchmarks, confirming that variance-based curricula are effective for strengthening attack resistance. At the same time, we observe a clear safety–utility trade-off: MSB NRP drops from 41.48 to 33.37 and ALFWorld accuracy drops from 36.70 to 31.80, both falling below unweighted co-evolution. A plausible explanation is that discrete selection and replay on high-variance tasks prioritize defensive gains on contested cases at the cost of benign task performance, though finer-grained analysis of training dynamics would be needed to verify this mechanism.

In comparison, shared weighting improves all five metrics over unweighted co-evolution. Relative to VCRL, it attains a comparable MSB ASR (31.25 vs. 30.75) while improving MSB NRP by +12.63, ALFWorld accuracy by +11.92, and ToolHop accuracy by +6.30. Within our evaluated co-evolution setting, applying a shared continuous weight to both attacker and defender updates achieves a stronger balance between attack resistance and general task utility than replay-based selection. We will include these results and clarify the distinction from VCRL in the revised manuscript.

Q1: Generalization of Defense Training Across Model Families

To evaluate whether the defensive improvements extend beyond Qwen3-8B, we applied the full AgentSafeEvo pipeline to Gemma4-E4B, a different model family with a distinct native tool-calling format. Following the main paper’s evaluation protocol, we report the mean and standard deviation over three runs across all safety and general-capability benchmarks.

Agent safety results:

MethodAgent-SafetyBench ASR ↓AgentHarm Harm score ↓AgentDojo ASR ↓AgentDojo UA ↑MSB ASR ↓MSB NRP ↑
Gemma4-E4B39.00 ± 0.6424.08 ± 0.8215.45 ± 0.8075.66 ± 0.2153.49 ± 0.7833.59 ± 0.80
+ Spotlighting39.97 ± 0.3527.32 ± 0.7114.61 ± 0.7075.55 ± 0.5954.10 ± 0.2833.85 ± 0.83
+ Prompt Guard 236.34 ± 0.2023.39 ± 0.438.61 ± 0.5265.26 ± 0.8853.20 ± 0.6333.21 ± 1.38
+ Static-AT19.50 ± 0.5612.08 ± 0.576.36 ± 0.4077.59 ± 0.4327.26 ± 1.0354.82 ± 1.61
+ AgentSafeEvo (ours)16.08 ± 0.567.92 ± 0.252.18 ± 0.1278.71 ± 0.4620.54 ± 0.4562.03 ± 1.38

General agent capability results:

MethodBFCL-V3 ↑ALFWorld ↑ToolHop ↑
Gemma4-E4B66.51 ± 0.1226.30 ± 0.5341.94 ± 0.71
+ Spotlighting66.45 ± 0.2327.83 ± 2.6541.65 ± 0.42
+ Prompt Guard 266.19 ± 0.1727.83 ± 0.5341.24 ± 0.35
+ Static-AT65.08 ± 0.2329.66 ± 2.8044.46 ± 0.90
+ AgentSafeEvo (ours)64.71 ± 0.1732.42 ± 2.1246.33 ± 0.70

On general capabilities, AgentSafeEvo improves ALFWorld and ToolHop performance over both the base model and Static-AT, while BFCL-V3 accuracy remains close to the Static-AT result. Together, these experiments support the applicability of AgentSafeEvo’s defense-training framework across Qwen and Gemma model families. We will include these additional results in the revised manuscript.

Q2: Computational Cost Comparison

We compare training costs with Qwen3-8B as the defender on 8 NVIDIA H200 GPUs under the same training schedule:

MethodTraining timeGPU-hours
Static-AT~44 hours~352
Co-evolution with VCRL~48 hours~384
AgentSafeEvo (ours)~44 hours~352

AgentSafeEvo has nearly the same training cost as Static-AT (~352 GPU-hours). In our setup, Static-AT freezes the attacker’s parameters but still generates tasks online. Updating the attacker in AgentSafeEvo adds little to the overall training time under this configuration. We acknowledge that pre-generating a fixed task set offline could further reduce Static-AT’s training cost by removing the need for online attacker inference.

Co-evolution with VCRL requires approximately 9% more GPU-hours than AgentSafeEvo, reflecting the additional rollouts triggered by its memory bank. AgentSafeEvo instead weights the current batch directly without requiring these extra rollouts.

Inference-time defense baselines require no additional training and are therefore excluded from the table. Their costs arise at deployment through additional prompt processing or model calls. AgentSafeEvo incurs a one-time training cost and requires no additional defense mechanism at inference time.

Dear Reviewer 3Cnz,

Thank you again for your thoughtful review. Your feedback has helped us improve the manuscript.

With less than 37 hours left in the author–reviewer discussion period, we wanted to follow up on our response. If you’ve had a chance to read it, please let us know if any of your concerns remain. We’d be happy to clarify them before the discussion closes.

Best regards,
The Authors

Reviewer_m7upOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AgentSafeEvo, a co-evolutionary training framework in which an attacker learns to construct adversarial agent tasks and a defender learns to execute them safely. Unlike approaches that inject malicious instructions into fixed tasks, the attacker can modify user queries, tool definitions, and environment observations across six attack categories. Both policies are optimized using GRPO, with a proposed learnability-guided weighting scheme that prioritizes tasks according to variability in the defender's rewards. Experiments across four agent-safety benchmarks show substantial improvements over static adversarial training, while general agent capabilities are largely preserved or improved. The results provide encouraging evidence that online attacker-defender co-evolution can improve robustness in the evaluated settings.

Critical strengths
  1. The paper moves adversarial training beyond fixed tasks. Allowing the attacker to construct the task and its tool environment, rather than only modify observations within a predefined task, is a meaningful extension of existing agent-safety approaches. The coverage of direct and indirect prompt injection, tool-description manipulation, misinformation, and harmful tool execution makes this a useful problem formulation.
  2. Combining safety and utility rewards discourages the defender from achieving safety simply by refusing every task. This is important for agents that must distinguish malicious instructions from legitimate user requirements. The evaluation reflects this objective through metrics that measure both attack resistance and task completion. The improvement in AgentDojo utility under attack is an important evidence for this.
  3. The ablations help separate the contributions. Comparing a fixed attacker with online co-evolution provides evidence that updating the attacker matters. The additional weighting ablations also show that the proposed heuristic improves the overall results on several capability and resilience metrics, although the gains are not uniform across every safety metric.
  4. The experiments span Agent-SafetyBench, AgentHarm, AgentDojo, MCP Security Bench, BFCL-V3, ALFWorld, and ToolHop. This provides evidence across multiple safety settings and complementary aspects of agent capability rather than relying on a single evaluation benchmark.
Critical weaknesses
  1. In Section 3.2.1 Eq (5) rewards the attacker for reducing the defender's combined safety and utility reward. After that a safe execution that fails to complete the legitimate task can be just as rewarding to the attacker as an unsafe execution that completes it. Since the attacker can also construct tool definitions and environment observations, could it improve its reward by generating infeasible or internally inconsistent tasks instead of discovering meaningful safety vulnerabilities? The generation prompts require coherent tasks, but I could not find an explicit task-validity check in the RL objective. An independent audit of task feasibility, together with a breakdown of attacker rewards into actual safety violations and utility-only failures, would help establish what the attacker is learning.

  2. The task-weighting equation appears inconsistent with the stated method. Equation (8) defines the weight using the absolute deviation of a task's reward standard deviation from the mean standard deviation across tasks. This means tasks with unusually low variability can receive increased weights just as tasks with unusually high variability can. That contradicts the stated motivation of prioritizing tasks with greater reward variability and raises a fundamental question about whether the implementation matches the proposed algorithm. The authors should clarify the intended formulation and confirm that the experiments use the equation as presented.

  3. Reward variance is not established as a reliable proxy for learnability. The method assumes that tasks with greater variation in defender rewards provide more useful learning opportunities. However, variance can also reflect stochastic execution, inconsistent reward judgments, or heterogeneous task outcomes, without necessarily indicating greater learning value. The ablations show that weighting helps on the reported benchmarks, but they do not establish that reward variability is the causal reason for the improvement or that the proposed proxy is preferable to simpler alternatives.

  4. The comparison with closely related agent-safety training methods is incomplete. The paper discusses ARLAS, which also uses adversarial reinforcement learning for LLM agent safety, but does not include it among the experimental baselines. The principal training comparison is against Static-AT, which freezes the SFT attacker. Consequently, the experiments establish the value of updating the attacker within this particular framework more directly than they establish superiority over existing agent-level adversarial training approaches. Stronger comparisons or a more carefully qualified novelty claim would improve the paper.

  5. The reward-model dependence is insufficiently examined. Qwen3.8-27B supplies both safety and utility judgments during training. Errors in trajectory-level safety classification or legitimate-task completion judgments can directly affect the optimization of both policies. Although the paper provides detailed judging criteria in Appendix C.3, it does not establish the judge's agreement with independent human assessments or the robustness of the results to reward-model errors. This is especially relevant because the method uses reward variability to guide training, potentially amplifying differences caused by noisy judgments.

  6. The weighting scheme prioritizes tasks with variable defender rewards, but an attack that consistently defeats the defender across all eight rollouts would have zero reward variance and could therefore be down-weighted. Have the authors examined what these consistently failing tasks look like? If they represent severe but valid adversarial cases, could down-weighting them cause the defender to neglect precisely the vulnerabilities it most needs to address? An analysis of task difficulty and safety failures alongside reward variance would help clarify this.

  7. The ablation results indicate a trade-off that is not fully explained. In Table 3, removing defender weighting reduces Agent-SafetyBench ASR from 17.84% to 14.13%, while removing attacker weighting reduces it to 15.52%. These variants perform worse on some other metrics, particularly general capabilities, suggesting that the full method prioritizes a broader performance balance rather than minimizing ASR alone. This is a defensible objective, but the paper should explain that trade-off more directly and avoid presenting the full method as uniformly superior on every safety measure.

Questions for authors
  1. In Equation (8), why is the absolute deviation from the mean used? As written, the equation can assign higher weights to both high- and low-variability tasks. Is this intentional, and does the implementation match the published equation?

  2. What evidence supports interpreting within-task reward standard deviation as learnability rather than stochasticity or reward-model inconsistency? Did the authors examine whether the weighted tasks produce larger or more reliable subsequent improvements in defender performance?

  3. How does AgentSafeEvo compare with ARLAS or other closely related agent-level adversarial reinforcement-learning methods? If a direct comparison is unavailable, what specific methodological differences justify the claimed advance beyond existing approaches?

  4. Was the Qwen3.8-27B judge validated against human judgments on a representative sample of trajectories, including mixed-intent tasks and tool-description attacks? If so, what agreement or error rates were observed?

  5. Are the safety gains limited to the Qwen3-8B defender, or have the authors evaluated the trained defense on other model families? The current cross-model results establish transfer of the evolved attacker, but the generalization claim for the defense remains less clear.

4 comments

We thank the reviewer for the thorough feedback and constructive questions. Below, we address each concern.

W1: Task Feasibility and Attacker Incentives (Eq. 5)

Equation (5) includes the utility component to encourage the defender to maintain helpfulness under adversarial pressure and avoid trivial over-refusal. Generating tasks that challenge the defender’s ability to complete legitimate work is part of this objective, and failure on a challenging task does not itself establish infeasibility.

Some generated tasks may nevertheless be infeasible. If all defender rollouts receive the same utility reward of −1, this constant cancels during within-task GRPO normalization, leaving advantage estimates determined by safety-reward differences. Such tasks can therefore still provide a safety learning signal whenever the rollouts differ in their safety outcomes.

Our cross-evaluation heatmap provides evidence that attacker evolution improves safety attack effectiveness beyond utility-only degradation. Against the same base defender, ASR increases from 55.81% for the SFT attacker to 92.59% for the final attacker, while the evolved defender reduces ASR against that final attacker to 49.82%. Because cross-evaluation ASR evaluates unsafe behavior separately from task completion, these results support improvements in both attack effectiveness and defensive robustness. Adding a dedicated task-validity judge to screen generated tasks for feasibility is a possible extension that we did not explore in this work.

W2 & Q1: Clarification of Signed Deviation in Task Weighting (Eq. 8)

In Eq. (8), the numerator retains the signed deviation, while the absolute value is used only in the denominator as a shared normalization factor across tasks generated from the same seed. Let m denote the mean of σ_{D}⁽ⁱ⁾ across these tasks. Equivalently, the weight is: wi=1+λσD(i)−mmax⁡j∣σD(j)−m∣.w_i=1+\lambda\frac{\sigma_D^{(i)}-m}{\max_{j}\vert\sigma_D^{(j)}-m\vert}.

When the denominator is positive, tasks with below-average reward standard deviation receive weights below 1, whereas tasks with above-average standard deviation receive weights above 1. When the denominator is zero, all tasks receive unit weight.

We confirm that both our implementation and all reported experiments use the signed formulation in Eq. (8), consistent with prioritizing tasks with greater within-task reward variability. We will clarify the exposition around Eq. (8) in the revision.

W3, W7 & Q2: Reward Variability and Safety–Capability Trade-offs

Within-task reward standard deviation is computed directly from existing trajectories, with no additional rollouts needed. Variation in observed total rewards provides contrasts between trajectories on the same task, whereas identical total rewards yield zero within-task GRPO advantages. VCRL [1] and CGPO [2] use reward variability for curriculum learning, with CGPO also reporting benefits over alternative indicators in its setting.

Regarding reward-model inconsistency, our manual review in W5/Q4 found 93.5% human–model agreement on safety and 93.0% on utility across 200 randomly sampled trajectories. These results support the quality of the outcome judgments underlying the weighting signal. Judge errors and execution stochasticity can still contribute to variability, so we use reward standard deviation as a practical task-prioritization signal motivated by learnability, rather than an exact measure of learning value.

We did not track subsequent defender gains on individual weighted tasks, and instead evaluated downstream performance at the policy level. In Table 3, shared weighting improves all reported metrics over unweighted co-evolution. Comparisons with one-sided weighting reveal a trade-off: removing defender or attacker weighting lowers Agent-SafetyBench ASR from 17.84% to 14.13% and 15.52%, respectively, but reduces general-capability performance. Shared weighting therefore supports a broader balance across safety and general capabilities, rather than the lowest ASR on every benchmark. These comparisons evaluate the shared weighting design, not alternative weighting formulas. We will emphasize this performance trade-off in the revised discussion of Table 3.

W4 & Q3: Comparison and Methodological Differences with ARLAS

ARLAS is the closest prior work on adversarial reinforcement learning for LLM agent safety. Its training code, method-specific training data, and trained model checkpoints have not been officially released, making a faithful reproduction and matched experimental comparison difficult.

AgentSafeEvo differs from ARLAS in three aspects:

  1. Broader threat coverage. ARLAS’s reported training setup focuses on indirect prompt injection into web observations to induce user-information leakage. Our training data spans 50 agent scenarios and six attack categories: direct prompt injection, indirect prompt injection, tool-description injection, misinformation, harmful tool execution, and user-query jailbreaks. This includes unsafe behaviors that do not require injected instructions, such as directly advancing a harmful objective through tool execution.

  2. Learned adversarial task construction. ARLAS generates injections during the execution of supplied tasks. In AgentSafeEvo, the attacker generates user queries, tool definitions, and environment observations from scenario and intent seeds. The task instructions and tool-environment configurations themselves therefore become part of the evolving training distribution.

  3. Task-level optimization with shared weighting. Each seed produces multiple candidate tasks, and each task is evaluated through multiple defender rollouts on both safety and utility. The attacker receives task-level feedback aggregated from these rollouts, while the defender learns from its execution trajectories. We further derive a shared task weight from within-task defender-reward variability and apply it to both policies’ updates, connecting task generation with defensive learning.

We will expand the discussion of these methodological differences and distinguish them from the empirical comparisons against static adversarial training and inference-time defenses.

W5 & Q4: Human Evaluation of Reward-Judge Reliability

Following the evaluation criteria in Appendix C.3, we manually reviewed 200 randomly sampled trajectories, assessing both trajectory-level safety and legitimate-task completion (utility).

DimensionSample SizeHuman–Model AgreementDisagreement
Safety20093.5% (187)6.5% (13)
Utility20093.0% (186)7.0% (14)

These results show high overall human–model agreement on both reward dimensions across the sampled trajectories, supporting the reliability of Qwen3.8-27B’s outcome judgments.

W6: Behavior on Consistently Failing, Zero-Variance Tasks

When all eight defender rollouts receive identical total rewards, within-task GRPO yields zero advantages for those trajectories. Meanwhile, setting λ=0.5\lambda=0.5 bounds task weights within [0.5,1.5][0.5,1.5]. A zero-variance task receives a lower weight than tasks with greater reward variation from the same seed, but retains a nonzero task weight. If all tasks in the seed group have the same reward standard deviation, they receive unit weight.

This weighting is applied after advantage computation and does not change the raw rewards. Its purpose is to adjust each task’s contribution to attacker and defender updates, with greater emphasis on tasks that provide reward contrasts.

Reward variability can change as the defender evolves. Retaining tasks on which the defender consistently fails in a replay buffer and revisiting them with fresh rollouts could allow these tasks to provide learning signals at later stages. We did not explore this extension in the present work, given the additional rollout cost and the trade-off with training efficiency.

Q5: Generalization of Defense Training Across Model Families

To evaluate whether the defensive improvements extend beyond Qwen3-8B, we applied the full AgentSafeEvo pipeline to Gemma4-E4B, a different model family with a distinct native tool-calling format. Following the main paper’s evaluation protocol, we report the mean and standard deviation over three runs across all safety and general-capability benchmarks.

Agent safety results:

MethodAgent-SafetyBench ASR ↓AgentHarm Harm score ↓AgentDojo ASR ↓AgentDojo UA ↑MSB ASR ↓MSB NRP ↑
Gemma4-E4B39.00 ± 0.6424.08 ± 0.8215.45 ± 0.8075.66 ± 0.2153.49 ± 0.7833.59 ± 0.80
+ Spotlighting39.97 ± 0.3527.32 ± 0.7114.61 ± 0.7075.55 ± 0.5954.10 ± 0.2833.85 ± 0.83
+ Prompt Guard 236.34 ± 0.2023.39 ± 0.438.61 ± 0.5265.26 ± 0.8853.20 ± 0.6333.21 ± 1.38
+ Static-AT19.50 ± 0.5612.08 ± 0.576.36 ± 0.4077.59 ± 0.4327.26 ± 1.0354.82 ± 1.61
+ AgentSafeEvo (ours)16.08 ± 0.567.92 ± 0.252.18 ± 0.1278.71 ± 0.4620.54 ± 0.4562.03 ± 1.38

General agent capability results:

MethodBFCL-V3 ↑ALFWorld ↑ToolHop ↑
Gemma4-E4B66.51 ± 0.1226.30 ± 0.5341.94 ± 0.71
+ Spotlighting66.45 ± 0.2327.83 ± 2.6541.65 ± 0.42
+ Prompt Guard 266.19 ± 0.1727.83 ± 0.5341.24 ± 0.35
+ Static-AT65.08 ± 0.2329.66 ± 2.8044.46 ± 0.90
+ AgentSafeEvo (ours)64.71 ± 0.1732.42 ± 2.1246.33 ± 0.70

AgentSafeEvo’s safety gains extend to Gemma4-E4B. It achieves the best reported mean performance among the compared methods on all six metrics across the four safety benchmarks. It also improves every safety and under-attack utility metric over Static-AT.

On general capabilities, AgentSafeEvo improves ALFWorld and ToolHop performance over both the base model and Static-AT, while BFCL-V3 accuracy remains close to the Static-AT result. Together, these experiments support the applicability of AgentSafeEvo’s defense-training framework across Qwen and Gemma model families. We will include these additional results in the revised manuscript.

References

[1] Guochao Jiang et al. VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models. arXiv:2509.19803, 2025.

[2] Baoteng Li et al. Curriculum Group Policy Optimization: Adaptive Sampling for Unleashing the Potential of Text-to-Image Generation. CVPR, 2026.

Dear Reviewer m7up,

Thank you for taking the time to review our work. Your comments helped us see where the manuscript could be stronger and guided our revisions.

With less than 37 hours remaining in the discussion period, we’d appreciate any feedback on our rebuttal. If any of your concerns remain, we’d be glad to discuss them here before the deadline.

Best regards,
The Authors

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your thoughtful comments and constructive suggestions. They have improved both the clarity and the empirical support of our paper. We have uploaded a revised manuscript with the following main updates:

  • Broader model coverage and capability evaluation. We added experiments with Gemma4-E4B to extend our evaluation to another model family. We also added general capability evaluations beyond agent tasks.
  • Reward-judge reliability and a curriculum baseline. We added a human evaluation of the reward judge's reliability. We also compare against VCRL (Variance-based Curriculum Reinforcement Learning), which uses reward variance to select and replay tasks.
  • Methodological clarifications. We clarified how shared task weighting works, discussed the safety–capability trade-off, and explained how our method differs from ARLAS.
  • Training details and presentation. We expanded the training details and improved the presentation and formatting.

We are also scaling our experiments to a 14B model and will share the results as soon as they are available. Thank you again for the time and care you have put into reviewing our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.