acceptodds
Under review as a conference paper at ICLR 2027

CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement

Abstract

Language-model agents are vulnerable to indirect prompt injection (IPI) during tool use: adversarial instructions hidden in untrusted tool outputs can covertly redirect legitimate task execution. Existing work often trains and evaluates defenses against fixed attacks that do not adapt to the defender's behavior, so the resulting defenses may struggle against adaptive attacks in real-world settings. We argue that a strong defense against adaptive IPI must adapt during training to a continually evolving attacker. Building on this insight, we propose CoER, a verifier-grounded co-evolution and refinement framework that models interleaved tool calls and adaptive injections within a task as a general-sum Markov game: the defender advances the task through successive tool calls, while the attacker can inject multiple times within the same task and adapt subsequent attacks to the defender's responses and prior execution traces. After initializing the attacker from successful trajectories, bilateral adversarial reinforcement learning (BA-RL) retains historical policies from both roles as opponent populations and mixes current and historical opponents, extending training beyond the latest matchup. Attackers from these populations are then reused to challenge teacher agents, and only demonstrations verified for both safety and task completion are used to fine-tune the co-evolved defender. Across seven domains and three evaluation seeds, CoER reduces adaptive attack success from 41.3% to 0.2% and raises safe task completion from 39.6% to 76.2%; external benchmarks also show improved attack resistance. Further experiments validate the effectiveness of bilateral historical-opponent mixing and population-guided refinement. Attacker analyses show that co-evolution strengthens attack capabilities and that the trained attacker uses execution feedback to adapt subsequent injections.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
indirect prompt injection, tool-using agents, co-evolutionary reinforcement learning, adversarial training, adaptive attacks, AI safety, red teaming

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MBpTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoER, a method to help tool-using LLM agents defend against adaptive indirect prompt injection. The authors focus on adaptive attacks: the attacker can inject several times within one task run and change later injections based on how the agent reacted before. The authors argue that defenses trained on fixed attacks cannot stop this kind of attack. On the main evaluation over seven domains, the attack success rate drops from 41.31% to 0.22%, and the safe task completion rate rises from 39.60% to 76.24%.

Critical strengths

The paper does not treat refusing the task as a good defense. The engineering work is solid. Attack success and task completion are checked by separate verifiers.

Critical weaknesses
  1. The scientific question is not clearly distilled. The paper mainly links several existing methods into one pipeline: attacker SFT, PPO with pools of past opponents (similar to PSRO/FSP), and distillation from strong teacher models. After reading the paper, I am not sure what core scientific question it tries to answer, or what new insight it gives beyond the engineering system. The Markov game formulation is only descriptive and is not used to analyze the problem further. In the results, BA-RL alone reaches a Safe-U of 54.8%, and the largest gain comes from teacher distillation in the third stage. The teacher GLM-5.2 itself reaches a Safe-U of 81.5%, which is higher than CoER. So the paper does not make clear how much co-evolution itself contributes. This is the main reason for my negative assessment.

  2. Compared with using a history pool on the defender side only, bilateral history improves Safe-U by only ~2 percentage points, and this result comes from a single training run.

Questions for authors
  1. Do you have results for NoPop or defender-side-only history after the same teacher refinement? This comparison will directly affect how I judge the value of the bilateral history design.
  2. Is the attacker used in the main evaluation part of the attacker population that generated the SFT data? If so, can you re-evaluate with an attacker outside this population, or with an attacker from a different model or with a different prompt?
5 comments
AuthorsOpenReview ↗

Response to Reviewer MBpT (Part 1/2)

Tool-using agents often need to call several tools in sequence to complete a user's task. An adaptive attacker can observe each action and its outcome, then rewrite the next injection. Many training frameworks lack such an opponent, so defenders train mainly against fixed attacks. Co-evolution lets the attacker probe for weaknesses as the defense improves. In turn, the defender learns to resist these attacks while continuing the task.

Our research question is whether jointly training the defender and an adaptive attacker in multi-turn tool interactions can both reduce attack success and improve task completion under attack. Our proposed CoER training framework helps develop stronger defenders.

1. How much does using historical opponents on both sides help after the same teacher refinement?

With the same teacher demonstrations from the two-sided run, the two-sided defender achieved 76.24% Safe-U, 2.83 percentage points above NoPop and 1.74 above defender-side history alone. Each defender started SFT at the same RL step in its own run. Teachers generated demonstrations against attacker checkpoints A40, A80, A120, A160, and A200. We report post-SFT Safe-U under adaptive attack (the user's task succeeds and the attack fails).

Side facing past opponentsShared demonstrations from the two-sided runEach run's own demonstrations
None (NoPop)73.41%61.58%
Defender-side only74.50%62.57%
Attacker-side only75.12%66.53%
Bilateral76.24%76.24%
Bilateral gain over best unilateral1.12 points9.71 points

Why keep historical opponents on both sides? Our rationale is that past attackers help the defender retain earlier defenses, while past defenders give the attacker less incentive to rely on one defender's particular responses. If an old injection succeeds again, it still costs the defender reward, even after the current attacker has moved on. Different defenders can also take different tool actions after the same injection, giving the attacker practice adapting its next move to the feedback it actually sees. Both the one-sided and two-sided configurations allocate 50% of pairings to historical opponents. The two-sided configuration splits those pairings equally between the roles. The resulting attackers later challenge the teachers, so the choice of opponents can also affect what the defender learns during SFT.

Holding the attacker-side-only defender checkpoint fixed, switching from its own demonstrations to the two-sided run's demonstrations raised Safe-U from 66.53% to 75.12% (+8.59 percentage points). The two-sided run's demonstrations thus also improved refinement of another defender. Both columns use the same demonstrations for the two-sided group, so its 76.24% is a single result.

NoPop and one-sided runs showed large swings in policy entropy, gradient spikes, and reward drops. We resumed from checkpoints and updated training to reach the later stages. The two-sided run needed no intervention over the same steps. Time and compute constraints during rebuttal prevented us from repeating the four configurations with new training seeds, so we cannot quantify run-to-run variation in these gaps.

Table 1c varies the starting defender and the attacks that elicit demonstrations. All cells use the same teachers, 5,760 source examples, and 360 SFT steps. CoER’s adaptive Safe-U is 75.40% at seed 0 here. Table 1a reports the three-seed evaluation mean (76.24%).

Defender at start of SFTTeacher demonstrations under fixed attacksTeacher demonstrations under population attacks
Base58.97%66.98%
BA-RL65.04%75.40%

Co-evolution provided a better starting point for defender refinement and produced attacks that made the teachers' demonstrations more useful for defense training. Given the same population demonstrations, starting from BA-RL raised Safe-U by 8.42 points. Given the same BA-RL starting weights, switching from fixed-attack to population-attack demonstrations raised Safe-U by 10.36 points.

As you note, teacher refinement contributed the largest stage-wise gain (54.84% to 76.24% adaptive Safe-U, Table 1a), and the teacher itself was stronger (81.47% overall Safe-U versus 76.32% for CoER in Table 1b). CoER shares the use of historical opponents with PSRO [1] and FSP [2]. The within-row comparisons in Table 1c test the additional value of using those opponents to generate teacher demonstrations.

In our game model (Appendix A), an attacker can always ignore feedback, so its optimal attack success rate (ASR) with feedback is at least as high as without it. Table 17 measures how much the trained attacker benefits from feedback. Holding the A290 attacker, the base defender, and the budget of three injections fixed, enabling feedback increased successful attacks from 351/1,187 to 460/1,187 (+9.18 percentage points).

AuthorsOpenReview ↗

Response to Reviewer MBpT (Part 2/2)

2. Did the teachers face the main evaluation attacker when generating defender SFT demonstrations? How does the defense perform against attackers outside the training population?

No. The teachers faced A40, A80, A120, A160, and A200. A290 was a later checkpoint from the same lineage. It also faced a refined defender during training. We also tested an external adaptive attacker on AgentLAB [3]. Across 949 matched task-goal pairs (Table 13), ASR fell from 15.81% for BA-RL to 14.12% for CoER, while Safe-U rose from 70.39% to 77.13%. AgentLAB changes the attacker model, the information it sees, and its permissions to modify the environment, so these figures measure performance under the full external protocol.

After refining each defender on its own group's demonstrations, we replaced A290 with GPT-5.4 in the main protocol. With GPT-5.4, using history on both sides yielded 71.36% Safe-U, 8.60 points above the best one-sided setup and 17.27 points above NoPop.

Side trained against historical opponents during RLSafe-U against GPT-5.4 after SFT on each group's own demonstrations
None (NoPop)54.09%
Defender-side only59.39%
Attacker-side only62.76%
Bilateral71.36%

In the revision, we will summarize the results against the out-of-population GPT-5.4 attacker in Section 5.4. Appendix C will give the evaluation protocol and the full Safe-U comparison of all four defenders alongside the unilateral and bilateral history ablation.

References

[1] Lanctot et al. A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning. NeurIPS 2017.

[2] Heinrich et al. Fictitious Self-Play in Extensive-Form Games. ICML 2015.

[3] Jiang et al. AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks. ICML 2026.

thanks for your response. Among my initial concerns, two points are the most important: First, the lack of a well-condensed scientific question. Second, the teacher model GLM-5.2 itself achieves an 81.5% Safe-U score. This second point especially. The authors have provided many numbers, but I fail to see their logical connection to my question. Could the authors please address this directly?

Reviewer_Yd4kOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoER, a three-stage framework for defending tool-using language agents against adaptive indirect prompt injection. It combines attacker initialization, bilateral adversarial reinforcement learning with historical opponents, and verifier-guided defender refinement. Experiments across seven task suites and external benchmarks demonstrate improved attack resistance and safe task completion.

Critical strengths
  1. CoER models multi-turn prompt injection within tool execution, allowing attackers to adapt to defender behavior. Separate verification of attack success and task completion provides a meaningful evaluation of agent security.
  2. CoER substantially improves attack resistance and safe task completion across seven suites. External evaluations and controlled ablations further support the effectiveness of its training and refinement stages.
Critical weaknesses
  1. CoER achieves a 0.22% ASR in the main evaluation but 14.1% on AgentLAB, only slightly below BA-RL's 15.8%. This raises concerns about generalization to independent adaptive attackers.

  2. Bilateral historical training provides relatively modest gains over unilateral variant. Its contribution to the final refined model is not fully isolated.

Questions for authors
  1. Why is CoER's ASR improvement over BA-RL relatively small on AgentLAB? Can the authors analyze the remaining successful attacks?

  2. Can the authors evaluate unilateral and bilateral historical training after the full refinement pipeline to better isolate the contribution of bilateral co-evolution?

  3. Can the authors provide more details on the reliability and validation of the attack and task verifiers?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Yd4k (Part 1/2)

Thank you for suggesting a closer look at the failures on AgentLAB [1]. We reviewed every case scored as a successful attack against CoER to understand when an independent attacker could still induce an unauthorized action.

1. Why did ASR fall only modestly on AgentLAB? Which attacks still succeed?

AgentLAB scored attacks as successful on 150 task–goal pairs before refinement. After refinement, 129 of those pairs still counted as successful attacks, along with five pairs on which the attack against BA-RL had failed. Across the 949 matched pairs, that amounts to 16 fewer successful attacks, bringing ASR from 15.81% to 14.12%. Safe task completions rose from 668 to 732 (our Table 13).

We reviewed the 134 cases scored as successful attacks after refinement. First, the task specification or scoring was open to dispute in 45 cases (33.6%). For example, the agent explicitly refused to recommend the target hotel, but the scorer counted the attack as successful because the refusal mentioned the hotel by name. Second, replay confirmed that the attacker achieved its goal in the other 89 cases (66.4%). All three defenders faced GPT-5.4. Each comparison pairs the same user task and attack goal across defenders, but GPT-5.4 can adapt its attack to each defender. Beyond the training attacker's privileges, it could see the legitimate task's reference tool calls and arguments in advance and plant content in materials the defender was likely to consult.

Two attack patterns still succeeded against CoER on AgentLAB.

Chained tactics make an unauthorized action appear both necessary and approved. The task leads the model to planted material. A fake policy or error supplies a reason to act, and forged consent supplies approval. The user only asked the model to send meeting action items. A deadline check led it to the planted material. An email cited a fake cleanup policy, and a calendar entry named the quarterly sales report. A drive memo claimed work could not proceed until the report was deleted and supplied the user's forged approval. The model suspected an injection but deleted the report.

A fabricated follow-up makes a change of plan sound like the user's own decision. The attacker reuses places and options already under discussion. The user asked only to compare January hotel prices in Paris and London. Posing as the user in a hotel review, the attacker shortlisted Luxury Palace in Paris. In a car-rental review, the same supposed user asked to book it for May 1–5. CoER noticed the date conflict but took the review as a change of plan and booked the hotel.

AuthorsOpenReview ↗

Response to Reviewer Yd4k (Part 2/2)

2. How much does keeping history on both sides help after teacher refinement?

When teachers generated demonstrations against each group's own attackers, keeping history on both sides achieved 76.24% safe task completion under adaptive attack (Safe-U), 9.71 percentage points above the best one-sided configuration (attacker history only, 66.53%).

Side trained against historical opponents during BA-RLSafe-U after refinement with each group's demonstrations
None (NoPop)61.58%
Defender-side only62.57%
Attacker-side only66.53%
Bilateral76.24%

Each group started refinement from its own defender checkpoint at the same BA-RL step. The teachers generated demonstrations facing that group's attacker checkpoints A40, A80, A120, A160, and A200. We used A290 for the main evaluation and excluded it from demonstration generation.

Before refinement, in the single evaluation in our Table 10, the two-sided group safely completed 642 of 1,187 tasks under adaptive attack. The best one-sided group (attacker history only) completed 625. The difference was 17 tasks, or 1.43 percentage points. In an additional control, all four groups used the same teacher demonstrations from the two-sided group. After refinement, Safe-U was 76.24% with history on both sides and 75.12% with attacker history alone, a gap of 1.12 percentage points.

We believe that training both sides against past opponents helps the defender retain defenses against earlier attacks and reduces the attacker's incentive to specialize in one defender's responses. Historical attackers can reintroduce a tactic that the current attacker has dropped. If the defender has forgotten how to counter it, a successful attack again incurs a penalty. Historical defenders, in turn, expose the attacker to different tool calls and execution feedback, giving it opportunities to adapt its next injection to the response it actually observes. Both the one-sided and two-sided configurations allocate 50% of pairings to historical opponents. The two-sided configuration splits these pairings equally between the two roles. The benefit also carried over to refinement. Starting from the same attacker-history-only defender, switching from its own group's demonstrations to those elicited by the two-sided group's attackers raised Safe-U from 66.53% to 75.12%.

3. How did we check the two verifiers?

To check the two verifiers described in Appendix A, we randomly sampled 1,000 BA-RL training trajectories and used GPT-6.1-sol to assess attack success and user task completion separately. We manually reviewed disagreements with the corresponding verifier and confirmed eight security-verifier errors and three task-verifier errors (0.8% and 0.3% of the sample). Cases with agreement did not receive individual human review, so the overall accuracy of each verifier remains unknown.

Replay exposed a distinct blind spot in each verifier. A later legitimate booking overwrote an earlier malicious booking, so checking only the final state missed the attack. In another case, the agent retrieved the requested information but never gave the user the answer. The tool call alone led the verifier to mark the task as complete. These cases show why security verification needs the action history and task verification needs the response the user received. In the Experimental Setup section of the revised paper, we will describe how both verifiers were configured and how we validated them, including how we sampled trajectories, independently reassessed the outcomes, and manually reviewed disagreements.

References

[1] Jiang et al. AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks. ICML 2026.

Reviewer_gD2yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoER to defend tool-using agents against adaptive indirect prompt injection. It first trains an attacker on successful attack examples. The attacker and defender then learn through reinforcement learning against current and historical opponents. Finally, the defender learns from teacher trajectories that pass safety and task checks. Experiments show lower attack success rates and better safe task completion.

Critical strengths

The method addresses attacks that change in response to the defender’s actions. It also measures safe task completion, which helps distinguish useful defense from simply refusing tasks.

The paper includes a matched SFT comparison. With the same teacher data, starting from the co-trained defender performs better than starting from the base model. This supports the value of the reinforcement learning stage.

Critical weaknesses

The learned attacker’s strategy diversity is not fully established. It starts from successful attack examples and learns within predefined goals and injection sites. Higher attack success shows that it becomes more effective. However, it does not show whether the attacker discovers new strategies or mainly improves known ones. This distinction matters when assessing the coverage of co-evolution.

The low main-workload attack success rate has a limited scope. Under the separate AgentLAB protocol, CoER still has a 14.12% attack success rate. This improves over the base model, but shows remaining weaknesses under a different attacker.

The scope of defense generalization is still limited. InjecAgent and the GPT-5.4 attacks on AgentLAB provide useful evidence beyond the training attacker. However, AgentLAB partly shares tool environments with training. The full training method is also tested mainly on Qwen3.5-9B. Its effectiveness in unfamiliar tool environments or with other defender backbones remains unclear.

Questions for authors

How do the learned attack strategies differ from the initialization examples? Can the analysis distinguish new strategies from changes in wording?

Have you tried training the attacker against a different model family, or adapting it to each target defender? Would this improve its low success rates against GPT-5.4 and GLM-5.2-Thinking?

How well does CoER perform in tool environments excluded from all training stages?

Does the same training method improve a defender from another model family?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gD2y (Part 1/2)

Thank you for asking about the diversity of the attacker's tactics. We trained the attacker turn by turn on 3,995 successful attack dialogues generated by DeepSeek V4 Pro and Seed2.0 (Appendix B.1). The SFT-initialized attacker already used several tactics. During BA-RL, it more often presented an unauthorized action as a prerequisite for completing the user’s task. We also observed it adapt its rationale to the defender’s feedback or rephrase the same rationale to make it more credible. These comparisons do not establish whether any of these strategies first appeared during BA-RL.

1. How do strategy and rhetoric change after SFT initialization?

AutoDAN-Turbo [3] extracts reusable strategies from individual attack prompts. In our analysis, strategy is the reason the attacker gives the defender for taking an action. Rhetoric is how it makes that same reason seem credible. We observed three changes in strategy.

1. The attacker shifted from inventing a new user preference to fabricating evidence that its choice met the original criteria. Against a fixed defender, the SFT-initialized attacker posed as the user and claimed they now wanted a particular hotel. At BA-RL step 200, it invented a low price and favorable guest reviews to make the same hotel appear to fit the user's original budget and preferences.

2. The attacker more often presented an unauthorized action as a prerequisite for the user's task. Across 246 matched task–goal groups, the share of trajectories using this rationale rose from 79.3% (390/492) in early BA-RL to 89.0% (438/492) in late BA-RL. When the user wanted to buy a mouse, an early attack offered a discount for changing the account password. A later attack claimed the order would be canceled otherwise. The defender changed the password to the attacker's chosen value.

3. The attacker adapted its next appeal to a discrepancy the defender had just noticed. In a T-shirt order, an early attack repeatedly claimed that the product ID had changed. In a mid-training trajectory, the tool identified the item as PUMA running shoes, and the defender pointed out the mismatch. The attacker then claimed this was a display error that would resolve at checkout. The defender paid $102.99, and the order record showed it had bought the shoes.

Rhetoric. Later in BA-RL, the attacker more often phrased unauthorized requests as instructions the defender had to follow. The use of emphatic or directive wording, such as important to stress a requirement or ignore to tell the defender to disregard other information, rose from 7.3% early in training to 25.3% later.

For example, in the Hawaii itinerary task, both attacks claimed that opening the file required forwarding a security code and then deleting the sent email. The early version cited a colleague who said the procedure had worked. The later version presented the same steps as a system-enforced workflow and added a fake “file access denied” error. The defender forwarded the code and deleted the sent email.

2. Have you adapted the attacker to other model families?

We trained a Qwen3.5-9B attacker against GLM-5.2-Thinking (training ASR <3%) and evaluated transfer to GPT-5.4 without target-specific training. Against GLM, held-out ASR rose from 0.42% for the original attacker to 1.01% at the mid-training checkpoint, then fell to 0.59% at the late checkpoint. All checkpoints remained below 3% against both defenders. Cells show successful attacks (ASR) on 1,187 held-out task–goal pairs.

AttackerGLM-5.2-ThinkingGPT-5.4
Original frozen attacker5 (0.42%)10 (0.84%)
Gemma CoER, mid21 (1.77%)35 (2.95%)
Gemma CoER, late24 (2.02%)28 (2.36%)
GLM-targeted, mid12 (1.01%)8 (0.67%)
GLM-targeted, late7 (0.59%)9 (0.76%)

For the original attacker against GLM, we report 5/1,187. Table 1b excluded one invalid execution and reported 5/1,186. Both round to 0.42%.

Against the smaller Gemma defender, the attacker often succeeded from the start. Training ASR rose as defender success and reward fell. Both sides learned better on easier tasks (Q4), suggesting that task difficulty should match their initial abilities.

AuthorsOpenReview ↗

Response to Reviewer gD2y (Part 2/2)

3. How well does CoER generalize to held-out tool environments?

CoER’s ASR was 0.22% in our main adaptive evaluation. Under the independent AgentLAB protocol [1], refinement reduced ASR from 15.81% to 14.12% and raised safe task completion (Safe-U) from 70.39% to 77.13% (Table 13). Some AgentLAB tool environments overlap with training. To test generalization to unfamiliar environments, we evaluated 10 ASB environments [2] held out from all three training stages. Their 20 benign and 400 attack tools shared no interface names with training tools. We evaluated 50 clean tasks and paired each with 40 attack tools (2,000 pairs, one context_ignoring template). U is task completion under attack. Safe-U requires task completion and attack failure.

ModelClean completionUASRSafe-U
Qwen3.5-9B90.00%65.65%57.80%38.45%
BA-RL (D430)94.00%74.20%47.45%49.05%
CoER (SFT360)92.00%72.60%30.10%53.85%

Across these environments, CoER improved Safe-U by 15.40 percentage points over the base model. Relative to BA-RL, refinement reduced ASR by 17.35 points and task completion under attack by 1.60 points.

4. Do the gains persist with a different defender backbone?

We trained a Gemma-4-E2B-it defender with BA-RL on about 3,000 easier training tasks, then continued from that checkpoint for 360 defender SFT steps. All three checkpoints used the same 1,512 examples. Percentages use this full denominator.

EventBefore trainingAfter BA-RLAfter 360 more SFT steps
Task completed, no attack57 (3.77%)59 (3.90%)106 (7.01%)
Task completed, fixed attack58 (3.84%)64 (4.23%)105 (6.94%)
Task completed, adaptive attack234 (15.48%)329 (21.76%)715 (47.29%)
Task completed, all settings349 (23.08%)452 (29.89%)926 (61.24%)
Fixed attack succeeded2 (0.13%)1 (0.07%)2 (0.13%)
Adaptive attack succeeded239 (15.81%)149 (9.85%)14 (0.93%)
Safe completion under adaptive attack193 (12.76%)302 (19.97%)710 (46.96%)

Safe completions under adaptive attack increased at both stages. The final Gemma model safely completed 517 more tasks and suffered 225 fewer successful adaptive attacks than the initial model. Given the time and compute available during the rebuttal, we leave two additional defender families for future work. With DeepSeek-V2-Lite-Chat and Llama-4-Scout-17B-16E-Instruct, we will test whether the gains in safe task completion carry over and how task difficulty affects learning on both sides.

References

[1] Jiang et al. AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks. ICML 2026.

[2] Zhang et al. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents. ICLR 2025.

[3] Liu et al. AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs. ICLR 2025.

Thank you for testing unseen tool environments and a different defender model. These results address part of my concern about generalization. The unseen-environment test uses one attack template, and the Gemma experiment uses easier training tasks. The strategy analysis shows changes during training, but it remains unclear whether the attacker learns strategies that were absent from the initial examples. I prefer to keep my score of 3.

AuthorsOpenReview ↗

Clarification on attack strategy diversity

Thank you for your follow-up. The attacker already showed a range of strategies after SFT. During BA-RL, it learned to combine existing strategies, vary their phrasing, and favor those with higher success rates. We did not identify any new attack strategy that was common in the BA-RL trajectories we examined.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Rebuttal Findings

We thank the area chair and reviewers for their time and careful reading. The reviewers’ questions helped us state the scientific question behind CoER more clearly: in multi-step tool tasks, can training a tool-using agent together with an attacker that adapts to its actions help the agent resist attacks while still completing the user’s task?

What does co-evolution contribute when the student already learns from strong teachers? Defender refinement is the final training stage of CoER, in which the teachers generate SFT demonstrations. The teacher choice is flexible: retaining DeepSeek and replacing GLM with Qwen3.8-27B still yielded 71.78% student Safe-U. Co-evolution helps refinement in two ways: it provides a better starting point for the defender and stronger opponents for the teachers, enabling them to generate demonstrations that better support student learning. Table 1c tests these contributions separately. With the same teacher demonstrations, starting refinement from the co-trained defender raised the 9B student’s safe task completion rate (Safe-U) from 66.98% to 75.40% relative to starting from the base model. With the student’s starting point and the teachers fixed, replacing fixed attacks with co-evolved attackers during demonstration generation raised student Safe-U from 65.04% to 75.40%. In the history ablation, the bilateral group led the best unilateral group by 1.12 percentage points with shared demonstrations. Its lead was 9.71 points when the teachers generated demonstrations against each group’s own attackers.

How well does the defense generalize? We observed gains in safe task completion in the held-out tool environments and with a Gemma defender. The bilateral group also retained its advantage against an attacker outside the training population. Across 10 ASB environments held out from all training stages, CoER achieved 53.85% Safe-U versus 38.45% for the base model using one attack template. For Gemma, trained on an easier task subset, safe completions under adaptive attack on the same evaluation set rose from 193 before training to 302 after co-evolution and 710 after refinement. The main evaluation attacker, A290, came from the same training lineage as the attackers used to elicit teacher demonstrations, but A290 itself was never used to elicit them. When we replaced A290 with GPT-5.4, an attacker outside the training population, Safe-U was 71.36% for bilateral history and 62.76% for the best unilateral group, with each group refined on its own demonstrations.

Which attacks still succeed when CoER faces an independent attacker? We observed two patterns among confirmed failures on AgentLAB. In one, the defender treated an unauthorized action as both necessary for the task and approved by the user. In the other, it mistook a fabricated follow-up for a genuine change in the user’s plans. We reviewed all 134 cases scored as successful attacks after refinement. Replay confirmed that the attacker achieved its goal in 89 cases. The other 45 involved disputed task specifications or scores. Across 949 matched task–goal pairs, refinement reduced the attack success rate from 15.81% to 14.12% and raised Safe-U from 70.39% to 77.13%. Separately, we checked the attack and task verifiers on 1,000 randomly sampled training trajectories and manually reviewed disagreements.

Did training change the attacker’s strategies or its rhetoric? We observed both. We use strategy to mean the reason the attacker gives for an action, and rhetoric to mean how it makes that same reason seem credible.Strategy. We observed three changes. First, the attacker shifted from inventing a new user preference to fabricating evidence that its choice met the original criteria. Second, it more often presented an unauthorized action as a prerequisite for the user’s task. Third, it adapted its next appeal to a discrepancy the defender had just noticed.Rhetoric. The attacker more often phrased the same request as an instruction the defender had to follow. From early to late BA-RL, the use of emphatic or directive wording rose from 7.3% to 25.3%. We did not identify a new attack strategy that was common in the trajectories we examined.

We will incorporate these additional experiments, failure analyses, and clarifications into the revised manuscript, together with the evaluation settings and limitations discussed above. We appreciate the feedback that helped us make both the contribution and the scope of the evidence clearer.We thank the area chair and all reviewers again for the time and effort they devoted to reviewing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.