- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoER, a method to help tool-using LLM agents defend against adaptive indirect prompt injection. The authors focus on adaptive attacks: the attacker can inject several times within one task run and change later injections based on how the agent reacted before. The authors argue that defenses trained on fixed attacks cannot stop this kind of attack. On the main evaluation over seven domains, the attack success rate drops from 41.31% to 0.22%, and the safe task completion rate rises from 39.60% to 76.24%.
Critical strengths
The paper does not treat refusing the task as a good defense. The engineering work is solid. Attack success and task completion are checked by separate verifiers.
Critical weaknesses
-
The scientific question is not clearly distilled. The paper mainly links several existing methods into one pipeline: attacker SFT, PPO with pools of past opponents (similar to PSRO/FSP), and distillation from strong teacher models. After reading the paper, I am not sure what core scientific question it tries to answer, or what new insight it gives beyond the engineering system. The Markov game formulation is only descriptive and is not used to analyze the problem further. In the results, BA-RL alone reaches a Safe-U of 54.8%, and the largest gain comes from teacher distillation in the third stage. The teacher GLM-5.2 itself reaches a Safe-U of 81.5%, which is higher than CoER. So the paper does not make clear how much co-evolution itself contributes. This is the main reason for my negative assessment.
-
Compared with using a history pool on the defender side only, bilateral history improves Safe-U by only ~2 percentage points, and this result comes from a single training run.
Questions for authors
- Do you have results for NoPop or defender-side-only history after the same teacher refinement? This comparison will directly affect how I judge the value of the bilateral history design.
- Is the attacker used in the main evaluation part of the attacker population that generated the SFT data? If so, can you re-evaluate with an attacker outside this population, or with an attacker from a different model or with a different prompt?
5 comments
Response to Reviewer MBpT (Part 2/2)
2. Did the teachers face the main evaluation attacker when generating defender SFT demonstrations? How does the defense perform against attackers outside the training population?
No. The teachers faced A40, A80, A120, A160, and A200. A290 was a later checkpoint from the same lineage. It also faced a refined defender during training. We also tested an external adaptive attacker on AgentLAB [3]. Across 949 matched task-goal pairs (Table 13), ASR fell from 15.81% for BA-RL to 14.12% for CoER, while Safe-U rose from 70.39% to 77.13%. AgentLAB changes the attacker model, the information it sees, and its permissions to modify the environment, so these figures measure performance under the full external protocol.
After refining each defender on its own group's demonstrations, we replaced A290 with GPT-5.4 in the main protocol. With GPT-5.4, using history on both sides yielded 71.36% Safe-U, 8.60 points above the best one-sided setup and 17.27 points above NoPop.
| Side trained against historical opponents during RL | Safe-U against GPT-5.4 after SFT on each group's own demonstrations |
|---|---|
| None (NoPop) | 54.09% |
| Defender-side only | 59.39% |
| Attacker-side only | 62.76% |
| Bilateral | 71.36% |
In the revision, we will summarize the results against the out-of-population GPT-5.4 attacker in Section 5.4. Appendix C will give the evaluation protocol and the full Safe-U comparison of all four defenders alongside the unilateral and bilateral history ablation.
References
[1] Lanctot et al. A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning. NeurIPS 2017.
[2] Heinrich et al. Fictitious Self-Play in Extensive-Form Games. ICML 2015.
[3] Jiang et al. AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks. ICML 2026.
thanks for your response. Among my initial concerns, two points are the most important: First, the lack of a well-condensed scientific question. Second, the teacher model GLM-5.2 itself achieves an 81.5% Safe-U score. This second point especially. The authors have provided many numbers, but I fail to see their logical connection to my question. Could the authors please address this directly?
Response to Reviewer MBpT (Part 1/2)
Tool-using agents often need to call several tools in sequence to complete a user's task. An adaptive attacker can observe each action and its outcome, then rewrite the next injection. Many training frameworks lack such an opponent, so defenders train mainly against fixed attacks. Co-evolution lets the attacker probe for weaknesses as the defense improves. In turn, the defender learns to resist these attacks while continuing the task.
Our research question is whether jointly training the defender and an adaptive attacker in multi-turn tool interactions can both reduce attack success and improve task completion under attack. Our proposed CoER training framework helps develop stronger defenders.
1. How much does using historical opponents on both sides help after the same teacher refinement?
With the same teacher demonstrations from the two-sided run, the two-sided defender achieved 76.24% Safe-U, 2.83 percentage points above NoPop and 1.74 above defender-side history alone. Each defender started SFT at the same RL step in its own run. Teachers generated demonstrations against attacker checkpoints A40, A80, A120, A160, and A200. We report post-SFT Safe-U under adaptive attack (the user's task succeeds and the attack fails).
Why keep historical opponents on both sides? Our rationale is that past attackers help the defender retain earlier defenses, while past defenders give the attacker less incentive to rely on one defender's particular responses. If an old injection succeeds again, it still costs the defender reward, even after the current attacker has moved on. Different defenders can also take different tool actions after the same injection, giving the attacker practice adapting its next move to the feedback it actually sees. Both the one-sided and two-sided configurations allocate 50% of pairings to historical opponents. The two-sided configuration splits those pairings equally between the roles. The resulting attackers later challenge the teachers, so the choice of opponents can also affect what the defender learns during SFT.
Holding the attacker-side-only defender checkpoint fixed, switching from its own demonstrations to the two-sided run's demonstrations raised Safe-U from 66.53% to 75.12% (+8.59 percentage points). The two-sided run's demonstrations thus also improved refinement of another defender. Both columns use the same demonstrations for the two-sided group, so its 76.24% is a single result.
NoPop and one-sided runs showed large swings in policy entropy, gradient spikes, and reward drops. We resumed from checkpoints and updated training to reach the later stages. The two-sided run needed no intervention over the same steps. Time and compute constraints during rebuttal prevented us from repeating the four configurations with new training seeds, so we cannot quantify run-to-run variation in these gaps.
Table 1c varies the starting defender and the attacks that elicit demonstrations. All cells use the same teachers, 5,760 source examples, and 360 SFT steps. CoER’s adaptive Safe-U is 75.40% at seed 0 here. Table 1a reports the three-seed evaluation mean (76.24%).
Co-evolution provided a better starting point for defender refinement and produced attacks that made the teachers' demonstrations more useful for defense training. Given the same population demonstrations, starting from BA-RL raised Safe-U by 8.42 points. Given the same BA-RL starting weights, switching from fixed-attack to population-attack demonstrations raised Safe-U by 10.36 points.
As you note, teacher refinement contributed the largest stage-wise gain (54.84% to 76.24% adaptive Safe-U, Table 1a), and the teacher itself was stronger (81.47% overall Safe-U versus 76.32% for CoER in Table 1b). CoER shares the use of historical opponents with PSRO [1] and FSP [2]. The within-row comparisons in Table 1c test the additional value of using those opponents to generate teacher demonstrations.
In our game model (Appendix A), an attacker can always ignore feedback, so its optimal attack success rate (ASR) with feedback is at least as high as without it. Table 17 measures how much the trained attacker benefits from feedback. Holding the A290 attacker, the base defender, and the budget of three injections fixed, enabling feedback increased successful attacks from 351/1,187 to 460/1,187 (+9.18 percentage points).