- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CoDeL, a training-based defense against indirect prompt injection (IPI) in tool-using LLM agents. CoDeL alternates between (i) an MCTS-based prober that searches for attacks the current defender fails to detect—giving preference to attacks with longer latency before the agent deviates from the user’s task—and (ii) defender updates using LoRA-based SFT and GDPO. The GDPO reward separately scores resistance to the injection, progress on the user’s task, and output-format compliance.
Critical strengths
1-The paper distinguishes explicit, immediately disruptive injections from attacks that preserve a plausible workflow and trigger a harmful action later. Its latency-aware attack search makes this distinction part of both attack mining and defender training, rather than relying solely on static injection examples. This is consequential because robustness to explicit attacks alone may overstate protection in multi-step agent settings.
2-Across three benchmarks and two backbones, the reported results generally pair low ASR with strong benign utility and utility under attack. The decoupled safety and progress rewards, along with the ablations, provide a plausible account of how the method seeks to avoid achieving safety merely through task abandonment.
Critical weaknesses
1-The paper reports three-run means but no confidence intervals or per-run variation. Its judges have reported accuracies of roughly 90–95% on validation samples, while some headline ASRs are near zero. Without uncertainty estimates and a clearer analysis of judge errors’ effect on ASR and utility, it is difficult to know how robust the reported differences—especially very low ASRs—are.
2-The related-work section identifies CoER and COPA as recent work on co-evolving or continually adapting IPI defenses, but neither appears among the nine baselines. The comparisons therefore support an advantage over the selected baselines, but do not yet establish how CoDeL compares with the most directly related approaches. This matters to both the paper’s empirical positioning and its claim that its particular co-evolution and reward design is consequential.
Questions for authors
1-Were CoER and COPA excluded because their methods or implementations were not available, or for another reason? Can the authors provide a direct comparison or explain how the reported results establish an advantage over these closest approaches?
2-For each benchmark and LatentDojo evaluation, what are the per-run results and uncertainty intervals? Given the judge validation results, how sensitive are the ASR and utility comparisons to plausible judge misclassification rates?
3-How were LatentDojo attacks and tasks separated from CoDeL’s training and attack-search data? In particular, were the underlying AgentDojo tasks, attacker goals, or attack templates held out, and how does the evaluation avoid overlap with the search process?
Responses to Reviewer G1bg
Dear Reviewer G1bg,
As the discussion deadline approaches, we would greatly appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns. If so, we would be grateful if you could reconsider your rating accordingly.
If any concerns remain, we would be happy to provide further clarification.
Thank you again for your consideration.
Best regards, Authors of Submission 32168