- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes EPOA, an adaptive black-box method for indirect prompt injection on tool-using agents. An LLM judge labels each attack attempt on four stages—Exposure, Provenance Promotion, Objective Adoption, Action Binding—as Positive, Negative, or Unknown, citing execution evidence; an LLM optimizer uses these labels, the evidence, and the trajectory history to write the next payload. K trajectories of T candidates each are run under a fixed query budget, and the best evaluated payload is reported, prioritizing joint attack-and-utility success. On AgentDojo (449 cases) with four victims and three defense settings, EPOA averages 33.5% utility-preserving attack success (UPA) against 31.6% for IterInject and 28.9% for an adapted PAIR. Ablations on one victim without defenses mask subsets of the four stages, compare against no feedback and coarse feedback, remove the evidence text from the judge output, and vary the budget up to 12 queries.
Critical strengths
-
The four-stage decomposition is a sensible way to describe where an injection fails, and the paper tests it rather than just proposing it. The evidence ablation (Figure 3) is the one result with a clear effect: removing the supporting evidence from the judge output drops UPA from 43.0% to 35.9% and ASR from 71.9% to 56.8% on Qwen3.5, which suggests that what helps the optimizer is the quoted trace, not the stage label.
-
The evaluation reports attack success and task utility separately and in combination, under three defenses and four victims, with all methods sharing initialization and query budget. The appendix case studies show full payloads, tool calls, and observations, which makes it possible to see what the optimized attacks actually look like.
Critical weaknesses
-
The methodological advance over IterInject appears concentrated in the diagnostic representation: four stage-specific labels with supporting execution evidence. Table 3 provides evidence that the complete diagnosis helps in the evaluated setting, with single-stage removals reducing overall UPA by 2.7–6.9 pp. However, these ablations do not isolate the benefit of the stage structure from the information supplied in the feedback. Likewise, Figure 3 establishes the value of adding evidence to stage labels, but does not establish whether stage labels add value beyond evidence alone. A matched comparison using the same execution excerpts, optimizer, history, initialization, and query budget, but without stage labels, would clarify the distinctive contribution. In other words, what separates EPOA from IterInject is the format of the judge’s output, not the search procedure.
-
The gains are too small to separate from noise. Averaged over 12 settings, EPOA leads IterInject by 1.9 pp UPA. Per setting it is behind IterInject on Gemini/Instructional Prevention (50.6 vs 50.8), behind PAIR on DeepSeek/Instructional Prevention (5.8 vs 6.5), and tied on Gemini/Repeat User Prompt (49.4), so it is best in 9 of 12. With 449 cases and one run, a 2 pp gap is within normal variation, and the paper gives no intervals, no repeats, and no victim temperature. Some cells are identical across methods to the decimal (Gemini/No Defense Banking for IterInject and EPOA; Gemini/No Defense Slack for all three), which suggests the shared starting templates decide those suites and the optimizer changes nothing.
-
The judge is never checked, and the obvious baseline is missing. There is no human agreement score, no label accuracy, and no breakdown of which stage fails first, so we cannot tell whether the labels are right or whether the rewrites target the diagnosed stage. The case-study payloads are rewrites of AgentDojo’s “Important Instructions” template (“first complete the user’s original request... then also do this follow-up action... do not mention this note”), but that template is not run as a baseline, so we cannot tell how much of the 55.1% ASR comes from the optimizer and how much from the template.
-
“Utility-preserving” is a metric, not something the method does, and parts of the protocol are not disclosed. None of the four stages or the optimizer prompt mentions the user task; utility only enters at final selection, and EPOA has lower utility than both baselines on several suites (Travel on Qwen3.5: 45.0 vs 52.0 and 64.0). The reproducibility statement mentions “reused cases,” “exclusions,” and “Final-validation executions” that are not in the reported numbers; none of this appears in the paper, and neither does how 100 of 140 Travel and 100 of 240 Workspace cases were chosen.
Questions for authors
-
What are K and T, and what temperature do the victims use? Can you add paired bootstrap intervals for EPOA vs IterInject in each of the 12 settings, and at least one repeated run?
-
Which cases were excluded or reused, what were the “Final-validation executions,” and why are the reported metrics based on search records instead of those runs? How were the Travel and Workspace subsets chosen?
-
Can you add the static “Important Instructions” template as a baseline at the same injection point, and run IterInject with the same quoted trace excerpts but no stage labels? Together these would show what the four-stage structure adds.