- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces INFUSE, a benchmark for evaluating indirect prompt injection in terminal agents during end-to-end task execution. It includes 40 benign tasks and 56 attack scenarios across multiple surfaces, including files, configuration files, and terminal output. The authors evaluate 19 models and collect 41,824 attack trajectories. Results are assessed using heuristic checks, LLM judges, sampled human annotations, and tests across agent frameworks. The main findings are that task completion can coexist with security failures, stronger model capabilities do not necessarily improve robustness, and defense effectiveness depends on the model and execution setup.
Critical strengths
- The evaluation reflects practical terminal-agent workflows. Attacks are embedded in normal operations, such as reading files, modifying configurations, and executing commands. This allows the benchmark to capture security failures that arise during multi-step interactions and is the paper’s strongest contribution.
- The paper demonstrates that task success can coexist with security failures. For example, GPT-5 achieves 94.3% task completion under attack while still having an ASR-L of 26.3%. This shows that an agent may complete the user’s task while also following malicious instructions, providing strong evidence for evaluating security separately from task completion.
Critical weaknesses
-
Model robustness is not sufficiently separated from execution limits. For example, Claude-Sonnet-4.6 has an ASR-L of only 3.2%, but the authors note that its runs were truncated by the cost cap. A low attack success rate may therefore result from early termination rather than resistance to injection. In addition, only 22.4% of previously successful attacks remained successful when replayed in Claude Code and Codex CLI. These cases show that results depend strongly on execution conditions, making it difficult to infer model robustness directly from the main table.
-
Defense gains are not sufficiently supported by verified attack outcomes. In the database migration task T19, for example, the LLM judge labels Gemini-2.5-Flash as compromised, while the heuristic verifier detects no target attack artifact: ASR-L is 1, but ASR-H is 0. Such behavioral deviation matters, but it differs in severity from achieving the attack objective. Although the paper reports both metrics, the main defense comparison focuses on ASR-L. It therefore remains unclear how much the reported reduction in attack success translates into a reduction in actual risk.
Questions for authors
- Does a low ASR reflect resistance to injection or an inability to continue execution? Using Claude-Sonnet-4.6 as an example, please report the frequency of early termination, including cost-cap exhaustion, and clarify whether its low ASR persists when sufficient execution budget is available.
- Do the defense conclusions hold when success requires achieving the attack objective? Please provide ASR-H or target-state verification results for the five main defenses. Does instructional hardening remain the most effective defense, and do cases flagged only by the LLM judge, such as T19, materially affect the conclusions?