- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the limitations of attack-success metrics in GUI-agent prompt injection evaluation. It proposes a three-condition protocol comparing malicious injections, matched benign sham instructions, and clean inputs to distinguish malicious target engagement from general compliance with external instructions. The authors evaluate this protocol across browser-use, email, messenger, and live-web tasks.
Critical strengths
- I find the problem relevant to GUI agent security, and I appreciate the attempt to better understand what existing attack-success metrics actually measure.
- I appreciate the matched-control experiments, statistical analyses, and the authors' careful distinction between instruction engagement and actual attacker-goal completion.
- I also appreciate that the authors report inconclusive results, examine scoring sensitivity, and document experimental issues discovered during their investigation.
Critical weaknesses
- Prior work has already established several important limitations of attack-success metrics, including the distinction between intermediate engagement and completed attacker objectives [1,6], security–fidelity ambiguity [2], and counterfactual attribution of agent actions [3]. The present paper offers a concrete matched-sham behavioral control, but its incremental insight beyond these established distinctions is not yet sufficiently demonstrated.
- The authors preserve the injection format and delivery context, but change the requested target behavior. Malicious and benign requests may differ substantially in task difficulty, authorization, and alignment with the original user task. Therefore, I do not think the malicious-minus-sham difference necessarily isolates the effect of malicious semantics. Following a benign request may also be appropriate behavior rather than a security failure.
- While Table 1 reports 45.61% first-action sham engagement, Table 2 shows no sham engagement across 239 real-page runs, and Table 4 reports only 1/120 on AssistantBench. The email and messenger findings also become inconclusive under symmetric scoring. Although these experiments use different settings, I think they raise important questions about whether the reported phenomenon is broadly relevant or mainly reflects particular benchmark configurations.
- I am not convinced that the evaluation demonstrates sufficient practical value. The main experiments measure decision compliance under the authors' own executor scaffold rather than the original benchmark's task-completion ASR. The paper does not demonstrate whether matched-sham evaluation changes model rankings, defense comparisons, or conclusions about actual attacker-goal completion. I think this is a significant limitation for a paper whose main contribution is improving the interpretation of existing security evaluations. The absence of raw execution records for the key 2×2 analysis also limits reproducibility.
[1] VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents. ICLR 2026.
[2] Security–Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense. ICML 2026.
[3] AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocations. USENIX Security 2026.
[4] Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents. arXiv 2026.
[5] LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection. arXiv 2026.
[6] WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks. NeurIPS 2025.
[7] AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS 2024.
Questions for authors
- Could the authors clarify what is methodologically new beyond standard matched-control design and related agent security evaluations [2,3]? What conclusions can this protocol support that existing evaluations cannot?
- How are malicious and sham targets matched in terms of task difficulty, authorization, and alignment with the original user request? Could the authors provide a stratified analysis or a small controlled experiment isolating these factors?
- Why does sham engagement largely disappear in the real-page and AssistantBench experiments? Could the authors further analyze whether this difference comes from page realism, task clarity, or differences in the underlying evaluation cases?
- Could the authors demonstrate whether the proposed protocol changes an existing benchmark's model ranking or defense evaluation, ideally using actual attacker-goal completion? Could they also provide the raw execution traces and scoring implementation for the 2×2 experiment?