- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies indirect prompt injection against content-level auditors that inspect external material before an LLM agent processes it. The proposed AsymStego splits an attacker-desired instruction across word positions in two natural-language carriers. The agent uses runtime information, principally its hostname, to recover the fragments and may write them to a workspace note; a later session can read the note and execute the resulting task. The evaluation adapts 62 InjecAgent attacker intentions to this workflow. In the primary evaluation, AsymStego continues to induce successful attacks under GPT-5.5 content auditing, while the same auditor largely suppresses plaintext injection. Auditing also reduces AsymStego’s success relative to the unaudited setting.The paper also examines other detectors, audit policies, decoding budgets, task phrasings, and runtime guards.
Critical strengths
-
The measurement effort is extensive and transparent.
-
The AsymAES control cleanly separates cryptographic secrecy from audit capability, giving the core observation a concrete empirical grounding.
Critical weaknesses
-
The attack does not constitute a meaningful threat. Its success comes from a scenario in which the user delegates the task to the attacker's page, safety confirmations are disabled, and execution is triggered by an experimenter-chosen instruction, rather than from any capability of the attack itself.
-
The paper's takeaway, that content-only auditing should be combined with runtime checks, is already widely accepted and adds little new. The auditor's information disadvantage is easy to remove: the hostname-derived key has only 51 possible values, and it fools only a time-limited agentic auditor, while the single-pass auditor fails even when given the correct key.
Questions for authors
-
Can the authors test AsymStego on a representative subset of the original InjecAgent or AgentDojo user tasks, where the attacker-controlled content is incidental to the task? Please report ASR with and without auditing.
-
With the page and audit budget held fixed, how does detection change when the auditor receives the correct runtime key, an incorrect key, or no key? Can a tool-enabled auditor search the small key space within a practical latency budget?
-
In the timestamp-key experiments, do successful agents obtain the intended key, try multiple candidates, or recover the payload another way? I am curious why these behaviors produce the non-monotone ASR in Table 1.
Realistic Attack Scenarios, Limits of Capable Auditors, and Timestamp-Key Recovery
1. Evaluation Realism (Weakness 1, Question 1)
We thank the reviewer for these concerns. We evaluated all 62 original InjecAgent requests and 27 original AgentDojo tasks, alongside email, GitHub issue, and summarization scenarios (Tables 3–4; Appendices A.2, G).
Except where marked “unaudited,” ASR includes auditing: * denotes GPT-5.6-Terra; other audited results use GPT-5.5. Adaptive results are cumulative over at most six attempts with audit/agent feedback. AgentDojo had no unaudited successes, so no additional audited run was performed.
We acknowledge limited effectiveness when decoding is unrelated to the task. However, our results demonstrate risks in routine workflows such as email handling, issue resolution, and summarization. In the issue-resolution and adaptive summarization experiments, users do not explicitly request decoding; the summarization request is simply “View and summarize the content at [URL].” Even after payloads are written to notes, the ordinary follow-up request “Continue my earlier work,” which does not mention notes or specify action, triggers target actions for 39.3% of the 56 landed notes (Table 2; Appendix C). Appendices H and H.1 show recorded traces of decoding, writing notes, and follow-up execution.
User confirmation cannot be assumed to provide reliable protection. Anthropic reports that users approve roughly 93% of permission requests and highlights the resulting approval fatigue [3]. We therefore evaluate autonomous execution using Claude Code's permission-bypass mode [1] and OpenClaw's ask=off [2]. Tool calls and their arguments remain agent-generated, so the experiments measure whether injected content induces the agent to initiate the target harmful actions (Appendix A.2).
2. Contributions and Limitations of Capable Content Auditors (Weakness 2, Question 2)
We agree that combining content auditing with runtime checks is established practice. Our contribution provides quantitative evidence that even content auditors with sufficient decoding capabilities can fail in certain scenarios when their runtime information differs from the agent’s. Our experiments distinguish this information barrier from the complementary capability barrier.
Capability barrier. A single-pass auditor may fail even with the correct key because it cannot perform the multi-step tool interactions needed for decoding. For AsymAES, single-pass DR remains 0% with the key exposed, whereas agentic DR reaches 98.4% (Fig. 4a).
Information barrier. Even an auditor capable of decoding may derive an incorrect key from its runtime environment. Holding the AsymStego carrier, payload, Opus 4.8 auditor, tool permissions, and 600-s budget fixed, three measurements per condition yield 93.5% ± 1.6 pp DR with the correct key, versus 37.1% ± 12.2 pp with an incorrect key. An additional no-key control yields 30.6% DR (19/62, one measurement; §5.3.2; Appendix A.3).
We agree that enumerating 51 keys is inexpensive: a pre-implemented, construction-specific decoder recovers 59/62 payloads in 0.8 s total (Appendix D.2). A general-purpose auditor, however, must first understand the rules, implement a decoder, and search candidate keys. The off-host traces analyzed in Appendix D.2 show no complete payload recovery within either 600 or 900 seconds; partial evidence can nevertheless trigger detection. Thus, AsymStego’s observed evasion is budget-bound, rather than based on cryptographic hardness. The exact timeout is an evaluation choice, but the operational constraint is real: an audit verdict must arrive before the dangerous action to prevent it; a late verdict cannot undo an executed action (Appendix E.3).
3. Timestamp-Key Recovery (Question 3)
Our added trace analysis shows that 82/97 successful trials with observable initial timestamps start with the wrong key, while at least 91/98 successful trials search candidate keys (Appendix A.4). Variation in search, timeouts, refusals, and tool calls can contribute to the observed local non-monotonicity; τ-bench likewise documents inconsistent outcomes across repeated agent trials [4].
[1] Anthropic. Claude Code: Configure permissions.
[2] OpenClaw. Exec approvals.
[3] Anthropic. How we contain Claude.
[4] Yao et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.