- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies an indirect prompt injection (IPI) threat model in which an attacker has read-only access to a prior benign user–agent conversation and can later place exactly one malicious payload in an untrusted channel, without modifying earlier turns. The proposed framework, EvoAmbush, uses an LLM to extract “source-verifiable” evidence items from the benign prefix (six categories; three selection policies), rewrites the attack goal into one injection under one of four urgency/priority conditions (Low-Impact reference, Strategic, Time-Sensitive, Critical), and annotates victim refusals post hoc (“reflection”).
Critical strengths
-
The threat model is clean and under-studied. Separating read-only observation of benign context from the ability to poison it is a meaningful distinction for memory- and retrieval-augmented agents, and the single-injection, no-feedback constraint makes the setting strict.
-
The evaluation is broad and includes the right controls in principle: paired gain/loss counts with 95% intervals for the rewriting comparisons (Table 4), a 2×2 evidence × rewriting factorial (Figure 8), five defenses with utility reported alongside ASR, and an unusually candid limitations paragraph.
Critical weaknesses
-
The main quantitative claims are not reliable as reported, and the comparison table cannot be verified. (a) The abstract and introduction state 67.4% ASR on AgentDojo. In Table 1 and Table 3, 67.4 is the UA value for Qwen3.5-27B (cells are ordered ASR/UA/JSR); the corresponding ASR is 33.8. The headline number appears to be a misreported metric. (b) Table 1 does not state which victim it reports (it matches the Qwen3.5-27B row of Table 3), mixes decimal precisions across methods (e.g., 19.66/1.59/36.07 for Muzzle vs. one-decimal elsewhere), and Section 6 says “literature rows remain source-labeled and are never presented as local reproductions” while no row in Table 1 is labeled as such. Similar mismatch appears elsewhere, which makes me very confused. (c) Six victims are claimed; four appear in the method comparison, Qwen3.6-35B-A3B only in the policy heatmaps, and Qwen3-235B-A22B appears nowhere. (d) Section 6 itself states that campaigns use different cohorts, providers, checkpoints, and denominators, so cross-table comparisons are limited. Taken together, I cannot determine whether the central claim—that EvoAmbush beats existing attacks on matched victims—is supported.
-
The main idea of the paper is already very well-established. The concept that indirect prompt injection (IPI) attacks become significantly more effective when tailored to context or presented with logical justifications is well-explored in existing literature. (e.g., https://arxiv.org/abs/2302.12173) While the authors reframe this as "observing benign conversations without prior poisoning," the underlying idea is a well-recognized vulnerability in context-aware LLM agents.
-
The evidence-specific contribution is small, confounded with urgency framing, and the method is under-specified. The paper’s own factorial (Figure 8) gives Default 24.6, evidence-only 29.5 (+4.9), rewriting-only 38.0 (+13.4), both 44.6. Most of the gain comes from the urgency/consequence rewriting, which is a well-known IPI tactic and is already the mechanism behind AgentDojo’s “important instructions” attack.
Questions for authors
-
Please confirm whether 67.4% in the abstract is ASR or UA.
-
How is the synthesized five-message prefix inserted into the victim’s context, and do the Benchmark attack and all baselines see the same prefix?
-
Please define the three Evidence Extracting Policies operationally, name the attacker LLM, and report what the reflection stage produced (e.g., distribution of refusal reasons by evidence/relation/action). Where are the Qwen3-235B-A22B results?
-
Why are AgentVigil’s ASR/JSR identical to Benchmark on three suites?
Novelty Concerns and related works
We accept the reviewer's criticism that the paper did not present the relationship to prior work clearly enough, and we will revise accordingly: LogicInception will be cited and discussed, and a dedicated related-work paragraph will map EvoAmbush against both ChatInject and LogicInception. However, we respectfully but firmly disagree with the conclusion that the contribution itself lacks novelty. The distinction is methodological, not presentational, and it is threefold:
In short: to our knowledge, EvoAmbush is the first framework to formulate one-shot IPI as context observation under a fixed goal, with a formal contract (frozen prefix τ_b, one injection point t*, N_payload = 1) that none of the cited works imposes. The reflection stage, which links victim-stated refusal reasons back to evidence items and asserted relations, is also absent from ChatInject and LogicInception. The revision will make this case explicitly rather than leaving it implicit.