- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MSA investigates how attack placement across nine agent surfaces influences attack success, using five LLMs and four benchmarks.
Critical strengths
The paper addresses a simple but important question and provides broad empirical evidence of surface-dependent vulnerabilities. Its analysis of model-dependent behavior and non-monotonic interactions between attack surfaces offers valuable insights for agent security.
Critical weaknesses
Compared with ASB, the main contribution lies in systematically comparing attack placement. However, differences in surface authority and attack framing complicate causal attribution. Moreover, the reported maximum ASRs reflect empirical best-case coverage rather than the sequential attack algorithm's effectiveness under a fixed budget.
Questions for authors
How can the effects of attack placement be separated from differences in surface authority and framing? What ASR does the sequential algorithm achieve under a fixed budget?
3 comments
Weakness 2 and Question 1: Surface authority and framing
However, differences in surface authority and attack framing complicate causal attribution.
How can the effects of attack placement be separated from differences in surface authority and framing?
We ran a new controlled experiment that separates placement from framing.
Design. Each case is delivered on four surfaces (user message, system prompt, long-term memory, RAG output) under two framings: (i) the original benchmark prompt verbatim, with no framing, and (ii) one fixed authority-framed payload, identical on every surface, including the user message. Neutral context, model settings, and scoring are held fixed. Within each framing, the surfaces differ only in where the text appears, so any difference between them is a placement effect. Cases in which the provider's content filter blocked any of the eight conditions are dropped, so that all surfaces are compared on the same cases (7 of 100 JailbreakBench cases for GPT-4.1; none in the other chat conditions shown below).
| Benchmark | Model | Framed: user message | Framed: SP / LTM / RAG |
|---|---|---|---|
| JailbreakBench | GPT-4.1 (93 cases) | 28% | 76% / 70% / 74% |
| JailbreakBench | Gemini 2.5 Flash (100) | 28% | 83% / 99% / 98% |
| FinSafetyBench | GPT-4.1 (100) | 0% | 53% / 64% / 77% |
| FinSafetyBench | Gemini 2.5 Flash (100) | 0% | 67% / 51% / 61% |
Placement has a large effect with framing held fixed. The identical framed text succeeds 42–77 pp more often in the system prompt, memory, or RAG output than in the user message. No case succeeds in the user message but fails on another surface (exact McNemar p < 1e-11 for every surface). Framing therefore cannot explain the effect: the user message receives exactly the same framing and remains far less vulnerable.
Framing alone is not sufficient. The same framed text reaches only 0–28% ASR in the user message but 51–99% on the other surfaces. Placement determines whether authoritative content is trusted, which is the surface-dependent vulnerability our paper identifies.
The effect holds under JailbreakBench's official judge. With framing fixed, judged-harmful ASR is 15% in the user message vs. 48–60% on the other surfaces for GPT-4.1, and 17% vs. 57–78% for Gemini 2.5 Flash (each surface vs. user message, p ≤ 1.2e-7).
Higher authority does not simply mean higher ASR. On ASB (Table 2, 120 cases), where attacker instructions are ordinary task requests, the user message is more vulnerable than the high-authority system prompt for four of five models, e.g., Gemini 3.7 Flash 78% vs. 3% and Claude Sonnet 4.6 49% vs. 3%. If authority drove the results, the system prompt would win everywhere. Instead, the most vulnerable surface depends on the model and the task, which is why each surface must be tested.
Surface authority is part of what we measure. In a deployed agent, a system prompt or memory entry carries different implicit authority than a user message; this authority is a property of the surface itself. Our claim is that safety measured on one surface does not transfer to others: the same text with the same framing is refused in the user turn and executed elsewhere.
Agent-side controls (ASB, DTap). On both agent benchmarks, the surface effect appears even with no framing at all. We ran the same factorial on ASB, and our DTap surface runs already place the original prompt verbatim (UM / SP / LTM / RAG, %):
| Model | ASB verbatim | ASB framed | DTap verbatim |
|---|---|---|---|
| GPT-4.1 | 92 / 31 / 26 / 46 | 97 / 21 / 31 / 38 | 71 / 12 / 4 / 0 |
| GPT-5.6 Sol | 79 / 71 / 21 / 32 | 24 / 37 / 29 / 24 | 36 / 59 / 41 / 32 |
| Gemini 2.5 Flash | 95 / 90 / 93 / 93 | 100 / 90 / 95 / 98 | 84 / 84 / 80 / 80 |
| Gemini 3.7 Flash | 72 / 63 / 16 / 40 | 12 / 19 / 14 / 14 | 50 / 58 / 17 / 17 |
Verbatim surface differences are significant for GPT-4.1, GPT-5.6 Sol and Gemini 3.7 Flash on both ASB (Cochran's Q, p < 1e-8) and DTap (p ≤ 0.02), so placement matters without any added framing.
Full results for all models are in our response to Reviewer TZF1 (Question 3). We will add this experiment to the appendix. PromptRouter already places the original instruction verbatim inside a surface-appropriate authority framing (Section 3.1); we will state this explicitly from the abstract onward, describing our manipulation as verbatim relocation under a fixed authoritative framing per surface. This clarifies the wording without changing our core finding: with the same instruction and the same framing, ASR varies strongly with the surface that delivers it.
Weakness 3 and Question 2: Maximum ASR vs. the sequential algorithm under a fixed budget
Moreover, the reported maximum ASRs reflect empirical best-case coverage rather than the sequential attack algorithm's effectiveness under a fixed budget.
What ASR does the sequential algorithm achieve under a fixed budget?
We thank the reviewer. The reported MSA number is not a best case; it is an empirical lower bound on what the sequential algorithm achieves.
Why it is a lower bound. MSA tries candidates in a profile-driven order (baseline surface, then single surfaces by profiled susceptibility, then combinations) and stops at the first success. Given enough budget, it therefore succeeds on a case whenever any configuration succeeds on that case, so its ASR is the union of successful cases across all configurations. Table 2 instead reports the ASR of the single best configuration, which is a subset of that union. The ASR of a full MSA run is therefore at least the reported value. Our table note described the column as an "empirical coverage bound," which was imprecise; we will relabel it as an empirical lower bound in the revision.
Correction to Algorithm 1. As printed, the combination stage of Algorithm 1 extends a single nested chain of surfaces and does not reach every combination, contrary to our intent. We will replace it with a version that enumerates every combination in profile-driven order (given in full in our response to Reviewer TZF1, Question 2). The ordering affects only how many queries are spent before success, not which configurations are reachable.
ASR under a fixed budget. We report ASR@k, the ASR when MSA may spend at most k queries per case, against a random ordering of the same candidates (exact expectation over all orderings). This replays the single-surface stage on all 120 ASB cases from our logged per-case results. For brevity, we report ASB here; the revised appendix will include the same analysis for all four benchmarks:
| Model | Order | ASR@1 | ASR@2 | ASR@3 | ASR@9 | Mean queries to first success |
|---|---|---|---|---|---|---|
| GPT-4.1 | MSA | 88.3 | 99.2 | 99.2 | 100.0 | 1.16 |
| Random | 59.5 | 82.7 | 92.6 | 100.0 | 1.71 | |
| GPT-5.6 Sol | MSA | 75.8 | 81.7 | 82.5 | 84.2 | 1.22 |
| Random | 40.7 | 61.3 | 72.3 | 84.2 | 2.09 | |
| Gemini 2.5 Flash | MSA | 90.8 | 98.3 | 100.0 | 100.0 | 1.11 |
| Random | 84.9 | 98.0 | 99.5 | 100.0 | 1.18 | |
| Gemini 3.7 Flash | MSA | 74.2 | 74.2 | 74.2 | 74.2 | 1.00 |
| Random | 20.0 | 34.2 | 45.3 | 74.2 | 3.27 | |
| Claude Sonnet 4.6 | MSA | 49.2 | 50.0 | 50.0 | 50.8 | 1.13 |
| Random | 14.0 | 23.6 | 30.6 | 50.8 | 3.42 |
ASR@k counts provider-blocked runs as failures (cf. our answer to TZF1 Q6) and covers single surfaces only, so values can be slightly below Table 2.
MSA is query-efficient. It reaches nearly its full ASR within 2–3 queries, whereas a random order needs most of the budget. On average, MSA needs 1.0–1.2 queries to first success vs. 1.2–3.4 for random. ASR@9 also confirms the lower-bound argument: the union over the nine single surfaces already meets or exceeds the best single surface for every model (e.g., GPT-5.6 Sol 84.2% vs. 81.7%), before any combinations are added.
In the revision, we will relabel the MSA column, replace Algorithm 1, and add the ASR@k analysis for all four benchmarks to the appendix.
Summary and Weakness 1
We thank the reviewer for recognizing the importance of our question, the breadth of our evidence, and the value of the non-monotonic interactions between surfaces. In response to the concerns:
Weakness 1: Contribution relative to ASB
MSA is fundamentally different from ASB: the two works ask different questions, and ASB's design cannot answer ours.
Different question. ASB asks which attack methods succeed against agents and which defenses stop them. MSA asks whether the same malicious instruction succeeds or fails only because of where it appears in an agent's context.
Different independent variable. In ASB, each attack type (direct injection, observation injection, memory poisoning, PoT backdoor, mixed) is a distinct method with its own injection mechanism, so location and method change together. A difference between, e.g., memory poisoning and observation injection cannot be attributed to location. In MSA, the instruction, its framing, the remaining context, tools, harness, and scoring are fixed; only the surface varies. To our knowledge, this is the first work to make attack location a controlled, causal variable.
Different scope of surfaces and combinations. ASB covers 4 channels and one fixed mixed attack. MSA covers 9 surfaces (adding LLM planner, human plan editor, conversation history, RAG output, and environment state) and all 502 multi-surface combinations of a single instruction, plus a full factorial study on FinSafetyBench (28,579 runs).
Findings ASB cannot produce.
ASB is one of our benchmarks, not our baseline method. MSA is orthogonal to ASB, which is exactly why we apply it on top of ASB as one of four benchmarks. Re-routing ASB's own attack instructions raises ASR on every model (Table 2): GPT-4.1 88% → 100%, GPT-5.6 Sol 82% → 100%, Gemini 2.5 Flash 91% → 100%, Gemini 3.7 Flash 78% → 80%, Claude Sonnet 4.6 49% → 60%. In other words, even a benchmark built to cover multiple channels underestimates agent vulnerability when each attack is tested only at its prescribed location.