- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ASPIRE, an automated red-teaming framework for open-ended prompt-injection vulnerability discovery in LLM agents. ASPIRE maintains an Agent Security Behavior Graph that models how untrusted content propagates through environments, tools, reasoning modules, and persistent states. Explore and Exploit experts use this graph to propose vulnerability hypotheses over consequences, injection methods, and environments, which are instantiated into executable tests and evaluated using trajectory evidence. The framework also uses cross-run memory to retain transferable search strategies. Experiments on AgentDojo and AgentDyn report improvements over payload-optimization baselines in attack success, vulnerability diversity, and attack-surface coverage.
Critical strengths
The paper addresses an important and timely problem: discovering indirect prompt-injection vulnerabilities in tool-using LLM agents.
ASPIRE takes a useful system-level perspective by modeling vulnerabilities as multi-step influence paths rather than isolated malicious prompts.
The Behavior Graph and Explore–Exploit design provide a structured way to explore, diagnose, and generalize vulnerabilities across tools, environments, and consequences.
The evaluation covers multiple benchmarks, agent architectures, attacker models, active defenses, and component ablations, while also considering vulnerability diversity, coverage, and cost in addition to attack success rate.
The detailed prompts and qualitative case studies improve the interpretability of the framework and illustrate practically relevant failure modes.
Critical weaknesses
1. The Relationship to Existing Fuzzing Paradigms Is Insufficiently Clarified
The paper presents ASPIRE as a paradigm shift from “attack payload optimization” to “open-ended vulnerability discovery.” From a broader security-testing perspective, however, ASPIRE closely resembles stateful, model-based, and grey-box workflow fuzzing:
- The Behavior Graph corresponds to a state or workflow model.
- The Explore–Exploit mechanism corresponds to frontier exploration and seed exploitation.
- The Realizer corresponds to a structured test-case generator or mutator.
- Execution trajectories serve as runtime feedback.
- functions as a domain-specific semantic coverage metric.
- Long-term memory resembles cross-campaign seed scheduling or a learned mutation policy.
ASPIRE therefore appears closer to an application of stateful, coverage-guided fuzzing to the semantic workflows of LLM agents than to an entirely new vulnerability-discovery paradigm. The paper should clarify whether its main contribution is a fundamentally new search paradigm or a new state representation, feedback signal, and test-generation mechanism specialized for agent security.
Compared with mature fuzzing systems, ASPIRE may also have several inherent limitations:
-
Its coverage feedback is less objective.
Traditional grey-box fuzzing relies on concrete branch, edge, or state-transition coverage. In ASPIRE, consequences, methods, environments, and the Behavior Graph are largely inferred and categorized by LLMs, making the coverage signal susceptible to synonymous duplicates, misclassification, and spurious coverage. -
Its testing throughput is likely much lower.
The system requires repeated calls to the Explore and Exploit experts, the Realizer, the Evaluator, the victim agent, and the memory module. The paper reports approximately 130K–680K tokens per discovered vulnerability, suggesting that the number of tests executable per unit of time may be substantially lower than that of conventional fuzzers. -
Its reproducibility and determinism are weaker.
LLM sampling, adaptive search, and cross-run memory can cause identical configurations to produce different results. Traditional fuzzing systems can generally save, deterministically replay, and minimize an input that triggers a failure. -
It lacks test-case minimization and root-cause deduplication.
The paper does not show how successful trigger queries, payloads, or interaction sequences are minimized. Nor does it establish that different triples correspond to distinct root causes. The same underlying vulnerability may be counted multiple times when expressed through different semantic wrappers. -
It may prune search paths prematurely.
After a small number of unsuccessful attempts, the Behavior Graph may label a path as blocked or saturated. Because LLM-based attacks have substantial variance, temporary failure does not establish that a path is unreachable. -
It may be less effective at discovering low-level boundary errors.
The Realizer primarily generates natural, business-native semantic content. It may therefore be less effective than conventional fuzzing at exploring malformed schemas, field-length limits, encoding discrepancies, parameter boundaries, unusual tool combinations, and protocol-state errors. -
It lacks mature scheduling, isolation, and replay mechanisms.
The paper does not sufficiently demonstrate systematic seed scheduling, state snapshots, environment resets, deterministic replay, vulnerability minimization, or root-cause bucketing.
Consequently, the current experiments show at most that ASPIRE outperforms prompt- or payload-mutation methods under fixed tasks. They do not establish that ASPIRE outperforms stateful or workflow fuzzing more generally. The evaluation should include a strong fuzzing baseline with access to the same tool descriptions, execution feedback, and computational budget. The comparison should report vulnerability-discovery curves, throughput, reproduction rates, the number of minimized and root-cause-deduplicated vulnerabilities, and cost efficiency.
2. The Experimental Protocol Is Not Comparable Across ASPIRE and the Baselines
The primary baselines optimize attack payloads under fixed user tasks, malicious objectives, and injection scenarios. In contrast, ASPIRE can jointly select or generate:
- the target consequence ;
- the injection method ;
- the injection environment ;
- the user trigger query ; and
- the attack payload .
ASPIRE can therefore select easier targets and workflows, whereas the baselines must optimize strings for a predetermined set of potentially difficult instances. The methods operate over different search spaces, instance-difficulty distributions, and ASR denominators. Consequently, the contrast in Table 1—near-zero ASR for the baselines and approximately 50% ASR for ASPIRE—does not directly demonstrate the effectiveness of the Behavior Graph or the Explore–Exploit mechanism.
A fairer evaluation should include at least the following:
- Fix the same instances and allow every method to optimize only .
- Allow the baselines to generate or modify and , as ASPIRE does.
- Give all methods the same static reconnaissance information, execution feedback, number of calls, and token budget.
- Report the cumulative number of unique vulnerabilities discovered under an equal budget, rather than comparing ASR over test cases generated by each method itself.
- Separately measure the contributions of objective selection, trigger-query generation, payload generation, and trajectory feedback.
3. The Evaluation Lacks Counterfactual Evidence That the Injection Caused the Consequence
The paper describes as a benign user query. However, the Realizer prompt permits or requires trigger queries that authorize high-impact operations such as placing orders, issuing refunds, inviting users, deleting resources, pushing changes, or transferring funds. In such cases, the malicious payload may only supply a particular account, recipient, or target object, while the high-impact action itself has already been authorized by the user query.
Checking whether tool arguments originate from the payload establishes correlation, but it does not demonstrate that the consequence would not have occurred without the injection. For each confirmed vulnerability, the evaluation should compare at least:
- the same with no injected content;
- the same with benign or semantically similar non-malicious content;
- the same with the malicious payload ; and
- where appropriate, counterfactual variants that swap payload entities or injection environments to verify that tool arguments are actually controlled by the injected content.
The paper should also formally define what counts as “benign,” “unauthorized,” and “outside the scope of the user’s authorization.” Without such definitions and controls, some reported successes may simply reflect the agent executing an overly broad user request that already authorizes a high-risk action.
4. The Evaluation Lacks Independent Repetitions and Statistical Uncertainty
ASPIRE is an adaptive search system composed of multiple stochastic LLM components, but the paper does not clearly report:
- the number of independent runs for each configuration;
- the number of successful tests and total tests in each run;
- temperatures, sampling parameters, and random seeds;
- the actual number of calls and token budget allocated to each method;
- means, standard deviations, or confidence intervals; or
- whether the reported ASR differences are statistically significant.
The tables report percentages to one decimal place without any uncertainty estimates. In particular, the 5–8 percentage-point improvements attributed to long-term memory and the cases in which a defense apparently increases ASR could result from run-to-run variance, changes in test selection, or adaptive budget reallocation.
The authors should conduct at least 3–5 independent runs and report confidence intervals, raw success and test counts, and vulnerability-discovery curves as a function of the testing budget.
5. The Long-Term Memory Evaluation May Suffer from Evaluation Leakage
The paper does not specify which agents, models, domains, or runs were used to construct long-term memory, nor whether memory was reset between experimental conditions. If LTM directly accumulates experience from previous tests on the same benchmark, domain, or target agent, Table 3 may measure continued optimization on the evaluation environment rather than cross-agent generalization.
The authors should clarify:
- how memory is initialized;
- the order in which experimental configurations are executed;
- whether memory is shared across models or ablation conditions;
- whether previous tests from the same benchmark are used to construct memory; and
- how memory transfers to entirely unseen agents, domains, and tool sets.
There is also an inconsistency between the algorithm and the textual description. The main text states that memory learns both productive and cautionary lessons from successes, failures, stalls, and turning points, whereas Algorithm 1 invokes UpdateAttackMemory only when status = Success. The paper should reconcile this discrepancy.
6. Vulnerability Counting, Deduplication, and Coverage Definitions Are Unclear
The paper treats every triple supported by at least one successful test as a unique vulnerability, but it does not explain:
- how semantically equivalent consequences are merged;
- whether the category sets are fixed in advance or dynamically expanded by an LLM;
- who performs categorization and deduplication;
- whether the results undergo human review or inter-annotator agreement analysis; or
- whether the same root cause triggered through different environments or payloads is counted multiple times.
Merely instructing the model not to generate synonymous consequences does not establish that the reported 96 triples represent 96 distinct vulnerabilities.
The reported coverage statistics also appear to use inconsistent counting conventions:
- Table 9 reports 27 total environments, whereas Figure 3 and Table 14 report up to 40 environments.
- Table 9 uses 61 as the denominator for access tools, whereas Figure 3 shows a maximum of 64.
- Table 9 uses 90 as the denominator for action tools, whereas Figure 3 shows a maximum of 60.
- Figure 3 shows a maximum of seven methods, whereas Table 14 reports six.
- The appendix states that more than 58 action tools are accessed under the defense settings, but Table 14 reports 52, 58, 52, and 50 action tools.
If these values correspond to different cross-domain deduplication levels or counting units, the paper should define those conventions explicitly and provide a complete mapping of categories, tools, and reported vulnerabilities.
Questions for authors
Since ASPIRE jointly selects or generates (C), (M), (E), (T), and (P), while the baselines mainly optimize (P) for fixed tasks, how are the ASR values in Table 1 directly comparable? Did all methods receive the same search space, execution feedback, number of trials, and token budget?
How many independent runs were conducted for each configuration? Could the authors provide the raw numbers of successful and total tests, together with variance or confidence intervals?
For confirmed vulnerabilities, did the authors run counterfactual controls using the same trigger query with no injection or with benign injected content? How is an “unauthorized” consequence distinguished from an action already permitted by a broadly phrased user request?
8 comments
Response 2
(continue)
- Why byte-style minimization and low-level boundary fuzzing are outside our threat model: Traditional fuzzers require input minimization to strip random mutation noise around a crash, and probe malformed schemas, field lengths, or encoding errors to find parser bugs. In contrast, indirect prompt injection targets semantic instruction–data confusion and authorization violations during legitimate tool workflows, not low-level software parsing bugs. Because ASPIRE synthesizes structured tests cleanly from an explicit hypothesis, each Vulnerability Report already isolates the exact triggering snippet, tool call, and state change (Section 3.3); aggressively stripping surrounding business context would destroy the naturalness required for realistic indirect injections.
W2 & Q1. We thank the reviewer for this constructive comment.
Table 1 had different user-task constraints because VeriGrey and AgentXploit, by design, only mutate payloads () on pre-given benchmark tasks using tool-sequence or outcome feedback, whereas ASPIRE reframes red-teaming to jointly discover user queries (), injection environments (), methods (), consequences (), and payloads () (detailed test and call budgets are given in W4 & Q2). Moreover, Table 1 already includes a domain-level Free-form attacker that receives the exact same tool schemas and environment slots as ASPIRE, freely selects , and receives full trajectory feedback after each sandboxed execution (for up to 10 rounds per domain vs. 8 rounds in ASPIRE), yet gets 0.0% ASR (clarified in Section 4.1, "Baselines", and Section 4.2). We evaluate this open-ended baseline directly rather than modifying VeriGrey or AgentXploit with custom and generators, which would alter those published methods into ad-hoc variants.
To further show that jointly searching for queries is not the primary reason for the performance gain, we have added a controlled experiment optimizing baselines on ASPIRE's successful tasks in Appendix B.7 (Table 17; briefly noted in Section 4.2). We uniformly sample 10 confirmed vulnerability tasks per domain discovered by ASPIRE across all 7 domains (70 verified-exploitable tasks in total). For every task, we hold ASPIRE's exact user query, injection environment, and target consequence strictly fixed, guaranteeing that every task is 100% reachable and workflow-aligned, and run VeriGrey (30 executions/task) and AgentXploit (up to 3 MCTS rounds/task) from scratch to optimize only the payload on Gemini-3.7-Flash (ReAct). Supplying ASPIRE's reachable contexts lifts VeriGrey and AgentXploit from 0.0%–0.7% to 28.6% (20/70) and 18.6% (13/70) ASR, yet both still fail on 71.4% (50/70) and 81.4% (57/70) of tasks where ASPIRE's realized payloads succeed on all 70/70 tasks. Note that this fixed-task setup evaluates VeriGrey and AgentXploit purely as local payload-search methods after giving them vulnerable user queries and entry environments already found by ASPIRE, which they cannot discover on their own.
Third, to isolate the contributions of the Behavior Graph and the Explore–Exploit mechanism within the same search space, our revised ablation studies in Section 4.3 (16-test per-round cap across 3 runs) evaluate all variants under identical open-ended search conditions: removing the Behavior Graph (w/o Graph) reduces ASR to 0.0%, and removing the dual experts (w/o Expert) lowers ASR from 52.4 ± 3.4% / 37.8 ± 2.6% to 37.2 ± 4.6% / 30.5 ± 3.8% (Section 4.3, Table 4; see also Explore-only at 29.6% ASR / 87 #(C, M, E) and Exploit-only at 32.5% ASR / 35 #(C, M, E) vs. Dual-experts at 46.1% ASR / 96 #(C, M, E) across all 7 domains in Section 4.3, Figure 4).
To further ensure that ASPIRE explores broadly rather than concentrating on a few easier targets, our protocol forbids re-testing any confirmed (consequence, environment) pair; accordingly, ASPIRE covers 92.6% (25 / 27) of all available action tools on AgentDojo (Appendix B.3, Table 11) with a balanced distribution across 43 consequences (normalized Shannon entropy 0.903, Table 2). Therefore, ASPIRE discovers 96 unique combinations (vs. near-zero for baselines), showing that the performance of ASPIRE does not rely on self-selected ASR denominators.
Finally, following the reviewer's helpful suggestion to evaluate metrics independent of self-generated ASR denominators, we report absolute discovery counts (96 unique combinations vs. near-zero for baselines) alongside cumulative discovery curves across LLM calls, tokens, and USD in Appendix B.2 (Figure 5). Together, these results indicate that the Behavior Graph and dual-expert design play a central role in ASPIRE's effectiveness.
Response 3
W3 & Q3. We thank the reviewer for this comment. In ASPIRE, the malicious consequence is strictly independent of the user query and exists solely in the injected payload.
First, ASPIRE enforces a hard information split in the Realizer (Appendix C.3, Prompt C.3), so that the user query () cannot mention any malicious goal, harmful target, or injection instruction. Specifically, initiates a normal user task without adversarial goals or target parameters, authorizing actions only on legitimate user-owned records; an unauthorized consequence occurs when causes the agent to invoke an unrequested sensitive tool or redirect the workflow onto out-of-scope targets originating from untrusted injection slot, as we have now defined explicitly in Section 3 ("Problem setup") and Appendix B.8. Thus, even when a user query uses action verbs or licenses a routine commit step over legitimate records, it only authorizes a benign user workflow, while the unauthorized target or operation comes solely from the injected payload. We manually inspect all confirmed cases on Gemini-3.7-Flash (ReAct), and 0.0% (0/269) of user queries contain contents related to malicious consequences or attacker parameters. Moreover, a vulnerability is confirmed only when rule-based checks verify that the sensitive arguments and unauthorized state change come from the untrusted payload rather than the user query. For example, in Case Study C.1 (Appendix C.4), the user only asks to check PR status, yet the agent registers an attacker's SSH public key found in an untrusted issue comment.
Second, to further verify that the malicious consequence comes solely from the injected payload rather than the user query, we have added paired counterfactual controls and an independent authorization audit across all confirmed cases in Appendix B.8 (Table 18):
- Paired counterfactual experiments (holding the exact user query fixed): Across all 269 confirmed cases on Gemini-3.7-Flash (ReAct; 182 on AgentDojo and 87 on AgentDyn), we fix the exact user query and target consequence, reset the sandbox, and test two controls alongside the original malicious payload ( trigger rate): (1) No Injection (running alone with slot empty in the clean environment), where the malicious consequence trigger rate drops to 0.0% (0/269); and (2) Benign Content Injection (replacing in slot with normal, format-matched non-malicious content), where the malicious consequence trigger rate also drops to 0.0% (0/269).
- Independent authorization audit: Manual inspection of the user query and full execution trajectory confirms that 100.0% (269/269) of confirmed cases exceed the legitimate scope authorized by alone.
W4 & Q2 We thank the reviewer for asking for these reporting details.
First, to answer Q2 directly, all experiments used default sampling (temperature = 1.0) under identical target-agent settings across methods, and we conducted 3 independent runs per configuration (Mean ± Std reported in Table 1 3 4 5 15). Because ASPIRE and the Free-form attacker search at the domain level whereas VeriGrey and AgentXploit must iterate over each predefined benchmark task individually, capping task-level baselines at ASPIRE's total test count would leave most benchmark tasks untested; we therefore allocate practical per-task iteration budgets covering all predefined benchmark tasks. Below is the budget comparison on Gemini-3.7-Flash (ReAct) across all 7 domains, added in Appendix B.2 ("Execution budgets and cumulative discovery curves", and Table 10):
| Method | Executed Tests (AgentDojo / AgentDyn / Total) | Total LLM Calls | ASR (AgentDojo / AgentDyn) |
|---|---|---|---|
| Free-form | 39 / 27 / 66 | ~0.5k | 0.0% / 0.0% |
| VeriGrey | 2,910 / 1,800 / 4,710 | ~32.5k | 0.7% / 0.0% |
| AgentXploit | ~2,820 / ~1,660 / ~4,480 | ~29.0k | 0.2% / 0.0% |
| ASPIRE | 344 / 233 / 577 | 4,983 | 52.4% / 37.8% |
On one hand, task-level baselines (VeriGrey and AgentXploit) execute 8x more tests (6x more LLM calls) by exhaustively iterating over fixed tasks, yet yield only 0.0%–0.7% ASR. On the other hand, the Free-form baseline generates far fewer tests (66 vs. 577) despite a 10-round budget; without a Behavior Graph to retain and branch from successful footholds, its search degenerates into disconnected ad-hoc probes.
Response 1
We sincerely thank the reviewer for the careful reviewing and insightful comments. We would like to address all comments respectively.
W1. We thank the reviewer for this insightful security-testing perspective.
First, relative to general software security testing, we fully agree that ASPIRE shares the high-level closed-loop spirit of stateful, feedback-guided fuzzing, and its core technical contribution lies in a new semantic state representation (the Agent Security Behavior Graph), multi-stage trajectory feedback signal, and decoupled test-generation mechanism specialized for LLM agent security. Meanwhile, relative to existing LLM agent prompt-injection research, which predominantly optimizes isolated payload strings (P) on predefined user tasks (T) and fixed slots (E), reframing red-teaming into open-ended joint discovery over full test tuples (C,M,E,T,P) represents a substantive shift for agentic vulnerability discovery. We have clarified this positioning explicitly in Section 2.
Second, regarding comparing against fuzzing baselines with the same tool schemas, feedback, and discovery metrics: Table 1 and Appendix B.2 (Figure 5 and Tables 8–10) already compare ASPIRE against two dedicated LLM-agent fuzzers, i.e. VeriGrey (grey-box fuzzing guided by tool-invocation sequences) and AgentXploit (black-box MCTS fuzzing), as well as an open-ended Free-form attacker and
w/o Graphablation that receive the exact same static reconnaissance, environment slots, and runtime trajectory feedback. Despite executing 8x more tests (6x more LLM calls) than ASPIRE, VeriGrey and AgentXploit stagnate at ≤0.7% ASR on Gemini-3.7-Flash (and reach only 28.6% / 18.6% even when given ASPIRE's verified-reachable (T,E,C) tasks in *Appendix B.7, Table 17. Unstructured open-ended exploration (Free-formandw/o Graph) also collapses to 0.0% ASR (W2 & Q1). We also report cumulative discovery curves of unique vulnerable (C,M,E) combinations against LLM calls, tokens, and USD in Figure 5, reproduction stability across 3 runs in Tables 1, 3–5, and 15, and root-cause-deduplicated consequence/tool counts in Table 11 16.Third, we clarify the specific design choices raised by the reviewer, distinguishing where ASPIRE already enforces deterministic guarantees from where agentic prompt injection intentionally diverges from traditional binary/API fuzzing:
W6). Every test executes from a clean sandbox reset, logs the complete tuple x=(C,M,E,T,P) and trajectory for exact replay, and exhibits low cross-run variance across 3 independent repetitions (W4 & Q2).untested,conditional,supported,blocked,saturated). When an attempt stalls at retrieval or execution commitment, it is markedconditionaland scheduled for targeted single-axis repair (refining T for retrieval or M/P for execution); a path is labeledblockedonly after repeated hard refusals (Section 3.3).(to continue)