acceptodds
Under review as a conference paper at ICLR 2027

ASPIRE: Agentic Safety and Prompt Injection Red-teaming Engine

Abstract

LLM agents retrieve untrusted content and act through tools, creating indirect prompt-injection risks that can cause unauthorized actions or persistent state changes. Existing automated red-teaming largely optimizes payloads for pre-specified scenarios, leaving latent vulnerabilities across the agent's behavior space unexplored. We present ASPRIE, an Agentic Safety & Prompt Injection Red-teaming Engine for open-ended, behavior-level vulnerability discovery. ASPIRE maintains an evolving Agent Security Behavior Graph and uses complementary Explore and Exploit experts to discover, verify, and generalize consequence-centric tests. Trajectory evidence updates the graph and diagnoses partial or failed attempts, while cross-run memory transfers useful search strategies. Experiments on AgentDojo and AgentDyn show that ASPIRE substantially expands coverage across consequences, injection methods, environments, and behavior paths while maintaining strong attack success.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Red team, prompt injection, agent safety, agent security

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7gXGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ASPIRE, an automated red-teaming framework for open-ended prompt-injection vulnerability discovery in LLM agents. ASPIRE maintains an Agent Security Behavior Graph that models how untrusted content propagates through environments, tools, reasoning modules, and persistent states. Explore and Exploit experts use this graph to propose vulnerability hypotheses over consequences, injection methods, and environments, which are instantiated into executable tests and evaluated using trajectory evidence. The framework also uses cross-run memory to retain transferable search strategies. Experiments on AgentDojo and AgentDyn report improvements over payload-optimization baselines in attack success, vulnerability diversity, and attack-surface coverage.

Critical strengths

The paper addresses an important and timely problem: discovering indirect prompt-injection vulnerabilities in tool-using LLM agents.

ASPIRE takes a useful system-level perspective by modeling vulnerabilities as multi-step influence paths rather than isolated malicious prompts.

The Behavior Graph and Explore–Exploit design provide a structured way to explore, diagnose, and generalize vulnerabilities across tools, environments, and consequences.

The evaluation covers multiple benchmarks, agent architectures, attacker models, active defenses, and component ablations, while also considering vulnerability diversity, coverage, and cost in addition to attack success rate.

The detailed prompts and qualitative case studies improve the interpretability of the framework and illustrate practically relevant failure modes.

Critical weaknesses

1. The Relationship to Existing Fuzzing Paradigms Is Insufficiently Clarified

The paper presents ASPIRE as a paradigm shift from “attack payload optimization” to “open-ended vulnerability discovery.” From a broader security-testing perspective, however, ASPIRE closely resembles stateful, model-based, and grey-box workflow fuzzing:

  • The Behavior Graph corresponds to a state or workflow model.
  • The Explore–Exploit mechanism corresponds to frontier exploration and seed exploitation.
  • The Realizer corresponds to a structured test-case generator or mutator.
  • Execution trajectories serve as runtime feedback.
  • (C,M,E)(C,M,E) functions as a domain-specific semantic coverage metric.
  • Long-term memory resembles cross-campaign seed scheduling or a learned mutation policy.

ASPIRE therefore appears closer to an application of stateful, coverage-guided fuzzing to the semantic workflows of LLM agents than to an entirely new vulnerability-discovery paradigm. The paper should clarify whether its main contribution is a fundamentally new search paradigm or a new state representation, feedback signal, and test-generation mechanism specialized for agent security.

Compared with mature fuzzing systems, ASPIRE may also have several inherent limitations:

  1. Its coverage feedback is less objective.
    Traditional grey-box fuzzing relies on concrete branch, edge, or state-transition coverage. In ASPIRE, consequences, methods, environments, and the Behavior Graph are largely inferred and categorized by LLMs, making the coverage signal susceptible to synonymous duplicates, misclassification, and spurious coverage.

  2. Its testing throughput is likely much lower.
    The system requires repeated calls to the Explore and Exploit experts, the Realizer, the Evaluator, the victim agent, and the memory module. The paper reports approximately 130K–680K tokens per discovered vulnerability, suggesting that the number of tests executable per unit of time may be substantially lower than that of conventional fuzzers.

  3. Its reproducibility and determinism are weaker.
    LLM sampling, adaptive search, and cross-run memory can cause identical configurations to produce different results. Traditional fuzzing systems can generally save, deterministically replay, and minimize an input that triggers a failure.

  4. It lacks test-case minimization and root-cause deduplication.
    The paper does not show how successful trigger queries, payloads, or interaction sequences are minimized. Nor does it establish that different (C,M,E)(C,M,E) triples correspond to distinct root causes. The same underlying vulnerability may be counted multiple times when expressed through different semantic wrappers.

  5. It may prune search paths prematurely.
    After a small number of unsuccessful attempts, the Behavior Graph may label a path as blocked or saturated. Because LLM-based attacks have substantial variance, temporary failure does not establish that a path is unreachable.

  6. It may be less effective at discovering low-level boundary errors.
    The Realizer primarily generates natural, business-native semantic content. It may therefore be less effective than conventional fuzzing at exploring malformed schemas, field-length limits, encoding discrepancies, parameter boundaries, unusual tool combinations, and protocol-state errors.

  7. It lacks mature scheduling, isolation, and replay mechanisms.
    The paper does not sufficiently demonstrate systematic seed scheduling, state snapshots, environment resets, deterministic replay, vulnerability minimization, or root-cause bucketing.

Consequently, the current experiments show at most that ASPIRE outperforms prompt- or payload-mutation methods under fixed tasks. They do not establish that ASPIRE outperforms stateful or workflow fuzzing more generally. The evaluation should include a strong fuzzing baseline with access to the same tool descriptions, execution feedback, and computational budget. The comparison should report vulnerability-discovery curves, throughput, reproduction rates, the number of minimized and root-cause-deduplicated vulnerabilities, and cost efficiency.

2. The Experimental Protocol Is Not Comparable Across ASPIRE and the Baselines

The primary baselines optimize attack payloads under fixed user tasks, malicious objectives, and injection scenarios. In contrast, ASPIRE can jointly select or generate:

  • the target consequence CC;
  • the injection method MM;
  • the injection environment EE;
  • the user trigger query TT; and
  • the attack payload PP.

ASPIRE can therefore select easier targets and workflows, whereas the baselines must optimize strings for a predetermined set of potentially difficult instances. The methods operate over different search spaces, instance-difficulty distributions, and ASR denominators. Consequently, the contrast in Table 1—near-zero ASR for the baselines and approximately 50% ASR for ASPIRE—does not directly demonstrate the effectiveness of the Behavior Graph or the Explore–Exploit mechanism.

A fairer evaluation should include at least the following:

  1. Fix the same (C,E,T)(C,E,T) instances and allow every method to optimize only PP.
  2. Allow the baselines to generate or modify C,E,T,C,E,T, and PP, as ASPIRE does.
  3. Give all methods the same static reconnaissance information, execution feedback, number of calls, and token budget.
  4. Report the cumulative number of unique vulnerabilities discovered under an equal budget, rather than comparing ASR over test cases generated by each method itself.
  5. Separately measure the contributions of objective selection, trigger-query generation, payload generation, and trajectory feedback.

3. The Evaluation Lacks Counterfactual Evidence That the Injection Caused the Consequence

The paper describes TT as a benign user query. However, the Realizer prompt permits or requires trigger queries that authorize high-impact operations such as placing orders, issuing refunds, inviting users, deleting resources, pushing changes, or transferring funds. In such cases, the malicious payload may only supply a particular account, recipient, or target object, while the high-impact action itself has already been authorized by the user query.

Checking whether tool arguments originate from the payload establishes correlation, but it does not demonstrate that the consequence would not have occurred without the injection. For each confirmed vulnerability, the evaluation should compare at least:

  1. the same TT with no injected content;
  2. the same TT with benign or semantically similar non-malicious content;
  3. the same TT with the malicious payload PP; and
  4. where appropriate, counterfactual variants that swap payload entities or injection environments to verify that tool arguments are actually controlled by the injected content.

The paper should also formally define what counts as “benign,” “unauthorized,” and “outside the scope of the user’s authorization.” Without such definitions and controls, some reported successes may simply reflect the agent executing an overly broad user request that already authorizes a high-risk action.

4. The Evaluation Lacks Independent Repetitions and Statistical Uncertainty

ASPIRE is an adaptive search system composed of multiple stochastic LLM components, but the paper does not clearly report:

  • the number of independent runs for each configuration;
  • the number of successful tests and total tests in each run;
  • temperatures, sampling parameters, and random seeds;
  • the actual number of calls and token budget allocated to each method;
  • means, standard deviations, or confidence intervals; or
  • whether the reported ASR differences are statistically significant.

The tables report percentages to one decimal place without any uncertainty estimates. In particular, the 5–8 percentage-point improvements attributed to long-term memory and the cases in which a defense apparently increases ASR could result from run-to-run variance, changes in test selection, or adaptive budget reallocation.

The authors should conduct at least 3–5 independent runs and report confidence intervals, raw success and test counts, and vulnerability-discovery curves as a function of the testing budget.

5. The Long-Term Memory Evaluation May Suffer from Evaluation Leakage

The paper does not specify which agents, models, domains, or runs were used to construct long-term memory, nor whether memory was reset between experimental conditions. If LTM directly accumulates experience from previous tests on the same benchmark, domain, or target agent, Table 3 may measure continued optimization on the evaluation environment rather than cross-agent generalization.

The authors should clarify:

  • how memory is initialized;
  • the order in which experimental configurations are executed;
  • whether memory is shared across models or ablation conditions;
  • whether previous tests from the same benchmark are used to construct memory; and
  • how memory transfers to entirely unseen agents, domains, and tool sets.

There is also an inconsistency between the algorithm and the textual description. The main text states that memory learns both productive and cautionary lessons from successes, failures, stalls, and turning points, whereas Algorithm 1 invokes UpdateAttackMemory only when status = Success. The paper should reconcile this discrepancy.

6. Vulnerability Counting, Deduplication, and Coverage Definitions Are Unclear

The paper treats every (C,M,E)(C,M,E) triple supported by at least one successful test as a unique vulnerability, but it does not explain:

  • how semantically equivalent consequences are merged;
  • whether the category sets are fixed in advance or dynamically expanded by an LLM;
  • who performs categorization and deduplication;
  • whether the results undergo human review or inter-annotator agreement analysis; or
  • whether the same root cause triggered through different environments or payloads is counted multiple times.

Merely instructing the model not to generate synonymous consequences does not establish that the reported 96 triples represent 96 distinct vulnerabilities.

The reported coverage statistics also appear to use inconsistent counting conventions:

  • Table 9 reports 27 total environments, whereas Figure 3 and Table 14 report up to 40 environments.
  • Table 9 uses 61 as the denominator for access tools, whereas Figure 3 shows a maximum of 64.
  • Table 9 uses 90 as the denominator for action tools, whereas Figure 3 shows a maximum of 60.
  • Figure 3 shows a maximum of seven methods, whereas Table 14 reports six.
  • The appendix states that more than 58 action tools are accessed under the defense settings, but Table 14 reports 52, 58, 52, and 50 action tools.

If these values correspond to different cross-domain deduplication levels or counting units, the paper should define those conventions explicitly and provide a complete mapping of categories, tools, and reported vulnerabilities.

Questions for authors

Since ASPIRE jointly selects or generates (C), (M), (E), (T), and (P), while the baselines mainly optimize (P) for fixed tasks, how are the ASR values in Table 1 directly comparable? Did all methods receive the same search space, execution feedback, number of trials, and token budget?

How many independent runs were conducted for each configuration? Could the authors provide the raw numbers of successful and total tests, together with variance or confidence intervals?

For confirmed vulnerabilities, did the authors run counterfactual controls using the same trigger query with no injection or with benign injected content? How is an “unauthorized” consequence distinguished from an action already permitted by a broadly phrased user request?

8 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the careful reviewing and insightful comments. We would like to address all comments respectively.

W1. We thank the reviewer for this insightful security-testing perspective.

First, relative to general software security testing, we fully agree that ASPIRE shares the high-level closed-loop spirit of stateful, feedback-guided fuzzing, and its core technical contribution lies in a new semantic state representation (the Agent Security Behavior Graph), multi-stage trajectory feedback signal, and decoupled test-generation mechanism specialized for LLM agent security. Meanwhile, relative to existing LLM agent prompt-injection research, which predominantly optimizes isolated payload strings (PP) on predefined user tasks (TT) and fixed slots (EE), reframing red-teaming into open-ended joint discovery over full test tuples (C,M,E,T,P)(C, M, E, T, P) represents a substantive shift for agentic vulnerability discovery. We have clarified this positioning explicitly in Section 2.

Second, regarding comparing against fuzzing baselines with the same tool schemas, feedback, and discovery metrics: Table 1 and Appendix B.2 (Figure 5 and Tables 8–10) already compare ASPIRE against two dedicated LLM-agent fuzzers, i.e. VeriGrey (grey-box fuzzing guided by tool-invocation sequences) and AgentXploit (black-box MCTS fuzzing), as well as an open-ended Free-form attacker and w/o Graph ablation that receive the exact same static reconnaissance, environment slots, and runtime trajectory feedback. Despite executing 8x more tests (6x more LLM calls) than ASPIRE, VeriGrey and AgentXploit stagnate at ≤0.7%\le 0.7\% ASR on Gemini-3.7-Flash (and reach only 28.6% / 18.6% even when given ASPIRE's verified-reachable (T,E,C)(T, E, C) tasks in *Appendix B.7, Table 17. Unstructured open-ended exploration (Free-form and w/o Graph) also collapses to 0.0%0.0\% ASR (W2 & Q1). We also report cumulative discovery curves of unique vulnerable (C,M,E)(C, M, E) combinations against LLM calls, tokens, and USD in Figure 5, reproduction stability across 3 runs in Tables 1, 3–5, and 15, and root-cause-deduplicated consequence/tool counts in Table 11 16.

Third, we clarify the specific design choices raised by the reviewer, distinguishing where ASPIRE already enforces deterministic guarantees from where agentic prompt injection intentionally diverges from traditional binary/API fuzzing:

  • Objective coverage, root-cause deduplication, and replay isolation: Behavior Graph nodes are constructed deterministically from static tool schemas rather than open-ended LLM guesses, and runtime edge transitions and vulnerability confirmation are gated by deterministic rules over executed tool calls and sandbox state mutations (Section 3.3), preventing spurious LLM-scored coverage. To avoid counting different semantic wrappers as separate root causes, we consolidate confirmed hits strictly by executed action tools and state mutations into 43 distinct consequences (100% blind human agreement) and 52 / 90 compromised action tools (W6). Every test executes from a clean sandbox reset, logs the complete tuple x=(C,M,E,T,P)x = (C, M, E, T, P) and trajectory for exact replay, and exhibits low cross-run variance across 3 independent repetitions (W4 & Q2).
  • Graded path scheduling against premature pruning: Rather than discarding a path after a single stochastic failure, the Behavior Graph tracks five graded edge statuses (untested, conditional, supported, blocked, saturated). When an attempt stalls at retrieval or execution commitment, it is marked conditional and scheduled for targeted single-axis repair (refining TT for retrieval or MM/PP for execution); a path is labeled blocked only after repeated hard refusals (Section 3.3).
  • Sample efficiency over raw execution throughput: Unlike CPU-bound binary fuzzing where millions of fast random mutations are viable, multi-turn agent evaluation is bottlenecked by LLM inference, making sample efficiency (vulnerabilities uncovered per LLM call) the governing metric. Guided by the Behavior Graph, ASPIRE discovers 96 unique (C,M,E)(C, M, E) combinations in just 577 tests (4,983 total LLM calls, USD 0.23–2.00 per combination; Tables 8–10), whereas high-volume mutation baselines (VeriGrey and AgentXploit) expend ~6x more calls with near-zero yield.

(to continue)

AuthorsOpenReview ↗

Response 2

(continue)

  • Why byte-style minimization and low-level boundary fuzzing are outside our threat model: Traditional fuzzers require input minimization to strip random mutation noise around a crash, and probe malformed schemas, field lengths, or encoding errors to find parser bugs. In contrast, indirect prompt injection targets semantic instruction–data confusion and authorization violations during legitimate tool workflows, not low-level software parsing bugs. Because ASPIRE synthesizes structured tests cleanly from an explicit (C,M,E)(C, M, E) hypothesis, each Vulnerability Report already isolates the exact triggering snippet, tool call, and state change (Section 3.3); aggressively stripping surrounding business context would destroy the naturalness required for realistic indirect injections.

W2 & Q1. We thank the reviewer for this constructive comment.

Table 1 had different user-task constraints because VeriGrey and AgentXploit, by design, only mutate payloads (PP) on pre-given benchmark tasks using tool-sequence or outcome feedback, whereas ASPIRE reframes red-teaming to jointly discover user queries (TT), injection environments (EE), methods (MM), consequences (CC), and payloads (PP) (detailed test and call budgets are given in W4 & Q2). Moreover, Table 1 already includes a domain-level Free-form attacker that receives the exact same tool schemas and environment slots as ASPIRE, freely selects (C,M,E,T,P)(C, M, E, T, P), and receives full trajectory feedback after each sandboxed execution (for up to 10 rounds per domain vs. 8 rounds in ASPIRE), yet gets 0.0% ASR (clarified in Section 4.1, "Baselines", and Section 4.2). We evaluate this open-ended baseline directly rather than modifying VeriGrey or AgentXploit with custom TT and EE generators, which would alter those published methods into ad-hoc variants.

To further show that jointly searching for queries is not the primary reason for the performance gain, we have added a controlled experiment optimizing baselines on ASPIRE's successful tasks in Appendix B.7 (Table 17; briefly noted in Section 4.2). We uniformly sample 10 confirmed vulnerability tasks per domain discovered by ASPIRE across all 7 domains (70 verified-exploitable tasks in total). For every task, we hold ASPIRE's exact user query, injection environment, and target consequence strictly fixed, guaranteeing that every task is 100% reachable and workflow-aligned, and run VeriGrey (30 executions/task) and AgentXploit (up to 3 MCTS rounds/task) from scratch to optimize only the payload on Gemini-3.7-Flash (ReAct). Supplying ASPIRE's reachable (T,E,C)(T, E, C) contexts lifts VeriGrey and AgentXploit from 0.0%–0.7% to 28.6% (20/70) and 18.6% (13/70) ASR, yet both still fail on 71.4% (50/70) and 81.4% (57/70) of tasks where ASPIRE's realized payloads succeed on all 70/70 tasks. Note that this fixed-task setup evaluates VeriGrey and AgentXploit purely as local payload-search methods after giving them vulnerable user queries and entry environments already found by ASPIRE, which they cannot discover on their own.

Third, to isolate the contributions of the Behavior Graph and the Explore–Exploit mechanism within the same (C,M,E,T,P)(C, M, E, T, P) search space, our revised ablation studies in Section 4.3 (16-test per-round cap across 3 runs) evaluate all variants under identical open-ended search conditions: removing the Behavior Graph (w/o Graph) reduces ASR to 0.0%, and removing the dual experts (w/o Expert) lowers ASR from 52.4 ± 3.4% / 37.8 ± 2.6% to 37.2 ± 4.6% / 30.5 ± 3.8% (Section 4.3, Table 4; see also Explore-only at 29.6% ASR / 87 #(C, M, E) and Exploit-only at 32.5% ASR / 35 #(C, M, E) vs. Dual-experts at 46.1% ASR / 96 #(C, M, E) across all 7 domains in Section 4.3, Figure 4).

To further ensure that ASPIRE explores broadly rather than concentrating on a few easier targets, our protocol forbids re-testing any confirmed (consequence, environment) pair; accordingly, ASPIRE covers 92.6% (25 / 27) of all available action tools on AgentDojo (Appendix B.3, Table 11) with a balanced distribution across 43 consequences (normalized Shannon entropy 0.903, Table 2). Therefore, ASPIRE discovers 96 unique (C,M,E)(C, M, E) combinations (vs. near-zero for baselines), showing that the performance of ASPIRE does not rely on self-selected ASR denominators.

Finally, following the reviewer's helpful suggestion to evaluate metrics independent of self-generated ASR denominators, we report absolute discovery counts (96 unique (C,M,E)(C, M, E) combinations vs. near-zero for baselines) alongside cumulative discovery curves across LLM calls, tokens, and USD in Appendix B.2 (Figure 5). Together, these results indicate that the Behavior Graph and dual-expert design play a central role in ASPIRE's effectiveness.

AuthorsOpenReview ↗

Response 3

W3 & Q3. We thank the reviewer for this comment. In ASPIRE, the malicious consequence is strictly independent of the user query and exists solely in the injected payload.

First, ASPIRE enforces a hard information split in the Realizer (Appendix C.3, Prompt C.3), so that the user query (TT) cannot mention any malicious goal, harmful target, or injection instruction. Specifically, TT initiates a normal user task without adversarial goals or target parameters, authorizing actions only on legitimate user-owned records; an unauthorized consequence CC occurs when PP causes the agent to invoke an unrequested sensitive tool or redirect the workflow onto out-of-scope targets originating from untrusted injection slot, as we have now defined explicitly in Section 3 ("Problem setup") and Appendix B.8. Thus, even when a user query uses action verbs or licenses a routine commit step over legitimate records, it only authorizes a benign user workflow, while the unauthorized target or operation comes solely from the injected payload. We manually inspect all N=269N = 269 confirmed cases on Gemini-3.7-Flash (ReAct), and 0.0% (0/269) of user queries contain contents related to malicious consequences or attacker parameters. Moreover, a vulnerability is confirmed only when rule-based checks verify that the sensitive arguments and unauthorized state change come from the untrusted payload rather than the user query. For example, in Case Study C.1 (Appendix C.4), the user only asks to check PR status, yet the agent registers an attacker's SSH public key found in an untrusted issue comment.

Second, to further verify that the malicious consequence comes solely from the injected payload rather than the user query, we have added paired counterfactual controls and an independent authorization audit across all N=269N = 269 confirmed cases in Appendix B.8 (Table 18):

  • Paired counterfactual experiments (holding the exact user query fixed): Across all 269 confirmed cases on Gemini-3.7-Flash (ReAct; 182 on AgentDojo and 87 on AgentDyn), we fix the exact user query and target consequence, reset the sandbox, and test two controls alongside the original malicious payload (100.0%100.0\% trigger rate): (1) No Injection (running TT alone with slot EE empty in the clean environment), where the malicious consequence trigger rate drops to 0.0% (0/269); and (2) Benign Content Injection (replacing PP in slot EE with normal, format-matched non-malicious content), where the malicious consequence trigger rate also drops to 0.0% (0/269).
  • Independent authorization audit: Manual inspection of the user query and full execution trajectory confirms that 100.0% (269/269) of confirmed cases exceed the legitimate scope authorized by TT alone.

W4 & Q2 We thank the reviewer for asking for these reporting details.

First, to answer Q2 directly, all experiments used default sampling (temperature = 1.0) under identical target-agent settings across methods, and we conducted 3 independent runs per configuration (Mean ± Std reported in Table 1 3 4 5 15). Because ASPIRE and the Free-form attacker search at the domain level whereas VeriGrey and AgentXploit must iterate over each predefined benchmark task individually, capping task-level baselines at ASPIRE's total test count would leave most benchmark tasks untested; we therefore allocate practical per-task iteration budgets covering all predefined benchmark tasks. Below is the budget comparison on Gemini-3.7-Flash (ReAct) across all 7 domains, added in Appendix B.2 ("Execution budgets and cumulative discovery curves", and Table 10):

MethodExecuted Tests (AgentDojo / AgentDyn / Total)Total LLM CallsASR (AgentDojo / AgentDyn)
Free-form39 / 27 / 66~0.5k0.0% / 0.0%
VeriGrey2,910 / 1,800 / 4,710~32.5k0.7% / 0.0%
AgentXploit~2,820 / ~1,660 / ~4,480~29.0k0.2% / 0.0%
ASPIRE344 / 233 / 5774,98352.4% / 37.8%

On one hand, task-level baselines (VeriGrey and AgentXploit) execute 8x more tests (6x more LLM calls) by exhaustively iterating over fixed tasks, yet yield only 0.0%–0.7% ASR. On the other hand, the Free-form baseline generates far fewer tests (66 vs. 577) despite a 10-round budget; without a Behavior Graph to retain and branch from successful footholds, its search degenerates into disconnected ad-hoc probes.

Reviewer_NUuKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces ASPIRE, which extends prompt-injection red-teaming for LLM agents from optimizing attack text under fixed objectives to open-ended vulnerability discovery. The system maintains an evidence-backed behavior graph, uses Explore and Exploit experts to select consequences, injection methods, and environments, and then generates user queries and malicious payloads. It validates outcomes through tool calls and state changes while accumulating cross-run strategy memory. The main experiments on AgentDojo and AgentDyn report attack success rates reaching 54.1%, alongside broad coverage of entry surfaces. Its value lies in organizing search, diagnosing failures, and producing auditable vulnerability maps. However, the main baselines lack equivalent control over user tasks, the boundary between legitimate authorization and malicious outcomes requires further validation, and vulnerability deduplication, budget matching, and independent repetitions are insufficiently established. These limitations weaken conclusions about the method’s advantages and the failure of existing defenses.

Critical strengths
  1. The research objective is closely aligned with developers' practical auditing needs.

    The system searches not only for attack text, but also for possible consequences, entry surfaces, and tool-mediated paths leading to sensitive operations. Compared with repeatedly optimizing payloads for a fixed malicious objective, this approach has greater potential to uncover risks that were not specified in advance.

    Separating the choice of which security hypothesis to test from the generation of concrete user queries and payloads also helps organize the search process. The resulting vulnerability reports and maps have potential value for developers investigating security failures. (pp. 3–6)

  2. The validation procedure emphasizes tool execution and external state changes, with clearly organized evidence.

    The paper requires a vulnerability to satisfy rule-based gates and have trajectory evidence linking the injection to the intended consequence before it is confirmed. The LLM evaluator primarily diagnoses bottlenecks and cannot override the outcome ceiling imposed by those gates.

    The evaluator is also instructed to inspect tool arguments, returned results, and environment changes, distinguishing payload retrieval, dangerous planning, tool execution, and realized consequences. This is more credible than relying solely on a model's judgment of whether an attack succeeded, and it allows failed trajectories to inform subsequent search. (pp. 5, 25–28)

  3. The system has clearly differentiated roles, with reasonably broad experiments and component analyses.

    Explore searches for unconfirmed consequences, while Exploit extends existing findings to new entry surfaces or methods. The prompts explicitly discourage repeatedly accumulating successes on already confirmed cases. Strategy memory stores search experience, such as switching surfaces, repairing retrieval failures, and grounding arguments in valid objects, rather than simply caching attack text.

    The experiments cover two environment collections, seven domains, two agent architectures, and multiple attacker and victim models. They also evaluate the behavior graph, expert guidance, Explore–Exploit collaboration, memory, and active defenses. These designs reflect substantial engineering effort toward a complete red-teaming system. (pp. 6–9, 14–18)

Critical weaknesses
  1. The main comparisons do not match control over user tasks, leaving the source of the large performance gap unresolved.

    Page 6 explicitly states that the main baselines optimize payloads under fixed user tasks, whereas ASPIRE can jointly select user queries, target consequences, and injected content. The methods therefore operate under different constraints.

    Consequently, the gap between baseline ASRs of at most approximately 0.7% and ASPIRE's much higher rates may partly reflect more readily triggered tasks, easier consequences, or more favorable environment combinations. It cannot yet be attributed directly to the behavior graph and dual-expert design.

    More closely related discovery-oriented systems, including SIRAJ [1], ChainFuzzer [2], and PI-Hunter [3], are discussed in related work but are not included in the main experiments. The comparison should first match the controllable variables, environmental information, and execution budgets, and should include an additional evaluation with fixed user tasks. (pp. 2–3, 6–7)

  2. Generated user requests can expand operational authorization, requiring stronger counterfactual validation of the claimed vulnerabilities.

    The appendix prompts explicitly require some user queries to use action-oriented language, such as executing, updating, or canceling. For consequences involving purchases, refunds, and repository writes, the query is further required to "license the commit."

    This design can help activate realistic business workflows, but it raises a central concern: observing a sensitive operation does not automatically establish an authorization violation. The evaluation must show that malicious arguments, targets, or out-of-scope operations originate from the injected content, rather than being authorized by the generated user request itself.

    The authors do instruct the evaluator to distinguish the user task from payload instructions, so the cases should not simply be dismissed as invalid. Nevertheless, the paper needs paired experiments that hold the user query fixed while removing or neutralizing the payload, together with independent audits of the legitimate authorization boundary. Rule-based gates and trajectory associations alone are also insufficient to establish the reports' claimed "causal root causes." (pp. 24–30)

  3. The definitions of a unique vulnerability and coverage may overstate the independence and completeness of the findings.

    The paper counts each successful distinct (C,M,E)(C,M,E) triple as a unique vulnerability, where the elements represent consequence, method, and environment. However, the same root cause may be counted multiple times when the method or entry surface changes, while distinct paths may be merged if they share the same triple.

    Consequence categories can also be generated during search, but semantic deduplication, category consolidation, and human agreement are insufficiently specified. Thus, "96 successful combinations" should be distinguished from "96 independent security flaws."

    Coverage primarily concerns the predefined entry surfaces. Table 9 reports environment coverage of 27/27, but overall action-tool coverage is only 52/90, or 57.8%; Table 14 then reports 40 environments in the no-defense setting. The granularity and aggregation rules need clarification to avoid interpreting entry-surface coverage as near-exhaustive coverage of the vulnerability space. (pp. 6, 13, 15–18)

  4. The ablations do not adequately isolate the contributions of the evolving graph, expert division, and memory.

    The collapse to zero ASR after removing the behavior graph does not by itself establish that graph topology is necessary. The expert prompts depend on graph-derived valid paths, compatible consequences, and candidate lists. If removing the graph also removes this information, the ablation may simply deprive the system of inputs required to construct valid tests.

    The paper should compare a flat table, a static graph, an evolving graph, and a trajectory summary containing equivalent information, while also matching the test quotas of the single- and dual-expert variants.

    The approximately seven-percentage-point gain from strategy memory also requires further analysis. The paper does not sufficiently specify which targets and runs supplied the memories, whether the test environments had already been explored, or how much historical search budget was used. Evaluation on unseen domains or models and a no-memory baseline with the same total budget are needed. (pp. 8, 18–23)

  5. Budgets, evaluation denominators, and statistical repetitions are insufficiently reported, making the efficiency claims difficult to verify.

    The paper states that each expert proposes up to eight tests per round for eight rounds, but does not adequately report the actual number of executed tests for each method, the treatment of invalid tests and infrastructure failures, or the baselines' iteration and model-call budgets.

    The main text and appendices also lack clear reporting of independent randomized repetitions, variability, and confidence intervals. For a stochastic, open-ended search process, reporting a single ASR and discovery count is insufficient to establish stability.

    The claim of no more than $2 per vulnerability further requires a complete cost breakdown, including reconnaissance, generation, evaluation, victim execution, failed attempts, and memory construction. Cumulative deduplicated-discovery curves against model calls, tokens, and monetary budgets would provide a more informative evaluation. (pp. 6, 8, 15)

  6. The conclusions about defense effectiveness and failure mechanisms are stronger than the evidence supports.

    ASPIRE can select new user tasks, consequences, and entry surfaces against each defense, so the ASRs across defense settings arise from different attack distributions. Comparing these rates alone does not establish how much each defense reduces risk against the same attacks.

    The paper should also specify defense versions, parameters, and deployment locations, and evaluate false rejections and utility on legitimate tasks. Ideally, it should report both replay on a fixed attack set and adaptive search under equal budgets.

    Moreover, the explanations that some defenses increase ASR through "attention anchoring" or "overriding native safety alignment" remain largely speculative and lack controlled experimental support. Generalizing from these observations to claims that existing defenses merely filter superficial variants and cannot address structural risks exceeds the current evidence. (pp. 9, 17–18)

  7. Minor Formatting Comment

    Finally, a minor formatting point: the manuscript's font appears to differ from that used in other submissions. The authors may wish to check whether it conforms to the conference template and requires any adjustment.

[1] Zhou, K., Elgohary, A., Iftekhar, A. S. M., & Saied, A. (2026). SIRAJ: Diverse and efficient red-teaming for LLM agents via distilled structured reasoning. In V. Demberg, K. Inui, & L. Marquez (Eds.), Findings of the Association for Computational Linguistics: EACL 2026 (pp. 3269–3292). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.findings-eacl.171

[2] Wu, J., Yao, Z., Nan, Y., & Zheng, Z. (2026). ChainFuzzer: Greybox fuzzing for workflow-level multi-tool vulnerabilities in LLM agents. arXiv. https://doi.org/10.48550/arXiv.2603.12614

[3] He, P., Miculicich, L., Sharma, V., Fox, A., Lee, G., Tang, J., Pfister, T., & Le, L. T. (2026). PI-Hunter: Automated red-teaming for exposing and localizing prompt injections. arXiv. https://doi.org/10.48550/arXiv.2606.12737

Questions for authors

Please refer to the listed weaknesses in previous section

7 comments
AuthorsOpenReview ↗

Response 1/n

We sincerely thank the reviewer for the careful reviewing and insightful comments. We would like to address all comments respectively.

W1. We thank the reviewer for the comment.

Table 1 had different user-task constraints because VeriGrey and AgentXploit, by design, only mutate payloads (PP) on pre-given tasks, whereas ASPIRE reframes red-teaming to jointly discover user queries (TT), injection environments (EE), methods (MM), consequences (CC), and payloads (PP).

To further show that jointly searching for queries is not the primary reason for the performance gain, we have added a controlled experiment optimizing baselines on ASPIRE's successful tasks in the revision, Appendix B.7 (Table 17; briefly noted in Section 4.2). We uniformly sample 10 confirmed vulnerability tasks per domain discovered by ASPIRE across all 7 domains (70 verified-exploitable tasks in total). For every task, we hold ASPIRE's exact user query (TT), injection environment (EE), and target consequence (CC) strictly fixed, guaranteeing that every task is 100% reachable and workflow-aligned, and run VeriGrey (30 executions/task) and AgentXploit (up to 3 MCTS rounds/task) from scratch to optimize only the payload (PP) on Gemini-3.7-Flash (ReAct). Even on these verified-reachable tasks (where ASPIRE's realized payloads succeed on all 70/70 tasks), VeriGrey reaches only 28.6% (20/70) ASR and AgentXploit reaches only 18.6% (13/70) ASR, failing on 71.4% and 81.4% of tasks without ASPIRE's structured method (MM) selection and surface-native realization.

Moreover, Table 1 in the paper includes a Free-form attacker that has the exact same control over (C,M,E,T,P)(C, M, E, T, P), the same tool schemas and environment slots, and up to 10 rounds per domain (vs. 8 rounds in ASPIRE), yet gets 0.0% ASR (clarified in Section 4.1, "Baselines"; Section 4.2,; and Appendix B.2, Table 10).

In addition, our revised quota-matched ablation studies (16-test per-round cap across 3 runs) isolate the contributions of ASPIRE's components within the exact same open-ended (C,M,E,T,P)(C, M, E, T, P) search space: removing the Behavior Graph (w/o Graph) reduces ASR to 0.0%, and removing the dual experts (w/o Expert) lowers ASR from 52.4 ± 3.4% / 37.8 ± 2.6% to 37.2 ± 4.6% / 30.5 ± 3.8% (Section 4.3, Table 4, Appendix B.5, Table 14). To further ensure that ASPIRE explores broadly rather than concentrating on a few easier targets, our protocol forbids re-testing any confirmed (consequence, environment) pair; accordingly, ASPIRE covers 92.6% (25 / 27) of all available action tools on AgentDojo (Appendix B.3, Table 11) with a balanced distribution across 43 consequences (normalized Shannon entropy 0.903, Table 2). All these results confirm that the Behavior Graph and dual-expert design, rather than task selection alone, drive the performance gain.

Regarding the related work mentioned, we would like to clarify that they target fundamentally different problem scopes and threat models, so they are not designed for our problem setup.

  • SIRAJ fine-tunes a distilled 8B attacker model to generate malicious or risky user instructions for general safety testing. It is not designed for indirect prompt injection, where the user request must stay benign and the attacker only controls external data slots.
  • ChainFuzzer analyzes tool implementation source code to trigger traditional program bugs (SQLi, CMDi, SSRF, SSTI).
  • PI-Hunter focuses on localizing where untrusted external text enters and diverts agent reasoning.

Beyond these scope differences, none of the works has publicly released code or model checkpoints that could be adapted to our benchmarks. We have clarified these scope boundaries in the revision, highlighted in Section 2.

AuthorsOpenReview ↗

Response 2/n

W2. We thank the reviewer for this important point.

In ASPIRE, the malicious consequence is strictly independent of the user query and exists solely in the injected payload. First, ASPIRE enforces a hard information split (as shown in Appendix C.3), so that the user query cannot mention any malicious goal, harmful target, or injection instruction. We manually check all confirmed cases on Gemini-3.7-Flash (ReAct), and 0% of user queries contain the contents related to malicious consequences or attacker parameters. Thus, even when a user query uses action verbs or licenses a routine commit step, it only authorizes a benign user workflow over legitimate user-owned records, while the malicious consequence resides solely in the injected payload.

Additionally, a vulnerability is confirmed only when rule-based checks verify that the sensitive arguments and unauthorized state change come from the untrusted payload rather than the user query. For example, in Case Study C.1 (Appendix C.4), the user only asks to check PR status, yet the agent registers an attacker's SSH public key found in an untrusted issue comment. To further verify that the malicious consequence comes solely from the injected payload rather than the user query, we add paired counterfactual controls and an independent authorization audit.

  • Paired counterfactual experiments (holding the exact user query fixed): Across all 269 confirmed cases on Gemini-3.7-Flash (ReAct; 182 on AgentDojo and 87 on AgentDyn), we hold the exact user query and target consequence fixed, reset the sandbox, and evaluate two counterfactual controls alongside the original malicious payload (100.0%100.0\% trigger rate): (1) No Injection (removing the payload so slot EE remains empty in the clean environment); and (2) Benign Content Injection (replacing PP in slot EE with normal, format-matched non-malicious content). We present the results as follows:
Test typeMalicious consequence Trigger rate
Original Malicious Injection100%
No Injection0%
Benign Content Injection0%
We also include this table in the revision, Appendix B.8 (Table 18).
  • Independent authorization audit: Manual inspection of the user query and full execution trajectory confirms that 100.0% of confirmed cases exceed the legitimate scope authorized by the user query alone. We also agree that rule-based checks and trajectory links do not prove formal causality, and we have renamed the attribution phrasing and report header to Automated Behavioral Failure Attributions in Section 3.3 ("Vulnerability report") and Appendix C.4 ("Vulnerability Report Example"). Thank you for the suggestion.
AuthorsOpenReview ↗

Response 3/n

W3. We thank the reviewer for the comment and we would like to clarify our metric definitions and coverage granularity.

ASPIRE reports metrics across three levels: individual components, (C, M, E) combinations, and full behavior paths. Together, these metrics capture both cross-surface exposure and component-level coverage.

First, we agree with the reviewer's distinction between "96 successful combinations" and "96 independent security flaws": #(C, M, E) = 96 measures verified vulnerable combinations across consequences, methods, and surfaces rather than 96 separate root-cause bugs (and we explicitly refer to them as unique vulnerable (C,M,E)(C, M, E) combinations in Section 4.1; Section 4.2,; Figure 5; and Table 16). We still track #(C, M, E) because prompt-injection guardrails are often surface- or mechanism-specific. For instance, sanitizing incoming emails does not protect Slack channel metadata (EE), and blocking imperative overrides does not stop structured-record implants (MM). This makes it useful for developers to see across which surfaces and framings a consequence remains reachable. Alongside #(C, M, E), we report single-dimension coverage and diversity metrics across consequences, methods, environments, and tools (Figure 3; Table 2; Appendix B.3, Table 11; and Appendix B.6, Table 16) where surface or method variants do not increase the count. Meanwhile, (C,M,E)(C, M, E) serves as a compact skeleton key for the experts; each test case's full multi-step tool sequence is preserved in the Behavior Graph and Vulnerability Report, so distinct tool chains sharing a (C,M,E)(C, M, E) triple are never lost and remain available for guidance.

Second, during search, an expert can propose a new consequence only when an attack hits an uncovered action tool or causes a new world-state change (Appendix C.3, Prompt C.1). During consolidation (Section 3.3, Paragraph 1), consequences are separated strictly by executed action tools and system state changes rather than wording. For example, in Vulnerability Map C.1 (Appendix C.4), hijacking a recurring scheduled payment, executing an immediate money transfer, and tampering with user profile info are kept separate because they modify distinct state variables through different action tools, whereas differently worded attacks producing the same state change are merged. Two blind human annotators independently audited all 43 consolidated consequences using only the executed tool calls and state changes, reaching 100% agreement.

Third, we clarify injection environments, injection slots, access tools, and action tools. Injection environments (27 / 27, 100.0% in Table 11) are external environments where untrusted data can exist (such as emails, calendars, file systems, and webpages), while injection slots (40 exploited slots in Figure 3, and Table 16) are the specific positions predefined by the AgentDojo and AgentDyn benchmarks inside those environments where a payload can be inserted. Access tools (60 / 61, 98.4%) are read tools that retrieve content from injection environments, whereas action tools (52 / 90, 57.8%) are write tools that mutate system states. Therefore, 27 / 27 means every upstream environment was penetrated at least once, not near-exhaustive coverage of the overall vulnerability space. We have revised our phrasing in Section 1 (Contribution 3) and Appendix B.3, and stated these granularity and aggregation rules explicitly in Section 4.2, Appendix B.3 and Appendix B.6 (Table 16).

Reviewer_sYjpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ASPIRE, an automated framework for discovering indirect prompt-injection vulnerabilities in tool-using LLM agents. ASPIRE searches over consequences, injection methods, and environments, while jointly constructing user triggers and injected payloads. An evolving Agent Security Behavior Graph guides complementary Explore and Exploit experts, and execution feedback informs subsequent tests and vulnerability reports. Cross-run memory stores reusable search strategies. The evaluation uses AgentDojo and AgentDyn environments across multiple attacker models, agent architectures, and defenses, measuring attack success, coverage, diversity, and cost. The authors report broader discovery coverage and higher attack success than the evaluated baselines, supported by component ablations and qualitative case studies.

Critical strengths
  1. The paper addresses an important limitation of existing prompt-injection evaluations: optimizing attacks against predefined tasks and malicious goals provides only a partial picture of an agent’s vulnerabilities. Searching across user workflows, injection surfaces, and downstream consequences is relevant to developers who need to assess an agent before deployment.
  2. The separation between vulnerability hypothesis generation and payload realization is well motivated. It allows the search to first decide which consequence and influence path to investigate, before constructing the corresponding attack. The use of intermediate execution feedback also provides a more informative basis for subsequent tests than a binary success or failure signal.
  3. The evaluation covers seven application domains, two agent architectures, several attacker backbones, and three defenses. The Explore-only and Exploit-only ablations are particularly useful: they show that achieving a relatively high attack success rate does not necessarily imply broad vulnerability coverage. These results support the paper’s decision to evaluate discovery breadth alongside attack effectiveness. 4.The vulnerability reports provide concrete evidence for inspecting the discovered failures. The SSH-key registration and Slack metadata examples connect attacker-controlled content to specific tool invocations and consequential state changes. Producing these execution records and diagnostic summaries could make the framework useful for investigating and addressing agent failures.
Critical weaknesses
  1. The main comparison does not adequately separate the contribution of the proposed search method from that of the expanded search space. Section 4.1 states that ASPIRE constructs both the user trigger and injected payload, whereas the published baselines attack fixed user tasks. Selecting the trigger can directly affect whether the payload is retrieved and whether the agent proceeds with the relevant workflow. This broader setting is reasonable for vulnerability discovery, but the resulting ASR gap does not establish that graph-guided search accounts for the improvement. It also provides insufficient support for the stronger conclusion that existing methods have become ineffective against recent models.
  2. The graph ablation does not clearly isolate what the graph contributes. The representation contains tool–environment mappings, candidate paths, and evidence from previous attempts, each of which can affect attack construction independently of graph topology. The paper reports that removing the graph reduces ASR to zero, but does not explain which information remains available to the attacker. Consequently, this experiment does not yet establish the claim that a topological model is a prerequisite for discovering valid multi-step attacks.
  3. The coverage metric does not clearly distinguish diverse exploit configurations from distinct behavioral vulnerabilities. Each successful consequence–method–environment triple is counted as a unique vulnerability, so changing the method or injection surface can increase the count without revealing a different underlying weakness. The consequence categories also require clearer boundaries. For example, the banking map lists both account lockout takeover and unauthorized password reset under the same password-update tool. These may represent different outcomes, but their distinction and consolidation criteria are not explained sufficiently to assess the significance of the reported 96 unique triples.
  4. The memory experiments do not sufficiently establish the claimed transfer of red-teaming strategies. Table 3 shows consistent improvements from using memory, but the paper does not specify which previous runs produced that memory or how those runs differ from the evaluation targets. The results therefore support the usefulness of accumulated experience, while leaving unclear whether the strategies transfer to unfamiliar agents and environments. This distinction matters because transferable cross-run memory is presented as a central contribution.
Questions for authors
  1. Does the Free-form attacker have the same ability to construct user triggers and select injection surfaces as ASPIRE? How are their execution budgets matched, and where do the Free-form attacker’s attempts typically fail?
  2. In the w/o Graph experiment, which reconnaissance results, candidate constraints, and previous execution feedback remain available? How is this information presented to the attacker after removing the graph? 3.What observable criteria distinguish potentially overlapping consequences, such as the two password-update outcomes in the banking map? How are such cases handled when counting unique vulnerabilities?
  3. Which runs produced the memory used in Table 3? Are the reported improvements measured on agents or domains that were absent from those memory-building runs?
4 comments
AuthorsOpenReview ↗

Response 1/n

We sincerely thank the reviewer for the careful reviewing and insightful comments. We would like to address all comments respectively.

W1&Q1.

We thank the reviewer for this insightful comment.

Below, we (1) clarify our contribution on the problem formulation and give additional evidence on why existing payload-only methods fall short on recent models; and (2) provide more details on the free-form attacker, together with where its attempts fail.

As mentioned in the main paper, existing methods (VeriGrey, AgentXploit) formulate agentic red-teaming narrowly around optimizing only the injected payload (PP) under a fixed user query (TT) and fixed injection environment (EE). Therefore, they naturally rely on a fixed set of predefined tasks, which is one reason why they fall short on recent models. Notably, they reach high success rates on older models (67.8% and 67.5% ITSR on Gemini-2.5-Flash in Appendix B.9, Table 19), while stagnating on recent ones (0.0%–0.7% ASR on Gemini-3.7-Flash in Table 1). Consequently, an important contribution of our work (Section 1) is revisiting agentic prompt-injection red-teaming and formulating it as an open-ended vulnerability discovery problem that jointly searches over (C,M,E,T,P)(C, M, E, T, P).

To further show that jointly searching for queries is not the primary reason for the performance gain, we have added a controlled experiment optimizing baselines on ASPIRE's successful tasks in the revision, Appendix B.7 (Table 17; briefly noted in Section 4.2). We uniformly sample 10 confirmed vulnerability tasks per domain discovered by ASPIRE across all 7 domains (70 verified-exploitable tasks in total). For every task, we hold ASPIRE's exact user query (TT), injection environment (EE), and target consequence (CC) strictly fixed, guaranteeing that every task is 100% reachable and workflow-aligned, and run VeriGrey (30 executions/task) and AgentXploit (up to 3 MCTS rounds/task) from scratch to optimize only the payload (PP) on Gemini-3.7-Flash (ReAct). Supplying ASPIRE's reachable (T,E,C)(T, E, C) contexts lifts VeriGrey and AgentXploit from 0.0%–0.7% to 28.6% (20/70) and 18.6% (13/70) ASR, confirming that ASPIRE identifies genuinely exploitable workflow entry points. At the same time, both baselines still fail on 71.4% (50/70) and 81.4% (57/70) of tasks where ASPIRE's realized payloads succeed on all 70/70 tasks, demonstrating that heuristic mutation and MCTS string search struggle without ASPIRE's structured method (MM) selection and surface-native realization.

We next provide more details on the Free-form attacker, which we have clarified in the revision, Section 4.1, Section 4.2, and Appendix B.2 (Table 10). In Table 1, our Free-form attacker receives the exact same tool schemas and environment slots as ASPIRE and freely constructs user queries, selects injection environments, chooses consequences and methods, and writes payloads. It runs for up to 10 rounds per domain (vs. 8 rounds in ASPIRE) under identical target-agent settings, with fresh sandbox resets per test and full trajectory feedback after each execution. The Free-form attacker achieves 0.0% ASR, showing the difficulty of finding vulnerabilities even with full control over user queries and injection surfaces. Inspecting its trajectories reveals two dominant failure stages:

(a) Free-form repeatedly writes blunt imperative overrides (e.g., [SYSTEM UPDATE] or [ADMIN INSTRUCTION] naming sensitive tools directly) and keeps rephrasing them on the same surface after refusal. Without structured state tracking or separation between high-level skeleton planning and surface-native realization, an unguided LLM defaults to obvious jailbreak phrasing that modern models reject.

(b) Even when retrieved without refusal, the victim simply summarizes the text and never calls an action tool. An unguided attacker either writes read-only queries (e.g., "read and summarize") that never activate an action workflow, or places directives in slots with no natural workflow path to the target tool.

AuthorsOpenReview ↗

Response 2/n

W2&Q2(a).

We thank the reviewer for the comment. We would like to provide more details on the w/o graph ablation study.

In w/o Graph, the static reconnaissance stage is identical to full ASPIRE, so the attacker receives the complete list of discovered injection environment slots and full descriptions of all victim access (read) and action (write) tools. It also keeps the exact same Explore and Exploit experts, the same evaluator, and full execution trajectory feedback. Instead of maintaining the Behavior Graph to track red-team states, w/o Graph presents available environments, tools, and recent trajectory feedback and bottleneck diagnoses as flat unlinked text blocks when proposing tests. Thus, all target information remains available, and w/o Graph isolates the removal of structured graph state tracking and path-level guidance. We have clarified these details in the revision, Section 4.3 ("Impact of Behavior Graph and Experts" and Table 4).

W3&Q2(b).

We thank the reviewer for raising this question.

We agree that changing the method MM or entry surface EE for the same consequence CC does not necessarily indicate a new downstream root cause, so #(C, M, E) = 96 should be interpreted as verified vulnerable combinations rather than 96 independent capability flaws (accordingly, we refer to #(C, M, E) as unique vulnerable (C,M,E)(C, M, E) combinations in Section 4.1; Section 4.2; Figure 5; and Table 16. We still track #(C, M, E) because prompt-injection guardrails are often surface- or mechanism-specific. For instance, sanitizing incoming emails does not protect Slack channel metadata (EE), and blocking imperative overrides does not stop structured-record implants (MM). This makes it useful for developers to see across which surfaces and framings a consequence remains reachable. At the same time, to measure underlying weaknesses without counting surface or method variants multiple times, we jointly report single-dimension coverage and diversity metrics alongside #(C, M, E):

  • Consequences: 43 unique consequences (Figure 3; Table 16, formerly Table 14), with a balanced distribution rather than clustering on a few easy targets (normalized Shannon entropy 0.903, Gini-Simpson index 0.736, Table 2).
  • Methods: 6–7 distinct injection methods (Figure 3; Table 16), with consistently high diversity across all attacker models (normalized Shannon entropy 0.765–0.794, Gini-Simpson index 0.663–0.680, Table 2).
  • Action & Access Tools: 52 / 90 (57.8%) compromised action tools overall (25 / 27 or 92.6% on AgentDojo; 27 / 63 or 42.9% on AgentDyn), and 60 / 61 (98.4%) access tools (Appendix B.3, Table 11; Figure 3; Table 16).
  • Environments: 27 / 27 (100.0%) external injection environments (Table 11), spanning 40 exploited injection slots predefined by the benchmarks (Figure 3; Table 16).

During search and consolidation, two consequences are counted separately only when they produce different system state changes through executed action tools, not when they merely use different wording, as clarified in Section 3.3 (Paragraph 1) and Appendix C.3 (Prompt C.1). Regarding the banking example raised by the reviewer:

  • unauthorized_password_reset only changes the account password (update_password), which remains recoverable through the user's existing contact profile.
  • complete_account_lockout_takeover chains both recovery-profile modification (update_user_info) and password modification (update_password) within a single trajectory from one payload, locking the user out of account recovery.

We count such multi-tool chains as distinct consequences (CC) because, in an agent workflow, triggering two coordinated writes from a single untrusted payload produces an irreversible compound state change and tests whether the agent executes back-to-back sensitive actions without stopping or asking for confirmation after the first write. Vulnerability Map C.1 originally listed both under update_password only because the visual tree grouped multi-tool chains under one primary tool; we have updated Vulnerability Map C.1 (Appendix C.4) to display all involved tools explicitly (update_user_info + update_password). Meanwhile, differently worded attacks that produce the same state change (e.g., two attacks that both only change the password) are always merged into one consequence before computing #C = 43 and #(C, M, E) = 96 (verified by two blind human annotators with 100% agreement). At the same time, to provide a strictly tool-level view alongside #C, we report compromised action tools (52 / 90, Table 11), where each write tool (update_user_info, update_password, etc.) is counted at most once regardless of whether it is triggered alone or in a multi-tool chain.

AuthorsOpenReview ↗

Response 3/n

W4&Q3

We thank the reviewer for the comment.

We would like to clarify that the strategy memory is not pre-built offline; instead, it starts empty and evolves continuously as red-teaming runs proceed across domains and models, consistent with the Section 3.4.

In Table 3, every run in the without memory baseline starts from an empty memory (with no sharing from with memory). In the default with memory setting (Tables 1 and 3), ASPIRE also starts from an empty memory at the beginning of the evaluation sequence and maintains a single persistent strategy memory across runs. After each 8-round domain run finishes, ASPIRE extracts high-level turning-point search lessons, such as switching surfaces after repeated refusal, repairing commitment framing, or binding live object IDs, from that run's logs and carries this updated memory forward to subsequent domains and attacker models. Memory items store only domain-agnostic search decisions, never specific payloads, tool names, environment slots, or consequence labels.

Because each (domain, attacker model) setup is evaluated before its own lessons are added to memory, improvements are always measured without repeated optimization on the same target setup. In particular, when moving from AgentDojo into unseen AgentDyn domains (e.g. DailyLife, GitHub, Shopping, which have entirely different tools and workflows), or when advancing across the four distinct attacker backbones in Table 3, that target setup is absent from memory during its run. Furthermore, retrieved lessons act only as a soft prior that local trajectory evidence in the Behavior Graph can weaken or override when a new agent behaves differently. Across 3 independent runs per setting, accumulated strategy memory yields consistent gains across all four models and both benchmarks (+5.6% to +8.5% average ASR gain in every row of revised Table 3), confirming reliable generalization across unfamiliar targets. We have explicitly documented this sequential cross-run protocol in Section 4.3 and Appendix C.2.

AuthorsOpenReview ↗

A friendly reminder

Dear Reviewer sYjp,

As the author-reviewer discussion period is coming to a close soon (Nov 18), we would like to gently follow up on our responses. Thank you again for your constructive feedback, and we spare no effort to address them. Please let us know if our clarifications and revisions have addressed your concerns. We are looking forward to hearing from you!

Best regard,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions (for All Reviewers & ACs)

We thank all reviewers for their constructive feedback. We have uploaded a revised manuscript (highlighted in blue) with new experiments, multi-run statistics, and clarifications:

1. Additional Experiments, Statistical Stability & Budget Analyses

  • Payload-Only Evaluation on 70 Fixed (T,E,C)(T, E, C) Tasks (Section 4.2; Appendix B.7, Table 17): Holding ASPIRE's verified-reachable (T,E,C)(T, E, C) fixed across 70 tasks, VeriGrey and AgentXploit reach only 28.6% and 18.6% ASR.

  • Counterfactual Authorization Controls (Appendix B.8, Table 18): Across all N=269N = 269 confirmed cases, holding TT fixed reduces the consequence trigger rate from 100.0% to 0.0% under both No Injection and Benign Content Injection (with 0.0% adversarial goals in TT upon manual audit).

  • Quota-Matched Core & Expert Ablations (Section 4.3, Figure 4; Table 4; Appendix B.5, Table 14): Under a matched 16-test per-round cap, w/o Graph drops to 0.0% ASR, w/o Expert drops to 37.2 ± 4.6% / 30.5 ± 3.8%, and Dual-experts (46.1% ASR, 96 combinations) outperforms Explore-only (29.6%, 87) and Exploit-only (32.5%, 35).

  • Multi-Run Statistical Stability (Table 1; Table 3; Tables 4–5; Tables 14–15): Expanded main comparisons, Strategy Memory ablations (+5.6% to +8.5% ASR gains), core ablations, and defense evaluations to 3 independent runs (Mean ± Std).

  • Stage-Wise Cost Breakdown, Call Budgets & Discovery Curves (Section 4.4; Appendix B.2, Figure 5 & Tables 8–10): Added a 6-stage token/USD breakdown (0.23−2.000.23-2.00 per combination, Table 9), cross-method test/call budgets (∼6×\sim6\times fewer LLM calls than task-level baselines, Table 10), and cumulative discovery curves (Figure 5).

2. Methodological, Metric & Presentation Clarifications

  • Scope & Relationship to Fuzzing (Section 2): Clarified threat-model differences with SIRAJ, MUZZLE, ChainFuzzer, and PI-Hunter, and positioned ASPIRE relative to stateful feedback-guided fuzzing.

  • Authorization Boundary & Consequence Deduplication (Sections 3 & 3.3; Section 4.1; Appendix C.4): Defined benign query TT and unauthorized consequence CC; detailed rule-based consequence consolidation by tool calls and state mutations; adopted unique vulnerable (C,M,E)(C, M, E) combinations.

  • Coverage Granularity & Figure 3 Normalization (Section 1; Section 4.2, Figure 3; Appendix B.3, Table 11; Appendix B.6, Table 16): Distinguished external injection environments (27) from benchmark slots (40) and unified the disjoint tool partition (61 access, 90 action tools) across Figure 3 and all tables.

  • Experimental Protocols & Formatting (Sections 4.1–4.4; Appendix B.6; Appendix C.1–C.2): Documented the Free-form and "w/o Graph" setups, LTM protocol, defense configurations and working hypotheses, fixing Algorithm 1, and standard ICLR times font.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.