acceptodds
Under review as a conference paper at ICLR 2027

From Search Success to Deployment Reliability: Rethinking Automated Red Teaming for Indirect Prompt Injection

Abstract

Automated red teaming for indirect prompt injection against LLM agents is typically evaluated by a simple question: does search find at least one successful payload within a query budget? Yet search ultimately delivers a single fixed payload, whose usefulness depends on whether it succeeds again. We show that these are fundamentally different quantities. In a controlled random-search study, 14–27% of candidates that succeed during search fail in all four independent holdout rollouts. More importantly, under stationary i.i.d. First-Hit search, increasing the budget improves the chance of finding a success, while pooled reliability can decrease. Motivated by this distinction, we separate attack coverage (whether search finds a successful payload) from committed-payload reliability (how reliably it succeeds again) and define deployment success as their product. Re-evaluating automated red-team methods with over 130K victim rollouts across two benchmarks, we find that search success can substantially misrepresent deployment performance: some automated methods fail to outperform a static template, and method rankings can reverse. These findings also change how the query budget should be spent. We introduce Search–Verify–Commit (SVC), a backend-agnostic protocol that trades off discovering new candidates against independently verifying existing ones before committing a single payload. Across extensive AgentDojo experiments, our StaticOpt and Adaptive instantiations improve deployment success over Search by 7.9 and 8.6 percentage points on average across backends and models. Our results suggest that automated red teaming should optimize the reliability of the attack it delivers, rather than search success alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Indirect Prompt Injection, Automated Red Teaming, LLM Agents, Agent Security, Deployment Reliability, Search-Verify-Commit

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PgGCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper separates two quantities that automated red-teaming for indirect prompt injection usually treats as one: whether search finds a payload that succeeds once (attack coverage, A) and how often that committed payload succeeds on a fresh rollout (conditional reliability, R). Their product D = AR is proposed as the deployment-relevant metric. Using random search on 128 AgentDojo task pairs with 100 candidates each and four holdout rollouts per candidate, the authors show that 14–27% of search-successful candidates fail every holdout, that pooled reliability falls as the search budget grows, and (Proposition 1) that this fall comes from task composition rather than from any within-task effect of budget. Re-evaluating Random, TAP, PISmith, and PIMiner on AgentDojo and AgentDyn with three victims and 20-rollout holdouts, they find that Random and TAP do not beat the static Important Instructions template on D and that method rankings can flip. They then propose Search–Verify–Commit (SVC), which spends part of the online query budget re-testing discovered candidates before committing one. At B = 100 on AgentDojo without defenses, these improve D over search-only commitment by 7.9 and 8.6 pp, and over a fixed five-repeat verification baseline by 1.6 and 2.3 pp.

Critical strengths
  1. The paper’s core distinction is established rather than asserted. The A/R/D decomposition is simple, the holdout protocol keeps selection and evaluation apart, and Proposition 1 together with the task-mixture identity explains a pattern that would otherwise look contradictory.

  2. The evaluation is careful in ways this subfield often is not. All SVC policies replay the same recorded search trace, so coverage is identical and the comparison isolates allocation and commitment.

Critical weaknesses
  1. The gain attributable to SVC’s design, as opposed to verifying at all, is small, and the headline numbers obscure this. The 7.9 and 8.6 pp improvements in the abstract are over SEARCH. The natural baseline is FIXED, which re-runs each candidate up to five times, and it captures most of the benefit: at B = 100, D goes 50.3 (SEARCH) → 56.6 (FIXED) → 58.2 (StaticOpt) → 58.9 (Adaptive), and FIXED6 reaches 56.9 (Table 20). Repeated evaluation before commitment is already practiced in the cited work (Hofer et al. average scores over target executions; Monteuuis et al. accept only unanimous repeats), so the part of SVC that is new is the allocation rule, and its measured benefit is 1.3–2.3 pp over FIXED, with Adaptive–StaticOpt at +0.7 pp (CI [−0.08, 1.42]). A protocol paper whose novel component contributes at this level does not clear the bar on its own; the evaluation framework is useful but is a re-statement of practices in Hofer et al. and Monteuuis et al. with cleaner bookkeeping.

  2. SVC is tested in a narrower setting than the paper’s own motivation. Section 3 motivates the problem with Spotlighting, but Experiment 3 runs only without defenses, only on AgentDojo, and only with the two open victims (GPT-5.4-mini is dropped). Reliability is defined relative to a fixed configuration with temperature 1.0 and thinking enabled on both victims; the paper does not show how much of the search-vs-holdout gap survives at low temperature, which is a common deployment setting and would change the value of verification.

Questions for authors
  1. Can you report the comparison against FIXED5 and FIXED6 with the same prominence as the comparison against SEARCH, and add a baseline that verifies all discovered candidates round-robin with no calibration or planning? That is the right null for the allocation rules.

  2. How do the search-vs-holdout gap (Figure 1a) and the SVC gains change at temperature 0 or with thinking disabled? If the gap largely closes, the paper’s practical scope should say so.

  3. Does the SVC gain hold under Spotlighting, where Section 3 shows lower reliability and sparser coverage? One backend × one model would be enough to answer this.

8 comments
AuthorsOpenReview ↗

Response to Reviewer PgGC (1 of 6)

We thank the reviewer for the careful reading and for the three concrete questions. We ran all requested experiments. In brief:

  • W1. We accept the reviewer's decomposition of the gains. The revised abstract and introduction report the total gains of fixed-schedule SVC policies alongside those of StaticOpt and Adaptive; Section 6.1 separately quantifies the additional gains from allocation. We also explain why verify-then-commit under a shared budget is not prior practice: FIXED is the simplest SVC policy, not an external baseline. At the end of Part 1 (1.4), we restate the paper's overall contribution, which runs from quantifying the reliability gap and re-evaluating existing methods to formalizing the budgeted selection problem and giving policies for it.
  • W2. We also ran SVC on AgentDyn: averaged over eight settings, StaticOpt and Adaptive improve D over SEARCH by 11.4 and 12.9 pp; Adaptive exceeds FIXED-MAX (FIXED6) and RR-MAX (RR-50) by 1.96 and 2.53 pp.
  • Q1. We added FIXED6 and three uncalibrated round-robin policies (RR). On AgentDojo at B = 100, these fixed-schedule policies (the verification schedule is fixed in advance, with no calibration or planning) account for most of the gain over SEARCH, yet the best policy in each of the eight backend–victim settings is StaticOpt or Adaptive.
  • Q2. At temperature 0 and with thinking disabled, the search-vs-holdout gap shrinks from 40.6 to 12.2 pp but does not close, and verification still adds 8–9 pp over SEARCH.
  • Q3. Under Spotlighting (PIMiner × Qwen3.6), Adaptive improves D by 19.5 pp over SEARCH and by 3.4–4.5 pp over fixed-depth verification.

The response has five parts, in the order W1, W2, Q1, Q2, Q3. The revised PDF has been uploaded; changes are in blue and listed at the end.

AuthorsOpenReview ↗

Response to Reviewer PgGC (2 of 6)

Part 1. W1: what is new in SVC, and how much each part contributes

On the SVC side, our contribution has three levels: we pose and formalize the problem of candidate discovery, verification, and final commitment under a shared budget, establish the SVC framework, and give concrete policies that solve it. The gains from the fixed-schedule policies and from the allocation rules come from an empirical comparison of different policies inside this framework. We first give the decomposition of the gains, and then explain the contribution of the problem and the framework themselves.

1.1 We accept the reviewer's decomposition and now report it that way

The 7.9 and 8.6 pp in the abstract are total gains over SEARCH, and most of that gap is closed by the fixed-schedule policies (FIXED and RR). The revised paper makes this comparison explicit for AgentDojo at B = 100, with gains averaged over the eight backend–victim settings:

  • the abstract and introduction report total gains over SEARCH side by side: up to +7.0 pp with fixed-schedule SVC policies, +7.9 pp with StaticOpt, and +8.6 pp with Adaptive;
  • Section 6.1 explicitly reports the additional gains from allocation: +0.9 pp (StaticOpt) and +1.6 pp (Adaptive) over the best fixed-schedule policy (RR-50).

The best fixed-depth and round-robin variants (FIXED6 and RR-50) now appear in the main results next to SEARCH, and the complete results for FIXED5, FIXED6, and RR-25/50/75 are in the new Appendix G.1.

1.2 What is new relative to Hofer et al. and Monteuuis et al.

Both works repeat executions, and our text should position our work relative to them more precisely. They use repetition to serve decisions different from the one SVC addresses.

  • Hofer et al. repeat executions in two places. Inside TAP, "reliability retries" are used to "average evaluator scores across multiple trials" to stabilize the judge signal that drives pruning and refinement; this is part of the search backend. After optimization, each resulting injection is evaluated several times; this measures the outcome. Neither step chooses among several environment-confirmed successful payloads, and neither is charged against a query budget shared with discovery.
  • Monteuuis et al. (CAS-gen) accept a jailbreak prompt only if all k_gen repeated trials succeed, and stop at the first prompt that passes. This is a unanimity filter: a candidate with one failure is rejected, nothing is returned if no candidate passes, and no evidence is used to rank candidates.

Existing methods optimize search success, not the reliability of the delivered payload. To our knowledge, no automated IPI red-teaming method takes the fresh-execution success probability of the payload it finally commits as the objective of its search or optimization. The four methods we evaluate all stop and commit at the first observed success, and the repeated executions in the two works above do not target it either. The paper's results show that this surrogate objective is not aligned with deployment success: under first-hit commitment, a larger search budget does not improve the reliability of the committed payload (Proposition 1), and method rankings by search success reverse (Table 1).

We formalize this decision as a terminal selection problem under a shared budget: with B queries in total and one payload to deliver, how should candidate discovery, reliability verification, and final commitment be decided jointly to maximize deployment success D? Definition 2 specifies the three actions (Search, Verify, and Commit), the shared budget constraint, and the evidence available to a policy; Eq. (6) defines the objective. Inside this framework, FIXED and RR are simple policies, while StaticOpt and Adaptive improve the allocation through cross-task calibration and current-task evidence, respectively. Concretely, under our verification and commitment policies:

  • verification is paid for from the same budget as search;
  • a candidate that fails a verification stays eligible and is ranked by posterior mean;
  • a payload is committed whenever one exists, so coverage is unchanged while reliability improves.

FIXED and RR are therefore not implementations of prior practice. They are the simplest policies inside Definition 2. We described FIXED as a baseline "motivated by" Monteuuis et al., which invited the reading that verify-then-commit under a shared budget already existed. The revision corrects this and presents FIRST-HIT, FIXED, RR, StaticOpt, and Adaptive as one family of SVC policies (Section 5.1). Read this way, the +6.3 to +7.0 pp is the measured value of the protocol, and the allocation rules add a smaller, backend-dependent increment on top. We do not rest the paper on the second increment alone.

AuthorsOpenReview ↗

Response to Reviewer PgGC (3 of 6)

1.3 What the allocation rules add on top

The reviewer is right that this second increment is modest on average: +1.6 and +2.3 pp over FIXED5, and +0.9 and +1.6 pp over the best round-robin split. Its value is robustness across backends. Part 3 gives the evidence:

  • no single simple rule is right everywhere: the best round-robin split differs by backend, and fixed-depth verification gains nothing on TAP × Qwen3.6;
  • in all eight AgentDojo settings, the best policy is StaticOpt or Adaptive;
  • the margin is largest where candidates are plentiful: Adaptive is 4.3–4.4 pp above FIXED5 on PIMiner with both victims.

1.4 On the evaluation framework

Repeated holdout evaluation already appears in some prior work, which we cite and discuss. The paper adds three things beyond it:

  1. the A/R/D decomposition, tied to an explicit commitment rule and query budget;
  2. Proposition 1 and the task-mixture identity, which show that a larger budget raises coverage without improving the reliability of the first-hit payload, and that pooled reliability can fall through task composition;
  3. the empirical consequence under one protocol across four methods, two benchmarks, and three victims: Random and TAP reach 43–82% coverage on AgentDojo with the open-weight victims, yet do not exceed a static template in D, and method rankings reverse.

To our knowledge, neither work reports these. The revised related-work paragraph states the distinction from both works explicitly.

More broadly, we see the paper's contribution as one line of argument rather than an evaluation framework plus an allocation rule. We quantify how often a search success fails to repeat and show that more search does not fix this (Proposition 1). We then propose A/R/D as a more complete set of evaluation metrics. Re-evaluating existing red-team methods of different types under these metrics shows that they optimize a surrogate that is misaligned with deployment success: rankings can reverse, and some automated methods do not outperform a static template. Finally, we formalize the resulting budgeted selection problem and give concrete policies for it.


Part 2. W2: scope — benchmarks and victims

AgentDyn. We ran the full SVC comparison on the 36 AgentDyn task pairs with all four backends and both open-weight victims (no defense, B = 100). The per-setting results are in Appendix G.5.

D (%) averaged over the eight backend–victim settings, and paired differences (pp, task-paired 95% bootstrap intervals):

PolicyDStaticOpt − policyAdaptive − policy
SEARCH52.47+11.35 [8.98, 13.68]+12.92 [10.35, 15.45]
FIXED563.16+0.66 [−0.82, 2.14]+2.22 [0.95, 3.58]
FIXED663.42+0.40 [−0.99, 1.77]+1.96 [0.97, 2.97]
RR-2562.55+1.27 [0.35, 2.26]+2.83 [1.28, 4.58]
RR-5062.85+0.97 [−0.02, 1.96]+2.53 [1.39, 3.70]
RR-7561.77+2.05 [0.49, 3.66]+3.61 [2.33, 4.95]
StaticOpt63.82—+1.56 [0.36, 2.81]
Adaptive65.38——
  • The gain holds on a second benchmark. Verification before commitment improves D in all eight settings, by 11.4 pp (StaticOpt) and 12.9 pp (Adaptive) on average.
  • Adaptive is the best policy in six of the eight settings. On average it exceeds every fixed-schedule policy by 2.0–3.6 pp and StaticOpt by 1.6 pp, with all intervals excluding zero.
  • StaticOpt is on par with the best fixed-schedule policies (FIXED6 and RR-50); those intervals include zero.

GPT-5.4-mini. We did not run SVC on GPT-5.4-mini because of its cost. Section 8 now states that all SVC experiments use the two open-weight victims.

Reviewer_TCEvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether a successful search-time indirect prompt injection (IPI) payload is reliable when replayed after search. It separates attack coverage A(B), the probability that a method finds a payload to commit, from conditional committed-payload reliability R(B), and defines deployment success as D(B)=A(B)R(B). Under stationary i.i.d. FIRST-HIT search, it proves that within-task first-hit reliability is invariant to the budget, while pooled reliability can change because the mix of tasks with a hit changes. The paper then evaluates Random, TAP, PISmith, and PIMiner across AgentDojo and AgentDyn, using Gemma4, Qwen3.6, and GPT-5.4-mini, and reports that search-centric rankings can reverse under deployment evaluation. Finally, it proposes Search–Verify–Commit (SVC), with StaticOpt and Adaptive instantiations that allocate a shared online victim-rollout budget between candidate discovery and verification. On AgentDojo without defenses, averaged equally over eight backend–model settings at B=100, deployment success is 50.28% for Search, 56.58% for Fixed, 58.20% for StaticOpt, and 58.87% for Adaptive. The reported gains over Search are 7.92 and 8.59 percentage points for StaticOpt and Adaptive, respectively.

Critical strengths
  1. The paper identifies a meaningful evaluation mismatch: observing one successful payload during search is not equivalent to delivering a payload that succeeds reliably on fresh executions. The A/R/D decomposition is clear and operationally interpretable.
  2. The empirical analysis is substantial. The paper reports more than 130K victim rollouts across two benchmarks and shows a concrete ranking reversal: on AgentDojo/Qwen3.6, TAP has higher coverage than Random (82.0% vs. 80.5%) but lower deployment success (30.9% vs. 39.9%).
  3. The evaluation separates search/selection observations from fresh holdouts. The SVC experiments use a shared online victim-rollout budget and compare against both a Search policy and a fixed repeated-verification baseline. Task-paired uncertainty intervals and extensive appendices improve transparency.
  4. The authors discuss important limitations, including fixed execution configurations, lack of defended SVC experiments, and PISmith training/evaluation overlap. The manuscript also provides detailed prompts/configurations and an anonymous code URL, which supports reproducibility.
Critical weaknesses

Potential training–evaluation overlap in the SVC experiments. The paper explicitly excludes the 32 Workspace evaluation task pairs from its Section 4 comparison to address overlap with PISmith's training data, but Section 6 appears to evaluate SVC on the full set of 128 AgentDojo task pairs, including the PISmith backend. It is therefore unclear whether the same overlap is present in the SVC results. Since candidate availability and reliability heterogeneity directly affect the gains from verification and selection, this issue could influence the aggregate results. Reporting the SVC comparisons on the 96 non-Workspace tasks would help establish whether the main conclusions are robust to this exclusion.

Questions for authors

Section 4 excludes the 32 Workspace evaluation task pairs to address PISmith training–evaluation overlap, whereas the SVC experiments appear to use all 128 AgentDojo task pairs. Could the authors clarify whether the Workspace pairs are included in the SVC results and, if so, report the main policy comparisons on the 96 non-Workspace tasks? Would the principal conclusions remain unchanged after this exclusion?

4 comments
AuthorsOpenReview ↗

Response to Reviewer TCEv (1 of 2)

We sincerely thank the reviewer for the thoughtful and encouraging review. We are very glad that the A/R/D decomposition, the separation of selection from fresh holdouts, and the transparency of the evaluation came across as intended. We also appreciate the careful reading behind this question: it points to a robustness check that belongs in the paper, and we have now run it.

Short answer. Yes, the SVC results include the 32 Workspace pairs. On the 96 non-Workspace pairs, all main conclusions hold and the gains are essentially unchanged: StaticOpt and Adaptive improve D over Search by +8.22 and +8.89 pp, compared with +7.92 and +8.59 pp on all 128 pairs.

Are the Workspace pairs included in the SVC results? Yes. Section 6 uses all 128 AgentDojo task pairs for every backend, including the 32 Workspace pairs. One small clarification: the main comparison in Section 4 also uses all 128 pairs, and the Workspace-excluded results are the sensitivity analysis in Appendix D.3. The submission did not extend that analysis to the SVC experiments. We fully agree that it should have, and we report it below.


SVC on the 96 non-Workspace pairs

Protocol. We follow Appendix D.3. For every backend and victim, we exclude the 32 Workspace pairs and keep the same 96 pairs (32 Banking, 24 Slack, 40 Travel). All metrics are recomputed from the recorded per-task results. The Search traces, Policy-Verify outcomes, committed payloads, and the 20 Final-holdout rollouts are unchanged, and no new victim rollouts were run. Intervals are task-paired 95% percentile bootstrap intervals.

Deployment success D (%) at B = 100, averaged equally over the eight backend–model settings:

PolicyAll 128 pairs96 non-Workspace pairs
Search50.2849.85
Fixed56.5856.30
StaticOpt58.2058.07
Adaptive58.8758.74

Paired differences in D at B = 100 (pp):

ComparisonAll 128 pairs96 pairs95% interval (96 pairs)
StaticOpt − Search+7.92+8.22[6.74, 9.73]
Adaptive − Search+8.59+8.89[7.25, 10.52]
StaticOpt − Fixed+1.62+1.78[0.76, 2.83]
Adaptive − Fixed+2.29+2.44[1.49, 3.42]

Do the principal conclusions remain unchanged? Yes.

  • Verifying before committing still improves deployment success. At B = 100, StaticOpt and Adaptive gain +8.22 and +8.89 pp over Search. The gains are also positive at smaller budgets: +2.70 and +3.24 pp at B = 20, and +7.02 and +7.36 pp at B = 50. All of these intervals exclude zero.
  • The advantage over fixed repeated verification is preserved. At B = 100 the margins are +1.78 and +2.44 pp, compared with +1.62 and +2.29 pp on all 128 pairs, and both intervals exclude zero.
  • Coverage is essentially identical: 80.47% on the 96 pairs and 80.37% on all 128.

PISmith specifically. Because the overlap concerns only the PISmith backend, we also examined it in isolation.

  • The reviewer's intuition is right about PISmith's absolute numbers. PISmith's Search D is 72.19% on the 32 Workspace pairs and 57.73% on the other 96. The other six settings show no such pattern (44.71% and 47.22%). This is consistent with PISmith benefiting from the overlap on Workspace, although we cannot separate it from differences between task domains.
  • The SVC gains with PISmith do not depend on Workspace. On the 96 pairs, both policies still improve over Search in the two PISmith settings at B = 100: +2.89 pp [1.22, 4.64] for StaticOpt and +2.68 pp [1.04, 4.40] for Adaptive.
  • The overall conclusions do not depend on PISmith at all. On the six settings of the other three backends, restricted to the same 96 pairs, at B = 100 StaticOpt and Adaptive gain +10.00 pp [8.16, 11.88] and +10.95 pp [8.88, 13.04] over Search, and +1.91 pp [0.74, 3.12] and +2.86 pp [1.79, 3.97] over Fixed.
AuthorsOpenReview ↗

Response to Reviewer TCEv (2 of 2)

Additional validation on a second benchmark

We also ran the SVC experiments on AgentDyn, which does not contain the Workspace suite used to train PISmith. Over the eight settings (36 task pairs, four backends, and two victims) at B = 100, StaticOpt and Adaptive improve mean deployment success over Search by 11.35 pp (95% CI [8.98, 13.68]) and 12.92 pp ([10.35, 15.45]). In the two PISmith settings alone, StaticOpt and Adaptive also improve over Search: by 4.58 and 4.31 pp on Qwen3.6, and by 8.33 and 9.44 pp on Gemma4. This further indicates that the gains of SVC are not specific to the AgentDojo evaluation set that includes Workspace. Complete results are in the new Appendix G.5.

Revision. The revised PDF adds Appendix G.2 with the complete 96-pair results for all three budgets and every backend–model setting. Section 6 references it and now states explicitly that the main SVC results include the Workspace pairs. The new Appendix G.5 reports the additional AgentDyn experiments, and Section 8 adds a corresponding statement.

Thank you again for the constructive suggestion, which makes the SVC results more convincing. We would be happy to provide any further breakdown.

Thank you for the detailed response and additional experiments.

The results on the 96 non-Workspace task pairs directly address my main concern. In particular, the improvements of StaticOpt and Adaptive over both Search and Fixed remain positive after excluding the potentially overlapping tasks. The additional AgentDyn experiments further strengthen the empirical evidence.

Based on the reported analyses, I consider my concern about training–evaluation overlap satisfactorily addressed. I appreciate the authors' efforts and maintain my recommendation.

AuthorsOpenReview ↗

Response to Reviewer TCEv’s Follow-up

Thank you for reading our response and for confirming that it has addressed your concern about training–evaluation overlap. We appreciate your constructive review and your continued support of the paper.

Reviewer_bcDxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies evaluating automated prompt injection methods by how reliably the selected attack succeeds on fresh executions, rather than whether search finds at least one successful attack within a budget. It shows that this distinction affects method rankings in some settings, and proposes allocating the query budget between finding attacks and repeatedly testing them.

Critical strengths
  • The reliability of discovered attacks is an interesting thing to measure, with potential practical relevance when the goal is to produce a reusable attack.
  • The experiments show that measuring this quantity can change the relative performance of some existing methods.
Critical weaknesses
  • The main distinction between finding a successful attack within a query budget and having that attack succeed again is fairly elementary. Best-of-k attack success rate is a standard metric that answers a legitimate question, and it is not clear that measuring a different quantity constitutes a significant methodological contribution.
  • Showing that this change affects rankings in some cases is interesting, but it is not clear that this would substantially improve the development of attack or defense methods. The practical value of the proposed evaluation seems insufficiently established.
  • The paper also proposes search-verification allocation methods that improve performance on the proposed metric, but the improvement over a simple repeated-testing baseline is relatively modest. Overall, this seems like an interesting experiment, but the significance of the contribution does not seem sufficient for ICLR.
Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bcDx (1 of 3)

We thank the reviewer for the time spent on our paper. The review raises no concern about soundness or presentation (both rated 3), so the disagreement is entirely about significance. On this question, we respect the reviewer's judgment but firmly disagree with the three points raised. Below, we respond to each point with evidence from the paper and additional experiments conducted during the rebuttal, and we hope the reviewer will reconsider the assessment in light of these results.


W1. The distinction is fairly elementary; best-of-k is a standard metric

We do not replace best-of-k. We show what it leaves out. Attack coverage A(B) in Definition 1 is exactly best-of-B success. We agree it answers a legitimate question, and we recommend that it continue to be reported. Our claim is that it answers a different question from the one that matters once search ends. In indirect prompt injection, the attacker plants one fixed payload in external content, and the victim's agent then executes it, typically with no feedback to the attacker and no chance to retry. Best-of-k describes the search phase. It does not describe the artifact that is delivered.

Being easy to state does not mean the distinction is accounted for in practice. All four automated methods we evaluate stop at the first observed success and commit that payload. If the field had already absorbed the distinction, the following would not be true:

  • 14–27% of search-successful payloads fail all four independent re-executions; overall, search-successful payloads succeed in only 43–62% of re-executions (Figure 1a).
  • TAP on Qwen3.6 reaches 82.0% best-of-100 success, yet its deployment success is 30.9% (Table 1).
  • Random and TAP reach 43–82% best-of-100 success on AgentDojo with Gemma4 and Qwen3.6, yet neither exceeds, in deployment success, a static template that uses no search at all. On Qwen3.6, TAP is 10.7 pp below the template (95% CI [−17.9, −3.4]).

The contribution is not the observation. It is three results that are not elementary.

  1. Proposition 1. Under stationary i.i.d. first-hit search, a larger budget does not improve the reliability of the committed payload at all. This reliability is exactly budget-invariant. More search buys coverage and nothing else.
  2. The task-mixture identity. Pooled reliability falls as the budget grows (from 0.60 to 0.45 on Gemma4; Δ = −0.14, 95% CI [−0.20, −0.07]) although no task's reliability changes. Without the decomposition, this reads as "more search yields worse payloads", which is the wrong conclusion.
  3. Reliability is not a rescaling of coverage. At B = 100, committed-payload reliability ranges from 36% to 82% across methods, victims, and benchmarks (Figure 2). Deployment success therefore cannot be inferred from best-of-k, and method rankings reverse.

The gap is not an artifact of sampling temperature. In a new experiment (full results in Appendix G.3 of the revised PDF), we repeat the study on Qwen3.6 at temperature 0. The gap between search success and re-execution success remains 25.4 pp with thinking enabled and 12.2 pp with thinking disabled, and 38.4 and 21.3 pp under Spotlighting.

AuthorsOpenReview ↗

Response to Reviewer bcDx (2 of 3)

W2. The practical value of the proposed evaluation seems insufficiently established

The paper shows three concrete consequences. Each one changes a decision that an attack or defense developer makes.

1. It changes which attack method to choose, and whether automation helps at all.

  • On AgentDojo with Qwen3.6, a developer who selects by best-of-100 picks TAP over Random (82.0% against 80.5%). TAP's deployment success is 9.0 pp lower (30.9% against 39.9%; 95% CI [1.8, 16.3]). The same reversal occurs on AgentDyn.
  • Under best-of-100, Random and TAP look far stronger than a static template. Under deployment success they are not.

2. It changes how an attack budget should be spent. Proposition 1 implies that additional search cannot fix reliability under first-hit commitment. Spending part of the same budget on verification does. At B = 100 and identical coverage, deployment success rises from 50.3% to 58.2–58.9% on average over eight backend–victim settings, and by 12–19 pp on PIMiner and on Random × Qwen3.6. This requires no change to the attack generator.

3. It changes how defenses are evaluated.

  • Best-of-k saturates with budget and hides what a defense does. For Random search on Qwen3.6 at B = 100, Spotlighting lowers best-of-100 success from 80.5% to 74.2%, a relative reduction of 8%. It lowers deployment success from 43.2% to 29.5%, a relative reduction of 32% (Appendix C.1). Judged by best-of-100, the defense looks nearly ineffective. Judged by what the attacker can actually deliver, it removes a third of the attack's success.
  • Unreliable test payloads make regression testing uninformative. If a committed payload succeeds with probability 0.5, which is typical of first-hit payloads in our study, three consecutive failures after a patch occur 12.5% of the time when nothing has been fixed. With a payload that succeeds with probability 0.9, the same observation occurs 0.1% of the time. This example illustrates the value of more reliable test payloads. In our additional experiment under Spotlighting with PIMiner × Qwen3.6, overall deployment success across tasks rises from 68.5% (Search) to 87.9% (Adaptive) at the same budget (new experiment; Appendix G.4 of the revised PDF).
AuthorsOpenReview ↗

Response to Reviewer bcDx (3 of 3)

W3. The improvement over a simple repeated-testing baseline is relatively modest

The contribution of SVC is not only the allocation rules. It has three levels: we pose and formalize the problem of candidate discovery, verification, and final commitment under a shared budget, establish the SVC framework, and give concrete policies that solve it. To our knowledge, existing automated IPI red-teaming methods all take search-time success as the objective of their optimization and stopping, and none jointly optimizes discovery, verification, and final commitment under a shared budget to maximize deployment success. The results under W1 show that search-time success is not aligned with deployment success. We formalize the objective that should be optimized as a terminal selection problem under a shared budget: with B queries in total and one payload to deliver, how should candidate discovery, reliability verification, and final commitment be decided jointly to maximize deployment success D? Definition 2 specifies the three actions Search, Verify, and Commit, the shared budget constraint, and the evidence available to a policy, and Eq. (6) defines the objective.

The repeated-testing baseline the reviewer refers to is itself part of this framework. Fixed is not an existing method. It is the simplest policy inside the SVC definition, and the revised Section 5.1 makes this explicit; StaticOpt and Adaptive improve the budget allocation through cross-task calibration and current-task evidence, respectively. The comparison that measures what the paper proposes is therefore Search vs. SVC, rather than only StaticOpt and Adaptive vs. Fixed.

The gains of SVC hold under three evaluation conditions. The table summarizes deployment success D (%) at B = 100. The AgentDyn and Spotlighting rows are new experiments run during the rebuttal; their full results are in Appendices G.5 and G.4 of the revised PDF, respectively, and the full results for the AgentDojo row are in Appendix G.1.

Evaluation conditionSearchBest fixed-schedule policyStaticOptAdaptiveAdaptive − SearchAdaptive − best fixed-schedule policy
AgentDojo, no defense (mean of 8 settings)50.2857.31 (RR-50)58.2058.87+8.59 [7.21, 9.98]+1.56 [0.60, 2.53]
AgentDyn, no defense (mean of 8 settings)52.4763.42 (Fixed6)63.8265.38+12.92 [10.35, 15.45]+1.96 [0.97, 2.97]
AgentDojo, Spotlighting defense (PIMiner × Qwen3.6)68.4884.53 (Fixed6)84.2687.93+19.45 [15.35, 23.67]+3.40 [1.48, 5.35]

Differences are in pp, with task-paired 95% bootstrap intervals in brackets. Fixed-schedule policies are the two kinds of SVC policy that fix the verification schedule in advance and need no calibration or planning: fixed depth (each candidate is executed at most 5 or 6 times; Fixed5, Fixed6) and round-robin (search with 25%, 50%, or 75% of the budget, then verify all candidates in turn; RR-25/50/75). The "best fixed-schedule policy" is chosen in hindsight among these five by mean D, so the comparison favors them.

  • Verifying candidates and committing based on the evidence yields a large gain that is not specific to one benchmark or setting. At identical coverage, Adaptive improves over Search by 8.6, 12.9, and 19.5 pp under the three conditions.
  • Even against the best fixed-schedule policy chosen in hindsight, Adaptive is higher under all three conditions, with every interval excluding zero. The average increments the reviewer refers to (1.6 and 2.3 pp for StaticOpt and Adaptive over Fixed5) are statistically reliable as well (intervals [0.8, 2.5] and [1.5, 3.1]).
  • The gain from the allocation rules is larger when candidates are plentiful. On PIMiner, the strongest backend, Adaptive exceeds Fixed5 by 4.3–4.4 pp on both AgentDojo victims.
  • Simple rules are not robust. On TAP × Qwen3.6 on AgentDojo, fixed-depth verification gains nothing over Search (−0.2 pp), while StaticOpt gains 6.5 pp. In all eight AgentDojo settings, the best policy is StaticOpt or Adaptive; on AgentDyn, Adaptive is the best policy in six of the eight settings.

SVC is one of three contributions. The evaluation framework with Proposition 1, and the re-evaluation of four methods on two benchmarks and three victims with over 130K victim rollouts, stand independently of the size of this increment.

We kindly ask the reviewer to reconsider the assessment in light of these results, and we are happy to answer any further questions. Thank you again for your time and feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful reading and continued discussion. Following the discussion, all three reviewers rate Soundness and Presentation 3. The remaining concerns focus primarily on significance. The revised PDF has been uploaded, with changes in blue. Below, we summarize the discussion with each reviewer.

Reviewer TCEv: the overlap concern is resolved.

We recomputed the Search–Verify–Commit (SVC) results on the 96 non-Workspace task pairs (Appendix G.2). At B = 100, StaticOpt and Adaptive retain gains of 8.22 and 8.89 percentage points (pp) over Search, and 1.78 and 2.44 pp over Fixed5. TCEv considered the overlap concern satisfactorily addressed, noted that the additional AgentDyn experiments strengthen the evidence, and maintained weak accept (rating 3).

Reviewer PgGC: experimental evidence acknowledged; disagreement on contribution remains.

We gave the Fixed5 and Fixed6 comparisons greater prominence, added three uncalibrated round-robin policies, and conducted SVC experiments on AgentDyn, under Spotlighting, and across temperature and thinking configurations (Appendices G.1 and G.3–G.5). PgGC acknowledged the new evidence and corrected gain reporting, considered the scope concerns mostly addressed, and raised Soundness from 2 to 3 while retaining the overall rating of 2. Section 8 now specifies that the decoding and Spotlighting studies each cover one backend–victim pair and that Spotlighting replays candidates generated without the defense.

The remaining disagreement is whether the contribution is significant given the modest allocation increment. Our contribution forms a connected chain. We quantify the search–holdout gap: 14–27% of search-successful candidates fail all four independent holdouts. Under stationary i.i.d. search, Proposition 1 establishes budget invariance of within-task first-hit reliability, while the task-mixture analysis explains changes in pooled reliability. Re-evaluation under A/R/D across four methods, two benchmarks, and three victims demonstrates ranking reversals and shows that Random and TAP do not outperform a static template on AgentDojo with the two open-weight victims. These findings motivate our formulation of discovery, verification, and terminal commitment as a joint decision problem under a shared online budget, with deployment success as the objective. We agree that re-testing is a natural step once committed-payload reliability is the target. The preceding results show why committed-payload reliability should be an explicit objective: existing methods optimize and stop on search success, and this surrogate is misaligned with deployment success. The allocation rules are the final component of this chain.

We acknowledge that fixed schedules capture most of the improvement and now report this prominently. At B = 100, Adaptive exceeds the best fixed schedule chosen in hindsight by +1.56 pp [0.60, 2.53] on AgentDojo, +1.96 [0.97, 2.97] on AgentDyn, and +3.40 [1.48, 5.35] under Spotlighting (95% task-paired bootstrap intervals). StaticOpt is on par with the best fixed schedule in the latter two conditions, as stated in Section 8.

Reviewer bcDx: practical relevance acknowledged; the significance concern remains.

We answered each concern with concrete evidence. Coverage A preserves best-of-k evaluation; reliability and deployment success additionally characterize the delivered payload. This changes method selection: the ranking of TAP and Random reverses when moving from coverage to deployment success on AgentDojo/Qwen3.6 at B = 100. It changes budget use: at B = 100 and equal coverage, Adaptive improves deployment success over Search by 8.6 pp, averaged over eight AgentDojo settings. It also changes defense assessment: for Random × Qwen3.6 on AgentDojo at B = 100, Spotlighting reduces best-of-100 success by 8% but deployment success by 32% in relative terms (Appendix C.1). The concern about modest gains over repeated testing overlaps with PgGC's concern and is addressed above.

bcDx acknowledged that our clarifications strengthen the case for practical relevance, but maintained the recommendation without specifying which answers remained unconvincing. We invited clarification of any unresolved points and have not received a further reply.

We respectfully ask that the final assessment weigh these concrete responses and the full contribution chain: reliability analysis, re-evaluation of existing methods, problem formulation, and policies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.