- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper separates two quantities that automated red-teaming for indirect prompt injection usually treats as one: whether search finds a payload that succeeds once (attack coverage, A) and how often that committed payload succeeds on a fresh rollout (conditional reliability, R). Their product D = AR is proposed as the deployment-relevant metric. Using random search on 128 AgentDojo task pairs with 100 candidates each and four holdout rollouts per candidate, the authors show that 14–27% of search-successful candidates fail every holdout, that pooled reliability falls as the search budget grows, and (Proposition 1) that this fall comes from task composition rather than from any within-task effect of budget. Re-evaluating Random, TAP, PISmith, and PIMiner on AgentDojo and AgentDyn with three victims and 20-rollout holdouts, they find that Random and TAP do not beat the static Important Instructions template on D and that method rankings can flip. They then propose Search–Verify–Commit (SVC), which spends part of the online query budget re-testing discovered candidates before committing one. At B = 100 on AgentDojo without defenses, these improve D over search-only commitment by 7.9 and 8.6 pp, and over a fixed five-repeat verification baseline by 1.6 and 2.3 pp.
Critical strengths
-
The paper’s core distinction is established rather than asserted. The A/R/D decomposition is simple, the holdout protocol keeps selection and evaluation apart, and Proposition 1 together with the task-mixture identity explains a pattern that would otherwise look contradictory.
-
The evaluation is careful in ways this subfield often is not. All SVC policies replay the same recorded search trace, so coverage is identical and the comparison isolates allocation and commitment.
Critical weaknesses
-
The gain attributable to SVC’s design, as opposed to verifying at all, is small, and the headline numbers obscure this. The 7.9 and 8.6 pp improvements in the abstract are over SEARCH. The natural baseline is FIXED, which re-runs each candidate up to five times, and it captures most of the benefit: at B = 100, D goes 50.3 (SEARCH) → 56.6 (FIXED) → 58.2 (StaticOpt) → 58.9 (Adaptive), and FIXED6 reaches 56.9 (Table 20). Repeated evaluation before commitment is already practiced in the cited work (Hofer et al. average scores over target executions; Monteuuis et al. accept only unanimous repeats), so the part of SVC that is new is the allocation rule, and its measured benefit is 1.3–2.3 pp over FIXED, with Adaptive–StaticOpt at +0.7 pp (CI [−0.08, 1.42]). A protocol paper whose novel component contributes at this level does not clear the bar on its own; the evaluation framework is useful but is a re-statement of practices in Hofer et al. and Monteuuis et al. with cleaner bookkeeping.
-
SVC is tested in a narrower setting than the paper’s own motivation. Section 3 motivates the problem with Spotlighting, but Experiment 3 runs only without defenses, only on AgentDojo, and only with the two open victims (GPT-5.4-mini is dropped). Reliability is defined relative to a fixed configuration with temperature 1.0 and thinking enabled on both victims; the paper does not show how much of the search-vs-holdout gap survives at low temperature, which is a common deployment setting and would change the value of verification.
Questions for authors
-
Can you report the comparison against FIXED5 and FIXED6 with the same prominence as the comparison against SEARCH, and add a baseline that verifies all discovered candidates round-robin with no calibration or planning? That is the right null for the allocation rules.
-
How do the search-vs-holdout gap (Figure 1a) and the SVC gains change at temperature 0 or with thinking disabled? If the gap largely closes, the paper’s practical scope should say so.
-
Does the SVC gain hold under Spotlighting, where Section 3 shows lower reliability and sparser coverage? One backend × one model would be enough to answer this.
8 comments
Response to Reviewer PgGC (2 of 6)
Part 1. W1: what is new in SVC, and how much each part contributes
On the SVC side, our contribution has three levels: we pose and formalize the problem of candidate discovery, verification, and final commitment under a shared budget, establish the SVC framework, and give concrete policies that solve it. The gains from the fixed-schedule policies and from the allocation rules come from an empirical comparison of different policies inside this framework. We first give the decomposition of the gains, and then explain the contribution of the problem and the framework themselves.
1.1 We accept the reviewer's decomposition and now report it that way
The 7.9 and 8.6 pp in the abstract are total gains over SEARCH, and most of that gap is closed by the fixed-schedule policies (FIXED and RR). The revised paper makes this comparison explicit for AgentDojo at B = 100, with gains averaged over the eight backend–victim settings:
- the abstract and introduction report total gains over SEARCH side by side: up to +7.0 pp with fixed-schedule SVC policies, +7.9 pp with StaticOpt, and +8.6 pp with Adaptive;
- Section 6.1 explicitly reports the additional gains from allocation: +0.9 pp (StaticOpt) and +1.6 pp (Adaptive) over the best fixed-schedule policy (RR-50).
The best fixed-depth and round-robin variants (FIXED6 and RR-50) now appear in the main results next to SEARCH, and the complete results for FIXED5, FIXED6, and RR-25/50/75 are in the new Appendix G.1.
1.2 What is new relative to Hofer et al. and Monteuuis et al.
Both works repeat executions, and our text should position our work relative to them more precisely. They use repetition to serve decisions different from the one SVC addresses.
- Hofer et al. repeat executions in two places. Inside TAP, "reliability retries" are used to "average evaluator scores across multiple trials" to stabilize the judge signal that drives pruning and refinement; this is part of the search backend. After optimization, each resulting injection is evaluated several times; this measures the outcome. Neither step chooses among several environment-confirmed successful payloads, and neither is charged against a query budget shared with discovery.
- Monteuuis et al. (CAS-gen) accept a jailbreak prompt only if all k_gen repeated trials succeed, and stop at the first prompt that passes. This is a unanimity filter: a candidate with one failure is rejected, nothing is returned if no candidate passes, and no evidence is used to rank candidates.
Existing methods optimize search success, not the reliability of the delivered payload. To our knowledge, no automated IPI red-teaming method takes the fresh-execution success probability of the payload it finally commits as the objective of its search or optimization. The four methods we evaluate all stop and commit at the first observed success, and the repeated executions in the two works above do not target it either. The paper's results show that this surrogate objective is not aligned with deployment success: under first-hit commitment, a larger search budget does not improve the reliability of the committed payload (Proposition 1), and method rankings by search success reverse (Table 1).
We formalize this decision as a terminal selection problem under a shared budget: with B queries in total and one payload to deliver, how should candidate discovery, reliability verification, and final commitment be decided jointly to maximize deployment success D? Definition 2 specifies the three actions (Search, Verify, and Commit), the shared budget constraint, and the evidence available to a policy; Eq. (6) defines the objective. Inside this framework, FIXED and RR are simple policies, while StaticOpt and Adaptive improve the allocation through cross-task calibration and current-task evidence, respectively. Concretely, under our verification and commitment policies:
- verification is paid for from the same budget as search;
- a candidate that fails a verification stays eligible and is ranked by posterior mean;
- a payload is committed whenever one exists, so coverage is unchanged while reliability improves.
FIXED and RR are therefore not implementations of prior practice. They are the simplest policies inside Definition 2. We described FIXED as a baseline "motivated by" Monteuuis et al., which invited the reading that verify-then-commit under a shared budget already existed. The revision corrects this and presents FIRST-HIT, FIXED, RR, StaticOpt, and Adaptive as one family of SVC policies (Section 5.1). Read this way, the +6.3 to +7.0 pp is the measured value of the protocol, and the allocation rules add a smaller, backend-dependent increment on top. We do not rest the paper on the second increment alone.
Response to Reviewer PgGC (3 of 6)
1.3 What the allocation rules add on top
The reviewer is right that this second increment is modest on average: +1.6 and +2.3 pp over FIXED5, and +0.9 and +1.6 pp over the best round-robin split. Its value is robustness across backends. Part 3 gives the evidence:
- no single simple rule is right everywhere: the best round-robin split differs by backend, and fixed-depth verification gains nothing on TAP × Qwen3.6;
- in all eight AgentDojo settings, the best policy is StaticOpt or Adaptive;
- the margin is largest where candidates are plentiful: Adaptive is 4.3–4.4 pp above FIXED5 on PIMiner with both victims.
1.4 On the evaluation framework
Repeated holdout evaluation already appears in some prior work, which we cite and discuss. The paper adds three things beyond it:
- the A/R/D decomposition, tied to an explicit commitment rule and query budget;
- Proposition 1 and the task-mixture identity, which show that a larger budget raises coverage without improving the reliability of the first-hit payload, and that pooled reliability can fall through task composition;
- the empirical consequence under one protocol across four methods, two benchmarks, and three victims: Random and TAP reach 43–82% coverage on AgentDojo with the open-weight victims, yet do not exceed a static template in D, and method rankings reverse.
To our knowledge, neither work reports these. The revised related-work paragraph states the distinction from both works explicitly.
More broadly, we see the paper's contribution as one line of argument rather than an evaluation framework plus an allocation rule. We quantify how often a search success fails to repeat and show that more search does not fix this (Proposition 1). We then propose A/R/D as a more complete set of evaluation metrics. Re-evaluating existing red-team methods of different types under these metrics shows that they optimize a surrogate that is misaligned with deployment success: rankings can reverse, and some automated methods do not outperform a static template. Finally, we formalize the resulting budgeted selection problem and give concrete policies for it.
Part 2. W2: scope — benchmarks and victims
AgentDyn. We ran the full SVC comparison on the 36 AgentDyn task pairs with all four backends and both open-weight victims (no defense, B = 100). The per-setting results are in Appendix G.5.
D (%) averaged over the eight backend–victim settings, and paired differences (pp, task-paired 95% bootstrap intervals):
| Policy | D | StaticOpt − policy | Adaptive − policy |
|---|---|---|---|
| SEARCH | 52.47 | +11.35 [8.98, 13.68] | +12.92 [10.35, 15.45] |
| FIXED5 | 63.16 | +0.66 [−0.82, 2.14] | +2.22 [0.95, 3.58] |
| FIXED6 | 63.42 | +0.40 [−0.99, 1.77] | +1.96 [0.97, 2.97] |
| RR-25 | 62.55 | +1.27 [0.35, 2.26] | +2.83 [1.28, 4.58] |
| RR-50 | 62.85 | +0.97 [−0.02, 1.96] | +2.53 [1.39, 3.70] |
| RR-75 | 61.77 | +2.05 [0.49, 3.66] | +3.61 [2.33, 4.95] |
| StaticOpt | 63.82 | — | +1.56 [0.36, 2.81] |
| Adaptive | 65.38 | — | — |
- The gain holds on a second benchmark. Verification before commitment improves D in all eight settings, by 11.4 pp (StaticOpt) and 12.9 pp (Adaptive) on average.
- Adaptive is the best policy in six of the eight settings. On average it exceeds every fixed-schedule policy by 2.0–3.6 pp and StaticOpt by 1.6 pp, with all intervals excluding zero.
- StaticOpt is on par with the best fixed-schedule policies (FIXED6 and RR-50); those intervals include zero.
GPT-5.4-mini. We did not run SVC on GPT-5.4-mini because of its cost. Section 8 now states that all SVC experiments use the two open-weight victims.
Response to Reviewer PgGC (1 of 6)
We thank the reviewer for the careful reading and for the three concrete questions. We ran all requested experiments. In brief:
The response has five parts, in the order W1, W2, Q1, Q2, Q3. The revised PDF has been uploaded; changes are in blue and listed at the end.