acceptodds
Under review as a conference paper at ICLR 2027

STRAP: STrategy-Driven Red-Teaming of Multimodal Web Agents via Latent Sequential Probe Injection

Abstract

Multimodal web agents retain browsing context, allowing content encountered on one page to influence actions on later pages. We study delayed prompt injection in a black-box setting where an attacker modifies earlier webpages but has no access to the user's task instruction or control over the target page. We introduce STRAP, a strategy-driven red-teaming framework that coordinates page-level probes to induce an adversarial action after intervening uncontrolled navigation. STRAP uses behavioral feedback to refine attack strategies and train a probe generator through supervised fine-tuning and preference optimization; at deployment, the generator conditions successive probes on observed interactions. A visual extension pairs text probes with adversarial image perturbations optimized using a proxy model. Across three task domains and three proprietary vision-language backends in WebVoyager, STRAP achieves 91.2–95.5% attack success with three controlled pages and three intervening uncontrolled pages, exceeding the strongest adapted baseline by approximately 40.0 percentage points on average across the nine settings under matched page budgets. Under four evaluated defenses on GPT-4o, attack success remains 74.8–84.5%. These results highlight the need to limit how untrusted webpage content influences subsequent agent decisions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial attack, web agents, red teaming

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BNvrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes STRAP, an attack on multimodal web agents. The attacker controls a few early webpages and injects text into hidden page elements, together with small changes to page images. The goal is to let the agent continue its task until a later page, such as checkout, where it follows the attacker's instruction. The text generator is trained on examples collected through an attacker–scorer–strategist loop. On WebVoyager with three commercial backends, the paper reports 91.2-95.5% attack success, and 74.8-84.5% under four tested defenses on GPT-4o.

Critical strengths
  1. The attack scenario makes sense. It is easy to imagine an attacker running a coupon or price-comparison site but having no way to modify a retailer's checkout page. It’s worth studying what happens when injection and action occur on different pages.
  2. The high success rates across three backends suggest that such attacks can be effective under the evaluated settings, where agents retain their full browsing history.
Critical weaknesses
  1. The paper does not establish that the attack is executable under the threat model. Section 3.1 says the attacker controls only early pages and does not know the user's task. Yet the generator takes ctmc_t^m, defined as the agent's accumulated context, and the prompt in Appendix F.1 includes "previous actions observed so far." A website owner can record activity on their own pages. This does not give them the agent's actions on other sites, its reasoning, or whether the user's task is progressing normally. Section 3.2 says the frozen prober at test time is conditioned on "the agent's actions observed so far," but does not specify which actions, or how a webpage owner would observe them. According to these details, the generator requires more information than the stated attacker has in the threat model. The paper does not clarify that its reported success rates are achievable under the settings in Section 3.1.
  2. The main experiments also assume that the agent encounters three attacker-controlled pages before reaching the target. Appendix Q tests what happens when some pages are skipped, but does not explain how the three injection points are placed in the trajectory to begin with, or whether this placement requires knowing the agent's route in advance.
  3. The evaluation does not check agent behaviors before the final target action. The paper claims the agent behaves normally until it reaches the target page. But the judge in Appendix I only checks whether the final address matches the attacker's target. It does not inspect the earlier trajectory. An attack that carries out the malicious action before checkout and eventually submits the target address would still pass this judge. The paper does not report how often the attack was actually innocuous through intermediate pages.
  4. The paper attributes STRAP's success against defenses to delayed execution, but the defenses may simply be ineffective against immediate attacks as well. AdvAgent already achieves about 93% success without defenses in its native setting (Table 6). How much does this drop when TaskShield or TraceAegis is applied? If it remains high, then STRAP's high success rate alone does not show that delaying the attack helps it evade detection.
  5. The introduction explains why prior attacks were insufficient and why the three properties are necessary. But prior attacks work well in their own settings and perform worse in this paper’s setting. The paper should frame its contribution as studying a different threat model, instead of addressing a failure of prior work.
  6. The paper reports high attack success rates, but it does not demonstrate that its proposed method is necessary for an injection to remain effective across multiple pages. Table 7 shows that text probes without a strategy already achieve 80-86% success rates, while Figure 3 only evaluates STRAP against distance. The paper should run the no-strategy variant from Table 7 or the adapted injection baselines from Table 1 through the same distance test. Without this comparison, it is unclear whether STRAP specifically improves persistence across intermediate pages or whether common prompt injections would perform similarly when the full history is retained.
  7. The detailed examples and published judge mainly focus on address replacement, even though the paper covers three domains. Testing other attack goals beyond changing form values would provide stronger evidence that the method works across different types of tasks.
  8. The main results assume that agents retain the full history. Although the truncation experiments in Appendix P are helpful, the large drop in success rates with shorter histories shows that memory management affects the attack's effectiveness. The paper should either narrow its claims or test the method on agents that discard or summarize older observations.
Questions for authors
  1. What exact inputs does the frozen prober receive at test time, and how would a webpage owner obtain each one?
  2. Can the authors report results using only webpage content and events from attacker-controlled pages, without access to other activities?
  3. Among successful attacks, how many satisfy the requirement that the agent stays on task and does not attempt the malicious action before the target page? How is this validated?
  4. Can the authors evaluate AdvAgent or Genesis in their original settings with TaskShield and TraceAegis, and compare those results with their undefended performance?
5 comments
AuthorsOpenReview ↗

Rebuttal to reviewer BNvr

We thank the reviewer for the constructive feedback and answer each point below.

W1: Clarification of the threat model "Section 3.1 says the attacker controls only early pages and does not know the user's task. Yet the generator takes ctmc_t^m, defined as the agent's accumulated context, and the prompt in Appendix F.1 includes 'previous actions observed so far.' A website owner can record activity on their own pages. This does not give them the agent's actions on other sites, its reasoning, or whether the user's task is progressing normally."

Response: Our notation implied more than the prober actually uses, and we will clarify this in the paper. The experiments do not rely on it: at test time the prober takes only inputs that the owner of the controlled pages has, and all reported results use only these (see Q1 and Q2 below). The agent's full context ctc_t (Sec. 4.1) stays in the agent's memory, where the attacker cannot access it (Sec. 3.1 and 4). The generator in Eq. 1 does not take ctc_t. It takes ctmc_t^m: only the observations and agent actions that occur on the attacker-controlled pages up to the mm-th. The attacker controls those pages and can log the agent's interaction events on them, so it can record ctmc_t^m. Anything the agent holds from uncontrolled pages (WmidW_{\text{mid}}, wtargetw_{\text{target}}) is not in ctmc_t^m. At test time, each probe is generated by Eq. 1 from ctmc_t^m and delivered on the controlled page, and the attacker observes only the agent's behavior on that page after delivery. We will rephrase "the agent context accumulated up to that page" (Eq. 1 and Appendix F.1) to define ctmc_t^m as the attacker-controlled-page portion of ctc_t.

The generator does not use the victim agent's actions on other sites, the agent's reasoning, or the user's task. As defined above, ctmc_t^m contains no actions on other sites. The generator is conditioned on the objective GG rather than the task TT (Sec. 3.2), so it uses neither the user's task nor its progress. At test time it does not use the agent's reasoning either. Its only feedback is the agent's actions logged on the attacker's own pages (ctmc_t^m), such as the confirmatory action. One caveat: the Scorer description and prompt (Appendices B and F.2) can also credit probe mentions in the agent's reasoning, which conflicts with Appendix J. We will make them consistent. The Scorer runs only in the offline data collection phase (Sec. 3.2, Appendix H) and is never used at test time.

Reviewer: "Section 3.2 says the frozen prober at test time is conditioned on 'the agent's actions observed so far,' but does not specify which actions, or how a webpage owner would observe them."

Response: The actions are confirmatory actions, e.g., clicking a non-critical element or icon whose identifier matches the injected content (Appendix J). A page owner records them with standard event listeners and server logs on its own page, and the Scorer reads only "the resulting trajectory segment" on that page (Sec. 3.2). So "the agent's actions observed so far" means the agent's actions recorded on earlier controlled pages (ctmc_t^m), not its full history. We will reword Sec. 3.2 accordingly and restrict {ai<t}\{a_{i<t}\} in Sec. 4.3 and the perturbation description to ctmc_t^m.

Reviewer: "According to these details, the generator requires more information than the stated attacker has in the threat model. The paper does not clarify that its reported success rates are achievable under the settings in Section 3.1."

Response: At test time, the attack stays within the threat model of Sec. 3.1: every input it uses is available to the attacker (see Q1), and the reported results use only these inputs. The closed loop with environment resets (Appendix H) is an offline sandbox step that only labels training data. Its Scorer checks (e.g., continued progress toward the trigger page, Appendix J) are not test-time inputs. At test time the frozen prober makes one pass over the controlled pages, with no resets and no Scorer.

Q1: "What exact inputs does the frozen prober receive at test time, and how would a webpage owner obtain each one?"

Response: The prober receives four inputs, and the page owner has each of them: (i) the strategy sms^m, from the attacker's own library, (ii) the objective GG and target-page description, which are the attacker's own goal, (iii) the previous probe pm−1p^{m-1}, the attacker's own output, and (iv) ctmc_t^m, the agent's actions logged on earlier controlled pages via standard event listeners and server logs on its own pages, as we will state in Sec. 3.1.

Q2: "Can the authors report results using only webpage content and events from attacker-controlled pages, without access to other activities?"

Response: Yes. The frozen prober receives only the inputs in Q1, so the main results (Tables 1-4) are already obtained using only webpage content and events from attacker-controlled pages.

AuthorsOpenReview ↗

Cont. rebuttal to reviewer BNvr

W2: Placement of the attacker-controlled pages

Response: Yes, the main results (Tables 1 to 3) use three controlled pages by default (Sec. 5.1). That number is an evaluation setting, and the attacker does not arrange it. All the attacker assumes is that the agent visits some of its pages, as in the reviewer's own example of a coupon or price-comparison site on the way to checkout. Figure 3a also tests one, three, four and five controlled pages, and a single controlled page is already enough for 79.4% ASR (GPT-4o, Sec. 6).

How the three injection points are placed in the trajectory to begin with

Response: As Appendix Q states, "the attacker does not control page order", only "which pages it owns among those the agent's own navigation happens to visit." Page positions are therefore set by the agent's navigation rather than by the attacker. In our experiments we model this by placing the controlled pages at random positions before the target page (Table 13, "Current setting (reference, random)").

Whether this placement requires knowing the agent's route in advance

Response: It does not. The threat model gives the attacker no route information: it "has no knowledge of the victim's task or of any deployed defenses" and "knows only the type of target page ... not the specific page or when the agent will reach it" (Sec. 3.1), and "knows neither the victim's task nor the uncontrolled pages the agent visits" (Introduction, contributions). The prober generates each probe when the agent lands on a controlled page, using only sms^m, GG, pm−1p^{m-1} and ctmc_t^m (see our response to W1). None of these depends on the route or on how many pages will follow. The results are also not tied to one favorable arrangement (Appendix Q, GPT-4o E-commerce): front-loaded 93.40, back-loaded 93.30 and interleaved 88.90, against 95.50 for the random reference, and 89.90 and 82.80 when 10% and 20% of the controlled pages are never visited.

AuthorsOpenReview ↗

Cont. rebuttal to reviewer BNvr

W3 / Q3: Behavior before the target page

"The evaluation does not check agent behaviors before the final target action. The paper claims the agent behaves normally until it reaches the target page. But the judge in Appendix I only checks whether the final address matches the attacker's target. It does not inspect the earlier trajectory."

Response: The Appendix I judge does see only the target page, as the reviewer notes, so we also checked earlier behavior on step-level logs (details below). The attack objective and the attack success rate are defined over three page categories. The paper partitions each trajectory into attacker-controlled pages, uncontrolled intermediate pages, and the target page "where the adversarial objective is evaluated" (Sec. 3), and "We evaluate success only by the completion of the adversary's objective at wtargetw_{\text{target}}" (Sec. 3.1). The attack is successful if probes are "latent (it does not trigger the adversarial objective at the injection page)" and "persistent (it remains effective after intervening uncontrolled pages)", activating "only at the target page" (Introduction, Sec. 3.2). The probes are delivered on attacker-controlled pages and adapted so that they survive the intermediate pages, remain latent, and finally trigger at the target page. An episode in which the agent takes an action so invasive that it never reaches the target page therefore counts as a failure in ASR rather than as an attack success. A probe is marked "unsuccessful" if "the agent ... can no longer reach wtargetw_{\text{target}}" (Sec. 3.2), the Scorer gives 0 if the agent "deviated excessively, abandoning its trajectory" or if "the probe directly triggered the adversarial action at the current page" (Appendix F.2), and the judge scores only "what the agent actually entered or confirmed", with failure if the original address remains (Appendix I, Rule 5). We inspected the step-level logs of 100 episodes across 3 tasks and found that the adversarial action occurred before wtargetw_{\text{target}} in 7 of them. These are failure cases and are not counted in ASR.

"An attack that carries out the malicious action before checkout and eventually submits the target address would still pass this judge."

Response: Such a premature trigger is not counted as an attack success. It violates the latency requirement ("latent (it does not trigger the adversarial objective at the injection page)", Introduction), and during training it earns no reward: the Scorer gives 0 if "the probe directly triggered the adversarial action at the current page rather than remaining latent until the trigger page" (Appendix F.2), and a probe is rewarded only if the agent did not complete "the adversarial action prematurely" (Appendix B). Because the judge itself does not look at earlier pages, we checked this case separately on step-level logs. All 7 inspected episodes with a premature trigger count as failures and are not in the reported ASR. We will describe this check and its sample of 100 episodes next to the judge in Appendix I.

"The paper does not report how often the attack was actually innocuous through intermediate pages." / Q3: "Among successful attacks, how many satisfy the requirement that the agent stays on task and does not attempt the malicious action before the target page? How is this validated?"

Response: In the 100 episodes whose logs we read (above), every counted success took the adversarial action only at the target page. The 7 episodes in which the action came earlier all count as failures and are not in the reported ASR. During training, the Scorer also gives 0 to a probe that completes the adversarial action before the target page (Appendices B and F.2), so the prober is trained to stay latent.

Reviewer_j6QvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies delayed prompt injection against multimodal web agents. STRAP places malicious content on earlier webpages to influence actions on a later target page. It combines strategy refinement, a trained text probe generator, and visual perturbations. Experiments evaluate its effectiveness across several model backends and defenses.

Critical strengths
  1. The method combines strategy learning and sequential feedback in a coherent attack framework.

  2. Experiments across several model backends show strong attack performance.

  3. Tests on context truncation, page ordering, and another agent framework provide useful robustness evidence.

Critical weaknesses
  1. The framework contains many components, but their individual benefits are not equally clear. This makes it difficult to identify the main source of improvement.

  2. The baseline comparison needs stronger control of training and search budgets. Matching the number of controlled pages alone may not provide a fully fair comparison.

  3. The defense results need more careful validation. The defenses are adapted to the agent pipeline, and it is unclear whether these adaptations preserve all features of the original methods.

  4. The evaluation uses tasks selected for perfect clean success. More detailed reporting of task counts, repeated runs, and uncertainty would help establish the reliability of the results.

Questions for authors
  1. Are victim queries, candidate generation, and training resources comparable across methods?
  2. How did you verify that the adapted defenses preserve the protection offered by their original implementations?
3 comments
AuthorsOpenReview ↗

Rebuttal to reviewer j6Qv

We thank the reviewer for the constructive feedback and address each point below.

W1: Clarification of the improvements from the different components of the method.

Response: The main source of improvement is the strategy-driven refinement loop (+5.7 to +8.2 points on GPT-4o), followed by DPO (+4.8 to +5.9). The confirmatory action, the visual probe and refinement on failure each add about +2.8 to +3.8. The confirmatory action matters for a second reason as well: it is the only feedback an attacker gets when it cannot see the target page. In the paper, these ablations were scattered across Tables 2, 3, 7, 8 and 10 with different bases. Below we consolidate all of them in one table (GPT-4o, ASR in %, each cell shows "without →\rightarrow with (gain)"). Every value is taken from the cited table, and the table will be added to the appropriate section of the paper.

ComponentWhat is removed ("without")E-commerceTravelEntertainmentSource
Strategy-guided refinementText probe, no strategy83.20 → 90.45 (+7.25)85.56 → 91.23 (+5.67)84.64 → 92.34 (+7.70)Tab. 7, 2
Strategy-guided refinementText+Image probe, no strategy87.24 → 94.23 (+6.99)88.43 → 94.30 (+5.87)87.35 → 95.50 (+8.15)Tab. 7, 2
DPOSFT-only prober (text)85.42 → 90.45 (+5.03)86.45 → 91.23 (+4.78)86.43 → 92.34 (+5.91)Tab. 3
Confirmatory actionAny response counted as success, no refinement88.34 → 91.45 (+3.11)87.45 → 91.23 (+3.78)89.12 → 92.34 (+3.22)Tab. 10
Refinement on failureConfirmation kept, no refinement91.45 → 94.23 (+2.78)91.23 → 94.30 (+3.07)92.34 → 95.50 (+3.16)Tab. 10
Visual probeText-only probe (with strategy)90.45 → 94.23 (+3.78)91.23 → 94.30 (+3.07)92.34 → 95.50 (+3.16)Tab. 2
Alignment loss LalignL_{\text{align}} (image probe)Only LcomplL_{\text{compl}}81.78 → 86.45 (+4.67)82.45 → 87.23 (+4.78)83.12 → 87.89 (+4.77)Tab. 8
Complementarity loss LcomplL_{\text{compl}} (image probe)Only LalignL_{\text{align}}83.12 → 86.45 (+3.33)84.01 → 87.23 (+3.22)84.56 → 87.89 (+3.33)Tab. 8
All componentsText probe, no strategy83.20 → 94.23 (+11.03)85.56 → 94.30 (+8.74)84.64 → 95.50 (+10.86)Tab. 7, 1

The same ordering holds on the other backends where the ablations were run: across all nine settings the strategy gain is 5.11 to 8.22 points and the visual-probe gain is 2.56 to 4.00 points (Sec. 6), and the confirmatory-action and refinement gains are 3.11 to 4.44 and 2.67 to 3.67 points (Table 10).

AuthorsOpenReview ↗

Cont. rebuttal to reviewer j6Qv

W2: "The baseline comparison needs stronger control of training and search budgets. Matching the number of controlled pages alone may not provide a fully fair comparison."

Response: In Table 1 the page budget is matched, and the number of candidate probes per page and the victim queries during the attack are also matched across methods. Each baseline keeps its own generation, optimization or evolution procedure (Appendix C), and AdvAgent's prompter is not retrained for the delayed setting (only its delivery framing is changed, Table 5). STRAP, in contrast, uses an offline training stage (20 GPT-4o candidate probes per page, then SFT and DPO on the 80% training split of the 860 tasks, Sec. 5.1) that the baselines do not have. We therefore (1) document the budgets of all methods in one table, (2) describe Table 1 as a comparison under a matched page budget, not as a fully matched-compute comparison, and (3) state this as a limitation.

MethodControlled pagesCandidates per pageVictim queries during searchTraining dataRetrained for delayed setting
STRAP320 (GPT-4o, temperature 1.0)Offline loop, then one run per task at test time688 tasks (80% of 860), SFT then DPOYes
AdvAgent (adapted)3SameSame, one run per task at test timeNativeOnly the delivery framing is changed. It is trained for immediate attack according to its native threat model and method
Genesis (adapted)3SameOnline evolution at attack time, one run per task at test timeNative procedureNo training required, online delivery
GCG, AgentAttack, InjecAgent, WASP, MUZZLE (adapted)3SameSame, one run per task at test timeNative procedureGCG re-optimizes its suffix for the new target string (Appendix C), and WASP and MUZZLE are trained for the delayed setting

Q1: "Are victim queries, candidate generation, and training resources comparable across methods?"

Response: The page budget, the candidate generation and the test-time cost are comparable: every method runs once per task on 3 controlled pages, with the same number of candidate probes per page and victim queries during the attack. STRAP's additional cost is incurred offline, during training, and at attack time it makes a single pass with a frozen prober. The baselines are not weak. AdvAgent and Genesis reach 88 to 93% when they may inject at the target page (Table 6), but 39 to 59% when they may not (Table 1). That drop shows how hard the delayed setting is for attacks not built for it. STRAP, by comparison, stays within 4 points of its own oracle that injects at the target page (Table 6). It is built and trained for this setting, and we state its offline training cost as a limitation.

W3 / Q2: "The defense results need more careful validation. The defenses are adapted to the agent pipeline, and it is unclear whether these adaptations preserve all features of the original methods."

Response: Each adaptation keeps the method's original prompt, procedure or threshold, and "no thresholds were tuned on STRAP attack data" (Appendix O, table below). Two results show that the defenses keep their effect. (1) The defenses are active against immediate attacks. In the native setting, TaskShield lowers AdvAgent from 93.22% to 36.67% and Genesis from 92.64% to 40.91%, and TraceAegis lowers them to 33.11% and 46.32% (GPT-4o). STRAP's higher ASR under defense is therefore not an artifact of weakened defenses. (2) In Table 4, every defense lowers every adapted baseline (for example GCG from 26.41% to 12.44% under TraceAegis, AdvAgent from 48.44% to 36.12% under IPIGuard, Genesis from 58.29% to 40.78% under IPIGuard), and STRAP stays highest under all four (84.50%, 82.30%, 79.60%, 74.80%). The table below lists what each adaptation keeps and what it changes.

DefenseHow applied (Appendix O)Preserved from the originalChanged in adaptation
TaskShieldTask-alignment prompt appended at each stepAuthors' released prompt templateNo change
Instruction DetectionUntrusted tags on webpage content before context ingestionMarking of webpage-sourced content as untrustedAlso applied to OCR text from screenshots
IPIGuardTool dependency graph over the trajectoryFlagging of anomalous invocationsBrowser actions (click, type, scroll) mapped to tool calls
TraceAegisSliding-window trajectory scorerAuthors' default threshold (0.6)No change
AuthorsOpenReview ↗

Cont. rebuttal to reviewer j6Qv

W4: "The evaluation uses tasks selected for perfect clean success. More detailed reporting of task counts, repeated runs, and uncertainty would help establish the reliability of the results."

Response: We address the three parts in turn.

Task counts. We will add the following table to the appendix. The train/test split is described in Sec. 5.1.

DomainTasks before filteringTasks after filtering (100% clean success)Train / test (80/20)
Entertainment523360288 / 72
E-commerce322260208 / 52
Travel367240192 / 48
Total1212860688 / 172

Each domain's ASR is therefore computed over 72, 52 or 48 test tasks (172 in total).

Repeated runs and uncertainty. Every reported ASR is the mean over 10 independent runs of each attack. Each run evaluates the test tasks once, so one run's ASR moves in steps of one task (1.92 points for the 52 E-commerce test tasks). We will report both the sample standard deviation over the 10 runs and the standard error of the mean (standard deviation divided by √10) for every entry in Tables 1 to 4, and list the per-run ASRs in the appendix. For example, STRAP on GPT-4o, E-commerce: 94.23 (sample standard deviation 1.28, standard error 0.41).

Reviewer_TtNHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies delayed indirect prompt injection against multimodal web agents. The attacker controls earlier pages but not the user’s task, intermediate pages, or the target page, and tries to induce an adversarial action only after intervening navigation.

Critical strengths

The disclosure of prompts, adapted payloads, algorithmic details, qualitative traces, and defense configurations is unusually extensive. The explicit trade-off reported for treating all pre-target content as untrusted—lowering attack success to 35.57% while reducing clean task success from 100% to 71.30%—is useful for defense design.

Critical weaknesses

The evaluation also has unresolved validity and reporting problems. GPT-5.2 judges outcomes that often could be checked deterministically, yet no human agreement or judge error analysis is provided, and the reproduced judge prompt appears address-specific despite travel and entertainment tasks. GPT-4o generates tasks and probes and is also a victim, only 100%-clean-success tasks are retained, and leakage controls across templates, sites, objectives, and products are not described.

Questions for authors
  1. Can all attacks be compared under matched target queries, candidate probes, sequential feedback, training examples, and optimization compute, including retraining AdvAgent for the delayed setting?

  2. What capability remains unique to STRAP relative to a matched Genesis-like sequential feedback baseline without a named strategy library?

3 comments
AuthorsOpenReview ↗

Rebuttal to reviewer TtNH

We thank the reviewer for the constructive feedback. Our responses follow.

W1: Validation of the LLM judge with a deterministic process, and additional prompts for other tasks.

1. "GPT-5.2 judges outcomes that often could be checked deterministically." We agree, and we checked the judge against both manual labels and a deterministic checker. On 100 manually labeled outcomes, the GPT-5.2 judge matches the labels on all 100 and the deterministic checker on 98. For the deterministic check, we built a regex checker that normalizes case, punctuation and standard abbreviations (e.g., "St" vs "Street") and requires the street number, street name, city and country of the submitted address to match the target address. Both evaluators receive the same input for every outcome, e.g., the attacker's target address and the address the victim agent submitted. To validate them, we manually checked 100 samples drawn from all three domains (e-commerce, travel and entertainment) and labeled each as match or no match. The table reports accuracy against these manual labels. These are accuracy scores on the address comparison, not attack success rates. We will add the table, the regex pattern, the checked samples and the scoring code to the appendix.

EvaluatorSamples checkedCorrectAccuracy (%)
GPT-5.2 judge100100100
Regex checker1009898

2. "Additional prompt." The prompt in Appendix I is the template for the address-replacement objective, which is used in all three domains (Appendix I: "The objective corresponds to the delivery address associated with a booking confirmation"). Some tasks use other objectives, e.g., modifying the order configuration or changing the quantity (see our response to Reviewer BNvr, W7), and the template is instantiated with each objective's target value. We will add the prompt for every objective and domain to Appendix I.

W2: "GPT-4o generates tasks and probes and is also a victim, only 100%-clean-success tasks are retained, and leakage controls across templates, sites, objectives, and products are not described."

Response: 1. "GPT-4o generates tasks and probes and is also a victim." We use GPT-4o for two offline purposes (Sec. 5.1): iteratively generating benign navigation tasks, which we verify manually, and producing 20 candidate probes per page during training. Neither role is present at test time. At test time the trained and frozen prober (initialized from Qwen3) generates each probe from the page context and the objective GG, and GPT-4o is only a victim backend. The generated tasks are benign navigation tasks and contain no attack content. The results also do not hinge on GPT-4o being the victim: on the two backends that play no role in task or probe generation, STRAP reaches 91.23–92.34% (Claude Sonnet 4.6) and 92.12–93.45% (Gemini 3 Flash), within about four points of GPT-4o (94.23–95.50%, Table 1).

2. "Only 100%-clean-success tasks are retained." The filter is applied before any attack is run and identically to all methods, and its purpose is to ensure that an attack failure cannot be caused by the agent failing the clean task.

DomainTasks before filteringTasks after filtering (100% clean success)Train / test (80/20)
Entertainment523360288 / 72
E-commerce322260208 / 52
Travel367240192 / 48
Total1212860688 / 172

Each domain's ASR is therefore computed over 72, 52 or 48 test tasks (172 in total).

3. "Leakage controls across templates, sites, objectives, and products are not described." We will add a data-split description to the appendix. The split is disjoint at the level of websites and products: a website used for training is never used for testing (for example, if the prober is trained on tasks from Walmart, it is tested on tasks from Best Buy), and the products in the test tasks do not appear in the training tasks. The table below reports the overlap between the 688 training tasks and the 172 test tasks along each axis the reviewer lists.

AxisOverlap between train and testControl
WebsitesNoneWebsite-disjoint split: test websites are never seen in training
ProductsNoneTest tasks use products that do not appear in training tasks
Task templatesSharedSame task structure (for example, buy an item and check out). Task metadata differs: the delivery addresses, quantities and other task parameters
AuthorsOpenReview ↗

Cont. rebuttal to reviewer TtNH

Q1: "Can all attacks be compared under matched target queries, candidate probes, sequential feedback, training examples, and optimization compute, including retraining AdvAgent for the delayed setting?"

Response: In Table 1, the page budget and the number of candidate probes per page are matched across methods. Each baseline keeps its own generation, optimization, or evolution procedure (Appendix C), and AdvAgent's prompter is not retrained for the delayed setting to maintain the original threat model (immediate attack), and only its delivery framing is changed (Table 5). STRAP, in contrast, uses an offline training stage (20 GPT-4o candidate probes per page, followed by SFT and DPO on the 80% training split of the 860 tasks, Sec. 5.1) and sequential confirmatory-action feedback across pages, which the baselines do not have. We therefore (1) document the budgets of all methods in one table, (2) describe Table 1 as a comparison under a matched page budget, not as a fully matched-compute comparison, and (3) state this as a limitation.

MethodControlled pagesCandidates per pageVictim queries during searchTraining dataTrained for the delayed setting
STRAP320 (GPT-4o, temperature 1.0)Offline loop, then one run per task at test time688 tasks (80% of 860), SFT then DPOYes
AdvAgent (adapted)320One run per task at test timeNative training for immediate attack under its native threat modelNo, only the delivery framing is changed (Table 5)
Genesis (adapted)320Online evolution at attack time, one run per task at test timeNative procedure, no training requiredNo
GCG, AgentAttack, InjecAgent, WASP, MUZZLE (adapted)320One run per task at test timeNative procedureGCG re-optimizes its suffix for the new target string (Appendix C), and WASP and MUZZLE are trained for the delayed setting

Q2: "What capability remains unique to STRAP relative to a matched Genesis-like sequential feedback baseline without a named strategy library?"

Response: Four things set STRAP apart: a feedback signal that needs no access to the target page (the confirmatory action), a prober trained offline across tasks so that test time is a single pass, strategy conditioning, and a visual probe coordinated with the text probe. The closest test in the paper removes strategy guidance, and with it sequential refinement, at the same page budget. Removing them costs 5.1 to 8.2 points for text probes (Table 2 vs. Table 7, Appendix E). We have not run a Genesis variant with our feedback channel, and we will list this as a limitation. The table below summarizes the differences.

CapabilityGenesis-like sequential-feedback baseline (no named library)STRAPEvidence in the paper
Feedback signal on early pagesBuilt for injection at the target page, which our attacker cannot accessConfirmatory action: a task-neutral action showing that the probe may affect the victim agent (Appendix J). The attacker does not control the target pageTable 10: confirmatory action plus refinement, GPT-4o E-commerce 88.34% to 94.23%
When search and feedback happenOnline at attack time, with repeated victim queriesOffline loop with environment resets (Appendix H), and at test time a frozen prober generates probes in one passSec. 4.1, Appendix H
How probes are producedEvolved payload: injected at the target page natively, reworded and replicated at each controlled page when adapted (Table 5)Learned prober trained with SFT and then DPO, pm=F(sm,ctm,G,pm−1)p^m = F(s^m, c_t^m, G, p^{m-1}), conditioned on the accumulated context, the previous probe and the objective GGTable 3: DPO adds 4.78 to 5.91 points over SFT alone
ModalityText payloadMultimodal: a text probe coordinated with a visual probe (Appendix M)Table 2: image probe adds 2.56 to 4.00 points on top of text
Strategy guidanceEvolves strategies but is not trained across tasksStrategy conditioning during data collection and probe generationTable 2: strategy guidance adds 5.11 to 8.22 points over unguided injection

A Genesis-like baseline thus needs feedback from the target page, which the attacker in our threat model cannot access, and searches online at attack time. STRAP gets its feedback on its own pages (confirmatory actions), learns offline, and generates probes in a single pass at test time.

Thanks for the detailed rebuttal. I maintain my score to support this paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

General response to all reviewers

We thank the reviewers for their careful reviews and were encouraged by these comments:

  • Reviewer BNvr: "The attack scenario makes sense. It is easy to imagine an attacker running a coupon or price-comparison site but having no way to modify a retailer's checkout page. It's worth studying what happens when injection and action occur on different pages."
  • Reviewer j6Qv: "The method combines strategy learning and sequential feedback in a coherent attack framework.", "Experiments across several model backends show strong attack performance." and "Tests on context truncation, page ordering, and another agent framework provide useful robustness evidence."
  • Reviewer TtNH: "The disclosure of prompts, adapted payloads, algorithmic details, qualitative traces, and defense configurations is unusually extensive." and the trust-boundary trade-off "is useful for defense design."

The claim in brief. Webpage content is a known attack surface, but the threat model is new. The attacker owns a page that the agent passes on its way, such as a coupon or comparison site, yet never sees the user's task and never touches the checkout page where the harm happens. We show that this attacker still succeeds (91.2 to 95.5% ASR on three backends, Table 1) and that delaying the trigger makes TaskShield and TraceAegis much less effective. These two defenses remove half or more of the success of AdvAgent and Genesis when those attacks trigger immediately, but only 11 to 21% of STRAP's (GPT-4o, BNvr W4). Losing access to the target page costs STRAP about 3 points against its oracle variant (94.23 vs. 97.32, Table 6), but costs AdvAgent 45 points and Genesis 34 (Table 6 vs. Table 1, GPT-4o, E-commerce). Prior attacks work well in the settings they were built for. Our point is that this setting is different and realistic, and to our knowledge no prior attack targets it.

The reviewers mainly asked whether the attack is executable under this threat model, whether it is the delay that weakens the defenses, and whether the evaluation can be trusted. Short answers follow, with details in the individual replies:

  1. Is the attack executable under the threat model? Yes. At test time the prober sees only its strategy, its goal, its previous probe and the events logged on the attacker's own pages, and every reported number uses only these. Our notation implied more than this, and we will clarify it in the paper. The controlled pages sit at random positions that the attacker does not choose (BNvr W1, W2, Q1, Q2).
  2. Latency, delay and persistence. In the step-level logs of 100 episodes, the adversarial action came before the target page in 7. All 7 count as failures and are not in the reported ASR, so every counted success in that sample stayed latent (BNvr W3). Under TaskShield and TraceAegis, AdvAgent and Genesis keep about 35 to 50% of their success when they trigger immediately, but 77 to 86% when the same attacks are delayed (BNvr W4). In a new distance test, the injection without strategy fades over intermediate pages, while full STRAP holds steady (BNvr W6).
  3. Components and budgets. The largest gain comes from the strategy-driven refinement loop, and each other component adds a smaller but consistent one (j6Qv W1). Table 1 matches the page budget and the candidates per page, but not STRAP's offline training. We will say so and list it as a limitation (j6Qv W2, TtNH Q1).
  4. Evaluation reliability. On 100 manually labeled outcomes, the GPT-5.2 judge agrees on all 100 and a deterministic checker on 98. Claude and Gemini, which play no role in task or probe generation, reach 91.2 to 93.5% ASR, against 94.2 to 95.5% on GPT-4o. We also report task counts, the split protocol, and standard deviation and standard error over 10 runs (j6Qv W4, TtNH W1, W2).
  5. Scope and attack goals. The evaluation covers several attack objectives (delivery address, order configuration and order quantity), with 93.5 to 94.7% ASR on each (GPT-4o, E-commerce). We will also state explicitly that the reported results apply to agents that keep the controlled-page observations in context, and move the context-truncation results into the main paper (BNvr W7, W8).

Planned changes to the paper. (i) The Introduction and contributions present a new threat model rather than a failure of prior work. (ii) We clarify that ctmc_t^m contains only observations and actions on the attacker's own pages. (iii) New tables cover the native-setting defenses, the distance test, the consolidated ablations, method budgets, judge validation, per-objective results, and task counts and splits. (iv) Table 1 is described as matched in page budget, with the offline training cost listed as a limitation. (v) Claims are scoped to agents that keep controlled-page content in context, and Table 12 moves to the main paper. We are happy to answer any remaining questions before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.