- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes STRAP, an attack on multimodal web agents. The attacker controls a few early webpages and injects text into hidden page elements, together with small changes to page images. The goal is to let the agent continue its task until a later page, such as checkout, where it follows the attacker's instruction. The text generator is trained on examples collected through an attacker–scorer–strategist loop. On WebVoyager with three commercial backends, the paper reports 91.2-95.5% attack success, and 74.8-84.5% under four tested defenses on GPT-4o.
Critical strengths
- The attack scenario makes sense. It is easy to imagine an attacker running a coupon or price-comparison site but having no way to modify a retailer's checkout page. It’s worth studying what happens when injection and action occur on different pages.
- The high success rates across three backends suggest that such attacks can be effective under the evaluated settings, where agents retain their full browsing history.
Critical weaknesses
- The paper does not establish that the attack is executable under the threat model. Section 3.1 says the attacker controls only early pages and does not know the user's task. Yet the generator takes , defined as the agent's accumulated context, and the prompt in Appendix F.1 includes "previous actions observed so far." A website owner can record activity on their own pages. This does not give them the agent's actions on other sites, its reasoning, or whether the user's task is progressing normally. Section 3.2 says the frozen prober at test time is conditioned on "the agent's actions observed so far," but does not specify which actions, or how a webpage owner would observe them. According to these details, the generator requires more information than the stated attacker has in the threat model. The paper does not clarify that its reported success rates are achievable under the settings in Section 3.1.
- The main experiments also assume that the agent encounters three attacker-controlled pages before reaching the target. Appendix Q tests what happens when some pages are skipped, but does not explain how the three injection points are placed in the trajectory to begin with, or whether this placement requires knowing the agent's route in advance.
- The evaluation does not check agent behaviors before the final target action. The paper claims the agent behaves normally until it reaches the target page. But the judge in Appendix I only checks whether the final address matches the attacker's target. It does not inspect the earlier trajectory. An attack that carries out the malicious action before checkout and eventually submits the target address would still pass this judge. The paper does not report how often the attack was actually innocuous through intermediate pages.
- The paper attributes STRAP's success against defenses to delayed execution, but the defenses may simply be ineffective against immediate attacks as well. AdvAgent already achieves about 93% success without defenses in its native setting (Table 6). How much does this drop when TaskShield or TraceAegis is applied? If it remains high, then STRAP's high success rate alone does not show that delaying the attack helps it evade detection.
- The introduction explains why prior attacks were insufficient and why the three properties are necessary. But prior attacks work well in their own settings and perform worse in this paper’s setting. The paper should frame its contribution as studying a different threat model, instead of addressing a failure of prior work.
- The paper reports high attack success rates, but it does not demonstrate that its proposed method is necessary for an injection to remain effective across multiple pages. Table 7 shows that text probes without a strategy already achieve 80-86% success rates, while Figure 3 only evaluates STRAP against distance. The paper should run the no-strategy variant from Table 7 or the adapted injection baselines from Table 1 through the same distance test. Without this comparison, it is unclear whether STRAP specifically improves persistence across intermediate pages or whether common prompt injections would perform similarly when the full history is retained.
- The detailed examples and published judge mainly focus on address replacement, even though the paper covers three domains. Testing other attack goals beyond changing form values would provide stronger evidence that the method works across different types of tasks.
- The main results assume that agents retain the full history. Although the truncation experiments in Appendix P are helpful, the large drop in success rates with shorter histories shows that memory management affects the attack's effectiveness. The paper should either narrow its claims or test the method on agents that discard or summarize older observations.
Questions for authors
- What exact inputs does the frozen prober receive at test time, and how would a webpage owner obtain each one?
- Can the authors report results using only webpage content and events from attacker-controlled pages, without access to other activities?
- Among successful attacks, how many satisfy the requirement that the agent stays on task and does not attempt the malicious action before the target page? How is this validated?
- Can the authors evaluate AdvAgent or Genesis in their original settings with TaskShield and TraceAegis, and compare those results with their undefended performance?
5 comments
Cont. rebuttal to reviewer BNvr
W2: Placement of the attacker-controlled pages
Response: Yes, the main results (Tables 1 to 3) use three controlled pages by default (Sec. 5.1). That number is an evaluation setting, and the attacker does not arrange it. All the attacker assumes is that the agent visits some of its pages, as in the reviewer's own example of a coupon or price-comparison site on the way to checkout. Figure 3a also tests one, three, four and five controlled pages, and a single controlled page is already enough for 79.4% ASR (GPT-4o, Sec. 6).
How the three injection points are placed in the trajectory to begin with
Response: As Appendix Q states, "the attacker does not control page order", only "which pages it owns among those the agent's own navigation happens to visit." Page positions are therefore set by the agent's navigation rather than by the attacker. In our experiments we model this by placing the controlled pages at random positions before the target page (Table 13, "Current setting (reference, random)").
Whether this placement requires knowing the agent's route in advance
Response: It does not. The threat model gives the attacker no route information: it "has no knowledge of the victim's task or of any deployed defenses" and "knows only the type of target page ... not the specific page or when the agent will reach it" (Sec. 3.1), and "knows neither the victim's task nor the uncontrolled pages the agent visits" (Introduction, contributions). The prober generates each probe when the agent lands on a controlled page, using only , , and (see our response to W1). None of these depends on the route or on how many pages will follow. The results are also not tied to one favorable arrangement (Appendix Q, GPT-4o E-commerce): front-loaded 93.40, back-loaded 93.30 and interleaved 88.90, against 95.50 for the random reference, and 89.90 and 82.80 when 10% and 20% of the controlled pages are never visited.
Cont. rebuttal to reviewer BNvr
W3 / Q3: Behavior before the target page
"The evaluation does not check agent behaviors before the final target action. The paper claims the agent behaves normally until it reaches the target page. But the judge in Appendix I only checks whether the final address matches the attacker's target. It does not inspect the earlier trajectory."
Response: The Appendix I judge does see only the target page, as the reviewer notes, so we also checked earlier behavior on step-level logs (details below). The attack objective and the attack success rate are defined over three page categories. The paper partitions each trajectory into attacker-controlled pages, uncontrolled intermediate pages, and the target page "where the adversarial objective is evaluated" (Sec. 3), and "We evaluate success only by the completion of the adversary's objective at " (Sec. 3.1). The attack is successful if probes are "latent (it does not trigger the adversarial objective at the injection page)" and "persistent (it remains effective after intervening uncontrolled pages)", activating "only at the target page" (Introduction, Sec. 3.2). The probes are delivered on attacker-controlled pages and adapted so that they survive the intermediate pages, remain latent, and finally trigger at the target page. An episode in which the agent takes an action so invasive that it never reaches the target page therefore counts as a failure in ASR rather than as an attack success. A probe is marked "unsuccessful" if "the agent ... can no longer reach " (Sec. 3.2), the Scorer gives 0 if the agent "deviated excessively, abandoning its trajectory" or if "the probe directly triggered the adversarial action at the current page" (Appendix F.2), and the judge scores only "what the agent actually entered or confirmed", with failure if the original address remains (Appendix I, Rule 5). We inspected the step-level logs of 100 episodes across 3 tasks and found that the adversarial action occurred before in 7 of them. These are failure cases and are not counted in ASR.
"An attack that carries out the malicious action before checkout and eventually submits the target address would still pass this judge."
Response: Such a premature trigger is not counted as an attack success. It violates the latency requirement ("latent (it does not trigger the adversarial objective at the injection page)", Introduction), and during training it earns no reward: the Scorer gives 0 if "the probe directly triggered the adversarial action at the current page rather than remaining latent until the trigger page" (Appendix F.2), and a probe is rewarded only if the agent did not complete "the adversarial action prematurely" (Appendix B). Because the judge itself does not look at earlier pages, we checked this case separately on step-level logs. All 7 inspected episodes with a premature trigger count as failures and are not in the reported ASR. We will describe this check and its sample of 100 episodes next to the judge in Appendix I.
"The paper does not report how often the attack was actually innocuous through intermediate pages." / Q3: "Among successful attacks, how many satisfy the requirement that the agent stays on task and does not attempt the malicious action before the target page? How is this validated?"
Response: In the 100 episodes whose logs we read (above), every counted success took the adversarial action only at the target page. The 7 episodes in which the action came earlier all count as failures and are not in the reported ASR. During training, the Scorer also gives 0 to a probe that completes the adversarial action before the target page (Appendices B and F.2), so the prober is trained to stay latent.
Rebuttal to reviewer BNvr
We thank the reviewer for the constructive feedback and answer each point below.
W1: Clarification of the threat model "Section 3.1 says the attacker controls only early pages and does not know the user's task. Yet the generator takes ctm, defined as the agent's accumulated context, and the prompt in Appendix F.1 includes 'previous actions observed so far.' A website owner can record activity on their own pages. This does not give them the agent's actions on other sites, its reasoning, or whether the user's task is progressing normally."
Response: Our notation implied more than the prober actually uses, and we will clarify this in the paper. The experiments do not rely on it: at test time the prober takes only inputs that the owner of the controlled pages has, and all reported results use only these (see Q1 and Q2 below). The agent's full context ct (Sec. 4.1) stays in the agent's memory, where the attacker cannot access it (Sec. 3.1 and 4). The generator in Eq. 1 does not take ct. It takes ctm: only the observations and agent actions that occur on the attacker-controlled pages up to the m-th. The attacker controls those pages and can log the agent's interaction events on them, so it can record ctm. Anything the agent holds from uncontrolled pages (Wmid, wtarget) is not in ctm. At test time, each probe is generated by Eq. 1 from ctm and delivered on the controlled page, and the attacker observes only the agent's behavior on that page after delivery. We will rephrase "the agent context accumulated up to that page" (Eq. 1 and Appendix F.1) to define ctm as the attacker-controlled-page portion of ct.
The generator does not use the victim agent's actions on other sites, the agent's reasoning, or the user's task. As defined above, ctm contains no actions on other sites. The generator is conditioned on the objective G rather than the task T (Sec. 3.2), so it uses neither the user's task nor its progress. At test time it does not use the agent's reasoning either. Its only feedback is the agent's actions logged on the attacker's own pages (ctm), such as the confirmatory action. One caveat: the Scorer description and prompt (Appendices B and F.2) can also credit probe mentions in the agent's reasoning, which conflicts with Appendix J. We will make them consistent. The Scorer runs only in the offline data collection phase (Sec. 3.2, Appendix H) and is never used at test time.
Reviewer: "Section 3.2 says the frozen prober at test time is conditioned on 'the agent's actions observed so far,' but does not specify which actions, or how a webpage owner would observe them."
Response: The actions are confirmatory actions, e.g., clicking a non-critical element or icon whose identifier matches the injected content (Appendix J). A page owner records them with standard event listeners and server logs on its own page, and the Scorer reads only "the resulting trajectory segment" on that page (Sec. 3.2). So "the agent's actions observed so far" means the agent's actions recorded on earlier controlled pages (ctm), not its full history. We will reword Sec. 3.2 accordingly and restrict {ai<t} in Sec. 4.3 and the perturbation description to ctm.
Reviewer: "According to these details, the generator requires more information than the stated attacker has in the threat model. The paper does not clarify that its reported success rates are achievable under the settings in Section 3.1."
Response: At test time, the attack stays within the threat model of Sec. 3.1: every input it uses is available to the attacker (see Q1), and the reported results use only these inputs. The closed loop with environment resets (Appendix H) is an offline sandbox step that only labels training data. Its Scorer checks (e.g., continued progress toward the trigger page, Appendix J) are not test-time inputs. At test time the frozen prober makes one pass over the controlled pages, with no resets and no Scorer.
Q1: "What exact inputs does the frozen prober receive at test time, and how would a webpage owner obtain each one?"
Response: The prober receives four inputs, and the page owner has each of them: (i) the strategy sm, from the attacker's own library, (ii) the objective G and target-page description, which are the attacker's own goal, (iii) the previous probe pm−1, the attacker's own output, and (iv) ctm, the agent's actions logged on earlier controlled pages via standard event listeners and server logs on its own pages, as we will state in Sec. 3.1.
Q2: "Can the authors report results using only webpage content and events from attacker-controlled pages, without access to other activities?"
Response: Yes. The frozen prober receives only the inputs in Q1, so the main results (Tables 1-4) are already obtained using only webpage content and events from attacker-controlled pages.