STRAP: STrategy-Driven Red-Teaming of Multimodal Web Agents via Latent Sequential Probe Injection

Multimodal web agents retain browsing context, allowing content encountered on one page to influence actions on later pages. We study delayed prompt injection in a black-box setting where an attacker modifies earlier webpages but has no access to the user's task instruction or control over the target page. We introduce STRAP, a strategy-driven red-teaming framework that coordinates page-level probes to induce an adversarial action after intervening uncontrolled navigation. STRAP uses behavioral feedback to refine attack strategies and train a probe generator through supervised fine-tuning and preference optimization; at deployment, the generator conditions successive probes on observed interactions. A visual extension pairs text probes with adversarial image perturbations optimized using a proxy model. Across three task domains and three proprietary vision-language backends in WebVoyager, STRAP achieves 91.2–95.5% attack success with three controlled pages and three intervening uncontrolled pages, exceeding the strongest adapted baseline by approximately 40.0 percentage points on average across the nine settings under matched page budgets. Under four evaluated defenses on GPT-4o, attack success remains 74.8–84.5%. These results highlight the need to limit how untrusted webpage content influences subsequent agent decisions.