acceptodds
Under review as a conference paper at ICLR 2027

BrowseSafe++: Benchmarking Prompt Injection Detection Across Web Contexts, Languages, and Browser Views

Abstract

Browser agents need to make decisions and execute actions on behalf of users while consuming untrusted webpages. This creates the chance for malicious instructions embedded in webpages to redirect legitimate tasks. Existing benchmarks establish this threat, but broader evaluation requires consistent coverage of the web domains, languages, and page views encountered in deployment. We introduce BrowseSafe++, extending BrowseSafe from 14,719 to 38,675 entries. We start with prompt injections observed in the wild from production website captures, preserve the original injections, and rewrite sensitive portions to protect privacy. Each entry pairs an attacked page with a control page that differs only by the injection, provides HTML, Markdown, accessibility tree, and screenshot views, and is linked to a legitimate task and expected outcome. The benchmark contains twelve domains, twenty-four attack types, and eighteen placements. Using these pairs as supervision, we train BrowseSafe-Guard, compact 0.8B text and multi-modal detectors that outperform frontier models while raising far fewer false alarms. We also introduce BrowseSafe-Adapt, which refines benchmark injections from browser-agent feedback and shows that attacks succeeding on a few percent of tasks when replayed succeed on up to 95% after adaptation. The benchmark, model, and SDK for adaptive framework will be released upon publication.

Primary area:
datasets and benchmarks
Keywords:
Indirect Prompt Injection, Browser Agents, Web Agents, AI Security, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_DhMFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BrowseSafe++, a benchmark for prompt injection detection in web agents. The proposed benchmark contains 38,675 attack/control pairs across different web domains, languages, and page views. The paper constructs the benchmark using prompt injections observed on real-world webpages and existing BrowseSafe samples. They also train BrowseSafe-Guard, a lightweight 0.8B detection model using pairwise supervision, and propose BrowseSafe-Adapt, an adaptive attack framework that refines injections based on agent feedback. Experiments show strong detection performance and substantially higher attack success rates after adaptive refinement.

Critical strengths
  1. The paper introduces a large-scale prompt injection benchmark covering diverse web domains, languages, attack types, and page views.

  2. The proposed BrowseSafe-Guard achieves strong detection performance with a relatively small 0.8B model and low false positive rates.

  3. The paper evaluates adaptive attacks against multiple agent models and shows that static attack evaluation can underestimate attack success rates.

Critical weaknesses
  1. The generalization ability of BrowseSafe-Guard is not sufficiently evaluated. The model is trained and tested on different splits of BrowseSafe++, which share the same data construction pipeline and similar attack distributions. Although identical injection texts are separated between training and testing, this does not guarantee generalization to other datasets. Evaluating BrowseSafe-Guard on independent benchmarks such as WASP, VPI, or WAInjectBench would provide stronger evidence.

  2. The paper lacks an ablation study on the pairwise ranking loss used to train BrowseSafe-Guard. The authors combine binary cross-entropy with a pairwise ranking loss, but do not clearly show how much the additional loss contributes to detection performance. Since the attack and control pairs are a key part of the proposed method, comparing BCE-only training with BCE plus pairwise ranking loss would help justify this design.

  3. The practical effectiveness of BrowseSafe-Guard in real agent workflows remains unclear. The paper reports strong detection accuracy and low false-positive rates, but does not sufficiently evaluate how deploying the guard affects actual agent task completion. Even a relatively low false-positive rate may cause legitimate actions to be blocked during multi-step tasks. An end-to-end evaluation measuring both attack prevention and task utility would help demonstrate the practical value of the proposed detector.

  4. Finally, a minor concern regarding the LLM rewriting. Although the paper claims to use prompt injections observed in real-world webpages, the surrounding webpage content is largely rewritten using LLMs. Such rewriting may change the original context and the relationship between the injection and the legitimate task. It remains unclear whether these reconstructed pages preserve the behavior of the original attacks. An evaluation comparing attack effectiveness before and after rewriting would help support the realism of the benchmark.

Questions for authors

Please refer to the weaknesses above.

5 comments

We thank the reviewer for their feedback and constructive suggestions. We appreciate that the reviewer recognized the broad coverage of BrowseSafe++, the strong detection performance of our lightweight 0.8B model, and the importance of evaluating adaptive attacks beyond static benchmarks. We will address the concerns regarding generalization, the contribution of pairwise supervision, practical deployment, and the realism of rewritten webpages. Revisions to the paper are highlighted in blue and will be updated on Nov 17.


W1: Generalization to Independent Benchmarks

"The generalization ability of BrowseSafe-Guard is not sufficiently evaluated. The model is trained and tested on different splits of BrowseSafe++, which share the same data construction pipeline and similar attack distributions. Although identical injection texts are separated between training and testing, this does not guarantee generalization to other datasets. Evaluating BrowseSafe-Guard on independent benchmarks such as WASP, VPI, or WAInjectBench would provide stronger evidence."

We evaluated Guard on WAInjectBench without any further training. WAInjectBench is a collection of attacks from earlier work and includes the attack texts of WASP and VPI, so it covers the three benchmarks the reviewer names. Guard generalizes to them. On the attacks that give the agent an instruction, it has the highest F1 of all trained detectors.

DetectorWASP recall (%)VPI recall (%)F1 on all WAInjectBench attacks with an explicit instruction
Llama Prompt Guard 2 (86M)0.00.04.3
PIGuard100.00.062.7
ProtectAI DeBERTa-v3 (v2)83.342.172.0
BrowseSafe-Guard71.4100.082.3

Guard also raises the fewest false alarms on benign data from other sources. NotInject is a set of benign prompts that contain trigger words.

DetectorFPR on WAInjectBench benign items (%)FPR on NotInject (%)
Llama Prompt Guard 2 (86M)0.24.7
PIGuard2.011.5
ProtectAI DeBERTa-v3 (v2)2.543.4
BrowseSafe-Guard0.10.6

By our definition, an injection is an instruction in page content that tries to redirect the agent (Appendix B), and this is what Guard is built to detect. WAInjectBench also contains attacks with no instruction to the agent, such as pop-ups and misleading form labels. These are outside our definition, and Guard is not built to flag them. We will state this scope in the paper.

The reviewer also notes that training and test pages share one construction pipeline. Guard also works on production pages that our pipeline never rewrote: 415 pages collected from November 1 to 10, 2026, after the benchmark was built. Guard finds as many of their injections as on rewritten pages in Markdown, and 7.0 points fewer in HTML.

PagesHTML recall (%)Markdown recall (%)
Rewritten pages89.485.0
Raw production pages82.484.9

This test uses a Guard trained at a smaller scale than the one in Table 2, so both rows are lower than for the Guard in Table 2.

Guard also generalizes to attacks it was not trained on. We train it without one part of the data and test it on that part: injections observed from April 2026 on, six of the twelve domains, or 22 of the 24 attack types. We compare it with a Guard trained on all data and tested on the same pages. Each held-out part has its own test pages, so the F1 of the Guard trained on all data differs between parts.

Held-out partTraining dataHTML F1Markdown F1Accessibility tree F1
Later injectionsAll periods99.292.993.3
Later injectionsEarlier periods only99.393.191.1
Six domainsAll domains95.588.989.1
Six domainsOther six domains only96.388.087.7
22 attack typesAll types93.586.487.1
22 attack typesTwo types only92.183.681.4

W2: Ablation of the Pairwise Ranking Loss

"The paper lacks an ablation study on the pairwise ranking loss used to train BrowseSafe-Guard. The authors combine binary cross-entropy with a pairwise ranking loss, but do not clearly show how much the additional loss contributes to detection performance. Since the attack and control pairs are a key part of the proposed method, comparing BCE-only training with BCE plus pairwise ranking loss would help justify this design."

We ran this ablation. We trained Guard three times with BCE only and three times with BCE plus the pairwise ranking loss, changed nothing else, and tested both on the 9,175 test pairs. All six runs use a smaller training scale than the Guard in Table 2, so their F1 is lower than in Table 2. The table gives the mean over the three training runs.

ViewTraining lossF1Recall at 1% FPR (%)
HTMLBCE only88.760.1
HTMLBCE + pairwise ranking loss89.363.5
MarkdownBCE only86.046.0
MarkdownBCE + pairwise ranking loss85.950.3
Accessibility treeBCE only86.965.4
Accessibility treeBCE + pairwise ranking loss87.567.1

At the default threshold, the F1 is about the same with both losses. The benefit of the pairwise ranking loss is at low false-positive rates, where a deployed detector operates: at 1% FPR, Guard detects 1.7 to 4.3 points more attacks with it, in every view. We will add this ablation to the appendix.

W3: Practical Effectiveness and Agent Task Utility

"The practical effectiveness of BrowseSafe-Guard in real agent workflows remains unclear. The paper reports strong detection accuracy and low false-positive rates, but does not sufficiently evaluate how deploying the guard affects actual agent task completion. Even a relatively low false-positive rate may cause legitimate actions to be blocked during multi-step tasks. An end-to-end evaluation measuring both attack prevention and task utility would help demonstrate the practical value of the proposed detector."

We ran this end-to-end evaluation on 1,024 tasks. Guard is placed in front of an agent (Gemini 3.5 Flash-Lite) and checks the Markdown view of every page before the agent reads it. A flagged page is replaced with a notice that it was blocked, and the agent continues with the task. Guard blocks 94.53% of the pages that carry an observed injection.

SettingAttack success of the observed injection (%)Clean-task completion (%)
Agent alone4.0097.85
Agent + Guard1.3791.21

With Guard, the observed injection succeeds on 1.37% of the tasks instead of 4.00%. The cost is 6.64 points of clean-task completion, because Guard flags 6.93% of clean pages. In a multi-step task the agent reads several pages, and one wrongly blocked page can stop the task.

Our experiment also uses Guard in the strictest way: a flagged page is blocked outright. Real products do not rely on a detector alone. They use it as one layer of a layered defense. Google describes this for Gemini, where prompt-injection classifiers are combined with model hardening and user confirmation [1], and Microsoft describes a defense-in-depth approach that combines detection with prevention and impact mitigation [2]. In such a cascade, a flag from the detector triggers the next layer, for example a second check or a confirmation by the user. A false positive then costs one extra step, not the task.

We will add this evaluation and this discussion to the paper.

References

  • [1] Google. Mitigating prompt injection attacks with a layered defense strategy. Google Security Blog, June 2025.
  • [2] Microsoft Security Response Center. How Microsoft defends against indirect prompt injection attacks. July 2025.
Reviewer_pVfpOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends BrowseSafe into BrowseSafe++, a prompt-injection detection benchmark with 38,675 attack/control pairs (77,350 pages), of which 32,187 carry injections observed on production webpages collected over 386 days (September 1, 2025 to September 21, 2026, 1,152,315 candidate pages, 603,000 flagged by a deployed classifier). Each entry pairs an attacked page with a control page that differs only by the injection, provides four page views (HTML, Markdown, accessibility tree, full-page screenshot), is linked to a legitimate user task and expected outcome, and is annotated with domain (12 types), attack type (24), placement (18), writing tone (4), risk category, attacker goal, and the languages of both the page and the injection (24.4% of pairs have an injection in a different language from the page). The construction pipeline (Figure 3) is seven-stage with explicit attrition: injection identification via a 3-model committee (GPT-5.5, Opus 4.8, Gemini 3.8 Flash, 4-of-6 vote) keeps 58,036 of 1,092,957 production pages (5.3%), page rewriting (Haiku 4.5 and Sonnet 4.6, reviewed by Sonnet 5) keeps 48,092, and multi-view audit + final review keep 32,187 production and 6,488 BrowseSafe pairs. The paper then uses the pairs for two applications. BrowseSafe-Guard is a 0.8B Qwen3.5-based classifier fine-tuned with a pair-wise ranking loss (BCE + Softplus margin between attack and control page scores) across HTML/Markdown/A11y views (one model) and screenshots (one model). It reaches 93.0-98.3 F1 at 1.4-7.4% FPR against GPT-5.5's 89.3-89.7 F1 at 19.1-21.0% FPR (Table 2) and 93.3 F1 at 6.1% FPR on screenshots against the best prompted multimodal at 78.7/28.1% (Table 3). Three secondary empirical findings are consequential: (1) model rankings change across page views (Kendall tau = 0.79 HTML-vs-Markdown), with Markdown costing 13.7-15.9 recall points on preference overrides, tool lures, and role manipulation vs 4.2 on ignore-previous; (2) screenshot detection costs 4.5-40.9 F1 relative to a model's best text view, and the modality cost varies by model (Flash-Lite loses 4.5 F1 while GPT-5.4 nano loses half); (3) detection does not imply resistance: 69.4% of injections that models follow are injections the same model flags when asked, and Sonnet 5 flags and follows the injection on 19.7% of attacked pages. BrowseSafe-Adapt is an adaptive attack framework (Figure 4) that starts from each entry's seed injection and revises it from execution feedback within a 64-attempt budget using quality-diversity search over strategy, agent-behaviour, and failed-condition groups. Across 7 target agents on 1,024 tasks each, seed ASR is 0.5-4.0% and adapted ASR is 30.7-95.4% within 64 attempts: Sonnet 5 at 30.7%, GPT-5.5 at 65.2%, Haiku 4.5 at 78.8%, Gemini 3.1 Pro at 92.9%, Gemini 3.5 Flash-Lite 93.6%, DeepSeek V3.1 95.0%, Kimi K2 95.4%. Vulnerable tasks are shared: 284 of 1,024 tasks (27.7%) succeed against all 7 targets. The paper will release the benchmark, Guard models, and Adapt SDK on publication.

Critical strengths

1. The attack/control pair design is a real methodological improvement over prior IPI benchmarks, and the paper directly operationalises what it buys. Prior IPI benchmarks (BIPIA, InjecAgent, AgentDojo, ASB, WAInjectBench, DTap-Bench, BrowseSafe v1, Table 1) use separately collected benign and attack pages, so a classifier can learn topic, style, or source differences instead of the injection itself. BrowseSafe++ eliminates this confound by construction: for each pair, attack and control differ only by the injection text, share the same task, task-relevant facts, DOM structure, CSS, and layout. The paper then uses this directly for two things that would be hard without pairs: (a) the Softplus ranking loss in Eq. 1 can only be lowered by responding to the injection because the two pages are otherwise identical, giving the detector a direct injection-specific signal rather than a page-level discriminator, and (b) at test time, every attack page has a benign counterpart with aligned content, so the measured 1.4-7.4% FPR on BrowseSafe-Guard is a per-page false-alarm rate on the very page that carries the attack, not a cross-page benign confound. The empirical result that Guard reaches 93.0-98.3 F1 at 1.4-7.4% FPR against GPT-5.5's 19.1-21.0% FPR is the paper's strongest single data point, and it is directly attributable to the pair design.

2. The paper documents three counterintuitive empirical findings that are concrete, portable, and will change how the field evaluates IPI defenses. First, detection does not imply resistance: across 10 models, 69.4% of injections that get followed during the task call are injections the same model flags when classifying, and Sonnet 5 flags-and-follows on 19.7% of attacked pages while GPT-5.5 is 2.0% (Figure 5c). This is a direct quantitative refutation of the implicit assumption behind detector-only benchmarks that strong detection implies safe agentic behaviour. Second, the model ranking changes across page views (Kendall tau = 0.79 HTML-vs-Markdown) and the view-cost is attack-type-dependent (13.7-15.9 recall points on preference overrides, tool lures, and role manipulation vs 4.2 on ignore-previous), so a leaderboard built on a single view does not transfer, and screening across any-of-three views raises GPT-OSS recall from 73.7% to 87.7%. Third, static replay dramatically understates adaptive risk: observed injections succeed on 0.5-4.0% of tasks seeded, but within a 64-attempt budget BrowseSafe-Adapt reaches 92.9-95.4% against 4 of 7 production agents including Gemini 3.1 Pro. The last two columns of Table 4 further split by goal and show that GPT-5.5 and Sonnet 5 remain comparatively resistant on tasks that require application-state change (28.5% and 7.9%) versus answer-only tasks (81.4% and 40.6%), which is a useful operational finding: declining to act is a meaningful last-ditch defense even when the agent is persuaded.

Critical weaknesses

1. The benchmark construction is heavily LLM-filtered with the same model families that are later evaluated as detectors, creating a validity-threatening circularity that the paper does not test. Stage 3 (injection identification) uses a committee of GPT-5.5, Claude Opus 4.8, and Gemini 3.8 Flash with a 4-of-6 vote gate, keeping only 58,036 of 1,092,957 production pages (5.3% retention). Stage 4 (rewrite) uses Claude Haiku 4.5 (with Sonnet 4.6 retries) and Claude Sonnet 5 for review, and Stage 6 (multi-view audit) uses Claude Sonnet 5 again. The Table 2 detectors include GPT-5.5 and Claude Sonnet 5, which are either the same model or the same family as the construction committee. This means (a) the test set is pre-filtered to injections that these LLM committees already recognise as operative, so Table 2's 89.3-89.7 F1 for GPT-5.5 and 87.4-89.5 for Sonnet 5 is a floor rather than a representative estimate (what counts as an injection is defined by their majority), and (b) the 94.7% of production pages excluded by Stage 3 contains unknown amounts of injections that the committee could not recognise. For a benchmark whose primary selling point is wild attacks observed in production, the paper owes either a human-only identification of a validation subset or an explicit committee-performance baseline, and currently gives neither. Human review is reported for Stage 4 ("about 10% of pages") and Stage 6 (independent reviewers for presence-in-every-view), but there is no human-only labeling of a stage 3 subset that could isolate the committee bias. This is a known problem in LLM-constructed benchmarks and does not disqualify the paper, but it caps what the Table 2 numbers establish about detector capability in the wild.

2. BrowseSafe-Guard is trained and tested on the same construction pipeline with no cross-benchmark transfer, so the 93.0-98.3 F1 cannot be read as general-purpose detector performance. The 29,500 training pairs and 9,175 test pairs come from the same seven-stage pipeline. The pair-ranking loss directly matches the pair structure of the test set, which gives Guard a distributional advantage no prompted model can share. The paper does not evaluate Guard on BIPIA, WAInjectBench, DTap-Bench, WASP, AgentDojo, or any BrowseSafe++-external IPI benchmark, so the question of whether Guard generalises or is a BrowseSafe++-specific predictor is open. A related concern: the "injection verbatim, surrounding content rewritten" design (Stage 4) means every production injection sits inside LLM-rewritten text while every control page is also fully LLM-rewritten, so the injection vs non-injection distinction co-varies with "production text pasted into LLM-generated page" vs "pure LLM-generated page". A detector could learn the style discontinuity rather than the injection semantics, and Guard's 98.3 F1 on HTML is high enough to be consistent with that. A single evaluation on an unseen benchmark, or on a held-out subset of pages with pre-rewrite controls, would resolve both concerns and would plausibly raise this paper from a Weak accept to a Clear accept.

Questions for authors
  1. Committee bias check. Can you report, on a stratified random sample of say 500 pages from the 94.7% that Stage 3 rejected, how many a human labeler would judge to contain an operative injection? This is the single most important validity check for the "wild-attack benchmark" claim, and it fits in the author response window.

  2. Guard cross-benchmark transfer. Can you report BrowseSafe-Guard's F1 and FPR on at least one external IPI benchmark that it was not trained on, such as BIPIA, DTap-Bench, or WAInjectBench? Any single number here would significantly strengthen the "outperforms frontier models" claim.

  3. Guard vs Adapt interaction. BrowseSafe-Adapt reaches 92.9-95.4% ASR against four production agents. Can you report the same ASR when BrowseSafe-Guard is placed in front of the target agent as a gating detector? If Guard breaks the attack chain, this is the paper's strongest practical result. If it doesn't, that is also an important finding and belongs in Section 4.3.

  4. Style confound. On the control page, the surrounding content is LLM-rewritten and no injection is present. On the attack page, the surrounding content is LLM-rewritten and a verbatim-production injection is inserted. A detector could in principle learn to flag "verbatim-production-style text pasted into LLM-generated text" as injection, independent of injection semantics. Have you measured Guard's performance on a held-out subset where (a) the control's surrounding content is production-captured instead of LLM-rewritten, or (b) the injection is paraphrased by the same LLM that rewrote the surroundings?

  5. Per-language performance. You report that 24.4% of pairs have an injection in a different language from its page, and Figure 2 shows seven non-English languages per side. Tables 2 and 3 pool across all languages. Could you report, in the appendix or main text, Guard's F1 broken out as English-page/English-injection vs mixed-language vs fully non-English? For a paper that emphasises multilingual coverage, this is the natural follow-up and the data already exist.

5 comments

We thank the reviewer for their detailed feedback and constructive suggestions. We are encouraged that the reviewer recognized our paired attack/control design as a "real methodological improvement" and highlighted the practical importance of our findings on detection versus resistance, page-view differences, and adaptive attack success. We will address the concerns regarding possible biases in the benchmark construction pipeline, generalization of BrowseSafe-Guard, and the practical implications of our results. Revisions to the paper are highlighted in blue and will be updated on Nov 17.


W1 & Q1: LLM Committee Bias and Missed Production Injections

W1: "The benchmark construction is heavily LLM-filtered with the same model families that are later evaluated as detectors, creating a validity-threatening circularity that the paper does not test."

"The test set is pre-filtered to injections that these LLM committees already recognise as operative, so Table 2's 89.3-89.7 F1 for GPT-5.5 and 87.4-89.5 for Sonnet 5 is a floor rather than a representative estimate [...] and the 94.7% of production pages excluded by Stage 3 contains unknown amounts of injections that the committee could not recognise."

Q1: "Can you report, on a stratified random sample of say 500 pages from the 94.7% that Stage 3 rejected, how many a human labeler would judge to contain an operative injection? This is the single most important validity check for the 'wild-attack benchmark' claim, and it fits in the author response window."

We tested the circularity in two ways. First, we added more detectors to Table 2, among them larger frontier models and models from families that took no part in construction. If the benchmark favored the families used in construction, their models would score above the others. They do not: in every view, Qwen 3.7 Max and Jev 1.13, which come from other families, score inside the range of the five models from the families used in construction.

DetectorFamily used in constructionHTML F1Markdown F1Accessibility tree F1
GPT-6 AstraYes91.691.391.5
GPT-5.5Yes89.389.089.7
Gemini 3.1 ProYes88.990.190.7
Claude Sonnet 5Yes88.289.587.4
Claude Opus 5.5Yes86.584.984.4
Qwen 3.7 MaxNo89.089.388.7
Jev 1.13No89.287.089.1

Second, we ran the audit the reviewer asks for. Two human annotators independently labeled a stratified random sample of 500 pages that Stage 3 rejected. They saw only the page, and no model output.

Pages rejected by Stage 3Judged injected by annotator 1Judged injected by annotator 2Judged injected by both
50011 (2.2%)24 (4.8%)0 (0.0%)

The two annotators never flagged the same page: every page that one of them flagged, the other judged benign. The sample contains no injection that both annotators recognize.

Most pages that Stage 3 rejects are not attacks at all. The classifier and the cues are built to miss little, so they also select many pages that only mention instructions to an AI. For 76% of the rejected pages, Stage 3 recorded one of four reasons, and none of them is an instruction to the agent that reads the page.

Reason recorded by Stage 3 for rejecting the pageShare of rejected pages (%)
Documentation or a tutorial about prompts24.5
A notice addressed to human visitors18.7
Code or a configuration file18.1
A chat transcript or system prompt shown as page content15.1

An injection that Stage 3 missed is absent from the benchmark. It does not change the label of any page in the benchmark, so the wild-attack claim stands: each of the 32,187 production pairs carries an injection observed on a production page.

W2 & Q2: Cross-Benchmark Generalization of BrowseSafe-Guard

W2: "BrowseSafe-Guard is trained and tested on the same construction pipeline with no cross-benchmark transfer, so the 93.0-98.3 F1 cannot be read as general-purpose detector performance. The 29,500 training pairs and 9,175 test pairs come from the same seven-stage pipeline. The pair-ranking loss directly matches the pair structure of the test set, which gives Guard a distributional advantage no prompted model can share."

Q2: "Can you report BrowseSafe-Guard's F1 and FPR on at least one external IPI benchmark that it was not trained on, such as BIPIA, DTap-Bench, or WAInjectBench? Any single number here would significantly strengthen the 'outperforms frontier models' claim."

We ran Guard on two external benchmarks, without any further training. WAInjectBench is the benchmark the reviewer names that is built for web agents, with 354 attacks and 2,707 benign items. NotInject is a set of 339 benign prompts that contain trigger words. For comparison, we ran three public trained detectors on the same items.

DetectorWAInjectBench F1WAInjectBench recall (%)WAInjectBench FPR (%)NotInject FPR (%)
BrowseSafe-Guard82.370.30.10.6
ProtectAI DeBERTa-v3 (v2)72.066.92.543.4
PIGuard62.752.52.011.5
Llama Prompt Guard 2 (86M)4.32.30.24.7

On WAInjectBench, Guard reaches an F1 of 82.3 and flags only 2 of the 2,707 benign items. On NotInject, its FPR is 0.6%. Of the four detectors, Guard has the highest F1 on WAInjectBench and the lowest FPR on both benchmarks.

BrowseSafe-Guard is a detector for prompt injections on web pages, and we evaluate it as one. By our definition, an injection is an instruction in page content that tries to redirect the agent (Appendix B). The 354 WAInjectBench attacks are the ones that match this definition: the four subsets whose attacks give the agent an explicit instruction (WASP, VPI web text, and the "with explicit instruction" parts of EIA and VWA-adv). WAInjectBench also contains attacks with no instruction to the agent, such as pop-ups and misleading form labels. These are not injections under our definition, and Guard is not built to flag them. We will state this scope in the paper.

The pairwise loss gives Guard no advantage at test time. Guard scores one page at a time and never sees the page's pair (Section 3.1).

W2 & Q4: LLM-Rewriting Style Confound

W2: "The 'injection verbatim, surrounding content rewritten' design (Stage 4) means every production injection sits inside LLM-rewritten text while every control page is also fully LLM-rewritten, so the injection vs non-injection distinction co-varies with 'production text pasted into LLM-generated page' vs 'pure LLM-generated page'. A detector could learn the style discontinuity rather than the injection semantics, and Guard's 98.3 F1 on HTML is high enough to be consistent with that."

Q4: "On the control page, the surrounding content is LLM-rewritten and no injection is present. On the attack page, the surrounding content is LLM-rewritten and a verbatim-production injection is inserted. A detector could in principle learn to flag 'verbatim-production-style text pasted into LLM-generated text' as injection, independent of injection semantics. Have you measured Guard's performance on a held-out subset where (a) the control's surrounding content is production-captured instead of LLM-rewritten, or (b) the injection is paraphrased by the same LLM that rewrote the surroundings?"

We ran both tests the reviewer names.

In (b), the rewriting LLM also paraphrases the injection, so the injection and the page are written by the same model. Guard detects these injections as often as the original ones, so it does not rely on a style difference. In (a), we use 415 production pages that were never rewritten, collected from November 1 to 10, 2026, after the benchmark was built. Guard finds as many of their injections as on rewritten pages in Markdown, and 7.0 points fewer in HTML.

TestHTML recall (%)Markdown recall (%)
Rewritten pages, original injection89.485.0
(b) Rewritten pages, injection paraphrased by the rewriting LLM90.986.4
(a) Raw production pages82.484.9

All rows use the same Guard, trained at a smaller scale than the one in Table 2, so all are lower than for the Guard in Table 2.

Reviewer_tQHTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces BrowseSafe++, an extension of BrowseSafe containing 38,675 attack/control webpage pairs, including 32,187 injections collected from production webpages. The original injection text is preserved, while the surrounding content is rewritten to remove sensitive information. Each pair includes four page views (HTML, Markdown, accessibility tree, and screenshot), along with task and attack annotations. The authors also introduce BrowseSafe-Guard, a lightweight prompt injection detector, and BrowseSafe-Adapt, an adaptive attack framework. Experiments show strong detection performance for BrowseSafe-Guard and substantially higher attack success rates after adaptation.

Critical strengths
  1. The benchmark includes a large number of real-world prompt injections across different domains and languages.

  2. The paired attack/control design is a nice idea, as it allows a more direct comparison between malicious and benign pages.

  3. The experiments cover different page views and adaptive attacks, providing a fairly comprehensive evaluation.

Critical weaknesses
  1. Candidate production pages are selected by a deployed LLM-based classifier and regular-expression cues, and then confirmed by a committee that includes GPT-5.5, which is later evaluated as a detector. Claude Sonnet 5, also an evaluated detector, must recognize the injection in every page view during validation, and pairs it fails on are relocated or removed. Injections that these models find difficult may therefore be under-represented, which could inflate the reported detection performance of these and similar models.

  2. The paper requires injections to appear in all four views, including screenshots, but the dataset also contains hidden placements such as HTML comments and data attributes. It is unclear how these injections remain represented across views without changing their original placement or visibility. This may limit the benchmark's coverage of realistic hidden injections.

  3. The model is mainly compared against prompted LLMs rather than existing trained detection models. There is also no ablation of the pairwise ranking loss, despite it being an important component of the proposed approach. Moreover, evaluation on an independent dataset or a temporal split would help establish whether the strong results generalize beyond the benchmark's construction pipeline.

  4. BrowseSafe-Adapt achieves impressive attack success rates, but there is no comparison against simpler alternatives under the same attack budget. Without ablations of the critic, archive, and revision strategies, it is difficult to determine which components contribute to the improvements.

Questions for authors
  1. How does the benchmark account for injections missed by the classifiers and LLMs used during data collection? Was any human audit performed on unflagged pages?
  2. How are hidden injections (e.g., HTML comments, data attributes) represented in all four views, especially screenshots? Do the placement labels reflect the original or the relocated position?
  3. How does BrowseSafe-Guard compare with existing trained detectors, and what does the pairwise loss contribute? Do the results hold on an independent dataset or a temporal split?
  4. How does BrowseSafe-Adapt compare with simpler revision strategies under the same budget, and does BrowseSafe-Guard detect the adapted injections?
  5. Per model, how does the follow rate differ between flagged and unflagged pages?
6 comments

We thank the reviewer for their feedback and constructive suggestions. We appreciate that the reviewer recognized the large-scale real-world coverage of BrowseSafe++, the value of our paired attack/control design, and the comprehensive evaluation across different page views and adaptive attacks. We will clarify the benchmark construction process and provide further analysis of the detection models and adaptive attack framework. Revisions to the paper are highlighted in blue and will be updated on Nov 17.


W1 & Q1: Selection Bias from LLM-Based Filtering

W1: "Candidate production pages are selected by a deployed LLM-based classifier and regular-expression cues, and then confirmed by a committee that includes GPT-5.5, which is later evaluated as a detector. Claude Sonnet 5, also an evaluated detector, must recognize the injection in every page view during validation, and pairs it fails on are relocated or removed. Injections that these models find difficult may therefore be under-represented, which could inflate the reported detection performance of these and similar models."

Q1: "How does the benchmark account for injections missed by the classifiers and LLMs used during data collection? Was any human audit performed on unflagged pages?"

BrowseSafe++ contains only the injections that our pipeline found, and we will state this limit in Section 2.2. No human audit was performed on unflagged pages, because it is not doable at this scale: the deployed agent read far more pages in 386 days than people can read, and injections are rare. Even among the 1,092,957 candidate pages, only 5.3% contain one.

We did audit the pages where injections missed by the LLMs would be: the pages that the Stage 3 committee rejected. Two human annotators independently labeled a stratified random sample of 500 of these pages. They saw only the page, and no model output.

Pages rejected by Stage 3Judged injected by annotator 1Judged injected by annotator 2Judged injected by both
50011 (2.2%)24 (4.8%)0 (0.0%)

The two annotators never flagged the same page: every page that one of them flagged, the other judged benign. The sample contains no injection that both annotators recognize.

Most pages that Stage 3 rejects are not attacks at all. For 76% of them, Stage 3 recorded that the page is documentation about prompts, a notice addressed to human visitors, code or a configuration file, or a chat transcript shown as page content.

A missed injection is absent from the benchmark. It does not change any page in it. Even with this limit, BrowseSafe++ has more injections than the benchmarks below, and it is the only one whose injections were observed in the wild.

BenchmarkWhat is countedCountObserved in the wild
BIPIAAttacker goals250No
AgentDojoSecurity test cases629No
WAInjectBenchMalicious text segments991No
InjecAgentTest cases1,054No
BrowseSafeMalicious entries7,297No
BrowseSafe++Distinct injections36,985Yes

The selection also does not inflate the reported performance of GPT-5.5 and Claude Sonnet 5. In construction, these models confirm a suspected injection. In the evaluation, they must detect one on an unknown page, with BrowseSafe's detection prompt. Table 2 shows that a page that passed construction is not an easy page in the evaluation. Stage 6 kept a production pair only if Claude Sonnet 5 judged its control page benign in every view. In the evaluation, Sonnet 5 still flags 18.4% of these same control pages in Markdown and 25.7% in the accessibility tree. Models from families that took no part in construction also reach the same F1: in HTML, 89.0 for Qwen 3.7 Max and 89.2 for Jev 1.13, against 89.3 for GPT-5.5 and 88.2 for Sonnet 5.

W2 & Q2: Hidden Injections Across Different Page Views

W2: "The paper requires injections to appear in all four views, including screenshots, but the dataset also contains hidden placements such as HTML comments and data attributes. It is unclear how these injections remain represented across views without changing their original placement or visibility. This may limit the benchmark's coverage of realistic hidden injections."

Q2: "How are hidden injections (e.g., HTML comments, data attributes) represented in all four views, especially screenshots? Do the placement labels reflect the original or the relocated position?"

Hidden injections are outside the scope of BrowseSafe++. An injection can only attack an agent that sees it. An injection in an HTML comment or a data attribute exists only in the raw HTML. It is not in the Markdown view, the accessibility tree, or the screenshot. Browser agents increasingly read these views instead of raw HTML: for example, AgentOccam reads the accessibility tree and UI-TARS reads screenshots (Section 1). Such an agent never sees a hidden injection, so we do not count it as an effective attack against today's agents.

The hidden placements in BrowseSafe++ come almost entirely from BrowseSafe entries as shown in Figure 2. The injections we observed in production are visible on the page: only 66 of the 32,187 production pairs (0.2%) carry a hidden placement label.

Placement labelPairs from BrowseSafePairs from production
HTML comment2397
Data attribute67834
Hidden text56125

For the pairs with a hidden placement, we move the injection text to a visible place on the page, so that all four views show it. The placement label is the original position. We will correct it in the benchmark release.

W3 & Q3: Trained Detector Baselines, Pairwise Loss, and Generalization

W3: "The model is mainly compared against prompted LLMs rather than existing trained detection models. There is also no ablation of the pairwise ranking loss, despite it being an important component of the proposed approach. Moreover, evaluation on an independent dataset or a temporal split would help establish whether the strong results generalize beyond the benchmark's construction pipeline."

Q3: "How does BrowseSafe-Guard compare with existing trained detectors, and what does the pairwise loss contribute? Do the results hold on an independent dataset or a temporal split?"

At their default thresholds, existing trained detectors perform poorly on full web pages. We ran four public prompt-injection detectors on the same test pages as Guard, at their default thresholds. Two of them flag almost every page, and the other two miss most injections.

DetectorHTML F1HTML FPRMarkdown F1Markdown FPRAccessibility tree F1Accessibility tree FPR
Llama Prompt Guard 2 (22M)16.41.120.72.922.61.7
Llama Prompt Guard 2 (86M)60.010.758.717.163.113.5
ProtectAI DeBERTa-v3 (v2)66.7100.062.164.766.596.7
PIGuard66.799.766.995.467.790.0
BrowseSafe-Guard 0.8B98.31.493.37.493.05.8

The pairwise ranking loss helps at low false-positive rates, where a deployed detector operates. We trained Guard three times with BCE only and three times with BCE plus the pairwise ranking loss, and tested both on the 9,175 test pairs. All six runs use a smaller training scale than the Guard in Table 2, so their F1 is lower than in Table 2. At the default threshold, the F1 is about the same with both losses. At 1% FPR, Guard detects 1.7 to 4.3 points more attacks with the pairwise ranking loss, in every view.

ViewTraining lossF1Recall at 1% FPR (%)
HTMLBCE only88.760.1
HTMLBCE + pairwise ranking loss89.363.5
MarkdownBCE only86.046.0
MarkdownBCE + pairwise ranking loss85.950.3
Accessibility treeBCE only86.965.4
Accessibility treeBCE + pairwise ranking loss87.567.1

Guard also works on independent datasets. We ran Guard on two external benchmarks, without any further training. WAInjectBench is built for web agents, and we use its 354 attacks that give the agent an explicit instruction and its 2,707 benign items. NotInject is a set of 339 benign prompts that contain trigger words. For comparison, we ran three public trained detectors on the same items.

DetectorWAInjectBench F1WAInjectBench recall (%)WAInjectBench FPR (%)NotInject FPR (%)
BrowseSafe-Guard82.370.30.10.6
ProtectAI DeBERTa-v3 (v2)72.066.92.543.4
PIGuard62.752.52.011.5
Llama Prompt Guard 2 (86M)4.32.30.24.7

Of the four detectors, Guard has the highest F1 on WAInjectBench and the lowest FPR on both benchmarks.

The results hold on a temporal split. We train Guard only on injections observed before April 2026 and test it on the 3,156 attack pages with later injections and their controls. Compared with training on all periods, F1 changes by at most 2.2 points. Both versions use 7,040 training pairs.

Training dataHTML F1Markdown F1Accessibility tree F1
All periods99.292.993.3
Before April 2026 only99.393.191.1
Reviewer_oztBOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BrowseSafe++, a benchmark for indirect prompt-injection detection in browser-agent settings. It expands BrowseSafe to 38,675 attack/control pairs, including 32,187 injections observed on production webpages and 6,488 retained from BrowseSafe. Each pair provides HTML, Markdown, accessibility-tree, and screenshot views, together with a legitimate task and expected outcome. The authors also train BrowseSafe-Guard, a 0.8B text detector and a multimodal screenshot detector, and propose BrowseSafe-Adapt, which revises injections using agent feedback. Experiments evaluate ten language models across multiple page views, four image-capable models on screenshots, and seven agents under adaptive attacks. The results show substantial benign false-positive rates, imperfect correspondence between detection and resistance, strong performance from the compact detector, and large increases in attack success after adaptive revision.

Critical strengths
  1. The paired attack/control construction is a strong methodological contribution. Because the control page is intended to differ only in the injection, the benchmark directly measures sensitivity to malicious instructions rather than topic or page-style artifacts.
  2. The multi-view and adaptive evaluations are highly relevant to deployed browser agents. The finding that models may detect an injection yet still follow it, and that adaptive revision raises attack success dramatically, provides useful evidence that static detection benchmarks underestimate operational risk.
Critical weaknesses
  1. The ecological validity of the “wild” benchmark is difficult to assess. Production pages are substantially rewritten by language models, while injection labels and view-level checks also rely heavily on model judgments; only approximately 10% of rewritten pages receive human review. The released examples may therefore reflect the rewriting and filtering pipeline as much as the original production distribution.
  2. The main results are largely in-distribution. The train/test split prevents exact normalized injections from crossing splits, but pages share domains, rendering conventions, attack taxonomies, and potentially similar rewrite templates. The paper does not provide domain-held-out, temporal-held-out, or cross-benchmark results. In addition, screenshot decisions with transport failures are excluded, and adaptive attack rates are reported without uncertainty estimates.
Questions for authors
  1. What is the inter-annotator agreement or estimated error rate for the model-based injection labels, rewritten-page quality checks, and screenshot-region judgments? How does performance change on the manually reviewed subset?
  2. Can you provide evaluations with stronger distribution shifts, such as holding out complete domains, attack types, production time periods, or rewrite templates?
  3. How many valid screenshot decisions are available per model, and do the reported F1/FPR values change when failed or incomplete requests are counted rather than excluded?
  4. For BrowseSafe-Adapt, how sensitive are the reported attack-success rates to the 64-attempt budget, attacker/critic model choice, and search seed? Please report confidence intervals or results over multiple seeds.
4 comments

We thank the reviewer for their feedback and constructive suggestions. We are encouraged that the reviewer found our paired attack/control construction a "strong methodological contribution". We will address the concerns regarding benchmark validity, generalization, and the reliability of our evaluation results. Revisions to the paper are highlighted in blue and will be updated on Nov 17.


W1 & Q1: Ecological Validity and Annotation Reliability

W1: "The ecological validity of the 'wild' benchmark is difficult to assess. Production pages are substantially rewritten by language models, while injection labels and view-level checks also rely heavily on model judgments; only approximately 10% of rewritten pages receive human review. The released examples may therefore reflect the rewriting and filtering pipeline as much as the original production distribution."

Q1: "What is the inter-annotator agreement or estimated error rate for the model-based injection labels, rewritten-page quality checks, and screenshot-region judgments? How does performance change on the manually reviewed subset?"

Datasets built from deployment data must remove private information before release. WildChat [1] and LMSYS-Chat-1M [2], both built from real user traffic, remove or replace personal information, and Stage 2 of BrowseSafe++ does the same by masking it on every page. Stage 4 goes further and writes the page's prose anew (Appendix G.3), for two reasons. Production pages carry third-party content, such as articles, comments, and messages, that is not ours to redistribute. And a verbatim sentence can be searched to find the original page and the people who wrote on it or are named on it.

Rewriting changes the prose around the injection but keeps the injection and the page structure. The injection is copied byte for byte from the production page. The DOM structure, CSS, and layout are kept: in a random sample of 1,000 pairs, the HTML tags of every page are identical before and after rewriting. The facts that the page's task depends on are kept as well.

To test whether rewriting changes the results, we ran Guard on 415 production pages that were never rewritten, collected from November 1 to 10, 2026, after the benchmark was built. Guard finds as many injections on these raw pages as on rewritten pages in Markdown, and 7.0 points fewer in HTML.

Page viewInjections found on rewritten pages (%)Injections found on raw production pages (%)
HTML89.482.4
Markdown85.084.9

This test uses a Guard trained at a smaller scale than the one in Table 2, so both columns are lower than for the Guard in Table 2.

The model-based labels are consistent across annotators. The six votes of the Stage 3 committee agree with a Fleiss' κ\kappa of 0.847. The view-level checks also do not rely on a model alone. The deterministic audits of Stage 6 check that every page view contains the complete injection, and the attack screenshot visibly differs from the control screenshot in all pairs.

A human review of about 10% of the pages is a large share at this scale. Safety benchmarks labeled by LLMs check far fewer items by hand: OR-Bench [3] labels 80K prompts and checks 100, and SALAD-Bench [4] labels 21,318 questions and checks 458 (2.1%). In BrowseSafe++, human reviewers check about 10% of the pages, and a model also reviews every page on three criteria (Section 2.2). Performance on the manually reviewed pages is close to the performance on all test pages.

Detector (HTML view)F1, all test pagesF1, manually reviewed pagesFPR, all test pagesFPR, manually reviewed pages
GPT-5.589.390.521.022.5
Claude Sonnet 588.285.425.324.8

References

  • [1] Zhao et al. WildChat: 1M ChatGPT Interaction Logs in the Wild. ICLR 2024.
  • [2] Zheng et al. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset. ICLR 2024.
  • [3] Cui et al. OR-Bench: An Over-Refusal Benchmark for Large Language Models. ICML 2025.
  • [4] Li et al. SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models. Findings of ACL 2024.

W2 & Q2: Generalization Beyond the Construction Pipeline

W2: "The main results are largely in-distribution. The train/test split prevents exact normalized injections from crossing splits, but pages share domains, rendering conventions, attack taxonomies, and potentially similar rewrite templates. The paper does not provide domain-held-out, temporal-held-out, or cross-benchmark results."

Q2: "Can you provide evaluations with stronger distribution shifts, such as holding out complete domains, attack types, production time periods, or rewrite templates?"

We ran the evaluations the reviewer lists. Holding out a time period or half of the domains changes Guard's F1 by at most 2.2 points. Holding out 22 of the 24 attack types costs up to 5.7 points. On an external benchmark, WAInjectBench, Guard outperforms three public trained detectors.

For each shift, we compare two Guards on the held-out pages of that shift: one trained without the held-out part and one trained on all data. Both use 7,040 training pairs. Each shift has its own held-out pages, so the F1 of the Guard trained on all data differs between shifts. For time, the held-out part is every injection observed from April 2026 on. For domain, it is six of the twelve domains: forums, web apps, code repositories, e-mail, marketplaces, and checkout. For attack type, Guard is trained on the two most common types, role manipulation and multi-stage chains, and tested on the other 22.

ShiftTraining dataHTML F1Markdown F1Accessibility tree F1
TimeAll periods99.292.993.3
TimeEarlier periods only99.393.191.1
DomainAll domains95.588.989.1
DomainOther six domains only96.388.087.7
Attack typeAll types93.586.487.1
Attack typeTwo types only92.183.681.4

For rewrite templates, all pages are rewritten by the same procedure, so there is no separate template to hold out. W1 & Q1 above test a stronger shift: 415 production pages that were never rewritten and were collected after the benchmark ended.

For a cross-benchmark result, we ran Guard on two external benchmarks, without any further training. WAInjectBench is built for web agents. We use its 354 attacks that give the agent an explicit instruction, which is our definition of an injection (Appendix B), and its 2,707 benign items. NotInject is a set of 339 benign prompts that contain trigger words. For comparison, we ran three publicly trained detectors on the same items.

DetectorWAInjectBench F1WAInjectBench recall (%)WAInjectBench FPR (%)NotInject FPR (%)
BrowseSafe-Guard82.370.30.10.6
ProtectAI DeBERTa-v3 (v2)72.066.92.543.4
PIGuard62.752.52.011.5
Llama Prompt Guard 2 (86M)4.32.30.24.7

Of the four detectors, Guard has the highest F1 on WAInjectBench and the lowest FPR on both benchmarks.

W3 & Q3: Screenshot Evaluation and Excluded Failures

W3: "Screenshot decisions with transport failures are excluded."

Q3: "How many valid screenshot decisions are available per model, and do the reported F1/FPR values change when failed or incomplete requests are counted rather than excluded?"

Every prompted model has a valid decision on 87.3–88.8% of the 18,350 test screenshots. The rest timed out in our cluster because their screenshots were large. Counting these failures as errors, or running them again, leaves the conclusions unchanged.

To confirm this, we count every failed request as a wrong decision. F1 drops by 6 to 9 points for every prompted model, and the order of the models stays the same. Guard has no failed requests, so the gap to it only widens.

ModelValid decisionsF1, failures excluded (Table 3)F1, failures counted as errorsFPR, failures excluded (Table 3)FPR, failures counted as errors
BrowseSafe-Guard 0.8B18,35093.393.36.16.1
Gemini 3.1 Flash-Lite16,28678.770.228.135.9
Claude Haiku 4.516,18664.857.133.841.4
GPT-5.4 nano16,22637.531.36.717.1
Qwen3-VL 30B16,01234.528.412.523.5

We also fixed the issue and ran the four prompted models again on all 18,350 test screenshots. Every screenshot now has a valid decision.

ModelF1, Table 3F1, complete runFPR, Table 3FPR, complete run
BrowseSafe-Guard 0.8B93.393.36.16.1
Gemini 3.1 Flash-Lite78.779.028.126.5
Claude Haiku 4.564.864.333.832.3
GPT-5.4 nano37.541.26.77.7
Qwen3-VL 30B34.532.112.511.3

In every setting Gemini 3.1 Flash-Lite is the best prompted model, and Guard stays well ahead of it.

W4 & Q4: BrowseSafe-Adapt Robustness and Uncertainty

W4: "Adaptive attack rates are reported without uncertainty estimates."

Q4: "For BrowseSafe-Adapt, how sensitive are the reported attack-success rates to the 64-attempt budget, attacker/critic model choice, and search seed? Please report confidence intervals or results over multiple seeds."

Over the 1,024 tasks per target, the 95% confidence interval of every attack-success rate is within ±3\pm 3 points.

TargetASR within 64 attempts (%)95% CI, lower95% CI, upper
Sonnet 530.727.933.6
GPT-5.565.262.368.1
Haiku 4.578.876.281.2
Gemini 3.1 Pro92.991.194.3
Gemini 3.5 Flash-Lite93.691.994.9
DeepSeek V3.195.093.596.2
Kimi K295.493.996.5

The rates grow with the attempt budget, because every extra attempt gives the attacker another chance. Our conclusions do not depend on the choice of 64. With half the budget, 32 attempts, the rates are only 1.3 to 8.8 points lower. Sonnet 5, GPT-5.5, and Haiku 4.5 are the three most robust targets, in this order, at every budget from 4 attempts on.

TargetASR within 1 attempt (%)4 attempts8 attempts16 attempts32 attempts64 attempts
Sonnet 52.16.416.124.928.430.7
GPT-5.50.530.234.754.762.665.2
Haiku 4.50.637.647.560.370.078.8
Gemini 3.1 Pro2.963.177.386.391.192.9
Gemini 3.5 Flash-Lite4.068.078.985.690.393.6
DeepSeek V3.12.976.584.391.393.795.0
Kimi K20.675.982.589.893.895.4

The rates also change little with the search seed and with the attacker model. We repeated the attack on a random 128 of the 1,024 tasks against Gemini 3.5 Flash-Lite.

SettingASR within 64 attempts (%)
Submitted setting, search seed 195.3
Submitted setting, search seed 296.1
Submitted setting, search seed 393.8
One attacker model (Gemini 3.1 Pro) instead of three96.1
Without the critic (and with a generic archive)92.2

Three search seeds differ by at most 2.3 points. One attacker model gives the same rate as three. Removing the critic (and our archive) lowers the rate by 1.6 to 3.9 points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.