- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces BrowseSafe++, a benchmark for prompt injection detection in web agents. The proposed benchmark contains 38,675 attack/control pairs across different web domains, languages, and page views. The paper constructs the benchmark using prompt injections observed on real-world webpages and existing BrowseSafe samples. They also train BrowseSafe-Guard, a lightweight 0.8B detection model using pairwise supervision, and propose BrowseSafe-Adapt, an adaptive attack framework that refines injections based on agent feedback. Experiments show strong detection performance and substantially higher attack success rates after adaptive refinement.
Critical strengths
-
The paper introduces a large-scale prompt injection benchmark covering diverse web domains, languages, attack types, and page views.
-
The proposed BrowseSafe-Guard achieves strong detection performance with a relatively small 0.8B model and low false positive rates.
-
The paper evaluates adaptive attacks against multiple agent models and shows that static attack evaluation can underestimate attack success rates.
Critical weaknesses
-
The generalization ability of BrowseSafe-Guard is not sufficiently evaluated. The model is trained and tested on different splits of BrowseSafe++, which share the same data construction pipeline and similar attack distributions. Although identical injection texts are separated between training and testing, this does not guarantee generalization to other datasets. Evaluating BrowseSafe-Guard on independent benchmarks such as WASP, VPI, or WAInjectBench would provide stronger evidence.
-
The paper lacks an ablation study on the pairwise ranking loss used to train BrowseSafe-Guard. The authors combine binary cross-entropy with a pairwise ranking loss, but do not clearly show how much the additional loss contributes to detection performance. Since the attack and control pairs are a key part of the proposed method, comparing BCE-only training with BCE plus pairwise ranking loss would help justify this design.
-
The practical effectiveness of BrowseSafe-Guard in real agent workflows remains unclear. The paper reports strong detection accuracy and low false-positive rates, but does not sufficiently evaluate how deploying the guard affects actual agent task completion. Even a relatively low false-positive rate may cause legitimate actions to be blocked during multi-step tasks. An end-to-end evaluation measuring both attack prevention and task utility would help demonstrate the practical value of the proposed detector.
-
Finally, a minor concern regarding the LLM rewriting. Although the paper claims to use prompt injections observed in real-world webpages, the surrounding webpage content is largely rewritten using LLMs. Such rewriting may change the original context and the relationship between the injection and the legitimate task. It remains unclear whether these reconstructed pages preserve the behavior of the original attacks. An evaluation comparing attack effectiveness before and after rewriting would help support the realism of the benchmark.
Questions for authors
Please refer to the weaknesses above.
5 comments
W2: Ablation of the Pairwise Ranking Loss
"The paper lacks an ablation study on the pairwise ranking loss used to train BrowseSafe-Guard. The authors combine binary cross-entropy with a pairwise ranking loss, but do not clearly show how much the additional loss contributes to detection performance. Since the attack and control pairs are a key part of the proposed method, comparing BCE-only training with BCE plus pairwise ranking loss would help justify this design."
We ran this ablation. We trained Guard three times with BCE only and three times with BCE plus the pairwise ranking loss, changed nothing else, and tested both on the 9,175 test pairs. All six runs use a smaller training scale than the Guard in Table 2, so their F1 is lower than in Table 2. The table gives the mean over the three training runs.
| View | Training loss | F1 | Recall at 1% FPR (%) |
|---|---|---|---|
| HTML | BCE only | 88.7 | 60.1 |
| HTML | BCE + pairwise ranking loss | 89.3 | 63.5 |
| Markdown | BCE only | 86.0 | 46.0 |
| Markdown | BCE + pairwise ranking loss | 85.9 | 50.3 |
| Accessibility tree | BCE only | 86.9 | 65.4 |
| Accessibility tree | BCE + pairwise ranking loss | 87.5 | 67.1 |
At the default threshold, the F1 is about the same with both losses. The benefit of the pairwise ranking loss is at low false-positive rates, where a deployed detector operates: at 1% FPR, Guard detects 1.7 to 4.3 points more attacks with it, in every view. We will add this ablation to the appendix.
W3: Practical Effectiveness and Agent Task Utility
"The practical effectiveness of BrowseSafe-Guard in real agent workflows remains unclear. The paper reports strong detection accuracy and low false-positive rates, but does not sufficiently evaluate how deploying the guard affects actual agent task completion. Even a relatively low false-positive rate may cause legitimate actions to be blocked during multi-step tasks. An end-to-end evaluation measuring both attack prevention and task utility would help demonstrate the practical value of the proposed detector."
We ran this end-to-end evaluation on 1,024 tasks. Guard is placed in front of an agent (Gemini 3.5 Flash-Lite) and checks the Markdown view of every page before the agent reads it. A flagged page is replaced with a notice that it was blocked, and the agent continues with the task. Guard blocks 94.53% of the pages that carry an observed injection.
| Setting | Attack success of the observed injection (%) | Clean-task completion (%) |
|---|---|---|
| Agent alone | 4.00 | 97.85 |
| Agent + Guard | 1.37 | 91.21 |
With Guard, the observed injection succeeds on 1.37% of the tasks instead of 4.00%. The cost is 6.64 points of clean-task completion, because Guard flags 6.93% of clean pages. In a multi-step task the agent reads several pages, and one wrongly blocked page can stop the task.
Our experiment also uses Guard in the strictest way: a flagged page is blocked outright. Real products do not rely on a detector alone. They use it as one layer of a layered defense. Google describes this for Gemini, where prompt-injection classifiers are combined with model hardening and user confirmation [1], and Microsoft describes a defense-in-depth approach that combines detection with prevention and impact mitigation [2]. In such a cascade, a flag from the detector triggers the next layer, for example a second check or a confirmation by the user. A false positive then costs one extra step, not the task.
We will add this evaluation and this discussion to the paper.
References
- [1] Google. Mitigating prompt injection attacks with a layered defense strategy. Google Security Blog, June 2025.
- [2] Microsoft Security Response Center. How Microsoft defends against indirect prompt injection attacks. July 2025.
We thank the reviewer for their feedback and constructive suggestions. We appreciate that the reviewer recognized the broad coverage of BrowseSafe++, the strong detection performance of our lightweight 0.8B model, and the importance of evaluating adaptive attacks beyond static benchmarks. We will address the concerns regarding generalization, the contribution of pairwise supervision, practical deployment, and the realism of rewritten webpages. Revisions to the paper are highlighted in blue and will be updated on Nov 17.
W1: Generalization to Independent Benchmarks
We evaluated Guard on WAInjectBench without any further training. WAInjectBench is a collection of attacks from earlier work and includes the attack texts of WASP and VPI, so it covers the three benchmarks the reviewer names. Guard generalizes to them. On the attacks that give the agent an instruction, it has the highest F1 of all trained detectors.
Guard also raises the fewest false alarms on benign data from other sources. NotInject is a set of benign prompts that contain trigger words.
By our definition, an injection is an instruction in page content that tries to redirect the agent (Appendix B), and this is what Guard is built to detect. WAInjectBench also contains attacks with no instruction to the agent, such as pop-ups and misleading form labels. These are outside our definition, and Guard is not built to flag them. We will state this scope in the paper.
The reviewer also notes that training and test pages share one construction pipeline. Guard also works on production pages that our pipeline never rewrote: 415 pages collected from November 1 to 10, 2026, after the benchmark was built. Guard finds as many of their injections as on rewritten pages in Markdown, and 7.0 points fewer in HTML.
This test uses a Guard trained at a smaller scale than the one in Table 2, so both rows are lower than for the Guard in Table 2.
Guard also generalizes to attacks it was not trained on. We train it without one part of the data and test it on that part: injections observed from April 2026 on, six of the twelve domains, or 22 of the 24 attack types. We compare it with a Guard trained on all data and tested on the same pages. Each held-out part has its own test pages, so the F1 of the Guard trained on all data differs between parts.