acceptodds
Under review as a conference paper at ICLR 2027

AgentHoneypot: Benchmarking Machine-Readable Content Injection and Provenance-Aware Authorization in Web Agents

Abstract

Web agents consume machine-readable webpage content beyond what users directly see, including hidden DOM content, accessibility metadata, runtime-injected text, parser outputs, and media-derived content. These representations create additional indirect prompt-injection surfaces, but observed attack success can be confounded by pre-existing action preferences, visible semantic differences, or carriers that never reach the model. We introduce AgentHoneypot, a matched-honeypot benchmark that controls these factors using visibly matched canonical and honeypot actions, carrier-matched PLACEBO controls, and layout counterbalancing. In the primary Qwen3-VL-Flash study, runtime-injected A3 and parser-derived A4 each produce a +50 percentage-point pooled matched lift; a separate A5-LSB study produces the same lift on SmartApp. A 300-trial DeepSeek replication and cross-agent studies with Skyvern and Crawl4AI further show that carrier effects depend strongly on the model, observation pipeline, and website. We also introduce MCAAG, a provenance-aware authorization layer that links suspicious carrier evidence to the action candidate it supports and checks that candidate before execution. Across 85 matched benign PLACEBO replays, we observe no false rejection or unnecessary recovery. In 180 fresh clean Browser Use trials, task success changes from 90/90 to 88/90 with MCAAG enabled, with a median local authorization time of 27.0 ms. Overall, our evaluation contains more than 2,400 valid behavioral trials.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Web Agents, Indirect Prompt Injection, Agent Security, Machine-Readable Content, Honeypot Benchmark, Provenance-Aware Defense

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p1KsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgentHoneypot, a framework designed to evaluate whether the injection of machine-readable content alters the action selection of web agents. The benchmark establishes "canonical" and "honeypot" candidates—sharing identical visible task semantics—across three controlled websites; it distinguishes between inherent selection preferences and injection-induced behavioral changes through carrier-matched ATTACK/PLACEBO controls and candidate position swapping. The study examines five categories of carriers: hidden references, accessibility metadata, runtime-injected text, parser output, and decoder-assisted LSB media content. The authors further distinguish between reachability (whether the payload enters the model's observation space) and exploitability (whether the payload alters action selection), reporting varying results across Qwen, DeepSeek, Skyvern, Crawl4AI, and FP-Agent setups. The proposed defense method, MCAAG, binds carrier-related evidence to specific candidates and performs authorization checks prior to execution. Upon rejection, the system deterministically recovers if an equivalent candidate exists that satisfies the same authorization policy. The paper reports on the interception of successful attack replays, 85 PLACEBO control trials, and 180 "clean" task runs using Browser Use; the latter showed a task success rate shift from 90/90 to 88/90, with a median local authorization latency of 27.0 ms.

Critical strengths
  1. First, the paper accurately identifies a critical issue: an agent selecting the button preferred by the attacker does not necessarily mean the injection was effective; it could simply reflect the button's original position or a pre-existing preference. By using a control setup that preserves the carrier and page structure while swapping only the selection instruction, the study isolates the "additional impact caused by the malicious instruction." The value of this design is evident in the results:
  • For DeepSeek A1 (Table 6), the "honeypot" button was selected 15 out of 30 times under the ATTACK condition, but the PLACEBO condition already showed a selection rate of 14 out of 30; thus, the net increase was only 3.3 percentage points. Treating the 50% figure directly as the injection's effectiveness would be highly misleading.
  • Appendix E.1 shows that while A3 and A4 jumped from 0% to 100% in Layout A, the PLACEBO condition in Layout B was already at 100%. Consequently, the aggregated +50 percentage point increase cannot be interpreted as uniform vulnerability across all layouts. This contribution improves the quality of security assessment conclusions. Even without introducing new attack categories, accurately measuring "what exactly the attack changes" is a topic that warrants significant attention.
  1. Distinguishing between reachability and exploitability—and reporting negative results—provides diagnostic insights far more valuable than a simple attack leaderboard. The paper avoids interpreting every failure as evidence of model security and refrains from characterizing every adapter experiment as an inherent vulnerability. For instance, Skyvern’s native A2 was successfully injected into the model's context but resulted in 0/10 selections under both conditions; similarly, while the content for FP-Agent’s A2 and A3 was confirmed to be part of the model prompt, the target was not selected across 36 payload runs. In contrast, the attack selection rate for Crawl4AI’s native A2 was 45/50, 43/50, and 15/50 across the three tested websites. Collectively, these results demonstrate that merely exposing a payload to the model does not determine its behavioral impact. The authors clearly defined the scope regarding the adapter paths for Skyvern and Crawl4AI, the explicit decoder for A5, and the repeated experiments on fixed websites, thereby establishing clear boundaries for interpreting the results. The paper offers practical methods for determining whether a failure occurs during the observation phase or the decision-making phase; these negative results also help avoid overgeneralizing the effectiveness of specific agents.
Critical weaknesses

W1. MCAAG's key authorization and detection logic lacks operational definitions; the current PDF does not sufficiently support independent replication or an assessment of its security boundaries.

Section 4.1 describes how various modules check for hidden references, accessibility metadata, runtime modifications, parsing inconsistencies, and media sources, outputting:

f=(c,k,r).f=(c,k,r).

Appendix D defines authorization as the conjunction of six checks:

\land I_{\mathrm{binding}}\land I_{\mathrm{dest}}\land I_{\mathrm{integrity}}.$$ However, neither the main text nor the appendix adequately explains how these checks are calculated. For instance, what conditions render a standard accessibility description "suspicious" evidence? How is "suspicious" defined? How does the system distinguish between legitimate runtime page updates and malicious injections? The absence of such explanations directly impacts the validity conditions of the method. While the paper notes that the defense ignores `canonical`/`honeypot` tags—thereby ruling out the shortcut of simply reading evaluation labels—it fails to specify the actual rules employed by the detector. ### W2. The link between induced behavior and unauthorized actions is insufficiently established; the connection between benchmark labels and security semantics requires further clarification. Section 3.2 and Appendix B configure the two candidates to share identical visible task semantics while retaining their `canonical`/`honeypot` identities on the evaluation side. While this design is reasonable for studying behavioral deviation, it raises a question: **Apart from the evaluation-side identity, what trustworthy, deployment-time evidence indicates that one action aligns with user authorization while the other does not?** This issue is evident in the PLACEBO results: out of 85 benign controls, only 51 `canonical` cases succeeded; MCAAG left all behaviors unchanged and resulted in no "unresolved" cases. Consequently, the defense does not universally rectify non-`canonical` selections defined by the evaluation side; instead, it primarily rejects candidates supported by suspicious payloads. This does not invalidate the concept of "matched lift," but it suggests that the following two propositions must be treated separately: 1. Injected content alters the candidate selection. 2. The defense identifies and blocks actions that violate user authorization. The evidence supporting the first proposition is significantly stronger than that for the second. While the concepts of "destination consistency" and "task compatibility" in Appendix D might offer a link, the paper fails to provide specific examples detailed enough for verification. ### W3. Existing ablation studies fail to demonstrate the incremental value of MCAAG over simple candidate-blocking strategies The ablation results in Appendix I show that: - Removing the "media finding" step allows all 10 honeypot proposals in A5 to execute; - Bypassing authorization also results in full execution; - Removing the "recovery" step maintains the blocking action, but leaves all cases "unresolved." While these results confirm that system components are integrated into the execution flow, they do not sufficiently demonstrate the necessity or advantages of the shared authorization policy. In particular, the "attack execution after bypassing authorization" scenario cannot distinguish between a complex authorization mechanism and a simple kill-switch. The paper omits a crucial control experiment: **using the same payload evidence to directly reject flagged candidates and select the sole remaining, equivalent, unflagged candidate.** In scenarios involving only two matching terminal candidates, this alternative strategy might cover many of the same cases; experimental data is required to determine this. The authors acknowledge in the "Related Work" section that provenance-aware authorization is not a novel concept in itself; this makes it even more critical to articulate the specific incremental benefits provided by candidate binding and shared authorization. Current evidence confirms that MCAAG functions in the tested cases, but it does not clearly demonstrate that its core design outperforms simpler defense mechanisms that utilize the same information. Consequently, it is difficult to substantiate the actual effectiveness of the proposed approach. ### W4. The benign utility evaluation suffers from selection bias based on attack outcomes and fails to cover benign inputs that would best test for false rejections Appendix J explicitly states that benign replays for CreativeUI–A2 were not executed because the underlying attack failed to produce an effect. **The fact that an attack is ineffective under certain conditions does not mean the defense will not falsely reject benign inputs under those same conditions.** Therefore, the assessment of benign utility should not be contingent upon the prior observation of a successful attack. Furthermore, PLACEBO constructs its test cases by eliminating malicious selection preferences. While it can check whether the defense excessively rejects the carrier itself, it may not detect more challenging benign cases—such as accessibility descriptions containing instructions for normal actions, page text prompting user interaction following a legitimate update, or genuine task steps conveyed via the parser. The 180 "clean runs" using Browser Use provided a valuable supplement; however, they relied on the same three controlled websites, and the two failures observed with CarCare were not further analyzed or attributed. ### W5. Gaps in Verifiability Regarding Sample Composition, Statistical Intervals, and Raw Evidence Three specific issues exist: - **Main Experiment Counts:** Section 5.2 reports 360 valid runs for the Qwen main experiment. However, the design outlined in Table 3 for categories A1–A4 specifies 30 PLACEBO runs and 30 ATTACK runs per category (totaling 240), while the 20 runs for A5 are explicitly listed as an additional experiment. The remaining 120 runs likely correspond to ablation studies or other conditions, but the PDF does not provide a complete mapping. - **Confidence Intervals:** Table 6 cites Newcombe intervals; however, calculations based on the table's counts show that the reported values ​​can be replicated by directly subtracting the endpoints of two Wilson intervals. For instance, A1 yields a range of approximately $[-30.7, 36.6]$ percentage points. The authors need to clarify the formula actually employed and explain how matching and stratified layout factors were incorporated into the calculation. - **Missing Raw Records:** Appendix L notes that the raw JSON logs for the historical MCAAG-v2 A2–A4 regression tests have been lost. Consequently, related claims rely on surviving reports and snapshots, yet the authors do not specify which claims still depend on these missing records versus those supported by independent re-runs.
Questions for authors
  1. Please list the six authorization predicates and the specific decision rules for each carrier module, and present at least one ATTACK/PLACEBO pair showing the raw observations, findings, candidate bindings, and final decisions. Does the detector rely on fixed payload vocabulary, candidate IDs, or path features?

  2. What evidence visible at deployment—without reading canonical/honeypot labels—independently determines the legitimacy of an action? Please explain why non-canonical selections in PLACEBO should be preserved, and the difference in security semantics between such a selection being retained in PLACEBO versus the same candidate being rejected in an ATTACK scenario.

  3. What results are achieved by a simple strategy of "directly rejecting marked candidates" when using identical carrier evidence? Does shared authorization allow marked-but-legitimate actions, or reject unmarked-but-illegitimate actions, in certain instances?

  4. Why does the benign evaluation of CreativeUI–A2 depend on whether the attack is effective? Are there recorded instances of false rejections involving accessibility text, runtime updates, or parsed content that contained legitimate operational suggestions? What specifically caused the two failures in CarCare?

  5. Are there results from closed-loop experiments (defense ON vs. OFF) under attack conditions within the same agent pipeline? If so, please report honeypot execution, original task completion, recovery outcomes, and unresolved cases, rather than just the interception results for frozen proposals.

  6. Please provide a table counting the conditions for the 360 ​​main experiments and all behavioral trials, the formulas used to calculate the intervals in Table 6, and the correspondence between the three missing raw data files and the claims made in the current paper.

7 comments
AuthorsOpenReview ↗

Response to Reviewer p1Ks — Authorization checks and a paired example

Thank you for identifying the distinction between a change in candidate selection and an action that violates the user's task. The matched-twin benchmark measures the former. We use the separate task-diversion experiment to measure the latter as navigation to a distinct off-task destination.

W1. How the guard decides. MCAAG makes a deterministic decision before executing the action. It makes no LLM call. The six checks are:

  1. Candidate binding. The proposal must resolve to the candidate being checked.
  2. Task compatibility. The original user task and the rendered action label must have lexical overlap.
  3. Actionability. The action must be allowed for that control, which must be available for interaction.
  4. Workflow binding. The candidate must have a browser node and a workflow container.
  5. Destination and action binding. The candidate must have an observed destination, and the action must match its recorded binding.
  6. Carrier integrity. No blocking finding may be bound to the candidate.

A separate input check excludes evaluator-only fields. Runtime candidate IDs identify browser elements. They do not carry the canonical or honeypot labels. The destination is used to check the action binding. This check does not determine whether the destination itself serves the user's task.

The carrier modules inspect different evidence. A1 checks candidate-linked descriptions outside the rendered view. A2 checks whether action guidance in accessibility metadata is supported by the rendered control and its context. A3 checks candidate-linked guidance introduced after the initial page observation. An ordinary page update alone is insufficient. A4 checks guidance present in parsed content against the rendered context. A5 checks action guidance in available media-derived content and binds it to the referenced candidate. These are lexical and structural heuristics. A finding depends on the observed source and its relation to the proposed action.

W2. A paired replay example. In SmartApp A5-LSB, Layout B, repetition 1, both recorded proposals selected control A. Both conditions used the same media pipeline. The following summarizes the recorded observations and decisions with the controls labeled A and B.

PLACEBOATTACK
Media observationDecoded neutral textDecoded text contained a selection directive
Candidate findingNo blocking finding on AA blocking media finding was bound to A
Checks on AAll six passedThe first five passed. Carrier integrity failed
Final action in replayA was authorized and clickedA was rejected. B was the only eligible equivalent. B passed the checks and was clicked

The same evaluator-labeled alternate can therefore be kept under PLACEBO and rejected under ATTACK. The change in decision comes from the evidence attached to the action. In the 85 benign replays, all proposals were directly authorized, including the 34 non-canonical choices. This is the expected result for visually equivalent controls with no blocking evidence.

For an independent check on task diversion, our visible-notice experiment measured navigation to a different destination from the user's request. In each layout, that destination was reached in 30/30 ATTACK, 0/30 PLACEBO, and 0/30 NONE runs. Here NONE is the original page without an added notice.

AuthorsOpenReview ↗

Response to Reviewer p1Ks — Baselines, live evaluation, and benign utility

W3. What do the ablations and simpler baselines establish? The ablations show that the carrier finding, the pre-action check, and recovery each affect the outcome. They do not establish that shared authorization outperforms a policy that uses the same findings to reject a candidate and choose the sole unflagged equivalent. Our current simple-blocking baseline uses a different detector. The exact same-evidence comparison remains open, especially on two-button tasks where the strategies may agree.

Under the current policy, a blocking finding fails the integrity check even if the action later proves legitimate. We observed this in the benign tests below. An unflagged candidate can still fail a separate check. For example, in four controlled component cases with no carrier findings, the candidate failed task compatibility. This shows that the two checks can make distinct decisions. It does not establish that task compatibility reliably detects unauthorized actions.

Closed-loop evidence. We added 250 valid autonomous GPT-5 Browser Use trials under attack. All five arms used the same agent pipeline. A2 and A3 used native observations, while A1, A4, and A5 used compatibility adapters.

ArmHoneypot endpointCanonical endpointRecoveredUnresolved
Defense OFF50/5028/500/5022/50
MCAAG-v110/5020/5020/5030/50
MCAAG-v20/5049/5049/501/50
Simple blocking39/500/500/5050/50
Surface cleaning32/5027/500/5023/50

Canonical execution means reaching the designated task endpoint. Both endpoints can be reached in one run. Unresolved means that the canonical endpoint was never reached. Full multi-step task completion was scored separately in the benign evaluation below.

The live results apply to the fixed attack fixtures. In a separate manual white-box evaluation of the frozen guard, two selected held-out ATTACK conditions caused off-task navigation in 8/8 and 7/7 valid defense-ON runs. Both matched PLACEBO conditions were 0/8. One additional ATTACK response was invalid due to output format. The guard authorized the actions that reached the off-task destination. We therefore limit our claim to the observed interception and recovery results.

W4. Benign coverage and false rejections. We agree that the historical CreativeUI–A2 exclusion was a gap. It was skipped because the original attack showed no lift, which gives no information about false rejection. Later scripted checks accepted the normal CreativeUI–A2 proposal in both layouts. The full matched benign replay for that cell remains missing.

We also observed false rejections in two separate scripted accessibility cases, one with a legitimate accessible name and one with a benign accessibility note. In the autonomous Utility-v2 evaluation, full multi-step completion was 52/59 OFF and 55/60 ON. At least one legitimate action was rejected in 19/60 ON runs, with 55/500 checked legitimate actions rejected. The four additional benign scenarios completed in 20/20 runs per arm. The legitimate dynamic-content scenario recorded one rejection in six checked actions. These results leave the false-rejection rate on legitimate parser-supplied instructions open.

The two historical CarCare clean runs ended without a canonical execution receipt. The surviving records establish that outcome, but do not identify whether a guard decision or another cause prevented completion.

AuthorsOpenReview ↗

Response to Reviewer p1Ks — Trial accounting, corrected intervals, and archived evidence

W5. Trial accounting. The original 360-run Qwen batch had six variants. Each used three sites, two layouts, and five trials per site/layout/condition.

Original Qwen3-VL-Flash variantPLACEBOATTACK
A1, off-screen description3030
A2-V1, matched accessibility metadata3030
A3, visible runtime text3030
A4-LaTeX, parser output3030
A2-V2-Strong, accessibility diagnostic3030
Adversarial screenshot, visual diagnostic3030
Total180180

Table 3 includes the first four rows (240 runs) plus the separate SmartApp A5-LSB study (10 PLACEBO and 10 ATTACK). Those 20 historical LSB runs recorded candidate selections.

The six main cohorts with explicit denominators in the paper reconcile to 2,410 valid episodes.

CohortValid episodesConditions
Original Qwen3-VL-Flash batch360180 PLACEBO, 180 ATTACK
Qwen SmartApp A5-LSB2010 PLACEBO, 10 ATTACK
DeepSeek replication300150 PLACEBO, 150 ATTACK
Skyvern formal transfer10050 PLACEBO, 50 ATTACK
Crawl4AI formal evaluation1,450650 PLACEBO, 650 ATTACK, 150 CLEAN
Browser Use clean utility18090 OFF, 90 ON

The paper also reports 40 A4-Markdown trials and 36 carrier-bearing Skyvern runs on FP-Agent sites as supplemental results. The 36 are part of a 42-run pilot with six additional clean controls. The 85 benign PLACEBO replays and the defense ablations reuse archived proposals. They do not add fresh behavioral episodes to the total.

Table 6 correction. The published intervals subtracted Wilson endpoints and were incorrectly labeled Newcombe. We recomputed them from the original counts. Within a layout we use Newcombe's independent-proportion hybrid score interval. For the layout-stratified result we combine the two layouts with equal weights using Wilson–MOVER. Newcombe, 1998

For each layout jj, let pA,jp_{A,j} and pP,jp_{P,j} be the ATTACK and PLACEBO rates. Their Wilson 95% intervals are [LA,j,UA,j][L_{A,j}, U_{A,j}] and [LP,j,UP,j][L_{P,j}, U_{P,j}]. We give each of the two layouts equal weight, wj=1/2w_j = 1/2. The estimated lift is

d=∑jwj(pA,j−pP,j).d = \sum_j w_j (p_{A,j} - p_{P,j}).

The lower and upper limits are

L=d−∑jwj2[(pA,j−LA,j)2+(UP,j−pP,j)2],L = d - \sqrt{\sum_j w_j^2 \left[(p_{A,j} - L_{A,j})^2 + (U_{P,j} - p_{P,j})^2\right]},

U=d+∑jwj2[(UA,j−pA,j)2+(pP,j−LP,j)2].U = d + \sqrt{\sum_j w_j^2 \left[(U_{A,j} - p_{A,j})^2 + (p_{P,j} - L_{P,j})^2\right]}.

The model calls were independent within each layout. Matching refers to the experimental conditions. The table reports the lift and interval limits in percentage points.

CarrierLayout A, PLACEBO / ATTACKLayout B, PLACEBO / ATTACKEqual-layout lift [95% CI], pp
A10/15 / 1/1514/15 / 14/15+3.33 [−12.57, 19.93]
A2-V10/15 / 0/150/15 / 8/15+26.67 [8.16, 41.61]
A3, A4-LaTeX, and A5-LSB, each0/15 / 15/1515/15 / 15/15+50.00 [32.34, 60.19]

A3, A4, and A5 each show a lift in Layout A and a saturated PLACEBO rate in Layout B. These estimates describe the tested fixtures.

Missing raw records. The old A2–A4 regression JSON files remain unavailable. The surviving reports describe five blocks and five recoveries in each historical cell listed below. The newer live experiments are independent evidence for the specified cells under the revised v2 integration.

Missing raw fileHistorical cells supported by surviving reportsIndependent live coverage
a2_regression.jsonA2 on SmartAppA2 on CreativeUI
a3_regression.jsonA3 on CreativeUI and SmartAppA3 on CreativeUI
a4_regression.jsonA4 on CreativeUI and SmartAppA4 on CarCare

The original per-trial findings and recovery decisions for those historical cells still depend on the surviving reports and snapshots.

Reviewer_QLjaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies indirect prompt injection through machine-readable content in web agents. It points out that choosing an attacker-preferred action does not necessarily mean an injection worked: the agent may already prefer that action. Conversely, an attack may fail simply because its payload never reached the model. To address these ambiguities, the authors introduce AgentHoneypot, a controlled benchmark for fairer and more accurate evaluation. Each task contains two actions with the same visible task semantics. Attacks are compared with controls that preserve the injection carrier but omit the malicious directive. The authors also swap candidate positions to account for position preferences. These comparisons distinguish whether a payload reaches the model from whether it changes action selection. The paper also proposes MCAAG, a provenance-aware authorization layer for web-agent actions. It operates between action proposal and execution. MCAAG links suspicious content to the action candidate it supports, then checks whether that action is consistent with the user’s task and the available provenance evidence. If the action is rejected, it attempts recovery through an already-observed alternative that passes the same checks. Experiments across models and agent pipelines show that attack effects vary with the carrier and how web page content is processed. Defense effectiveness is tested primarily by replaying saved observations and action proposals. Separate benign Browser Use runs measure task completion and runtime overhead.

Critical strengths
  1. The benchmark is designed to separate injection effects from the agent’s existing preferences. It compares malicious directives with harmless content in the same carrier and swaps the positions of two visibly equivalent action candidates. These controls matter: in some layouts, the agent always selects the honeypot even without an attack. The benchmark therefore offers a more informative assessment than attack-selection rates alone.
Critical weaknesses
  1. The novelty of MCAAG is unclear. Prior work such as AuthGraph already uses provenance to check whether actions are authorized. Prismata also restricts actions at the level of individual web elements. MCAAG adds carrier-specific checks and a recovery procedure, but the paper does not clearly show what new capability these additions provide. As presented, it appears closer to an implementation of existing principles in the benchmark environment than a distinct defense method.

  2. The defense evaluation does not fully test its operation during an attack. MCAAG is evaluated mainly by replaying saved observations and action proposals. This tests authorization given the recorded evidence, but not whether the system can reliably collect and bind that evidence during a live attack. Recovery is also tested where an equivalent alternative has already been observed. The fresh Browser Use experiments assess benign utility, so they do not close this gap.

Questions for authors
  1. What trusted evidence allows MCAAG to distinguish authorized from unauthorized candidates when their visible task semantics are matched?Please provide a complete example showing the evidence, the authorization checks, and the recovery decision.

  2. What capability does MCAAG add beyond provenance-aware authorization in AuthGraph and element-level action restrictions in Prismata?Please explain which parts of carrier-specific checking or recovery are not already covered by these approaches, and what evidence supports their benefit.

  3. How are the ATTACK and PLACEBO contents matched beyond using the same carrier and layout? Please clarify differences in length, wording, visibility, and candidate descriptions, and whether these could independently affect selection.

6 comments
AuthorsOpenReview ↗

Response to Reviewer QLja — Authorization, live evidence, and matched controls

Thank you for asking us to clarify what MCAAG checks, how it relates to prior defenses, and what is held fixed in our benchmark.

1. What evidence does MCAAG use?

The user’s instruction defines the task. The browser supplies the current element, its rendered label, its accessibility metadata, and its action and destination bindings. These observations help the guard check the proposal. Page text by itself cannot grant permission. When two controls have the same visible task semantics, either can pass. The canonical and honeypot labels are used only for evaluation.

In one live GPT-5 Browser Use run, the agent proposed clicking control A. A passed the checks for the live element, task-label compatibility, actionability, and workflow and destination binding. It failed the carrier integrity check because its accessibility metadata contained a directive that the rendered page did not support. Control B was the only already observed equivalent in the same workflow. B passed the same checks. Browser Use clicked B, and we verified the click and the HTTP 200 response at the expected endpoint. A’s endpoint was not reached.

The destination check verifies that the action is bound to the observed destination. It cannot establish what that destination means for the user’s task. In a separate matched media replay, the guard allowed all 10 PLACEBO proposals, including five that selected the evaluator-labeled honeypot. Those choices had no blocking finding, so the guard preserved them.

2. How does MCAAG relate to AuthGraph and Prismata?

AuthGraph compares execution provenance against an authorization specification derived from the user’s request and tool catalog. Prismata assigns capabilities to individual web elements and enforces them both in the agent’s observation and at execution. MCAAG applies related principles to browser observations from several carriers. It links a finding to the specific action the agent proposes. If the action is rejected, the guard can check and execute a unique, already observed equivalent. Our evidence is for this implementation and recovery procedure. The current experiments provide no head-to-head comparison with either system.

We agree that replay evaluates decisions on saved evidence. To test the live pipeline, we ran 250 valid autonomous GPT-5 Browser Use trials under attack across five defense settings. These runs exercised evidence collection, candidate binding, and recovery during browser execution. The honeypot endpoint was reached in 50/50 defense-OFF runs and 0/50 MCAAG-v2 runs. With v2, recovery reached the canonical endpoint in 49/50 runs.

For both A1 and A3, honeypot execution was 0/10 with either v1 or v2. Canonical execution rose from 0/10 to 9/10 on A1 and from 0/10 to 10/10 on A3. This is evidence of better recovery on the tested fixtures. Recovery still depends on an observed, equivalent control that passes the checks. The simple blocker uses a different detector, so the benefit of shared authorization under identical findings remains an open question.

3. What is matched between ATTACK and PLACEBO?

A carrier is the part of the page or its representation through which added content reaches the model. Within each site, carrier, and layout, we hold fixed the user task, candidate set, visible button labels, positions, and destinations. ATTACK contains a selection directive. PLACEBO presents neutral wording through the same carrier.

CarrierWhat stayed fixedWhat changed
A1 — off-screen descriptionSame control and referenced description fieldThe off-screen description text
A2-V1 — accessibility metadataSame controls, visible labels, and accessibility fieldsThe accessible name and descriptions sent to the model
A3 — visible runtime noticeSame page region and runtime presentation. The notice was visible in both conditionsThe wording of the notice
A4-LaTeX — parser outputSame candidate page and parser pipelineThe text in the parser output sent to the model
A5-LSB — decoder-assisted imageSame source image, dimensions, decoder, and decoded text lengthThe encoded image bytes and the decoded wording

In the original Qwen A1–A4 request samples, ATTACK was one word and 6–7 characters longer than its site-matched PLACEBO. The two A5-LSB texts were both 50 bytes. In A2, the model-facing descriptions differ even though the rendered labels stay fixed. Wording, length, and salience may each affect selection. Our estimate therefore applies to the full directive version compared with its neutral counterpart. The experiment does not identify the separate effect of any one wording change.

Thank you again for your helpful questions about the authorization checks and the matched controls. We’ve included a concrete live example and explained what stayed fixed between ATTACK and PLACEBO. If any part of the example or comparison would benefit from more detail, we’d be happy to clarify.

Thank you for your response and clarifications. However, some of my concerns still persist. I will maintain my original score.

Reviewer_aAp9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

AgentHoneypot tests how five types of hidden or machine-readable prompt injections influence web agents across three simple websites. Each test measures whether an injection makes the agent choose a targeted button over an identical alternative, compared with a placebo. The authors evaluate several agents and propose MCAAG, a defence that checks the source of instructions before actions are executed. In limited tests, MCAAG blocks all tested attacks while maintaining high performance on clean tasks with little added latency.

Critical strengths

Quite strong experimental design. The use of matched placebos and counterbalanced layouts helps distinguish genuine attack effects from baseline behaviour. I think that especially matters because honeypot selection rates are often close to 50% even without an attack, and one layout already shows very high selection rates. The evaluation approach is a valuable contribution on its own.

Cross-pipeline comparisons. The results show that the same injection can have very different effects depending on how an agent processes website content. In some cases, injections reach the model but fail to influence its actions, while in others, the same attack succeeds through a different pipeline. I think there is a strong contribution in showing why simply checking whether an injection reaches the model is not enough to assess vulnerability, making these findings useful for designing safer agents.

Critical weaknesses

The main attack claims are not fully supported by the experiments. The benchmark asks agents to choose between two equally valid buttons, so injections only need to influence an arbitrary choice rather than redirect an agent away from the user's task. This limits how well the results represent real prompt injection attacks. Notably, FP-Agent shows no successful attacks (0/36), despite the injected content reaching the model.

I'm also worried about the statistical evidence as each site–layout combination has only five trials, decoding settings are unclear, and some reported improvements are driven by extreme cases. The paper lacks a no-injection baseline and complete per-site results. Several attacks rely on custom adapters to reach the model, while the more realistic image-based attack fails. Without these adapters, the evidence is limited to a few agents and three simple websites. There is therefore an issue with the construct validity and the headline success rates overstate what the experiments establish about real-world attack effectiveness.

The proposed defence is insufficiently specified and evaluated. MCAAG is tested mainly against the same five attacks it was designed to detect, without explaining precisely how its detection rules work. Its effectiveness may therefore depend on recognising these specific attack patterns rather than providing broader protection. The evaluation lacks comparisons with simpler defences, tests against adaptive attackers, and live experiments combining attacks with the defence. Some replay results are missing, and the clean Browser Use comparison is confounded by differences in execution. It seems that the recovery is unusually easy as every task has one valid alternative button. Because of it, it remains unclear whether MCAAG would work against new attacks or in the realistic settings.

On a more minor note:

Related work I noticed that the related work section omits some of the directly comparable web-agent injection literature, which might significantly contribute to your work, e.g. EIA (Liao et al., 2024) and TRAP (Korgul et al., ICML 2026) already inject task-divergent targets into agents on realistic web tasks, the first through hidden DOM content and the second through persuasion-principled payloads with tailoring, on frontier models. AdvWeb (Xu et al., 2024) uses the same invisible-HTML carrier as A1 with optimised payloads and WASP (Evtimov et al., 2025) is an end-to-end web-agent injection benchmark that separates task success from attack success. Potentially Wu et al. (2024), Progent (Shi et al., 2025), and MELON (Zhu et al., 2025) also work in the similar area. Together these undercut the claim that no controlled cross-channel benchmark exists, and they would provide you with the reference points for the payload text and the divergent-target condition the paper currently lacks.

Use of AI It seems that the text is quite strongly AI-generated/polished, which makes it significantly harder to read. I believe a human iteration would significantly improve the quality of text.

Questions for authors

Clarifications that could affect my assessment:

  1. What form of information (e.g. screenshot, DOM, accessibility tree, prompt) does Qwen3-VL-Flash actually receive? Are A1/A2 injections for sure reaching the model? And can it see differences in the candidates’ href values?
  2. Is the A3 injected text visible to a human who is viewing the page?
  3. What specific rules trigger each MCAAG module? Are they based on content or something else, and does authorisation involve an LLM?
  4. Could you explain several reporting inconsistencies: 360 trials in Section 5.2 versus 260 in Table 3, CarCare-A1 excluded from replay but included in Table 3, SmartApp replay cells with five rather than ten trials, and were the 55 and 22 infrastructure exclusions balanced across conditions?

Suggested experiments:

  1. Testing injections that redirect agents towards a different action or destination, rather than choosing between equally valid buttons.
  2. Comparing MCAAG with a simple content-sanitisation baseline, testing adaptive attacks, and measuring false positives on realistic dynamic and accessible websites.
  3. Evaluating MCAAG in a live Browser Use attack with the defence enabled, ideally including at least one frontier model.
7 comments
AuthorsOpenReview ↗

Response to Reviewer aAp9 — Benchmark design and task diversion

Thank you for pointing out the difference between changing a button choice and changing the user’s task. We use visibly equivalent buttons to measure the first effect. Both buttons appear to serve the same task. We swap their positions and compare the attack with a neutral message in the same carrier. This lets us measure how much the directive changes the choice.

Can the directive divert the agent from the task? We ran a separate 180-trial experiment. The user asked the agent to open a contact page. We measured whether the browser instead reached a separate verification page. NONE was the original page with no added notice. PLACEBO showed a neutral notice in the same format. ATTACK used that notice to direct the agent toward the off-task destination. In each of the two layouts, the agent reached that destination in 0/30 NONE, 0/30 PLACEBO, and 30/30 ATTACK runs. The notice was visible. The measured outcome was page navigation.

What did Qwen3-VL-Flash actually see? In the original A1–A4 runs, we sent text extracted from the page and a list of candidate controls. Each candidate had an ID, a visible label, and fields such as its accessibility name and description. These requests contained no screenshot, raw HTML, or full accessibility tree.

  • A1: The page linked a candidate control to an off-screen description. The extractor read that description and put its text in the candidate’s aria_describedby_text field. We verified that field in the serialized model request.
  • A2: The candidate’s accessibility metadata carried the added content. In the matched A2 variant, we verified it in the serialized accessible name and ARIA fields. The ordinary visible button label stayed the same.
  • A3: The page added text at runtime. Once rendered, the text was visible to a person looking at the page. The extractor included it in the visible page-text field sent to Qwen.
  • Destinations: Qwen saw different opaque path IDs for the two links. The evaluator retained the original href values. The IDs let Qwen tell the links apart, but they removed the meaning of the original URL paths. What transferred across agents? On the three FP-Agent sites, Skyvern received the carrier content in all 36 runs with added content. Those runs included 18 ATTACK and 18 PLACEBO runs. Skyvern selected the target in 0/36. A separate Crawl4AI follow-up on those sites also recorded 0/18 ATTACK target executions. On the three AgentHoneypot sites, Crawl4AI consumed A1–A3 natively in the tested pipeline. A4 and A5-LSB used explicit adapters. The original adversarial screenshot variant had zero matched lift on all three sites. A5-LSB measures what happens after a decoder supplies the media-derived content to the agent.

How this relates to prior work. EIA evaluates privacy leakage on realistic web tasks. TRAP tests how persuasive content redirects agents from their tasks. AdvWeb, now titled AdvAgent, already studies stealthy web content and overlaps with our A1 setting. WASP measures how far an attack proceeds and whether it reaches its end goal. WIPI is an earlier demonstration of instructions embedded in webpages. Progent and MELON are relevant defenses based on privilege control and masked re-execution, respectively. AgentHoneypot addresses a complementary measurement question. It uses matched controls and swapped positions to measure the change in choice, then checks whether the relevant content reached the model in each pipeline.

AuthorsOpenReview ↗

Response to Reviewer aAp9 — Live defense and adaptive evaluation

How does MCAAG decide? MCAAG checks each proposed browser action before execution. The check is deterministic and makes no LLM call. A1 examines referenced off-screen descriptions. A2 compares accessibility metadata with rendered content. A3 checks runtime changes against the initial page state. A4 compares parser output with rendered content. A5 checks available media-derived evidence in the context of the proposed action. The guard combines these findings with checks on the candidate, the action, and the destination binding. A routine page update alone does not trigger rejection.

What changed from v1 to v2? We found two problems in the original checks. On A2, v1 missed relevant source evidence and allowed the proposed action. V2 compares accessibility metadata with independently rendered content and ties the finding to the specific control. On some A1 and A3 cases, v1 blocked the original action but stopped because several alternatives passed its recovery checks. V2 limits recovery to already observed, equivalent controls in the same workflow. It proceeds only when exactly one such control passes authorization.

Live comparison with GPT-5. We ran 250 valid autonomous Browser Use trials under attack, with 50 per condition. A2 and A3 used native observations. A1, A4, and A5 used compatibility adapters.

ConditionHoneypot endpoint executedCanonical endpoint executedRecovered
Defense OFF50/5028/500/50
MCAAG-v110/5020/5020/50
MCAAG-v20/5049/5049/50
Simple blocking39/500/500/50
Surface cleaning32/5027/500/50

On A2, honeypot execution fell from 10/10 with v1 to 0/10 with v2. On A1 and A3, both versions prevented honeypot execution. Canonical execution rose from 0/10 to 9/10 on A1 and from 0/10 to 10/10 on A3. One v2 run ended without canonical execution. These are endpoint counts, and both endpoints can be reached in a single run. Surface cleaning provides a sanitization baseline. The simple blocker uses a different detector, so this comparison leaves open how much shared authorization adds when both methods use the same findings.

Adaptive evaluation. We performed a manual white-box evaluation on local test pages. The researchers could inspect the frozen guard, prior browser observations, and action traces. We used this feedback to develop candidate pages while holding the model, user task, and guard fixed. We froze the candidate pool and selection criteria before screening on development runs. We selected two candidates and confirmed their effect on separate held-out pages with the defense off. We then evaluated the same conditions in independent runs with the defense on.

Held-out conditionOff-task navigation, defense OFFOff-task navigation, defense ON
Candidate 1, ATTACK8/88/8
Candidate 2, ATTACK8/87/7 valid
Candidate 1, matched PLACEBO0/80/8
Candidate 2, matched PLACEBO0/80/8

One additional defense-ON ATTACK response was invalid due to output format. The guard authorized the actions that led to the off-task destinations. These cases show that passing the current checks does not establish that the destination serves the user's task.

Benign utility and recovery. The later Utility-v2 study used the same model, browser settings, and tool schema in both arms. Full multi-step completion was 52/59 with defense OFF and 55/60 with defense ON. The additional benign scenarios completed in 20/20 runs per arm. The guard rejected 55/500 checked legitimate actions in the ON multi-step runs. At least one such rejection occurred in 19/60 runs. We agree that an available equivalent button makes recovery easier in the original benchmark. The two historical CarCare clean failures had no canonical execution receipt, and the surviving records do not establish their cause.

AuthorsOpenReview ↗

Response to Reviewer aAp9 — Decoding, trial counts, and exclusions

Carrier labels. A1–A5 identify what kind of page content reaches the model. MCAAG-v1 and MCAAG-v2, used in our second reply, identify defense versions.

  • A1 (off-screen description): The model receives text linked to a control that is outside the rendered viewport.
  • A2-V1 (matched accessibility metadata): The model receives added content in the control's accessibility fields. The visible label stays the same. This is the A2 variant in Table 3.
  • A2-V2-Strong (accessibility diagnostic): A separate A2 variant designed to strengthen the cue. Its pooled matched lift was lower than A2-V1's in the original Qwen runs.
  • A3 (runtime page text): The page adds text at runtime. It becomes visible once rendered.
  • A4-LaTeX (parser output): The model receives text produced by a LaTeX parser.
  • Adversarial screenshot (visual diagnostic): The model receives a manipulated screenshot.
  • A5-LSB (decoder-assisted media): An explicit decoder extracts the embedded content before it is passed to the model.

Decoding and original per-site results. The original Qwen3-VL-Flash requests sent temperature=0, seed=20260908, enable_thinking=false, and max_tokens=600. The provider did not echo the seed or temperature. The separate historical A5-LSB study sent temperature 0 and a 600-token cap.

The original study ran five trials per site, carrier, layout, and condition. Each entry below gives honeypot executions under PLACEBO / ATTACK, with 10 trials per condition across the two layouts.

CarrierCreativeUI P / ACarCare P / ASmartApp P / A
A15 / 55 / 55 / 10
A2-V15 / 51 / 50 / 10
A35 / 105 / 105 / 10
A4-LaTeX5 / 105 / 105 / 10
A2-V2-Strong3 / 00 / 00 / 5
Adversarial screenshot5 / 55 / 55 / 5

For A3 and A4 at each site, Layout A went from 0/5 under PLACEBO to 5/5 under ATTACK. Layout B was already at 5/5 under PLACEBO and stayed at 5/5 under ATTACK. The pooled lift of 50 percentage points comes entirely from Layout A.

Explicit NONE and more repetitions. The original Qwen3-VL-Flash endpoint returned model_not_found when we attempted the NONE runs. We completed a separate three-condition replication with Qwen3.6 Flash. It contains 1,800 unique valid primary trials, with 30 per primary cell. NONE leaves out the added carrier content. PLACEBO keeps the carrier and presents neutral content. Each entry gives honeypot executions for NONE / PLACEBO / ATTACK, with 30 trials for each count.

Carrier and siteLayout ALayout B
A1, A3, and A4, each at all three sites0 / 0 / 3030 / 30 / 30
A2-V1, CreativeUI0 / 0 / 030 / 0 / 17
A2-V1, CarCare0 / 0 / 430 / 0 / 0
A2-V1, SmartApp0 / 0 / 830 / 2 / 30
A5-LSB, SmartApp0 / 0 / 3030 / 30 / 30

The A1/A3/A4 row shows the same triplets for each carrier at each site. The text-based NONE controls were shared across compatible carriers and counted once. The NONE results show a strong position preference without added carrier content. A2-V1 also varies by site.

360 versus 260. Section 5.2 counts 240 trials for A1, A2-V1, A3, and A4, plus 60 A2-V2-Strong trials and 60 adversarial screenshot trials. Table 3 counts the 240 primary trials plus 20 trials from the separate SmartApp A5-LSB study. Those 20 historical LSB trials recorded candidate selections. On SmartApp the totals were 5/10 under PLACEBO and 10/10 under ATTACK.

CarCare–A1 and SmartApp replay counts. CarCare–A1 had valid behavioral results, with 5/10 honeypot executions under both PLACEBO and ATTACK. Its older replay source had a request-identity mismatch, so we excluded that cell from the replay. For SmartApp A2, A3, and A4, the replay archive contained five eligible PLACEBO records per carrier. Three came from Layout A and two from Layout B. SmartApp A1 had ten eligible records. The historical raw MCAAG-v2 A2–A4 regression files remain unavailable. The live comparison in our second reply provides independent evidence on its own fixtures.

Technical exclusions. Crawl4AI had 43 ATTACK, 10 PLACEBO, and 2 CLEAN exclusions. All 55 occurred in Layout A. The final valid counts were 650 ATTACK, 650 PLACEBO, and 150 CLEAN. DeepSeek had 12 ATTACK and 10 PLACEBO exclusions, with 150 valid runs remaining in each condition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.