- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AgentHoneypot, a framework designed to evaluate whether the injection of machine-readable content alters the action selection of web agents. The benchmark establishes "canonical" and "honeypot" candidates—sharing identical visible task semantics—across three controlled websites; it distinguishes between inherent selection preferences and injection-induced behavioral changes through carrier-matched ATTACK/PLACEBO controls and candidate position swapping. The study examines five categories of carriers: hidden references, accessibility metadata, runtime-injected text, parser output, and decoder-assisted LSB media content. The authors further distinguish between reachability (whether the payload enters the model's observation space) and exploitability (whether the payload alters action selection), reporting varying results across Qwen, DeepSeek, Skyvern, Crawl4AI, and FP-Agent setups. The proposed defense method, MCAAG, binds carrier-related evidence to specific candidates and performs authorization checks prior to execution. Upon rejection, the system deterministically recovers if an equivalent candidate exists that satisfies the same authorization policy. The paper reports on the interception of successful attack replays, 85 PLACEBO control trials, and 180 "clean" task runs using Browser Use; the latter showed a task success rate shift from 90/90 to 88/90, with a median local authorization latency of 27.0 ms.
Critical strengths
- First, the paper accurately identifies a critical issue: an agent selecting the button preferred by the attacker does not necessarily mean the injection was effective; it could simply reflect the button's original position or a pre-existing preference. By using a control setup that preserves the carrier and page structure while swapping only the selection instruction, the study isolates the "additional impact caused by the malicious instruction." The value of this design is evident in the results:
- For DeepSeek A1 (Table 6), the "honeypot" button was selected 15 out of 30 times under the ATTACK condition, but the PLACEBO condition already showed a selection rate of 14 out of 30; thus, the net increase was only 3.3 percentage points. Treating the 50% figure directly as the injection's effectiveness would be highly misleading.
- Appendix E.1 shows that while A3 and A4 jumped from 0% to 100% in Layout A, the PLACEBO condition in Layout B was already at 100%. Consequently, the aggregated +50 percentage point increase cannot be interpreted as uniform vulnerability across all layouts. This contribution improves the quality of security assessment conclusions. Even without introducing new attack categories, accurately measuring "what exactly the attack changes" is a topic that warrants significant attention.
- Distinguishing between reachability and exploitability—and reporting negative results—provides diagnostic insights far more valuable than a simple attack leaderboard. The paper avoids interpreting every failure as evidence of model security and refrains from characterizing every adapter experiment as an inherent vulnerability. For instance, Skyvern’s native A2 was successfully injected into the model's context but resulted in 0/10 selections under both conditions; similarly, while the content for FP-Agent’s A2 and A3 was confirmed to be part of the model prompt, the target was not selected across 36 payload runs. In contrast, the attack selection rate for Crawl4AI’s native A2 was 45/50, 43/50, and 15/50 across the three tested websites. Collectively, these results demonstrate that merely exposing a payload to the model does not determine its behavioral impact. The authors clearly defined the scope regarding the adapter paths for Skyvern and Crawl4AI, the explicit decoder for A5, and the repeated experiments on fixed websites, thereby establishing clear boundaries for interpreting the results. The paper offers practical methods for determining whether a failure occurs during the observation phase or the decision-making phase; these negative results also help avoid overgeneralizing the effectiveness of specific agents.
Critical weaknesses
W1. MCAAG's key authorization and detection logic lacks operational definitions; the current PDF does not sufficiently support independent replication or an assessment of its security boundaries.
Section 4.1 describes how various modules check for hidden references, accessibility metadata, runtime modifications, parsing inconsistencies, and media sources, outputting:
Appendix D defines authorization as the conjunction of six checks:
\land I_{\mathrm{binding}}\land I_{\mathrm{dest}}\land I_{\mathrm{integrity}}.$$ However, neither the main text nor the appendix adequately explains how these checks are calculated. For instance, what conditions render a standard accessibility description "suspicious" evidence? How is "suspicious" defined? How does the system distinguish between legitimate runtime page updates and malicious injections? The absence of such explanations directly impacts the validity conditions of the method. While the paper notes that the defense ignores `canonical`/`honeypot` tags—thereby ruling out the shortcut of simply reading evaluation labels—it fails to specify the actual rules employed by the detector. ### W2. The link between induced behavior and unauthorized actions is insufficiently established; the connection between benchmark labels and security semantics requires further clarification. Section 3.2 and Appendix B configure the two candidates to share identical visible task semantics while retaining their `canonical`/`honeypot` identities on the evaluation side. While this design is reasonable for studying behavioral deviation, it raises a question: **Apart from the evaluation-side identity, what trustworthy, deployment-time evidence indicates that one action aligns with user authorization while the other does not?** This issue is evident in the PLACEBO results: out of 85 benign controls, only 51 `canonical` cases succeeded; MCAAG left all behaviors unchanged and resulted in no "unresolved" cases. Consequently, the defense does not universally rectify non-`canonical` selections defined by the evaluation side; instead, it primarily rejects candidates supported by suspicious payloads. This does not invalidate the concept of "matched lift," but it suggests that the following two propositions must be treated separately: 1. Injected content alters the candidate selection. 2. The defense identifies and blocks actions that violate user authorization. The evidence supporting the first proposition is significantly stronger than that for the second. While the concepts of "destination consistency" and "task compatibility" in Appendix D might offer a link, the paper fails to provide specific examples detailed enough for verification. ### W3. Existing ablation studies fail to demonstrate the incremental value of MCAAG over simple candidate-blocking strategies The ablation results in Appendix I show that: - Removing the "media finding" step allows all 10 honeypot proposals in A5 to execute; - Bypassing authorization also results in full execution; - Removing the "recovery" step maintains the blocking action, but leaves all cases "unresolved." While these results confirm that system components are integrated into the execution flow, they do not sufficiently demonstrate the necessity or advantages of the shared authorization policy. In particular, the "attack execution after bypassing authorization" scenario cannot distinguish between a complex authorization mechanism and a simple kill-switch. The paper omits a crucial control experiment: **using the same payload evidence to directly reject flagged candidates and select the sole remaining, equivalent, unflagged candidate.** In scenarios involving only two matching terminal candidates, this alternative strategy might cover many of the same cases; experimental data is required to determine this. The authors acknowledge in the "Related Work" section that provenance-aware authorization is not a novel concept in itself; this makes it even more critical to articulate the specific incremental benefits provided by candidate binding and shared authorization. Current evidence confirms that MCAAG functions in the tested cases, but it does not clearly demonstrate that its core design outperforms simpler defense mechanisms that utilize the same information. Consequently, it is difficult to substantiate the actual effectiveness of the proposed approach. ### W4. The benign utility evaluation suffers from selection bias based on attack outcomes and fails to cover benign inputs that would best test for false rejections Appendix J explicitly states that benign replays for CreativeUI–A2 were not executed because the underlying attack failed to produce an effect. **The fact that an attack is ineffective under certain conditions does not mean the defense will not falsely reject benign inputs under those same conditions.** Therefore, the assessment of benign utility should not be contingent upon the prior observation of a successful attack. Furthermore, PLACEBO constructs its test cases by eliminating malicious selection preferences. While it can check whether the defense excessively rejects the carrier itself, it may not detect more challenging benign cases—such as accessibility descriptions containing instructions for normal actions, page text prompting user interaction following a legitimate update, or genuine task steps conveyed via the parser. The 180 "clean runs" using Browser Use provided a valuable supplement; however, they relied on the same three controlled websites, and the two failures observed with CarCare were not further analyzed or attributed. ### W5. Gaps in Verifiability Regarding Sample Composition, Statistical Intervals, and Raw Evidence Three specific issues exist: - **Main Experiment Counts:** Section 5.2 reports 360 valid runs for the Qwen main experiment. However, the design outlined in Table 3 for categories A1–A4 specifies 30 PLACEBO runs and 30 ATTACK runs per category (totaling 240), while the 20 runs for A5 are explicitly listed as an additional experiment. The remaining 120 runs likely correspond to ablation studies or other conditions, but the PDF does not provide a complete mapping. - **Confidence Intervals:** Table 6 cites Newcombe intervals; however, calculations based on the table's counts show that the reported values can be replicated by directly subtracting the endpoints of two Wilson intervals. For instance, A1 yields a range of approximately $[-30.7, 36.6]$ percentage points. The authors need to clarify the formula actually employed and explain how matching and stratified layout factors were incorporated into the calculation. - **Missing Raw Records:** Appendix L notes that the raw JSON logs for the historical MCAAG-v2 A2–A4 regression tests have been lost. Consequently, related claims rely on surviving reports and snapshots, yet the authors do not specify which claims still depend on these missing records versus those supported by independent re-runs.Questions for authors
-
Please list the six authorization predicates and the specific decision rules for each carrier module, and present at least one ATTACK/PLACEBO pair showing the raw observations, findings, candidate bindings, and final decisions. Does the detector rely on fixed payload vocabulary, candidate IDs, or path features?
-
What evidence visible at deployment—without reading canonical/honeypot labels—independently determines the legitimacy of an action? Please explain why non-canonical selections in PLACEBO should be preserved, and the difference in security semantics between such a selection being retained in PLACEBO versus the same candidate being rejected in an ATTACK scenario.
-
What results are achieved by a simple strategy of "directly rejecting marked candidates" when using identical carrier evidence? Does shared authorization allow marked-but-legitimate actions, or reject unmarked-but-illegitimate actions, in certain instances?
-
Why does the benign evaluation of CreativeUI–A2 depend on whether the attack is effective? Are there recorded instances of false rejections involving accessibility text, runtime updates, or parsed content that contained legitimate operational suggestions? What specifically caused the two failures in CarCare?
-
Are there results from closed-loop experiments (defense ON vs. OFF) under attack conditions within the same agent pipeline? If so, please report honeypot execution, original task completion, recovery outcomes, and unresolved cases, rather than just the interception results for frozen proposals.
-
Please provide a table counting the conditions for the 360 main experiments and all behavioral trials, the formulas used to calculate the intervals in Table 6, and the correspondence between the three missing raw data files and the claims made in the current paper.
7 comments
Response to Reviewer p1Ks — Baselines, live evaluation, and benign utility
W3. What do the ablations and simpler baselines establish? The ablations show that the carrier finding, the pre-action check, and recovery each affect the outcome. They do not establish that shared authorization outperforms a policy that uses the same findings to reject a candidate and choose the sole unflagged equivalent. Our current simple-blocking baseline uses a different detector. The exact same-evidence comparison remains open, especially on two-button tasks where the strategies may agree.
Under the current policy, a blocking finding fails the integrity check even if the action later proves legitimate. We observed this in the benign tests below. An unflagged candidate can still fail a separate check. For example, in four controlled component cases with no carrier findings, the candidate failed task compatibility. This shows that the two checks can make distinct decisions. It does not establish that task compatibility reliably detects unauthorized actions.
Closed-loop evidence. We added 250 valid autonomous GPT-5 Browser Use trials under attack. All five arms used the same agent pipeline. A2 and A3 used native observations, while A1, A4, and A5 used compatibility adapters.
| Arm | Honeypot endpoint | Canonical endpoint | Recovered | Unresolved |
|---|---|---|---|---|
| Defense OFF | 50/50 | 28/50 | 0/50 | 22/50 |
| MCAAG-v1 | 10/50 | 20/50 | 20/50 | 30/50 |
| MCAAG-v2 | 0/50 | 49/50 | 49/50 | 1/50 |
| Simple blocking | 39/50 | 0/50 | 0/50 | 50/50 |
| Surface cleaning | 32/50 | 27/50 | 0/50 | 23/50 |
Canonical execution means reaching the designated task endpoint. Both endpoints can be reached in one run. Unresolved means that the canonical endpoint was never reached. Full multi-step task completion was scored separately in the benign evaluation below.
The live results apply to the fixed attack fixtures. In a separate manual white-box evaluation of the frozen guard, two selected held-out ATTACK conditions caused off-task navigation in 8/8 and 7/7 valid defense-ON runs. Both matched PLACEBO conditions were 0/8. One additional ATTACK response was invalid due to output format. The guard authorized the actions that reached the off-task destination. We therefore limit our claim to the observed interception and recovery results.
W4. Benign coverage and false rejections. We agree that the historical CreativeUI–A2 exclusion was a gap. It was skipped because the original attack showed no lift, which gives no information about false rejection. Later scripted checks accepted the normal CreativeUI–A2 proposal in both layouts. The full matched benign replay for that cell remains missing.
We also observed false rejections in two separate scripted accessibility cases, one with a legitimate accessible name and one with a benign accessibility note. In the autonomous Utility-v2 evaluation, full multi-step completion was 52/59 OFF and 55/60 ON. At least one legitimate action was rejected in 19/60 ON runs, with 55/500 checked legitimate actions rejected. The four additional benign scenarios completed in 20/20 runs per arm. The legitimate dynamic-content scenario recorded one rejection in six checked actions. These results leave the false-rejection rate on legitimate parser-supplied instructions open.
The two historical CarCare clean runs ended without a canonical execution receipt. The surviving records establish that outcome, but do not identify whether a guard decision or another cause prevented completion.
Response to Reviewer p1Ks — Trial accounting, corrected intervals, and archived evidence
W5. Trial accounting. The original 360-run Qwen batch had six variants. Each used three sites, two layouts, and five trials per site/layout/condition.
| Original Qwen3-VL-Flash variant | PLACEBO | ATTACK |
|---|---|---|
| A1, off-screen description | 30 | 30 |
| A2-V1, matched accessibility metadata | 30 | 30 |
| A3, visible runtime text | 30 | 30 |
| A4-LaTeX, parser output | 30 | 30 |
| A2-V2-Strong, accessibility diagnostic | 30 | 30 |
| Adversarial screenshot, visual diagnostic | 30 | 30 |
| Total | 180 | 180 |
Table 3 includes the first four rows (240 runs) plus the separate SmartApp A5-LSB study (10 PLACEBO and 10 ATTACK). Those 20 historical LSB runs recorded candidate selections.
The six main cohorts with explicit denominators in the paper reconcile to 2,410 valid episodes.
| Cohort | Valid episodes | Conditions |
|---|---|---|
| Original Qwen3-VL-Flash batch | 360 | 180 PLACEBO, 180 ATTACK |
| Qwen SmartApp A5-LSB | 20 | 10 PLACEBO, 10 ATTACK |
| DeepSeek replication | 300 | 150 PLACEBO, 150 ATTACK |
| Skyvern formal transfer | 100 | 50 PLACEBO, 50 ATTACK |
| Crawl4AI formal evaluation | 1,450 | 650 PLACEBO, 650 ATTACK, 150 CLEAN |
| Browser Use clean utility | 180 | 90 OFF, 90 ON |
The paper also reports 40 A4-Markdown trials and 36 carrier-bearing Skyvern runs on FP-Agent sites as supplemental results. The 36 are part of a 42-run pilot with six additional clean controls. The 85 benign PLACEBO replays and the defense ablations reuse archived proposals. They do not add fresh behavioral episodes to the total.
Table 6 correction. The published intervals subtracted Wilson endpoints and were incorrectly labeled Newcombe. We recomputed them from the original counts. Within a layout we use Newcombe's independent-proportion hybrid score interval. For the layout-stratified result we combine the two layouts with equal weights using Wilson–MOVER. Newcombe, 1998
For each layout , let and be the ATTACK and PLACEBO rates. Their Wilson 95% intervals are and . We give each of the two layouts equal weight, . The estimated lift is
The lower and upper limits are
The model calls were independent within each layout. Matching refers to the experimental conditions. The table reports the lift and interval limits in percentage points.
| Carrier | Layout A, PLACEBO / ATTACK | Layout B, PLACEBO / ATTACK | Equal-layout lift [95% CI], pp |
|---|---|---|---|
| A1 | 0/15 / 1/15 | 14/15 / 14/15 | +3.33 [−12.57, 19.93] |
| A2-V1 | 0/15 / 0/15 | 0/15 / 8/15 | +26.67 [8.16, 41.61] |
| A3, A4-LaTeX, and A5-LSB, each | 0/15 / 15/15 | 15/15 / 15/15 | +50.00 [32.34, 60.19] |
A3, A4, and A5 each show a lift in Layout A and a saturated PLACEBO rate in Layout B. These estimates describe the tested fixtures.
Missing raw records. The old A2–A4 regression JSON files remain unavailable. The surviving reports describe five blocks and five recoveries in each historical cell listed below. The newer live experiments are independent evidence for the specified cells under the revised v2 integration.
| Missing raw file | Historical cells supported by surviving reports | Independent live coverage |
|---|---|---|
a2_regression.json | A2 on SmartApp | A2 on CreativeUI |
a3_regression.json | A3 on CreativeUI and SmartApp | A3 on CreativeUI |
a4_regression.json | A4 on CreativeUI and SmartApp | A4 on CarCare |
The original per-trial findings and recovery decisions for those historical cells still depend on the surviving reports and snapshots.
Response to Reviewer p1Ks — Authorization checks and a paired example
Thank you for identifying the distinction between a change in candidate selection and an action that violates the user's task. The matched-twin benchmark measures the former. We use the separate task-diversion experiment to measure the latter as navigation to a distinct off-task destination.
W1. How the guard decides. MCAAG makes a deterministic decision before executing the action. It makes no LLM call. The six checks are:
A separate input check excludes evaluator-only fields. Runtime candidate IDs identify browser elements. They do not carry the canonical or honeypot labels. The destination is used to check the action binding. This check does not determine whether the destination itself serves the user's task.
The carrier modules inspect different evidence. A1 checks candidate-linked descriptions outside the rendered view. A2 checks whether action guidance in accessibility metadata is supported by the rendered control and its context. A3 checks candidate-linked guidance introduced after the initial page observation. An ordinary page update alone is insufficient. A4 checks guidance present in parsed content against the rendered context. A5 checks action guidance in available media-derived content and binds it to the referenced candidate. These are lexical and structural heuristics. A finding depends on the observed source and its relation to the proposed action.
W2. A paired replay example. In SmartApp A5-LSB, Layout B, repetition 1, both recorded proposals selected control A. Both conditions used the same media pipeline. The following summarizes the recorded observations and decisions with the controls labeled A and B.
The same evaluator-labeled alternate can therefore be kept under PLACEBO and rejected under ATTACK. The change in decision comes from the evidence attached to the action. In the 85 benign replays, all proposals were directly authorized, including the 34 non-canonical choices. This is the expected result for visually equivalent controls with no blocking evidence.
For an independent check on task diversion, our visible-notice experiment measured navigation to a different destination from the user's request. In each layout, that destination was reached in 30/30 ATTACK, 0/30 PLACEBO, and 0/30 NONE runs. Here NONE is the original page without an added notice.