- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a class of information integrity issues affecting browser agents. In scenarios where scripts from multiple sources can concurrently modify the same page, agents typically perceive only the final DOM or AXTree, making it impossible to discern which entity modified a specific value, tag, or link during execution. Consequently, third-party code can influence the agent by altering the state of task-relevant web pages without injecting explicit natural-language instructions. The authors propose ProvDOM, a browser execution record based on PageGraph that tags nodes with their creation and modification origins, integrated with a task-aware HTML chunk relevance ranker. Downstream policies leverage this information to block untrusted content, restrict actions, limit data access via restricted readers, and restore affected UI controls. The paper categorizes these attacks as "In-Page State Forgery" and evaluates them across four vectors: link hijacking, value tampering, tag tampering, and usability manipulation. Security experiments utilized 12 tasks within a self-hosted WebArena Shopping environment, employing GPT-5.6 Luna as the primary agent; each task-defense-attack configuration was replicated six times. As shown in Table 2, the "Guarded Quarantine" defense yielded Attack Success Rates (ASR) of 0.0%, 0.0%, 3.4%, and 10.5% across the four attack categories, with a benign Task Success Rate (TSR) of 79.2% (compared to 84.7% without defense). It is important to note that these attack results are conditioned on the successful benign execution of each respective method; thus, they cannot be directly interpreted as a comparison based on identical samples. Relevance experiments demonstrated that chunk ranking achieves favorable target retention and ranking efficiency on Mind2Web, though its transfer to WebArena relies heavily on in-domain training. Overall, while the manuscript supports the premise that execution provenance serves as a valuable security signal, the evidence is insufficient to demonstrate that it provides universal, stable, and low-cost protection against the described threats posed by third-party scripts.
Critical strengths
- Execution provenance provides information that cannot be recovered from the final page state alone, offering clear technical value. The paper focuses not merely on whether the model correctly understands the webpage, but on whether the information received by the agent is sufficient to distinguish between a trustworthy state and a tampered one. Two visually identical DOM nodes may result from different execution processes; analyzing only the final page state makes it impossible to reliably recover this distinction. Browser execution logs thus provide additional evidence that the system can leverage. This aspect grounds the contribution in a clear mechanistic basis. It implies that defenses need not wait for tampered text to exhibit specific linguistic characteristics, nor must they assume that malicious content resides in a separate, isolated region. Although the underlying logging relies on the existing PageGraph framework, the paper demonstrates the research value of translating execution evidence into constraints on agent observations and actions. S2. Policy ablation reveals a critical distinction between integrity protection and task usability. Table 2 goes beyond a simple comparison between a fully protected system and an undefended agent; it also illustrates the varying performance of strategies such as Annotate, Drop, Quarantine, and Guarded Quarantine. I find the following points particularly noteworthy:
- Even when provenance annotations are visible, the agent may still utilize tampered information.
- Preventing a restricted reader from executing instructions embedded in data does not guarantee the authenticity of the returned data.
- Removing affected controls might prevent erroneous actions, but it could also render the task impossible to complete. For instance, the paper reports that compared to standard Quarantine, Guarded Quarantine reduced the Attack Success Rate (ASR) from 34.5% to 0.0% under value-manipulation attacks and increased the Task Success Rate (TSR) from 1.7% to 89.5% under availability-manipulation attacks. Although the nature of the samples under different conditions limits causal interpretation of these differences, these results help pinpoint the effective mechanisms at play and underscore why security evaluations must consider both attack success and task completion.
Critical weaknesses
W1. There is a substantial gap between direct execution attribution and the scope of attacker influence described in the paper.
ProvDOM records the source of code that directly executes DOM operations, rather than the source of the data or calls that triggered those operations. While §2.1 explicitly acknowledges this limitation, the issue is not a lack of disclosure by the authors, but rather that this limitation is not adequately reflected in the core security conclusions.
Figures 3–4 in the paper provide key evidence: a price modification triggered by a third party is ultimately executed by first-party code; consequently, the new price text and associated changes are still attributed to the first party. Under the trust determination criteria in §2.3, this state would not be flagged as untrusted solely based on the aforementioned source recording. This demonstrates that "an attacker influenced the task evidence" is not equivalent to "the system observed an untrusted modification."
§6 also acknowledges that the current method does not propagate indirect effects and cannot recover first-party nodes that have been completely removed from the final DOM—yet the description of attack capabilities in §3 includes the removal of controls required for the task. These boundaries pertain to the threat under discussion itself, rather than being entirely unrelated extended scenarios.
This implies that the current results support defenses against specific modifications directly attributed to third parties, but are insufficient to support broad conclusions regarding the overall "In-Page State Forgery" threat described in the paper. I believe the authors need to clearly articulate the security properties that are guaranteed or empirically verified, and distinguish between cases that are covered by the mechanism, those explicitly excluded, and those that remain unverified. If this can be clarified, I would be happy to Improve my current score.
W2. The reasoning from source labels to "secure recovery" is insufficient, and there are unresolved inconsistencies in the description of trust boundaries.
§2.3 employs an "exact-origin" policy, implying that only nodes where both creation and all modifications originate from the top-level origin are considered trusted. Conversely, Appendix B defines "first-party" based on site and explicitly states that the paper uses site rather than origin as the execution boundary. While these two choices could apply at different levels, the manuscript fails to explain their relationship or their actual implementation in the experiments. For scripts across different subdomains of the same site, these two choices could yield different trust determinations. More importantly, while the execution source indicates "who made the modification," it cannot—on its own—determine whether that modification should be reverted. Regarding numerical processing, the authors acknowledge that old values may become obsolete due to legitimate third-party updates and thus refuse to restore them directly; yet, for disabled controls, they adopt a strategy of restoring "trusted behavior." The manuscript fails to provide sufficient justification for how to determine whether a previously operable state remains consistent with the current application state and user permissions.
The paper also lacks sufficiently clear rules for restoring "safe accessible names" and "trusted behavior"—including requirements for historical state data, conflict resolution between legitimate updates and restoration actions, and consistency between observation and execution. This is a critical omission, as the primary benefits of Guarded Quarantine stem from these proactive handling strategies. Without clearly defined correctness criteria for the restoration process itself, the observed ASR and TSR figures are insufficient to prove that the system preserves correct web semantics. While the current data does not demonstrate that restoration causes errors, it also fails to adequately establish the safety of the mechanism.
W3. Varying sample sets across methods limit the comparability of attack rates and conclusions regarding security utility
Section 4 specifies that attack metrics for each defense are calculated only based on samples where that specific defense successfully completed the task during benign execution. While this conditional approach helps exclude tasks that were inherently impossible to complete, it implies that the ASR and TSR-under-attack values for different methods may correspond to distinct subsets of samples.
Table 2 reveals significant disparities in benign TSR: 84.7% for the baseline (no defense), 51.4% for UCM, 79.2% for Guarded Quarantine, and 75.0% for Guarded Quarantine with correlation filtering. Consequently, pre-selecting the same 12 tasks does not guarantee that the specific execution instances contributing to the metrics share the same difficulty or composition. The current data is insufficient to attribute these performance differences solely to the defense mechanisms.
Furthermore, ASR and TSR are not complementary metrics. In the label manipulation scenario, the baseline agent (no defense) shows an ASR of 96.4% and a TSR of 60.7%, demonstrating that achieving the attack objective and successfully completing the task can occur simultaneously. Therefore, a high task completion rate cannot be directly equated to a high secure completion rate. The report should simultaneously present the actual denominator for each cell, unconditional results based on the fixed task distribution, comparisons within the subset of tasks where benign success is achieved, and the joint metric of "task success without a targeted attack occurring." Since six repetitions per configuration cannot eliminate inter-task variance, uncertainty estimates that account for task clustering should be provided. While existing results offer valuable insights, the magnitude of the relative advantage remains unclear (this section was somewhat confusing to me; a brief clarification would be helpful, though it is not a major issue).
W4. Safety validation is limited to a narrow range of tasks and system configurations; superiority over the baseline is insufficient to support broad comparative conclusions.
The safety experiments cover only 12 tasks within a single self-hosted shopping site; Table 4 corresponds to six task templates, focusing primarily on order inquiries and catalog navigation. The primary defense evaluation relies on a single proxy model. Other models mentioned in the appendix are used mainly for filtering benign tasks and cannot be considered a cross-model validation of the defense mechanism.
The selected tasks were required to be compatible with the specific attack types under study and to pass a screening for benign task completion. While this design facilitates controlled comparisons, the results represent a specific set of vulnerable, completable workflows and are insufficient to support broad generalizations regarding browser agent tasks in general.
The only external defense directly compared is UCM; however, §5 explicitly acknowledges that proactive runtime DOM rewriting falls outside UCM's original threat model. Consequently, while these results illustrate the applicability boundaries of different safety signals, they are insufficient on their own to establish ProvDOM's overall superiority over other applicable defense strategies.
Questions for authors
-
How are modifications triggered by third parties but executed by first-party code handled within the full Guarded Quarantine system? Please specify the set of states the system claims to protect, as well as any attack scenarios described in existing literature that fall outside the current scope of protection.
-
Do the experiments actually employ "exact-origin" or "site" boundaries? What rules govern "restoring trusted behavior" and generating secure names? What evidence demonstrates that these operations do not revert legitimate state updates or re-enable actions that should currently be unavailable?
-
Could you provide the actual sample counts for each cell in Table 2, the intersection of successful samples across methods, and the unconditional results based on a fixed task distribution? What is the joint success rate for "task success without achieving the attack objective"? Do these comparisons hold true when accounting for uncertainty in task clustering?
-
UCM does not cover proactive runtime DOM rewriting. In the current experiments, which results distinguish the benefits of "new origin signals" from those of "stricter action restrictions, blocking, and recovery strategies"? Is there a control condition with matching execution capabilities to support this attribution?
-
Are different datasets used for checkpoint selection versus final reporting in Mind2Web? What do the 75% of chunks retained online actually represent in terms of element and token counts? What is the net impact of filtering on the secure completion rate when evaluated on the same execution set?
8 comments
Q2 / W2. Trust boundary and the rules of Guarded Quarantine.
Our experiments use exact origins (scheme, host, and port), as stated in Section 2.3. Appendix B gives general background on sites and origins, but it incorrectly stated that the paper enforces the site as its boundary. Thanks for the careful read; we corrected Appendix B and Sections 2 and 3 to state that trust is decided by exact origin, so code from a different origin of the same site is also untrusted. The site-based definitions in Appendix B still describe first- and third-party in the usual web sense. In our experiments, the attack scripts were served from a different site than the storefront, so both definitions classify every injected script as third-party.
Guarded Quarantine never restores earlier content from history. It works only on the current DOM and its provenance record, and it applies three rules:
- Values. The reader does not return a value that a third party edited, and we do not substitute an older value, which may be stale.
- Labels. If a third party changed a link’s label but not its destination, the agent sees a placeholder name together with the trusted destination path, for example
link "label hidden -- untrusted" → /sales/order/view/161, and the link stays clickable. Any other control whose accessible name comes from masked content is shown as[untrusted name hidden]. - Disabled controls. At each step, if provenance records a third-party edit to a first-party control and the control is currently disabled, Guarded Quarantine removes the disabling state from the live DOM and recaptures the observation.
Guarded Quarantine changes the live DOM only to remove a disabling state; the value and label rules change only what the agent observes. Actions then run on the live DOM, so observation and execution stay consistent: a link whose label was hidden still goes to the destination path the agent was shown, and actions on masked content remain blocked (Appendix D.4).
We treat values and controls differently because the risks differ. Restoring an old value would present possibly stale data to the agent as fact. Re-enabling a control only lets the agent attempt a first-party action. A disabled state in the page is not an access-control mechanism, and a well-built application still checks permissions on the server when the action is submitted.
The rule for controls has one limitation: it can re-enable a control that the first party legitimately disabled, if a third-party script also edited that control. We cannot rule this out in general. Because ProvDOM records which attribute each actor set (Figure 4), a stricter rule could undo only the disabling edits made by the third party itself.
Q3 / W3. Denominators and secure completion.
Table 2 conditions each method’s attack metrics on its own benign successes, and we agree that the different denominators complicate direct comparisons. Value manipulation applies to the seven retrieval tasks in Table 4 and label manipulation to the five navigation tasks (Appendices F.4 and F.5), while hijacking and availability manipulation apply to all 12. With six runs each, this gives 72, 42, 30, and 72 attacked runs. Each entry below gives the number of attacked runs whose paired benign run succeeded:
| Method | Hijack | Evidence: value | Evidence: label | Availability |
|---|---|---|---|---|
| No Defense | 61/72 | 33/42 | 28/30 | 61/72 |
| UCM | 37/72 | 26/42 | 11/30 | 37/72 |
| Annotate | 57/72 | 30/42 | 27/30 | 57/72 |
| Drop | 56/72 | 30/42 | 26/30 | 56/72 |
| Quarantine | 58/72 | 29/42 | 29/30 | 58/72 |
| Guarded Quarantine | 57/72 | 28/42 | 29/30 | 57/72 |
| GQ + Relevance | 54/72 | 33/42 | 21/30 | 54/72 |
To compare methods on the same runs, we report secure completion: the fraction of all 216 attacked runs, pooled across the four attacks, in which the agent completes the task and the attack fails. This is the joint metric you suggest, and it counts every run, whether or not its paired benign run succeeded. ASR and TSR below are computed on the same 216 runs, and the confidence intervals are 95% bootstrap intervals that resample tasks.
| Method | Secure completion (95% CI) | ASR | TSR |
|---|---|---|---|
| No Defense | 6.5% [2.3, 12.0] | 89.8% | 14.8% |
| UCM | 2.8% [0.0, 8.3] | 77.8% | 6.0% |
| Annotate | 5.1% [0.5, 11.6] | 90.3% | 12.0% |
| Drop | 19.4% [10.2, 29.2] | 39.4% | 19.4% |
| Quarantine | 22.7% [13.0, 33.8] | 42.1% | 22.7% |
| Guarded Quarantine | 56.9% [38.4, 74.5] | 8.8% | 56.9% |
| GQ + Relevance | 56.0% [37.5, 73.6] | 9.7% | 56.9% |
As you note, ASR and TSR are not complementary. Without a defense, the agent completes 14.8% of attacked runs but only 6.5% securely; under label manipulation, for example, ASR and TSR are 96.4% and 60.7% (Table 2). Under Guarded Quarantine, every completed run is also secure. Its interval does not overlap with that of any other method except GQ + Relevance, so its improvement in secure completion over those methods holds under task-level uncertainty.
Thank you for the careful review. Your reading of Figures 3 and 4 and of the policy ablations is precise, and your questions helped us state the paper’s claims more exactly. We answer each question below, together with the related weakness. All changes in the revised PDF are highlighted in blue.
Q1 / W1. What ProvDOM protects, and what it does not.
We agree that “an attacker influenced the task evidence” is not the same as “the system observed an untrusted modification,” and we revised the security claims in the abstract, Sections 1, 3, and 6 to match what the mechanism provides. We now separate three categories.
Covered and evaluated. A third-party script directly edits a first-party node that remains in the final DOM, including replacing its text. ProvDOM records the origin of the code that performs each such write, so the trust rule flags the node. Our evaluation tests direct third-party edits to link destinations, displayed values, labels, and the disabled state of controls (Table 2), and our security claims are limited to this category.
Covered by the mechanism, but not evaluated. Content that a third-party script creates outright is untrusted under the same rule, for example, an adversarial pop-up inserted by a third-party script (Section 3). We have not evaluated such cases, nor combined attacks such as disabling a control while also forging its label.
Outside the current scope:
In every category, the trust decision depends only on provenance, so it is unaffected by whether the attacker shows changes only to automated visitors.