acceptodds
Under review as a conference paper at ICLR 2027

What the DOM Forgets: Provenance, Relevance and Browser-Agent Security

Abstract

Browser agents operate on browsers optimized around human limitations, despite facing very different constraints. Humans need browsers to be fast and highly interactive, and often struggle to make accurate security decisions from the single URL presented in the address bar. As a result, browsers discard enormous amounts of information that is too costly or confusing for human users, including which actors created or edited which elements in the page. The different tradeoffs faced by agentic browsers make it practical to track this per-element edit history and use it to make better security and privacy decisions. We propose ProvDOM, an agent-optimized system for improving agentic security in two steps. First, ProvDOM tracks which actors (i.e., script origins) created and edited each element in the page. This is achieved by instrumenting the browser's execution runtime, and so is transparent to the webpage. Second, ProvDOM supplies these per-element annotations to a task-conditioned relevance scorer that distinguishes page content relevant to the user's goals from content that is irrelevant or maliciously injected. We then demonstrate that ProvDOM protects against attacks that current systems fail to defend against. We define a category of attack we call In-Page State Forgery, in which an attacker rewrites task-relevant portions of the page to alter an agent's decision making, exploiting multi-principal composition against agents fed a representation built for humans. Because ProvDOM uniquely tracks which page content was created or modified by which actors, it defends against such agent-confusion attacks where current approaches fail.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Browser-Agent Security, Data Provenance, Web Agents, Indirect Prompt Injection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8ST3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a class of information integrity issues affecting browser agents. In scenarios where scripts from multiple sources can concurrently modify the same page, agents typically perceive only the final DOM or AXTree, making it impossible to discern which entity modified a specific value, tag, or link during execution. Consequently, third-party code can influence the agent by altering the state of task-relevant web pages without injecting explicit natural-language instructions. The authors propose ProvDOM, a browser execution record based on PageGraph that tags nodes with their creation and modification origins, integrated with a task-aware HTML chunk relevance ranker. Downstream policies leverage this information to block untrusted content, restrict actions, limit data access via restricted readers, and restore affected UI controls. The paper categorizes these attacks as "In-Page State Forgery" and evaluates them across four vectors: link hijacking, value tampering, tag tampering, and usability manipulation. Security experiments utilized 12 tasks within a self-hosted WebArena Shopping environment, employing GPT-5.6 Luna as the primary agent; each task-defense-attack configuration was replicated six times. As shown in Table 2, the "Guarded Quarantine" defense yielded Attack Success Rates (ASR) of 0.0%, 0.0%, 3.4%, and 10.5% across the four attack categories, with a benign Task Success Rate (TSR) of 79.2% (compared to 84.7% without defense). It is important to note that these attack results are conditioned on the successful benign execution of each respective method; thus, they cannot be directly interpreted as a comparison based on identical samples. Relevance experiments demonstrated that chunk ranking achieves favorable target retention and ranking efficiency on Mind2Web, though its transfer to WebArena relies heavily on in-domain training. Overall, while the manuscript supports the premise that execution provenance serves as a valuable security signal, the evidence is insufficient to demonstrate that it provides universal, stable, and low-cost protection against the described threats posed by third-party scripts.

Critical strengths
  1. Execution provenance provides information that cannot be recovered from the final page state alone, offering clear technical value. The paper focuses not merely on whether the model correctly understands the webpage, but on whether the information received by the agent is sufficient to distinguish between a trustworthy state and a tampered one. Two visually identical DOM nodes may result from different execution processes; analyzing only the final page state makes it impossible to reliably recover this distinction. Browser execution logs thus provide additional evidence that the system can leverage. This aspect grounds the contribution in a clear mechanistic basis. It implies that defenses need not wait for tampered text to exhibit specific linguistic characteristics, nor must they assume that malicious content resides in a separate, isolated region. Although the underlying logging relies on the existing PageGraph framework, the paper demonstrates the research value of translating execution evidence into constraints on agent observations and actions. S2. Policy ablation reveals a critical distinction between integrity protection and task usability. Table 2 goes beyond a simple comparison between a fully protected system and an undefended agent; it also illustrates the varying performance of strategies such as Annotate, Drop, Quarantine, and Guarded Quarantine. I find the following points particularly noteworthy:
  • Even when provenance annotations are visible, the agent may still utilize tampered information.
  • Preventing a restricted reader from executing instructions embedded in data does not guarantee the authenticity of the returned data.
  • Removing affected controls might prevent erroneous actions, but it could also render the task impossible to complete. For instance, the paper reports that compared to standard Quarantine, Guarded Quarantine reduced the Attack Success Rate (ASR) from 34.5% to 0.0% under value-manipulation attacks and increased the Task Success Rate (TSR) from 1.7% to 89.5% under availability-manipulation attacks. Although the nature of the samples under different conditions limits causal interpretation of these differences, these results help pinpoint the effective mechanisms at play and underscore why security evaluations must consider both attack success and task completion.
Critical weaknesses

W1. There is a substantial gap between direct execution attribution and the scope of attacker influence described in the paper.

ProvDOM records the source of code that directly executes DOM operations, rather than the source of the data or calls that triggered those operations. While §2.1 explicitly acknowledges this limitation, the issue is not a lack of disclosure by the authors, but rather that this limitation is not adequately reflected in the core security conclusions.

Figures 3–4 in the paper provide key evidence: a price modification triggered by a third party is ultimately executed by first-party code; consequently, the new price text and associated changes are still attributed to the first party. Under the trust determination criteria in §2.3, this state would not be flagged as untrusted solely based on the aforementioned source recording. This demonstrates that "an attacker influenced the task evidence" is not equivalent to "the system observed an untrusted modification."

§6 also acknowledges that the current method does not propagate indirect effects and cannot recover first-party nodes that have been completely removed from the final DOM—yet the description of attack capabilities in §3 includes the removal of controls required for the task. These boundaries pertain to the threat under discussion itself, rather than being entirely unrelated extended scenarios.

This implies that the current results support defenses against specific modifications directly attributed to third parties, but are insufficient to support broad conclusions regarding the overall "In-Page State Forgery" threat described in the paper. I believe the authors need to clearly articulate the security properties that are guaranteed or empirically verified, and distinguish between cases that are covered by the mechanism, those explicitly excluded, and those that remain unverified. If this can be clarified, I would be happy to Improve my current score.

W2. The reasoning from source labels to "secure recovery" is insufficient, and there are unresolved inconsistencies in the description of trust boundaries.

§2.3 employs an "exact-origin" policy, implying that only nodes where both creation and all modifications originate from the top-level origin are considered trusted. Conversely, Appendix B defines "first-party" based on site and explicitly states that the paper uses site rather than origin as the execution boundary. While these two choices could apply at different levels, the manuscript fails to explain their relationship or their actual implementation in the experiments. For scripts across different subdomains of the same site, these two choices could yield different trust determinations. More importantly, while the execution source indicates "who made the modification," it cannot—on its own—determine whether that modification should be reverted. Regarding numerical processing, the authors acknowledge that old values ​​may become obsolete due to legitimate third-party updates and thus refuse to restore them directly; yet, for disabled controls, they adopt a strategy of restoring "trusted behavior." The manuscript fails to provide sufficient justification for how to determine whether a previously operable state remains consistent with the current application state and user permissions.

The paper also lacks sufficiently clear rules for restoring "safe accessible names" and "trusted behavior"—including requirements for historical state data, conflict resolution between legitimate updates and restoration actions, and consistency between observation and execution. This is a critical omission, as the primary benefits of Guarded Quarantine stem from these proactive handling strategies. Without clearly defined correctness criteria for the restoration process itself, the observed ASR and TSR figures are insufficient to prove that the system preserves correct web semantics. While the current data does not demonstrate that restoration causes errors, it also fails to adequately establish the safety of the mechanism.

W3. Varying sample sets across methods limit the comparability of attack rates and conclusions regarding security utility

Section 4 specifies that attack metrics for each defense are calculated only based on samples where that specific defense successfully completed the task during benign execution. While this conditional approach helps exclude tasks that were inherently impossible to complete, it implies that the ASR and TSR-under-attack values ​​for different methods may correspond to distinct subsets of samples.

Table 2 reveals significant disparities in benign TSR: 84.7% for the baseline (no defense), 51.4% for UCM, 79.2% for Guarded Quarantine, and 75.0% for Guarded Quarantine with correlation filtering. Consequently, pre-selecting the same 12 tasks does not guarantee that the specific execution instances contributing to the metrics share the same difficulty or composition. The current data is insufficient to attribute these performance differences solely to the defense mechanisms.

Furthermore, ASR and TSR are not complementary metrics. In the label manipulation scenario, the baseline agent (no defense) shows an ASR of 96.4% and a TSR of 60.7%, demonstrating that achieving the attack objective and successfully completing the task can occur simultaneously. Therefore, a high task completion rate cannot be directly equated to a high secure completion rate. The report should simultaneously present the actual denominator for each cell, unconditional results based on the fixed task distribution, comparisons within the subset of tasks where benign success is achieved, and the joint metric of "task success without a targeted attack occurring." Since six repetitions per configuration cannot eliminate inter-task variance, uncertainty estimates that account for task clustering should be provided. While existing results offer valuable insights, the magnitude of the relative advantage remains unclear (this section was somewhat confusing to me; a brief clarification would be helpful, though it is not a major issue).

W4. Safety validation is limited to a narrow range of tasks and system configurations; superiority over the baseline is insufficient to support broad comparative conclusions.

The safety experiments cover only 12 tasks within a single self-hosted shopping site; Table 4 corresponds to six task templates, focusing primarily on order inquiries and catalog navigation. The primary defense evaluation relies on a single proxy model. Other models mentioned in the appendix are used mainly for filtering benign tasks and cannot be considered a cross-model validation of the defense mechanism.

The selected tasks were required to be compatible with the specific attack types under study and to pass a screening for benign task completion. While this design facilitates controlled comparisons, the results represent a specific set of vulnerable, completable workflows and are insufficient to support broad generalizations regarding browser agent tasks in general.

The only external defense directly compared is UCM; however, §5 explicitly acknowledges that proactive runtime DOM rewriting falls outside UCM's original threat model. Consequently, while these results illustrate the applicability boundaries of different safety signals, they are insufficient on their own to establish ProvDOM's overall superiority over other applicable defense strategies.

Questions for authors
  1. How are modifications triggered by third parties but executed by first-party code handled within the full Guarded Quarantine system? Please specify the set of states the system claims to protect, as well as any attack scenarios described in existing literature that fall outside the current scope of protection.

  2. Do the experiments actually employ "exact-origin" or "site" boundaries? What rules govern "restoring trusted behavior" and generating secure names? What evidence demonstrates that these operations do not revert legitimate state updates or re-enable actions that should currently be unavailable?

  3. Could you provide the actual sample counts for each cell in Table 2, the intersection of successful samples across methods, and the unconditional results based on a fixed task distribution? What is the joint success rate for "task success without achieving the attack objective"? Do these comparisons hold true when accounting for uncertainty in task clustering?

  4. UCM does not cover proactive runtime DOM rewriting. In the current experiments, which results distinguish the benefits of "new origin signals" from those of "stricter action restrictions, blocking, and recovery strategies"? Is there a control condition with matching execution capabilities to support this attribution?

  5. Are different datasets used for checkpoint selection versus final reporting in Mind2Web? What do the 75% of chunks retained online actually represent in terms of element and token counts? What is the net impact of filtering on the secure completion rate when evaluated on the same execution set?

8 comments

Thank you for the careful review. Your reading of Figures 3 and 4 and of the policy ablations is precise, and your questions helped us state the paper’s claims more exactly. We answer each question below, together with the related weakness. All changes in the revised PDF are highlighted in blue.

Q1 / W1. What ProvDOM protects, and what it does not.

We agree that “an attacker influenced the task evidence” is not the same as “the system observed an untrusted modification,” and we revised the security claims in the abstract, Sections 1, 3, and 6 to match what the mechanism provides. We now separate three categories.

Covered and evaluated. A third-party script directly edits a first-party node that remains in the final DOM, including replacing its text. ProvDOM records the origin of the code that performs each such write, so the trust rule flags the node. Our evaluation tests direct third-party edits to link destinations, displayed values, labels, and the disabled state of controls (Table 2), and our security claims are limited to this category.

Covered by the mechanism, but not evaluated. Content that a third-party script creates outright is untrusted under the same rule, for example, an adversarial pop-up inserted by a third-party script (Section 3). We have not evaluated such cases, nor combined attacks such as disabling a control while also forging its label.

Outside the current scope:

  • Third-party influence carried out by first-party code. In Figures 3 and 4, the third party supplies the price, but the first-party applyPromo function writes it, so the write is attributed to the first party and Guarded Quarantine does not flag the forged price (Section 6).
  • Indirect effects, such as a third-party CSS change on an ancestor (Section 6).
  • First-party nodes that a third party removes entirely, since provenance is attached to the final DOM (Section 6). Section 3 now notes that removal is outside our protection.
  • Content served by the site itself, including content written by others within the first-party origin, such as user comments, syndicated material, or AI-generated summaries (Section 6).
  • A compromised first-party server, which our threat model excludes (Section 3).

In every category, the trust decision depends only on provenance, so it is unaffected by whether the attacker shows changes only to automated visitors.

Q2 / W2. Trust boundary and the rules of Guarded Quarantine.

Our experiments use exact origins (scheme, host, and port), as stated in Section 2.3. Appendix B gives general background on sites and origins, but it incorrectly stated that the paper enforces the site as its boundary. Thanks for the careful read; we corrected Appendix B and Sections 2 and 3 to state that trust is decided by exact origin, so code from a different origin of the same site is also untrusted. The site-based definitions in Appendix B still describe first- and third-party in the usual web sense. In our experiments, the attack scripts were served from a different site than the storefront, so both definitions classify every injected script as third-party.

Guarded Quarantine never restores earlier content from history. It works only on the current DOM and its provenance record, and it applies three rules:

  • Values. The reader does not return a value that a third party edited, and we do not substitute an older value, which may be stale.
  • Labels. If a third party changed a link’s label but not its destination, the agent sees a placeholder name together with the trusted destination path, for example link "label hidden -- untrusted" → /sales/order/view/161, and the link stays clickable. Any other control whose accessible name comes from masked content is shown as [untrusted name hidden].
  • Disabled controls. At each step, if provenance records a third-party edit to a first-party control and the control is currently disabled, Guarded Quarantine removes the disabling state from the live DOM and recaptures the observation.

Guarded Quarantine changes the live DOM only to remove a disabling state; the value and label rules change only what the agent observes. Actions then run on the live DOM, so observation and execution stay consistent: a link whose label was hidden still goes to the destination path the agent was shown, and actions on masked content remain blocked (Appendix D.4).

We treat values and controls differently because the risks differ. Restoring an old value would present possibly stale data to the agent as fact. Re-enabling a control only lets the agent attempt a first-party action. A disabled state in the page is not an access-control mechanism, and a well-built application still checks permissions on the server when the action is submitted.

The rule for controls has one limitation: it can re-enable a control that the first party legitimately disabled, if a third-party script also edited that control. We cannot rule this out in general. Because ProvDOM records which attribute each actor set (Figure 4), a stricter rule could undo only the disabling edits made by the third party itself.

Q3 / W3. Denominators and secure completion.

Table 2 conditions each method’s attack metrics on its own benign successes, and we agree that the different denominators complicate direct comparisons. Value manipulation applies to the seven retrieval tasks in Table 4 and label manipulation to the five navigation tasks (Appendices F.4 and F.5), while hijacking and availability manipulation apply to all 12. With six runs each, this gives 72, 42, 30, and 72 attacked runs. Each entry below gives the number of attacked runs whose paired benign run succeeded:

MethodHijackEvidence: valueEvidence: labelAvailability
No Defense61/7233/4228/3061/72
UCM37/7226/4211/3037/72
Annotate57/7230/4227/3057/72
Drop56/7230/4226/3056/72
Quarantine58/7229/4229/3058/72
Guarded Quarantine57/7228/4229/3057/72
GQ + Relevance54/7233/4221/3054/72

To compare methods on the same runs, we report secure completion: the fraction of all 216 attacked runs, pooled across the four attacks, in which the agent completes the task and the attack fails. This is the joint metric you suggest, and it counts every run, whether or not its paired benign run succeeded. ASR and TSR below are computed on the same 216 runs, and the confidence intervals are 95% bootstrap intervals that resample tasks.

MethodSecure completion (95% CI)ASRTSR
No Defense6.5% [2.3, 12.0]89.8%14.8%
UCM2.8% [0.0, 8.3]77.8%6.0%
Annotate5.1% [0.5, 11.6]90.3%12.0%
Drop19.4% [10.2, 29.2]39.4%19.4%
Quarantine22.7% [13.0, 33.8]42.1%22.7%
Guarded Quarantine56.9% [38.4, 74.5]8.8%56.9%
GQ + Relevance56.0% [37.5, 73.6]9.7%56.9%

As you note, ASR and TSR are not complementary. Without a defense, the agent completes 14.8% of attacked runs but only 6.5% securely; under label manipulation, for example, ASR and TSR are 96.4% and 60.7% (Table 2). Under Guarded Quarantine, every completed run is also secure. Its interval does not overlap with that of any other method except GQ + Relevance, so its improvement in secure completion over those methods holds under task-level uncertainty.

Reviewer_785hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Browser states are represented as Document-Object Model (DOM), which are rendered to show the users. The authors observe that there is no provenance to DOM states, therefore it is impossible to distinguish changes from authorized parties and unauthorized parties. This paper proposed to add provenance to browser DOM, recording the history of DOM updates and the origin of the updates. The provenance logs are then used to calculate a score to expose unauthorized updates.

Critical strengths

This paper is based on a powerful observation: there are important information missing from browser state that are needed to detect malicious behavior from third-party or injected JavaScript behaviors. I believe this is the right direction to solve the problem. In addition, the proposed solution will be a new foundation to a range of web security problems. The ProvDOM technique can be used to directly address problems much wider in scope than browser agents and "in-page state forgery" attacks.

Critical weaknesses

Although the paper's technical direction is promising, the technical depth are not enough, either on the web security aspect or the AI aspect. The core approach belongs to web security, where the authors need to analyze the design space and argue why the exact provenance taken are at the right balance, solving the problem while keeping the browser usable. The full log of DOM updates can be huge. Simply using PageGraph from Brave will not be enough when more fine-grained provenance is needed from DOM updates.

Web is known for its lack of granularity in access control. I agree with the authors that we need to add fine-grained origins, but Definition 1 is still not clear for what type of attacks it helps to expose. The revised origin should be discussed with the background of potential threats, e.g., showing the difference of code from inline script, external resources, third-party components, etc. Once an external code is integrated into the DOM's origin, it may need more tracking than what the browser can offer to identify whether it is from an actor a'. Intuitively the revised origin works, but it is very challenging to correctly track it in the browser engine.

The in-page state forgery is not a separate class of attacks. It is the effect of executing third-party code. Your solution can actually solve the broader range of problems without necessarily narrowing the scope on this specific subset of problems.

In the technical approach of the main paper, how the provenance data is used is not discussed, for example, the analysis method or AI analysis techniques. It is expected to be an important technical part of the solution.

Questions for authors

The authors may consider to focus on the DOM provenance problem and make this paper into a web security paper, as your idea will make a clear impact for a bigger problem space.

If the authors prefers to continue focusing on AI solution on DOM provenance, the paper's Section 3 is not so important. It is better to be replaced with the technical AI approach as the main body of the technical solutions.

5 comments

Thank you for recognizing the broader value of ProvDOM and its potential to serve as a foundation for other web security problems. All changes in the revised PDF are highlighted in blue.

Scope.

Our central argument is that securing web agents requires co-design of ML and web security. Existing agent defenses predominantly analyze the final page representation through injection detection, task-alignment checks, or page structure (Section 1, Table 1). Our work shows why agent security also needs the browser layer: a third-party script can modify first-party content in place without leaving evidence of the responsible actor in the final DOM. ProvDOM addresses this information gap by recovering execution provenance from the browser and making it available to agent-facing enforcement policies.

We agree that execution provenance has applications beyond browser agents. We focus on agents because they provide both a direct consumer of this information and a concrete setting in which provenance can inform observations and permitted actions. Our contribution is therefore not a replacement for general web security mechanisms, but the integration of browser-observed provenance with agent security policies and task-aware relevance selection.

Design space and security–performance trade-offs.

We agree that the design space deserves explicit analysis. ProvDOM makes three design choices, each discussed in Sections 2.1 and 6, which we summarize here.

The first choice is what relationship to record. At one end, existing agent defenses infer trust from the final page state, for example from page structure (Table 1). This is cheap but cannot see an in-place edit, which is why UCM leaves attack success between 69.2% and 100% in our evaluation (Section 4). At the other end, Information Flow Control (IFC) tracks which data influenced each value. It is the most complete option, but full IF instrumentation of a browser engine is impractical. ProvDOM records execution provenance, which code directly performed each DOM operation (Section 2.1). This captures the case our attacks exercise, a third-party script using its own DOM authority to change first-party state, and the information is already present in browser execution traces.

The second choice is whom to attribute an operation to. We use the resolved origin of the acting code (Definition 1) and an exact-origin trust rule (Section 2.3). Finer distinctions within an origin, such as editorial content versus user comments or AI-generated summaries, cannot be established from browser observation alone. They require publishers to declare and authenticate authoring roles, which we discuss in Section 6.

The third choice is what to record for each node: the creating origin, the ordered list of editing origins, and the type of each operation, such as a text replacement, attribute update, insertion, or removal (Section 2.1). We do not yet propagate provenance through indirect effects, such as a third-party CSS change on an ancestor, or recover first-party nodes that a third party removed entirely (Section 6).

On keeping the browser usable, Chromium’s AdTracker shows that execution attribution is useful enough to ship, but human-browsing latency budgets force it to limit coverage (Section 2.1). An agent is not held to that budget. In our prototype, recording adds 8.7 s per step and 147 MB of memory per page (Section 4). On log size, we record each page in a fresh browser profile (Appendix D.1), and ProvDOM reduces the recording to the per-node labels above and discards the full graph after extraction.

On whether PageGraph is fine-grained enough: PageGraph records each individual DOM operation and the script that performed it, which is the granularity at which DOM authority is exercised. Finer provenance, meaning which data influenced a value, would require information-flow instrumentation, an option we considered but rejected: browser-wide information-flow tracking would incur substantial runtime and resource costs, require extensive changes to the browser engine, and offer uncertain benefits for detecting the direct third-party edits we study (Appendix D.1).

We made these performance and representation trade-offs more explicit in Section 2.1 and Appendix D.1.

Definition 1 and tracking it in the browser engine.

Definition 1 maps every actor that can modify the document to an origin, and each case addresses a different threat:

  • The document parser, and inline scripts in the document’s own network response, resolve to the document’s origin. This is the publisher’s own content.
  • A script fetched over the network resolves to the origin it was fetched from. This exposes a third-party script that edits first-party content directly, as in the action hijacking of Figure 1. It also applies when one script inserts another external script: the inserted script resolves to its own source.
  • Code or markup supplied by another actor resolves to that actor’s origin. This covers inline scripts created by another script, code run through eval, and markup a script writes into the page, for example through innerHTML. Such code has no URL of its own, so a naive rule would attribute it to the document. Definition 1 attributes it to the actor that supplied it, so a third-party script cannot make its edits look first-party by injecting inline code.

We appreciate and agree with the reviewer that correctly (i) tracking and attributing DOM modifications to the responsible scripts, and (ii) tracking the ancestry of script units, is a difficult task, and not something any browser currently does in full. ProvDOM builds on PageGraph, which extensively modifies and integrates with Chromium, and which has been developed, maintained, and used by domain experts at a prominent browser vendor, to fill this gap in the browser space, and to allow security and privacy researchers to correctly attribute fine-grained actions and edits to specific scripts. PageGraph records which script performed each DOM operation and how each script entered the page. The second part is what makes the third case of Definition 1 work: every script node has an incoming edge from the actor that added it, so code without its own URL can be traced to the actor that supplied it. We have also tested and validated PageGraph ourselves, to ensure that it correctly records, tracks, and attributes DOM modifications in the way ProvDOM requires.

For example, consider the script below, run on the HTML fragment that follows. The script runs (1), creates an image element (2), and sets its src attribute (3), which causes the image (4) to be fetched from the network (5). It then inserts the image into the page (6) and removes the existing “loading...” paragraph (7).

const sectionElm = document.getElementsByTagName("section")[0];
const pElm = document.getElementById("loading-text");

const imgElm = document.createElement("img");
imgElm.src = "https://site.example/sample.png";

sectionElm.appendChild(imgElm);
pElm.parentNode.removeChild(pElm);
<section>
    <p id="loading-text">loading...</p>
</section>

PageGraph records the following, as a directed multigraph:

  1. A “script” node, with an incoming edge from the actor that added the script to the page, such as the HTML parser or another script.

  2. An “HTML element” node for the image, with an incoming “create” edge from the script node. The node records the element’s Blink-assigned ID and tag name (img).

  3. An “add attribute” edge from the script node to the image node, recording the attribute name (src) and value (the image URL).

  4. A “resource” node for the image being fetched, recording its URL.

  5. A “request” edge from the image node to the resource node, recording metadata such as HTTP headers, and a “response” edge back from the resource node, recording the hash of the response and relevant headers.

  6. An “insert node” edge from the script node to the image node, recording, among other values, the Blink IDs of the image’s new parent and previous sibling.

  7. A “remove node” edge from the script node to the <p id="loading-text"> element.

Every DOM operation in steps 2, 3, 6, and 7 points back to the script node from step 1, so each change is attributed to the script that made it, and step 1 links that script to whoever added it to the page.

How ProvDOM uses provenance.

The main paper covers this in three places. Section 2.3 defines the trust rule and the policies that act on provenance, Section 2.2 describes the component we train, a task-conditioned relevance scorer, and Section 4 evaluates both (Tables 2 and 3, Figure 2).

As described in Section 2.3, all our policies share one exact-origin trust rule. A node is trusted only if (i) it was created by the top-level document’s origin and (ii) every later edit was attributed to that same origin; any other node is untrusted. In Figure 1, a third-party script rewrites the destination of a “Reserve Now!” link but leaves its text unchanged, so ProvDOM labels the link created_by=https://seaside-hotel.com, edited_by=https://attack.example, and the link becomes untrusted. Section 2.3 then defines five policies on top of this rule. They use the same provenance labels and differ only in how they act on them:

  • Drop removes untrusted content from the agent’s observation and blocks actions on it.
  • Annotate shows untrusted content to the agent with its provenance labels and leaves it readable and actionable, delegating the decision to the model.
  • Quarantine masks untrusted content with a placeholder, blocks actions on it, and lets the agent retrieve task-required values through a restricted reader. The reader is a separate LLM that is instructed to ignore any instructions in the data, and its answer is coerced to the requested type (Appendix F.3).
  • Guarded Quarantine adds the provenance-aware integrity enforcement defined in Section 2.3, which uses the modification history to handle third-party edits to first-party content. It replaces tampered labels with safe accessible names and restores first-party controls a third party disabled. If a third party modified a value, such as a price or shipping method, the reader does not return it.
  • Guarded Quarantine + Relevance Filtering also removes task-irrelevant content before applying Guarded Quarantine.

Enforcement is applied to the DOM and then projected onto the accessibility tree the agent observes (Appendix D.4). The learned part is our relevance scorer, ChunkRank, a cross-encoder that jointly encodes the user’s task, recent actions, and a serialized HTML chunk, trained with graded relevance labels and ranking losses (Section 2.2, Appendix D.3.2). Under relevance filtering, it decides what content the agent sees, never what it trusts. We also corrected the abstract, which described the relevance scorer as detecting malicious content.

We keep the trust decision itself deterministic for a reason. Provenance is recorded by the browser rather than inferred, so the decision needs no learned model, and a fixed rule cannot be argued out of its decision by page content, which the attacker controls. Our results show the difference directly. Under Annotate, the model sees correct labels but still acts on manipulated content. Moving from Quarantine to Guarded Quarantine adds a policy rule that withholds values a third party edited instead of letting the reader return them, which cuts value-manipulation ASR from 34.5% to 0.0%.

Reviewer_1P7xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a modification to browsers which captures provenance information for information on the page. The paper shows that this provenance information allows for defense against in-place modifications of a page made by third party scripts. The paper evaluates their approach by collecting the suggested information on two browser agent benchmarks.

Critical strengths

The problem the paper tackles is very well motivated and not solved by approaches from prior work.

The paper's approach to solving the problem is reasonable and novel. Several papers mentioned in the related work suggest additions to model context to improve various browser agent security problems, but this is the first I am aware of that has an appraoch to handle malicious third party scripts.

Critical weaknesses

The paper should more thoroughly document the impact on benign users. It is unclear from the paper if the benign tasks tested against include legitimate third party scripts. One failure mode here would be models generally being suspicious about third parties. The paper also does not document the memory/latency costs of the approach. This should be done both for the browser provenance collection itself, as well as for the increase in token cost.

Fundamentally, I would expect that a smart model should be able to figure out "relevance" itself. I worry that the tasks are too simple/the models tested too weak to demonstrate limitations with the relevance component.

Questions for authors

Can the authors provide information about the other impacts to benign users?

3 comments

Thank you for your thoughtful review and for recognizing the novelty of our problem and approach. We address your concerns about benign functionality, runtime overhead, and relevance filtering below. All changes in the revised PDF are highlighted in blue.

Handling legitimate third-party scripts.

We agree that third-party content is sometimes legitimate, and some tasks depend on it. ProvDOM records which origin directly created or modified a node, not whether the modification was malicious. Provenance therefore provides an attribution signal, while determining which actors to trust remains a separate policy decision.

Our evaluated policies use a conservative exact-origin trust boundary (Section 2.3). Quarantine masks untrusted content, blocks actions on it, and allows the agent to retrieve task-required information through a restricted reader that returns typed values. Guarded Quarantine additionally withholds values modified by other origins to prevent the reader from returning manipulated information. Our benign evaluation achieves 79.2% task success under Guarded Quarantine, compared with 84.7% without defense (Table 2). These experiments do not specifically assess workflows that depend on legitimate third-party modifications, but we designed provenance labeling with this case in mind. When a third-party script modifies a first-party node, ProvDOM keeps the node's first-party creator and also records the third-party origin that edited it. The labels therefore remain correct when a legitimate third party modifies first-party content.

Because ProvDOM records the resolved origin responsible for each direct DOM modification, the trust policy can be made more selective without changing the provenance mechanism. Curated service metadata, such as Disconnect's Services list, could help identify and categorize third-party services. For example, a publisher could explicitly authorize a payment or content-delivery provider to update specific fields. A policy could then permit retrieval of values modified by that authorized provider, which Guarded Quarantine currently withholds, while continuing to quarantine modifications from unauthorized origins.

We would not grant unrestricted trust to listed services, however. Service lists can be incomplete, and even established services or their infrastructure can be abused. For example, malicious advertisements distributed through established advertising networks affected visitors to major websites, including the New York Times and BBC, in 2016 [1]. Authorization should therefore be scoped to specific origins and permitted operations rather than treating all modifications from a known service as trustworthy. This is a possible policy extension, not a mechanism implemented or evaluated in our current system. We added this extension and its limitations to Sections 2.3 and 6.

[1] Alex Hern. “Major sites including New York Times and BBC hit by ‘ransomware’ malvertising.” The Guardian, March 16, 2016. https://www.theguardian.com/technology/2016/mar/16/major-sites-new-york-times-bbc-ransomware-malvertising

Documenting the memory and latency cost.

ProvDOM affects cost in three places: provenance recording, relevance scoring, and the agent’s input tokens. Provenance recording adds some overhead, and it is what enables the security gains in Table 2. Relevance filtering, in turn, reduces how much the agent has to read.

Provenance recording. ProvDOM builds on PageGraph, which records script executions and DOM operations as the page runs. For each operation, we identify the responsible actor, resolve it to an origin, and attach that origin to the affected node (Section 2.1, Appendix D.1). In our prototype this adds 8.7 s per step and 147 MB of memory per page. We kept this cost down by design. ProvDOM records only which code performed each DOM operation, not which values influenced it, so it avoids the overhead of full information-flow tracking (Section 2.1). The remaining overhead fits the agent setting. Chromium’s AdTracker shows that execution attribution is useful enough to ship, but human-browsing latency budgets force it to limit coverage (Section 2.1). An agent is the direct consumer of provenance and is not held to the same budget, so it can absorb richer instrumentation.

Relevance scoring. Our chunk-level design makes finding the relevant parts of a page much cheaper than the element-level baseline. MindAct scores every element separately, while our scorer, ChunkRank, a 307M-parameter cross-encoder (Appendix D.3.2), scores structure-preserving HTML chunks of up to 1,024 tokens (Section 2.2). As we report in Section 4, this reduces the average ranking workload from about 2,700 elements to 100 chunks, or about 27 times fewer relevance decisions. It also cuts average scoring latency from 2,722 ms to 839 ms, a 3.2× speedup. ChunkRank is also far more accurate: on the Mind2Web test split, it raises MRR from 25.2 to 92.1 over MindAct (Table 5). Scoring also saves cost downstream, since filtered regions never reach the agent model.

Input token cost. Under Drop, Quarantine, and Guarded Quarantine, provenance adds essentially no tokens to the agent’s input. ProvDOM attaches provenance to every node (Section 2), but in these policies the labels are read by the policy, not by the model, and untrusted regions are removed or replaced with short placeholders (Appendix F.1). In our runs, input tokens change by less than 0.3%. Relevance filtering reduces the input further: before the page is shown to the agent, we remove the 25% of chunks that the scorer ranks as least relevant to the task (Appendix D.3.3), which cuts input tokens by another 3.6%. Only Annotate shows the labels to the agent (Appendix F.2), which increases its input by 3.6%. We also clarified in Section 2 that both annotations are applied before the agent’s observation is built, and corrected the introduction, which overstated how much they add to the agent’s input.

We documented the overhead and measurement methodology in Section 4.

Relevance filtering with stronger models.

Even if a strong model can identify relevant content on its own, it has to read the whole page to do so. That spends input tokens on content unrelated to the task and exposes the model to regions that can distract or mislead it. Our filter runs as a preprocessing step, so only the selected content enters the model’s context and counts toward its input (Sections 2.2 and 4, Appendix D.3.3).

Leaving security-relevant judgments to the model is also unreliable in our results: under Annotate, the agent is told which content a third party modified, yet it still follows the hijacked link, with an attack success rate of 96.5% (Table 2). A stronger model does not remove the need for provenance either.

Tasks.

Following prior work [2, 3], we select tasks that the agent can complete without attacks, so that attack results are measured against a successful benign baseline. For the retrieval tasks, we screened all 192 WebArena-Verified Shopping tasks for cases where the agent must navigate to a page and report a single value shown there, so that value manipulation succeeds exactly when the agent reports the value we inserted. We excluded:

  • 111 tasks graded on completing actions such as purchases or reaching a page, since replacing a displayed value would not change the graded outcome.
  • 34 tasks where the agent must compute the answer, since different changes to the inputs could produce different final answers.
  • 22 tasks answerable before any navigation, because the answer appears on the starting product page (11) or in the My Orders list (11).
  • 7 tasks with no answer text to replace, including tasks whose expected answer is null.
  • 2 tasks (146 and 147) whose answers require units that the instruction does not ask for.

Because some tasks fall under more than one exclusion, this left 27 candidates. We ran each three times without attacks or defenses and kept those completed at least once, generally keeping two tasks per template (Appendix E.1), which gives the seven retrieval tasks in Table 4. On the final 12 tasks, the undefended agent reaches 84.7% benign task success (Table 2).

[2] Wu et al. Dissecting Adversarial Robustness of Multimodal LM Agents. ICLR 2025.

[3] Zhang et al. Attacking Vision-Language Computer Agents via Pop-ups. ACL 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.