- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper applies the dual-LLM approach for defending against prompt injection attacks to Computer Use agents. By default, the full screen is treated as untrusted and masked. Several ways are used to identify some parts of the screens as trusted and unmasked. The privileged LLM sees only the unmasked (trusted) screen. Identifying which parts as trusted require initial efforts to identify certain web domains and email contacts as trusted, and analysis of screens to generate rules for what are trusted. There are additional shortcuts to identify which parts are trusted. There are also endorsements that allow some additional texts be treated as trusted. Experiments demonstrate that the proposed approach, SA-CUA, preserves more utility tha CaMel-CUA.
Critical strengths
-
The idea of using integrity models and dual-LLM is principled. The idea of starting with masking the whole screen and unmasking trusted ones prioritize security.
-
Experimental results show that the proposed SA-CUA, preserves substantially more utility tha CaMel-CUA.
Critical weaknesses
- The proposed system is quite heavy-weight, in the sense that a lot of work needs to be done specifically for the tasks at hand. For example, one needs to specify trusted web domains and email contacts; rule sets for which regions are trusted are hostname and URL specific; several additional shortcuts are needed. One has the feeling that a lot of these are results of making SA-CUA achieve higher Benign Task Completion on the benchmarks. The generalibility to broader tasks is questionable. The experimental results in the paper have some evidences for this, as SA-CUA does not perform well on OSWorld and UCM-B.
Questions for authors
- Is it really feasible to deploy such approaches for CUAs that need to browse the web for many tasks?
2 comments
Dear Reviewer XZtX,
Thank you again for reviewing our paper. As the discussion period is about to end, we wanted to kindly follow up on our response to your comments. If it addresses your concerns, we would appreciate it if you would consider raising your score.
Best regards,
The Authors
Thank you for taking the time to review our paper. We address the raised points below.
"One needs to specify trusted web domains and email contacts." True, but this is set up once per user or organization, not per task, like an email address book or a company network allowlist. In our experiments, the trusted-domain list is one global setting for the whole run, and the contact list is shared by all tasks of a suite. Any defense separating trusted from untrusted content needs such a specification; otherwise, "trusted" is undefined. SA-CUA requires it only once, before deployment, never during a task.
"Rule sets for which regions are trusted are hostname and URL specific." Rule sets are not specific to a hostname and URL: they are selected by URL template within a site, and per application for native windows, and are reused on pages they were never proposed on. Across all SA-CUA turns, masking used a rule set not proposed on that very page in 93.4% of turns for the adversarial suites (100% for pages with adversarial content), 97.2% for OSWorld, and 24.5% for UCM-B. UCM-B is low because its static benchmark sites have hard-coded content and often no second page of the same template. Applying rules to new pages is the normal case, not an exception.
"Several additional shortcuts are needed." The shortcuts are general rules, not per-task fixes. They incorporate prior knowledge the RP-LLM (rule-proposing LLM) lacked when proposing rules: where an injection cannot be (website infrastructure / consent banners), what cannot carry an injection (capacity), and what was written by a trusted party (ownership). We agree they reduce the over-masking our conservative rules cause. But each shortcut unmasks only content that is safe to see. None relies on what the content says about itself.
"One has the feeling that a lot of these are results of making SA-CUA achieve higher Benign Task Completion." This is correct. SA-CUA builds on the idea that the channels through which adversarial content reaches the agent can be identified, and an agent that cannot see an injection cannot fall for it. Its default is therefore a fully masked screen: ASR is 0, but benign completion suffers. Every further component recovers utility from this safe default by unmasking only content that is safe to see (see 3.), so SA-CUA's worst case is lost utility, never compromised security. The one-time setup this requires is comparable to tool-use agents, where each platform must offer a typed tool interface and each tool needs its own security policy. Most desktop applications and many websites offer no such interface, which is why computer-use agents exist. Ideally, websites would expose to agents where UGC (user-generated third-party content) sits and which values are type-constrained, reducing this effort.
"The generalizability to broader tasks is questionable [...] SA-CUA does not perform well on OSWorld and UCM-B." We agree these were our weakest results and the most mentioned point across reviews. We analysed all failure cases and identified several issues; see the general comment for details and updated results. The utility degradation in the updated results is model-dependent and ranges from 0 to 16.7pp on OSWorld and from 9.1 to 24.3pp on UCM-B.
Question: "Is it really feasible to deploy such approaches for CUAs that need to browse the web for many tasks?" Partially. SA-CUA is best for targeted tasks where the user knows which surfaces are trusted (e.g., their own mail, files, chat or company tools) and the agent uses untrusted content in a controlled way. Generalization is affected mainly in
This limits dual-LLM designs in general, not only SA-CUA: letting untrusted content determine the agent's actions is exactly what a prompt injection does. For web browsing, feasibility thus depends on the task. It is feasible if the task is specified, i.e. the prompt determines what the agent should do and untrusted content only supplies data that is capacity-bound or verifiable from trusted sources. Explorative tasks, where the agent must rely on untrusted content to decide what to do, cannot be made safe by construction, as this influence is exactly what a prompt injection exploits. However, whether a task is explorative depends on the prompt: a more detailed prompt (e.g., stating the setup steps instead of referring to the README) can make the same task specified. We will add this to the limitations paragraph.
We hope this addresses your concerns and are happy to answer any follow-up questions. If your concerns are resolved, we would be grateful if you would consider raising your score.