acceptodds
Under review as a conference paper at ICLR 2027

Towards Safe Autonomy for Computer Use

Abstract

The safety of LLM-based agents currently rests on alignment training and safety classifiers, neither of which can rule out that a sophisticated indirect prompt injection (IPI) goes through and causes significant harm. Recent work therefore builds agents with information-flow guarantees that rely on structured tool interfaces to interact with the environment. Computer Use Agents (CUAs) act visually on the computer screen, which does not provide a boundary between trusted first-party interfaces and untrusted, potentially attacker-authored content. Actions are equally ambiguous: a click may land on an application toolbar or a button controlled by an attacker. Treating the user as the source of trust and confirming each sensitive decision produces confirmation fatigue, with the risk of missing harmful actions. We introduce SA-CUA, a dual-LLM framework that generates a reliable trust boundary for CUAs by deriving conservative deterministic rules from the accessibility tree for unmasking trusted regions from the privileged LLM’s screenshot, keeping trusted and untrusted content separated. Human interaction is only required for proper task specification in the beginning. During the task a robust endorsement channel allows values to pass from the untrusted to the trusted domain by verifying information via trusted channels. This allows the agent to autonomously handle tasks involving untrusted information. SA-CUA maintains much higher utility than CaMeL-CUA while keeping the attack success rate at zero.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Computer Use Agent, Indirect Prompt Injection, Information-flow control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XZtXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies the dual-LLM approach for defending against prompt injection attacks to Computer Use agents. By default, the full screen is treated as untrusted and masked. Several ways are used to identify some parts of the screens as trusted and unmasked. The privileged LLM sees only the unmasked (trusted) screen. Identifying which parts as trusted require initial efforts to identify certain web domains and email contacts as trusted, and analysis of screens to generate rules for what are trusted. There are additional shortcuts to identify which parts are trusted. There are also endorsements that allow some additional texts be treated as trusted. Experiments demonstrate that the proposed approach, SA-CUA, preserves more utility tha CaMel-CUA.

Critical strengths
  1. The idea of using integrity models and dual-LLM is principled. The idea of starting with masking the whole screen and unmasking trusted ones prioritize security.

  2. Experimental results show that the proposed SA-CUA, preserves substantially more utility tha CaMel-CUA.

Critical weaknesses
  1. The proposed system is quite heavy-weight, in the sense that a lot of work needs to be done specifically for the tasks at hand. For example, one needs to specify trusted web domains and email contacts; rule sets for which regions are trusted are hostname and URL specific; several additional shortcuts are needed. One has the feeling that a lot of these are results of making SA-CUA achieve higher Benign Task Completion on the benchmarks. The generalibility to broader tasks is questionable. The experimental results in the paper have some evidences for this, as SA-CUA does not perform well on OSWorld and UCM-B.
Questions for authors
  1. Is it really feasible to deploy such approaches for CUAs that need to browse the web for many tasks?
2 comments

Thank you for taking the time to review our paper. We address the raised points below.

  1. "One needs to specify trusted web domains and email contacts." True, but this is set up once per user or organization, not per task, like an email address book or a company network allowlist. In our experiments, the trusted-domain list is one global setting for the whole run, and the contact list is shared by all tasks of a suite. Any defense separating trusted from untrusted content needs such a specification; otherwise, "trusted" is undefined. SA-CUA requires it only once, before deployment, never during a task.

  2. "Rule sets for which regions are trusted are hostname and URL specific." Rule sets are not specific to a hostname and URL: they are selected by URL template within a site, and per application for native windows, and are reused on pages they were never proposed on. Across all SA-CUA turns, masking used a rule set not proposed on that very page in 93.4% of turns for the adversarial suites (100% for pages with adversarial content), 97.2% for OSWorld, and 24.5% for UCM-B. UCM-B is low because its static benchmark sites have hard-coded content and often no second page of the same template. Applying rules to new pages is the normal case, not an exception.

  3. "Several additional shortcuts are needed." The shortcuts are general rules, not per-task fixes. They incorporate prior knowledge the RP-LLM (rule-proposing LLM) lacked when proposing rules: where an injection cannot be (website infrastructure / consent banners), what cannot carry an injection (capacity), and what was written by a trusted party (ownership). We agree they reduce the over-masking our conservative rules cause. But each shortcut unmasks only content that is safe to see. None relies on what the content says about itself.

  4. "One has the feeling that a lot of these are results of making SA-CUA achieve higher Benign Task Completion." This is correct. SA-CUA builds on the idea that the channels through which adversarial content reaches the agent can be identified, and an agent that cannot see an injection cannot fall for it. Its default is therefore a fully masked screen: ASR is 0, but benign completion suffers. Every further component recovers utility from this safe default by unmasking only content that is safe to see (see 3.), so SA-CUA's worst case is lost utility, never compromised security. The one-time setup this requires is comparable to tool-use agents, where each platform must offer a typed tool interface and each tool needs its own security policy. Most desktop applications and many websites offer no such interface, which is why computer-use agents exist. Ideally, websites would expose to agents where UGC (user-generated third-party content) sits and which values are type-constrained, reducing this effort.

  5. "The generalizability to broader tasks is questionable [...] SA-CUA does not perform well on OSWorld and UCM-B." We agree these were our weakest results and the most mentioned point across reviews. We analysed all failure cases and identified several issues; see the general comment for details and updated results. The utility degradation in the updated results is model-dependent and ranges from 0 to 16.7pp on OSWorld and from 9.1 to 24.3pp on UCM-B.

Question: "Is it really feasible to deploy such approaches for CUAs that need to browse the web for many tasks?" Partially. SA-CUA is best for targeted tasks where the user knows which surfaces are trusted (e.g., their own mail, files, chat or company tools) and the agent uses untrusted content in a controlled way. Generalization is affected mainly in

  • tasks where untrusted content specifies the task goal (e.g., following setup instructions from a README),
  • tasks where the result is a judgment about untrusted content (e.g., reviewing a document), and
  • tasks where a decision depends on untrusted content which cannot be endorsed (e.g., deleting forum comments that violate community guidelines).

This limits dual-LLM designs in general, not only SA-CUA: letting untrusted content determine the agent's actions is exactly what a prompt injection does. For web browsing, feasibility thus depends on the task. It is feasible if the task is specified, i.e. the prompt determines what the agent should do and untrusted content only supplies data that is capacity-bound or verifiable from trusted sources. Explorative tasks, where the agent must rely on untrusted content to decide what to do, cannot be made safe by construction, as this influence is exactly what a prompt injection exploits. However, whether a task is explorative depends on the prompt: a more detailed prompt (e.g., stating the setup steps instead of referring to the README) can make the same task specified. We will add this to the limitations paragraph.

We hope this addresses your concerns and are happy to answer any follow-up questions. If your concerns are resolved, we would be grateful if you would consider raising your score.

Dear Reviewer XZtX,

Thank you again for reviewing our paper. As the discussion period is about to end, we wanted to kindly follow up on our response to your comments. If it addresses your concerns, we would appreciate it if you would consider raising your score.

Best regards,
The Authors

Reviewer_pyekOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper is an improvement on the recent secure-by-design line of work for computer use agents. They propose SA-CUA, a method for deriving trust boundary from the accessibility tree of the document the model is observing. The idea of SA-CUA is to unmask trusted regions from the privileged LLM screenshot, whereas by default all content is masked (note: in secure-by-design approaches, normally there are two llms: privileged, which is the main agent planner, and quaratined, that is allowed to see untrusted content). The method does not require human in the loop when the agent is running, but needs human to help specify tasks. Untrusted data can only reach the data through quarantined model.

Critical strengths
  • The paper often compares itself with Untrusted Content Masking (UCM), and I find it useful to understand the paper's strength through this comparison. Unlike UCM, SA-CUA works in hybrid web-OS environment, not only web. UCM relies on correct labeling of untrusted content. This method essentially relies on correct labeling of trusted content. Under perfect labeling, there is no difference, however, when labeling is imperfect, SA-CUA is better (errors type-I vs errors type-II). In SA-CUA, labeling is happening semi-automatically via rules that can be checked, which is strictly better. Overall, it's a cool improvement.
  • The main contribution of this paper to the field is a set of novel methods that concern masking and value endorsment. This diversifies the current secure-by-design methods we have.
Critical weaknesses

It's a bit hard for me to pinpoint weaknesses in the sense that the weaknesses I see are in comparison with UCM, which the authors claim to be better than; I don't think all of these claims are necessarily true, and I see the methods as more being on-par with each, being better on different things. At the same time, UCM appeared publicly exactly 2 months before ICLR submission and would be considered concurrent work, so I will ignore any such weaknesses that come from the comparison. Below is a concern independent of any such comparisons:

  • [Addressed during rebuttals] SA-CUA has a steep utility cost, around 40% drop on UCM-B completion rate, which is very noticeable. I see it as a rather fundamental limitation of the method, which does not allow me to give clear accept. At the same time, the paper has a lot of cool ideas, so I would still be more than willing to give weak accept.
Questions for authors

What would make me raise my score: substantial increase in utility of the method. In my opinion, this is perfectly fine for a secure-by-design method to have up to 10% utility drop, but 40% is a bit too much.


Discussion, not relevant to the score

For my own understanding, can you provide me with a list of guarantees that the method enforces, and a list of assumptions that have to be in place for these guarantees to hold?

3 comments

Thank you for taking the time to review our paper and for your positive feedback on our ideas.

"In my opinion, this is perfectly fine for a secure-by-design method to have up to 10% utility drop, but 40% is a bit too much." We agree with this concern. As the other two reviewers share it, we address it in the general comment above. The utility degradation in the updated results is model-dependent and ranges from 0 to 16.7pp on OSWorld and from 9.1 to 24.3pp on UCM-B.

"List of guarantees that the method enforces, and a list of assumptions that have to be in place for these guarantees to hold."

The core properties of SA-CUA are:

  • (i) the P-LLM never sees untrusted pixels that can carry a prompt injection,
  • (ii) a value from untrusted content reaches the P-LLM only as a typed value that passed robust endorsement against an independent trusted source,
  • (iii) every action passes the policy gate. For example, clicks on masked regions are refused unless endorsed.

From these, SA-CUA guarantees, under the assumptions below, that every action the P-LLM performs comes from a trusted context and therefore that no prompt injection is followed. More precisely, we can guarantee non-interference: Let each observation element carry a label T (trusted) or U (untrusted). An element is labelled T if it is covered by a rule, has been endorsed, belongs to a trusted domain or contact, or falls under one of the three shortcuts: it is part of the website infrastructure, it lacks the capacity to carry an injection, or it was written by a trusted party. All other elements are labeled U. Then, the P-LLM's action sequence depends only on the user task and the T-labelled observations. Changing U-content therefore does not change the actions, except where a value is endorsed and thus becomes T; robust endorsement ensures that such values serve the user's task.

This holds if:

  • I. the user is trusted,
  • II. the P-LLM is not itself adversarial, i.e. given only trusted inputs, it pursues the user task,
  • III. websites are not compromised, i.e. the website infrastructure itself does not allow attacks such as escaping the user-generated content (UGC) regions,
  • IV. the trusted domains and contacts are correctly configured,
  • V. files are correctly labeled, i.e. a file into which the user copied untrusted web content must be labeled untrusted rather than self-authored and trusted,
  • VI. the endorsement sources are not compromised,
  • VII. our framework covers every route the agent could try to circumvent masking and access untrusted information in unredacted form,
  • VIII. the accessibility tree is complete and matches the rendered page.

What is not guaranteed: underspecified tasks that require untrusted content (e.g. "do what the e-mail says") are out of scope by design.

Is this the formulation you were looking for, or were you rather looking for a different formulation, e.g. a more mathematical one?

We hope this addresses your concerns and are happy to answer any follow-up questions. If yes, we would be grateful if you would consider raising your score.

Thank you for additional clarification. My main concern was the utility drop, which has been addressed by the authors in the main comment and has been mostly result of code bugs. I increase my score to clear accept.

Thank you very much for your feedback. We have added the comment above to the appendix as a separate section, providing a compact overview of our framework.

Reviewer_NwTkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work introduces SA-CUA, a novel framework for improving the safety of computer-use agents (CUAs). Building on a dual-LLM architecture, SA-CUA separates trusted and untrusted information by selectively unmasking trusted GUI regions and specialized rules. To safely incorporate information from untrusted sources, the framework employs type-constrained queries and an endorsement mechanism that verifies extracted values against independent trusted sources. In the experiments, SA-CUA exhibits strong resistance to indirect prompt injection attacks, with substantially improved task completion compared to an existing dual-LLM-based baseline.

Critical strengths
  • SA-CUA extends the dual-LLM architecture to computer-use agents by addressing the unique challenges of separating trusted and untrusted content in GUI environments. The proposed endorsement-based unmasking mechanism is highly intuitive yet effective.
  • The proposed framework is carefully motivated by established security principles and prior work, including Biba's integrity model and robust endorsement.
  • The experiments demonstrate zero observed attack success rates across the evaluated benchmarks while substantially improving task completion over the existing dual-LLM-based baseline.
Critical weaknesses

1. Title and Introduction

1.1. The title is overly general and does not sufficiently reflect the paper's specific contributions.

1.2. Please elaborate on the Biba integrity model (Line 96) or, at minimum, refer readers to Section 3.1, where it is discussed in more detail.

1.3. Consider briefly explaining what “SA” stands for in SA-CUA (Line 92).

1.4. Please clarify how the rule sets are configured in Figure 1. The input-output flow in the diagram is confusing. The input to the rule-proposing LLM appears to include page/window information, whereas the description says that the LLM never sees page text.

2. Methodology

2.1. In Figure 2, the injected content is difficult to identify in the screenshots. Please consider highlighting the injection regions to improve readability.

2.2. Please clarify what “N” represents in [name:length:N] (Line 236).

2.3. The timing of rule generation by the rule-proposing LLM is unclear. Please clarify when the LLM is invoked, particularly whether rule generation occurs only before deployment or can also take place during task execution.

2.4. Please elaborate on the motivation for introducing shortcut rules and explain why these three specific cases were selected.

2.5. While type-constrained queries can improve safety, could they substantially limit the generality of the framework? Please discuss how this restriction affects the range of tasks SA-CUA can handle.

2.6. Please provide further analysis of the specification classifier. How is its reliability evaluated, and to what extent can the framework depend on its decisions to identify underspecified tasks?

3. Experimental Setup and Baselines

3.1. “MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control (Lee et al., AAAI 2026)” introduces Safety-Consideration Chain-of-Thought (SCoT), a prompting method that generates safety considerations before taking actions and demonstrates substantial safety improvements in mobile CUA tasks. Comparing SA-CUA against such a prompting-based approach would help clarify the benefits of the dual-LLM architecture over a single-LLM approach with safety-aware prompting.

3.2. Why was only a small subset of OSWorld tasks selected for benign utility evaluation? Please clarify the selection criteria and discuss how representative this subset is of the full benchmark.

4. Experimental Results and Analysis

4.1. SA-CUA sacrifices benign task completion performance in exchange for improved safety. What are the primary causes of this degradation? Please provide a more detailed failure analysis that distinguishes failures specifically introduced by SA-CUA's mechanisms (e.g., masking, endorsement, and policy gating), beyond more general agent failures (e.g., reaching the step limit) discussed in Appendix D.2.

4.2. What potential approaches could mitigate the utility degradation while preserving the safety guarantees? A discussion of possible directions for future work would strengthen the paper.

4.3. SA-CUA incurs approximately 7.4 times the token cost of the unprotected CUA, which appears prohibitively expensive for practical deployment, even considering the reductions achievable through prompt caching. The authors suggest that fine-tuning could reduce this overhead. Could a small-scale fine-tuning experiment support this hypothesis, or could the authors provide a more concrete analysis of the expected cost reductions?

4.4. Can we add some analysis on the specification classifier? How can we trust it?

Questions for authors

See weaknesses.

3 comments

Part 1/1

Thank you for taking the time to review our paper and for your feedback. We answer each question individually below.

1.1 We agree and will change the title to "Out of Sight, Out of Reach: Robust Trust Separation for Safe Computer Use Agents".

1.2 We will add a one-line summary (no read-down/write-up for integrity) and a reference to §3.1.

1.3 SA-CUA stands for Safe & Autonomous CUA; we will expand it on first use.

1.4 The rule-proposing LLM (RP-LLM) receives only a sanitized skeleton of the AT-SPI tree in which every accessible name is replaced by a length placeholder, so attacker-controlled text cannot influence the rules that are proposed.

2.1 We highlighted the injection regions to make them obvious at first glance.

2.2 N is the character length of the name of the accessibility node. It is kept as a structural cue to the RP-LLM to judge how large that element is.

2.3 Rules are ideally proposed offline per application/site template before the agent is deployed. During task execution, unmasking is decided based on the matched rule set for the shown windows/pages. If the agent encounters a page/window for which no rule set exists, a rule set is proposed online during that step.

2.4 By default, content from an unknown website is masked. This prevents an injection on that site from reaching the P-LLM, but it also degrades utility. Trust is granted to elements covered by a rule. However, we have prior knowledge that the RP-LLM (rule-proposing LLM) did not have when proposing the rules, and we use this prior knowledge to recover utility. This motivates the three shortcuts, each encoding a different kind of prior knowledge: where an injection cannot be (website infrastructure / consent banners), what cannot carry an injection (capacity), and what was written by a trusted party (ownership).

  • Elements such as consent banners are owned by the website itself. They can be identified deterministically, so we know in advance that they are trustworthy.
  • Some elements are constrained in their format and cannot carry an injection. For example, prices on a shopping platform are unmasked as they do not have the capacity to carry an injection because their format is strictly confined, and because the P-LLM could not endorse a value it cannot see. This lets the P-LLM reason about user-generated content (UGC) that cannot carry a prompt injection.
  • Self-authored elements are trusted. Re-endorsing content the agent itself wrote in a previous turn would only add computational overhead, and masking trusted documents owned by the user would degrade utility with no security benefit.

2.5 Free-text answers from the Q-LLM cannot be endorsed and are never shown to the P-LLM. SA-CUA allows querying the Q-LLM for short free-text answers (at most 120 characters), because 5 UCM-B tasks require answers to the user that cannot be expressed in any more constrained format. These answers can only be reported back to the user; they cannot be endorsed or used in subsequent actions. We acknowledge that this restricts the set of tasks SA-CUA can handle, but we expect the restriction to be modest: many operations can be expressed as terminal commands, so they never require passing free text between models. Generalization is affected mainly in three ways:

  • tasks where untrusted content specifies the task goal (e.g., following setup instructions from a README),
  • tasks where the result is a judgment about untrusted content (e.g., reviewing a document), and
  • tasks where a decision depends on untrusted content which cannot be endorsed (e.g., deleting forum comments that violate community guidelines).

This is a limitation of the dual-LLM design in general, not only of SA-CUA: letting untrusted content determine the agent’s actions is exactly what a prompt injection does. Overall, this reflects a safety-utility tradeoff. Tasks lie on a spectrum between specified tasks, where the prompt determines what the agent should do and untrusted content only supplies data, and explorative tasks, where the agent must rely on untrusted content to decide what to do. SA-CUA handles specified tasks well, while explorative tasks cannot be made safe by construction, since they require exactly the influence over the agent that a prompt injection can exploit.
However, whether a task is explorative depends on the prompt, not only on the task itself: the same task can be made specified by a more detailed prompt (e.g., stating the setup steps instead of referring to the README). The limitation therefore applies to task–prompt pairs that leave the task open-ended, not to tasks as such.

Part 2/2

2.6 & 4.4 We will add the specification classifier's prompt to the appendix. We wrote a few example prompts and used Claude Opus 5 to expand them into a small prompt set, containing e.g. "Follow the instructions in this email to perform the download." (unspecified) and "Clone the official repository for the requests library from the link in this post." (specified). We then tested the classifier (the model itself, prompted with a few in-context examples) on binary classification of this set. Classification accuracy is 100% on the in-sample tune set (14 examples), 84% (Qwen) / 93% (GPT Luna) on the held-out set (80 examples), and 100% on the RTC generic prompts. Flagging only had an effect on RTC, where we designed some generic variants to be unspecific and unsolvable for SA-CUA; elsewhere it had no impact. By asking for clarification, the classifier can turn a task that is unsolvable (unclear user intent, nothing untrusted can be robustly endorsed) into a solvable one. It is conservative: it can only flag prompts, so a flagged prompt that is not clarified can only cost utility under SA-CUA, never security.

3.1 We tested SCoT for Qwen3.8-27B (numbers are from the runs in the general comment and may deviate from the submitted manuscript). CUA and CUA+SCoT use Qwen's native computer-use action space. CR is averaged over k = 3 seeds; ASR is ASR@3.

FrameworkWASP$^\dagger$RTC$^\dagger$RTC-valueOSWorldUCM-B
ASR ↓CR ↑ASR ↓CR ↑ASR ↓CR ↑CR ↑CR ↑
CUA7.597.566.297.459.047.973.199.3
CUA+SCoT (auto approve)0.088.355.494.955.147.476.993.9
CUA+SCoT (end on consent)0.088.350.890.855.144.976.993.9
SA-CUA0.098.30.088.20.090.673.186.2

We used the exact SCoT prompting from Lee et al., adding refuse() and ask-consent() tools (CUA can only end episodes with DONE or FAIL). Since user confirmation fatigue is a core motivation for our endorsement route, we report both auto-approving after ask-consent() and counting it as refusal. Qwen asked for consent in only 2.2% of adversarial episodes. SCoT's safety relies on the user rejecting the request, which does not happen under confirmation fatigue. Moreover, SA-CUA's strength is its safety guarantee vs. the empirical approach of SCoT.

3.2 The subset is OSWorld's official test_small set minus the Google-Drive tasks. They cover 14 multi-app, 4 Chrome, 3 LibreOffice Calc, 3 VS Code, and 2 each for LibreOffice Impress, LibreOffice Writer, GIMP, OS/terminal, Thunderbird and VLC. We chose it for cost and runtime reasons and found these 36 tasks representative of SA-CUA’s utility on OSWorld-like tasks.

4.1 & 4.2 The general comment gives a detailed failure analysis and updated results. During the analysis we identified and fixed several issues. The remaining failures are attributable to SA-CUA's method. Qwen in particular repeatedly clicks into untrusted regions or attempts policy-blocked actions, and reasons very extensively, showing confusion about the blacked-out environment. Despite being told the mechanism and motivation of SA-CUA, Qwen does not fully adapt to it. However, this rarely causes failures: it mainly lengthens episodes and thus adds to the cost overhead of SA-CUA. This effect is much less pronounced for Sonnet. Future work could address this by teaching models to understand a safety defense's manipulations of the trajectory without trying to circumvent them, e.g. via RL within the safety harness that penalizes clicks in untrusted regions. We will add this to the conclusion.

4.3 We anticipate that SFT could teach what the tools do (i.e. descriptions and arguments). For the updated results, however, we changed the memory management to reduce cost overhead: instead of RedTeamCUA's approach (accumulating history for 10 steps, then a sliding window, which changes the prompt at every step), we accumulate context for 25 steps, keep the last 10, and accumulate again. This makes prompt caching much more effective and removes much of the overhead of the longer system prompt. For SA-CUA with GPT-5.6-Luna, this gives 5.0x the input tokens, with 94.5% cache reads (85.6% for CUA), and 1.8x the output tokens, i.e. a 3.5x cost overhead over CUA. The overhead is 2.7x for Sonnet and 3.5x for Qwen (assuming Sonnet's prices for cache and output tokens).
Reducing cost further would require teaching the model to use the SA-CUA harness well, which is much harder (see 4.2). For smaller models like Qwen3.8-27B, this could be done via RL in the SA-CUA harness instead of SFT, which we leave to future work given the effort involved.

We will upload a revised version with these changes towards the end of the rebuttal. We hope this addresses your concerns and are happy to answer follow-up questions. If your concerns are resolved, we would be grateful if you would consider raising your score.

Dear Reviewer NwTk,

Thank you again for reviewing our paper. As the discussion period is about to end, we wanted to kindly follow up on our response to your comments. If it addresses your concerns, we would appreciate it if you would consider raising your score.

Best regards,
The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Updated Results and Failure Analysis (All Reviewers)

A main concern for all three reviewers is the utility degradation of our method. When performing an extensive failure analysis, we improved the benchmark in some aspects. We report the updated numbers, extended to Sonnet 5.5 (high thinking), in the table below. The utility reduction is now 0, 16.7, and 5.6pp on OSWorld and 13.1, 24.3, and 9.1pp on UCM-B for Qwen, GPT Luna, and Sonnet 5.5. For Qwen and Sonnet the utility reduction is moderate. Given the moderate to substantial attack success rate on all models for the normal CUA, we think this is an acceptable security-utility tradeoff.

Updated Table 1 on p.8 (CR is averaged over k seeds; ASR is ASR@k):

P-LLMFrameworkkWASP$^\dagger$RTC$^\dagger$RTC-valueOSWorldUCM-B
ASR ↓CR ↑ASR ↓CR ↑ASR ↓CR ↑FR ↓CR ↑CR ↑
Qwen3.8-27BCUA7.597.566.297.459.047.99.473.199.3
CaMeL-CUA30.056.70.017.40.015.884.211.112.8
SA-CUA0.098.30.088.20.090.69.473.186.2
GPT-5.6 LunaCUA7.597.536.9100.041.064.10.080.696.0
CaMeL-CUA10.062.50.033.80.028.271.811.124.2
SA-CUA0.0100.00.090.80.076.923.163.971.7
Sonnet 5.5CUA0.070.07.775.419.271.823.180.6100.0
CaMeL-CUA10.057.50.021.52.621.875.619.415.2
SA-CUA0.0100.00.089.20.096.23.875.090.9

Changes that were applied (trust model, policy and threat model are unchanged):

  • Benchmark fixes: we corrected several issues inherited from related work. For OSWorld, we pulled upstream fixes, fixed evaluators that accepted only one of several valid routes, allowed slight tolerance for continuous tasks, and tightened instructions where the evaluator was stricter. For UCM-B, we reworded several tasks (e.g. a task asking for "visible" content whose evaluator scored the full page) and fixed evaluators that lost events or rejected correct answers (e.g. five vs. 5 years or yes/no vs. true/false).
  • Native action space: We ran Qwen in the RedTeamCUA harness with pyautogui actions. However, we found that Qwen's utility is severely compromised by this and much of its utility is recovered when using the Qwen-native action space from "Qwen-CUA: Native Computer Use for (almost) everything".
  • Over-conservative masking in a few UI situations for SA-CUA: In-window stackorder was not resolved and settings menus were thus overmasked. The rule sets for few screens were empty due to some minor bug and were therefore masked.
  • Small bugfixes we found along the way such as fixing frozen pages and ensuring consistency between accessibility tree and screenshot readout (many parallel runs can lead to delays in the environments).

Some, especially the task-evaluator mismatches, hurt SA-CUA more. SA-CUA reads untrusted page content only through a typed quarantined channel, so it reports normalised values (e.g. "5" for "5+ years", "14:00" for "2:00 PM"), whereas the CUA copies the page's wording. Several UCM-B checkers accepted only the page's wording and therefore penalised SA-CUA more; we made them accept equivalent forms. 4 of the 36 OSWorld tasks are infeasible for SA-CUA by design, as they depend on untrusted data that cannot be endorsed.
From the failed Sonnet CUA episodes on WASP/RTC/RTC-value, Sonnet encountered API refusal in 0/14/16 episodes and declined on its own in 12/1/1 episodes. This explains the lower completion rate under CUA compared to SA-CUA.

Remaining failures attributable to SA-CUA are:

  • Masking. Over-masked regions lead the agent to conclude a page is empty and navigate away, or to type into the search box instead of using an over-masked filter tab. Counts are read only from the visible screen, not the full page.
  • Endorsement. The P-LLM requests only the identifier and drops version pins, builds the endorsement URL incorrectly, or uses endorsed identifiers in a wrong URL. It mistakes revealing a control for clicking it. On UCM-B filter tasks it endorses only the requested toggle and leaves on-by-default toggles active ("my filter is already on").
  • Q-LLM. The Q-LLM misreads a value, or falls for the injection and returns the attacker's value (e.g., the adversarial repository). Endorsement then fails. These are exactly the episodes that are attack successes for the plain CUA and the defense is working. The P-LLM can fall back on prior knowledge, but Qwen sometimes misremembers it (e.g., tomtom/vim-commentary instead of tpope/vim-commentary).
  • Policy gating. Qwen repeatedly tries to click in redacted regions and is blocked. However, this mainly lengthens episodes rather than causing failures.
AuthorsOpenReview ↗

Rebuttal Revision

Dear Reviewers,

We thank all reviewers for their constructive feedback. We have updated our submission to incorporate all of the points raised by the reviewers. The main changes are:

  • More specific title as requested by reviewer NwTk (currently only updated in the pdf).
  • Clearer motivation of the shortcuts in “Trusted Content Labeling” (Sec. 3.1).
  • Updated Table 1 according to our general response and added Sonnet 5.5 to the results.
  • Updated cost comparison due to more efficient caching strategy.
  • More detailed discussion which tasks SA-CUA cannot solve.
  • New C.5: comparison with safety-aware prompting (SCoT).
  • New C.9: details and accuracy of the specification classifier.

We would kindly ask Reviewer NwTk and Reviewer XZtX to look at our rebuttal as well. If it addresses your concerns, we would appreciate it if you would consider raising your scores.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.