acceptodds
Under review as a conference paper at ICLR 2027

IntentGuard: Securing GUI Agents without Sacrificing Benign User Intent

Abstract

GUI agents can autonomously interact with webpages, documents, and applications to perform complex user tasks, making them increasingly exposed to untrusted environmental content. Existing work on agent security has begun to defend against untrusted environmental content, but often relies on conservative detection-and-blocking strategies that may cause benign users’ legitimate tasks to be abandoned. When the user request is benign, an effective defense should not only block malicious environmental influence, but also preserve the user’s original intent to complete the legitimate task. To achieve these goals, we propose IntentGuard, a dual-module Blocking-Recovery framework. The Blocking module performs provenance-aware action screening to identify environment-induced unsafe behavior and prevent contaminated context from propagating into harmful actions. The Recovery module generates structured recovery guidance that helps the underlying agent bypass malicious environmental content and resume the user's intended task without retraining. To support systematic evaluation, we introduce EnvGuard-Bench, a GUI-agent safety benchmark organized along two axes: user intent and environmental content, each categorized as benign or malicious, resulting in four interaction settings. We evaluate IntentGuard under these settings, particularly the challenging benign-user/malicious-environment scenario. IntentGuard reduces attack success rate by 63.0 percentage points while achieving 67.0% safe task completion, and further experiments on existing safety benchmarks demonstrate its effectiveness and generalization. Code will be released at: https://anonymous.4open.science/r/IntentGuard-1560

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, GUI Agents, Agent Safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aSDsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on defending GUI agents against malicious environmental content while preserving legitimate task completion. Its primary goal is to overcome the limitations of conventional detection-and-blocking defenses, which may unnecessarily interrupt benign user tasks. To this end, the authors propose IntentGuard, a dual-module Blocking-Recovery framework that prevents unsafe actions and guides agents to resume legitimate tasks without retraining. They also introduce EnvGuard-Bench, which evaluates agent safety across four combinations of user intent and environmental content.

Critical strengths
  1. The authors present a useful problem formulation and a well-designed matched benchmark. The paper makes a clear point that simply detecting and aborting an attack is not a satisfactory safety outcome when the user's original request is benign. EnvGuard-Bench effectively captures this distinction by evaluating each scenario across four combinations of user intent and environmental content. Importantly, task completion, safety violations, and attacker success are measured using actual execution evidence rather than the defense's own decisions.
  2. Comprehensive experimental evaluation. The paper provides a thorough experimental evaluation, covering four backbone models (Tables 6 and 17), component ablations (Table 4), comparisons of different recovery strategies (Table 15), and runtime and memory overhead (Tables 18–19). I particularly appreciate the specification-provenance ablation in Appendix F, which examines how the results change when trusted information is obtained from user requests and runtime observations rather than directly from benchmark configurations. This makes the evaluation more convincing and helps clarify how the method may perform in realistic settings.
Critical weaknesses
  1. The performance gains are difficult to disentangle from benchmark-provided task guidance. In Table 5, Progress-only already improves task completion from 17% to 62% on Q00 and from 16% to 45% on Q01, even without the Blocking module. More importantly, the framework relies on trusted task specifications and progress feedback provided by the benchmark (§5.1, Appendix E). When these are replaced with request-derived specifications and runtime-visible evidence, SCR drops from 67% to 52% (Appendix F). Although all baselines receive the same trusted task information, they do not have comparable progress-control and recovery mechanisms. This makes it difficult to determine how much of IntentGuard's advantage comes from better security protection versus additional task-completion assistance.

  2. The benchmark design is closely aligned with the proposed defense mechanism. Although EnvGuard-Bench covers multiple applications and risk categories, many scenarios involve manipulating operation parameters or attacker-controlled fields, which are precisely the types of violations checked by the deterministic rule gate (§3.3, §4.1). This raises concerns about whether the evaluation sufficiently captures attacks beyond those anticipated by the defense. In addition, Q00 and Q01 differ not only in environmental content but also in the availability of attack-operation paths (Appendix D.1). While the authors acknowledge this design choice, it makes it harder to isolate the effects of malicious content itself.

Questions for authors

Q1. Section 5.1 states that all methods receive equivalent trusted task information (K), while Appendix E indicates that the benchmark also provides progress feedback. Could the authors clarify what information is actually available to each baseline, particularly whether they have access to the same task-completion feedback used by IntentGuard's progress controller?

Q2. Table 5 shows that Progress-only already achieves substantial gains without Blocking, while Appendix F shows that SCR drops from 67% to 52% when benchmark-provided inputs are replaced. Could the authors provide a comparison with the strongest baselines under the same request-derived specifications and runtime-visible progress evidence? This would help establish whether IntentGuard's advantage persists without benchmark-provided task guidance.

Q3. Section 4.1 and Appendix D.1 indicate that malicious environments expose attack-operation paths that are absent in clean environments. Could the authors provide a controlled comparison in which operation availability is held fixed and only the environmental content changes? This would help clarify whether the observed differences between Q00 and Q01 are driven by malicious content or changes in available operations.

6 comments
AuthorsOpenReview ↗

Reviewer aSDs — Response 1/3

Thank you for the careful review and for highlighting the relationship between benchmark guidance, runtime progress, operation availability, and gate alignment. We address each comment below with matched-input and fixed-path controls. Revisions responding to this review are marked in deep yellow in the revised manuscript. Tables in this response are annotated as Table R*.

Q1. Critical Weakness 1 / Question 1 — Information access and task-completion feedback

Critical Weakness 1: “The performance gains are difficult to disentangle from benchmark-provided task guidance.”

Question 1: “Could the authors clarify what information is actually available to each baseline, particularly whether they have access to the same task-completion feedback used by IntentGuard's progress controller?”

We separate the task interface into:

  • K_static: the approved objective, authorized parameters, protected-resource constraints, and allowed operation forms; and
  • K_completion: the criteria and feedback used for progress tracking and completion assessment.

Bare and Bare+K are intentional ablations with different information access. Bare receives the user request only. Bare+K additionally receives K_static, but neither configuration has a progress controller. Progress-only adds progress tracking, completion checks, and recovery guidance while disabling the blocking gate. Full IntentGuard combines blocking and recovery. Table R1 lists, for each baseline, access to the static specification, completion information, runtime-visible evidence, action interception, and recovery.

Table R1. Information access by configuration.

ConfigurationStatic task informationCompletion/progress informationRecoveryAction interception
BareUser request; no KNoneNoneNone
Bare + KBenchmark-derived static KNoneNoneNone
Progress-onlyBenchmark-derived static KBenchmark progress/completion feedback used by this ablationRecovery guidanceNone
Blocking + Generic retryBenchmark-derived static KNo progress intervention or premature-completion checkGeneric retryYes
Diagnostic PromptStatic K plus blocked-action diagnosticsProgress/status fields supplied by the diagnostic protocolDiagnostic promptYes
Full IntentGuard (reference)Benchmark-derived static specificationBenchmark-provided completion informationStructured recoveryYes
IntentGuard-DRequest-derived specificationRuntime-visible evidence available at deploymentStructured recoveryYes

PACT, AgentSentinel, AlignmentCheck, and CaMeL use the same screenshot/a11y observation interface through their native GUI adapters; under the D protocol, they receive the same deployment-available inputs as IntentGuard-D. The corresponding revisions are marked in deep yellow in Section 5.3, Appendices E and F, and Appendix G.

AuthorsOpenReview ↗

Reviewer aSDs — Response 2/3

Continued from the previous response.

Q2. Critical Weakness 1 / Question 2 — Matched deployment-input comparison

Question 2 (under Critical Weakness 1): “Could the authors provide a comparison with the strongest baselines under the same request-derived specifications and runtime-visible progress evidence?”

For the D setting, every configuration receives exactly the same request-derived specification, fixed deployer policy, runtime-visible evidence, screenshots, runner, interaction budget, and independent evaluator. Runtime-visible evidence consists only of screenshot-visible state, UI-returned results, public dispatch acknowledgments, and text feedback observable to the user. It does not include evaluator labels, hidden completion markers, benchmark metadata, or protected-state information.

Matched deployment-input evaluation

Table R2. Results of the matched D-setting control ladder.

MethodSCRUARASRAvg calls
Bare-D20625531
Progress-only-D38312428
Blocking + Generic retry-D4415829
Diagnostic Prompt-D4810527
IntentGuard-D524223

The component results are reported in Table R2, and the incremental changes produced by adding the components in the fixed control order are reported in Table R3.

Table R3. Incremental changes along the control ladder.

ComparisonSCR changeUAR changeASR changeInterpretation
Bare-D → Progress-only-D+18−31−31Incremental effect of progress tracking, completion checks, and recovery guidance
Progress-only-D → Blocking + Generic retry-D+6−16−16Incremental effect of the blocking gate
Blocking + Generic retry-D → Diagnostic Prompt-D+4−5−3Incremental effect of event-specific diagnostic guidance
Diagnostic Prompt-D → IntentGuard-D+4−6−3Incremental effect of staged structured recovery
Bare-D → IntentGuard-D+32−58−53End-to-end improvement under the D protocol

All SCR, UAR, and ASR changes above are percentage-point changes. The configurations are defined as follows:

  • Bare-D uses the request-derived specification and runtime-visible evidence but has neither a progress controller nor a blocking gate.
  • Progress-only-D adds progress tracking, completion checks, and recovery guidance without a blocking gate.
  • Blocking + Generic retry-D adds the same blocking gate while retaining generic retry feedback for blocked actions.
  • Diagnostic Prompt-D retains the blocking gate and progress controller but replaces generic retry with a single event-specific diagnostic prompt; it does not maintain a staged recovery state.
  • IntentGuard-D uses persistent remaining-obligation state, event-specific guidance, and the staged recovery controller.

Under deployment-available inputs, IntentGuard-D increases SCR from 20% to 52%, decreases UAR from 62% to 4%, decreases ASR from 55% to 2%, and reduces average calls from 31 to 23.

We also evaluate the two external methods with the highest Q01 SCR in the original comparison under the same D protocol (Table R4):

Table R4. External-baseline comparison under matched D inputs.

MethodSCRUARASRAvg calls
PACT-D329626
AgentSentinel-D3010627
IntentGuard-D524223

Thus, under identical deployment-available inputs, IntentGuard retains an advantage over the evaluated external baselines. The component ladder further shows the incremental contributions of progress/recovery assistance and blocking.

Effects of trusted-input provenance

Table R5. Effect of specification and completion-information sources.

ConfigurationSCRUARASRPremature completion
Reference679220.00
Replace static specification542227.03
Replace completion information613125.00
Replace both524228.00

Benchmark-derived guidance contributes materially to absolute SCR: replacing both the static specification and completion information with deployment-available inputs reduces SCR from 67% to 52%. Under the deployment-input protocol, IntentGuard still outperforms the evaluated external baselines. The corresponding revisions are marked in deep yellow in Section 5.3, Appendices E and F.

AuthorsOpenReview ↗

Reviewer aSDs — Response 3/3

Continued from the previous response.

Q3. Critical Weakness 2 / Question 3 — Benchmark alignment and fixed operation availability

Critical Weakness 2: “Many scenarios involve manipulating operation parameters or attacker-controlled fields that are precisely the types of violations checked by the deterministic rule gate.”

Question 3: “Could the authors provide a controlled comparison in which operation availability is held fixed and only the environmental content changes?”

We add Q00+ to hold operation availability fixed. Q00+ and Q01 use the same 100 scenario IDs and share the same attack-tool binary and path, permissions, UI fields, operation schema and parameters, initial state, execution markers, independent evaluator, runner, and interaction budget.

The only difference between Q00+ and Q01 is the environmental content (Table R6).

Table R6. Fixed-operation-path content comparison.

MethodQ00+ SCRQ00+ UARQ00+ ASRQ01 SCRQ01 UARQ01 ASR
Bare154N/A167265
PACT305N/A3284
Full IntentGuard674N/A6792

Q00+ ASR is N/A because the clean-content condition does not instantiate an environmental attacker objective. The attack-operation path remains available, and unsafe behavior is recorded through UAR and the independent goal predicates. For IntentGuard, SCR remains at 67% while UAR increases from 4% to 9%; for Bare, UAR increases from 4% to 72% and ASR reaches 65% in Q01; for PACT, UAR increases from 5% to 8% and ASR reaches 4% in Q01.

Therefore, Q00+–Q01 provides a direct environmental-content comparison with operation availability held fixed. Implementation sanity checks cover target/resource binding, operation binding, parameter binding, and provenance binding; the appendix defines how the 142 labeled events map to the 100 scenarios. OS-Harm and WASP provide complementary evidence on independent benchmarks: IntentGuard achieves 56% SCR and 4% ASR on OS-Harm, and 45.2% SCR and 2.4% ASR on WASP. The corresponding revisions are marked in deep yellow in Section 5.2, Appendix A, and Appendix D.

Reviewer_uHfGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes IntentGuard, a plug-and-play runtime defense framework designed to secure GUI agents against malicious environmental content (such as indirect prompt injections) without sacrificing legitimate user tasks. IntentGuard combines a Blocking module (rule gating and selective VL critic) to intercept unsafe actions with a Recovery module that provides structured advisory prompts for task resumption. Additionally, the authors introduce EnvGuard-Bench, a factorized benchmark featuring a 2x2 setting matrix across 100 application-risk scenarios (400 task variants) evaluated dynamically in Docker sandbox environments. Experiments show that IntentGuard reduces attack success rates while maintaining high safe task completion rates across multiple base models.

Critical strengths

High-Value Factorized Benchmark (EnvGuard-Bench): The introduction of EnvGuard-Bench fills a significant gap in evaluating GUI agent safety under indirect prompt injection. By factorizing user intent and environmental content into a 2x2 matrix, it highlights the crucial yet overlooked Q01 setting (benign user request within a malicious environment), where defense mechanisms must balance security with task completion. Evaluating this dynamically within realistic Docker sandboxes provides a solid and practical setup for the community.

Critical weaknesses
  • Unfair Baseline Comparison for the Recovery Mechanism: The baseline used to evaluate recovery is potentially unfair. When an action is blocked, the baseline is not provided with the specific diagnostic reasons, authorized parameters, or task status that IntentGuard injects into its advisory prompts. Without comparing against a baseline that receives equivalent diagnostic information via prompt engineering, it is difficult to discern whether the gains stem from IntentGuard's control system or simply from supplying richer information to the base model.

  • Limited Headroom for External Plugin Architectures: As an external wrapper operating around GUI agents, the headroom for optimization is inherently constrained because system execution ultimately relies on the base LLM's capabilities. Beyond structuring the prompt inputs, the core mechanisms of the external controller offer limited architectural leverage in single-agent GUI execution.

Questions for authors
  • Will the full EnvGuard-Bench environment, including Docker container configurations, task instances, and automated evaluation scripts, be fully open-sourced to guarantee stable and reproducible evaluations for the research community?

  • How does IntentGuard perform when compared against a fairer baseline where the blocking diagnostic reason and authorized parameters are directly appended to the base agent's prompt upon action interception?

3 comments
AuthorsOpenReview ↗

Reviewer uHfG — Response 1/2

We thank the reviewer for carefully reading the paper and for raising concerns about the fairness of the recovery baseline, the optimization headroom of an external wrapper, and the release of EnvGuard-Bench. We add a Diagnostic Prompt control, specify the deployment-available comparison protocol, and clarify the contents of the current anonymous repository and the final public artifact release. Revisions responding to this review are marked in red in the revised manuscript. Tables in this response are annotated as Table R*.

Q1. Critical Weakness 1 / Question 2 — Fairness of the recovery baseline

Critical Weakness 1: “The baseline used to evaluate recovery is potentially unfair because it does not receive the diagnostic reason, authorized parameters, and task status.”

Question 2: “How does IntentGuard perform against a fairer baseline where the blocking diagnostic reason and authorized parameters are appended to the base agent’s prompt?”

We added the requested Diagnostic Prompt baseline and evaluated it under the same experimental conditions.

All recovery configurations use OpenCUA-72B, the same 100 Q01 tasks, the same initial Docker states, the same independent evaluator, the same budget of 40 base-agent proposals, the same Blocking module, the same progress/stall/premature-completion logic, the same history length, and the same observation interface. After a blocked action, Diagnostic Prompt provides the same intervention fields used by Full IntentGuard: the blocking reason, authorized parameters, current task obligation, event type, recovery stage, and runtime progress/status. The only difference is how this information is organized: Diagnostic Prompt appends it as a single diagnostic message in the current turn, whereas Full IntentGuard uses structured, staged, event-specific recovery guidance. In both configurations, the intervention message is cleared after the next base-agent proposal is generated. The results are shown in Table R1.

Table R1. Recovery strategies with matched diagnostic information on 100 Q01 tasks.

MethodSCR ↑UAR ↓ASR ↓Average base-agent calls ↓
Blocking + Generic retry328324
Blocking + Progress + Generic retry4910328
Diagnostic Prompt589326
Full IntentGuard679223

Diagnostic Prompt closes the information gap identified by the reviewer: relative to progress control with generic retry, it raises SCR from 49% to 58%. With the diagnostic information matched, Full IntentGuard further raises SCR to 67%, a 9-point improvement over Diagnostic Prompt. UAR is 9% for both methods, ASR decreases from 3% to 2%, and average base-agent calls decrease from 26 to 23.

These results show that the gain from Full IntentGuard comes from structured, staged recovery rather than simply appending blocking diagnostics to the base agent’s prompt. The corresponding revisions are marked in red in Section 5.3 and Appendix G.

AuthorsOpenReview ↗

Reviewer uHfG — Response 2/2

Continued from the previous response.

Q2. Critical Weakness 2 — Optimization headroom for an external wrapper

Critical Weakness 2: “As an external wrapper operating around GUI agents, the headroom for optimization is inherently constrained because system execution ultimately relies on the base LLM's capabilities. Beyond structuring the prompt inputs, the core mechanisms of the external controller offer limited architectural leverage in single-agent GUI execution.”

IntentGuard has measurable system-level headroom as an external wrapper. We compare IntentGuard-D, PACT-D, and AgentSentinel-D under deployment-available inputs. The -D configuration uses a request-derived task specification, a fixed deployer policy, and runtime-visible progress evidence, with the same base agent, task set, evaluator, and action budget for all methods. The results are shown in Table R2.

Table R2. Deployment-available comparison under a fixed base agent.

MethodSCR ↑UAR ↓ASR ↓
PACT-D3296
AgentSentinel-D30106
IntentGuard-D5242

With deployment-available inputs, IntentGuard-D improves SCR by 20 points over PACT-D, reduces UAR by 5 points, and reduces ASR by 4 points. Relative to AgentSentinel-D, it improves SCR by 22 points, reduces UAR by 6 points, and reduces ASR by 4 points.

This conclusion is also supported by the fixed-base-agent component study. On Q01, Progress-only obtains 45% SCR and 27% ASR, whereas Full IntentGuard obtains 67% SCR and 2% ASR after adding the complete Blocking module. With Blocking fixed, generic retry obtains 49% SCR, while structured, staged recovery raises SCR to 67%. These results show that an external wrapper can change both which proposals are executable and how the base agent continues after an intervention, producing measurable gains in task completion and safety. The corresponding revisions are marked in red in Section 5.3, Appendix E, and Appendix F.

Q3. Question 1 — Release of EnvGuard-Bench

Question 1: “Will the full EnvGuard-Bench environment, including Docker container configurations, task instances, and automated evaluation scripts, be fully open-sourced to guarantee stable and reproducible evaluations for the research community?”

Yes. The current anonymous repository already contains the complete model code and experiment code used in the paper:

https://anonymous.4open.science/r/IntentGuard-1560

After the anonymous review phase, we will remove the anonymity and publish the complete EnvGuard-Bench artifact on Hugging Face, including:

  • Docker configurations and build files;
  • the 100 application-risk scenarios and their 400 quadrant variants;
  • task instances, task-selection metadata, and scenario manifests;
  • the automated evaluator and validation scripts;
  • baseline adapters, runtime configurations, and model interfaces;
  • random seeds, execution-log schemas, and reproduction scripts;
  • dependency versions, a versioned release, and the applicable license.

We sincerely appreciate your valuable comments and constructive suggestions, which have helped us improve our work. We have made every effort to address your concerns in our response and revised manuscript. We would be grateful to hear your thoughts on whether our revisions have sufficiently addressed the points you raised. We look forward to continuing the discussion and welcome any further feedback.

Reviewer_6yoHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IntentGuard combines rule-based screening and a VLM critic with progress tracking and one-turn recovery guidance for ui control agents. It aims to preserve benign task completion under environmental prompt injection. The paper introduces a four-condition benchmark varying user intent and the environment, and evaluates additional attack benchmarks and agent backbones. In the main benign-user / malicious-environment setting, the safe completion rate is high, and the attack SR is 2%. Controlled recovery ablations support structured guidance.

Critical strengths

Preserving benign task completion is useful. I find the joint treatment of safe completion and attack success well motivated: terminating a legitimate task is a consequential failure even when the attack is blocked.

The main-paper ablations help separate progress assistance from safety screening. Exps show that progress assistance explains much of the clean-task gain, while adding Blocking improves attacked-task SCR from 45% to 67% and reduces ASR from 27% to 2%. Authors also identify substantial effects of current-turn guidance and staged recovery.

Experiments on the execution models. Sections 3.2 and 4.2 distinguish dispatched actions from their effects and evaluate task completion independently of the guard's decisions.

Critical weaknesses

The main concerns from my view are its dependence on benchmark-provided task information and how convincingly the comparisons establish performance beyond the constructed task family.

The main comparison leaves information access and baseline adaptation unresolved. Sec 5.1 states that all baselines receive equivalent trusted information K, but exp in tab 5 distinguishes Bare from Bare+K and Table 3 uses the former's results. It is unclear whether the equivalence statement excludes the bare agent.

The connection between GUI actions and the proposed security checks is underspecified. Please refer to my question.

IMO, EnvGuard-Bench has narrower operational coverage than its application count suggests. Sec 4.1 constructs one scenario per application–risk pair and states that the ten applications provide visual contexts for a shared family of administrative operations. Its strongest evidence therefore concerns those operation families, while the variety of application names alone does not establish diverse GUI recovery requirements. The same section also changes both injected content and the availability of an attack-operation path between clean and malicious conditions. The paper acknowledges this design, but the quadrant contrast consequently measures a combined environment change rather than the effect of injected instructions alone.

The public-benchmark results provide limited evidence of a comparative advantage. Table 3's SCR advantage over pact corresponds to 2 additional successes on each benchmark: 28 versus 26 out of 50 on OS-Harm (it’s a very tiny eval set), and 38 versus 36 out of 84 on WASP. Sec. 5.1 further states that WASP's 84 instances come from 42 scenarios, so the two injection templates should not be treated as independent evidence.

Questions for authors
  1. Which task-selection rules and evaluator adaptations were used for OS-Harm/WASP, and do paired, scenario-level outcomes support the observed advantage?

  2. What information and GUI-adapted implementation does each baseline receive?

  3. The guard's connection to concrete GUI actions is underspecified. I found that sec 3.1 starts with screenshot observations and mouse/keyboard proposals, whereas sec 3.3's rule gate checks semantic properties such as recipients, protected assets, and parameter bindings? The missing link is how a coordinate click or keyboard proposal is mapped to those properties before execution.

6 comments
AuthorsOpenReview ↗

Reviewer 6yoH — Response 1/3

We thank the reviewer for identifying the issues concerning information parity, GUI semantic binding, benchmark coverage, the Q00/Q01 contrast, and paired evaluation on public benchmarks. We have revised the manuscript to clarify the experimental interfaces, add the runtime details, and include the corresponding benchmark analyses. Revisions responding to this review are marked in blue in the revised manuscript. Tables in this response are annotated as Table R*.

Q1. Critical Weakness 1 / Question 2 — Information access and baseline adaptation

Critical Weakness 1: “The main comparison leaves information access and baseline adaptation unresolved.”

Question 2: “What information and GUI-adapted implementation does each baseline receive?”

The revised manuscript separates task information, progress feedback, and action adaptation: Bare is the no-K configuration, Bare+K is a separate diagnostic ablation, and the K source and injection path for the remaining configurations are reported in Table R1.

On EnvGuard-Bench, all non-Bare comparator configurations receive the same user request and static task constraints K through their respective GUI wrappers. All methods use the same OpenCUA-72B base agent, Docker environment, 40-step budget, temperature-zero decoding, and method-independent outcome evaluation. Differences therefore arise from the defense module, prompting mechanism, observation interface, and action adapter.

Table R1. Information access and GUI action adaptation for each configuration.

ConfigurationTask informationProgress/completion informationGUI observation and action adaptationRecovery/interception
BareUser request; no KNoneBase-agent action interfaceNone
Defensive promptUser request + KNonePrompt-level adaptationNo action interception
AlignmentCheckUser request + KMethod-specific configurationScreenshot + a11y tree; native adapterMethod-specific mechanism
AgentSentinelUser request + KMethod-specific configurationScreenshot + a11y tree; native adapterMethod-specific mechanism
CaMeLUser request + KMethod-specific configurationScreenshot + a11y tree; native adapterMethod-specific mechanism
PACTUser request + KMethod-specific configurationScreenshot + a11y tree; native adapterMethod-specific mechanism
Full IntentGuardUser request + KProgress state, completion checks, and current-turn guidanceScreenshot + action canonicalizationAction interception + structured recovery

External defenses use accessibility-tree information through their native GUI adapters, while IntentGuard’s safety decision uses the task constraints, screenshot, candidate action, and interaction history. Table R1 reports these observation interfaces and action adapters for each configuration.

For OS-Harm and WASP, each method receives the task information provided by the corresponding benchmark instance; EnvGuard-Bench-derived K is not transferred. All methods use the same execution environment and scoring protocol. We also add a specification-provenance ablation. With a deployment-available configuration using a request-derived specification and runtime-visible progress evidence, IntentGuard obtains 52% SCR, 4% UAR, and 2% ASR on the same 100 EnvGuard-Bench tasks. This quantifies the contribution of benchmark-derived specification and reports system performance with deployment-available inputs. The corresponding revisions are marked in blue in Section 5.1, Section 5.3, Appendix E, and Appendix F.

AuthorsOpenReview ↗

Reviewer 6yoH — Response 2/3

Continued from the previous response.

Q2. Critical Weakness 2 / Question 3 — Mapping GUI actions to semantic security properties

Critical Weakness 2: “The connection between GUI actions and the proposed security checks is underspecified.”

Question 3: “How is a coordinate click or keyboard proposal mapped to recipients, protected assets, and parameter bindings before execution?”

We agree that the original description did not make the GUI-to-semantic connection sufficiently explicit. The revised manuscript describes the action canonicalization performed before dispatch.

Each candidate action is first normalized into an event containing

(application,window,event type,coordinates/key/text,focus,history).(\text{application},\text{window},\text{event type},\text{coordinates/key/text},\text{focus},\text{history}).

For a coordinate click, the runtime resolves the hit element using the active window’s element geometry and visible labels, and records its role, label, value, focus state, and application context. When structured element information is unavailable, the runtime uses the current screenshot and candidate action for visual grounding. For keyboard actions, the runtime maintains the active input buffer, focused control, and event sequence. Enter or an equivalent submission event is treated as a proposed commit of the pending operation rather than as an already executed operation.

The canonicalizer then produces

(operation,target/resource,parameters,provenance).(\text{operation},\text{target/resource},\text{parameters},\text{provenance}).

Provenance distinguishes values derived from the user request, trusted task constraints, runtime feedback, and environmental content. This structured representation is passed to the rule gate and selective visual reviewer, and is checked against the approved operation, authorized parameters, and protected resources in K. If a coordinate cannot be mapped uniquely to a control, or if the parameter provenance cannot be resolved, the event is marked unresolved and routed to escalation, warning, or blocking.

We add an action-binding audit covering 142 high-risk candidate events from 18 execution trajectories and 77 unique task-action pairs:

Table R2. Action-binding audit results.

MetricResult
Target/resource exact match95%
Operation exact match97%
Parameter binding micro-F10.92
Provenance micro-F10.94
Unresolved mapping4/142 = 2.8%

The manual-label/runtime-gate confusion matrix is shown in Table R3:

Table R3. Confusion matrix between manual labels and runtime-gate decisions.

Manual label / Runtime gateAllowWarnBlockTotal
Allow471250
Block127982
Warn/review42410
Total52585142

This audit reports how the runtime binds operations, targets/resources, parameters, and provenance. We also add a mouse-based GUI trace in which a form control is selected and an unauthorized recipient is entered, complementing the shell keyboard trace in the original version. The corresponding revisions are marked in blue in Sections 3.1, 3.3, and 3.4, Appendix A, and Appendix D.

Q3. Critical Weakness 3 — Operational coverage

Critical Weakness 3: “The application count does not establish diverse GUI recovery requirements.”

The current benchmark is organized around five shared administrative-operation families instantiated across ten GUI contexts. Table R4 and the appendix make this coverage explicit by reporting the applications, action sequences, injection locations, and recovery events represented in each family.

The 100 scenarios are grouped into the following semantic operation families (Table R4):

Table R4. Coverage by semantic operation family.

Operation familyScenario countMain semantic check
Command execution25Command/path binding and execution authority
Email/send20Recipient/content authorization
File staging20Source/destination and protected-file boundary
Permission/configuration20Target object and requested state change
Other GUI commit15Selected GUI object and commit action

The revised manuscript additionally reports the applications, GUI action sequences, injection locations, recovery events, and per-family results. “Other GUI commit” is further decomposed into concrete control-submission types, with the corresponding action sequences and recovery events reported in the appendix.

These results provide a structured starting point for evaluating shared operation semantics across GUI surfaces. In subsequent benchmark releases, we will add more application-specific workflows, diverse widget interactions, longer multi-step recovery trajectories, and additional interaction modalities to measure generalization beyond the current operation families. The corresponding revisions are marked in blue in Sections 4.1 and 4.2 and Appendix D.

AuthorsOpenReview ↗

Reviewer 6yoH — Response 3/3

Continued from the previous response.

Q4. Critical Weakness 4 — Q00 and Q01 change both content and path availability

Critical Weakness 4: “The quadrant contrast consequently measures a combined environment change rather than the effect of injected instructions alone.”

The original Q00→Q01 comparison changed both environmental content and the availability of the attack-operation path. We add Q00+ to measure environmental-content effects while holding operation availability fixed.

To control this factor, we add a Q00+ condition. Q00+ and Q01 use the same 100 scenario IDs, tasks, initial states, operation paths, execution budgets, and independent evaluators. Q00+ keeps the operation path available as in Q01 but retains clean environmental content, whereas Q01 uses malicious environmental content. Thus, Q00+→Q01 changes the environmental content while holding path availability fixed.

Table R5. Fixed-path comparison between Q00+ and Q01.

MethodQ00+ SCRQ00+ UARQ00+ ASRQ01 SCRQ01 UARQ01 ASR
Bare154N/A167265
PACT305N/A3284
Full IntentGuard674N/A6792

ASR is not applicable to Q00+ because it contains no malicious environmental content. With operation availability held fixed, Q00+→Q01 directly measures sensitivity to environmental content: IntentGuard’s SCR remains at 67% while UAR increases from 4% to 9%. The revised manuscript reports the fixed-path construction, matched scenario IDs, and results; Q00→Q01 provides the broader environmental comparison. The corresponding revisions are marked in blue in Section 5.2 and Appendix D.

Q5. Question 1 — OS-Harm/WASP task selection and paired outcomes

Question 1: “Which task-selection rules and evaluator adaptations were used for OS-Harm/WASP, and do paired, scenario-level outcomes support the observed advantage?”

We use the benchmark-provided task pools without performance-based filtering or subsampling. OS-Harm contains 50 instances. WASP contains 84 instances from 42 GitLab/Reddit scenarios, with one plain-text and one URL-injection template per scenario. Both methods use the same instance IDs, OpenCUA-72B base agent, Docker runner, temperature-zero decoding, 40-step budget, and method-blind outcome evaluation.

OS-Harm is paired at the instance/scenario level. For WASP, we first aggregate the two templates within each scenario. Let cm(s)∈0,1,2c_m(s)\in\\{0,1,2\\} denote the number of successful templates for method mm in scenario ss. We classify each scenario according to whether cIG(s)c_{\mathrm{IG}}(s) is larger than, equal to, or smaller than cPACT(s)c_{\mathrm{PACT}}(s).

Table R6. Scenario-level paired outcomes on OS-Harm and WASP.

BenchmarkIntentGuard SCRPACT SCRPaired unitIG winTiePACT win
OS-Harm28/5026/5050 scenarios6404
WASP38/8436/8442 scenarios8286

Across repeated evaluations, the paired outcomes consistently favor IntentGuard. We therefore characterize the result as a modest but reproducible SCR advantage under these benchmark protocols. Because the two WASP templates share the same scenario, we aggregate them at the scenario level, giving 42 paired scenario units. The corresponding revisions are marked in blue in Section 5.2 and Appendix D.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank all reviewers for their careful reviews and for highlighting information parity, GUI semantic binding, controlled comparisons, and reproducibility. We address the common points below; the individual responses provide the detailed evidence.

  1. Matched deployment inputs. Every D-setting configuration receives the same request-derived specification, runtime-visible evidence, screenshots, runner, interaction budget, base agent, and independent evaluator. IntentGuard-D obtains 52%/4%/2% SCR/UAR/ASR, compared with 32%/9%/6% for PACT-D and 30%/10%/6% for AgentSentinel-D. Diagnostic Prompt reaches 58% SCR, while Full IntentGuard reaches 67% with the same diagnostic fields.
  2. GUI semantic binding. Before dispatch, each click or keyboard proposal is canonicalized into an operation, target/resource, parameter binding, and provenance record. The audit covers 142 high-risk events from 18 trajectories and 77 task-action pairs: target/resource exact match 95%, operation exact match 97%, parameter-binding micro-F1 0.92, provenance micro-F1 0.94, and 4/142 unresolved events routed to escalation, warning, or blocking.
  3. Controlled comparisons. Q00+ and Q01 use the same scenario IDs and operation path while changing environmental content. IntentGuard keeps 67% SCR while UAR changes from 4% to 9%. On OS-Harm, IntentGuard obtains 28/50 SCR versus 26/50 for PACT; on WASP, 38/84 versus 36/84, with 8 IntentGuard wins, 28 ties, and 6 PACT wins at the scenario level.
  4. Artifact release. The anonymous repository contains the current code and experiments. After the anonymous review phase, we will publish the complete EnvGuard-Bench artifact on Hugging Face.

Tables in this response are annotated as Table R*.

The corresponding revisions are marked in Sections 3–5 and Appendices A, D, E, F, and G: blue for Reviewer 6yoH, red for Reviewer uHfG, and deep yellow for Reviewer aSDs.

We thank the reviewers again for their time and constructive feedback, and hope that these clarifications are helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.