- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on defending GUI agents against malicious environmental content while preserving legitimate task completion. Its primary goal is to overcome the limitations of conventional detection-and-blocking defenses, which may unnecessarily interrupt benign user tasks. To this end, the authors propose IntentGuard, a dual-module Blocking-Recovery framework that prevents unsafe actions and guides agents to resume legitimate tasks without retraining. They also introduce EnvGuard-Bench, which evaluates agent safety across four combinations of user intent and environmental content.
Critical strengths
- The authors present a useful problem formulation and a well-designed matched benchmark. The paper makes a clear point that simply detecting and aborting an attack is not a satisfactory safety outcome when the user's original request is benign. EnvGuard-Bench effectively captures this distinction by evaluating each scenario across four combinations of user intent and environmental content. Importantly, task completion, safety violations, and attacker success are measured using actual execution evidence rather than the defense's own decisions.
- Comprehensive experimental evaluation. The paper provides a thorough experimental evaluation, covering four backbone models (Tables 6 and 17), component ablations (Table 4), comparisons of different recovery strategies (Table 15), and runtime and memory overhead (Tables 18–19). I particularly appreciate the specification-provenance ablation in Appendix F, which examines how the results change when trusted information is obtained from user requests and runtime observations rather than directly from benchmark configurations. This makes the evaluation more convincing and helps clarify how the method may perform in realistic settings.
Critical weaknesses
-
The performance gains are difficult to disentangle from benchmark-provided task guidance. In Table 5, Progress-only already improves task completion from 17% to 62% on Q00 and from 16% to 45% on Q01, even without the Blocking module. More importantly, the framework relies on trusted task specifications and progress feedback provided by the benchmark (§5.1, Appendix E). When these are replaced with request-derived specifications and runtime-visible evidence, SCR drops from 67% to 52% (Appendix F). Although all baselines receive the same trusted task information, they do not have comparable progress-control and recovery mechanisms. This makes it difficult to determine how much of IntentGuard's advantage comes from better security protection versus additional task-completion assistance.
-
The benchmark design is closely aligned with the proposed defense mechanism. Although EnvGuard-Bench covers multiple applications and risk categories, many scenarios involve manipulating operation parameters or attacker-controlled fields, which are precisely the types of violations checked by the deterministic rule gate (§3.3, §4.1). This raises concerns about whether the evaluation sufficiently captures attacks beyond those anticipated by the defense. In addition, Q00 and Q01 differ not only in environmental content but also in the availability of attack-operation paths (Appendix D.1). While the authors acknowledge this design choice, it makes it harder to isolate the effects of malicious content itself.
Questions for authors
Q1. Section 5.1 states that all methods receive equivalent trusted task information (K), while Appendix E indicates that the benchmark also provides progress feedback. Could the authors clarify what information is actually available to each baseline, particularly whether they have access to the same task-completion feedback used by IntentGuard's progress controller?
Q2. Table 5 shows that Progress-only already achieves substantial gains without Blocking, while Appendix F shows that SCR drops from 67% to 52% when benchmark-provided inputs are replaced. Could the authors provide a comparison with the strongest baselines under the same request-derived specifications and runtime-visible progress evidence? This would help establish whether IntentGuard's advantage persists without benchmark-provided task guidance.
Q3. Section 4.1 and Appendix D.1 indicate that malicious environments expose attack-operation paths that are absent in clean environments. Could the authors provide a controlled comparison in which operation availability is held fixed and only the environmental content changes? This would help clarify whether the observed differences between Q00 and Q01 are driven by malicious content or changes in available operations.
6 comments
Reviewer aSDs — Response 2/3
Continued from the previous response.
Q2. Critical Weakness 1 / Question 2 — Matched deployment-input comparison
Question 2 (under Critical Weakness 1): “Could the authors provide a comparison with the strongest baselines under the same request-derived specifications and runtime-visible progress evidence?”
For the D setting, every configuration receives exactly the same request-derived specification, fixed deployer policy, runtime-visible evidence, screenshots, runner, interaction budget, and independent evaluator. Runtime-visible evidence consists only of screenshot-visible state, UI-returned results, public dispatch acknowledgments, and text feedback observable to the user. It does not include evaluator labels, hidden completion markers, benchmark metadata, or protected-state information.
Matched deployment-input evaluation
Table R2. Results of the matched D-setting control ladder.
| Method | SCR | UAR | ASR | Avg calls |
|---|---|---|---|---|
| Bare-D | 20 | 62 | 55 | 31 |
| Progress-only-D | 38 | 31 | 24 | 28 |
| Blocking + Generic retry-D | 44 | 15 | 8 | 29 |
| Diagnostic Prompt-D | 48 | 10 | 5 | 27 |
| IntentGuard-D | 52 | 4 | 2 | 23 |
The component results are reported in Table R2, and the incremental changes produced by adding the components in the fixed control order are reported in Table R3.
Table R3. Incremental changes along the control ladder.
| Comparison | SCR change | UAR change | ASR change | Interpretation |
|---|---|---|---|---|
| Bare-D → Progress-only-D | +18 | −31 | −31 | Incremental effect of progress tracking, completion checks, and recovery guidance |
| Progress-only-D → Blocking + Generic retry-D | +6 | −16 | −16 | Incremental effect of the blocking gate |
| Blocking + Generic retry-D → Diagnostic Prompt-D | +4 | −5 | −3 | Incremental effect of event-specific diagnostic guidance |
| Diagnostic Prompt-D → IntentGuard-D | +4 | −6 | −3 | Incremental effect of staged structured recovery |
| Bare-D → IntentGuard-D | +32 | −58 | −53 | End-to-end improvement under the D protocol |
All SCR, UAR, and ASR changes above are percentage-point changes. The configurations are defined as follows:
Bare-Duses the request-derived specification and runtime-visible evidence but has neither a progress controller nor a blocking gate.Progress-only-Dadds progress tracking, completion checks, and recovery guidance without a blocking gate.Blocking + Generic retry-Dadds the same blocking gate while retaining generic retry feedback for blocked actions.Diagnostic Prompt-Dretains the blocking gate and progress controller but replaces generic retry with a single event-specific diagnostic prompt; it does not maintain a staged recovery state.IntentGuard-Duses persistent remaining-obligation state, event-specific guidance, and the staged recovery controller.
Under deployment-available inputs, IntentGuard-D increases SCR from 20% to 52%, decreases UAR from 62% to 4%, decreases ASR from 55% to 2%, and reduces average calls from 31 to 23.
We also evaluate the two external methods with the highest Q01 SCR in the original comparison under the same D protocol (Table R4):
Table R4. External-baseline comparison under matched D inputs.
| Method | SCR | UAR | ASR | Avg calls |
|---|---|---|---|---|
| PACT-D | 32 | 9 | 6 | 26 |
| AgentSentinel-D | 30 | 10 | 6 | 27 |
| IntentGuard-D | 52 | 4 | 2 | 23 |
Thus, under identical deployment-available inputs, IntentGuard retains an advantage over the evaluated external baselines. The component ladder further shows the incremental contributions of progress/recovery assistance and blocking.
Effects of trusted-input provenance
Table R5. Effect of specification and completion-information sources.
| Configuration | SCR | UAR | ASR | Premature completion |
|---|---|---|---|---|
| Reference | 67 | 9 | 2 | 20.00 |
| Replace static specification | 54 | 2 | 2 | 27.03 |
| Replace completion information | 61 | 3 | 1 | 25.00 |
| Replace both | 52 | 4 | 2 | 28.00 |
Benchmark-derived guidance contributes materially to absolute SCR: replacing both the static specification and completion information with deployment-available inputs reduces SCR from 67% to 52%. Under the deployment-input protocol, IntentGuard still outperforms the evaluated external baselines. The corresponding revisions are marked in deep yellow in Section 5.3, Appendices E and F.
Reviewer aSDs — Response 3/3
Continued from the previous response.
Q3. Critical Weakness 2 / Question 3 — Benchmark alignment and fixed operation availability
Critical Weakness 2: “Many scenarios involve manipulating operation parameters or attacker-controlled fields that are precisely the types of violations checked by the deterministic rule gate.”
Question 3: “Could the authors provide a controlled comparison in which operation availability is held fixed and only the environmental content changes?”
We add Q00+ to hold operation availability fixed. Q00+ and Q01 use the same 100 scenario IDs and share the same attack-tool binary and path, permissions, UI fields, operation schema and parameters, initial state, execution markers, independent evaluator, runner, and interaction budget.
The only difference between Q00+ and Q01 is the environmental content (Table R6).
Table R6. Fixed-operation-path content comparison.
| Method | Q00+ SCR | Q00+ UAR | Q00+ ASR | Q01 SCR | Q01 UAR | Q01 ASR |
|---|---|---|---|---|---|---|
| Bare | 15 | 4 | N/A | 16 | 72 | 65 |
| PACT | 30 | 5 | N/A | 32 | 8 | 4 |
| Full IntentGuard | 67 | 4 | N/A | 67 | 9 | 2 |
Q00+ ASR is N/A because the clean-content condition does not instantiate an environmental attacker objective. The attack-operation path remains available, and unsafe behavior is recorded through UAR and the independent goal predicates. For IntentGuard, SCR remains at 67% while UAR increases from 4% to 9%; for Bare, UAR increases from 4% to 72% and ASR reaches 65% in Q01; for PACT, UAR increases from 5% to 8% and ASR reaches 4% in Q01.
Therefore, Q00+–Q01 provides a direct environmental-content comparison with operation availability held fixed. Implementation sanity checks cover target/resource binding, operation binding, parameter binding, and provenance binding; the appendix defines how the 142 labeled events map to the 100 scenarios. OS-Harm and WASP provide complementary evidence on independent benchmarks: IntentGuard achieves 56% SCR and 4% ASR on OS-Harm, and 45.2% SCR and 2.4% ASR on WASP. The corresponding revisions are marked in deep yellow in Section 5.2, Appendix A, and Appendix D.
Reviewer aSDs — Response 1/3
Thank you for the careful review and for highlighting the relationship between benchmark guidance, runtime progress, operation availability, and gate alignment. We address each comment below with matched-input and fixed-path controls. Revisions responding to this review are marked in deep yellow in the revised manuscript. Tables in this response are annotated as Table R*.
Q1. Critical Weakness 1 / Question 1 — Information access and task-completion feedback
We separate the task interface into:
K_static: the approved objective, authorized parameters, protected-resource constraints, and allowed operation forms; andK_completion: the criteria and feedback used for progress tracking and completion assessment.BareandBare+Kare intentional ablations with different information access.Barereceives the user request only.Bare+Kadditionally receivesK_static, but neither configuration has a progress controller.Progress-onlyadds progress tracking, completion checks, and recovery guidance while disabling the blocking gate. Full IntentGuard combines blocking and recovery. Table R1 lists, for each baseline, access to the static specification, completion information, runtime-visible evidence, action interception, and recovery.Table R1. Information access by configuration.
PACT, AgentSentinel, AlignmentCheck, and CaMeL use the same screenshot/a11y observation interface through their native GUI adapters; under the D protocol, they receive the same deployment-available inputs as IntentGuard-D. The corresponding revisions are marked in deep yellow in Section 5.3, Appendices E and F, and Appendix G.