- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies whether GUI agents distinguish an available action from an authorized action. It introduces AXIS, which crosses request formulations with screenshot conditions, and AXIS-Core, a LoRA-based approach that supervises risk and authorization assessments before action selection. Experiments report improved static decision accuracy, stronger discrimination between authorized and unauthorized requests, and limited changes in native grounding and navigation performance.
Critical strengths
The authorization-contrast pairs are a useful evaluation design: holding the screenshot and operation fixed while changing the stated authorization can distinguish indiscriminate execution from indiscriminate withholding. AXIS-Core’s improvements on these pairs provide meaningful evidence that its gains are not solely due to increased refusal.
Critical weaknesses
-
The paper is difficult to read, substantially hindering technical assessment. Sentences are frequently long and densely packed, with awkward clause boundaries and several ideas compressed into a single statement. Even basic descriptions require repeated reading to determine the intended meaning. Overlapping terminology and definitions scattered across the main text and appendices add to this burden. The paper needs substantial rewriting with shorter sentences, clearer logical connections, and consistent terminology so that readers can reliably assess its methods and claims.
-
The experimental design does not isolate the effects claimed. The H/L comparison changes authorization-relevant information, not just wording: H explicitly states overreach, whereas L leaves authorization unstated. Their execution gap therefore cannot be attributed specifically to lexical salience. Likewise, the clarification metric excludes authorization-confirmation questions. An agent that switches from asking what the user wants to asking for permission could show a large “clarification deficit” while still withholding action appropriately. These confounds undermine the paper’s central diagnosis. The training improvements do not resolve them; the claims require better-controlled comparisons and measures of execution without prior clarification or authorization confirmation.
Questions for authors
-
Clarification and reanalysis: If intent clarification, authorization confirmation, and refusal are considered jointly, how often do agents actually commit to an action without first requesting the missing information or authorization? Does the screen-dependent effect remain under this measure?
-
Clarification of the claimed effect: Given that H and L differ in explicit evidence of overreach, what supports interpreting their execution gap as lexical salience rather than sensitivity to different authorization information? Is there an existing comparison that holds authorization semantics fixed while varying wording?
4 comments
Thank you for your insightful comments.
We have uploaded a revised manuscript. We revised the prose throughout and marked new analyses in blue.
Presentation. We shortened sentences, separated the experimental setup from its interpretation, and used consistent terms in the main text and appendix. Section 3.1 now distinguishes asking what the user means (CLARIFY) from asking for confirmation before proceeding (PAUSE), and Section 3.4 defines the response measures before Section 4 presents the findings.
Clarification and confirmation. You raised the possibility that agents switched from asking about intent to asking for permission. Confirmation requests do become more frequent on task-relevant screens (Appendix B.3, Table 7). However, responses that proceed with no clarification, confirmation, or refusal also rise in all seven configurations, by 30.7–52.0 percentage points in the five unguarded models (Holm-adjusted p < 10⁻⁶; Figure 2b). The drop in clarification is therefore not only a switch to asking for permission.
Wording and non-consent. Section 3.3 varies these two factors independently across all 75 scenarios, holding the screen, target, and operation fixed. Adding explicit non-consent to a neutral request significantly reduces direct proceeding in five of seven configurations after Holm correction, whereas changing the wording alone, with consent still unstated, does so in two (Figure 2c). Appendix B.4, Table 9 repeats the comparison counting only responses that explicitly report or commit to execution. Under that measure, adding non-consent to the neutral request lowers the rate in all seven configurations.
Authorization pairs. On the 50 neutral AndroidControl pairs you highlighted, the best of the five prompting baselines resolves 19 when given the full policy and four demonstrations, whereas AXIS-Core resolves 47–49 across three training seeds (Sections 6.1 and 6.3).
We hope the revised manuscript has addressed your concerns. If you have any further concerns, we would be glad to address them before the discussion period ends.
I think the new experiment addresses my concern about the unfair comparison. And I appreciate the authors’ efforts to improve the readability of the abstract and introduction though leave much to improve. I will therefore will raise my score to 2.
W1. Fairness of the comparison
I think the new experiment addresses my concern about the unfair comparison.
We are glad the new experiment addressed your concern about the fairness of the comparison. Thank you for reconsidering the paper and raising your score.
W2. Readability of the manuscript
And I appreciate the authors’ efforts to improve the readability of the abstract and introduction though leave much to improve.
Thank you for recognizing the improvements to the abstract and introduction. We agree that the manuscript still needs a thorough edit. Much of the rebuttal period went into running and checking the additional experiments, and we are still revising the later sections in depth. The examples below show how we are applying your three suggestions throughout the paper.
1. Shorter sentences
- Before (original §5.4). A single sentence used two colons to connect four counts of completed benign actions, an explanation of the model’s behavior, and a drop in grounding accuracy. It reported 13/18 (ID), 11/18 (linguistic-OOD), 50/95 (AndroidControl), and 42/85 (AITW + GUIOdyssey) before introducing the “high-impact, therefore withhold” interpretation.
- After (working draft, §6.4 and §7.2). After reporting the counts in §6.4, we explain the pattern in a separate sentence, “Without rationale supervision, it associates high-impact operations with stopping.” Section 7.2 discusses the grounding result. We made a similar change in Appendix A.1, where the original description of request construction used three semicolons and a colon in one sentence. The revised passage walks through the procedure in six shorter sentences (lines 951 to 957).
2. Clearer logical connections
- Before (original §3.2, §4.1, and §4.2). Section 3.2 mixed predictions with metric definitions before the reader had seen the decision rule, which first appeared in §4.1. Section 4.2 also introduced each comparator and predicted its failure in the same sentence.
- After (working draft, §3.1, §3.4, §4, and §5). Section 3.1 introduces the decision rule (Eq. 1), and §3.4 defines the response measures. Section 4 then reports the drop in clarification, considers whether agents simply switched to asking for permission, and tests that explanation (lines 244 to 255). Section 5 connects this finding to the supervision each comparator receives (lines 294 to 300), before §6 presents the results.
3. Consistent terminology
- Before. The original draft alternated among “execution justification,” “warrant,” and “authorization.” It also used several labels for screen conditions (“correct,” “matching,” “real-neutral,” and “unrelated”), wording (“low-salience,” “Low-lex,” and “neutral”), and the paraphrase set (“linguistic-OOD” and “paraphrase”). Readers also had to keep track of three acronyms for the contrasts (CD, LSG, and NSE).
- After. The working draft uses “stated authorization,” “task-relevant” and “unrelated” screens, “neutral” and “harm-framed” requests, and “the paraphrase set.” Section 3.1 distinguishes CLARIFY from PAUSE. Section 3.4 defines the three response measures (asks what is meant, proceeds, and proceeds directly), and we use the same terms in §4, Figure 2, and Appendix B.
We will continue this work across the remaining sections and appendices, then proofread the manuscript as a whole. We will make the paper easier to read and the argument easier to follow.
We hope this ongoing rewriting, together with the final proofreading pass, fully addresses your remaining concern about the writing, and we would be grateful if you could take it into account in your evaluation.
Thank you for identifying the need to separate authorization information from wording and to account for authorization-confirmation questions.
1. Separating wording from explicit non-consent.
We added two request variants for each of the 75 scenarios, producing a 2×2 design that crosses harm-related wording with an explicit statement of non-consent. The screenshot, target, and requested operation remain fixed. All four conditions were evaluated together.
The table reports the percentage of responses that proceed directly, excluding intent clarification, authorization-confirmation requests, and safety refusals.
Each cell includes all 75 scenarios. Under neutral wording, adding explicit non-consent reduces direct proceeding in six configurations, significantly in five after Holm correction. Without explicit non-consent, harm-related wording reduces it in six configurations, significantly in two after correction.
The original H/L contrast therefore combines two effects. The revised interpretation will distinguish responses to stated non-consent from responses to harm-related wording, with their relative influence varying across models.
2. Accounting for confirmation as well as clarification.
We recomputed the screen comparison over all 1,050 original responses, retaining the original action-response definition and excluding every response labeled as intent clarification, authorization confirmation, or safety refusal.
All five unguarded models retain an increase of 30.7–52.0 percentage points, with Holm-adjusted p<10−6. The guarded increases are 4.0 and 18.7 points and do not reach significance after correction.
Authorization-confirmation questions do increase on matching screens, but this change does not fully account for the screen effect. The remaining difference concerns responses that proceed without either type of question or a safety refusal.
This endpoint measures action-oriented responses, including accepted next-action statements; it is not a device-execution rate. We will state this definition explicitly and report clarification, confirmation, refusal, and direct proceeding separately.
3. Writing and presentation.
We agree that several passages are hard to follow even on a careful reading. Some sentences introduce the experimental setup, the comparison, and its interpretation all at once. We will rewrite these passages in shorter sentences, making clear what was compared before explaining what the result shows. We will also define key terms briefly at first use in the main text and use the same terms throughout.