acceptodds
Desk rejected at ICLR 2027

“Allow” to Achieve, Over-Privileged Inadvertently: The Unintended Cost of Task-Completion-Driven Pop-up Decisions in Mobile GUI Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Mobile GUI agents routinely encounter permission dialogs while executing user tasks, yet it remains unclear whether they grant only permissions appropriate to the delegated task. We study this capability, which we term **Permission Literacy**, through a four-level framework based on task relevance and privacy risk, validated by three independent experts. Across 1,072 trials involving four frontier multimodal models and five Android applications, we find that agents frequently grant permissions that fall outside the scope of the current task. Controlled interventions further show that authorization decisions are systematically shaped by both the agent-visible requester and the surrounding task context: under the same Calendar task, changing only the requester from Calendar to PiMusic reduces grants from 26/32 to 0/32, while holding a permission popup fixed and changing the task context also substantially shifts authorization behavior. Prompt-level interventions can sharply reduce unnecessary grants, but often at the cost of suppressing legitimate ones, revealing a persistent safety–utility calibration problem. These results motivate treating permission approval as an explicit, task-scoped authorization decision rather than relying solely on task-completion-oriented behavior. We instantiate this design principle with **Dynamic Minimal Permission Scope (DMPS)**, a lightweight authorization policy that separates permission approval from GUI execution. Together, our findings identify permission handling as a distinct safety requirement for autonomous GUI agents. The source code is available at https://anonymous.4open.science/r/Pop-up-Decisions-in-Mobile-GUI-Agents-F2D1.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Mobile GUI Agents, Permission Literacy, Permission Authorization, Least Privilege, Agent Security, Multimodal Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_arqjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

They study Permission Literacy, through a four-level framework based on task relevance and privacy risk, validated by three independent experts

Critical strengths

-- The task setupp of studying popup based attack is practical.

Critical weaknesses

-- The paper was really hard to follow. Even the title itself was not self-conclusive. 'Task Completion-Driven Pop-up Decisions' is too verbose.

-- The paper has too much jargon terms. It needs significant rewriting effort to meet the bar ICLR expects.

-- Task setup is too narrowly scoped and is of smaller size.

-- The evaluation suite is not robust. I would like to see wider range of evaluation and more evaluation with experts.

Questions for authors

-- see weakness

1 comment
AuthorsOpenReview ↗

Follow-up on Permission Decisions in Mobile GUI Agents (Submission 24819)

Follow-up on Permission Decisions in Mobile GUI Agents (Submission 24819)

Dear Reviewer,

Thank you again for your feedback and for recognizing the practical importance of studying permission decisions in mobile GUI agents.

We would like to highlight a few findings from our response that we believe are particularly important to the understanding of this problem.

1. Permission decisions are systematically influenced by both requester identity and task context.

Our controlled experiments reveal two distinct effects:

  • Under the same Calendar task and otherwise matched popup conditions, changing only the visible requester from Calendar to PiMusic reduces authorization from 26/32 to 0/32.
  • With the permission popup held fixed, changing the active task from PiMusic to Calendar increases grant rates across all four evaluated models.

Together, these results show that permission decisions are shaped by both who requests access and what the agent is currently trying to accomplish. Importantly, application trust and task-specific permission necessity are two separate considerations that an agent needs to account for.

2. Simply making agents more conservative does not necessarily improve permission decisions.

Our structured-prompting experiments reveal a substantial safety–utility trade-off. For Gemini, the L2–L4 grant rate decreases from 69.2% to 0%, but its L1 grant rate also drops from 100% to 31.8%. GPT exhibits a similar reduction in legitimate L1 authorization, from 67.6% to 16.2%.

We believe this is an important practical finding: reducing unnecessary permission grants is not sufficient if agents also lose the ability to recognize permissions that are genuinely needed for the delegated task.

3. We further strengthened the empirical evaluation and clarified its validation.

The study covers 67 AndroidLab tasks, five applications, four multimodal models, and 1,072 trials. We also report an independent assessment by three experts on 24 permission scenarios, achieving Fleiss' κ = 0.869 for the binary Allow versus Deny/Defer decision.

In addition, we introduced fixed-policy baselines for DMPS, including Always Allow and Always Deny, and clarified its decision agreement on 268 unique task–popup combinations. These results provide a more concrete comparison of task-scoped authorization decisions.

We have also taken your comments on readability seriously and focused on simplifying terminology, removing unnecessary coined expressions, and presenting the empirical findings more directly.

We hope these additional results help convey the broader insights of our study: permission over-granting is not simply a matter of agents being too permissive; it also reflects how they interpret requester identity, task context, and the trade-off between granting useful access and avoiding unnecessary exposure.

We would greatly appreciate it if you could take another look at our responses when you have a chance. As the discussion deadline is approaching, we would also welcome any further questions or thoughts, so that we have an opportunity to continue the discussion and clarify any remaining points before it closes.

Thank you again for your time and consideration!

Best regards,
The Authors

Reviewer_w3MHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates the permission literacy of mobile GUI agents. Using tasks from the AndroidLab benchmark, the authors inject application-specific permission popups into different tasks and compare the agents' responses against expert-validated gold labels. The paper also introduces DMPS, a rule-based authorization policy that makes permission decisions separately from GUI execution.

Critical strengths

The topic is timely and important, and the paper is easy to follow. I especially appreciate the Introduction and the motivation in the Related Work section: the gap is well defined, and the paper offers a plausible root cause, namely that over-granting is common in human behavior and may be inherited by agents trained on human data.

The problem and the evaluation framework are clearly defined.

Critical weaknesses
  • My main concerns are about how risk levels are manipulated across tasks and about the corresponding results: The rationale for distinguishing L2, L3 and L4 is unclear. Expert agreement on the fine-grained levels is not high. Both the labels and the popup designs are defined per application rather than per task. This makes these results hard to evaluate, while they constitute a large part of the paper,

  • It is unclear how the coefficients and thresholds in DMPS were determined.

Questions for authors
  1. According to Table 1, whether a permission should be allowed or denied depends on whether it is necessary to complete the task. Why, then, is a multi-level classification needed? Moreover, expert agreement on the finer-grained levels is not high. I can think of one possible justification, although I'd more like to hear authors' thoughts on that. Finer-grained levels could serve as a dimension for capturing differences in users' autonomy boundaries in future work. Although individual preferences are not the focus of this study, users may differ in which permissions they are willing to let an agent grant. For example, in certain scenarios a user may accept the agent allowing an L2 request but not L3 or L4 requests.

  2. Labels are assigned per popup rather than per task–popup pair, whereas the necessity of a permission can depend on the specific task. To what extent does this labeling scheme affect the results? For example, after popups are assigned by application and level, how closely does the actual level of each task–popup pair align with the original application-level label?

  3. How were the coefficients and thresholds in DMPS determined, and why was it designed this way? Were these values tuned on the same 24 scenarios used for evaluation?

  4. Since the evaluation is based on actual agent runs rather than model question answering, it would be beneficial to also consider the effect of the agent harness, not only the underlying model. This should at least be discussed as a limitation or as future work.

2 comments
AuthorsOpenReview ↗

Follow-up on Permission Decisions in Mobile GUI Agents (Submission 24819)

Follow-up on Permission Decisions in Mobile GUI Agents (Submission 24819)

Dear Reviewer,

Thank you again for your positive assessment and for raising several important methodological questions. We appreciate your recognition of the practical relevance of Permission Literacy, and your feedback has helped us further clarify the scope and interpretation of our evaluation framework.

We would like to highlight a few clarifications from our additional analyses and responses.

1. The purpose of the four-level permission taxonomy

We clarified that the primary authorization analysis distinguishes L1 permissions from L2–L4 permissions. The finer-grained categories serve a diagnostic purpose: they distinguish optional convenience, permissions with little functional relevance, and highly sensitive permissions disproportionate to the delegated task.

Our expert evaluation also supports this distinction, although agreement is stronger for the binary Allow versus Deny/Defer decision (Fleiss' κ = 0.869) than for the finer-grained four-level classification (κ = 0.574). We have made this limitation explicit rather than treating the four levels as equally definitive.

2. Application-level labels and task-level necessity

We agree that permission necessity ultimately depends on the specific task, not simply the application.

To examine this issue, we analyzed 268 unique task–popup combinations. The task-scoped analysis shows that L1 cases generally receive higher necessity scores than L3 and L4 cases, although individual tasks can differ from their representative application-level scenarios.

Importantly, we clarified that these scores are generated by the DMPS heuristic itself, rather than independent expert annotations. They are therefore exploratory evidence, not independent validation of the original labels.

3. DMPS design and parameter sensitivity

We further clarified that DMPS coefficients and thresholds were manually specified as rule-design constants, rather than learned or tuned on the 24 expert-annotated scenarios.

We also added a threshold sensitivity analysis. The results show that L2–L4 decisions remain stable across a reasonable range of the required-permission threshold, while more conservative thresholds increasingly suppress legitimate L1 grants. This illustrates the safety–utility trade-off underlying the policy.

Compared with Always Allow and Always Deny, DMPS achieves 89.9% agreement with our operational labels on 268 unique task–popup combinations. We emphasize that this measures decision agreement, not independent expert validation or end-to-end task success.

Finally, we clarified that all four models were evaluated under a consistent harness, while acknowledging that we did not independently vary harness configurations. Cross-harness evaluation remains an important direction for future work.

We hope these clarifications address your methodological questions and make the design choices and limitations of our framework easier to assess.

We would greatly appreciate any further feedback you may have as the discussion approaches its deadline. If there are still aspects of the taxonomy, task-level evaluation, or DMPS design that you would like us to clarify, we would be happy to elaborate.

Thank you again for your valuable feedback and consideration!

Best regards,
The Authors

Thank you for the rebuttal. I like this work anyway. Just to make sure to prepare well the clarification mentioned above in the camera ready if possible. I will keep my original positive score. Good luck!

Reviewer_sLMfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

App-use agents make bad decisions, when they receive a pop-up permission request while doing a task.

Critical strengths
  • Highlights a practical problem in current app-use agents.
  • Hopefully will stimulate research to address this problem.
  • Innovative and valuable, I might share this with other researchers
  • The paper provides a benchmark
Critical weaknesses
  • Beyond the fact that current agents accept too many permission requests, I'm not sure that I learned too much more or gained much more insight.
  • There is no plausible solution proposed in the paper, not even a reasonable baseline, as far as I can tell.
  • Some of the writing was not fun to read. I don't want to read AI-styled writing. This might not rise to the level of warranting rejection on its own.
  • Hard for me to evaluate the benchmark. I have some doubts about some aspects of the benchmark, I'm not sure whether it is the right one to drive progress in the field, but I'm not sure whether this is fair.

Other comments for the authors (not critical weaknesses):

l51: I recommend that you avoid using typewriter font (\tt) for emphasis. Use it only for code.

l95: I'm confused. Why would Clock request microphone access? Is this a real example? Or are you imagining a hypothetical where an application requests more than it needs, but this is not intended to be realistic? I've never experienced this happening.

l97: But it's not task-irrelevant: being able to turn off an alarm is arguably relevant to setting an alarm. I think it's subjective / a matter of opinion/preference what is the best outcome here. That is a good argument for not auotmatically clicking allow, because it will allow not just for this alarm but for all alarms, which the user might or might not want. But I don't think it's right to dismiss this as "task-irrelevant".

l100: No need to explain how elements are tagged or provided to the agent, that is a distraction from the point you're trying to make with this example.

l105: "Excessive risk" seems debateable: the Clock application is provided by Google and hence presumably has a relatively high trust, so probably is a relatively low risk. I think this illustrates how hard it is to label these scenarios.

l268: I'm a bit skeptical about this methodology. It's not clear how the permission request is selected, but I suspect it is random independent of the app that is being interacted with. If that's correct, it seems likely that the agent will mostly see out-of-distribution permission requests that are unlikely to ever occur from that app. So the benchmark seems like it is evaluating situations that are unlikely to occur in real life. They could occur if the app were malicious, but l217 says that the paper focuses on benign apps and unintended over-privilege. Given that focus, to make the benchmark as applicable/valid as possible, I think the benchmark should be designed to select permission requests that the specific app might realistically request.

How do we evaluate how realistic the tasks and permission requests are, in the benchmark?

l268: How do you select/generate the justification to display? I expect this might have an influence on agent behavior. Are they real justifications from this app? Synthetic/invented justifications? How salient/persuasive are they, in terms of explaining why the request is relevant to the operation being performed?

l269: I don't understand what a "permission level" is. Please try explaining -- define all non-standard terms before first use.

l299: To be as useful for others as possible, a well-designed benchmark should have a single number/metric that captures how well the model did on the benchmark, and higher numbers usually indicate better performance. It does not appear that this benchmark has been designed in that way. There are multiple numbers (e.g., one Grant Rate per scenario), so it is hard to compare models or summarize how well a model does. Well-designed benchmarks are helpful to drive progress.

l322: I find this confusing. Maybe I don't know Android well enough. What's the motivation for studying permission requests from a different app than the one I'm using, that interrupt me? If I'm using the Calendar app, why would a permission request from PIMusic appear? Is this relevant to practice?

l356: This text makes no sense to me. You are reporting empirical findings; it doesn't make sense to me to "define" App-Trust Bias. This should be reporting the results of experiments, not introducing new terms and definitions. I find the sentence hard to follow, with stacked noun phrases (it reads a little like LLM-speak, and I find it unpleasant to read).

Sec 5.2: I have the same question why a scenario with a cross-app permission request would arise in practice and why this is practically relevant. It feels like it might be an invented scenario.

l426: "strictly necessary" seems like too extreme of a criterion, and might bias the agent towards denying (causes fear not well-informed judgement). I suspect this could cause loss of utility of the agent. I encourage you to find a way to evaluate that.

Sec 5.3: It seems like prompt-level mitigation was ineffective. It didn't lead the agent to make wiser judgements, it just made the agent more fearful / more reluctant to approve permission requests. I disagree with the conclusion on l453, I do not think that is well supported by the evidence, and I would prefer that the paper not include such statements.

l443, l453, etc.: Please avoid AI writing style. In a conference paper, I want to read writing from human writers, not the stylistic quirks of a LLM. I read LLM output all the time, and it's painful and tiring and tedious. If you used LLMs to help you write/revise the paper, I think it's harming the paper and making me dislike reading the paper a little bit. Saying things like "does not yield uniformly calibrated authorization across" is unhelpful -- it's like saying "it's not perfect" -- well sure, I could already guess it won't not be perfect. Don't tell me what it's not, tell me what it is. Tell me what useful/interesting conclusions you can draw, tell me what are the answers to the research questions you've raised. I'm sure it was your intention to improve clarity, but I suspect it might have made it worse.

l463: "permission-conditioned evidence", "justification alignment", etc. - please use simpler, direct language. Optimize for reader comprehension, not for trying to make the paper sound sophisticated. Be specific and say something, don't waffle. Be concrete, not vague. What's a "required path"?

Sec 5.4: I don't see any evaluation results for DMPS. Consider adding it to Table 3, with the same metrics? Based on Table 11, it sounds like it only successfully complete 64% of tasks where a permission request should be granted? So it is too fearful/cautious about accepting permission requests?

Table 4: I'm questioning how relevant some of these are. Does the Clock app really request contacts and location access at runtime? Does the Contacts app really request runtime permission for calendar, files, location? Can PiMusic really request contacts and location at runtime? Does Zoom really request SMS?

Appendix: What are the tasks the agent is asked to do? Shouldn't the level of a permission request depend not just on the app, but also on the task that the agent was asked to do?

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Follow-up on Permission Decisions in Mobile GUI Agents (Submission 24819)

Follow-up on Permission Decisions in Mobile GUI Agents (Submission 24819)

Dear Reviewer,

Thank you again for your thoughtful follow-up and for taking the time to discuss the remaining questions with us. We particularly appreciate your openness to reconsidering the paper and are glad that our clarification regarding the Clock microphone example addressed your concern.

We would like to briefly clarify one point regarding the significance of requester identity, which we understand may not have been sufficiently clear in our original presentation.

1. Application trust versus task-specific permission necessity

We fully agree that an agent should consider the trustworthiness of the requesting application. Our intention is not to suggest that Calendar and PiMusic should be treated equally, or that application identity should be ignored.

Rather, we want to distinguish application trust from whether a particular permission is necessary for the task delegated by the user. Even a trusted application may request access for additional features that are not needed for the current task.

Our controlled experiment illustrates this distinction: under the same Calendar task and otherwise matched popup conditions, changing only the visible requester from Calendar to PiMusic reduces authorization from 26/32 to 0/32. We view this as evidence that requester identity substantially influences authorization, rather than evidence that the two applications should receive identical treatment.

2. Why study decision factors rather than only benchmark performance or defenses?

We see the controlled requester-identity and task-context experiments as a way to understand how current agents make permission decisions, rather than simply measuring how often they over-grant.

Our prompting experiments further reveal an important trade-off: making agents more conservative can reduce unnecessary grants, but may also prevent legitimate permissions from being granted. We believe understanding these decision patterns is useful for developing more reliable permission-handling mechanisms in the future.

We also clarified the role of DMPS as a preliminary task-scoped authorization prototype, rather than presenting it as a complete defense or deployment-ready solution.

We hope these clarifications make the motivation and empirical contribution of the paper clearer. We sincerely appreciate your constructive engagement with our work and your willingness to consider our responses.

As the discussion deadline is approaching, we would be grateful if you could let us know if any remaining concerns or questions would benefit from further clarification. We would be happy to address them while there is still time.

Thank you again for your time and thoughtful consideration!

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Closing Note on Our Rebuttal

Rebuttal Summary

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their valuable feedback. During the discussion period, we conducted additional experiments and analyses, clarified the evaluation methodology, and further strengthened the presentation of our findings. We summarize the additional work below.

Paper Summary

This paper studies Permission Literacy in mobile GUI agents, examining whether agents grant only permissions necessary for users' delegated tasks. Across 1,072 trials involving four multimodal models, five Android applications, and 67 AndroidLab tasks, we identify two key factors influencing permission decisions: requester identity and task context. We also reveal a safety–utility trade-off in prompt-based mitigation and explore DMPS as a preliminary task-scoped authorization prototype.

Discussion Period Summary

During the discussion, we addressed the reviewers' main concerns through the following additional experiments and analyses:

  • Permission decision factors (Reviewers sLMf, arqj). We strengthened the evidence for the effects of requester identity and task context. Under the same Calendar task, changing only the requester from Calendar to PiMusic reduces grants from 26/32 to 0/32. Conversely, changing the task from PiMusic to Calendar increases grant rates across all four models, with consistent results in a second permission-family replication.

  • Safety–utility trade-off (Reviewers sLMf, arqj). We expanded the structured-prompting analysis. For Gemini, L2–L4 grants decrease from 69.2% to 0%, but legitimate L1 grants also drop from 100% to 31.8%. These results show that simply making agents more conservative does not necessarily improve task-specific authorization.

  • Scenario design and expert validation (Reviewers sLMf, w3MH, arqj). We clarified the construction of synthetic permission scenarios, the distinction between application trust and task-specific necessity, and the four-level permission taxonomy. We additionally analyzed 268 unique task–popup combinations and reported independent assessments by three experts, achieving Fleiss' κ = 0.869 for binary Allow versus Deny/Defer decisions.

  • DMPS evaluation (Reviewers sLMf, w3MH). We added comparisons against Always Allow and Always Deny. DMPS achieves 89.9% agreement with operational labels, compared with 25.0% and 75.0%, respectively. A threshold sensitivity analysis further illustrates the trade-off between preventing unnecessary grants and preserving legitimate L1 authorization.

  • Presentation and evaluation scope (Reviewers sLMf, w3MH, arqj). We simplified terminology, replaced abstract expressions with direct descriptions of experimental findings, and clarified the role of controlled cross-application scenarios and the consistent agent harness used in our evaluation.

The additional results and clarifications are available in the corresponding review discussions. We believe these efforts substantially strengthen the empirical findings and clarify the contribution of this work to understanding permission decisions in mobile GUI agents.

We sincerely appreciate your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.