- Beyond the fact that current agents accept too many
permission requests, I'm not sure that I learned too much
more or gained much more insight.
- There is no plausible solution proposed in the paper,
not even a reasonable baseline, as far as I can tell.
- Some of the writing was not fun to read. I don't want to
read AI-styled writing. This might not rise to the level
of warranting rejection on its own.
- Hard for me to evaluate the benchmark. I have some
doubts about some aspects of the benchmark, I'm not sure
whether it is the right one to drive progress in the
field, but I'm not sure whether this is fair.
Other comments for the authors (not critical weaknesses):
l51: I recommend that you avoid using typewriter font (\tt) for
emphasis. Use it only for code.
l95: I'm confused. Why would Clock request microphone access?
Is this a real example? Or are you imagining a hypothetical where
an application requests more than it needs, but this is not
intended to be realistic? I've never experienced this happening.
l97: But it's not task-irrelevant: being able to turn off an
alarm is arguably relevant to setting an alarm. I think it's
subjective / a matter of opinion/preference what is the best
outcome here. That is a good argument for not auotmatically
clicking allow, because it will allow not just for this alarm
but for all alarms, which the user might or might not want.
But I don't think it's right to dismiss this as "task-irrelevant".
l100: No need to explain how elements are tagged or provided
to the agent, that is a distraction from the point you're trying
to make with this example.
l105: "Excessive risk" seems debateable: the Clock application
is provided by Google and hence presumably has a relatively
high trust, so probably is a relatively low risk. I think this
illustrates how hard it is to label these scenarios.
l268: I'm a bit skeptical about this methodology. It's not
clear how the permission request is selected, but I suspect
it is random independent of the app that is being interacted
with. If that's correct, it seems likely that the agent will
mostly see out-of-distribution permission requests that are
unlikely to ever occur from that app. So the benchmark seems
like it is evaluating situations that are unlikely to occur
in real life. They could occur if the app were malicious,
but l217 says that the paper focuses on benign apps and
unintended over-privilege. Given that focus, to make the
benchmark as applicable/valid as possible, I think the
benchmark should be designed to select permission requests
that the specific app might realistically request.
How do we evaluate how realistic the tasks and permission
requests are, in the benchmark?
l268: How do you select/generate the justification to display?
I expect this might have an influence on agent behavior.
Are they real justifications from this app? Synthetic/invented
justifications? How salient/persuasive are they, in terms of
explaining why the request is relevant to the operation being
performed?
l269: I don't understand what a "permission level" is.
Please try explaining -- define all non-standard terms before
first use.
l299: To be as useful for others as possible, a well-designed
benchmark should have a single number/metric that captures
how well the model did on the benchmark, and higher numbers
usually indicate better performance. It does not appear
that this benchmark has been designed in that way. There are
multiple numbers (e.g., one Grant Rate per scenario), so it
is hard to compare models or summarize how well a model does.
Well-designed benchmarks are helpful to drive progress.
l322: I find this confusing. Maybe I don't know Android
well enough. What's the motivation for studying permission
requests from a different app than the one I'm using, that
interrupt me? If I'm using the Calendar app, why would a
permission request from PIMusic appear? Is this relevant
to practice?
l356: This text makes no sense to me. You are reporting
empirical findings; it doesn't make sense to me to "define"
App-Trust Bias. This should be reporting the results of
experiments, not introducing new terms and definitions.
I find the sentence hard to follow, with stacked noun
phrases (it reads a little like LLM-speak, and I find it
unpleasant to read).
Sec 5.2: I have the same question why a scenario with a
cross-app permission request would arise in practice and
why this is practically relevant. It feels like it might
be an invented scenario.
l426: "strictly necessary" seems like too extreme of a
criterion, and might bias the agent towards denying (causes
fear not well-informed judgement). I suspect this could
cause loss of utility of the agent. I encourage you to
find a way to evaluate that.
Sec 5.3: It seems like prompt-level mitigation was
ineffective. It didn't lead the agent to make wiser
judgements, it just made the agent more fearful / more
reluctant to approve permission requests. I disagree
with the conclusion on l453, I do not think that is
well supported by the evidence, and I would prefer
that the paper not include such statements.
l443, l453, etc.: Please avoid AI writing style. In
a conference paper, I want to read writing from human
writers, not the stylistic quirks of a LLM. I read LLM
output all the time, and it's painful and tiring and
tedious. If you used LLMs to help you write/revise
the paper, I think it's harming the paper and making me
dislike reading the paper a little bit. Saying things
like "does not yield uniformly calibrated authorization
across" is unhelpful -- it's like saying "it's not perfect"
-- well sure, I could already guess it won't not be
perfect. Don't tell me what it's not, tell me what
it is. Tell me what useful/interesting conclusions
you can draw, tell me what are the answers to the
research questions you've raised. I'm sure it was
your intention to improve clarity, but I suspect it
might have made it worse.
l463: "permission-conditioned evidence", "justification
alignment", etc. - please use simpler, direct language.
Optimize for reader comprehension, not for trying to
make the paper sound sophisticated. Be specific and say
something, don't waffle. Be concrete, not vague. What's
a "required path"?
Sec 5.4: I don't see any evaluation results for DMPS.
Consider adding it to Table 3, with the same metrics?
Based on Table 11, it sounds like it only successfully
complete 64% of tasks where a permission request should
be granted? So it is too fearful/cautious about
accepting permission requests?
Table 4: I'm questioning how relevant some of these are.
Does the Clock app really request contacts and location access
at runtime? Does the Contacts app really request runtime
permission for calendar, files, location? Can PiMusic
really request contacts and location at runtime? Does
Zoom really request SMS?
Appendix: What are the tasks the agent is asked to do?
Shouldn't the level of a permission request depend not
just on the app, but also on the task that the agent was
asked to do?
Follow-up on Permission Decisions in Mobile GUI Agents (Submission 24819)
Follow-up on Permission Decisions in Mobile GUI Agents (Submission 24819)
Dear Reviewer,
Thank you again for your feedback and for recognizing the practical importance of studying permission decisions in mobile GUI agents.
We would like to highlight a few findings from our response that we believe are particularly important to the understanding of this problem.
1. Permission decisions are systematically influenced by both requester identity and task context.
Our controlled experiments reveal two distinct effects:
Together, these results show that permission decisions are shaped by both who requests access and what the agent is currently trying to accomplish. Importantly, application trust and task-specific permission necessity are two separate considerations that an agent needs to account for.
2. Simply making agents more conservative does not necessarily improve permission decisions.
Our structured-prompting experiments reveal a substantial safety–utility trade-off. For Gemini, the L2–L4 grant rate decreases from 69.2% to 0%, but its L1 grant rate also drops from 100% to 31.8%. GPT exhibits a similar reduction in legitimate L1 authorization, from 67.6% to 16.2%.
We believe this is an important practical finding: reducing unnecessary permission grants is not sufficient if agents also lose the ability to recognize permissions that are genuinely needed for the delegated task.
3. We further strengthened the empirical evaluation and clarified its validation.
The study covers 67 AndroidLab tasks, five applications, four multimodal models, and 1,072 trials. We also report an independent assessment by three experts on 24 permission scenarios, achieving Fleiss' κ = 0.869 for the binary Allow versus Deny/Defer decision.
In addition, we introduced fixed-policy baselines for DMPS, including Always Allow and Always Deny, and clarified its decision agreement on 268 unique task–popup combinations. These results provide a more concrete comparison of task-scoped authorization decisions.
We have also taken your comments on readability seriously and focused on simplifying terminology, removing unnecessary coined expressions, and presenting the empirical findings more directly.
We hope these additional results help convey the broader insights of our study: permission over-granting is not simply a matter of agents being too permissive; it also reflects how they interpret requester identity, task context, and the trade-off between granting useful access and avoiding unnecessary exposure.
We would greatly appreciate it if you could take another look at our responses when you have a chance. As the discussion deadline is approaching, we would also welcome any further questions or thoughts, so that we have an opportunity to continue the discussion and clarify any remaining points before it closes.
Thank you again for your time and consideration!
Best regards,
The Authors