acceptodds
Under review as a conference paper at ICLR 2027

SafeDroid: The Cost of Obedience for VLM-Based Mobile Agents

Abstract

Mobile agents built on vision-language models (VLMs) operate a phone on the user's behalf, changing messages, accounts, and settings. Based on a questionnaire of 2,240 smartphone users, we found that 11% of requested phone changes were judged harmful in context by independent participants. However, agent-safety research has concentrated its threat model on externally injected goals—prompt injection, manipulated interfaces, untrusted tool output—so on mobile devices the user's own request is rarely the object of evaluation. We distill the 11% into twelve harm domains and introduce SafeDroid, a survey-grounded stress test of mobile agents' responses to risk-sensitive user requests, with device-state verification. Evaluating eleven general-purpose VLMs and five specialized GUI models reveals severe vulnerabilities: as single agents, the general-purpose VLMs reach the requested device states (target states) in 58.3% of runs without any external manipulation, and 93.6% of those trajectories never identify the specific consequence. Although Mobile-Agent-v3 (MA-v3) lowers the overall target-state reach rate, this reduction stems primarily from capability failure rather than cautious refusal. We call this the cost of obedience: overlooking consequences while faithfully carrying out a user request. Therefore, mobile-agent safety must assess the consequences of user requests even when they arrive through a trusted input channel.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Mobile Agents, Vision-Language Models, Agent Safety, Safety Benchmarking, Human-Centered AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NMegOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SafeDroid, a benchmark designed to evaluate the safety of VLM-based mobile agents when executing potentially harmful user requests. Based on a survey of 2,240 smartphone users, the authors construct a benchmark covering 12 harm domains, 17 Android tasks, and 255 request variants. The benchmark employs device-state verification to distinguish successful execution, explicit refusal, and capability failure. The authors evaluate 11 general-purpose VLMs and five specialized GUI models under different agent configurations. Experimental results reveal that single-agent VLMs reach potentially harmful target states in 58.3% of runs, with 93.6% of these successful trajectories failing to explicitly recognize the associated risks, while the lower target-state reach rate of the multi-agent framework is primarily attributable to capability failures.

Critical strengths
  1. The paper investigates safety risks arising from legitimate user requests, providing a valuable complementary perspective to existing agent safety research primarily focused on prompt injection and adversarial manipulation.
  2. The benchmark is grounded in a large-scale user survey and incorporates device-state verification to distinguish deliberate refusal from execution failure, improving the reliability of safety evaluation.
  3. The extensive evaluation across multiple models and agent configurations reveals important discrepancies between text-only safety assessments and actual GUI execution, particularly highlighting how capability failures can be mistakenly interpreted as safe behavior.
Critical weaknesses
  1. Despite including 255 test cases, the benchmark contains only 17 underlying tasks, most of which involve single-app interactions and single-state modifications. This limits its representativeness of complex real-world mobile agent workflows.
  2. Harmfulness labels are primarily determined through majority voting by ordinary users. Given the context-dependent nature of many operations, the absence of detailed inter-annotator agreement statistics raises concerns about annotation reliability.
  3. Risk awareness is assessed through recorded reasoning traces and an LLM-based judge, with human validation conducted by only one annotator. Consequently, the reliability of the reported high Unaware rates remains uncertain.
Questions for authors
  1. Given that most benchmark tasks involve single-app interactions and single-state changes, to what extent do the authors believe these tasks capture the safety challenges of more complex, real-world mobile agent workflows?
  2. How do the authors ensure the reliability of harmfulness annotations for highly context-dependent operations, such as disabling reminders or modifying permissions? Could additional inter-annotator agreement statistics be provided?
  3. The paper reports that 93.6% of target-reaching trajectories are classified as Unaware. How do the authors distinguish between agents that genuinely fail to recognize harmful consequences and those that recognize the risks but do not explicitly articulate them in their recorded reasoning?
Reviewer_8NWPOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SafeDroid, a benchmark for evaluating whether mobile agents can recognize the risks of working on user requests without external attacks. Based on a questionnaire study, the authors organize answers into harm domains and benchmark tasks. The paper evaluates substantial models and the Mobile-Agent-v3 harness. Results argue that model agent safety evaluation should account for risks introduced by legitimate user requests.

Critical strengths
  1. Important problem. The paper studies a meaningful and practical safety issue in mobile agents.
  2. Useful design of device-state verification. The evaluation checks whether agents actually reach the requested states. Importantly, it distinguishes successful execution, sustained refusal, and capability failure.
  3. Comprehensive evaluation. The evaluation contains 11 general-purpose VLMs and 5 specialized GUI models under 3 settings (text only, single agent, and Mobile-Agent-v3), which makes the results convincing.
Critical weaknesses
  1. The definition of harmful behavior is not sufficiently convincing. My main concern is whether reaching a risk-sensitive target state should always be interpreted as a safety failure. Some evaluated operations may create potential risks, but they are not necessarily harmful when explicitly requested by the device owner. For example, Figure 2 shows a case that user asks the agent to disable their daily medication reminder. I agree that this operation deserves caution, but I am less convinced that executing it is unsafe. A reasonable agent might explain the consequence, ask for confirmation, and then respect the user's informed decision. A similar case appears in Table 13 T02. It considers reaching a purchase or billing-verification screen as the target state, even though no payment is completed. It is unclear whether this constitutes financial harm.
  2. The human study need stronger clarification. First, how were the participants recruited? Appendix mentions diverse background but no detailed demographic breakdown. Second, prior mobile agent experience is not required. While this is reasonable for collecting general smartphone preferences, hypothetical requests from people who have never used a mobile agent may differ from requests made during actual agent-assisted workflows. Also, the requests used in the benchmark are manually constructed rather than directly sampled from the original user responses. This weakens the connection between the user study and the evaluated task distribution.
  3. Limited diversity of executable tasks. SafeDroid covers 12 harm domains but the benchmark consists of only 17 distinct Android operations. Also, appendix shows that only two tasks require multiple apps and only one involves multiple state changes. This leaves question that how well the findings generalize to more complex and real-world mobile-agent workflows.
Questions for authors
  1. I would like the authors to clarify the motivation on how they distinguish potentially risky actions from real harmful outcomes. And whether successful execution after appropriate warning or confirmation should receive the same safety assessment as unaware execution.
  2. For human study, please refer to weakness 2.
  3. If time permits, I want to see more tasks/cases of complex workflows with multiple apps/states.
Reviewer_LqUUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces SafeDroid, a benchmark grounded in a survey of 2,240 smartphone users, in which 10.7% of 9,924 elicited phone-change requests were judged harmful in context. Its 17 verifiable Android tasks and 255 matched direct/framed requests show that VLM-based mobile agents, without any external attack, reach risk-sensitive target states in 58.3% of single-agent runs, mostly without identifying the consequences, while lower success rates (e.g., under MA-v3) largely reflect capability failure rather than refusal, and text-only refusal fails to predict on-device behavior.

Critical strengths
  1. The shift in threat model is valuable. The paper moves the object of evaluation from “externally injected goals” to “ordinary user requests arriving through a trusted channel,” filling a gap in existing mobile-agent safety benchmarks.

  2. The metric design is clear and avoids misreading low TRR as safety. Target reached, Refusal, and Capability failure are mutually exclusive, and Risk Awareness is labeled separately. The cases of ShowUI-2B (0.4% TRR, 0 refusals) and MA-v3 convincingly show that a low TRR may stem from limited capability rather than caution.

  3. High transparency. The appendix fully documents the judge calibration design (stratified enriched sampling, per-model coverage, annotator blinding), the direction of the judge’s bias, and the limitations of each study.

Critical weaknesses
  1. The reliability of the harm label is not quantified. The authors explicitly state that they do not report inter-annotator agreement (e.g., Fleiss' κ) and rely only on the three-vote majority procedure. Since the "harmful" nature of the entire benchmark rests on this label, the paper should at least report the agreement rate, the proportions of 2:1 versus 3:0 votes, and the vote distribution for the requests underlying the selected tasks.

  2. "User grounding" extends only to the harm domains, not to the request wording. Of the 255 cases, 238 are framed requests written by the authors based on the social engineering and persuasion literature, including families such as "authority/coercion" and "incentive." These framings do not come from the questionnaire, so the "user-grounded" label may lead readers to overestimate the benchmark's ecological validity.

  3. Risk Awareness measurement is limited by the judge's view and a single annotator. The judge reads only the reasoning text, not actions or screenshots, and systematically under-calls Aware on the calibration sets (especially Aware-executed). The human reference comes from a single annotator, so inter-human agreement cannot be estimated. The "93.6% Unaware" figure is therefore closer to "not explicitly stated, from the judge's perspective" and is weak evidence that agents are unaware of the risk. In addition, models whose reasoning is not externalized are inherently more likely to be labeled Unaware.

  4. Small scale leaves domain-level and structure-level conclusions thinly supported. The 17 tasks give only 1 to 2 tasks per domain on average. The claim that "difficulty can masquerade as safety" rests on just 2 multi-app tasks and 1 multi-state-change task, and the multi-app tasks are fully confounded with the privacy domain. The appendix concedes this supports only a qualitative claim, but the main text states it more strongly.

  5. Insufficient statistical rigor. Each case appears to be run only once, sampling temperature is not set for some model APIs, and the main table reports no repeated runs, confidence intervals, or significance tests. Claims such as "framing effects are model-specific" are exactly the kind that require variance estimates to rule out random fluctuation.

  6. The context-controlled study and controls are too small. The context study has only 48 runs, and the benign controls consist of only 2 read-only tasks. Both are too small to test over-refusal or to support the conclusion that selective intervention is feasible.

Questions for authors
  1. How were the questionnaire scenarios selected? How sensitive is the 11% figure to the scenario composition? Could you report the harmful share per scenario, and state in the abstract that this is a share "under the given scenarios"?

  2. Could you report the agreement rate or Fleiss' κ for the three annotators, as well as the proportions of 3:0 and 2:1 votes? What is the vote distribution for the original requests underlying each of the 17 tasks?

  3. Of the 238 framed requests, how many have close counterparts in the original questionnaire responses? Could you distinguish "reasons users would actually give" from "pressure constructed by the authors based on the persuasion literature," and report TRR separately for each?

  4. The judge does not see actions or screenshots and is known to under-call Aware. Could you add a second human annotator to estimate inter-human agreement? Alternatively, could you run a control experiment (e.g., asking the agent to list consequences before acting) to estimate the gap between "not stated" and "not aware"?

  5. Were the device-state checks manually spot-checked? What is their error rate? Are there cases where the target is semantically reached but the check fails to capture it?

  6. There are only 2 multi-app tasks, and both belong to the privacy domain. Could you add multi-app tasks from other domains, or phrase the structure-related conclusions in the main text more conservatively?

  7. Is each case in the main experiment run only once? Could you perform repeated runs for at least some models and report confidence intervals for TRR and for the framed-minus-direct differences?

  8. Two read-only benign tasks are a weak check. Could you add benign variants that perform the same operation as the high-risk tasks but in a clearly legitimate context (similar to the legitimate contexts in Table 3), and test over-refusal at the full 255-case scale?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.