- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the misuse risk of phone-use agents on social media. The key observation is that, by operating apps through native GUIs, these agents can access functions platforms rarely expose via public APIs (reporting, danmaku, direct messaging)—abuse that traditionally relied on human labor and that agents could make far cheaper and more scalable. Existing benchmarks either use mock apps lacking such functions or cover only narrow abuse scenarios. The paper makes two contributions. BadPhoneAgent is a high-fidelity evaluation environment spanning 11 Chinese/English social apps and 18 abuse-relevant functions, with 150 policy-grounded test cases (8 categories, 35 subcategories); a semi-automated framework replicates real apps on a lightweight virtual platform, enabling large-scale repeated evaluation without real accounts and direct task-success checks from app state. VEIL (View Editing for Intent Laundering) is an attack exploiting a structural flaw: agents judge safety from what they observe, yet act on the true phone state. Controlling the local framework, the attacker modifies the model’s inputs (instruction, history, screenshot) without changing app state by (a) splitting a harmful request into individually benign subtasks and clearing history between them, and (b) using SIFT/template matching (no model inference) to mask screenshot regions linking the subtasks. Main findings: substantial risk exists even without attacks (GPT-6-Astra completes 52% of harmful tasks while refusing only 44%; several open-source models refuse none); VEIL raises GPT-6-Astra’s ASR from 52% to 88% and Qwen3.8-27B to 94.7%—surpassing all commercial models at 1.46× human speed—while dropping refusal rates sharply (Claude Fable 5 from 44.7% to 1.3%); real-device validation shows 91.5% virtual–physical agreement (κ=0.80) with VEIL still effective; ablations confirm all three components cumulatively raise ASR, supported by an information-theoretic argument; and two existing defenses, effective on direct requests, are weakened by VEIL.
Critical strengths
- The attack mechanism captures a real and fundamental architectural flaw. The observation–action gap, where the agent judges safety based on what it observes yet acts on the true underlying state, is an attack surface specific to phone-use agents and conceptually very clean.
- Real-device validation is the biggest plus. 600 virtual–real device pairs, 91.5% agreement, and κ=0.80 strongly address the question most often raised against benchmark papers, namely whether the virtual environment reflects real-world risks.
- The attack components are cleverly designed and low-cost. Screenshot matching uses SIFT+RANSAC / normalized cross-correlation, runs on pure CPU in tens of milliseconds with no model inference required, and the paper deliberately argues that a masked screenshot is not itself a sign of an attack (cloud-based MLLMs already routinely mask for privacy). This makes the attack both practical and hard to defend against by merely “detecting that an edit occurred.”
Critical weaknesses
-
Insufficient defense exploration weakens the constructive value. Only two off-the-shelf defenses (a safety reminder and OS-Sentinel) are tested, and both fail under VEIL. The paper proposes no new defense idea targeting the observation–action gap, not even conceptual ones (e.g., app-side state verification, cross-subtask consistency checks, action–observation consistency validation). An attack paper claiming to "promote safety" should at least outline viable defensive directions.
-
Attack novelty may be a combination of known techniques. Task decomposition plus multi-turn history isolation is essentially close to existing jailbreak ideas such as Crescendo; the genuinely new element is "screenshot masking" as visual-channel sanitization. Reviewers may see the core contribution as a framing of the observation–action gap rather than a fundamentally new attack primitive. The incremental novelty needs to be argued more clearly.
-
On-device ASR is "attempted ASR," which risks overstatement. To prevent real harm, the on-device experiments intercept actions without actually submitting them, so on-device ASR measures whether the agent attempts execution, not whether the operation takes effect. Platform-side rate limiting, CAPTCHAs, and risk controls are not evaluated at all (the authors concede in A.1 that no forced logouts or CAPTCHAs were encountered during testing, so they were not assessed). The "feasibility of abuse at scale" conclusion therefore skips the interception stage most likely to occur in real deployment.
-
Weak rigor in the human speed baseline. Only 3 volunteers, with each task timed only on the second (post-familiarization) run—a tiny sample with no reported variance. Figures such as "1.46× the human baseline" rest on this and are not very convincing.
-
Room for improvement in the judge and statistical rigor. Task success is determined by scripts from app state (a good choice), but refusal judgment relies on humans plus DeepSeek-V4-Flash (97.7% agreement, but no κ and no disagreement analysis).
Questions for authors
-
Have you explored any defense that is more principled with respect to the observation–action gap? For example, app/OS-side action–intent consistency verification, cross-subtask state-correlation checks, or requiring safety judgments to be based on the true app state rather than the view the model observes? Even a conceptual discussion would substantially strengthen the contribution.
-
On-device ASR is "attempted" (actions are intercepted and never actually submitted). If actions were truly submitted, to what extent would platform-side risk controls (rate limiting, CAPTCHAs, account bans, report-validity review) reduce the actual abuse success rate? Could you provide even a small-scale evaluation that actually reaches the platform?
-
Compared with Crescendo (multi-turn gradual escalation) and general task-decomposition jailbreaks, is VEIL's core increment simply "sanitization of the screenshot visual channel"? Could you quantify this more clearly through ablation—specifically, how large is the gap from existing multi-turn jailbreak methods when screenshot editing is removed and only decomposition + history isolation are used?
-
Is the 0% refusal rate of GUI-Owl / Qwen3-VL because these specialized GUI models have no safety alignment at all, or because the harness/prompt format fails to trigger their safety behavior? This bears on whether the conclusion should be attributed to the "model" or to the "evaluation setup."
-
Can a design with 3 volunteers and single-run timing support the speed conclusions? Could you add more participants and report variance, or at least state the speed claims more conservatively?
-
The 97.7% agreement between humans and DeepSeek-V4-Flash was measured on how many samples and which configurations? What is κ? Do disagreement cases concentrate in a particular task type?
-
Once the code and data are released, could these 150 policy-aligned cases be used directly for real abuse, or be absorbed into model training data and thereby lose their validity? Are there plans for access control or a gated release?
6 comments
Responses (2/3)
- Q3 and weakness2. Could you quantify this more clearly through ablation, specifically how large the gap from existing multi-turn jailbreak methods is when screenshot editing is removed and only decomposition and history isolation are used?
We compare our decomposition and history isolation with Crescendo (a multi-turn gradual-escalation attack) and DrAttack (a task-decomposition jailbreak). The results are shown below.
| Model | Baseline | Crescendo | DrAttack | Ours (without screenshot editing) |
|---|---|---|---|---|
| GPT-6-Astra | 52.0% | 52.7% (+0.7) | 65.3% (+13.3) | 84.7% (+32.7) |
| Qwen3.8-27B | 67.3% | 70.0% (+2.7) | 75.3% (+8.0) | 91.3% (+24.0) |
Even without screenshot editing, our decomposition with history isolation clearly outperforms Crescendo and DrAttack. This shows that VEIL's gains do not come only from sanitizing the screenshot visual channel.
- Additional clarification on Weakness2. About the novelty of our work
We would like to clarify that our novelty lies in benchmarking an overlooked real-world misuse risk and jailbreaking frontier models through an underexplored attack surface.
Existing benchmarks lack apps and functions needed to evaluate social media misuse by phone-use agents, such as reporting. Our benchmark covers them through a high-fidelity simulator and 150 test cases manually built from platform policies. With it, we find substantial risks even without attacks, with GPT-6-Astra and Claude Fable 5 completing 52.0% and 42.7% of harmful tasks.
On the attack side, we identify and exploit an underexplored attack surface in phone-use agents, namely the observation-state gap. Exploiting this gap enables effective attacks on models with stronger safeguards, which task decomposition and history isolation alone cannot achieve. As shown in the General Response, on GPT-6-Astra with a safety prompt, decomposition with history isolation raises ASR by 17%, whereas adding screenshot editing adds 43.3%.
- Q4. Is the 0% refusal rate of GUI-Owl / Qwen3-VL because these specialized GUI models have no safety alignment at all, or because the harness/prompt format fails to trigger their safety behavior? This bears on whether the conclusion should be attributed to the "model" or to the "evaluation setup."
Thank you for this insightful question. Our additional experiments show that both the evaluation setup and the model contribute to the 0% refusal rate.
We evaluate two configurations. In S1, the default setting in the paper, each model runs with its official system prompt [1,2] for phone-use tasks. In S2, we remove the GUI-related system prompt and send each harmful request as a plain conversational query.
| Model | Refusal rate (S1) | Refusal rate (S2) |
|---|---|---|
| GUI-Owl-1.5-8B | 0.0% | 41.3% |
| Qwen3-VL-8B | 0.0% | 78.0% |
Without the GUI system prompt, the refusal rates of GUI-Owl and Qwen3-VL rise to 41.3% and 78.0%, respectively. This indicates that the phone-use prompt format affects safety behavior.
Under S2, GUI-Owl also refuses far less often than Qwen3-VL. A possible reason is that GUI-Owl is fine-tuned from Qwen3-VL on GUI data [2], and prior work shows that fine-tuning even on benign data can degrade safety alignment [3]. We have added this analysis to the revised manuscript.
[1] Qwen3-VL Technical Report
[2] Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents
[3] Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
Responses (3/3)
- Q5 and weakness4. Can a design with 3 volunteers and single-run timing support the speed conclusions? Could you add more participants and report variance, or at least state the speed claims more conservatively?
Thank you for this valuable suggestion. We extended the study to five volunteers, each performing every task three times. The first run is used to familiarize the volunteer with the task, and the last two are timed. The resulting human execution time is 42.57 ± 4.1 seconds per task (mean and standard deviation across the ten runs).
We have also stated the speed claims more conservatively. For example, instead of comparing the agent's speed with humans, we now report the raw execution times for reference to avoid potential misunderstandings.
- Q6 and weakness5. The 97.7% agreement between humans and DeepSeek-V4-Flash was measured on how many samples and which configurations? What is κ? Do disagreement cases concentrate in a particular task type?
The 97.7% agreement is measured on all runs that the evaluation scripts classify as unsuccessful attacks (2,438 in total) across every experiment in the paper. Human labels were produced by two annotators working independently, who agreed on 98.9% of runs, and a third reviewer resolved the remaining disagreements.
Cohen's κ between DeepSeek-V4-Flash and the human labels is 0.92.
The disagreements do not concentrate in a single category. Across Harassment, Fraud and Scams, Illicit Transactions, Hate and Extremism, Dangerous Behavior, Privacy Violations, Platform Manipulation, and Misinformation, the numbers of disagreements are 12, 11, 4, 6, 5, 4, 10, and 4, respectively.
- Additional clarification on Weakness5. Human annotation of refusal rates.
All refusal rates reported in the paper come from the human annotation described above, not from DeepSeek-V4-Flash (in Section 3.3). We report the DeepSeek-V4-Flash results only as a reference for future work, to show how accurate an LLM judge would be for researchers who cannot run human annotation.
- Q7. About the release of code and dataset.
Thank you for this valuable suggestion. We plan to make the simulator publicly accessible. For the jailbreak samples, we follow common practice in safety research and release them only to applicants who agree to use them for research purposes and not to train models that are evaluated on this benchmark.
- On presentation.
We thank the reviewers for their feedback on presentation. We have fixed the noted typos and inconsistent citation styles, and we will carefully polish the writing throughout the paper to improve clarity and readability in the revised manuscript.
Responses (1/3)
We thank Reviewer GBbj for the thorough review and the constructive suggestions. We respond to each question in detail below.
Thank you for this valuable and constructive suggestion. We feed the judge model GPT-6-Sol the agent's full execution trajectory, including every action (e.g., tapping and typing) together with the corresponding app state as unedited screenshots, and ask it whether the task is harmful. This simulates a cross-subtask state check on the app/OS side. On VEIL executions with Qwen3.8-27B (ASR=94.7%), the judge identifies 63.3% of all 150 task executions as harmful, suggesting that app-side state verification and cross-subtask consistency checks are promising directions.
However, such defenses still face practical challenges. Subtasks issued across time need to be linked before they can be checked together, and social media platforms handle an enormous volume of user activity, which makes judging every trajectory costly. Evaluating app-side and system-level defenses is beyond the scope of this work, but we have added this discussion to the revised manuscript and hope it inspires future work.
Thank you for this valuable suggestion. We conducted an additional real-device evaluation with actual submission on 137 reversible tasks under VEIL with Qwen3.8-27B (excluding 13 irreversible tasks, such as reporting). All content submitted by the agent in this evaluation was successfully published and visible to another account, without being blocked or removed by the platforms. We have added these results and discussion to the revised manuscript and hope they inspire future work.
We would also like to clarify that by "at scale" we mean automating the GUI interactions that such abuse has so far relied on human labor to perform. Our focus is on the extent to which phone-use agents reduce this labor, with their limited safety alignment raising concerns about misuse. Rate limiting and account bans also constrain abuse carried out by humans. Evaluating these controls is beyond the scope of our work. To avoid potential misunderstanding, we have revised the related claims in the manuscript to the "potential for automated misuse in real-world apps" and discuss these factors in the limitations.