- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents MyPhoneBench, a benchmark for evaluating privacy behavior in phone-use agents during benign tasks. It introduces the iMy privacy contract and instrumented Android applications to assess over-permissioning, unnecessary data disclosure, and optional-field filling. Experiments on five frontier models across 10 apps and 300 tasks reveal trade-offs between task success and privacy compliance, with form minimization emerging as the most challenging dimension. The paper also examines cross-session preference reuse and highlights limitations of success-only agent evaluation.
Critical strengths
-
Important and timely research problem. The paper addresses privacy risks in GUI-based phone-use agents, an important yet underexplored setting. Its focus on privacy violations arising during benign task execution, rather than adversarial attacks, provides a valuable perspective on agent privacy evaluation.
-
Fine-grained and verifiable privacy auditing. The paper introduces field-level auditing that captures agents' data-handling behavior during execution, including unnecessary permission requests, repeated disclosures, and optional-field filling. This provides better visibility into privacy violations than conventional task-success-based evaluations.
-
Substantial engineering effort and reproducibility. The authors develop 10 instrumented Android mock applications, integrate a privacy contract and structured privacy probes, and implement deterministic evaluation mechanisms. The resulting benchmark represents a meaningful engineering effort and provides a reproducible foundation for studying privacy behavior in mobile agents.
Critical weaknesses
-
Limited validity of the privacy definition. The benchmark relies heavily on predefined rules, particularly the LOW/HIGH classification and task-necessity labels. While the authors acknowledge that this is a simplified privacy taxonomy, I am still concerned about how these rules are established. Who determines whether certain data should be classified as LOW or HIGH? How is task necessity defined? Without further justification or validation, it is unclear whether the benchmark captures realistic privacy expectations or mainly evaluates compliance with its own predefined rules.
-
Potential bias in privacy evaluation. Privacy scores are calculated only over the probes encountered during execution. This means that agents reaching different stages of a task may be evaluated on different privacy situations. Although PQSR partially addresses the issue of early task failures, it does not fully resolve differences in probe exposure, even among successful agents. I would like to see more analysis of how task progress, UI layout, and probe placement affect the reported privacy scores and model rankings.
-
The analysis lacks depth. Most findings are based on comparisons of scores across models and applications, with limited investigation into why these differences occur. For example, the authors attribute unnecessary optional-field filling to a completion-oriented bias, but this could also result from misunderstanding the UI, failing to recognize optional fields, or following general task-completion instructions. The current experiments demonstrate that these failures occur, but provide limited evidence explaining their causes. I expected more controlled analysis to draw meaningful conclusions beyond model rankings.
-
Missing important baselines. The paper evaluates five frontier models using essentially the same iMy contract and agent framework, but does not explore whether simple privacy-aware interventions could improve their behavior. For example, how would the results change with explicit privacy-aware prompting or a rule-based privacy filter? Without such comparisons, it is difficult to tell whether the observed failures reflect limitations of the models themselves or simply insufficient privacy guidance during execution. These baselines would also help clarify what the benchmark teaches us about improving agent privacy.
-
Unclear connection between cross-session preference reuse and privacy evaluation. I find the inclusion of cross-session preference reuse somewhat difficult to justify within the main scope of this privacy benchmark. The experiments mainly test whether agents can save and correctly reuse user preferences, which seems closer to evaluating memory functionality than privacy protection. An agent could achieve a high score on this metric while still storing information without proper consent or ignoring deletion requests. If user-controlled memory is intended to be a core privacy dimension, I would expect evaluations of consent, deletion, modification, or unauthorized retention. Otherwise, it is unclear why successful preference reuse should be treated as a privacy-related capability alongside over-permissioning, trap resistance, and form minimization.
Questions for authors
Please see the Critical Weaknesses section for specific concerns. I appreciate the authors' effort to make agent privacy measurable through concrete form-filling scenarios. However, this simplification also limits the benchmark's ability to capture the complexity of privacy risks in real-world agent interactions. More importantly, the analysis remains largely descriptive, with limited insight into why privacy failures occur, while privacy evaluation is also heavily intertwined with task completion. Overall, despite the meaningful problem setting and engineering effort, I believe the paper still falls short of the depth and rigor expected of a high-quality benchmark.