acceptodds
Under review as a conference paper at ICLR 2027

Do Mobile Agents Need the Phone Screen?

13 views

Abstract

Recent advances in mobile agents are dominated by the GUI paradigm, in which agents perceive UI information and emit screen interactions. However, mobile platforms also expose a command-line interface (CLI) with direct access to device state and services, so we study whether mobile agents can complete tasks without UI information. We evaluate three off-the-shelf coding agents across four model APIs on AndroidWorld and MobileWorld against two mobile-post-trained GUI specialists and two generalists, reproduced under a unified protocol, and construct hand-verified oracle solutions. We find that of AndroidWorld and of MobileWorld tasks are solvable without UI information, and every remaining task involves media or UI-only state, demonstrating that most benchmark tasks do not require the GUI. To separate interface from model, we fix the backbone at Opus 4.7: on CLI-solvable tasks, Claude Code matches its GUI counterpart ( vs. and vs. ), so the GUI agent leads only on GUI-only tasks. Without mobile post-training, Claude Code beats every reproduced GUI specialist on both benchmarks ( and ), and MobileCLI, our harness tuned on MobileWorld, lifts that backbone from to on held-out AndroidWorld's CLI-solvable tasks. Across an OS upgrade from Android 13 to 17, CLI agents change by at most points, while the two GUI specialists drop and and the Opus 4.7 GUI agent only , suggesting that mobile post-training fits screens rather than tasks. For everyday intents the touchscreen was not designed to serve, we introduce the Beyond-GUI Intent Suite: 45 templates in five categories. Every CLI agent outperforms every GUI baseline in all five. The CLI is thus a first-class complement to the GUI, and the interface should be chosen per task. We release all agents, oracles, the suite, and infrastructure at https://anonymous.4open.science/r/CLI_Agent-47D8.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agent, Mobile device control, CLI vs GUI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XGfxOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper asks a question: whether mobile use agents have to be based on GUI models.

The authors use state-of-the-art LLMs (MiniMax M2.7, Opus4.7, Sonnet, GPT) with coding agent harnesses to prove that, in AndroidWorld and MobileWorld, such coding agents can complete most of the tasks. And they have shown better generalization capabilities.

Critical strengths
  1. Good performance on Current Benchmarks: Shows that 88.8% of AndroidWorld and 86.3% of MobileWorld tasks can be solved via ADB commands without screen perception. The models are also off-the-shelf and zero-shot: no fine-tuning is performed for mobile use.
Critical weaknesses
  1. Unrealistic Root/System-Level Privilege Assumptions: The approach relies on privileged ADB and root access in emulators, which is prohibited on standard unrooted consumer phones and makes real-world deployment largely impractical.

  2. Reliance on Open-Source Apps and Exposed Schemas: Benchmark apps in AndroidWorld and MobileWorld provide full source code and accessible backend states, whereas real-world commercial apps are closed-source, obfuscated, and strictly sandboxed. The proposed method can in no way operate on real apps. The authors also do not provide such experiment details.

  3. Engineering Wrapper Rather Than Core ML Research: Adapting off-the-shelf coding agents via system prompts and bash tool wrappers is primarily a system engineering effort rather than a fundamental algorithmic or research contribution.

  4. Inability to Handle Visual or Closed-UI Tasks: Pure CLI agents are completely blind to tasks requiring visual media understanding or internal UI states that are not written to local databases.

  5. The comparison is not fair. All GUI models compared are models with smaller sizes. Whereas the coding LLMs are large models, even state-of-the-art proprietary models.

  6. There is no data on the time delay caused by ADB commands.

Questions for authors

Please address the weaknesses.

  1. Can you benchmark the GUI agents with smaller-sized LLMs using coding agent harnesses (e.g., Qwen3.8-27B / Qwen3.5-9B)?

  2. Can you provide ADB command time-delay data?

1 comment
Reviewer_XGfxOpenReview ↗

thanks for your response

thanks for your effort. however, my concerns still remain without real-device experiments. I will update my rating to 2: weak rejection.

Reviewer_AqjEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether mobile agents can complete Android tasks through a command-line interface (CLI) without observing screenshots or other UI information. The authors compare several off-the-shelf coding agents with GUI agents on AndroidWorld and MobileWorld, construct hand-verified oracle solutions, and show that existing CLI agents can successfully complete a substantial proportion of the tasks. They further introduce MobileCLI, a purpose-built harness tuned on MobileWorld and evaluated on AndroidWorld, and conduct cross-version evaluation across OS and app upgrades, oracle replay, and trajectory-level error analysis. The paper also introduces the Beyond-GUI Intent Suite, comprising 45 task templates covering bulk operations, multi-condition filtering, aggregation and top-K, cross-app workflows, and hidden device state. The results show that most tasks in the evaluated benchmarks can be completed without UI information, and that CLI systems are competitive and offer advantages on certain tasks and under version changes. The authors therefore advocate treating CLI as an important complement to GUI.

Critical strengths
  1. The research question is important. Mobile-agent research has largely centered on screen observation and GUI actions. Through oracle solutions and agent evaluations on two existing benchmarks, this paper quantifies how many mobile tasks can be completed without UI information under the evaluated access conditions. This helps clarify the capabilities measured by existing benchmarks and provides empirical evidence for expanding the execution interfaces available to mobile agents.

  2. The experimental evaluation is comprehensive. The paper covers two benchmarks, multiple models and harnesses, and reproduced GUI baselines, providing an extensive and thorough evaluation. The matched-backbone comparison with Opus 4.7 controls for model identity. Evaluating MobileCLI on AndroidWorld after developing it on MobileWorld provides evidence of cross-benchmark transfer. The cross-version experiments and model-free oracle replay provide complementary evidence of robustness.

Critical weaknesses
  1. Task-level analysis across experiments remains insufficient. The paper discusses set-level operations, cross-app workflows, and hidden device state on the Beyond-GUI Intent Suite, and demonstrates the viability of execution without UI information and robustness to version changes on existing benchmarks. However, these findings are not sufficiently connected through a unified analysis of task attributes on the two existing benchmarks. For example, which task attributes are associated with differences in CLI and GUI success, and do these patterns hold across benchmarks? The current analysis explains fairly well what CLI agents can do, but offers only preliminary insight into when CLI execution is preferable. It is reasonable for the authors to leave systematic interface routing to future work. Nevertheless, analyzing the existing trajectories along these dimensions could further strengthen the paper's insights for researchers and developers.

  2. Differences in access privileges and harness design limit the attribution of performance differences. As acknowledged in Appendix and Limitations, CLI configurations can access app-private state, and MobileWorld additionally provides infrastructure-level access to backend databases and containers. GUI configurations primarily operate through the user-visible interface. Even with the Opus backbone held fixed, access to state, operational privileges, and harness design still vary together. The on-device versus backend-assisted task split in Table 10 is informative, but the two groups contain different tasks and therefore do not quantify how performance on the same task changes when additional access is removed. The results therefore offer limited insight into whether their advantages arise from the I/O interface, additional access, or harness design, although they indeed support the competitiveness of the evaluated CLI systems.

Questions for authors
  1. According to Appendix B.2, Cache-match requires every expected substring to appear in the final answer, but does not describe how incorrect extra items are excluded. For the task of listing expenses above average spending, listing all expenses could also satisfy this condition. Meanwhile, the State-check deletion example only requires the number of expenses below 100 cents to be zero. Under this description, deleting all expenses could also pass, although non-target records should be preserved. Does the actual Cache-match implementation reject incorrect extra items, and does the deletion verifier also check that all non-target records are preserved? If additional checks are already implemented, please describe them. Otherwise, please report how many existing trajectories may be incorrectly scored as successful and how adding these checks changes success rates in Table 3 and the main conclusions. The completeness of these checks directly affects the reliability of Table 3.
Reviewer_GvDbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a question whether mobile agents need to see the screen to complete Android tasks. The authors compare CLI coding agents with GUI agents on AndroidWorld and MobileWorld, using shared evaluation criteria and manually verified reference solutions. They find that 88.8% and 86.3% of tasks can be solved without UI information. With Opus 4.7, CLI and GUI agents perform similarly on these tasks. The paper also introduces a MobileCLI harness and the Beyond-GUI Intent Suite with 45 templates. Experiments show improved performance with MobileCLI.

Critical strengths

The paper shows that most tasks in both benchmarks can be solved without seeing the screen. The hand-checked solutions and comparison using the same model make this finding convincing.

The new task suite covers useful things that current benchmarks miss, like bulk operations and hidden device state. It helps explain where CLI agents are actually useful.

Critical weaknesses

CLI agents get broader access to app data and backend services. The paper discusses this, but even the on-device comparison retains elevated privileges. This makes it harder to separate the benefits of CLI from those of broader access.

The Android upgrade results are interesting, but the models differ in more than their training. More evidence is needed to say that mobile post-training explains the GUI specialists’ performance drops.

Questions for authors

How many of the on-device tasks can be solved with non-root ADB access? Does the CLI advantage still hold on that subset?

For tasks solved by directly editing app databases, did you check that the changes appear correctly in the app and that unrelated data remain intact?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.