acceptodds
Under review as a conference paper at ICLR 2027

Do Mobile Agents Need the Phone Screen?

Abstract

Recent advances in mobile agents are dominated by the GUI paradigm, in which agents perceive UI information and emit screen interactions. However, mobile platforms also expose a command-line interface (CLI) with direct access to device state and services, so we study whether mobile agents can complete tasks without UI information. We evaluate three off-the-shelf coding agents across four model APIs on AndroidWorld and MobileWorld against two mobile-post-trained GUI specialists and two generalists, reproduced under a unified protocol, and construct hand-verified oracle solutions. We find that of AndroidWorld and of MobileWorld tasks are solvable without UI information, and every remaining task involves media or UI-only state, demonstrating that most benchmark tasks do not require the GUI. To separate interface from model, we fix the backbone at Opus 4.7: on CLI-solvable tasks, Claude Code matches its GUI counterpart ( vs. and vs. ), so the GUI agent leads only on GUI-only tasks. Without mobile post-training, Claude Code beats every reproduced GUI specialist on both benchmarks ( and ), and MobileCLI, our harness tuned on MobileWorld, lifts that backbone from to on held-out AndroidWorld's CLI-solvable tasks. Across an OS upgrade from Android 13 to 17, CLI agents change by at most points, while the two GUI specialists drop and and the Opus 4.7 GUI agent only , suggesting that mobile post-training fits screens rather than tasks. For everyday intents the touchscreen was not designed to serve, we introduce the Beyond-GUI Intent Suite: 45 templates in five categories. Every CLI agent outperforms every GUI baseline in all five. The CLI is thus a first-class complement to the GUI, and the interface should be chosen per task. We release all agents, oracles, the suite, and infrastructure at https://anonymous.4open.science/r/CLI_Agent-47D8.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.