acceptodds
Under review as a conference paper at ICLR 2027

PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents

Abstract

Mobile GUI agents operate through a perception–action loop: at each step they screenshot the device, invoke a vision–language model (VLM), and emit an action. It is slow, costly, and brittle, yet most of what it does is navigation—and everyday navigation is static, ordered, and endlessly repeated. We present PhoneCLI, which compiles an app's GUI navigation into callable commands, without any app-internal API, runtime instrumentation, or model training. Offline, PhoneCLI explores a target app from the outside and distills its screens, interactive elements, and navigation edges into a semantically annotated map; each screen yields one deterministic command: a replay sequence that reaches it. Online, the agent selects a command, verifies it before execution, and then executes it deterministically in sub-second time at zero VLM cost; open-ended interaction and every failure of the compiled path fall back to the embedded VLM interpreter, exactly the pure VLM agent, so compilation can only help. On AndroidLab, PhoneCLI improves the task success rate while reducing steps and token consumption, and it transfers to AndroidWorld's official M3A agent with consistent efficiency gains. What PhoneCLI compiles is the app's navigation rather than one run, so it serves new tasks, not only repeated ones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agent, Phone-Use Agent, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BMChOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents PhoneCLI, a framework that compiles the navigation structure of mobile apps into deterministic callable commands. It first explores an app offline to build a semantically annotated screen graph, then converts navigation paths into replayable action sequences. At runtime, an LLM selects and verifies a command, which is executed without additional VLM calls, and tasks not covered by the compiled commands fall back to the standard GUI agent.

Critical strengths

1.Stable and repetitive navigation can be compiled into deterministic commands, which is a natural fit for mobile workflows with fixed navigation paths, such as system settings. This avoids repeatedly invoking the VLM for the same navigation steps.

2.PhoneCLI is model-agnostic and can be attached to different GUI-agent backbones without retraining them.

Critical weaknesses

The scalability of the command catalog remains unclear. The paper already observes a substantial performance drop when the map budget increases from 50 to 100 screens, due to increased routing ambiguity. Since real applications can contain far more screens and navigation paths than the explored depth-3, 50-screen maps, it is unclear how PhoneCLI would scale to substantially larger apps.

Questions for authors

Could the authors clarify how the “guaranteed floor” claim is consistent with the error analysis, where PhoneCLI loses 10 tasks that are solved by the VLM-only baseline due to routing or command-selection errors?

Reviewer_YasZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhoneCLI compiles static navigation paths of mobile apps into deterministic callable commands offline. At runtime, matched tasks are replayed directly, while failures and open-ended interactions gracefully fall back to a pure VLM agent. Experiments show improved success rates with fewer steps and lower token consumption.

Critical strengths
  1. No app APIs, instrumentation, or model training required. New apps can be onboarded in minutes, and the fallback design introduces no new failure modes.
  2. Removes the most token-heavy repeated navigation from the VLM loop. Efficiency gains persist even with strong backbone models, which is highly valuable for real-world deployment.
Critical weaknesses
  1. Offline UI traversal, trajectory replay, and graph-guided navigation are all well-explored directions. The work reads more like an incremental engineering refinement than a conceptual advance, which is thin for ICLR.
  2. No head-to-head comparison with the closest prior work (PreAct, UI-KOBE). Tested apps are mostly simple system utilities; performance on complex third-party apps remains unvalidated.
Questions for authors
  1. Why is there no direct comparison with PreAct and UI-KOBE? This would be the most direct evidence of the paper’s incremental contribution.
  2. Current tests are limited to system utility apps. How would the method perform on third-party apps with deep navigation hierarchies and abundant dynamic elements?
  3. The paper emphasizes compiling app navigation rather than individual tasks to support unseen tasks. Could you walk through a concrete example of how the routing system selects and combines commands for a cross-page task never seen during exploration?
Reviewer_QyDEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies mobile GUI agents. The agent is an VLM with no training. The core idea is to compile an app's navigation into callable commands ahead of time. An breadth-first crawler explores each app (depth 3, at most 50 screens) and records screens, interactive elements and navigation edges. An LLM then labels each element as STABLE or DYNAMIC, assigns aliases and semantic types, and describes each screen. Every screen is compiled into a fixed replay sequence of force_stop, launch and taps at fixed coordinates. Online, a router LLM picks one of four decisions (OP, MACRO_VLM, NEED_VLM, FINISH) from a compressed command catalog, and a second LLM call verifies the selected command. The system then restarts the app, replays the command, checks the landing screen, and passes a landing hint to the VLM. Every other case falls back to a plain VLM agent with memory and stuck detection. On AndroidLab (138 tasks, 9 apps), PhoneCLI raises success from 50.7% to 63.0% with Qwen3.7-Plus, from 41.3% to 44.5% with GLM-4.6V, and from 68.8% to 69.6% with Kimi-K3, with fewer steps and tokens. Attached to the official M3A agent on AndroidWorld, it raises success from 61.2% to 65.5%.

Critical strengths

The problem is practical. Mobile apps expose no programmatic interface and navigation is highly repetitive, so removing it from the VLM loop is a simple and sensible idea. The method needs no training, is model-agnostic, and attaches to M3A without changing its prompts or action space. Token and step reductions appear across three backbones and two benchmarks. The ablations cover component decomposition, individual mechanisms and map size, and the failure analysis and case study show that the main gain comes from locating the task's entry point.

Critical weaknesses

W1. Each task is run once. The +0.8 gain on Kimi-K3 equals one task, and the +3.2 gain on GLM-4.6V equals four or five tasks. The paper claims p<0.05p < 0.05 against the strongest baseline but does not name the test, and a one-task difference from a single run cannot be significant.

W2. The 27 saved and 10 lost tasks relative to VLM-only task. THe noise seems to exist in several tasks. The author also refer to the noise as "single-run VLM variance".

W3. In Table 3, removing any single mechanism costs 12 to 16 points, and some variants score below using no map at all. For example, removing only the landing hint gives 50.0%, below VLM-only at 50.7%. This is hard to explain and is more likely noise.

W4. The 44 query tasks are judged by Qwen3.7-Plus, which is also the evaluated backbone, so it is likely to have bias.

W5. The abstract and main text repeatedly state that replay is "sub-second". The compiled command in Figure 6, however, contains waits of 0.5 s, 3.0 s, 1.5 s and 1.5 s, about 6.5 s in total. The paper motivates the work by the slowness of current agents but reports no wall-clock latency at all. Offline map building takes about ten minutes per app, and the LLM annotation tokens are excluded from the cost comparison.

Questions for authors

Q1. Can the authors run each task several times, report means with intervals, and name the test behind p<0.05p < 0.05?

Q2. How does PhoneCLI compare with AutoDroid, UI-KOBE or PreAct under the same backbone?

Q3. How do the results change with a judge different from the backbone?

Q4. What is the actual wall-clock time per task, including replay waits? What is the total cost once offline map building time and tokens are included?

Comments on the paper

1 comment
Area_Chair_gjuKOpenReview ↗

Reminder: Please Review Author Rebuttals and Participate in Discussions

Dear Reviewers,

Thank you for your time on the reviews. The authors have now submitted their rebuttals, and I encourage you to read their responses and participate in the discussion.

Please consider whether the rebuttals address your concerns and, where clarification is still needed, engage with the authors through the discussion forum. I also encourage you to exchange perspectives with your fellow reviewers and update your review or rating if appropriate.

Your participation will help ensure a thorough and fair evaluation of each paper. Thank you again for your time and effort.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.