acceptodds
Under review as a conference paper at ICLR 2027

MobiCLI: Benchmarking CLI and Hybrid GUI-CLI Interaction for Mobile Agents

Abstract

Mobile agents have largely been developed and evaluated through GUI-only interaction, overlooking command-line interfaces (CLIs) that can improve execution efficiency and support operations beyond GUI actions. To bridge this gap, we introduce MobiCLI, a benchmark for evaluating CLI and hybrid GUI–CLI interaction on mobile devices. We identify five representative mobile scenarios where CLI capabilities are particularly useful: Batch Operations, File Management, Computational Tasks, Data Processing, and System Operations, and construct 122 tasks across 18 applications. We evaluate strong general-purpose models and representative phone-use agents on MobiCLI, with the best model achieving only 55.7% success. Our experiments reveal three key findings. First, CLI access substantially improves both success and efficiency for strong general-purpose models, exposing the limitations of GUI-only interaction. Second, existing phone-use agents generally fail to effectively exploit CLI capabilities. Third, hybrid tasks are substantially harder than CLI tasks, highlighting the challenge of GUI–CLI coordination. To improve hybrid execution, we further introduce MobiMux, a mobile agent framework that combines task verification with continual experience learning. MobiMux improves success rates by up to 16.4 percentage points on general-purpose models and achieves the best overall performance on MobiCLI.

Primary area:
datasets and benchmarks
Keywords:
Mobile Agent, Benchmark, GUI-CLI Interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VWvkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MobiCLI is a benchmark for mobile agents that can use both GUI actions and shell commands. It contains 122 tasks across 18 apps, including 63 tasks solvable through CLI alone and 59 that still require GUI interaction. The authors evaluate eight models with and without CLI access and find large gains for strong general models, while existing phone-use agents benefit little. They also introduce MobiMux, which checks task completion and reuses experience collected on training tasks. With Kimi K2.6, it raises success from 43.4% to 57.4%. Ablations examine the contributions of verification and experience

Critical strengths

The benchmark includes tasks solvable entirely through the shell and tasks that still need GUI actions. This gives a useful setting for studying how mobile agents combine the two interfaces.

The paired evaluations show a large benefit from adding shell access and interface documentation. Testing eight models also reveals clear differences in how effectively agents use these capabilities, making the benchmark useful for further development.

Critical weaknesses

The experience pool is frozen during testing, but Section 5.1 gives little detail about the training tasks. It is unclear whether their templates overlap with MobiCLI, so the amount of genuine transfer remains uncertain.

Hybrid tasks have lower success and longer trajectories, but the two subsets are not matched for complexity. This makes it hard to tell how much of the gap comes from interface coordination, which limits that interpretation of the results.

Questions for authors

How were the MobiMux training tasks separated from evaluation tasks, particularly at the template level?

Can you separate interface coordination errors from general planning and step-budget failures in the hybrid-task results?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We thank the reviewer for the positive assessment. We appreciate the recognition of MobiCLI's coverage of both CLI and Hybrid tasks, as well as the value of our paired evaluations. We address the reviewer's concerns with additional experimental evidence and clarifications below.

Training–Evaluation Separation (W1, Q1)

We clarify that MobiMux uses a separate set of training tasks to construct its experience pool, with isolated namespaces and task-specific assets to prevent instance-level leakage. More importantly,we introduce structural changes across several dimensions, including input structure, task composition, boundary conditions, and GUI–CLI switching points. Their task structures and execution requirements are deliberately varied to encourage transferable experience rather than template memorization.

To further assess potential overlap, we compute the nearest-neighbor similarity between training and evaluation tasks using two complementary metrics. The median semantic similarity is 0.565 using all-MiniLM-L6-v2. Additionally, the median lexical similarity is only 0.106, measured using TF-IDF with word unigrams and bigrams. These results indicate limited semantic and lexical overlap.

We additionally compare MobiMux with representative experience-learning and skill-reuse methods using Seed2.0 Pro:

MethodSR (%) ↑
Vanilla26.2
+ ReasoningBank [1]30.3
+ AWM [2]32.8
+ AutoRPA [3]27.9
+ SkillDroid [4]20.5
+ MobiMux (Ours)42.6

MobiMux outperforms all baselines. Notably, AutoRPA [3] and SkillDroid [4] primarily focus on reusing executable procedures for recurring task patterns, whereas MobiMux extracts reusable knowledge at multiple levels. These results further support the effectiveness of MobiMux in transferring accumulated experience across different tasks.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Coordination Errors (W2, Q2)

To better disentangle these factors, we conduct three additional analyses.

(1) Controlling for execution length. To account for differences in task complexity, we use the median number of steps across successful executions by general-purpose models as a proxy for each task's reference solution length. We then group tasks by reference length and compare CLI and Hybrid success rates within comparable length ranges. After stratification, the Hybrid tasks' success rate remains only 0.41 times that of CLI tasks, suggesting that differences in execution length alone do not fully explain the performance gap.

(2) Hybrid-task decomposition. To further isolate the challenges introduced by integrating GUI and CLI operations, we construct two simplified variants for each Hybrid task: (B) CLI-only subtask, where the agent performs only the CLI-related operations and reports the results in its final response without interacting with the app; and (C) GUI-only subtask, where the agent is directly provided with the correctly processed results and only needs to perform the remaining GUI operations. We compare these variants against (A) the original end-to-end Hybrid task.

ModelCLI-only subtask (B)GUI-only subtask (C)B × COriginal (A)A − B × C
Kimi K2.60.900.490.460.29−0.17
Seed2.0 Pro0.630.440.300.17−0.13

Here, B × C is computed per task and then averaged, measuring the fraction of tasks for which both independently evaluated subtasks succeed. Notably, Kimi K2.6 achieves 46% joint subtask success but only 29% end-to-end success, while Seed2.0 Pro drops from 30% to 17%. These gaps demonstrate that successfully executing individual GUI and CLI components does not necessarily translate into successful end-to-end execution. Integrating the two interfaces introduces additional challenges.

(3) Coordination-specific failure analysis. We further inspect failed Hybrid trajectories using three general-purpose models with different performance levels and one phone-use agent. We identify coordination errors involving (i) inappropriate interface selection for a sub-operation or (ii) failure to recognize or utilize states produced through the other interface.

ModelCoordination Errors / Failed Hybrid Tasks
Claude Opus 4.850.0%
Seed2.0 Pro40.8%
Qwen3.5-35B-A3B42.4%
GUI-Owl-1.5-32B-Think55.9%

Coordination errors are observed in 40.8%–55.9% of failed Hybrid trajectories, providing direct evidence that interface coordination is a substantial challenge.

Finally, we clarify that task-type labels (CLI or Hybrid) are not disclosed to agents during evaluation. Thus, agents must autonomously determine the appropriate interface for each operation throughout the entire benchmark, including CLI tasks. Hybrid tasks additionally evaluate the ability to integrate operations and maintain state consistency across interfaces.

[1] Ouyang S, Yan J, Hsu I, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory[C]//International Conference on Learning Representations. 2026, 2026: 94327-94354.

[2] Wang Z Z, Mao J, Fried D, et al. Agent Workflow Memory[C]//Proceedings of the 42nd International Conference on Machine Learning. PMLR, 2025, 267: 63897-63911.

[3] Chen M, Hu X, Yu Z, et al. AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions[C]//Proceedings of the 43rd International Conference on Machine Learning. PMLR, 2026, 306: 14562-14588.

[4] Chen Q, Bellucci A, Sun Z, et al. SkillDroid: Compile Once, Reuse Forever[J]. arXiv preprint arXiv:2604.14872, 2026.

Dear Reviewer,

Thank you for your time and effort in reviewing our paper. We have provided a detailed response to your comments. As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. Please let us know if any questions remain.

Thank you again for your valuable time and feedback.

Dear Reviewer VWvk,

We apologize for reaching out again. We sincerely appreciate your thoughtful feedback and have taken your concerns very seriously. We have carefully addressed your comments and conducted additional experiments to provide further evidence.

As the discussion period is approaching its end, we would be truly grateful if you could find a moment to review our responses.

Thank you very much for your time and effort in reviewing our work. We sincerely appreciate your consideration.

Reviewer_Bmc8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark for evaluating mobile agents beyond GUI-only actions by incorporating CLI operations. The benchmark consists of 122 tasks across 18 applications. The authors conduct experiments on both major phone use agents and major general-purpose models and report their findings. They also propose a method called MobiMux, which adds an extra layer of verification before answer submission and incorporates a mechanism to extract, store, and reuse existing experiments at different levels.

Overall, this is a well-motivated paper that examines mobile control from a new angle by extending GUI-only actions with CLI options. However, its scientific rigor needs improvement, and the MobiMux framework offers limited novelty, as its underlying ideas have been extensively studied recently. I recommend a weak reject.

Critical strengths
  • The motivation is clear and compelling: current phone use agents are mostly GUI-based, but CLI is also a powerful tool that should be incorporated, studied, and evaluated given recent advances in LLMs/Agents. The entire pipeline and process for building the benchmark around this motivation are clearly defined and easy to follow, and the experimental design is consistent with the observations and proposed methods.
  • The results are sound and well analyzed. The findings and analysis align with recent industrial observations, providing directions for future work, such as optimizing hybrid mode usage of GUI + CLI.
Critical weaknesses
  • Given the extensive data and taxonomy in this benchmark paper, the taxonomy, scenario selection, and related aspects require more detailed definitions and explanations to clarify the scientific basis for their selection and definition. Examples include the 5 scenarios for shell usage, app categories, and failure taxonomy: how they are selected and vetted to ensure it's not biased and have decent distribution, etc.
Questions for authors

How will CLI be provided/implemented at a larger scale? The process appears to be manual, which would clearly become a bottleneck when scaling to more apps/platforms.

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of the motivation, the benchmark construction process, and the sound experimental analyses. We provide further clarifications below.

Scientific Basis (Summary, Weakness)

Our benchmark construction follows the common practices adopted in established mobile benchmarks. Specifically, we explain the rationale as follows:

(1) Scenario selection: Our scenario selection follows a widely adopted practice in existing mobile-agent benchmarks [1, 2], where task scenarios are identified based on practical use cases and observed limitations of existing benchmarks (§3.2). Following this methodology, we identify representative scenarios in which CLI provides complementary capabilities to GUI interaction.

(2) Application selection: We select applications based on two primary considerations (§3.2): coverage of common everyday mobile activities, and reproducibility through offline execution and deterministic state verification. Our benchmark covers the major application domains represented in existing mobile benchmarks [1] [2] [3], and the distribution is relatively balanced (Figure 3).

Beyond these considerations, MobiCLI additionally emphasizes interface diversity to evaluate different forms of CLI and GUI–CLI interaction. Specifically, the selected applications support heterogeneous interfaces, including local file systems (e.g., Markor and Obsidian), application APIs (e.g., Mattermost and Mastodon), and Android system interfaces (e.g., Calendar, Contacts, and Settings).

(3) Failure Taxonomy: Unlike the predefined task scenarios, we derive our failure taxonomy from recurring patterns observed in failed agent trajectories [1] [2]. These categories capture distinct failure mechanisms rather than exhaustively enumerate all possible errors. Each category is defined with explicit criteria to ensure consistent failure attribution. We assign each failed trajectory a primary failure cause for analysis.

CLI Scalability (Question)

We clarify that MobiCLI does not require manually implementing a dedicated CLI for every application or task. Instead, it provides agents with a shared shell execution interface (run_shell) that exposes existing tools and interfaces, as detailed below.

(1) General-purpose shell utilities, including standard Unix commands, Python, and curl, are installed in the environment and shared across all tasks without application-specific implementation.

(2) Existing application interfaces can be accessed directly, such as file-based applications (e.g., Markor and Obsidian) through standard filesystem commands, and services with REST APIs (e.g., Mattermost and Mastodon) through curl.

(3) Android-native services such as Calendar, Contacts, and SMS, are supported through lightweight wrappers that expose existing Android capabilities through a consistent CLI interface. These wrappers are reusable across applications and tasks, rather than being customized for benchmark tasks.

Overall, MobiCLI primarily relies on reusable tools and underlying interfaces rather than application-specific CLI implementations. Therefore, extending MobiCLI to additional applications does not require extensive manual implementation and is unlikely to become a scalability bottleneck.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

MobiMux Contribution (Summary)

MobiMux introduces a structured experience-learning framework tailored to the challenges of mobile GUI–CLI execution. Unlike general-purpose experience-learning methods [4] [5], MobiMux explicitly organizes reusable knowledge by its scope and intended use: General experience captures transferable strategies, App experience provides application-specific guidance, CLI experience captures tool-specific knowledge, and Verify experience supports task-completion checking. App and CLI experiences are selectively loaded as skills, while verification experiences are maintained separately for the Verifier.

MobiMux also connects execution, verification, and experience learning: the Verifier detects incomplete execution and requests corrections before submission, while its feedback also informs the Curator's extraction of reusable experiences.

Moreover, MobiMux substantially outperforms existing baselines. We compare MobiMux with representative experience-learning and reuse methods, including ReasoningBank [4], AWM [5], AutoRPA [6], and SkillDroid [7], covering different approaches to extracting, organizing, and reusing knowledge from past agent interactions.

MethodSR (%) ↑
Seed2.0 Pro26.2
+ ReasoningBank [4]30.3
+ AWM [5]32.8
+ AutoRPA [6]27.9
+ SkillDroid [7]20.5
+ MobiMux (Ours)42.6

MobiMux achieves the highest success rate of 42.6%, outperforming the strongest baseline (AWM) by 9.8 percentage points. Our component ablations further show that removing either verification or experience learning reduces the success rate (Table 7). These results support the effectiveness of MobiMux's structured experience organization and integrated verification mechanism in mobile GUI–CLI environments.

[1] Rawles C, Clinckemaillie S, Chang Y, et al. Androidworld: A dynamic benchmarking environment for autonomous agents[C]//International Conference on Learning Representations. 2025, 2025: 406-441.

[2] Kong Q, Zhang X, Yang Z, et al. Mobileworld: Benchmarking autonomous mobile agents in agent-user interactive and mcp-augmented environments[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 6142-6167.

[3] Xu Y, Liu X, Sun X, et al. Androidlab: Training and systematic benchmarking of android autonomous agents[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 2144-2166.

[4] Ouyang S, Yan J, Hsu I, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory[C]//International Conference on Learning Representations. 2026, 2026: 94327-94354.

[5] Wang Z Z, Mao J, Fried D, et al. Agent Workflow Memory[C]//Proceedings of the 42nd International Conference on Machine Learning. PMLR, 2025, 267: 63897-63911.

[6] Chen M, Hu X, Yu Z, et al. AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions[C]//Proceedings of the 43rd International Conference on Machine Learning. PMLR, 2026, 306: 14562-14588.

[7] Chen Q, Bellucci A, Sun Z, et al. SkillDroid: Compile Once, Reuse Forever[J]. arXiv preprint arXiv:2604.14872, 2026.

Dear Reviewer,

Thank you for your time and effort in reviewing our paper. We have provided a detailed response to your comments. As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. Please let us know if any questions remain.

Thank you again for your valuable time and feedback.

Reviewer_jwAQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MobiCLI, a benchmark for evaluating CLI and hybrid GUI–CLI interaction in mobile environments. It contains 122 tasks across 18 applications and five scenarios, with controlled initialization, deterministic evaluators, and detailed interface documentation. Experiments show substantial benefits from adding CLI access to general-purpose models, limited gains for the evaluated mobile-use models, and lower success rates on the Hybrid subset. The paper also proposes MobiMux, an agentic framework combining pre-submission verification with reusable experience, improving success from 26.2% to 42.6% for Seed 2.0 pro and from 43.4% to 57.4% for Kimi-k2.6.

Critical strengths
  1. Useful coverage of mobile workloads beyond conventional screen interaction. Section 3.2 includes batch operations, file management, computation, data processing, and system operations. Table 2 reports that 57.4% of tasks involve multiple applications or interfaces. The examples in Appendix A.5 illustrate meaningful workflows, such as converting structured data before importing it into an application.
  2. A controlled evaluation environment. Section 3.2 also describes a shared base snapshot, task-specific initialization, and deterministic checks over application and system state. Appendix A.1 further describes controlled backend deployments. These design choices provide a promising foundation for reproducible evaluation and reduce dependence on model-based judging, although evaluator validity still requires additional evidence.
  3. Informative analysis beyond aggregate success. Tables 3 and 5 and Figures 6–7 examine shell adoption, execution errors, documentation retrieval, and failure categories. These analyses distinguish agents that never invoke the shell from those that invoke it without improving task completion, providing useful directions for model adaptation and environment design.
Critical weaknesses
  1. The Hybrid definition does not establish a coordination requirement. Section 3.1 requires GUI use but does not exclude GUI-only solutions. Without a CLI-only ablation, Table 4 cannot distinguish coordination difficulty from task complexity. Following WeaveBench [1], the authors should validate channel dependencies and clarify how shell-mediated GUI actions are classified.
  2. Novelty relative to closely related work remains unclear. Beyond the GUI Paradigm [2] already studies mobile CLI agents and CLI-favorable tasks, while OSWorld-MCP [3], ToolCUA [4], and WeaveBench [1] investigate tool selection and hybrid execution. The paper should identify which mobile capabilities or workflow dependencies MobiCLI uniquely evaluates.
  3. CLI gains conflate broader access with better execution. Tasks are deliberately CLI-favorable (§3.2), and some require information unavailable through the standard GUI (Appendix A.5). Thus, gains may reflect additional access, wrappers, or documentation rather than CLI execution itself. Coverage analysis and jointly feasible task comparisons, as explored in [5,6], would clarify these contributions.
  4. MobiMux’s coordination and generalization benefits are insufficiently demonstrated. Tables 3 and 7 report aggregate gains without Hybrid-specific results. Training–test template separation and experience-management details are also unclear. Comparisons with simpler memory baselines and component ablations would better establish its contribution relative to experience-reuse methods such as SkillDroid [7] and AutoRPA [8].
  5. Cost and evaluation validity need stronger evidence. MobiMux increases average steps (Table 3), but total inference and experience-construction costs, or token consumption, are not reported. Deterministic evaluators also need negative tests or trajectory audits to establish semantic correctness.

[1] Li et al. WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces. 2026. arXiv:2606.09426v3.

[2] Gu et al. Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen? 2026. arXiv:2606.19388v1.

[3] Jia et al. OSWorld-MCP: Benchmarking MCP Tool Invocation in Computer-Use Agents. 2025. arXiv:2510.24563v2.

[4] Hu et al. ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents. 2026. arXiv:2605.12481v2.

[5] Zhou et al. GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents. 2026. arXiv:2606.24551v1.

[6] Bechard et al. Terminal Agents Suffice for Enterprise Automation. 2026. arXiv:2604.00073v4.

[7] Chen et al. SkillDroid: Compile Once, Reuse Forever. 2026. arXiv:2604.14872v1.

[8] Chen et al. AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions. ICML 2026. arXiv:2605.21082v1.

Questions for authors
  1. Interface necessity and novelty: Which Hybrid tasks genuinely require both channels, and what capabilities do they add beyond the CLI-Advantage Suite? Could the authors provide a CLI-only ablation under comparable conditions?
  2. Source of improvements: How much of the CLI gain remains on jointly feasible tasks? Could MobiMux’s results be broken down by CLI/Hybrid tasks and corrected failure types?
  3. Continual Experience Learning: This appears to be a form of context management. Could you elaborate on the rationale behind the design of these four types of experiences? Additionally, are there any visualizations or qualitative case studies that demonstrate how they function within specific tasks? If possible, please also address the remaining concerns raised in the Weaknesses section.
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We thank the reviewer for the detailed and constructive feedback. We appreciate the recognition of MobiCLI's diverse and realistic task coverage, controlled and reproducible evaluation design, and informative analyses. We provide additional evidence and clarifications to address the concerns below.

Interface necessity (W1, Q1)

We would like to clarify that MobiCLI aims to evaluate the complementarity of GUI and CLI interaction in Hybrid tasks, rather than requiring every Hybrid task to be strictly infeasible under either interface alone. In fact, MobiCLI focuses on common everyday mobile scenarios, where most tasks can be completed through GUI-only interaction. However, CLI provides more efficient and reliable alternatives for computational, repetitive, or data-intensive subtasks, reducing lengthy interaction sequences and potential errors.

CLI-only ablation (W1, Q1)

We conducted additional CLI-only experiments on four general-purpose models, where agents were restricted to CLI actions. As shown in the table below, all models exhibit performance degradation compared to the GUI+CLI setting. Further analysis reveals two main reasons for this degradation: (1) certain operations require GUI interactions and cannot be completed through the available CLI interface, and (2) some models, such as Claude Opus 4.8, occasionally hallucinate command execution results.

ModelGUI-onlyCLI-only (new)GUI+CLI
Claude Opus 4.87.4%35.2%55.7%
Seed2.0 Pro11.5%23.8%26.2%
Kimi K2.67.4%36.9%43.4%
Qwen3.5-35B-A3B2.5%13.1%13.9%

Related work (W2, Q1)

We thank the reviewer for pointing out the relevant concurrent work [1], which was released on arXiv shortly before our submission. After carefully reviewing it, we would like to clarify the key distinctions below:

(1) Research Focus: Beyond the GUI Paradigm [1] treats GUI and CLI as two mutually exclusive execution paradigms: an agent operates either through GUI or through CLI. In contrast, MobiCLI focuses on their complementarity. We equip a single agent with both GUI and CLI capabilities, requiring it to decide when to use each interface and how to coordinate them within the same task.

(2) Comprehensiveness and Complexity: Compared with the 45-task CLI-Advantage Suite [1], which focuses on CLI tasks, MobiCLI comprises 122 tasks, including 63 CLI tasks and 59 Hybrid tasks. Moreover, in addition to the operations in [1], MobiCLI covers more diverse and challenging tasks absent from [1], including multi-source batch processing, complex file management, algorithmic optimization, sophisticated data transformation, visual content processing, structured document editing, and software diagnostics. Some tasks also involve on-demand skill retrieval for APIs and shell interfaces, requiring agents to acquire interface knowledge and coordinate heterogeneous operations. Notably, 92 tasks combine multiple capabilities, and 70 span multiple applications or interfaces. Claude Opus 4.8 achieves only 55.7% success with GUI+CLI access, highlighting the benchmark's difficulty and its broader evaluation of mobile CLI capabilities and GUI–CLI complementarity.

(3) Evaluation: While [1] primarily evaluates GUI and CLI agents separately on existing mobile benchmarks, MobiCLI provides a more comprehensive evaluation under GUI-only and GUI+CLI settings, offering deeper insights into agents' ability to leverage and coordinate complementary GUI–CLI capabilities.

(4) Agent Framework (MobiMux): Beyond benchmark construction, we further propose MobiMux, a hybrid GUI–CLI agent framework that integrates task verification with structured experience learning. It improves success rates from 26.2% to 42.6% for Seed 2.0 Pro and from 43.4% to 57.4% for Kimi K2.6.

Regarding OSWorld-MCP [2], ToolCUA [3], and WeaveBench [4]:

(5) Mobile-Specific Hybrid Interaction: These works primarily investigate tool selection and hybrid execution in desktop environments. However, hybrid GUI–CLI interaction remains underexplored on mobile devices. To address this gap, MobiCLI specifically examines how CLI operations complement GUI interactions in mobile workflows. It evaluates phone-native services, application-specific interfaces, and device-level operations.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

CLI gains (W3, Q2)

We conducted a coverage analysis and found that 116 out of 122 tasks (95.1%) are feasible through standard GUI interactions. Only a few system-related tasks require information or operations unavailable through the standard GUI. For the remaining 116 jointly feasible tasks, we further compare GUI-only and GUI+CLI performance:

ModelGUI-onlyGUI+CLIΔ
Claude Opus 4.87.8%54.3%+46.5
Kimi K2.67.8%42.2%+34.4
Seed 2.0 Pro12.1%25.9%+13.8
Qwen3.5-35B-A3B2.6%12.1%+9.5

The improvements persist on jointly feasible tasks. This demonstrates that the gains cannot be explained solely by access to information unavailable through the GUI. Instead, CLI augmentation provides more efficient and reliable execution paths.

We also want to clarify that the lightweight wrappers primarily abstract cumbersome Android service interfaces into a consistent CLI format, rather than providing specific solutions. Similarly, skill documents describe available interfaces, enabling agents to discover and invoke them appropriately. The benchmark does not provide executable solutions.

MobiMux's improvements (W4, Q2)

We would like to clarify that MobiMux is designed to improve overall task execution in GUI–CLI environments, rather than exclusively targeting GUI–CLI coordination. Importantly, CLI and Hybrid denote task categories rather than execution settings. Both are evaluated under the same GUI+CLI setting, where agents have access to both interfaces and must determine how to accomplish each task without being informed of its category. Thus, even CLI tasks require agents to identify appropriate interaction strategies and effectively utilize available tools.

To provide a more fine-grained analysis, we further break down MobiMux's performance by task type. For Seed2.0 Pro, MobiMux improves CLI SR from 34.9% to 54.0% (+19.0 points) and Hybrid SR from 16.9% to 30.5% (+13.6 points). Similarly, for Kimi K2.6, CLI SR increases from 57.1% to 68.3% (+11.1 points), while Hybrid SR increases from 28.8% to 45.8% (+16.9 points). These consistent improvements across both task types demonstrate that MobiMux provides broad benefits for task execution in GUI–CLI environments.

MobiMux's evaluation (W4, W5)

Following the reviewer's recommendation, we conducted additional comparisons with AutoRPA [5] and SkillDroid [6]. The results are summarized below.

MethodSR (%) ↑Avg. Tokens ↓
Seed2.0 Pro26.2196k
+ AutoRPA27.9562k
+ SkillDroid20.5137k
+ MobiMux (Ours)42.6306k

MobiMux achieves the highest success rate of 42.6%, substantially outperforming AutoRPA (27.9%) and SkillDroid (20.5%).

We note that AutoRPA and SkillDroid are primarily designed to reuse executable GUI workflows across structurally similar tasks. However, such opportunities are limited in MobiCLI, where tasks involve diverse operations and interaction requirements. Consequently, their memory reuse mechanisms were rarely activated in our evaluation. We also note that SkillDroid's original execution agent relies exclusively on accessibility-tree text without screenshot-based perception, which limits its ability to handle visually dependent tasks.

We additionally report average token consumption aggregated across all recorded LLM calls. Although SkillDroid consumes fewer tokens, its success rate is considerably lower. These results highlight the performance–cost trade-off of MobiMux, whose primary objective is to improve task reliability rather than minimize inference overhead.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

MobiMux's details (Q3, W4)

(1) Rationale behind the design: Our design addresses the different scopes of reusable knowledge in mobile GUI–CLI execution. Specifically:

  • General experience captures transferable, cross-task strategies and is persistently included in the Actor's system prompt.
  • App experience captures application-specific interaction knowledge, organized as per-app skills and loaded on demand.
  • CLI experience captures tool-specific command usage, organized as per-tool skills and loaded on demand.
  • Verify experience captures application-specific completion criteria and common verification pitfalls, and is provided to the Verifier rather than the Actor.

This separation allows MobiMux to preserve broadly applicable strategies while selectively retrieving specialized knowledge, avoiding unnecessary context accumulation and keeping execution guidance separate from completion verification.

(2) Visualization: The original manuscript provides concrete examples of all four experience types in Table 8. To further improve clarity, we have additionally included a workflow visualization in the supplementary material.

(3) Training–test template separation: MobiMux uses a separate set of training tasks to construct its experience pool, with isolated namespaces and task-specific assets to prevent instance-level leakage. More importantly, we introduce structural variations across multiple dimensions, including input structure, task composition, boundary conditions, and GUI–CLI switching points. These variations are deliberately designed to encourage transferable experience rather than template memorization.

To quantify potential overlap, we compute the nearest-neighbor similarity between training and evaluation tasks using two complementary metrics. The median semantic similarity is 0.565 using all-MiniLM-L6-v2. Additionally, the median lexical similarity is only 0.106, measured using TF-IDF with word unigrams and bigrams. These results indicate limited semantic and lexical overlap.

Furthermore, as reported in our response regarding memory baselines, MobiMux substantially outperforms other methods, suggesting that its structured experience organization, combined with task verification, provides benefits beyond other experience-reuse mechanisms.

Evaluator validity (W5)

We clarify that all task evaluators were validated during benchmark construction using both positive and negative test cases. Specifically, each task includes at least one positive case representing correct completion and one negative case deliberately designed to simulate a likely failure mode for that particular task. For tasks with more complex evaluation criteria, we construct more negative cases to cover different failure conditions, with up to nine negative cases for a single task.

[1] Gu et al. Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen? 2026. arXiv:2606.19388v1.

[2] Jia et al. OSWorld-MCP: Benchmarking MCP Tool Invocation in Computer-Use Agents. 2025. arXiv:2510.24563v2.

[3] Hu et al. ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents. 2026. arXiv:2605.12481v2.

[4] Li et al. WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces. 2026. arXiv:2606.09426v3.

[5] Chen M, Hu X, Yu Z, et al. AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions[C]//Proceedings of the 43rd International Conference on Machine Learning. PMLR, 2026, 306: 14562-14588.

[6] Chen Q, Bellucci A, Sun Z, et al. SkillDroid: Compile Once, Reuse Forever[J]. arXiv preprint arXiv:2604.14872, 2026.

[7] Ouyang S, Yan J, Hsu I, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory[C]//International Conference on Learning Representations. 2026, 2026: 94327-94354.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their valuable time and constructive feedback. We appreciate their recognition of the importance and motivation of studying CLI capabilities for mobile agents (all reviewers), the benchmark's diverse and useful task coverage (jwAQ, VWvk), the controlled and reproducible evaluation design (jwAQ), and the informative experimental findings and analyses (all reviewers).

MobiCLI identifies 5 representative mobile scenarios where CLI capabilities are particularly useful and introduces a benchmark of 122 tasks across 18 applications to evaluate CLI and hybrid GUI–CLI interaction. Throughout evaluation, a single agent is equipped with both GUI and CLI capabilities without being informed of the task type. Beyond benchmarking existing general-purpose and phone-use agents, we propose MobiMux, which combines task verification with structured experience learning to improve execution reliability in GUI–CLI environment. Our central goal is to investigate how CLI complements GUI interaction in practical mobile workflows.

The main new evidence and clarifications in this rebuttal are:

  1. CLI benefits (jwAQ). Our coverage analysis shows that 116/122 tasks (95.1%) are GUI-feasible. On these tasks, GUI+CLI still improves success by 9.5–46.5 percentage points over GUI-only execution across general-purpose models.
  2. Hybrid difficulty (jwAQ, VWvk). Our failure analysis reveals that 40.8%–55.9% of failed Hybrid trajectories exhibit identifiable interface coordination errors. Furthermore, among tasks with comparable execution lengths, Hybrid tasks achieve substantially lower success rates than CLI tasks (0.41×). Decomposing Hybrid tasks into independently evaluated GUI and CLI subtasks further reveals a 13–17 percentage-point gap between joint subtask success and end-to-end success.
  3. MobiMux improves task execution in GUI–CLI environments (jwAQ, VWvk). Additional task-type breakdowns confirm consistent improvements on both subsets. MobiMux also outperforms ReasoningBank, AWM, AutoRPA, and SkillDroid (42.6% vs. 30.3%, 32.8%, 27.9%, and 20.5% SR, respectively). We clarify training–evaluation separation through structural variations and isolated task assets, supported by nearest-neighbor similarity analysis (median semantic similarity 0.565, lexical similarity 0.106).
  4. Benchmark design and scalability are further clarified (jwAQ, Bmc8). We elaborate on the rationale for scenario and application selection, failure taxonomy, distinctions from related work, and scalable CLI support through existing shell utilities, application interfaces, and reusable service wrappers.

We are grateful for the reviewers' detailed suggestions. We hope these additional results and clarifications address the key concerns, and we welcome further discussion. Thank you again for your thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.