- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MobiCLI is a benchmark for mobile agents that can use both GUI actions and shell commands. It contains 122 tasks across 18 apps, including 63 tasks solvable through CLI alone and 59 that still require GUI interaction. The authors evaluate eight models with and without CLI access and find large gains for strong general models, while existing phone-use agents benefit little. They also introduce MobiMux, which checks task completion and reuses experience collected on training tasks. With Kimi K2.6, it raises success from 43.4% to 57.4%. Ablations examine the contributions of verification and experience
Critical strengths
The benchmark includes tasks solvable entirely through the shell and tasks that still need GUI actions. This gives a useful setting for studying how mobile agents combine the two interfaces.
The paired evaluations show a large benefit from adding shell access and interface documentation. Testing eight models also reveals clear differences in how effectively agents use these capabilities, making the benchmark useful for further development.
Critical weaknesses
The experience pool is frozen during testing, but Section 5.1 gives little detail about the training tasks. It is unclear whether their templates overlap with MobiCLI, so the amount of genuine transfer remains uncertain.
Hybrid tasks have lower success and longer trajectories, but the two subsets are not matched for complexity. This makes it hard to tell how much of the gap comes from interface coordination, which limits that interpretation of the results.
Questions for authors
How were the MobiMux training tasks separated from evaluation tasks, particularly at the template level?
Can you separate interface coordination errors from general planning and step-budget failures in the hybrid-task results?
4 comments
Rebuttal by Authors (2/2)
Coordination Errors (W2, Q2)
To better disentangle these factors, we conduct three additional analyses.
(1) Controlling for execution length. To account for differences in task complexity, we use the median number of steps across successful executions by general-purpose models as a proxy for each task's reference solution length. We then group tasks by reference length and compare CLI and Hybrid success rates within comparable length ranges. After stratification, the Hybrid tasks' success rate remains only 0.41 times that of CLI tasks, suggesting that differences in execution length alone do not fully explain the performance gap.
(2) Hybrid-task decomposition. To further isolate the challenges introduced by integrating GUI and CLI operations, we construct two simplified variants for each Hybrid task: (B) CLI-only subtask, where the agent performs only the CLI-related operations and reports the results in its final response without interacting with the app; and (C) GUI-only subtask, where the agent is directly provided with the correctly processed results and only needs to perform the remaining GUI operations. We compare these variants against (A) the original end-to-end Hybrid task.
| Model | CLI-only subtask (B) | GUI-only subtask (C) | B × C | Original (A) | A − B × C |
|---|---|---|---|---|---|
| Kimi K2.6 | 0.90 | 0.49 | 0.46 | 0.29 | −0.17 |
| Seed2.0 Pro | 0.63 | 0.44 | 0.30 | 0.17 | −0.13 |
Here, B × C is computed per task and then averaged, measuring the fraction of tasks for which both independently evaluated subtasks succeed. Notably, Kimi K2.6 achieves 46% joint subtask success but only 29% end-to-end success, while Seed2.0 Pro drops from 30% to 17%. These gaps demonstrate that successfully executing individual GUI and CLI components does not necessarily translate into successful end-to-end execution. Integrating the two interfaces introduces additional challenges.
(3) Coordination-specific failure analysis. We further inspect failed Hybrid trajectories using three general-purpose models with different performance levels and one phone-use agent. We identify coordination errors involving (i) inappropriate interface selection for a sub-operation or (ii) failure to recognize or utilize states produced through the other interface.
| Model | Coordination Errors / Failed Hybrid Tasks |
|---|---|
| Claude Opus 4.8 | 50.0% |
| Seed2.0 Pro | 40.8% |
| Qwen3.5-35B-A3B | 42.4% |
| GUI-Owl-1.5-32B-Think | 55.9% |
Coordination errors are observed in 40.8%–55.9% of failed Hybrid trajectories, providing direct evidence that interface coordination is a substantial challenge.
Finally, we clarify that task-type labels (CLI or Hybrid) are not disclosed to agents during evaluation. Thus, agents must autonomously determine the appropriate interface for each operation throughout the entire benchmark, including CLI tasks. Hybrid tasks additionally evaluate the ability to integrate operations and maintain state consistency across interfaces.
[1] Ouyang S, Yan J, Hsu I, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory[C]//International Conference on Learning Representations. 2026, 2026: 94327-94354.
[2] Wang Z Z, Mao J, Fried D, et al. Agent Workflow Memory[C]//Proceedings of the 42nd International Conference on Machine Learning. PMLR, 2025, 267: 63897-63911.
[3] Chen M, Hu X, Yu Z, et al. AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions[C]//Proceedings of the 43rd International Conference on Machine Learning. PMLR, 2026, 306: 14562-14588.
[4] Chen Q, Bellucci A, Sun Z, et al. SkillDroid: Compile Once, Reuse Forever[J]. arXiv preprint arXiv:2604.14872, 2026.
Dear Reviewer,
Thank you for your time and effort in reviewing our paper. We have provided a detailed response to your comments. As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal. Please let us know if any questions remain.
Thank you again for your valuable time and feedback.
Dear Reviewer VWvk,
We apologize for reaching out again. We sincerely appreciate your thoughtful feedback and have taken your concerns very seriously. We have carefully addressed your comments and conducted additional experiments to provide further evidence.
As the discussion period is approaching its end, we would be truly grateful if you could find a moment to review our responses.
Thank you very much for your time and effort in reviewing our work. We sincerely appreciate your consideration.
Rebuttal by Authors (1/2)
We thank the reviewer for the positive assessment. We appreciate the recognition of MobiCLI's coverage of both CLI and Hybrid tasks, as well as the value of our paired evaluations. We address the reviewer's concerns with additional experimental evidence and clarifications below.
Training–Evaluation Separation (W1, Q1)
We clarify that MobiMux uses a separate set of training tasks to construct its experience pool, with isolated namespaces and task-specific assets to prevent instance-level leakage. More importantly,we introduce structural changes across several dimensions, including input structure, task composition, boundary conditions, and GUI–CLI switching points. Their task structures and execution requirements are deliberately varied to encourage transferable experience rather than template memorization.
To further assess potential overlap, we compute the nearest-neighbor similarity between training and evaluation tasks using two complementary metrics. The median semantic similarity is 0.565 using all-MiniLM-L6-v2. Additionally, the median lexical similarity is only 0.106, measured using TF-IDF with word unigrams and bigrams. These results indicate limited semantic and lexical overlap.
We additionally compare MobiMux with representative experience-learning and skill-reuse methods using Seed2.0 Pro:
MobiMux outperforms all baselines. Notably, AutoRPA [3] and SkillDroid [4] primarily focus on reusing executable procedures for recurring task patterns, whereas MobiMux extracts reusable knowledge at multiple levels. These results further support the effectiveness of MobiMux in transferring accumulated experience across different tasks.