- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper asks a question: whether mobile use agents have to be based on GUI models.
The authors use state-of-the-art LLMs (MiniMax M2.7, Opus4.7, Sonnet, GPT) with coding agent harnesses to prove that, in AndroidWorld and MobileWorld, such coding agents can complete most of the tasks. And they have shown better generalization capabilities.
Critical strengths
- Good performance on Current Benchmarks: Shows that 88.8% of AndroidWorld and 86.3% of MobileWorld tasks can be solved via ADB commands without screen perception. The models are also off-the-shelf and zero-shot: no fine-tuning is performed for mobile use.
Critical weaknesses
-
Unrealistic Root/System-Level Privilege Assumptions: The approach relies on privileged ADB and root access in emulators, which is prohibited on standard unrooted consumer phones and makes real-world deployment largely impractical.
-
Reliance on Open-Source Apps and Exposed Schemas: Benchmark apps in AndroidWorld and MobileWorld provide full source code and accessible backend states, whereas real-world commercial apps are closed-source, obfuscated, and strictly sandboxed. The proposed method can in no way operate on real apps. The authors also do not provide such experiment details.
-
Engineering Wrapper Rather Than Core ML Research: Adapting off-the-shelf coding agents via system prompts and bash tool wrappers is primarily a system engineering effort rather than a fundamental algorithmic or research contribution.
-
Inability to Handle Visual or Closed-UI Tasks: Pure CLI agents are completely blind to tasks requiring visual media understanding or internal UI states that are not written to local databases.
-
The comparison is not fair. All GUI models compared are models with smaller sizes. Whereas the coding LLMs are large models, even state-of-the-art proprietary models.
-
There is no data on the time delay caused by ADB commands.
Questions for authors
Please address the weaknesses.
-
Can you benchmark the GUI agents with smaller-sized LLMs using coding agent harnesses (e.g., Qwen3.8-27B / Qwen3.5-9B)?
-
Can you provide ADB command time-delay data?
thanks for your response
thanks for your effort. however, my concerns still remain without real-device experiments. I will update my rating to 2: weak rejection.