- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Mobile Skills Bench (MSB), a benchmark for evaluating long-horizon interactive mobile agents with reusable skills. MSB includes 145 tasks across 24 simulated apps, supports both Pure GUI and Hybrid execution, and compares no-skill, textual-skill, and multimodal-skill settings across nine models.
Critical strengths
The benchmark idea is interesting. MSB provides a controlled setting to study whether reusable skills actually help long-horizon mobile agents, including textual vs. multimodal skills and different execution modes.
Critical weaknesses
1.The realism and external validity of the benchmark are unclear. All tasks run on simulated applications, and the skills are generated from the simulator source code and manually reviewed. This tight coupling between the environment and the skill library may overestimate how useful such skills would be in real-world mobile applications.
2.The evaluation focuses mainly on general-purpose foundation models, while specialized GUI/mobile-agent models are not included. It is therefore unclear whether the conclusions about skill effectiveness generalize to agents explicitly trained for GUI interaction.
Questions for authors
Do the authors plan to evaluate smaller models more systematically? Since skills are intended to provide reusable procedural guidance, it would be useful to know whether they can meaningfully improve weaker models, or whether the observed benefits are largely limited to already capable agents.
7 comments
Reply part 2
| Model | Execution | Task success (%), w/o skills | Task success (%), w/ skills | Required checks completed (%), w/o skills | Required checks completed (%), w/ skills |
|---|---|---|---|---|---|
| UI-TARS-72B-DPO | Pure GUI | 0.0 ± 0.0 | 0.2 ± 0.4 | 7.3 ± 1.2 | 7.2 ± 1.1 |
| UI-Venus-Navi-72B | Pure GUI | 0.0 ± 0.0 | 0.0 ± 0.0 | 1.6 ± 0.6 | 1.1 ± 0.6 |
| GUI-Owl-1.5-32B-Think | Pure GUI | 0.0 ± 0.0 | 0.0 ± 0.0 | 3.7 ± 0.7 | 4.2 ± 1.2 |
| GUI-Owl-1.5-32B-Think | Hybrid | 0.5 ± 0.8 | 0.2 ± 0.4 | 1.8 ± 1.0 | 3.7 ± 1.6 |
| Gemini 3.8 Flash | Pure GUI | 22.8 ± 1.4 | 34.0 ± 1.1 | 43.0 ± 0.9 | 49.2 ± 0.7 |
| Claude Haiku 4.5 | Pure GUI | 0.0 ± 0.0 | 0.7 ± 0.7 | 3.8 ± 0.3 | 5.0 ± 0.7 |
| Qwen3-VL 235B | Pure GUI | 0.0 ± 0.0 | 0.0 ± 0.0 | 1.0 ± 0.5 | 1.6 ± 0.1 |
Mean ± SD over three runs. Skills refer to multimodal skills. Rows in italics are general-purpose models from the paper. UI-TARS and UI-Venus have no tool actions and therefore run under Pure GUI only.
- GUI specialization does not transfer to MSB. The specialists solve at most 0.5% of the tasks and complete only 1.1–7.3% of the required checks. This places them on par with the weakest general-purpose models and far below Gemini 3.8 Flash. This is expected given what MSB targets: long-horizon, difficult tasks that require multi-step plans across apps, questions to the user about missing information, and state carried from one app to another. The specialists ground well on a single screen, but this does not carry over: they make any required progress on only 19–43% of tasks and complete at least half of the required checks on only 1–5%. Their failure modes differ. UI-Venus almost never asks the user (≤0.2% of episodes) and stops after about 34 steps. UI-TARS asks often (39–53%) but rarely completes the later sub-goals. GUI-Owl runs 74–76 steps on average without converging.
- Skills do not help them, because they do not read skills. They read a skill in at most 1.8% of episodes. Adding skills changes their task success by at most 0.3 pp and their completion by at most 1.9 pp, which is comparable to the run-to-run variation (SD up to 1.6 pp).
- For agents that do not read skills, the catalogue is only extra context. The skill catalogue adds about 15k characters (48 entries) to the system prompt. UI-TARS never opens a skill. Yet with the catalogue present, it asks the user less often (52.9% → 39.3% of episodes) and produces more unparseable outputs (0.06% → 0.61%) in all three runs. GUI-Owl under Hybrid execution shows the same rise in unparseable outputs. This is consistent with the longer context diluting the model's attention without delivering any knowledge from the skills.
In summary, skills benefit capable agents that consult them, and current GUI specialists do not consult them.
Revisions. The GUI-specialist evaluation is in Appendix B.2, and the completion measure is in Appendix B.3. Sec. 4.1 and RQ2 in Sec. 4.2 refer to both.
Q. Do skills meaningfully help weaker models, or are the benefits limited to already capable agents?
At present, the benefit is largely limited to capable agents, and we now state this explicitly. Because task success is near zero for weak models, we compare the share of required checks completed. The table below reports this share under Pure GUI (no skills → multimodal skills; mean ± SD over three runs) and includes the three GUI-specialist models from W2:
| Model | w/o skills (%) | w/ skills (%) | Change (pp) | Skill read rate |
|---|---|---|---|---|
| Gemini 3.8 Flash | 43.0 ± 0.9 | 49.2 ± 0.7 | +6.2 | 97.5% |
| Claude Haiku 4.5 | 3.8 ± 0.3 | 5.0 ± 0.7 | +1.2 | ≤ 23% |
| Qwen3-VL 235B | 1.0 ± 0.5 | 1.6 ± 0.1 | +0.6 | ≤ 23% |
| UI-TARS-72B-DPO | 7.3 ± 1.2 | 7.2 ± 1.1 | −0.1 | 0% |
| UI-Venus-Navi-72B | 1.6 ± 0.6 | 1.1 ± 0.6 | −0.5 | 0.2% |
| GUI-Owl-1.5-32B-Think | 3.7 ± 0.7 | 4.2 ± 1.2 | +0.5 | 1.8% |
Only Gemini 3.8 Flash gains beyond the run-to-run variation. We identify two bottlenecks:
- Weak models rarely consult skills. Claude Haiku 4.5 and Qwen3-VL read a skill in at most 23% of episodes, and the GUI specialists in at most 1.8%. For the specialists, the catalogue even changes behavior without delivering knowledge (W2).
- Mid-tier models read but cannot execute. Claude Sonnet 5 and Kimi K2.6 read skills in 74–96% of episodes, yet gain at most 2.2 pp even in the episodes where they read.
Making skills useful for weaker agents therefore requires agents that learn to look up and follow procedural knowledge; better skills alone are not enough.
Revisions. Takeaway 2 in Sec. 4.2 now states this limitation, and Appendix B.3 (Tab. 8) reports the completion measure for all models.
We hope these revisions address the reviewer's concerns, and we would be glad to clarify anything further.
Reply part 3
References
- Qin et al. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv:2501.12326, 2025.
- Gu et al. UI-Venus Technical Report: Building High-performance UI Agents with RFT. arXiv:2508.10833, 2025.
- Xu et al. Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents. arXiv:2602.16855, 2026.
Reply part 1
We sincerely thank the reviewer for the careful reading and the insightful comments, and we are delighted that the reviewer appreciates the controlled design of MSB. All changes made in response to this review are marked in purple in the revised PDF.
W1. All apps are simulated and the skills are generated from the simulator's source code.
The benefit of skills depends neither on access to the source code nor on the apps being simulated. Skills written only from recorded trajectories still raise success, and the skill effect carries over to real apps.
Why simulated apps. Real app services cannot be reset to a fixed starting state, their backend state cannot be read for verification, and actions such as payments cannot be undone. Repeatable, automatically verified comparisons therefore require simulated apps, as in WebArena and AppWorld.
Skills without source access. Skills still help when they are written without the source code. To test this, we rebuilt the entire skill library without it. A model (Claude Opus 5.5) wrote every guide, screen manual and reference image only from the recorded no-skill trajectories of the nine evaluated models. From the verifier it received only whether each episode succeeded. The guides were then checked and corrected by an agent that operates the simulated phone through gestures only, as a user would. Each task is evaluated with a library built without its own trajectories. The catalogue and the protocol are unchanged.
Task success of Gemini 3.8 Flash (%, mean ± SD over 3 runs):
Real-app check. The skill effect also carries over from simulated to real apps. To test this, we repeated seven MSB tasks with real Android apps on an Android 15 emulator: the Etar calendar, Google Clock, Contacts and Messages, Files, and Android Settings. These are the tasks whose apps exist on a phone and whose state can be verified exactly. The agent, user simulator, skills and verifiers are the same. With Gemini 3.8 Flash under Pure GUI (7 tasks × 3 runs per condition), success rises from 38% (8/21) without skills to 52–57% (11–12/21) with skills. This gain is similar to that on the same tasks in the simulator (24% → 48–52%). Given the small sample, we read this as a consistency check rather than a precise estimate. It nevertheless shows that the skills work on real apps even though they were written for the simulated ones.
Revisions. The skills written without source access are in Appendix B.1, and RQ2 in Sec. 4.2 summarizes them. The real-app check is in Appendix B.4.
W2. The evaluation focuses on general-purpose models; it is unclear whether the conclusions hold for agents trained for GUI interaction.
The conclusions extend to GUI-specialized agents. The three specialists we evaluated solve almost no tasks, and skills do not help them because they do not read the skills.
We evaluated three widely used open GUI-specialist models on all 145 tasks, with three runs per setting: UI-TARS-72B-DPO (Qin et al., 2025), UI-Venus-Navi-72B (Gu et al., 2025) and GUI-Owl-1.5-32B-Think (Xu et al., 2026). To avoid penalizing them for an unfamiliar output format, we prompt each model with its official template, action space and coordinate convention. We then translate its actions into MSB actions. Calibration and parsing checks confirm that this translation is faithful: taps land on their targets, and fewer than 0.7% of steps are unparseable (Appendix B.2).
Because their task success is near zero, we also report the share of required checks completed. This measure counts only the verifier checks that the agent has to change and excludes checks already satisfied at the start. It is averaged over tasks. 0% means no required change was made.