acceptodds
Under review as a conference paper at ICLR 2027

Mobile Skills Bench: Benchmarking Long-Horizon Interactive Mobile Agents with Multimodal Skills under Hybrid Execution

Abstract

Mobile agents are increasingly expected to carry out everyday workflows that span multiple applications, changing interfaces, and evolving user requests. However, such long-horizon tasks remain challenging for current agents. Agent skills offer a promising remedy, because they package reusable procedural knowledge, and multimodal skills further add visual grounding through reference images. Yet their benefit remains unclear, since no benchmark compares skill designs under controlled conditions on long, interactive mobile tasks with verifiable outcomes and hybrid execution. To address this gap, we introduce Mobile Skills Bench (MSB), a controlled testbed for evaluating agent skills, which comprises 145 long-horizon tasks across 24 simulated applications with a verifiable backend, provides a user simulator, and supports both Pure GUI and Hybrid (GUI, MCP, and CLI) execution. MSB further includes a multimodal skill library of 73 procedures with 166 reference images, organized by function, by function–app pair, and by app. We then evaluate nine models in three agent harnesses. Without skills, the nine models achieve an average task success of only 28.9% under Hybrid and 12.5% under Pure GUI execution. The models often fail to ask for missing information, claim completion prematurely, or stall without recovery. With textual skills, the two averages rise to 34.2% and 15.0%, although the gain varies with the model and execution mode. However, adding the tested static reference images yields no consistent further gain. Finally, the three library organizations reach similar task success, but organizing the library by app uses 31% fewer input tokens on average than organizing it by function or by function–app pair, which suggests a simple principle for designing skill libraries in mobile scenarios.

Primary area:
datasets and benchmarks
Keywords:
AI Agents, Mobile Agents, VLM Agents, Multimodal Skills, Long-Horizon Tasks, Agent Skills, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Yce5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Mobile Skills Bench (MSB), a benchmark for evaluating long-horizon interactive mobile agents with reusable skills. MSB includes 145 tasks across 24 simulated apps, supports both Pure GUI and Hybrid execution, and compares no-skill, textual-skill, and multimodal-skill settings across nine models.

Critical strengths

The benchmark idea is interesting. MSB provides a controlled setting to study whether reusable skills actually help long-horizon mobile agents, including textual vs. multimodal skills and different execution modes.

Critical weaknesses

1.The realism and external validity of the benchmark are unclear. All tasks run on simulated applications, and the skills are generated from the simulator source code and manually reviewed. This tight coupling between the environment and the skill library may overestimate how useful such skills would be in real-world mobile applications.

2.The evaluation focuses mainly on general-purpose foundation models, while specialized GUI/mobile-agent models are not included. It is therefore unclear whether the conclusions about skill effectiveness generalize to agents explicitly trained for GUI interaction.

Questions for authors

Do the authors plan to evaluate smaller models more systematically? Since skills are intended to provide reusable procedural guidance, it would be useful to know whether they can meaningfully improve weaker models, or whether the observed benefits are largely limited to already capable agents.

7 comments
AuthorsOpenReview ↗

Reply part 1

We sincerely thank the reviewer for the careful reading and the insightful comments, and we are delighted that the reviewer appreciates the controlled design of MSB. All changes made in response to this review are marked in purple in the revised PDF.


W1. All apps are simulated and the skills are generated from the simulator's source code.

The benefit of skills depends neither on access to the source code nor on the apps being simulated. Skills written only from recorded trajectories still raise success, and the skill effect carries over to real apps.

Why simulated apps. Real app services cannot be reset to a fixed starting state, their backend state cannot be read for verification, and actions such as payments cannot be undone. Repeatable, automatically verified comparisons therefore require simulated apps, as in WebArena and AppWorld.

Skills without source access. Skills still help when they are written without the source code. To test this, we rebuilt the entire skill library without it. A model (Claude Opus 5.5) wrote every guide, screen manual and reference image only from the recorded no-skill trajectories of the nine evaluated models. From the verifier it received only whether each episode succeeded. The guides were then checked and corrected by an agent that operates the simulated phone through gestures only, as a user would. Each task is evaluated with a library built without its own trajectories. The catalogue and the protocol are unchanged.

Task success of Gemini 3.8 Flash (%, mean ± SD over 3 runs):

ExecutionSkill formatNo skillsTrajectory-derived skills
Pure GUItext22.8 ± 1.427.8 ± 3.1
Pure GUImultimodal22.8 ± 1.426.9 ± 2.5
Hybridtext48.5 ± 2.153.3 ± 2.2
Hybridmultimodal48.5 ± 2.158.3 ± 2.4
  • Gains in all four settings. Trajectory-derived skills raise mean success by +4.1 to +9.9 pp (+6.0 pp on average, paired estimates). The gain is significant under Hybrid execution with multimodal skills.
  • The agent reads them in 90–96% of episodes.

Real-app check. The skill effect also carries over from simulated to real apps. To test this, we repeated seven MSB tasks with real Android apps on an Android 15 emulator: the Etar calendar, Google Clock, Contacts and Messages, Files, and Android Settings. These are the tasks whose apps exist on a phone and whose state can be verified exactly. The agent, user simulator, skills and verifiers are the same. With Gemini 3.8 Flash under Pure GUI (7 tasks × 3 runs per condition), success rises from 38% (8/21) without skills to 52–57% (11–12/21) with skills. This gain is similar to that on the same tasks in the simulator (24% → 48–52%). Given the small sample, we read this as a consistency check rather than a precise estimate. It nevertheless shows that the skills work on real apps even though they were written for the simulated ones.

Revisions. The skills written without source access are in Appendix B.1, and RQ2 in Sec. 4.2 summarizes them. The real-app check is in Appendix B.4.


W2. The evaluation focuses on general-purpose models; it is unclear whether the conclusions hold for agents trained for GUI interaction.

The conclusions extend to GUI-specialized agents. The three specialists we evaluated solve almost no tasks, and skills do not help them because they do not read the skills.

We evaluated three widely used open GUI-specialist models on all 145 tasks, with three runs per setting: UI-TARS-72B-DPO (Qin et al., 2025), UI-Venus-Navi-72B (Gu et al., 2025) and GUI-Owl-1.5-32B-Think (Xu et al., 2026). To avoid penalizing them for an unfamiliar output format, we prompt each model with its official template, action space and coordinate convention. We then translate its actions into MSB actions. Calibration and parsing checks confirm that this translation is faithful: taps land on their targets, and fewer than 0.7% of steps are unparseable (Appendix B.2).

Because their task success is near zero, we also report the share of required checks completed. This measure counts only the verifier checks that the agent has to change and excludes checks already satisfied at the start. It is averaged over tasks. 0% means no required change was made.

AuthorsOpenReview ↗

Reply part 2

ModelExecutionTask success (%), w/o skillsTask success (%), w/ skillsRequired checks completed (%), w/o skillsRequired checks completed (%), w/ skills
UI-TARS-72B-DPOPure GUI0.0 ± 0.00.2 ± 0.47.3 ± 1.27.2 ± 1.1
UI-Venus-Navi-72BPure GUI0.0 ± 0.00.0 ± 0.01.6 ± 0.61.1 ± 0.6
GUI-Owl-1.5-32B-ThinkPure GUI0.0 ± 0.00.0 ± 0.03.7 ± 0.74.2 ± 1.2
GUI-Owl-1.5-32B-ThinkHybrid0.5 ± 0.80.2 ± 0.41.8 ± 1.03.7 ± 1.6
Gemini 3.8 FlashPure GUI22.8 ± 1.434.0 ± 1.143.0 ± 0.949.2 ± 0.7
Claude Haiku 4.5Pure GUI0.0 ± 0.00.7 ± 0.73.8 ± 0.35.0 ± 0.7
Qwen3-VL 235BPure GUI0.0 ± 0.00.0 ± 0.01.0 ± 0.51.6 ± 0.1

Mean ± SD over three runs. Skills refer to multimodal skills. Rows in italics are general-purpose models from the paper. UI-TARS and UI-Venus have no tool actions and therefore run under Pure GUI only.

  • GUI specialization does not transfer to MSB. The specialists solve at most 0.5% of the tasks and complete only 1.1–7.3% of the required checks. This places them on par with the weakest general-purpose models and far below Gemini 3.8 Flash. This is expected given what MSB targets: long-horizon, difficult tasks that require multi-step plans across apps, questions to the user about missing information, and state carried from one app to another. The specialists ground well on a single screen, but this does not carry over: they make any required progress on only 19–43% of tasks and complete at least half of the required checks on only 1–5%. Their failure modes differ. UI-Venus almost never asks the user (≤0.2% of episodes) and stops after about 34 steps. UI-TARS asks often (39–53%) but rarely completes the later sub-goals. GUI-Owl runs 74–76 steps on average without converging.
  • Skills do not help them, because they do not read skills. They read a skill in at most 1.8% of episodes. Adding skills changes their task success by at most 0.3 pp and their completion by at most 1.9 pp, which is comparable to the run-to-run variation (SD up to 1.6 pp).
  • For agents that do not read skills, the catalogue is only extra context. The skill catalogue adds about 15k characters (48 entries) to the system prompt. UI-TARS never opens a skill. Yet with the catalogue present, it asks the user less often (52.9% → 39.3% of episodes) and produces more unparseable outputs (0.06% → 0.61%) in all three runs. GUI-Owl under Hybrid execution shows the same rise in unparseable outputs. This is consistent with the longer context diluting the model's attention without delivering any knowledge from the skills.

In summary, skills benefit capable agents that consult them, and current GUI specialists do not consult them.

Revisions. The GUI-specialist evaluation is in Appendix B.2, and the completion measure is in Appendix B.3. Sec. 4.1 and RQ2 in Sec. 4.2 refer to both.


Q. Do skills meaningfully help weaker models, or are the benefits limited to already capable agents?

At present, the benefit is largely limited to capable agents, and we now state this explicitly. Because task success is near zero for weak models, we compare the share of required checks completed. The table below reports this share under Pure GUI (no skills → multimodal skills; mean ± SD over three runs) and includes the three GUI-specialist models from W2:

Modelw/o skills (%)w/ skills (%)Change (pp)Skill read rate
Gemini 3.8 Flash43.0 ± 0.949.2 ± 0.7+6.297.5%
Claude Haiku 4.53.8 ± 0.35.0 ± 0.7+1.2≤ 23%
Qwen3-VL 235B1.0 ± 0.51.6 ± 0.1+0.6≤ 23%
UI-TARS-72B-DPO7.3 ± 1.27.2 ± 1.1−0.10%
UI-Venus-Navi-72B1.6 ± 0.61.1 ± 0.6−0.50.2%
GUI-Owl-1.5-32B-Think3.7 ± 0.74.2 ± 1.2+0.51.8%

Only Gemini 3.8 Flash gains beyond the run-to-run variation. We identify two bottlenecks:

  1. Weak models rarely consult skills. Claude Haiku 4.5 and Qwen3-VL read a skill in at most 23% of episodes, and the GUI specialists in at most 1.8%. For the specialists, the catalogue even changes behavior without delivering knowledge (W2).
  2. Mid-tier models read but cannot execute. Claude Sonnet 5 and Kimi K2.6 read skills in 74–96% of episodes, yet gain at most 2.2 pp even in the episodes where they read.

Making skills useful for weaker agents therefore requires agents that learn to look up and follow procedural knowledge; better skills alone are not enough.

Revisions. Takeaway 2 in Sec. 4.2 now states this limitation, and Appendix B.3 (Tab. 8) reports the completion measure for all models.

We hope these revisions address the reviewer's concerns, and we would be glad to clarify anything further.

AuthorsOpenReview ↗

Reply part 3

References

  • Qin et al. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv:2501.12326, 2025.
  • Gu et al. UI-Venus Technical Report: Building High-performance UI Agents with RFT. arXiv:2508.10833, 2025.
  • Xu et al. Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents. arXiv:2602.16855, 2026.
Reviewer_C6x1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Mobile Skills Bench evaluates mobile-agent skills on 145 tasks across 24 simulated apps built on MobileGym. Nine models are compared under Pure GUI and Hybrid execution, using no skills, textual skills or multimodal skills. Textual skills improve average success, while the tested reference images provide no consistent additional gain. The paper also compares three library organizations.

Critical strengths
  1. Backend verification checks both requested changes and unintended edits: 239 of the 873 checks cover unrelated records. This is useful for evaluating multi-app workflows.

  2. The comparisons use matched tasks and resettable app states. Textual and multimodal skills share procedural text, and the authors include vendor-harness and user-simulator checks alongside the nine-model evaluation.

Critical weaknesses
  1. The image result needs more explanation. Live screenshots and textual screen manuals are already available, so the reference images may supply little additional information. The analysis does not adequately separate this possibility from retrieval problems or ineffective image use. Some skill conclusions also depend on the harness: GPT-5.6 Sol's Pure GUI gain changes from -0.5 percentage points in the authors' harness to +11.0 in Codex. The appendix acknowledges this exception, but the main-text claim that only Gemini Flash models benefit substantially needs the same qualification. Paired intervals would also help interpret the small image effects.

  2. Readability needs improvement. The introduction is unnecessarily long, overlaps with Related Work, and spends too much space previewing findings before listing the contributions. Figure 1 has an AI-like illustrative style, with decorative graphics taking space away from readable technical detail. Its small labels and app screens are difficult to inspect. Figure 3 combines too many models, conditions and annotations in one panel. Shortening the introduction and simplifying these figures would make the paper much easier to follow.

Questions for authors
  1. Can you identify tasks where the reference images add information beyond live screenshots and screen manuals, and report their effect on those tasks?

  2. Which skill conclusions hold across harnesses? Can you add paired intervals for the main skill and image comparisons?

  3. Can you shorten the introduction and simplify Figures 1 and 3?

3 comments
AuthorsOpenReview ↗

Reply Part 1

We sincerely thank the reviewer for the careful and insightful review, and we are very grateful for the many concrete suggestions. They have substantially improved the paper. All changes made in response to this review are marked in blue in the revised PDF.


W1 / Q1. Do the reference images add information beyond the live screenshots and the screen manuals?

The images rarely help, for two reasons: few of them add information, and the agents seldom fail because they cannot recognize a screen. We traced each step from what the images contain to how they affect the outcome.

  • Few images add new information. We compared each of the 166 images with its skill text and with the live screenshot at the moment of delivery. Only 40 images add task-relevant information beyond both, and they account for 24% of all deliveries. Examples are icon-only buttons and filled versus outlined state icons. The rest add little because the shared library shows generic screens, whose controls the screen manuals already describe or the screenshot already labels.
  • Even where images add information, success does not change. On the 114 tasks whose skills contain such an image, adding images changes success by +1.4 ± 1.6 pp under Pure GUI and −0.2 ± 1.9 pp under Hybrid execution (95% paired intervals, pooled over the five skill-reading models). The agents receive these images whenever they read the corresponding skill, which happens in about 90% of episodes. Exposure is therefore limited by the same reading behavior that limits the textual skills.
  • Nor do the images change behavior. At a step that receives an image, the next action matches that of the paired textual-skill episode as often as at a step without one (82–86% versus 83%). In a manual review of 50 image-bearing steps, no image helped.
  • Failures rarely depend on recognizing the screen, which is the second reason. We attributed every failed episode to the stage that the agent was working on when the episode ended. About 23% of these episodes end with a premature completion claim, and about 70% end with the stage unfinished because a required screen is never reached, a value is wrong, or a change is not made. Control-level or visual problems decide an estimated 5–11% of failed episodes. Even in these episodes, the agent usually fails to operate a control that is already visible, such as a focused text field or a date picker. A screenshot of the same screen cannot fix such failures. Adding images leaves all of these shares unchanged.

Revisions. Appendix A.3 reports this analysis, and Takeaway 3 in Sec. 4.2 states the cause.


W1 / Q2. Which skill conclusions hold across harnesses? Can you add paired intervals?

Most conclusions hold across harnesses. Only the absolute success rates and the Pure GUI skill gain of GPT-5.6 Sol depend on the harness. The paired intervals support the qualified takeaway on textual skills and show no detectable effect of images.

Harness dependence. We agree that the main text stated a harness-dependent result too generally. Takeaway 2 now says that only the two Gemini Flash models gain substantially under Pure GUI in our harness, and that GPT-5.6 Sol gains 11.0 pp under Pure GUI in Codex.

To check which conclusions depend on the harness, we ran the same 145 tasks, user simulator, and verifier through the vendors' own agent harnesses. Claude Opus 5 ran in Claude Code and GPT-5.6 Sol in Codex, without skills and with textual skills. Because of budget constraints, each vendor-harness setting uses a single run, whereas our harness uses three.

ModelHarnessPure GUI, none → textHybrid, none → text
Claude Opus 5Ours20.2 → 21.6 (+1.4)46.2 → 52.9 (+6.7)
Claude Opus 5Claude Code24.1 → 24.8 (+0.7)49.0 → 55.2 (+6.2)
GPT-5.6 SolOurs9.9 → 9.4 (−0.5)34.3 → 42.5 (+8.2)
GPT-5.6 SolCodex15.9 → 26.9 (+11.0)42.8 → 56.6 (+13.8)
AuthorsOpenReview ↗

Reply part 2

The comparative conclusions hold in both kinds of harness:

  • Success remains limited. Even in the vendors' own harnesses, success stays below 57% under Hybrid execution and 27% under Pure GUI.
  • Hybrid execution outperforms Pure GUI in every model, harness, and skill condition.
  • Skills help more under Hybrid execution than under Pure GUI in all four model–harness pairs.
  • Without skills, Claude Opus 5 outperforms GPT-5.6 Sol in both kinds of harness.

Two results depend on the harness.

  • Absolute success rates. Both models score higher in their vendors' harnesses: Claude Opus 5 by 2–4 pp and GPT-5.6 Sol by 6–18 pp. This is expected. Each vendor harness is built and tuned for its own models, whereas our harness treats all nine models identically, as a fair comparison between models requires. We therefore compare models and skill conditions only within the same harness.
  • The skill gain of GPT-5.6 Sol under Pure GUI. Textual skills change its success by −0.5 pp in our harness but by +11.0 pp in Codex. In Codex, GPT-5.6 Sol opens SKILL.md and a procedure document in every episode. The size of a model's skill gain can therefore depend on the harness, but the comparative conclusions listed above hold in both.

The vendor harnesses were run only without skills and with textual skills, so the image comparison is evaluated in our harness alone.

Paired intervals. The intervals confirm that textual skills help reliably only in some settings, and that reference images have no detectable effect. As suggested, we computed 95% paired bootstrap intervals. For each task, we average each condition over its runs and take the difference. We then resample the 145 tasks 2,000 times. Pairing each task with itself removes the large variation in difficulty across tasks.

Effect of textual skills (textual skills − no skills):

ModelPure GUIHybrid
Gemini 3.8 Flash+9.2 ± 5.0+13.8 ± 5.6
Gemini 3.7 Flash+8.3 ± 5.5+15.2 ± 6.1
Claude Opus 5+1.2 ± 4.5+6.9 ± 4.8
Gemini 3.1 Pro+2.8 ± 4.1+0.9 ± 4.8
GPT-5.6 Sol−0.7 ± 4.3+8.0 ± 5.6
Claude Sonnet 5−1.2 ± 2.6+0.7 ± 4.4
Kimi K2.6+1.4 ± 3.8+1.4 ± 6.2
Claude Haiku 4.5+0.2 ± 0.4−0.7 ± 1.6
Qwen3-VL 235B0.0 ± 0.0+0.9 ± 1.1

Additional effect of reference images (multimodal skills − textual skills):

ModelPure GUIHybrid
Gemini 3.8 Flash+2.1 ± 3.1−3.2 ± 3.7
Gemini 3.7 Flash+1.1 ± 3.3−1.1 ± 3.8
Claude Opus 5−0.2 ± 2.8+0.2 ± 3.0
Gemini 3.1 Pro0.0 ± 2.5+1.1 ± 3.4
GPT-5.6 Sol+1.4 ± 2.6−2.5 ± 3.8
Claude Sonnet 5−0.2 ± 1.6+1.4 ± 2.9
Kimi K2.6−0.7 ± 4.2+0.7 ± 5.5
Claude Haiku 4.5+0.5 ± 1.2−0.5 ± 1.1
Qwen3-VL 235B0.0 ± 0.00.0 ± 1.5

Change in task success (pp) ± half-width of the 95% paired bootstrap confidence interval over the 145 tasks. Bold: the interval excludes zero.

  • Textual skills: under Pure GUI, the interval excludes zero only for the two Gemini Flash models. Under Hybrid execution, it also excludes zero for Claude Opus 5 and GPT-5.6 Sol. This supports the qualified Takeaway 2.
  • Reference images: all 18 intervals include zero. The image effect is therefore not detectable, and the negative values do not indicate that images hurt.

Revisions. Takeaway 2 in Sec. 4.2 is qualified as described above. The vendor-harness results are in Appendix C.3.2, and the paired intervals are in Tab. 4 of Appendix A.2.


W2 / Q3. Can you shorten the introduction and simplify Figures 1 and 3?

We made all three changes.

  • Introduction, about 30% shorter. We condensed the two paragraphs that surveyed prior benchmarks into one paragraph that keeps only the argument for a new benchmark, and moved their details to Related Work. We also folded the preview of findings into the contribution list, so the contributions now follow directly after MSB is introduced.
  • Figure 1, redrawn. All illustrations are removed, and the figure uses one consistent type scale. The labels and app screens are also larger and aligned.
  • Figure 3 (now Fig. 4), split. The figure now has one panel for Pure GUI and one for Hybrid execution. This removes the hatching, the rotated per-bar numbers, and the read-rate row. The exact values remain in Tab. 3, and the read rates are in Fig. 6.

Revisions. We made revisions in Sec. 1, Fig. 1, and Fig. 4 accordingly.

We hope these revisions address the reviewer's concerns, and we would be glad to clarify anything further.

Dear Reviewer 3NtAf,

Thank you for your time and thoughtful reviews. Please be reminded that the author-reviewer discussion period ends on November 18.

Since the authors have procvided their feedback, please review the feedback and actively participate in the discussion. If you have any remaining questions or concerns, please raise them before the deadline so the authors have sufficient time to respond. Thanks.

Best regards,

AC

Reviewer_NtAfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mobile Skills Bench (MSB), a new multimodal environment and benchmark for robustly and reproducibly evaluating mobile use agentic models on long-horizon tasks requiring user interaction. The environment comes with 73 "skills" that can help agents complete tasks more effectively. The authors evaluate 9 models, focusing particularly on how useful these skills are but also on differences in pure GUI vs hybrid (with MCP and CLI options) execution, different ways of organizing the skill library, etc. Models generally don't perform well on MSB and the authors make some interesting observations on the above research questions (and more).

Critical strengths

MSB appears to be the first mobile use benchmark that simultaneously satisfies multiple desiderata: user interaction, hybrid execution, long trajectories, deterministic evaluation, skills use, and multimodality. The formalism in section 3.2, even if short, is valuable and is often skipped in similar benchmark papers. The empirical findings are clearly stated and discussed (albeit with a bit of repetition, which could be reduced), and some of these are not exactly aligned with expectation, which is good. I suspect MSB will find good use by many agent developers.

Critical weaknesses

I see two main weaknesses, some of which can be addressed with some additional space (which the authors get for the camera-ready version).

(1) The paper is missing a clear running example, which makes it hard for the reader to ground exactly what the tasks look like, how might agents struggle on them, etc. Please add a running example in the main text! There is one in Appendix C, which I found very useful to actually understand what the tasks and failure modes look like. A succinct version of this example can be added to the main paper. Some space can be made by reducing some repetitiveness in how the the empirical findings are stated multiple times in the paper.

(2) Looking at the example revealed that the long-horizon tasks in MSB aren't long because a succinctly stated task is complex. This is the incorrect impression I got from the example in Fig 1, whose instruction simply states a super high level goal "Plan my trip and add it to my calendar." Instead, as Appendix C clarifies (and is alluded to in the task definition in section 3.2), the tasks are really specified step-by-step and build upon previous steps. This is a very different from long-horizon "autonomous" tasks (e.g., deep research) where a simply specified instruction leads to a really long agent trajectory. This has two implications:

  • The nature of the MSB tasks should be further clarified in the main text. Having a running example as noted above will help, and so will some stats about M_q, the number of follow-up request per task. The latter should go in section 3.3, which should also be expanded with more detail.

  • This makes the design choice of handling follow-up requests in a fixed manner, independent of intermediate task success, not ideal. If the agent gets one step wrong, a real user wouldn't blindly continue with a follow-up request, but the simulated user does. This can also bring down agent scores, as a single mistake can trigger a long chain of wasted follow-up work. This aspect is inherent in MSB and can't really be easily changed at this point. I would at least like the authors to address this limitation clearly in the main paper.

MINOR: Many of the motivations and design choices of the authors resemble that of a non-mobile agentic benchmark over day-to-day apps, namely AppWorld (Trivedi et al, 2024) and AppWorld-UL (Chen et al. 2026), which may be worth citing while clarifying that they target API-based agents, not mobile agents.

In RQ1 discussion in section 4.2, the reference to Tab. 2 and Fig. 4 is confusing, as the table isn't present in the main text and some of the takeaways (e.g., 48.5% success rate) seems to be coming from Fig 3. Please clarify/fix.

Questions for authors
  • Typically how many follow-up requests are there in each task?

  • How tightly are the outputs of a request coupled with the follow-up requests? E.g., if an agent gets one intermediate output wrong, is it pretty much guaranteed that it will fail the full task?

  • For RQ4, can you please clarify what is being measured (besides overall accuracy) with and without skills? What exactly is "step cost" and how does it relate to the number of skill calls?

2 comments
AuthorsOpenReview ↗

Reply Part 1

We sincerely thank the reviewer for the thorough and thoughtful review, and we truly appreciate the time and care devoted to our paper. The suggestions were very helpful. We respond to each point below. All changes are marked in red in the revised PDF.


W1: Running example

We agree that the paper is much easier to follow with a running example in the main text, and we have added one. Built on the eBay task of Appendix D, it shows the initial request and the three follow-ups together with the facts the agent must ask the user for at each stage. It also shows the intended outcomes, the verifier checks, and a typical failure without skills.

Revisions. The running example is in Sec. 3.2 (Fig. 3).


W2: Incremental requests and fixed follow-ups

In short:

  1. Task length does not come only from follow-ups. Tasks whose full goal is stated in the first message take as many steps as staged tasks.
  2. An earlier mistake is not penalized twice. A task counts as successful only if every check passes on the final state. An earlier mistake therefore causes the task to fail only once, regardless of what follows.
  3. Fixed follow-ups are a deliberate design. They let MSB measure what the agent accomplishes without correction by the user, as explained below.

Task structure. The long horizons in MSB do not come from the follow-ups alone. MSB contains two kinds of tasks. The 26 single-request tasks state the full goal in the first message, yet they take at least as many steps as the 119 staged tasks:

Single-request tasksStaged tasks
Number of tasks26119
RequestsFull goal in the first message1–5 follow-ups
Median steps of successful episodes (Pure GUI / Hybrid)*60 / 4442–58 / 22–45
  • For the four models with the highest success; the range for staged tasks spans tasks with one to three follow-ups.

Single-request tasks therefore test long-horizon execution on its own, while staged tasks additionally test whether the agent carries results across requests. We agree, however, that Fig. 1 understated how incremental many tasks are.

Why follow-ups do not depend on intermediate success. Fixed follow-ups let MSB judge the outcome the user cares about, keep the comparison between agents fair, and match how users delegate multi-app tasks.

(a) MSB judges the final state, which is what matters to the user. Judging each request on its own can be misleading. An agent may notice and fix an earlier problem while handling a later request, so a request that looks incomplete when reported can still end up correct. MSB therefore evaluates the final state of all apps after the last request.

Example. In the eBay task of Appendix D, an agent might order two pairs of trainers instead of three. When the user later asks for a return, the agent sees the wrong quantity on the order page and places an additional order for the missing pair. The final state is correct, although a check right after the purchase would have marked it as a failure.

(b) Fixed follow-ups keep the comparison fair. Every agent receives the same follow-up requests in the same order and with the same wording, and the simulator never helps the agent. The user simulator only answers the agent's questions with facts that the user holds but omitted from the request. It never explains how an app works, never points out mistakes, and never volunteers hints. A simulator that points out mistakes would help weaker agents more because they make more mistakes. The evaluation would then partly measure the simulator's guidance rather than the agent's own ability. Replacing the simulator with a different model leaves the main conclusions unchanged (Appendix C.3.3).

AuthorsOpenReview ↗

Reply Part 2

(c) Users rarely check each request in complex tasks. When a task spans several apps, users usually do not verify every request before giving the next one, and many outcomes are hard for them to see. In 69.7% of MSB tasks, the results are spread across two or more apps. Moreover, 239 of the 873 verifier checks concern records that should stay unchanged, such as other conversations or orders. Users would not normally inspect these records. Our simulator likewise sees only the conversation.

Example. A user asks the agent to rebook a train ticket, then to schedule a meeting after the new arrival time, and then to add the meeting to the calendar. A real user would not open the railway app to confirm the rebooking before asking for the meeting. They rely on the agent's report and expect the ticket, the meeting, and the calendar to be correct in the end.

This setup follows established multi-turn benchmarks. In MT-Bench (Zheng et al., 2023), Multi-IF (He et al., 2024), and BFCL (Patil et al., 2025), the follow-up turns are predefined rather than conditioned on the model's earlier responses. Likewise, τ-bench (Yao et al., 2025) and AppWorld (Trivedi et al., 2024) judge success from the final environment state.

Revisions. The caption of Fig. 1 now states that requests arrive in stages, Sec. 3.3 distinguishes the two kinds of tasks, and Fig. 2c gives the number of follow-ups per task. Sec. 3.2 states the fixed-follow-up design and its rationale.


Questions

Q1: Number of follow-up requests. Each task has 1.5 follow-up requests on average, distributed as follows:

Follow-up requests01234–5
Number of tasks264452212

Revisions. This distribution is now in Fig. 2c and Sec. 3.3.

Q2: Coupling between outputs and follow-ups. In practice, the two are tightly coupled. In our runs, an incorrect intermediate output almost always makes the task fail. Many follow-ups refer to the results of earlier requests, and success requires the final state of every app to be correct. In principle, however, the task does not fail automatically: an agent can still correct an earlier mistake while handling a later request, and the episode then succeeds. This coupling is intended, since MSB evaluates whether the user's overall goal is achieved rather than each request in isolation.

Q3: What RQ4 measures. RQ4 measures how many steps an episode takes, and reading a skill counts as a step.

  • Besides success: RQ4 compares the mean number of steps per episode with and without skills. The mean covers all episodes, including failed ones. Appendix A.4 also reports the time and inference cost per episode. It further compares paired runs of the same task to separate tasks solved in both conditions, tasks failed in both, and tasks newly solved with skills.
  • Step cost: a step is one model call, which produces one action of the agent. An action can be a tap, a tool call, a question to the user, or a shell command. Step cost is the number of such steps per episode.
  • Relation to skill invocations: reading a skill document is itself a shell command, so each read takes one step and counts toward the same 100-step budget. Skill reads are therefore part of the step cost.

Revisions. The RQ4 paragraph of Sec. 4.2 now states this definition.


Minor points

  • Related work. We have cited AppWorld and AppWorld-UL in Sec. 2 and clarified that they target API-based agents, whereas MSB targets mobile agents that act through the GUI, MCP tools, and CLI commands.
  • References in RQ1. The 48.5% figure comes from the main results figure (Fig. 3 in the submission, Fig. 4 in the revision), and the corresponding table is Tab. 2 in Appendix A.1. We have corrected the references in the RQ1 paragraph accordingly.

We hope these revisions address the reviewer's concerns, and we would be glad to clarify anything further.


References

  • Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks Track, 2023.
  • He et al. Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following. arXiv:2410.15553, 2024.
  • Patil et al. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. ICML, 2025.
  • Yao et al. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. ICLR, 2025.
  • Trivedi et al. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. ACL, 2024.

Comments on the paper

1 comment

We sincerely thank the AC and all reviewers for their time and constructive feedback, which have helped us make the paper considerably stronger. The main concerns and our revisions are listed below. Changes are marked in red (NtAf), blue (C6x1), and purple (Yce5) in the revised PDF.

1. Presentation and clarity

  • Related work and references (NtAf). We cited AppWorld and AppWorld-UL and fixed the figure and table references in RQ1 (Sec. 2, Sec. 4.2).
  • Introduction and figures (C6x1). We shortened the introduction by about 30%, redrew Fig. 1, and split the main results figure into one panel per execution mode (Sec. 1, Fig. 1, Fig. 4).
  • Step cost in RQ4 (NtAf). We defined a step as one model call and stated that each skill read counts as one step (Sec. 4.2).
  • Running example (NtAf). We added the eBay task to the main text, with its requests, the facts the agent must ask for, the verifier checks, and a typical failure (Sec. 3.2, Fig. 3).
  • Incremental task structure (NtAf). We clarified that requests arrive in stages and added the number of follow-ups per task. The 26 single-request tasks are as long as the staged ones, so the long horizon does not come from follow-ups alone (Fig. 1, Fig. 2c, Sec. 3.3).

2. Robustness of the skill findings

  • Skills derived from the simulator's source code (Yce5). We rebuilt the entire skill library from recorded trajectories alone, without source access. These skills still raise the success of Gemini 3.8 Flash in all four settings, by +6.0 pp on average (App. B.1).
  • Simulated apps only (Yce5). We repeated seven tasks with real Android apps on an emulator. Skills raise success from 38% to 52–57%, similar to the gain in the simulator (App. B.4).
  • Paired intervals (C6x1). We added 95% paired bootstrap intervals for all skill and image comparisons. Textual skills help significantly for four models under Hybrid execution (App. A.2, Tab. 4).

3. Scope of the findings

  • Fixed follow-ups (NtAf). We now state explicitly that this is a deliberate design: MSB judges the final state, and fixed follow-ups keep the comparison between agents fair (Sec. 3.2).
  • Harness dependence (C6x1). We ran Claude Opus 5 in Claude Code and GPT-5.6 Sol in Codex. The comparative conclusions hold in both kinds of harness, and Takeaway 2 is now qualified for the Pure GUI gain of GPT-5.6 Sol (App. C.3.2).
  • Value of reference images (C6x1). We traced each image from its content to its effect on the outcome. Few images add information beyond the screenshot and the screen manual, and Takeaway 3 now states this cause (App. A.3).
  • GUI-specialist models (Yce5). We evaluated UI-TARS, UI-Venus, and GUI-Owl on all tasks. They solve almost no tasks, and skills do not help them because they rarely read the skills (App. B.2).
  • Benefit for weaker models (Yce5). We added a finer measure, the share of required checks completed, for all models. The benefit is largely limited to capable agents, because weak models rarely read skills and mid-tier models cannot carry them out (App. B.3).

We look forward to further discussion with the reviewers and are happy to address any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.