acceptodds
Under review as a conference paper at ICLR 2027

From Benchmarks to the Wild: Practical Design Choices for Deployable Mobile GUI Agents

Abstract

Recent years have witnessed rapid advances in large multimodal model (LMM)-based mobile GUI agents that have achieved strong performance on related benchmarks such as ScreenSpot and AndroidWorld. However, it remains challenging to deploy such benchmark-capable agents in real-world scenarios where they must operate under dynamic UI states, diverse user instructions, third-party application variations, and strict latency budgets that existing benchmarks often underrepresent. To investigate how to bridge this gap, we present an empirical study of practical mobile GUI agent design through deployment-oriented experiments across Android and HarmonyOS, covering 50+ mainstream applications. Our analysis focuses on two important dimensions: data construction strategies that make agents robust beyond benchmark-like settings, and latency-aware designs that keep accurate agents responsive as well. For robustness, to handle the wide variation of user inputs and environment states in real use, we propose three complementary data construction schemes that reflect diverse, realistic workloads and transfer GUI agents from fixed benchmarks to dynamic scenarios, improving real-world task success rate by 10.2 percentage points. For efficiency, we find that heavy reasoning or multi-agent workflows emphasized by existing agent paradigms can introduce unacceptable latency; by streamlining agent input/output and workflows, we identify the Pareto-optimal inference-time designs for balancing task accuracy and latency. Together, these findings provide practical guidelines for building mobile GUI agents that are robust, efficient, and deployable in real-world environments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Models, GUI Agents, Agent Deployability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vTW4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies mobile GUI agents in real-world settings, focusing on robustness and the trade-off between accuracy and latency. Using Qwen3-VL-8B as the base model, the authors examine trajectory diversity, anomaly-based data augmentation, instruction granularity, reasoning strategies, screenshot resolution, and agent workflows on a custom evaluation suite covering 54 applications across Android and HarmonyOS. The combined data construction strategy improves task success rate by 10.2 percentage points. Based on these results, the authors offer recommendations for training data and inference configurations intended for deployment.

Critical strengths

The paper seeks to address practically relevant challenges, particularly changing application states and response latency in real-world mobile use. The experiments examine several aspects of data construction and inference design, reporting both success rate and latency. The findings provide some useful engineering guidance for the deployment setting studied.

Critical weaknesses
  1. The introduction does not fully develop the argument connecting the challenges to the proposed designs. It identifies several challenges in real-world deployment and outlines corresponding design choices, but largely presents experimental conclusions in place of explaining the underlying technical rationale. It does not sufficiently explain how these choices address the identified challenges or why they should improve upon existing approaches.
  2. The contribution beyond prior work is not clearly established. The choices studied—including trajectory diversification, anomaly augmentation, mixed-granularity instructions, and image resolution—are largely straightforward engineering decisions. Prior work such as UI-TARS [1] and OS-ATLAS [2] has already explored related issues in GUI data construction, generalization, and reasoning design, yet the paper does not adequately distinguish its contributions from these studies. As an empirical study, it also leaves unclear which findings go beyond established practice, challenge existing assumptions, or reveal previously unrecognized patterns. The research contribution is therefore difficult to assess.
  3. The absence of external baselines makes the practical significance of the improvements difficult to judge. All experimental comparisons are between configurations trained by the authors using Qwen3-VL-8B. Representative models or systems such as UI-TARS[1], Mobile-Agent[3], and ScaleCUA[4] are not evaluated directly. These internal ablations can establish improvements over the authors’ own baseline, but do not show whether the same choices benefit existing strong methods. They also leave the final system’s accuracy and latency relative to existing approaches unknown.
  4. There is no validation on public benchmarks, and the custom evaluation is not sufficiently reproducible. All results are obtained on the authors’ own evaluation suite, with no results reported on public benchmarks such as AndroidWorld. This makes comparisons with prior work difficult. The manuscript also provides insufficient detail or resources concerning the task list, data splits, environment versions, and evaluation implementation to support independent reproduction, limiting the verifiability of the results.
  5. The evaluation does not adequately support the claims of robustness in real-world online environments. Of the 295 evaluation tasks, 80% are assessed through a trajectory-fusion-based UI state-transition framework, while only 20% are executed on real devices. Although trajectory-fusion evaluation uses data collected from real applications, it is not equivalent to live interaction. The paper does not sufficiently establish how well this framework covers dynamic UI changes, runtime anomalies, or valid execution paths absent from the recorded trajectories. Nor does it report agreement between this evaluation and execution on real devices. The evidence for robustness under actual deployment conditions therefore remains limited.

[1] Qin, Y., Ye, Y., Fang, J., et al. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv preprint arXiv:2501.12326, 2025.

[2] Wu, Z., Wu, Z., Xu, F., et al. OS-ATLAS: Foundation Action Model for Generalist GUI Agents. International Conference on Learning Representations, 2025, pp. 5090–5108.

[3] Ye J, Zhang X, Xu H, et al. Mobile-agent-v3: Fundamental agents for gui automation[J]. arXiv preprint arXiv:2508.15144, 2025.

[4] Liu Z, Xie J J, Ding Z, et al. Scalecua: Scaling open-source computer use agents with cross-platform data[C]//International Conference on Learning Representations. 2026, 2026: 62707-62766.

Questions for authors
  1. What is the clearest new finding beyond existing practices in data construction and efficiency optimization for GUI agents?
  2. Why are results on public benchmarks such as AndroidWorld not provided? Can the authors release the 295 evaluation tasks, training and test splits, application and operating system versions, and the extended MobiFlow evaluation resources? If the data cannot be released, what alternative resources could support independent reproduction?
  3. What model, training data, and configuration does 'Existing Approaches' in Figure 1 refer to? Can the authors evaluate representative GUI agents in the same environment under matched latency or compute budgets?
Reviewer_5VwgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies practical design choices for mobile GUI agents, focusing on robustness and response speed. The authors train Qwen3-VL-8B using different execution paths, UI anomaly augmentation, and mixtures of goal and procedure instructions. They also compare reasoning depth, screenshot resolution, and agent workflows. The data covers 54 apps across Android and HarmonyOS, with evaluation on 295 tasks. Most tests use fused recorded trajectories; the rest use real devices and human judgment. The authors report a 10.2% point success improvement and find that moderate augmentation, observation reasoning, and resized screenshots offer a useful balance of accuracy and latency.

Critical strengths

Collecting data across 54 apps on Android and HarmonyOS makes the study relevant to practical phone use. The comparison using equally sized training sets gives useful evidence that alternative execution paths improve transfer and robustness.

The paper measures accuracy alongside latency and compares concrete choices such as reasoning depth and image resolution. This is useful for building responsive agents, especially when heavier inference brings only small accuracy gains.

Critical weaknesses

Most evaluation uses recorded UI transitions, while details of the user-feedback comparison are thin. The paper does not clearly connect these measurements to the reported improvements in live use, which weakens its central deployment claim.

Mean latency on one A100 is not enough to verify the stated 1.5s response target. Tail latency and communication overhead are not reported, so the evidence for responsiveness in actual deployment remains incomplete.

Questions for authors

Which gains in Figure 1 were measured through live execution, and how were the user-feedback groups selected and compared?

Do the latency numbers include network and serving delays? What fraction of steps meet the 1.5-second target?

Reviewer_sXGcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical study of mobile GUI agent design using Qwen3-VL-8B, with data collected across 54 applications on Android and HarmonyOS and evaluation on 295 tasks. The proposed data strategies increase task success rate from 53.9% to 64.1%. The inference experiments examine reasoning depth, screenshot resolution, and workflow decomposition, identifying a configuration combining observation-based reasoning, moderate screenshot resolution, and an end-to-end workflow with an average per-step latency of 1.53 seconds.

Critical strengths

The paper systematically examines several practical design choices within a common experimental framework, including trajectory diversity, data augmentation, instruction granularity, reasoning depth, screenshot resolution, and workflow decomposition. Reporting both task success rate and per-step latency makes the trade-offs concrete and provides useful implementation guidance for practitioners working under similar deployment constraints.

Critical weaknesses
  1. Limited novelty and incremental contribution. The benefits of task and environment diversity for GUI-agent generalization have already been discussed in technical reports such as Qwen-UI-Agent, UI-Venus-1.5, Holo3, and Fara1.5 [1–4]. Although the paper provides controlled comparisons of specific design choices, it does not sufficiently establish what new understanding these experiments contribute beyond existing findings. The resulting recommendations on data construction and inference configuration are useful engineering observations, but their incremental scientific contribution appears limited.
  2. Insufficient evidence for the deployment claims. The central conclusions rely on a custom evaluation set of 295 tasks, but the paper provides insufficient concrete task examples and construction details to independently assess its difficulty, coverage, and separation from the training data. Moreover, 80% of the evaluation uses a trajectory-derived state-transition environment, limiting the evidence for reliability under uncontrolled real-world conditions. All experiments use a single base model, Qwen3-VL-8B, leaving the generality of the recommendations unclear. Finally, the recommended configuration’s average per-step latency of 1.53 seconds does not demonstrate compliance with the stated 1.5-second target, and average inference latency alone does not establish responsiveness under production serving conditions.

[1] Hanzhang Zhou et al. Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents. arXiv:2607.28227v1, 2026.

[2] Venus Team et al. UI-Venus-1.5 Technical Report. arXiv:2602.09082v2, 2026.

[3] H Company. Holo3: Open Foundation Models for Navigation and Computer Use Agents. 2026.

[4] Awadallah et al. Fara1.5: Scalable Learning Environments for Computer Use Agents. arXiv:2606.20785, 2026.

Questions for authors
  1. What specific findings go beyond the conclusions about task and environment diversity already discussed in [1–4]? Please distinguish new empirical insights from confirmation of existing recommendations.
  2. Could you provide representative evaluation tasks, including instructions, initial screenshots, valid execution paths, and success criteria? Please also clarify how training trajectories, evaluation graph-construction data, and test tasks are separated, and how actions outside the constructed state graph are handled.
  3. Do the principal findings hold for stronger or more recent base models, or for an independently developed GUI-specialized model? Additional evidence would help determine whether the reported trade-offs are general or specific to Qwen3-VL-8B.
  4. What evidence demonstrates that the proposed configuration meets the stated production latency target? Please clarify whether the measurements include network communication, action execution, and application response time, and whether latency under concurrent load or tail latency was evaluated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.