- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies mobile GUI agents in real-world settings, focusing on robustness and the trade-off between accuracy and latency. Using Qwen3-VL-8B as the base model, the authors examine trajectory diversity, anomaly-based data augmentation, instruction granularity, reasoning strategies, screenshot resolution, and agent workflows on a custom evaluation suite covering 54 applications across Android and HarmonyOS. The combined data construction strategy improves task success rate by 10.2 percentage points. Based on these results, the authors offer recommendations for training data and inference configurations intended for deployment.
Critical strengths
The paper seeks to address practically relevant challenges, particularly changing application states and response latency in real-world mobile use. The experiments examine several aspects of data construction and inference design, reporting both success rate and latency. The findings provide some useful engineering guidance for the deployment setting studied.
Critical weaknesses
- The introduction does not fully develop the argument connecting the challenges to the proposed designs. It identifies several challenges in real-world deployment and outlines corresponding design choices, but largely presents experimental conclusions in place of explaining the underlying technical rationale. It does not sufficiently explain how these choices address the identified challenges or why they should improve upon existing approaches.
- The contribution beyond prior work is not clearly established. The choices studied—including trajectory diversification, anomaly augmentation, mixed-granularity instructions, and image resolution—are largely straightforward engineering decisions. Prior work such as UI-TARS [1] and OS-ATLAS [2] has already explored related issues in GUI data construction, generalization, and reasoning design, yet the paper does not adequately distinguish its contributions from these studies. As an empirical study, it also leaves unclear which findings go beyond established practice, challenge existing assumptions, or reveal previously unrecognized patterns. The research contribution is therefore difficult to assess.
- The absence of external baselines makes the practical significance of the improvements difficult to judge. All experimental comparisons are between configurations trained by the authors using Qwen3-VL-8B. Representative models or systems such as UI-TARS[1], Mobile-Agent[3], and ScaleCUA[4] are not evaluated directly. These internal ablations can establish improvements over the authors’ own baseline, but do not show whether the same choices benefit existing strong methods. They also leave the final system’s accuracy and latency relative to existing approaches unknown.
- There is no validation on public benchmarks, and the custom evaluation is not sufficiently reproducible. All results are obtained on the authors’ own evaluation suite, with no results reported on public benchmarks such as AndroidWorld. This makes comparisons with prior work difficult. The manuscript also provides insufficient detail or resources concerning the task list, data splits, environment versions, and evaluation implementation to support independent reproduction, limiting the verifiability of the results.
- The evaluation does not adequately support the claims of robustness in real-world online environments. Of the 295 evaluation tasks, 80% are assessed through a trajectory-fusion-based UI state-transition framework, while only 20% are executed on real devices. Although trajectory-fusion evaluation uses data collected from real applications, it is not equivalent to live interaction. The paper does not sufficiently establish how well this framework covers dynamic UI changes, runtime anomalies, or valid execution paths absent from the recorded trajectories. Nor does it report agreement between this evaluation and execution on real devices. The evidence for robustness under actual deployment conditions therefore remains limited.
[1] Qin, Y., Ye, Y., Fang, J., et al. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv preprint arXiv:2501.12326, 2025.
[2] Wu, Z., Wu, Z., Xu, F., et al. OS-ATLAS: Foundation Action Model for Generalist GUI Agents. International Conference on Learning Representations, 2025, pp. 5090–5108.
[3] Ye J, Zhang X, Xu H, et al. Mobile-agent-v3: Fundamental agents for gui automation[J]. arXiv preprint arXiv:2508.15144, 2025.
[4] Liu Z, Xie J J, Ding Z, et al. Scalecua: Scaling open-source computer use agents with cross-platform data[C]//International Conference on Learning Representations. 2026, 2026: 62707-62766.
Questions for authors
- What is the clearest new finding beyond existing practices in data construction and efficiency optimization for GUI agents?
- Why are results on public benchmarks such as AndroidWorld not provided? Can the authors release the 295 evaluation tasks, training and test splits, application and operating system versions, and the extended MobiFlow evaluation resources? If the data cannot be released, what alternative resources could support independent reproduction?
- What model, training data, and configuration does 'Existing Approaches' in Figure 1 refer to? Can the authors evaluate representative GUI agents in the same environment under matched latency or compute budgets?