acceptodds
Under review as a conference paper at ICLR 2027

MobiGraph: Towards a Practical Evaluation Paradigm for Mobile Agents via Trajectory-Fused State Graphs

Abstract

Mobile agents can autonomously complete user-assigned tasks through GUI interactions. However, existing mainstream evaluation benchmarks, such as AndroidWorld, operate by connecting to a system-level Android emulator and provide evaluation signals based on the state of system resources. In real-world mobile agent scenarios, many third-party applications do not expose system-level APIs to determine whether a task has succeeded, leading to a mismatch between benchmarks and real-world usage. To address these issues, we propose MobiGraph, an evaluation framework that supports task construction for arbitrary third-party apps. Using an efficient graph-construction algorithm based on multi-trajectory fusion, MobiGraph can effectively compress the state space, support dynamic interaction, and better align with real-world third-party application scenarios. MobiGraph covers 20 widely used third-party applications and comprises 240 diverse real-world tasks, with enriched evaluation metrics. Compared with AndroidWorld, MobiGraph 's evaluation results show higher alignment with human assessments and can guide the evolution of future GUI-based models under real workloads.

Primary area:
datasets and benchmarks
Keywords:
Benchmarking, Evaluation Framework, GUI-Agent, Smartphone Control, MLLM, Multimodal model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9JzMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MobiGraph, a mobile GUI agent benchmark that constructs interactive, task-oriented state graphs by fusing multiple human operation trajectories. It supports multiple valid execution paths without relying on system-level APIs and covers 240 tasks across 20 third-party applications. The authors introduce additional evaluation metrics and specialized scenarios to assess different agent capabilities. Experiments on general-purpose and GUI-specialized models demonstrate meaningful performance differences, and the results show stronger alignment with human assessments than AndroidWorld.

Critical strengths
  1. The paper addresses the limitations of existing mobile agent benchmarks in evaluating real world third party applications. By collecting and merging multiple human operation trajectories, MobiGraph constructs state transition graphs that support dynamic interaction and multiple valid task completion paths without relying on system APIs.

  2. MobiGraph covers 240 tasks across 20 applications and introduces diverse evaluation metrics and specialized scenarios. Experiments on general models and GUI models reveal performance differences across various scenarios. The reported higher alignment with human assessments supports its practical value. The release of code and data further facilitates reproducibility and future research.

Critical weaknesses
  1. The paper constructs task state transition graphs by merging states with identical semantic labels, but provides limited validation of the correctness of this process. States with similar interface structures may not necessarily share the same transition relations, potentially introducing execution paths that are invalid in real applications. Further analysis of state merging accuracy and the validity

  2. Although the paper provides comprehensive evaluations across different models and scenarios, it lacks sufficient ablation studies on the core trajectory fusion method. In particular, compared with evaluating multiple reference trajectories independently, the paper does not sufficiently demonstrate the improvements brought by trajectory fusion in terms of valid execution path coverage and evaluation effectiveness. This makes it difficult to assess the actual contribution of trajectory fusion to the effectiveness of the benchmark.

Questions for authors
  1. How do the authors ensure that states with identical semantic labels can safely share the same transition relations? Have the authors conducted any validation to verify that the execution paths introduced by trajectory fusion remain valid in real applications?
  2. Could the authors provide an ablation study comparing trajectory fusion with evaluating multiple reference trajectories independently? In particular, how much does trajectory fusion improve valid execution path coverage and evaluation effectiveness compared with using multiple trajectories without fusion?
Reviewer_FLdnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MobiGraph, an evaluation framework for mobile GUI agents designed to bridge the gap between online emulators and static offline benchmarks. By introducing Trajectory-Fused State Graphs (TCSGs), the authors merge human interaction trajectories sharing identical semantic labels to construct task-oriented state-transition graphs. This enables offline multi-path exploration and dynamic feedback without relying on system-level APIs. The benchmark evaluates various general LLMs and GUI-specialized models across 20 third-party applications with 240 tasks, covering a base scenario and four specialized variants.

Critical strengths

Innovative Offline Graph Modeling for Multi-Path Execution: The key technical contribution lies in constructing Trajectory-Fused State Graphs (TCSGs) from offline human trajectories. By filtering non-task UI elements and merging semantically equivalent states, MobiGraph enables dynamic, multi-path interaction in an offline environment without reliance on system-level APIs or emulators.

Critical weaknesses
  • Limited Benchmark Task Scale Compared to Established Offline Datasets: Mobile GUI agent evaluation, especially offline evaluation, is a well-studied topic where the primary advantage is low execution cost and high speed, allowing for massive data scaling. However, MobiGraph contains only 240 tasks across 20 apps. Compared to existing offline benchmarks (e.g., AITW with thousands of trajectories), this small task scale undermines the core benefit of offline benchmarking.

  • Divergence from Industry Trends and Lack of Novel Insights: Current GUI agent research is rapidly moving toward real-time online environments for large-scale reinforcement learning (RL) and evaluation. Simulating online environments via small-scale offline graphs offers diminishing utility in this context. Furthermore, the empirical findings—such as GUI-specialized models outperforming general LLMs on basic operations while general LLMs show better robustness in complex reasoning—largely align with existing industry consensus and provide limited new guidance for future agent design.

Questions for authors
  • How do the authors plan to scale up the task suite to compete with large-scale offline datasets, and what is the human annotation overhead required for defining semantic tags across diverse apps?

  • Given the strong trend toward training and evaluating GUI agents in real-time online environments (e.g., online RL), how can an offline graph-based evaluation framework continue to provide unique and indispensable value?

Reviewer_GWjQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MobiGraph constructs interactive evaluation environments by fusing multiple human demonstrations into task-specific state-transition graphs. It aims to support third-party applications and multiple completion paths without requiring a live Android emulator. The benchmark covers 20 applications and 240 tasks, with metrics for success, progress, coverage, and efficiency, as well as specialized evaluation scenarios. The authors evaluate general-purpose and GUI-specialized models and report stronger alignment with user feedback than AndroidWorld.

Critical strengths
  • Task-focused multi-trajectory fusion offers a practical trade-off between single-reference offline evaluation and expensive live-device evaluation. Sharing transitions across equivalent states can provide richer interaction coverage without constructing an entire application graph.
  • The benchmark covers third-party applications and exposes several dimensions of execution quality. The human-alignment study is also a valuable direction for validating whether benchmark scores reflect practical agent performance.
Critical weaknesses
  • Correctness within the constructed graph is distinct from fidelity to the underlying application. For actions outside the collected transitions, keeping the interface unchanged, displaying a blank screen, or presenting a predefined prompt can change error recovery and exclude valid but uncollected strategies. The paper does not sufficiently establish whether these approximations preserve agent rankings.
  • The evidence for task-oriented completeness is limited. Multiple successful trajectories can provide broad coverage without establishing that all relevant solutions or state distinctions have been represented. The reported model-level alignment with user feedback does not directly resolve this issue or establish that state merging preserves task-relevant behavior.
Questions for authors
  • For independently collected successful trajectories excluded from construction, what fractions of actions and complete trajectories are supported by the graph?
  • How are visually or semantically similar states distinguished when their underlying business states imply different valid transitions or outcomes?
  • Can you provide uncertainty estimates and matched evaluation conditions for the human-alignment comparison? In particular, how do harness differences and graph coverage affect the comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.