- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces MobiGraph, a mobile GUI agent benchmark that constructs interactive, task-oriented state graphs by fusing multiple human operation trajectories. It supports multiple valid execution paths without relying on system-level APIs and covers 240 tasks across 20 third-party applications. The authors introduce additional evaluation metrics and specialized scenarios to assess different agent capabilities. Experiments on general-purpose and GUI-specialized models demonstrate meaningful performance differences, and the results show stronger alignment with human assessments than AndroidWorld.
Critical strengths
-
The paper addresses the limitations of existing mobile agent benchmarks in evaluating real world third party applications. By collecting and merging multiple human operation trajectories, MobiGraph constructs state transition graphs that support dynamic interaction and multiple valid task completion paths without relying on system APIs.
-
MobiGraph covers 240 tasks across 20 applications and introduces diverse evaluation metrics and specialized scenarios. Experiments on general models and GUI models reveal performance differences across various scenarios. The reported higher alignment with human assessments supports its practical value. The release of code and data further facilitates reproducibility and future research.
Critical weaknesses
-
The paper constructs task state transition graphs by merging states with identical semantic labels, but provides limited validation of the correctness of this process. States with similar interface structures may not necessarily share the same transition relations, potentially introducing execution paths that are invalid in real applications. Further analysis of state merging accuracy and the validity
-
Although the paper provides comprehensive evaluations across different models and scenarios, it lacks sufficient ablation studies on the core trajectory fusion method. In particular, compared with evaluating multiple reference trajectories independently, the paper does not sufficiently demonstrate the improvements brought by trajectory fusion in terms of valid execution path coverage and evaluation effectiveness. This makes it difficult to assess the actual contribution of trajectory fusion to the effectiveness of the benchmark.
Questions for authors
- How do the authors ensure that states with identical semantic labels can safely share the same transition relations? Have the authors conducted any validation to verify that the execution paths introduced by trajectory fusion remain valid in real applications?
- Could the authors provide an ablation study comparing trajectory fusion with evaluating multiple reference trajectories independently? In particular, how much does trajectory fusion improve valid execution path coverage and evaluation effectiveness compared with using multiple trajectories without fusion?