SimuWoB: Simulating Real-World Mobile Apps for Fast and Faithful GUI Agent Benchmarking

Mobile GUI agents powered by large language models have progressed rapidly, creating a growing need for realistic, challenging, and efficient evaluation. Existing interactive benchmarks often rely on open-source apps, file-centric tasks, or manually engineered reward functions, limiting their coverage of real-world app semantics and complex, long-horizon workflows. To address these limitations, we introduce SimuWoB, a fully synthetic benchmark comprising 120 challenging tasks across 63 simulated mobile applications. We develop a two-stage environment synthesis framework that constructs interactive mobile-style applications and injects task-specific logic together with executable state-based validators. Each environment is deployed as a backend-free webpage accessible via URL, enabling lightweight, reproducible, and parallelizable evaluation. Across five recent mobile GUI agents, the average success rate is only 27.92%, dropping to 17.82% on long-horizon tasks, indicating substantial room for improvement on complex workflows. Evaluation result comparison with real-world sample tasks demonstrate that agent assessments based on our synthetic environment generalize well. We further provide diagnostic insights across key capability dimensions and discuss implications for future mobile GUI agent development.