acceptodds
Under review as a conference paper at ICLR 2027

SimuWoB: Simulating Real-World Mobile Apps for Fast and Faithful GUI Agent Benchmarking

Abstract

Mobile GUI agents powered by large language models have progressed rapidly, creating a growing need for realistic, challenging, and efficient evaluation. Existing interactive benchmarks often rely on open-source apps, file-centric tasks, or manually engineered reward functions, limiting their coverage of real-world app semantics and complex, long-horizon workflows. To address these limitations, we introduce SimuWoB, a fully synthetic benchmark comprising 120 challenging tasks across 63 simulated mobile applications. We develop a two-stage environment synthesis framework that constructs interactive mobile-style applications and injects task-specific logic together with executable state-based validators. Each environment is deployed as a backend-free webpage accessible via URL, enabling lightweight, reproducible, and parallelizable evaluation. Across five recent mobile GUI agents, the average success rate is only 27.92%, dropping to 17.82% on long-horizon tasks, indicating substantial room for improvement on complex workflows. Evaluation result comparison with real-world sample tasks demonstrate that agent assessments based on our synthetic environment generalize well. We further provide diagnostic insights across key capability dimensions and discuss implications for future mobile GUI agent development.

Primary area:
datasets and benchmarks
Keywords:
Mobile GUI Agents, Benchmark, Environment Synthesis, State-Based Task Validation, Long-Horizon Tasks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jxFdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents SimuWoB, a synthetic benchmark for mobile GUI agents. The authors build a two‑stage synthesis pipeline to create 63 simulated mobile apps with 120 tasks, plus state‑based automatic validators. All environments run as simple web URLs for reproducible, parallel testing. They evaluate five modern GUI agents, reporting 27.92% overall success and just 17.82% for long‑horizon tasks. A small comparison against real‑world mobile examples is used to argue reasonable transferability, alongside diagnostic analysis pointing out agent capability gaps.

Critical strengths
  1. Practical benchmark design. Synthetic, backend‑free web environments with built‑in reward validators remove much of the overhead of testing on real mobile devices or APKs, which is a real pain point for GUI‑agent researchers.
  2. Clear empirical observation: existing SOTA agents perform poorly on long‑horizon mobile workflows. The diagnostic breakdown gives useful directions for follow‑up work.
Critical weaknesses
  1. Weak generalization evidence. Only limited comparison against real‑world mobile cases is provided. It remains unclear whether SimuWoB’s synthetic UI logic and task difficulty properly mirror real‑world mobile apps. Large distribution mismatch would mean benchmark results do not reliably predict real‑phone performance.
  2. Limited transparency for benchmark construction. Little information is given on how the 63 apps and 120 tasks were sampled. Without knowing sampling rules, it is hard to verify task diversity and rule out hidden biases in synthetic task generation.
Questions for authors
  1. Can you describe your app and task sampling strategy? How do you guarantee synthetic tasks reflect realistic user intents and common mobile UI patterns?
  2. Your real‑world transfer evaluation is small. Could you provide quantitative measurements of the distribution gap between SimuWoB and real‑world mobile tasks?
  3. What failure modes exist for your state‑based validator? Can an agent pass state checks without actually completing the intended user goal?
Reviewer_3UVwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SimuWoB, a synthetic benchmark containing 120 tasks across 63 simulated mobile applications. Its two-stage construction pipeline builds interactive mobile-style applications and then injects task-specific data, logic, and executable state-based validators. The environments run as backend-free webpages, and evaluated agents interact through screenshots and mobile-style actions. The paper evaluates five agents, reports human performance, and examines long-horizon, information-processing, and fine-grained interaction failures. It also compares agent performance with an independent real-app validation set and reports strong correlation at the agent-by-category level.

Critical strengths
  1. A practical approach to constructing controllable interactive environments. Sections 3.2 and 3.4 combine application synthesis, task injection, state-based evaluation, and lightweight reset interfaces. Agents receive screenshots rather than privileged DOM access, and evaluator-side functions are hidden from them. This provides a useful way to study mobile-style workflows under controlled conditions, and Section 2 acknowledges the tradeoff between native-system fidelity and control over benchmark construction.
  2. Meaningful quality-control evidence for generated environments and validators. Section 3.3 reports expert inspection, repair statistics, and inter-annotator agreement across 197 candidate tasks. All released tasks are manually checked, and an additional audit of 150 trajectories, balanced between validator-positive and validator-negative examples, finds complete agreement with blinded human judgments. This is substantially more informative than assuming that executable validators are automatically correct.
  3. Useful task coverage and an explicit attempt at external validation. Tasks originate from a user-study pipeline, cover both Chinese and English interfaces, and include extended information-gathering workflows. Human results help establish that the benchmark is executable despite low agent success rates, and the independent real-app evaluation and correlation analysis in Appendix C provide encouraging evidence that the benchmark captures capability differences observed outside the simulator.
Critical weaknesses
  1. The contribution relative to closely related work needs clearer positioning. MobileGym [1] already provides browser-hosted mobile environments with deterministic state-based evaluation and real-device validation. InfiniteWeb [2] is particularly relevant to the construction methodology: it generates task specifications, builds environment functionality to support those tasks, and synthesizes task-specific evaluators. This overlaps with SimuWoB's joint construction of environments, task logic, and validators, although SimuWoB's evaluation tasks originate from a user study rather than LLM-generated task proposals. Neither work is discussed, and Table 1 lists SimuWoB as its only web-simulation benchmark. A direct comparison should clarify what SimuWoB's two-stage design, human quality control, task provenance, and real-app validation add to these approaches. Mobile-style interaction alone does not distinguish it from MobileGym. MemGUI-Bench [3] is also relevant to the memory-related diagnostic claims.
  2. Overall scores appear to conflate untested tasks with failures. Section 4.1 limits the two local models to 74 tasks without return-value requirements, yet their overall scores in Table 3 imply a denominator of 120 (14.86 × 74/120 ≈ 9.17 for GUI-Owl; 31.76 × 74/120 ≈ 19.58 for MAI-UI). These scores contribute to the headline average of 27.92%, mixing interface coverage with measured execution performance. Table 3 already reports a mean of 36.35% on the common 74-task subset, with a best–worst gap of 35.14 percentage points. This comparison should be primary, with full-suite coverage reported separately and category-level denominators clarified.
  3. Fidelity and diagnostic claims exceed what the current analyses establish. Table 5 reproduces the pooled Spearman correlation of 0.912, but its 15 observations share agents and task sets, and the bootstrap resampling unit is unspecified. Category-specific correlations are also high (approximately 0.872, 0.975, and 0.895), but each uses only five agents. Because the simulated and real tasks differ, this supports aggregate agreement more directly than task-level fidelity. The validation-set descriptions also conflict: 30 tasks/23 apps in Section 4.1 versus 20 tasks/17 apps in Appendix C. Similarly, category-level gaps and the Figure 6 example cannot isolate memory limitations from execution length, grounding errors, or return-value requirements. A quantitative failure audit or controlled memory intervention, informed by approaches such as MemGUI-Bench [3], would strengthen the diagnosis; otherwise, these explanations should remain hypotheses.
  4. Efficiency and synthesis-quality claims need more reproducible evidence. Backend-free deployment plausibly reduces overhead, but eight parallel workers do not quantify an advantage over emulator-based evaluation: reset latency, environment-step latency, throughput, and resource use are not measured comparatively. Reported repair effort also excludes synthesis and initial inspection costs. Appendix F provides encouraging synthesis evidence, with scores of 85.0 versus 69.2 across 20 apps. Although it holds the generation and judge models constant and describes broad evaluation criteria, it omits model identities, detailed scoring rules, generation budgets, and variability. Matched-budget comparisons and human verification would help separate the benefit of stage separation from additional refinement effort.
  5. Model coverage needs a clearer rationale and stronger mobile-use baselines. The paper does not specify an evaluation cutoff date or justify its model selection. Relevant alternatives have published mobile-use results: the UI-Venus-2 report [4] lists MobileWorld results for Seed-2.0/2.1-Pro and Kimi-K2.6/K3, and evaluates UI-Venus-2 on both AndroidWorld and MobileWorld. Qwen3.8-Max [5] and Qwen3.8-27B [6] report AndroidWorld scores of 85.3% and 81.9%, respectively. These results establish relevance to mobile GUI evaluation, although differing setups preclude direct cross-model comparisons. Evaluating a small selection under SimuWoB's unified protocol, or explaining concrete availability and integration constraints, would clarify whether the reported difficulty extends beyond the current agents.

References:

[1] Wu et al. MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research. 2026. arXiv:2605.26114v3.

[2] Zhang et al. InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training. 2026. arXiv:2601.04126v3.

[3] Liu et al. MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments. 2026. arXiv:2602.06075v3.

[4] Venus Team et al. UI-Venus-2 Technical Report. 2026. arXiv:2609.00028v1. See Table 1 and Section 4.2.1 for mobile-use results.

[5] Qwen Team. Qwen3.8-Max: A New Bar for Coding and Cowork. Official English blog. August 3, 2026. See the AndroidWorld row in the full benchmark table.

[6] Qwen Team. Qwen3.8-27B. Hugging Face model card. 2026. See the AndroidWorld row under VL Performance.

Questions for authors
  1. Contribution: How does SimuWoB differ from MobileGym [1] and, specifically, InfiniteWeb's task-specification, environment-construction, and evaluator-generation pipeline [2]? Could Table 1 or a dedicated comparison distinguish methodological advances from differences in task provenance and quality control?
  2. Score accounting: How are untested tasks handled in overall and category-level scores? Could the common 74-task comparison be made primary, with full-suite coverage reported separately?
  3. Validation and diagnosis: Could you reconcile the real-app dataset sizes and clarify the correlation analysis and bootstrap procedure? What quantitative failure analysis or controlled intervention supports the memory-based explanation of long-horizon failures?
  4. Efficiency and reproducibility: Could you quantify environment overhead against an emulator baseline and total construction cost, and specify the models, scoring rules, budgets, and variability for Appendix F, including a matched-budget comparison?
  5. Model coverage: What was the evaluation cutoff date, and could representative additional models with AndroidWorld or MobileWorld results, such as Seed-2.1-Pro, Kimi-K3, UI-Venus-2 [4], Qwen3.8-Max [5], or Qwen3.8-27B [6], be evaluated? If not, what constraints prevented their inclusion?
Reviewer_c1xhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SimuWoB, a new fully-synthetic benchmark for evaluating mobile GUI agents with high coverage and efficient evaluation. The authors collect real use cases and use them to drive the generation of 60+ synthetic mobile apps and 120 tasks. The use of coding agents to generate SimuWoB is clever and careful, which is followed by a manual inspection to ensure task and environment quality. Experiments with several closed and open UI agents show that this benchmark is substantially more challenging than others like AndoidWorld, while human score remains high at ~90%. Among other insights, the authors identify "find-grained visual control" as a new challenge for VLLM agents in such scenarios.

Critical strengths

The proposed purely synthetic benchmark generation methodology, starting from real tasks of human interest, appears to work quite well. Creating similar benchmarks in the past, prior to the era of strong coding agents, has often taken many months of human effort (e.g., AppWorld, Trivedi et al., 2024). SimuWoB shows that today's coding agents can be used to streamline this task which preserving difficulty, richness, and coverage.

The evaluated agents score rather poorly on these tasks, even though the human baseline is as high as ~90%. This will make SimuWoB a useful environment and benchmark to develop mobile agents on.

Critical weaknesses

All tasks in SimuWoB are single-app tasks. This limits the scope of agent evaluation to a relatively narrow slice of real use tasks people like to perform, e.g., using their messages for communication of input / output to other apps. That said, challenging single-app tasks are also valuable, and the authors' evaluation shows they are indeed not easy even for frontier models.

Some part of the paper are unclear, making it difficult to judge them. I will state two examples here. (1) It's unclear what "backend-free" means. This is a term used often in the paper, but the tasks and apps clearly require an initial state, state updates (e.g., placing an order, sending a message, adding a song, etc.). Isn't some backend database needed to store and update such state, and then to evaluate / validate it once the agent finishes? (2) It's stated that a validation GUI agent interactively attempts every candidate tasks. But if the tasks are challenging for frontier models, wouldn't the validation agent need some privileged / hidden information to actually succeed at all 120 tasks? This is confusing and leaves unclear how the generated apps and tasks are checked for correctness and consistency.

Questions for authors
  • See above: does the validation GUI agent have access to any privileged / hidden information that makes it possible for it to validate all 120 tasks?

  • Please clarify what "backend free" means, taking into account that some start and end states must be maintained, and updated during agent execution, for all this to work.

  • Which agent harness(es) were used for each of the evaluated agents? This is important to know, as an unoptimized or careless harness can make agent performance artificially worse, sometimes by up to 20+ percentage points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.