- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents SimuWoB, a synthetic benchmark for mobile GUI agents. The authors build a two‑stage synthesis pipeline to create 63 simulated mobile apps with 120 tasks, plus state‑based automatic validators. All environments run as simple web URLs for reproducible, parallel testing. They evaluate five modern GUI agents, reporting 27.92% overall success and just 17.82% for long‑horizon tasks. A small comparison against real‑world mobile examples is used to argue reasonable transferability, alongside diagnostic analysis pointing out agent capability gaps.
Critical strengths
- Practical benchmark design. Synthetic, backend‑free web environments with built‑in reward validators remove much of the overhead of testing on real mobile devices or APKs, which is a real pain point for GUI‑agent researchers.
- Clear empirical observation: existing SOTA agents perform poorly on long‑horizon mobile workflows. The diagnostic breakdown gives useful directions for follow‑up work.
Critical weaknesses
- Weak generalization evidence. Only limited comparison against real‑world mobile cases is provided. It remains unclear whether SimuWoB’s synthetic UI logic and task difficulty properly mirror real‑world mobile apps. Large distribution mismatch would mean benchmark results do not reliably predict real‑phone performance.
- Limited transparency for benchmark construction. Little information is given on how the 63 apps and 120 tasks were sampled. Without knowing sampling rules, it is hard to verify task diversity and rule out hidden biases in synthetic task generation.
Questions for authors
- Can you describe your app and task sampling strategy? How do you guarantee synthetic tasks reflect realistic user intents and common mobile UI patterns?
- Your real‑world transfer evaluation is small. Could you provide quantitative measurements of the distribution gap between SimuWoB and real‑world mobile tasks?
- What failure modes exist for your state‑based validator? Can an agent pass state checks without actually completing the intended user goal?