- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces AutoGUIWorld, a method for synthesizing GUI interaction trajectories using image generation models, without deploying or running the corresponding software environments. It combines structured GUI sampling, task generation, high-level action planning, step-by-step screenshot generation, and state-transition quality checks to construct a dataset for training GUI agents.
Critical strengths
The method combines image generation models with task planning to generate GUI interaction trajectories without actually running the target software, offering a promising approach to expanding the application coverage of training data.
Critical weaknesses
- Step-by-step generation and per-step consistency checks do not guarantee consistency across an entire trajectory over a long horizon. The paper does not sufficiently explain how cumulative errors are detected and controlled, nor does it provide a systematic evaluation of trajectory length and long-horizon consistency.
- Model generation or model-based judgments are involved in initial GUI generation, action planning, and state-transition quality checks. Although the authors introduce structured constraints, staged checks, repair, and filtering, these mechanisms do not sufficiently establish that the generated results conform to the behavioral patterns of real-world applications. For example, the paper does not sufficiently verify whether the initial screenshots satisfy the specified GUI constraints, nor does it present an independent correctness-verification mechanism for the action sequences generated by the Meta Planner. For state-transition quality checks, even if the program strictly enforces label-processing and filtering rules, the semantic judgments still rely on the VLM itself. Therefore, the possibility that the checker incorrectly labels an erroneous trajectory as valid cannot be ruled out.
Questions for authors
- Beyond step-by-step generation and per-step quality checks, is there a consistency-verification mechanism that operates at the trajectory level?
- For initial screenshot generation, Meta Planner planning, and state-transition quality checks, do the authors employ any sufficiently rigorous verification mechanisms that are independent of generative models?
We thank the reviewer for the careful review and for recognizing the promise of expanding application coverage without running the target software. We agree that model-based generation calls for verification that does not itself depend on generative models. Below we show that such verification exists at the trajectory level and for each component the reviewer listed. Several of these results were omitted from the submission because of space constraints; we will add all of them to the revision.
Q1: Is there trajectory-level verification? Yes. To measure how reliably generated trajectories follow real application logic, we ran an execution-based verification study on generated trajectories spanning all length ranges. For each trajectory, a Codex agent with GPT-5.6 Sol executed the same task in a virtual machine running the real application until it obtained a successful execution. We then compared the generated trajectory, i.e., its full action sequence and step descriptions, with the real execution and checked whether the two follow the same task logic. This detects errors that step-level checks cannot see, such as a trajectory that relies on a feature the real software does not provide. Trajectories that failed were excluded from training. The study quantifies reliability; the generation pipeline itself does not require virtual machines.
W1: Cumulative errors and trajectory length. The inconsistency rate does not increase with length: 3.4% (1–5 steps), 3.1% (6–20), 4.3% (21–40), and 2.3% (41+). The step-level flag rate follows the same pattern by step position: 3.4%, 2.7%, 3.0%, and 2.8% at steps 1–5, 6–10, 11–20, and 21+. Errors therefore do not accumulate with length, at either the task level or the step level, including in trajectories longer than 40 steps. For reference, generated trajectories average 11.9/11.4/7.1 steps on Ubuntu/Windows/macOS, with 90th percentiles of 24/25/14 (Section 4.4).
Three design choices explain this stability. First, each frame is edited from the actual previous frame, so unrelated content is carried forward by the image itself, and Voyager conditions on the generated screenshot rather than the idealized plan. Second, the action sequence is fixed by the plan, so drift from the plan surfaces as missing targets or invalid actions at later steps, which the step-level checks flag. Third, if generation fails, only the valid prefix is kept.
Q2 / W2: Verification independent of generative models.
We hope these results address the reviewer's concerns about verification, and we would be grateful if the reviewer would take them into account in the final assessment.