acceptodds
Under review as a conference paper at ICLR 2027

AutoGUIWorld: Image Generators as Visual World Models for GUI Agent

Abstract

Scaling diverse training data for graphical user interface (GUI) agents is constrained by the environments available for interaction. Environment-based data engines typically construct platform- or task-specific interfaces and collect trajectories through execution, making broader coverage a recurring investment in development time and engineering effort. Yet screenshot-based agents learn from visible action consequences, motivating interaction fidelity as a data-generation target: preserving action-consistent visual feedback and task-state evolution without reconstructing the underlying software. The discrete, structured nature of GUI transitions makes image generators a natural candidate for implicit visual world models. We introduce AutoGUIWorld, an environment-free data engine that turns this premise into screenshot–action–screenshot trajectories. Structured world sampling combines platform, appearance, and interface-state constraints with Image2’s generative stochasticity to produce diverse visual seeds, from which tasks are generated. A meta planner then specifies atomic actions and their intended consequences; Image2 chain-edits the current screenshot into the next observation, while a grounding module annotates action targets. This factorization separates semantic planning from visual transition synthesis and expands data coverage without constructing or running the target applications.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Use Agent, World Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JEv7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AutoGUIWorld, a method for synthesizing GUI interaction trajectories using image generation models, without deploying or running the corresponding software environments. It combines structured GUI sampling, task generation, high-level action planning, step-by-step screenshot generation, and state-transition quality checks to construct a dataset for training GUI agents.

Critical strengths

The method combines image generation models with task planning to generate GUI interaction trajectories without actually running the target software, offering a promising approach to expanding the application coverage of training data.

Critical weaknesses
  1. Step-by-step generation and per-step consistency checks do not guarantee consistency across an entire trajectory over a long horizon. The paper does not sufficiently explain how cumulative errors are detected and controlled, nor does it provide a systematic evaluation of trajectory length and long-horizon consistency.
  2. Model generation or model-based judgments are involved in initial GUI generation, action planning, and state-transition quality checks. Although the authors introduce structured constraints, staged checks, repair, and filtering, these mechanisms do not sufficiently establish that the generated results conform to the behavioral patterns of real-world applications. For example, the paper does not sufficiently verify whether the initial screenshots satisfy the specified GUI constraints, nor does it present an independent correctness-verification mechanism for the action sequences generated by the Meta Planner. For state-transition quality checks, even if the program strictly enforces label-processing and filtering rules, the semantic judgments still rely on the VLM itself. Therefore, the possibility that the checker incorrectly labels an erroneous trajectory as valid cannot be ruled out.
Questions for authors
  1. Beyond step-by-step generation and per-step quality checks, is there a consistency-verification mechanism that operates at the trajectory level?
  2. For initial screenshot generation, Meta Planner planning, and state-transition quality checks, do the authors employ any sufficiently rigorous verification mechanisms that are independent of generative models?
1 comment

We thank the reviewer for the careful review and for recognizing the promise of expanding application coverage without running the target software. We agree that model-based generation calls for verification that does not itself depend on generative models. Below we show that such verification exists at the trajectory level and for each component the reviewer listed. Several of these results were omitted from the submission because of space constraints; we will add all of them to the revision.

What is verifiedVerification independent of the generatorResult
Trajectory-level task logic, incl. Meta PlannerReal execution of the same tasks in VMs, compared with generated trajectories2.3–4.3% inconsistent; no increase with length
Transition checkerHuman labels on 2,354 transitionsRecall 100%, precision 79.0%
Final training dataHuman audit of 1,000 transitions0.4%, all minor
Trained policyExecution-based benchmark evaluatorsGains on all four benchmarks

Q1: Is there trajectory-level verification? Yes. To measure how reliably generated trajectories follow real application logic, we ran an execution-based verification study on generated trajectories spanning all length ranges. For each trajectory, a Codex agent with GPT-5.6 Sol executed the same task in a virtual machine running the real application until it obtained a successful execution. We then compared the generated trajectory, i.e., its full action sequence and step descriptions, with the real execution and checked whether the two follow the same task logic. This detects errors that step-level checks cannot see, such as a trajectory that relies on a feature the real software does not provide. Trajectories that failed were excluded from training. The study quantifies reliability; the generation pipeline itself does not require virtual machines.

W1: Cumulative errors and trajectory length. The inconsistency rate does not increase with length: 3.4% (1–5 steps), 3.1% (6–20), 4.3% (21–40), and 2.3% (41+). The step-level flag rate follows the same pattern by step position: 3.4%, 2.7%, 3.0%, and 2.8% at steps 1–5, 6–10, 11–20, and 21+. Errors therefore do not accumulate with length, at either the task level or the step level, including in trajectories longer than 40 steps. For reference, generated trajectories average 11.9/11.4/7.1 steps on Ubuntu/Windows/macOS, with 90th percentiles of 24/25/14 (Section 4.4).

Three design choices explain this stability. First, each frame is edited from the actual previous frame, so unrelated content is carried forward by the image itself, and Voyager conditions on the generated screenshot rather than the idealized plan. Second, the action sequence is fixed by the plan, so drift from the plan surfaces as missing targets or invalid actions at later steps, which the step-level checks flag. Third, if generation fails, only the valid prefix is kept.

Q2 / W2: Verification independent of generative models.

  • Meta Planner. The execution study tests each planned sequence against a successful execution in the real application, a reference independent of any generative model. Plans that use nonexistent features or follow incorrect logic are detected and excluded; they account for 2.3–4.3% of trajectories across length bins.
  • State-transition checks. We validated the Gemini checker against human labels on 2,354 transitions. It flagged all 98 erroneous transitions (recall: 100%; 95% lower bound: 96.3%) and erred only toward rejection (precision: 79.0%). The case the reviewer raises, an erroneous transition labeled as valid, did not occur in this sample. A separate audit of 1,000 transitions from the final training set, stratified by platform and reviewed by three annotators, found only 4 minor issues (0.4%): an imperfect box or locally blurred text, with no substituted or hallucinated content.
  • Initial screenshots. Non-generative measurements (OmniParser UI detection, OCR, and distribution distances to real ScaleCUA screenshots; Sections 4.3–4.4) show that seeds are comparable to real interfaces. We will add a human check of seeds against their sampled specifications to the appendix.
  • Trained policy. The trained agent is evaluated by executing tasks in real environments with programmatic evaluators, independent of any generative model, and it improves on all four benchmarks.

We hope these results address the reviewer's concerns about verification, and we would be grateful if the reviewer would take them into account in the final assessment.

Reviewer_dkZ5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the issue of limited diversity in available trajectories within the research domain of GUI agents and proposes AutoGUIWorld, a data generation framework designed to synthesize GUI interaction trajectories without deploying actual software. The improved performance on OSWorld achieved using a fine-tuned Qwen3.5 model validates the benefits of the synthesized interaction trajectories generated by AutoGUIWorld.

Critical strengths
  • Proposes synthesizing trajectories without deploying software by combining task planning and image generation.
  • Collects the AutoGUIWorld dataset/benchmark, comprising 79,266 grounded and filtered step-level samples across different OS platforms.
  • Demonstrates experimental performance improvements from the generated interactions on the OSWorld and ScienceBoard benchmarks.
Critical weaknesses
  • Presents a potentially unfair comparison with AgentNet due to inconsistent experimental settings.
  • Relies heavily on generative models, where artifacts in text generation may negatively affect downstream performance.
Questions for authors
  • Lacks an ablation study to validate the contributions of individual components, including image generation.
2 comments

We thank the reviewer for the positive assessment and helpful comments. We address W1 and W2 below and will follow up on the ablation question in a separate comment.

W1 (comparison with AgentNet)

Each setting difference gives the AgentNet run more resources or a more favorable selection than AutoGUIWorld:

  • Data and training. AgentNet uses ~350k training examples, 4.4× more than AGW-35B (~80k), and trains longer (686 vs. 469 steps).
  • Reasoning supervision. AgentNet is trained in its native format, which includes reasoning annotations, so its model reasons before acting. AGW-35B is trained and evaluated without reasoning.
  • Checkpoint selection. We compare AGW-35B's single final checkpoint against AgentNet's best checkpoint, chosen separately for each benchmark.
  • Drag encoding. The harnesses serialize drag differently (one target coordinate vs. explicit start and end points). This changes the format, not which drags can be expressed.

The base model, benchmark task sets, evaluators, and score aggregation are identical. Despite these advantages, AGW-35B outperforms AgentNet on all four benchmarks:

BenchmarkAGW-35B (final)AgentNet (best ckpt)Gap
OSWorld40.840.1+0.7
WAA27.923.4+4.6
macOSWorld45.041.6+3.5
ScienceBoard32.216.8+15.4

The pattern of gaps is informative. On OSWorld, whose applications are common desktop software, AutoGUIWorld matches real demonstrations while using 4.4× less data. The gap is largest on ScienceBoard, whose specialized scientific software is costly to deploy and therefore hard to cover by demonstration collection. There, AgentNet improves only modestly over the base (14.0% → at most 16.8%, and 11.9% at its final checkpoint), whereas AGW-35B reaches 32.2%. This is consistent with our central claim. Because AutoGUIWorld does not need to run each application, it can generate grounded interactions for applications and interface states beyond the reach of real-demonstration collection. This broadens the policy's coverage and transfers to unseen real-environment tasks. In the revision, we will state the setting differences explicitly in the main text and present this comparison as evidence of data efficiency and coverage.

W2 (text artifacts)

We verified this concern directly:

  • Human audit. Three annotators inspected 1,000 transitions from the final training set, stratified by platform. Each item included adjacent before/after frames, the action, and the target box. Only 4 (0.4%) had any issue, all minor: a box not fully enclosing an icon, or locally blurred text. No item showed text substitution or hallucinated content.
  • Checker reliability. On 2,354 human-labeled transitions, the Gemini checker flagged all 98 erroneous ones (recall: 100%). Its errors were only conservative (precision: 79.0%), so generation artifacts are removed before training.
  • Downstream effect. Residual minor defects barely change the base policy's predicted actions. On ScreenSpot-Pro, text-target accuracy rises from 41.2% to 72.0% (+30.7 points), more than icon accuracy (+16.7). If text artifacts were propagating into the policy, text grounding would be expected to suffer first. The interactive gains are measured by execution in real environments.

Follow-up on Q (component ablations)

We ablated the key components of the pipeline:

  • Meta Planner and Voyager. Without them (Image2 prompted only with the task and "generate the GUI after the next operation"), only about 40% of generated cases pass the checker, compared with about 89% for the full pipeline.
  • Transition filtering. Adding back the ~5k filtered transitions lowers the four-benchmark mean from 36.5% to 30.8% (OSWorld 37.6, WAA 25.6, macOSWorld 36.8, ScienceBoard 23.1), removing 45% of the gain over the base model.
  • Image generation. Image generation produces all training observations; training on them raises the four-benchmark mean from 23.6% to 36.5%.

We will add these ablations to the revision.

Reviewer_d6moOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AutoGUIWorld generates multi-step GUI demonstrations without executing the target software during data collection. A planner specifies actions and their expected effects, and a pretrained image generator repeatedly edits the preceding screenshot to produce the trajectory. Action grounding and filtering yield 79,266 step-level training examples across Ubuntu, Windows, macOS, and Chrome. The authors fine-tune Qwen3.5-35B-A3B on this dataset and report improvements on four interactive benchmarks and ScreenSpot-Pro. The contributions are the generation pipeline, the resulting dataset, and evidence that these synthetic demonstrations improve downstream GUI-agent performance.

Critical strengths
  1. The generated demonstrations provide useful training supervision. Fine-tuning improves performance across several operating systems and scientific applications. OSWorld mean task score increases from 33.0% to 40.8%, and ScienceBoard task success increases from 14.0% to 32.2%. These evaluations involve application execution under controlled benchmark conditions, providing meaningful evidence that the generated experiences help an agent perform tasks. This is the strongest reason for acceptance.
  2. The work provides a practical method for constructing multi-step demonstrations without deploying each target application. The pipeline combines planning, sequential image editing, grounding, and filtering into a substantial training corpus. Producing usable action-labeled sequences from generated screens offers another route to expanding GUI training data, including for applications that are difficult to set up for collection. The complete system and its demonstrated training value constitute a useful contribution.
Critical weaknesses
  1. The experiments provide limited evidence about what the proposed world-model construction adds. The downstream gains establish that the complete dataset is useful, but there are no controlled ablations isolating coherent trajectories or transition filtering. It therefore remains unclear whether sequential simulation contributes beyond additional visual examples, grounding supervision, and planner-generated actions. This limits what the paper establishes beyond the usefulness of this particular dataset.
  2. The consequences of the simulation-to-reality gap for the trained agent remain insufficiently analyzed. Appendix F documents text substitutions and incorrect state changes, but the reported audit does not establish their frequency after final filtering or connect them to downstream behavior. Aggregate benchmark improvements can coexist with weaknesses on tasks requiring exact values and persistent state. Understanding these consequences is important for assessing where this source of training data is reliable.
Questions for authors
  1. Component contributions — clarification and suggested experiment. Are there additional results isolating the benefits of coherent trajectories and transition filtering? Otherwise, comparisons against independent screenshot–action examples and training without transition filtering would help explain which components drive the gains.
  2. Simulation errors — clarification. What inaccuracies remain after final filtering, and has analysis of downstream failures linked any to these errors? Are failures concentrated in tasks requiring exact text, numbers, or preservation of earlier state?
  3. Practical generalization and rollout length — clarification and suggested evaluation. Do the existing results reveal how performance varies with workflow length or the need to recover from unexpected states? Evaluating unfamiliar workflows and perturbed application states would strengthen the practical-usefulness claim.
  4. Longer-term effects on the agent — suggested future study. With increasing synthetic training, does the agent retain its earlier capabilities and ability to handle diverse workflows? Extending the checkpoint analysis with capability-retention and recovery tests would help identify possible over-specialization or forgetting. This is distinct from errors accumulating within a generated trajectory; the current results do not establish collapse.
2 comments

We thank the reviewer for the careful reading and constructive suggestions. Below we address W2/Q2–Q4; we will follow up on W1/Q1 in a separate comment.

W2 / Q2 (residual errors after filtering)

What 3.04% measures. A single Gemini pass serves as both audit and filter. The 3.04% is the fraction of raw desktop transitions it flagged, and all flagged transitions are removed before training. The figure describes raw Image2 outputs, not the final training data.

Human audit of the final data. We sampled 1,000 transitions from the final training set, stratified by platform share. Three annotators inspected each pair of adjacent frames together with the action and target box. They checked transition continuity, realism, completeness of the action effect, and hallucination. Only 4 transitions (0.4%; 95% CI: 0.16–1.02%) had any problem, and all were minor: a box not fully enclosing the target icon, or locally blurred text. None showed the failure types documented in the transition-fidelity appendix (content substitution, over-completion, missing effects, persistent-content mutation). These failure types characterize raw outputs, which the filter removes.

Checker reliability. On 2,354 human-labeled transitions from pipeline development, Gemini flagged all 98 erroneous ones (recall: 100%; 95% lower bound: 96.3%). Its only disagreements with the annotators were conservative: 26 of its 124 flags were correct transitions (precision: 79.0%; false-positive rate: 1.2%). This is consistent with the prompt's instruction to be "strict and literal." Over-rejection discards some valid data but does not admit hallucinated transitions.

Effect on the policy. We ran the base policy on screenshots with such minor defects and on their corrected versions. It predicts essentially the same actions on both, with only slight differences in token probabilities, because action prediction depends mainly on the target region.

Where errors concentrate. Errors concentrate by domain rather than by task requirement. They occur mainly in terminal workflows, where long commands must be rendered exactly, and in several specialized applications. Domain experts' comparison with real interfaces suggests that this relates to the generator's familiarity with the software. Because these errors are filtered, they mainly reduce the number of retained samples in these domains.

We will add these analyses to the appendix and revise the statement in Section 4.2 that checker error has not been calibrated by a human audit.

Q3 (workflow length, recovery, unfamiliar workflows)

Length. Multi-application tasks require longer workflows and carrying information across applications. They improve from 12.0% to 20.9% on OSWorld and from 13.8% to 27.6% on macOSWorld. ScienceBoard tasks, run with a budget of up to 100 model responses, improve in all five domains.

Recovery. All benchmarks are evaluated in closed loop: the agent acts in the real environment and receives the resulting screenshot. It must therefore respond to actual states, including deviations caused by its own earlier actions. The gains reflect behavior under these unplanned states, although we do not measure recovery separately.

Unfamiliar workflows. No benchmark tasks or environments are used in data generation; training tasks are generated from synthetic scenes. Benchmark workflows are therefore unseen at the task level, although many applications overlap with our training coverage. We have not evaluated perturbed application states. We agree that such an evaluation would strengthen the practical-usefulness claim, and we will add it to the limitations as future work.

Q4 (longer-term effects)

Within our training budget (three epochs, 469 steps), we see no sign of degradation. Every interactive benchmark stays above the base model at every evaluated checkpoint. The four-benchmark mean increases at every checkpoint (23.6% → 29.3% → 31.3% → 32.0% → 33.2% → 33.9% → 36.5%), and all ScreenSpot-Pro checkpoints exceed the base. These GUI-specific results come from a single run, so they do not establish that general capabilities are retained or how the model behaves under longer synthetic training. We agree that capability-retention and recovery tests over longer training are a valuable future study, and we will note this in the discussion.

Presentation. We will streamline Section 4 by moving secondary qualifications to the appendix and stating the relationship between auditing and filtering directly.

Follow-up on W1 / Q1 (component contributions)

We thank the reviewer for the suggestion. We ran two ablations.

Removing the Meta Planner and Voyager. We replaced them with a "sequence-gen" baseline in which Image2 receives only the task instruction and the instruction "generate the GUI after the next operation" at each step. Only about 40% of about 6k sequence-gen cases pass the checker, compared with about 89% for the full pipeline (Table 2). Planner-guided generation is therefore what produces action-consistent trajectories.

Training without transition filtering. We added the ~5k checker-rejected transitions (with repaired boxes) back into the training data and retrained under the same setup:

OSWorldWAAmacOSWorldScienceBoardMean
Base33.019.428.114.023.6
w/o transition filtering37.625.636.823.130.8
AGW-35B40.827.945.032.236.5

These transitions make up only about 6% of the data, yet they remove 45% of the average gain and half of the gain on ScienceBoard and macOSWorld. Transition quality is therefore critical, especially for specialized scientific software.

Together, the two ablations show that both the controlled generation design and transition filtering are essential to the downstream gains. We will add them to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.