acceptodds
Withdrawn from ICLR 2027

THE GUI IS NOT THE STATE: DIAGNOSING STATE ALIASING IN GUI WORLD MODELS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

GUI World Models (GUI-WMs) are increasingly used to predict future states for agent planning and simulation, yet most existing formulations condition only on the current GUI observation and action. We identify state aliasing, where the visible interface omits transition-relevant environment state, so identical observable conditions can correspond to different valid futures. To diagnose this failure mode, we introduce StateAliasBench, a diagnostic benchmark that explicitly isolates such ambiguities via strict pairing. We further propose lightweight predictivestate recovery that infers structured state from history and augments otherwise frozen GUI-WMs through a deterministic state interface. Family-specific specialists provide state recovery across heterogeneous state types, and multi-teacher distillation consolidates them into a single unified estimator. Experiments show that existing GUI-WMs exhibit systematic failures under observation-only conditioning, while predictive-state augmentation substantially restores state-sensitive prediction across evaluated WMs, preserves generative fidelity, and improves downstream performance of GUI agents on AndroidWorld. These results suggest that reliable GUI world modeling should account not only for what is visible, but also for the hidden transition state that determines what happens next.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI World Models, State Aliasing, Predictive State Recovery, GUI Agents, Predictive State Representatio

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_D3AdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies state aliasing in GUI world models. It introduces StateAliasBench, with 800 strict pairs across four state families, and learns a history-based structured-state estimator using specialist teachers and multi-teacher distillation. A deterministic, model-specific interface converts recovered state into constraints for frozen world models. Evaluation covers two world models, state-recovery robustness, image fidelity, and AndroidWorld.

Critical strengths
  1. The strict-pair diagnostic makes an important input-sufficiency problem concrete and auditable.

  2. The experimental evidence and documentation are substantial. The 39-page appendix provides construction rules, training details, information-access audits, interface templates, and downstream protocols, improving the completeness and quality of the presentation.

Critical weaknesses
  1. The interpretation and downstream scope of the gains need qualification. AndroidWorld gains are encouraging but modest. Improvements of 1.72 and 2.58 percentage points correspond to two and three additional successes out of 116 tasks; the paired Qwen analysis reports three rescues and no regressions. These results do not yet establish broad downstream reliability.

  2. The submission provides neither a code attachment nor an accessible repository. No downloadable benchmark/checkpoint link was found, despite references to released records and data. This limits reproduction of the benchmark, parser, and model-specific compilation.

  3. StateAliasBench and the collected histories are author-constructed assets whose public availability could not be verified. They are not established private industrial datasets, and no public/private industrial evaluation is reported.

  4. The JSON example at lines 2144-2146 extends beyond the PDF margin, preventing reconstruction of the exact request from the printed example.

Questions for authors

See Critical Weaknesses.

Reviewer_QxbXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies state aliasing in GUI world models, where identical current observations and actions can lead to different outcomes because relevant environment state is hidden. It introduces StateAliasBench, comprising 800 verified pairs across four state families and 25 causal templates. The paper also formalizes predictive state and proposes a history-based state estimator, with family-specific specialists distilled into a unified model. A deterministic interface translates recovered state into conditioning for frozen GUI world models. Experiments demonstrate substantial improvements on the diagnostic benchmark and modest improvements for two world-model-assisted agents on AndroidWorld.

Critical strengths
  1. The paper identifies a limitation of the observation–action pairs commonly used in GUI agents. Its benchmark confirms identical serialized observation-only inputs within each of 800 pairs, while independently validating that the paired cases require different outcomes. These findings provide convincing empirical evidence that the current GUI observation can omit information needed to predict the next observation.
  2. The paper provides useful diagnostic decomposition. Observation-only, raw-history, oracle-state, specialist-state, and distilled-state conditions help distinguish information availability, state recovery, and downstream realization.
  3. While observation-only conditioning achieves 0% PairCorrect across all four state families, the proposed predictive-state augmentation pipeline reaches 79.38% macro PairCorrect on Code2World-8B and 80.63% on gWorld-8B. These results demonstrate that recovering historical state and incorporating it through the proposed interface effectively distinguishes futures that cannot be resolved from the current observation alone.
Critical weaknesses
  1. The main gains depend heavily on a handcrafted consequence-and-rendering interface. Table 17 shows that, with specialist predictions fixed, Temporal Deadline PairCorrect increases from 0% under canonical state conditioning to 84.0% and 88.5% under the full interface. The largest improvement occurs in the final step, which changes several interface components simultaneously. The authors acknowledge this limitation, but the results still leave unclear how much the proposed mechanism improves state-conditioned prediction versus supplying the semantic answer for the model to render. This distinction materially affects the interpretation of “restoring” world-model prediction.
  2. The comparison against alternative history representations remains incomplete. Raw History supplies the complete interaction history, but directly presenting history to a frozen world model is a limited control for a separately trained state estimator with explicit schema supervision. Existing work such as GUI-Rise [1] demonstrates the relevance of compact history summarization, although it targets agent decisions rather than world-model prediction. An adapted summarization or prompted state-extraction baseline would help establish whether the proposed structured supervision provides benefits beyond history compression and interface engineering.
  3. The downstream evidence is too limited to establish a reliable practical improvement. In Table 4, adding state conditioning improves AndroidWorld success from 58.62% to 60.34% for GPT-5.4 and from 44.83% to 47.41% for Qwen3-VL-8B. Across 116 tasks, these correspond to only two and three additional successes. No repeated-run uncertainty is reported. The custom Relative Gain values of 66.4% and 74.8% normalize these improvements by the small benefit of adding the world model, making them appear substantially larger than the absolute gains.

References [1] GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation. NeurIPS, 2025.

Questions for authors
  1. Can you add a compact-history or prompted state-extraction baseline and evaluate it through the same downstream interface? Reporting both state-recovery accuracy and final PairCorrect would clarify whether the advantage comes from the learned state representation, distillation, or output conditioning.
  2. Can you provide paired success counts for both AndroidWorld agents and quantify uncertainty in the absolute improvements? Repeated evaluation on a manageable subset would strengthen the practical claim. Please also report how often each state family is applicable and contributes to changed decisions.
Reviewer_XYsVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies state aliasing in GUI world models: identical current observations and actions can produce different outcomes because relevant information is hidden in the interaction history. It introduces StateAliasBench, comprising 800 strict pairs across four state families and 25 templates. A learned estimator recovers structured state from history, and a deterministic interface converts that state into conditioning for two frozen GUI world models. The resulting pipeline substantially improves state-sensitive prediction on the constructed benchmark. The authors also evaluate state recovery under distribution shifts and report modest downstream improvements on AndroidWorld.

Critical strengths
  1. The benchmark construction is careful. The authors verify that paired branches have identical complete model-visible requests, obtain labels from independent environment evidence, and validate semantic outcomes rather than relying on screenshot differences alone. The replay and train–evaluation separation audits make this a credible diagnostic of missing state information.

  2. The paper evaluates state recovery separately from final prediction. This distinction helps identify where errors arise. The comparison against joint supervised training also provides evidence for the distillation component: final world-model macro PairCorrect improves from 76.75% to 79.38% under the same conditioning interface, although the improvement is not uniform across families.

Critical weaknesses
  1. The AndroidWorld improvements correspond to two additional solved tasks for GPT-5.4 and three for Qwen3-VL-8B, out of 116 tasks. No repeated-run uncertainty is reported. The three Qwen rescues all concern file operations or receipts. Moreover, the cross-application tests for Ordered Collection and Temporal Deadline retain the same state-producing applications and change the final observation context. These results support some robustness within the specified schemas, but provide limited evidence that the approach extends beyond the hand-written transition rules.

  2. The deterministic interface does more than serialize recovered current state: it computes the action’s semantic consequence and specifies how that consequence should appear in the generated GUI. Table 17 makes this dependency clear. With the same recovered states, canonical conditioning achieves only 1.0% and 12.5% PairCorrect for Persistent Artifact, and 0% for Temporal Deadline. The full interface raises these to 78.5%–88.5%. The largest improvement comes from a step that jointly changes several aspects of the model-facing request. This supports the effectiveness of the complete engineered pipeline, but leaves unclear what predictive capability the world model contributes beyond rendering an already resolved outcome. That distinction matters to the paper’s contribution.

Questions for authors
  1. Given that the full interface already determines the scored semantic outcome, how does the learned world model compare with a simpler renderer receiving the same resolved specification?

  2. Are the AndroidWorld gains reproducible across repeated task instantiations? How do they compare with giving the recovered state directly to the agent or action selector?

  3. What evidence supports transfer to new state-producing applications or transition mechanisms, particularly for Ordered Collection and Temporal Deadline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.