- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents DreamUI, a framework that uses a generative UI world model as a reinforcement learning environment for training mobile agents. The main motivation is clear and practical: training directly on real apps is often cumbersome due to reset difficulties, login requirements, and the cost of building and maintaining large-scale mock application environments. DreamUI addresses this by generating the next interface state from the task instruction, the current screenshot, and the agent’s action. Specifically, the model produces HTML, which is then rendered into the next UI, enabling an interactive simulated environment for agent training.
To improve training stability, the paper proposes two additional designs. Environment-Aware Trajectory Masking aims to filter out trajectories affected by errors in the world model, reducing the impact of low-quality simulated interactions. Rubrics-Based Reward System provides more fine-grained supervision by decomposing tasks into intermediate criteria, so that agents can receive partial rewards even when they do not fully complete the task. Empirically, DreamUI-World-8B achieves 82.9% instruction accuracy and 75.6% visual similarity on MWMBench, suggesting that the generated environments are reasonably faithful. When used for training, DreamUI-4B improves downstream performance from 69.8% to 73.3% on AndroidWorld and from 29.0% to 36.2% on MobileWorld.
Critical strengths
-
The paper is well written, with a clear motivation and a strong focus on an important problem. It does a good job of identifying the practical challenges of scaling training in real mobile environments, as well as the limitations of existing mock-app setups, which often depend on manually curated databases and hard-coded state transitions.
-
The proposed approach is quite novel. It introduces a UI World Model, built around generated HTML, to serve as a closed-loop reinforcement learning environment. This makes it possible to dynamically generate interface states and task progressions without relying on interaction with real physical devices.
-
The paper also directly tackles the reliability issues that come with simulated environments. Trajectory Masking is designed to reduce the negative impact of incorrect environment feedback during policy training, while the Rubrics-Based Reward mechanism helps address the sparse-reward problem in long-horizon tasks by providing more fine-grained supervision.
Critical weaknesses
-
The paper motivates generative UI world models mainly through their advantages over manually built mock apps, but this central comparison is not evaluated systematically. In particular, the paper does not provide matched comparisons on app coverage, realism, long-horizon reliability, and downstream RL performance. Although the introduction states that mock-app training saturates at around 3,000 tasks while DreamUI scales to 5,000 tasks, the corresponding quantitative results are not reported in the main experiments or appendix.
-
It remains unclear whether DreamUI simulates an underlying application state machine or primarily generates locally plausible action-conditioned screens. Since the model treats the current screenshot as the complete state, it may suffer from state aliasing: visually identical screens with different latent application states cannot be distinguished. Moreover, the framework does not appear to maintain an explicit navigation stack or persistent application state. Consequently, when the agent performs a Back action, there is no guarantee that the model restores the actual previously visited state, including its content, scroll position, and task-relevant modifications, rather than generating another plausible-looking screen. The paper does not introduce explicit constraints or evaluations for state reversibility, back-stack fidelity, or cycle consistency. Therefore, the current visual-fidelity metrics are insufficient to establish that the generated transitions form a closed and logically consistent application state machine.
Questions for authors
-
Could the authors provide a controlled comparison between DreamUI and mock-app environments under matched task counts and training budgets, including app coverage, environment realism, long-horizon reliability, and downstream RL performance?
-
Does DreamUI maintain any latent application state or navigation stack beyond the current screenshot? For example, after , how does the model ensure that it restores the original state A, including its content, scroll position, and previous modifications?
-
(suggestions for future work) As a broader question, concurrent work such as REA (https://github.com/morluto/rea) has substantially improved the realism and scalability of automatically generated mock apps. Under this development, what unique advantages does a generative UI world model retain over executable mock apps, particularly considering their differences in diversity, transition correctness, controllability, latency, and verifiability? Do the authors view these two paradigms as competing approaches, or could they be combined into a hybrid environment?
4 comments
Response to Reviewer Hi5W (2/2)
[Weakness 2 / Question 2] State Restoration Capability of DreamUI-World
We thank the reviewer for raising this interesting question. We considered incorporating historical observations to support accurate state restoration, but ultimately chose not to adopt history conditioning as the default setting for agent RL, based on its computational cost and the observed downstream results. We describe the evaluation behind this choice below.
To examine state restoration, we provide previously visited observations as additional context to DreamUI-World without updating its parameters. Appendix J (Figures 22–24) illustrates three backtracking trajectories. With historical context, the model closely reconstructs earlier interfaces, preserving key visible content, layout, and item ordering in these examples. These results show that DreamUI-World can use historical observations to support state restoration in a training-free manner.
For agent RL, the essential requirement is to preserve task-relevant facts and valid action-dependent transitions. Exact reproduction of every incidental UI detail is not required. Meanwhile, retaining extensive observation history increases context length and can make rollout generation substantially slower. We therefore examine whether the additional context improves downstream policy learning by comparing agent RL in DreamUI environments with and without history conditioning.
| Context Setting | AndroidWorld | MobileWorld (GUI-Only) | Time / Traj. (s) |
|---|---|---|---|
| w/o history | 73.3 | 36.2 | 358 |
| w history | 73.7 | 34.2 | 484 |
History conditioning increases the time per trajectory by 35.2%, without a clear overall improvement in downstream performance in this comparison. We additionally find that backtracking actions account for only approximately 3% of all actions in our RL rollouts. Together, these observations motivate our default RL setting without historical observation context, balancing rollout efficiency and downstream performance in the evaluated setting. History conditioning remains an option when restoring previously visited interfaces is a priority.
We appreciate the reviewer’s emphasis on state restoration and consistency across interactions. While this work focuses on using world models as effective training environments for agent RL, we recognize the importance of efficient history representations for accurate state restoration. Developing memory mechanisms that preserve task-relevant state and support fast, reliable backtracking remains a broader research challenge in world modeling, and we view this as an important direction for future work.
[Question 3] Discussion for REA
We thank the reviewer for highlighting REA. Advances in automatic app reconstruction reduce the effort required to build executable environments. We view executable mock apps and generative UI world models as complementary approaches: mock apps provide a stable application framework with explicit state and executable operations, while world models introduce task-conditioned GUI dynamics and contextual variation.
We see an agentic workflow as a promising way to combine them. Given application context describing supported functionality and workflows, an environment-building agent could establish the application scaffold and initial configuration. During interaction, an orchestration agent could invoke the UI world model as a state-conditioned interface-generation tool, generating the next UI from the current application state, task instruction, and GUI action. The executable component would maintain application state across interactions.
Task-relevant content could be instantiated on demand and retained in this application state, avoiding the need to manually populate or modify a predefined database when expanding task coverage. This would allow interfaces and contextual details to vary within a consistent application framework, combining the state management of executable environments with the flexible GUI generation of world models.
My questions have been clarified, and I have no further questions. Since I had already given a score leaning toward acceptance, I will keep my rating unchanged.
We sincerely thank the reviewer for the time and effort devoted to reviewing our work. We are glad to hear that our responses have clarified your questions. Your valuable feedback has helped us improve the quality and clarity of our manuscript.
Response to Reviewer Hi5W (1/2)
Dear Reviewer Hi5W,
Thank you for the thoughtful comments. It is encouraging that you found our paper well written, with a clear motivation that targets the limitations of both real mobile environments and existing mock apps. We are grateful that you considered our HTML-based UI world model a novel closed-loop RL environment, and that you highlighted how trajectory masking and the rubrics-based reward mitigate unreliable feedback and sparse rewards in long-horizon tasks. We respond to each of your comments below.
We thank the reviewer for suggesting this comparison. We conduct an additional experiment using 3,000 training tasks in each environment, with application types matched as closely as possible across the two environments. Both task sets cover five functional categories: System and Utilities, Productivity and Lifestyle, Communication and Social, Media and Creation, and Browsing and Navigation. The mock-app environment covers 34 apps, while DreamUI-World covers 31 apps. During RL, we set an identical maximum number of interaction turns for both environments and train each policy until performance saturates.
Success Rate (%)
As shown in the table, with matched task counts and training budgets, DreamUI achieves success rates of 72.4% on AndroidWorld and 33.3% on MobileWorld, compared with 71.1% and 29.9% for the mock-app environment, respectively. These results provide additional evidence that DreamUI is an effective environment for downstream agent RL. We further discuss environment realism and long-horizon reliability below.
Environment realism. The mock-app baseline reproduces application interfaces and behaviors through implemented logic and configured data. DreamUI instead learns action-conditioned UI transitions from real mobile interaction trajectories and generates interfaces conditioned on the task, current observation, and action. It is designed to preserve task-essential facts while introducing plausible variations in unspecified content and appearance, as illustrated in Figure 1. The two approaches thus offer complementary strengths: mock apps provide explicitly specified interfaces and behaviors, while DreamUI supports contextual variation and instruction-driven construction of task-relevant screens.
Long-horizon reliability. Mock apps maintain application state through explicit program logic, making transitions within their implemented scope inspectable and controllable. In DreamUI, the multi-step trajectories in Appendix E illustrate how generated interfaces preserve task-relevant information and reflect successive actions as tasks progress. To support reliable policy training, environment-aware trajectory masking additionally examines complete rollouts for action–outcome mismatches, missing task-relevant entities, inconsistent transitions, and persistent rendering failures, excluding flagged trajectories from policy optimization. This mechanism allows DreamUI to use generated multi-step interactions for agent RL while screening out detected environment errors.