acceptodds
Under review as a conference paper at ICLR 2027

WM-R1: Training GUI Agents to Reason and Leverage World Models with Reinforcement Learning

Abstract

GUI agents trained with reinforcement learning (RL) have showcased strong environment learning capabilities on mobile platforms. However, RL typically demands extensive real-environment interactions, leading to high resource costs and instability, especially in GUI scenarios. To address these, we propose WM-R1, the first reinforcement learning framework that trains mobile GUI agents with world models instead of real environments. Specifically, world models serve as the source of state transitions during all rollouts, replacing the real Android environment within the training loop. WM-R1 also embeds world models directly into the thinking process, enabling agents to reason about the consequences of candidate actions before committing to the final action. Crucially, WM-R1 eliminates the need for real-environment interaction, supports massively parallelized and step-level granularized trajectory generation grounded in world models, and introduces a multi-dimensional rule-based reward that jointly optimizes task success, trajectory efficiency, and world model utilization. For efficient training, we curate a high-quality dataset of 2000 challenging tasks. Experiments on Android mobile benchmarks demonstrate that WM-R1-trained agents significantly outperform GRPO-only baselines and inference-time simulation methods.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, GUI Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EiDzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WM-R1, a new approach to training GUI agents with reinforcement learning, where a pretrained world model replaces real environment interactions during training. Instead of relying on real Android devices to collect trajectories, the method uses Code2World to simulate the next GUI state after each action, allowing the agent to perform multi-step rollouts and learn through GRPO. The agent can also use the world model during reasoning to preview candidate actions and adjust its decisions based on the simulated outcomes. The authors further introduce a reward function that considers task success, trajectory length, and world-model usage. Experiments on several GUI benchmarks show promising improvements over the selected baselines at different model scales. The paper also includes multiple ablation studies to examine the contributions of individual components, along with training cost comparisons and additional experiments on Qwen3-VL models to evaluate the effectiveness and generality of the approach.

Critical strengths

One strength of this paper is that it addresses a practical challenge in GUI agent RL: collecting multi-step trajectories through real environment interactions is costly and difficult to scale. WM-R1 uses a world model to simulate these interactions during training and allows the agent to preview possible actions during inference. While using world models for agent training and reasoning is not entirely new, I think applying this idea to multi-step GUI RL is a useful contribution. It provides a promising alternative to training agents through real environment interactions. Another strength is the relatively thorough experimental evaluation. The paper reports consistent improvements across several GUI benchmarks and model sizes. The authors also examine the effects of CoT, trajectory-length rewards, and world-model usage rewards through ablation studies, along with additional hyperparameter and cross-model experiments. I particularly appreciate the training time comparison, as it helps demonstrate the practical benefit of the approach beyond improvements in task success rates. Taken together, these results give me reasonable confidence in the value of the proposed method.

Critical weaknesses

The main contribution of this paper is to use the World Model to replace the real environment for multi-step reinforcement learning training, and to utilize the World Model to assist decision-making during the reasoning stage. However, the existing ablation experiments mainly focus on components such as CoT, trajectory length rewards, and World Model invocation rewards, lacking experiments that can directly verify the independent contributions of these two core designs. Although this paper achieved good results on multiple Benchmarks, some GUI Agent methods that have already demonstrated higher performance were absent from the main experiments, such as MobileRL, GUI-Owl, and MAI-UI. The models of these works have similar scale to some of the settings in this paper, especially showing strong performance on AndroidWorld. Although the training data, training costs, and evaluation settings of different methods may vary, and a completely fair comparison cannot be made directly, the authors should at least discuss these methods and explain the advantages and limitations of WM-R1 compared to them. The current experiments can support the conclusion that WM-R1 outperforms the Baseline selected in the paper, but they are not sufficient to support a more general SOTA claim. This makes the performance positioning of this paper in the existing GUI Agent research less clear.

Questions for authors

The core idea of this paper is to use the World Model to replace the real environment for multi-step reinforcement learning training. However, both the training stage and the inference stage employ the World Model. Therefore, it is currently difficult to determine how much benefit the simulation training has brought. Can the author directly deploy the Agent trained by the World Model to the real GUI environment, without invoking the World Model to preview the actions, to evaluate its task success rate and compare it with the corresponding Baseline? I believe this experiment can more directly verify whether the strategies learned in the simulated environment can be transferred to the real environment, and also help to distinguish the benefits of the training stage from the improvement brought by the additional use of the World Model in the inference stage.

I have some questions regarding the selection of the Baseline in the paper and the evaluation settings of some Benchmarks. Firstly, existing methods such as MobileRL, GUI-Owl, and MAI-UI reported significantly higher success rates on AndroidWorld, with some models exceeding 70%, but these methods were not included in the main comparative experiments. The authors could explain the reason and further clarify the applicable scope of the SOTA statement in the paper? Secondly, the paper describes both ScreenSpot-Pro and ScreenSpot-V2 as mobile Grounding evaluations, but ScreenSpot-Pro itself is mainly targeted at desktop professional software and does not have a standard Mobile subset. Could the authors clarify the specific data division used here? Additionally, I also hope to further confirm whether the final results of AndroidWorld come from official task checks in the real Android environment and whether all methods have adopted the same evaluation protocol. These issues will directly affect my judgment on the comparability of the experimental results and the performance conclusions of the paper.

1 comment

Dear Reviewer, thank you again for your thoughtful comments and suggestions. In our response and revised manuscript, we have added experiments to distinguish the contributions of world-model training and inference, expanded comparisons with stronger baselines, and clarified the evaluation protocols. We would greatly appreciate any further thoughts you may have when you have a chance to review these updates.

Reviewer_1PysOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WM-R1, a reinforcement learning framework for mobile GUI agents. During training, it uses predictions from a frozen Code2World model in place of real-environment transitions. The policy also learns to call the world model during reasoning to check what might happen after an action. Training uses GRPO with rewards for task success, trajectory length, and world-model use. The paper reports gains on navigation and grounding benchmarks. It also includes experiments with different model sizes, component ablations, and an analysis of computational cost.

Critical strengths

A useful problem and a clear overall approach. Reducing the need for live Android interactions is a useful direction for GUI-agent training. The paper combines training on simulated trajectories with learning to preview actions (§4). These two parts fit together well.

A broad set of experiments with promising results. The evaluation covers navigation and grounding tasks, several model sizes, and component ablations (Tables 1–2). For example, Table 1 reports a 39.8% success rate on AndroidWorld for WM-R1-7B, compared with 30.8% for UI-R1-7B. These results suggest that the approach could be useful in practice. Still, I would like to see clearer evidence of which parts of the method lead to these gains.

Critical weaknesses

The simulator's reliability during training is unclear. The full trajectory comes from the world model (§3), but the paper does not measure how closely these trajectories match real execution over multiple steps. The example in §5.5 does not show how often simulation errors occur or how they affect learning.

The contribution of simulated training is unclear. Tables 1–2 do not isolate simulated training from changes in reasoning, rewards, and test-time world-model use. The results support the full system, but do not show how much of the gain comes from simulated training alone.

Questions for authors

Q1. How reliable are simulated trajectories over multiple steps? Could the authors compare simulation with real Android execution using the same initial states and actions? How does agreement change over multiple steps, and how often does simulated success turn into failure in real execution?

Q2. How much of the improvement comes from simulated training? Could the authors compare simulated and real-environment training while keeping the tasks, policy backbone, rewards, reasoning setup, and test-time world-model access and budget the same?

Q3. How much does the method depend on world-model quality? Both training and planning use the same frozen Code2World model (§5.1). Could the authors show how real-environment success varies with simulator quality across apps or task types? This would help clarify when simulation helps and when its errors hurt performance.

The paper addresses a useful problem and reports promising results. My main concerns are the reliability of simulated trajectories and the contribution of simulated training itself. I currently lean toward weak rejection, but clearer evidence on these points could change my recommendation.

Reviewer_wH3GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WM-R1, a reinforcement learning framework for mobile GUI agents that uses a pretrained, frozen world model (Code2World-8B) as the training environment instead of a real Android environment. Given the current screenshot and an action, the world model generates the next UI state, enabling entire GRPO rollouts to be conducted in simulation. In addition, WM-R1 integrates the world model into the agent's reasoning via a <call wm> mechanism, allowing the agent to simulate candidate actions and revise them before execution. The method combines this environment-based training paradigm with a composite reward that accounts for task performance, trajectory length, and world-model usage. The authors train Qwen2.5-VL-3B and 7B policies on 2,000 filtered tasks collected from AndroidCode, GUI-Odyssey, and GUI-R1. Experiments show that WM-R1 outperforms GRPO baselines, including UI-R1, and using the world model only at inference time. Evaluations on AndroidWorld, GUI-Odyssey, ScreenSpot-Pro/V2, and AndroidControl, together with ablations, analyses of world-model usage, training-cost comparisons, and experiments with additional backbones, provide further evidence of the method's effectiveness.

Critical strengths
  1. A relatively novel approach. The paper explores using a pretrained world model not only for inference-time action simulation but also as a training environment for RL-based mobile GUI agents. This combination offers a potentially useful direction for reducing reliance on real-device interactions during policy optimization.
  2. Effectiveness across model scales and families. The method shows consistent improvements across Qwen2.5-VL (3B and 7B) and Qwen3-VL (4B and 8B), demonstrating its effectiveness across the evaluated model families and scales.
Critical weaknesses
  1. The problem setting and the scope of the claims need to be clarified. The abstract and introduction motivate WM-R1 by the cost, irreversibility, and noise associated with real-environment interaction, suggesting that the method aims to reduce reliance on real-environment rollouts during RL training. However, the main baseline, UI-R1, performs offline GRPO without real-environment interaction (Appendix A.4, Table 5). Therefore, the reported improvements demonstrate the benefit of world-model-based rollouts over offline training, but do not establish whether WM-R1 can effectively replace or reduce real-environment interaction in RL. The authors should clarify the intended problem setting and align their claims with the experimental evidence. In particular, they could either narrow the claims to offline RL settings or compare WM-R1 against methods trained with real-environment rollouts under comparable tasks and training budgets.
  2. The impact of world-model errors on policy learning is insufficiently investigated. Both policy training and task-success evaluation rely on simulated interactions, but the paper does not quantitatively evaluate how simulation errors accumulate over long-horizon trajectories, nor does it verify whether tasks deemed successful in simulation also succeed on real devices. This leaves the reliability of world-model-based RL insufficiently established.
Questions for authors
  1. How does WM-R1 compare with RL methods trained using real-environment rollouts under comparable tasks? Other questions, see weakness please.
2 comments

Dear Reviewer, thank you very much for your detailed review. We have addressed your concerns about comparisons with real-environment RL and the impact of world-model errors through additional experiments and clarifications in the revised manuscript. If time permits, we would be grateful for any additional feedback on our response.

Dear Reviewer,

I hope you are well. With less than a day left in the discussion period, I wanted to kindly follow up. Could you please let us know whether any concerns remain after our response? We would appreciate a brief reply and will respond promptly to any remaining points.

Thank you for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.