acceptodds
Under review as a conference paper at ICLR 2027

WISH: A Training-Free World-Model Invocation and Selection Harness for GUI Agents

Abstract

GUI agents can use world models to compare the predicted outcomes of candidate actions, but the evidence they need varies across screens and candidate actions. Existing methods commonly provide the agent with one world model per run and use a fixed pipeline or external rule to decide when to call it. We introduce **WISH**, a training-free harness that lets a frozen GUI agent autonomously decide whether to request world-model predictions and which world model each candidate action needs. To guide these decisions, we abstract recurring evidence needs from teacher explanations into conditions the agent can assess using the current GUI and candidate actions, and combine this guidance with supporting components in WISH. Before requesting a world model's predictions, the agent judges whether those predictions could change its choice among candidate actions. If so, it selects a world model to provide the information each candidate needs. With Qwen3-VL-8B on AndroidWorld, WISH achieves 50.0% task success, exceeding the strongest reproduced baseline by 7.8 percentage points. WISH requests world-model predictions on 21.7% of steps, compared with 23.1% for that baseline, and improves task success over the evaluated baselines across two GUI agents and three benchmarks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
GUI Agent, World Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RXMxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WISH lets a frozen GUI agent decide whether to call a world model and which type of world model to use for each candidate action. The decision rules are derived from teacher traces and combined with candidate filtering, memory, action checks, and recovery.

Critical strengths
  1. The paper identifies a real issue: an agent may be confidently wrong, making entropy-based gating unreliable.
  2. Different world models provide different kinds of evidence, so selecting among them is well motivated.
  3. The trace analysis is useful and makes the system easier to understand.
Critical weaknesses
  1. My main concern is that WISH changes much more than world-model routing. It also adds memory, action repair, safety checks, and recovery, while the baselines do not receive these improvements.
  2. The paper should separately test the “whether” decision with a fixed world model and the “which” decision with an existing gate.
  3. Selective world-model use does not necessarily mean low cost, since WISH introduces several additional LLM calls per step. Runtime and token usage are missing.
Questions for authors

see weakness

2 comments

My concerns have been addressed, and I'd like to maintain my score

Thank you for the follow-up and for taking the time to review our responses. We appreciate your confirmation that your concerns have been addressed.

Reviewer_hf2QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WISH is a training-free control harness that allows a frozen GUI agent to decide both whether it needs a world model and which type of world model should predict the outcome of each candidate action, rather than always using one fixed world model. In addition, WISH additionally introduces the training-free harness, including action selection, execution checks, memory progress, and recovery. The evaluation shows the improvements of task success on the benchmarks AndroidWorld, AndroidLab, and MobileWorld.

Critical strengths
  • Well-motivated problem. I am personally interested in the motivation of this paper. The agent should not blindly consume every available prediction. It should reason about what evidence would change its decision.
  • The “whether + which” decomposition is simple and intuitive.
  • Results are consistent across multiple settings.
Critical weaknesses
  • The “training-free” claim is arguable. The guidance is derived from teacher traces, so this is not “zero adaptation”. It is more like offline teach-guided prompt/rule distillation without parameter training.
  • Some component evidence is suggestive rather than statistically compelling. For example, in table 9, the state-action memory and recovery after repeated no-progress improves the success but no direct gains or loses.
  • Handcrafted rules may limit generality.
Questions for authors

Q1: How much of WISH’s improvement comes specifically from world-model invocation and selection, versus the other components of the harness? Q2: Can you isolate the contribution of the “which world model” decision? Q3: Is the learned/engineered routing principle general, or tied to the specific teacher traces and GUI setup?

Reviewer_sNWiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces WISH, a training-free harness that enables a frozen GUI agent to decide whether to request world-model predictions and which world model to use for each candidate action. It offers three prediction formats: local textual changes, complete textual states, and rendered HTML interfaces. Guidance manually abstracted from teacher explanations supports these decisions, alongside candidate preparation, execution checks, memory, and recovery. Experiments with Qwen3-VL-8B and Qwen3-VL-32B-Instruct show improvements over the evaluated baselines on AndroidWorld, AndroidLab, and MobileWorld.

Critical strengths
  1. The proposed harness makes invocation and model-assignment decisions before receiving predictions. This permits different candidates on the same screen to receive different forms of evidence without running every available world model, which is interesting.
  2. WISH achieves the highest reported performance in all six evaluated settings and indicates consistent improvements across benchmarks and model scales. On AndroidWorld, success increases from the strongest baseline’s 42.2% to 50.0% for Qwen3-VL-8B and from 44.0% to 54.3% for Qwen3-VL-32B-Instruct. Improvements also extend to AndroidLab and MobileWorld.
Critical weaknesses

The central idea is letting the agent decide whether to obtain extra evidence and which source to query. WM-R1 [1] and MWM [2] already gate world-model calls. WISH’s distinct elements are a “whether” test and per-candidate model assignment. Both are implemented as hand-written prompts abstracted from teacher reasons, and their isolated contribution appears small. In Table 5, the fixed-rule harness without any world model reaches 47/116, and adding the agent’s world-model choice raises this to only 50/116. The full system’s 58/116 relies largely on action repair, memory, recovery, and task-specific checks. The main-table baselines lack these components. The paper should (i) compare against a simple description-based selector, with each world model described in one sentence, inside the same harness, and (ii) report MWM baselines with the same non-WM components. This would separate gains from world-model orchestration from gains from GUI engineering.

[1] WM-R1: Training GUI Agents to Reason and Leverage World Models with Reinforcement Learning. arXiv, 2026. [2] How Mobile World Model Guides GUI Agents? arXiv, 2026.

Questions for authors

Could you report the performance of the full WISH system with world-model calls disabled, while holding candidate preparation, action repair, memory, recovery, and execution checks fixed? This ablation would help distinguish gains from GUI engineering from those attributable to world-model orchestration. Within that same fixed harness, can you compare WISH's teacher-derived guidance against a simple description-based selector, with each world model described in one sentence, and against MWM entropy-gated self-reflection? The main-table baselines lack WISH's non-WM components, so the reported margins currently mix the two sources of gain. A clear advantage over both would establish the guidance as a contribution beyond WM-R1 [1] and MWM [2].

[1] WM-R1: Training GUI Agents to Reason and Leverage World Models with Reinforcement Learning. arXiv, 2026. [2] How Mobile World Model Guides GUI Agents? arXiv, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.