acceptodds
Under review as a conference paper at ICLR 2027

LatentCUA: A Decoupled World Model for GUI Prediction and Planning

Abstract

Computer-use agents typically choose actions from the current screen without explicitly reasoning about their future effects. We introduce LatentCUA, a compact world model that enables an agent to compare candidate actions through latent prediction. LatentCUA learns a semantic representation of GUI screens, predicts how this representation evolves under an action, and scores imagined future states against the agent's goals. Its modules are trained in separate stages, allowing representation learning, dynamics prediction, and state evaluation to use supervision suited to each role. During planning, a vision-language agent proposes candidate action sequences and LatentCUA ranks their predicted outcomes before execution. Offline ablations validate these design choices, with the final configuration providing the strongest overall balance of representation quality, multi-step prediction, and candidate discrimination. On OSWorld, direct world-model selection improves lower-performing planners, while selective assistance improves all five tested planners. We further provide a decision-sufficiency framework that relates representation, prediction, and scoring errors to action-selection quality.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Computer-use agent, world model

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_mHihOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LatentCUA, which predicts the consequences of GUI actions in a compact latent space to support action selection. It separately trains a screen encoder, an action-conditioned predictor, and a goal-conditioned scorer, then ranks three-step branches proposed by a vision-language planner. The authors analyze the modules through offline ablations and evaluate five planners on 361 OSWorld tasks. Direct world-model selection improves three lower-performing planners, while selective assistance with a shared threshold improves all five by 3.0–8.1 percentage points.

Critical strengths
  1. Representation, prediction, and scoring are trained separately, and Tables 1–3 test semantic retention, action sensitivity, and candidate discrimination. These experiments make the design choices testable rather than relying on prediction error alone.
  2. The OSWorld results demonstrate the value of cooperation with existing planners: direct selection can hurt strong planners, whereas selective assistance with a shared threshold improves all five. These multi-planner online gains are the main reason to support acceptance.
Critical weaknesses
  1. The online experiments do not separate gains from learning action consequences from gains due to additional candidate scoring. Offline dynamics ablations support the predictor design, but an online reranking control without future-state prediction on the same candidate trees is missing. The contribution of dynamics to complete-task improvement therefore remains uncertain.
  2. The quality–cost advantage of compact latent states lacks direct validation. There is no matched comparison with text- or screenshot-based prediction, nor measured inference cost for the complete system. The motivation of retaining decision-relevant information while reducing prediction cost is therefore only partially supported.
Questions for authors
  1. With the same candidate trees and selection rules, how does LatentCUA compare with direct action scoring without future-state prediction?
  2. With the same candidate count and planning depth, what are the task score and measured per-step cost relative to a text- or screenshot-based prediction method?
1 comment
AuthorsOpenReview ↗

Response to Reviewer mHih: learned dynamics versus extra scoring, and quality versus cost

Thank you for noting that Tables 1 to 3 make the design choices testable and that the multi-planner online results support the paper. We address both weaknesses and questions below.

W1/Q1. Gains from learned dynamics versus additional candidate scoring. We agree that this control is needed. With Gemini 3 Flash and the same candidate trees on the 361 OSWorld tasks, we let the planner re-rank its own candidates in a second pass, using the current state but no predicted futures.

Gemini 3 Flash on OSWorldChange from Agent (pp)
Planner re-ranks its own candidates, three seeds−4.2 ± 2.1
Scorer with predicted future states, Assisted gate+8.1

Scoring the same candidates again from the current state does not help. The Scorer also cannot gain anything without the dynamics. All candidates in a step share the same goal and history, and the Scorer receives no action, so the predicted future states are the only input that differs between candidates. Without them, every candidate gets the same score, Assisted never overrides the planner, and the result equals Agent. The gain over Agent therefore comes through the learned dynamics.

Offline results point in the same direction. Table 5 keeps the task goal, short-horizon goals, and history fixed and changes only the future states. Predicted futures keep most of the ranking signal of actual futures (68.6% versus 72.0% same-task three-way accuracy, and 70.3% versus 71.9% same-page pairwise accuracy).

W2/Q2. Quality and cost versus text- or screenshot-based prediction. Table 5 bounds the quality side. A perfect screenshot predictor would at best match the GT image column, which is 4.9 to 9.5 points above our predicted latents in outcome discrimination. The question is whether this gap is worth the cost of generating text or screenshots for every branch and step.

A fair comparison would keep the planner, candidate trees, and gate fixed and change only the world model. This is not possible with the existing systems. CUWM is trained on Microsoft Office applications and ViMo on Android apps, so both would need to be retrained on Ubuntu data with a different action space. Both also output screenshots, which our Scorer does not read, so a new scorer would be needed as well. R-WoM relies on an external tutorial corpus and uses an LLM both to simulate and to judge outcomes, so it would replace our Scorer as well as our Predictor. WMA is built for web navigation. It reads the page's accessibility tree, predicts a text description of the next page with a fine-tuned Llama-3.1-8B, and scores it with a prompted LLM, so it would need new desktop training data and would also replace our Scorer. In each case the comparison would change more than the world model.

For the complete system, LatentCUA's world model takes 4.28 s per decision on one RTX 4090: 1.16 s for screen encoding, 0.25 s for latent prediction of all branches, and 2.87 s for scoring.

Reviewer_aexROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LatentCUA is a CUA world model with three stages: a screen encoder that encodes screenshots into embeddings, a predictor that predicts future (+3) screenshot embeddings given action inputs, and a scorer that scores the future embeddings against the goal of the agent. The three components are decoupled, and each is trained with the other two parts frozen. The specific loss design and system components are shown to be useful through ablation studies of each part, and the whole system is shown to improve the performance of a CUA agent when used in a specific way to selectively override the agent planner's decisions.

Critical strengths
  • The methods are well written and not hard to understand.
  • Interesting system design and thoughtful design of the loss functions.
Critical weaknesses
  • The writing of the results discussion can be improved.
    • The figure captions are not helpful enough to understand the whole thing.
    • The metrics used need better explanation, including their definitions and the intuitions behind their relations to the overall system performance.
    • There are unexplained shorthands (e.g., Claim 3) / unlinked information in the appendix (e.g., the M1-M6 metrics for the scorers).
  • What is the rationale behind the Assisted setting? This needs some more explanation and perhaps experiments. Since this setting achieves the best performance in the planning results directly informed by agent performance, it makes the performance claim made by the paper somewhat entangled with the agent design (Agent S3). Is it possible to show that the same conclusion also holds for other CUA agents / models?
  • Would it be possible to compare the proposed planner / world model / agent system with other existing CUA world model approaches? All experiments so far are ablation studies / comparisons of system design approaches.
  • Cost seems to be what could make the latent world model appealing, but no cost comparison with existing approaches, or as an ablation, is made.
Questions for authors

Some of the design choices have no motivation (e.g., ablation results supporting the selection):

  • History length = 13 for the predictor
  • History length = 8 for the scorer
  • The 2x rule in the assisted planner replacement strategy

Is it possible to show their optimality?

1 comment
AuthorsOpenReview ↗

Response to Reviewer aexR: clarity, the Assisted rule, baselines, cost, and design choices

Thank you for the positive assessment of the system design and the loss functions. We answer each point below.

W1. Clarity of the results discussion. We will make each table and figure readable on its own. The definitions of M1 to M6 are now split between the Table 3 caption and the table, and we will move them all into the caption. Claim 3 is stated in Section 4, and where Section 6 uses it we will reference it with a link. The captions of Figures 1 and 2 will walk through one full decision.

W2. Rationale for Assisted and dependence on Agent S3. Claim 3 shows that Direct selection helps only when the world model's expected regret is smaller than the planner's. A strong planner usually ranks a good action first, so a world-model error can replace a correct choice. Assisted therefore keeps the planner's first choice unless the world model prefers another action by a clear margin. Table 4 matches this. Direct lowers the two strongest planners by 6.0 and 4.3 points, while Assisted improves all five by 3.0 to 8.1 points.

The same gate works for five planners from four model families, with base scores from 20.8 to 63.5, so the conclusion is not tied to one planner model. We build on Agent S3 because it already provides a grounding model, a structured action format, and short-horizon goals, which are the inputs our world model needs. Another agent system would need new fine-tuning data for its interface and would change the action distribution, so the world model would have to be retrained. We will state this scope in the paper.

W3. Comparison with other world models. A fair comparison would change only the world model and keep the planner, candidates, and gate fixed. This is not possible with existing systems. CUWM (Microsoft Office) and ViMo (Android) output screenshots, so they would need retraining on Ubuntu data and a new scorer. R-WoM (retrieved tutorials) and WMA (web accessibility trees) simulate outcomes in text and judge them with an LLM, so they would also replace our Scorer. As a partial reference on quality, scoring the real future screenshots, which is the best case for any screenshot predictor, is 4.9 to 9.5 points above our predicted latents (Table 5).

W4. Cost. Rolling out all branches in the latent space for three steps takes 0.25 s per decision on one RTX 4090, or 1.4% of the 18.2 s decision latency. A text or screenshot world model would instead generate a description or an image for every branch and step. With encoding and scoring, the world model takes 4.28 s per decision, and our reply to Reviewer Ddda gives the full breakdown.

Q. History lengths and the 2× rule. We swept the history length for each module.

Module and metricResult by history length (frames)
Predictor, one-step relative MSE ↓1: 0.821, 5: 0.67, 9: 0.48, 13: 0.478
Scorer, M1@8 accuracy ↑1: 18.5%, 4: 55.0%, 8: 69.7%, 10: 65.2%

Predictor error stops improving after 13 frames, and Scorer accuracy peaks at 8 frames, so the deployed lengths are the best in the tested range.

The 2× rule overrides the planner when the world model's preferred action has more than twice the softmax score of the planner's first choice, which equals a logit margin of log 2. A margin of 1× gives Direct, and an infinite margin gives Agent.

OSWorld score (override rate)1× (Direct)1.5×2× (deployed)3×∞ (Agent)
Gemini 3 Flash49.160.563.2 (25.5%)63.155.1 (0%)
GPT-5.4-mini59.268.167.0 (21.2%)62.963.5 (0%)

Margins from 1.5× to 3× keep Gemini 3 Flash 5.4 to 8.1 points above Agent. For GPT-5.4-mini, 1.5× is slightly better than 2×, and 3× is 0.6 points below Agent. Therefore, we do not claim that 2× is optimal. The best margin depends on the world model's error, which is unknown at test time, so we use one fixed margin for all planners instead of tuning it for each one.

Reviewer_DddaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LatentCUA, a decoupled latent world model for GUI prediction and planning. It sequentially trains a Screen Encoder, an action-conditioned Predictor, and a goal-conditioned State Scorer to evaluate candidate action branches proposed by a VLM planner before execution. Offline experiments validate the individual modules, and online evaluation on OSWorld shows that direct world-model selection improves the three weaker planners, while selective assistance improves all five tested planners by 3.0 to 8.1 percentage points.

Critical strengths
  1. World modeling is a promising approach for agent decision making, allowing agents to compare the expected outcomes of candidate actions before execution.
  2. The separate training approach gives each stage a focused learning objective while preserving the representations and capabilities learned in earlier stages.
  3. The paper provides extensive module-level ablations and online evaluation with five planners. These results help characterize the individual modules and show that selective world-model assistance can improve task performance.
Critical weaknesses
  1. The study argues that explicit future state generation is too costly, yet provides no runtime or computational cost measurements. At inference, each decision step still requires generating a depth-three candidate tree, encoding history frames, rolling out latent states, and scoring each branch with a VLM judge. A latency breakdown and end-to-end runtime comparison against the baseline agent are needed to assess whether the accuracy gains justify the additional test-time overhead.
  2. It remains unclear how much the learned latent dynamics contribute to the online OSWorld gains. During planning, the state scorer receives planner-generated short-horizon goal descriptions for each candidate alongside the predicted future states. Although the scorer receives no candidate action directly, these goals provide another path for candidate information. The scorer may therefore rank candidates by judging the relevance of these text goals to the task instruction and history, without making substantial use of the predicted states. Online ablations comparing against a scorer without predicted states are needed to contextualize the contribution of latent state prediction.
  3. Several empirical design choices are not clearly justified. For example, the predictor uses 13 history frames while the scorer uses 8, and the ablations only compare these settings against a single frame. Important details of the online planning loop, such as the choice of the 3-step horizon, the sensitivity to the fixed override threshold, and how multi-step branch scores are aggregated to select a root action, are also not clearly specified.
  4. The main paper is not self-contained, and the presentation in sec 3 can be improved. The training losses and several core mechanisms (e.g., competitive cross-attention) are introduced only as symbolic names or brief summaries in the main text. The authors are encouraged to put essential technical definitions into the main text and include a concrete running example showing how candidate generation, latent rollout, and scoring work together within each step.
Questions for authors

Please refer to weaknesses. Additinally,

  1. How much of the OSWorld improvement remains when the Scorer receives the same history and goals without predicted future states?
  2. What is the additional per-decision and end-to-end latency compared with the agent-only baseline (i.e., W/O world model)?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Ddda: latency, the Scorer without predicted states, and planning details

Thank you for the detailed feedback. We address the runtime cost, the role of predicted states, the planning details, and the clarity of Section 3 below.

W1/Q2. Online latency and computational cost. LatentCUA adds two costs to a standard agent: the planner generates a depth-three candidate tree instead of a single action, and the world model scores the branches. We profiled LatentCUA (Assisted) and a single-action agent without the world model on the same 361 OSWorld tasks, with Gemini 3 Flash as the planner and the world model on one RTX 4090.

Per decision or per taskSingle-action agentLatentCUA (Assisted)
Planner call, median8.0 s13.7 s
World model: encoding / prediction / scoringn/a1.16 / 0.25 / 2.87 s
Decision latency, median10.2 s18.2 s
Runtime per task, median126 s486 s

The world model takes 4.28 s per decision, or 23% of the decision latency. Latent prediction for all branches takes only 0.25 s, and most of the world-model time goes to screen encoding and scoring with the ShowUI-2B backbone. The larger part of the added latency comes from the planner generating the tree and retrying malformed outputs (13.7 s versus 8.0 s).

W2/Q1. Can the Scorer rank candidates from their goal descriptions alone? No. At each decision step, the Scorer receives one short-horizon goal for that step, together with the task goal and the eight-frame history, and every candidate is scored against this same goal. The Scorer also receives no candidate action. The predicted future states are therefore the only input that differs between candidates. If they are removed, for example by giving the Scorer copies of the current state, every candidate gets the same score, Assisted never overrides the planner, and the result equals Agent. The 8.1 pp gain of Assisted over Agent for Gemini 3 Flash (Table 4) can therefore come only through the predicted states. Section 3 says that each tree node carries a short-horizon goal, which suggests a separate goal per candidate. We will state clearly that the Scorer uses one goal per step.

Offline results show that the predicted states carry useful information. Table 5 keeps the goals and history fixed and changes only the future states, and predicted futures keep most of the ranking signal of actual futures.

Outcome-ranking testActual future latentsPredicted future latents
Same-task three-way accuracy72.0%68.6%
Same-page pairwise accuracy71.9%70.3%

W3. History length, rollout depth, and action selection.

History. We swept the history length for each module.

Module and metricResult by history length (frames)
Predictor, one-step relative MSE ↓1: 0.821, 5: 0.67, 9: 0.48, 13: 0.478
Scorer, M1@8 accuracy ↑1: 18.5%, 4: 55.0%, 8: 69.7%, 10: 65.2%

Predictor error falls quickly up to 9 frames and gives no further gain beyond 13 frames, so we use 13. Scorer accuracy peaks at 8 frames and drops at 10, so we use 8.

Rollout depth. We chose three steps from experience. Each rollout step needs an action from the planner, and actions more than three steps ahead are very hard to propose because they depend on screens the planner has not seen. Within three steps, the Predictor's relative MSE rises slowly from 0.478 to 0.565, compared with 1.0 for copying the current state (Table 2).

Action selection. Every root-to-leaf chain is rolled out to the same depth of three steps, and shorter chains are padded with null actions. The Scorer scores the state at each of the three depths, and the chain score is their weighted sum with weights 0.5, 0.3, and 0.2 for depths one, two, and three. The agent executes the root action of the highest-scoring chain, then replans from the next real screenshot. Assisted gives each root action the score of its best chain and overrides the planner's first action only when the best root's logit exceeds it by log 2. We will add these details to Section 3.

Override threshold. We swept the margin for the two strongest planners. A margin of 1× gives Direct, and an infinite margin gives Agent.

OSWorld score (override rate)1× (Direct)1.5×2× (deployed)3×∞ (Agent)
Gemini 3 Flash49.160.563.2 (25.5%)63.155.1 (0%)
GPT-5.4-mini59.268.167.0 (21.2%)62.963.5 (0%)

Margins of 1.5× and 2× improve over Agent for both planners, and only 3× falls slightly below Agent for GPT-5.4-mini (0.6 points), so the gain does not depend on the exact threshold.

W4. Clarity of Section 3. We will move the core loss definitions and the description of competitive cross-attention into the main text. We will also add a worked example to the appendix that follows one decision from candidate generation and short-horizon goals, through latent rollout and branch scoring, to the execution of the selected first action.

Reviewer_Ci3MOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LatentCUA, which combines a screenshot encoder that produces semantic latent slots, an predictor of future states, and a scorer to help a vision-language planner compare candidate GUI actions. On OSWorld, selective assistance improves the tested planners, while direct world-model selection helps some and hurts others. The paper does not compare against closely related GUI world models, leaving its relative advantage unclear.

Critical strengths
  • The prediction ablations (Table 2) test whether the world model distinguishes the consequences of alternative actions, not just predicts plausible future screens. This is important for establishing whether its predictions can support planning.
  • The online evaluation tests LatentCUA with planners of differing capability and distinguishes direct world-model selection from selective assistance. This makes the integration strategy more informative than assuming the world model should always override the planner.
Critical weaknesses
  • Prior approaches already predict the consequences of GUI actions and use imagined outcomes to guide computer-use planning. For example, CUWM (Computer-Using World Model) predicts the next UI state from a current state and candidate action, then uses test-time action search. R-WoM (Retrieval-augmented World Model) grounds simulated computer-use rollouts in retrieved tutorials and ranks them for planning. The novelty of LatentCUA is thus limited. Seperately, the evaluation lacks external world-model baselines, for example, CUWM and ViMo with public repo. The experiments should include adapted comparisons or a clear justification for why such comparisons are infeasible.

  • The paper does not isolate the benefit of predicting future states. For example, it is unclear whether the improvements in task performance come from the imagined future states or simply from an additional evaluation of candidate actions using the current state.

  • Table 4 mentioned that the world model can hurt stronger planners, but it is unclear under what conditions this occurs and whether the selective-assistance rule reliably mitigates such failures.

  • The complete workflow is complex, and it is unclear the online runtimt cost and computational overhead of using LatentCUA in practice.

Questions for authors

See the Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ci3M: novelty, predicted futures, the gate, and runtime

Thank you for the careful review. We address the four weaknesses below.

W1. Novelty and external baselines. We agree that prior GUI world models also predict action outcomes for planning (CUWM, R-WoM, WMA, and ViMo, Section 2). These methods generate each predicted outcome as text or a screenshot, which is expensive for every branch and step. LatentCUA differs in two ways.

  1. Prediction in a latent space. Like JEPA world models such as V-JEPA and DINO-WM, LatentCUA predicts future states in a learned latent space instead of generating pixels or text. Rolling out every branch of a depth-three tree for three steps takes 0.25 s per decision (W4).
  2. How the latent is built and judged. The Encoder compresses each screenshot into 32 latent slots, trained with self-supervision and alignment to other modalities: feature reconstruction of the screenshot, alignment with screen narrations, and matching of the visible GUI elements. An adapter then maps the slots into the visual-token space of ShowUI-2B, so the Scorer reads a predicted state the way the model reads a screenshot and can use its pretrained knowledge of GUIs to judge progress toward the goal. Neither prediction nor scoring generates pixels or text.

A fair baseline would change only the world model and keep the planner, candidates, and gate fixed. This is not possible with existing systems. CUWM (Microsoft Office) and ViMo (Android) output screenshots, so they would need retraining on Ubuntu data and a new scorer. R-WoM (retrieved tutorials) and WMA (web accessibility trees) simulate outcomes in text and judge them with an LLM, so they would also replace our Scorer. Each comparison would change more than the world model.

W2. Do the gains come from predicted futures? Three results support this.

  1. Without the sibling loss, same-screen outcome matching drops from 73.5% to 38.4% (Table 2, chance 21.0%), so the Predictor uses the action.

  2. In Table 5, the Scorer gets the same goals and history, and only the next three states change. Predicted futures keep most of the signal of actual futures when ranking successful against failed trajectories.

    Outcome-ranking testActual future latentsPredicted future latents
    Same-task three-way accuracy72.0%68.6%
    Same-page pairwise accuracy71.9%70.3%
  3. Scoring without predicted futures does not help. When Gemini 3 Flash re-ranks its own candidates, success changes by −4.2 ± 2.1 pp (three seeds). The Scorer also cannot rank without predicted futures. All candidates in a step share the same goal, and the Scorer receives no action, so without predicted states every candidate gets the same score and Assisted reduces to Agent.

W3. Strong planners and the gate. Claim 3 (Section 4) shows that Direct helps only when the world model's expected regret is smaller than the planner's. A strong planner usually ranks a good action first, so a world-model error can replace a correct choice. Assisted therefore keeps the planner's first choice unless the world model prefers another action by a fixed logit margin of log 2, so small gaps that may come from model error do not trigger an override. The same margin is used for all planners.

PlannerAgent scoreDirect changeAssisted changeOverride rate
Gemini 3 Flash55.1−6.0 pp+8.1 pp25.5%
GPT-5.4-mini63.5−4.3 pp+3.5 pp21.2%
Kimi-K2.533.4+4.7 pp+5.3 pp45.6%
Qwen3-VL-30B20.8+4.5 pp+3.4 pp38.3%
Qwen3-VL-Flash26.7+2.7 pp+3.0 pp30.1%

Assisted improves all five planners, including the two that Direct hurts, and overrides the strong planners less often.

W4. Online cost. We profiled LatentCUA (Assisted) and a single-action agent on the 361 OSWorld tasks, with Gemini 3 Flash as the planner and the world model on one RTX 4090.

Per decision or per taskSingle-action agentLatentCUA (Assisted)
Planner call, median8.0 s13.7 s
World model: encoding / prediction / scoringn/a1.16 / 0.25 / 2.87 s
Decision latency, median10.2 s18.2 s
Runtime per task, median126 s486 s

The world model takes 4.28 s per decision (23% of the latency), and latent prediction takes only 0.25 s. Most of the added latency comes from the planner generating a depth-three tree instead of one action (13.7 s versus 8.0 s). Direct and Assisted cost the same, since the gate only changes which action is executed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.