- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LatentCUA, which predicts the consequences of GUI actions in a compact latent space to support action selection. It separately trains a screen encoder, an action-conditioned predictor, and a goal-conditioned scorer, then ranks three-step branches proposed by a vision-language planner. The authors analyze the modules through offline ablations and evaluate five planners on 361 OSWorld tasks. Direct world-model selection improves three lower-performing planners, while selective assistance with a shared threshold improves all five by 3.0–8.1 percentage points.
Critical strengths
- Representation, prediction, and scoring are trained separately, and Tables 1–3 test semantic retention, action sensitivity, and candidate discrimination. These experiments make the design choices testable rather than relying on prediction error alone.
- The OSWorld results demonstrate the value of cooperation with existing planners: direct selection can hurt strong planners, whereas selective assistance with a shared threshold improves all five. These multi-planner online gains are the main reason to support acceptance.
Critical weaknesses
- The online experiments do not separate gains from learning action consequences from gains due to additional candidate scoring. Offline dynamics ablations support the predictor design, but an online reranking control without future-state prediction on the same candidate trees is missing. The contribution of dynamics to complete-task improvement therefore remains uncertain.
- The quality–cost advantage of compact latent states lacks direct validation. There is no matched comparison with text- or screenshot-based prediction, nor measured inference cost for the complete system. The motivation of retaining decision-relevant information while reducing prediction cost is therefore only partially supported.
Questions for authors
- With the same candidate trees and selection rules, how does LatentCUA compare with direct action scoring without future-state prediction?
- With the same candidate count and planning depth, what are the task score and measured per-step cost relative to a text- or screenshot-based prediction method?
Response to Reviewer mHih: learned dynamics versus extra scoring, and quality versus cost
Thank you for noting that Tables 1 to 3 make the design choices testable and that the multi-planner online results support the paper. We address both weaknesses and questions below.
W1/Q1. Gains from learned dynamics versus additional candidate scoring. We agree that this control is needed. With Gemini 3 Flash and the same candidate trees on the 361 OSWorld tasks, we let the planner re-rank its own candidates in a second pass, using the current state but no predicted futures.
Scoring the same candidates again from the current state does not help. The Scorer also cannot gain anything without the dynamics. All candidates in a step share the same goal and history, and the Scorer receives no action, so the predicted future states are the only input that differs between candidates. Without them, every candidate gets the same score, Assisted never overrides the planner, and the result equals Agent. The gain over Agent therefore comes through the learned dynamics.
Offline results point in the same direction. Table 5 keeps the task goal, short-horizon goals, and history fixed and changes only the future states. Predicted futures keep most of the ranking signal of actual futures (68.6% versus 72.0% same-task three-way accuracy, and 70.3% versus 71.9% same-page pairwise accuracy).
W2/Q2. Quality and cost versus text- or screenshot-based prediction. Table 5 bounds the quality side. A perfect screenshot predictor would at best match the GT image column, which is 4.9 to 9.5 points above our predicted latents in outcome discrimination. The question is whether this gap is worth the cost of generating text or screenshots for every branch and step.
A fair comparison would keep the planner, candidate trees, and gate fixed and change only the world model. This is not possible with the existing systems. CUWM is trained on Microsoft Office applications and ViMo on Android apps, so both would need to be retrained on Ubuntu data with a different action space. Both also output screenshots, which our Scorer does not read, so a new scorer would be needed as well. R-WoM relies on an external tutorial corpus and uses an LLM both to simulate and to judge outcomes, so it would replace our Scorer as well as our Predictor. WMA is built for web navigation. It reads the page's accessibility tree, predicts a text description of the next page with a fine-tuned Llama-3.1-8B, and scores it with a prompted LLM, so it would need new desktop training data and would also replace our Scorer. In each case the comparison would change more than the world model.
For the complete system, LatentCUA's world model takes 4.28 s per decision on one RTX 4090: 1.16 s for screen encoding, 0.25 s for latent prediction of all branches, and 2.87 s for scoring.