- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes a latent world-action model for GUI agents that combines candidate-action scoring with auxiliary next-state prediction and contrastive learning. It avoids explicit screenshot generation during inference and evaluates performance on offline datasets and AndroidWorld.
Critical strengths
-
Avoiding future-screen reconstruction is a reasonable approach to reducing the substantial inference cost of pixel-space world models.
-
The paper evaluates multiple agent backbones and reports consistent online improvements, supported by component ablations.
Critical weaknesses
-
Limited offline gains relative to added cost. In Table 1, base agents often achieve comparable performance with fewer tokens and lower latency. For example, Qwen3-VL-8B improves by only 1.53 and 1.10 SR points on AndroidControl and AITZ. These results leave the practical value of adding world modeling insufficiently established.
-
The larger online gains are insufficiently explained. Section 4.2 offers an intuitive trajectory-level explanation but no analysis establishing it. Since the benchmarks, metrics, and gating thresholds differ, it remains unclear what drives the much larger online improvement.
-
Insufficiently explained figures. Figure 1 does not clearly identify the implementations behind “Code2Image” and “Diffusion2Image,” and the naming is inconsistent with Code2World elsewhere. Figure 2 does not clearly specify the datasets, sampled pairs, or analysis procedure behind the similarity distributions.
-
Disorganized presentation. The introduction relies heavily on Figures 1, 2, and 4 without adequately explaining their experimental context. Figure 4 appears several pages after its first reference, making the central motivation difficult to follow.
-
Unclear treatment of identity shortcut. Although the paper briefly defines the term, its explanation and measurement are scattered across the introduction, method, and appendix. A clearer verbal example and operational definition are needed, particularly for the latent-space version.
Questions for authors
-
Given the modest gains in Table 1, when does the proposed method justify its additional token and latency costs over the base agent? Are these gains robust across repeated evaluations?
-
What explains the much larger online gains? Can the authors distinguish trajectory-level benefits from differences in datasets, evaluation criteria, and gating thresholds?
-
Section B.4.2 specifies 116 task instances, but Table 2 percentages appear consistent with 112 tasks—for example, 61.61% corresponds to 69/112. Were any tasks excluded?
I will consider raise my score if questions are addressed.
5 comments
[Q2] Presentation Issues
We thank the reviewer for these helpful comments on presentation and apologize for the unclear parts.
[Insufficiently Explained Figures]
Figure 1. "Code2Image" was a typo for Code2World, and we have corrected it.
Figure 2. It compares three groups of image pairs:
- Natural Video: consecutive frames sampled at 1-second intervals from 30 REDS videos [1], yielding 120 pairs.
- GUI Transition: current and ground-truth next screenshots from 320 steps across 85 AndroidControl episodes.
- GUI WorldModel: Diffusion2Image's predicted next screenshots and the ground-truth next screenshots on the same 320 steps.
Each image is encoded by DINOv2 (dinov2-base) and SigLIP2 (siglip2-base-patch16-224), and we compute the cosine similarity of the -normalized features for each pair. We have added these details to Appendix B.2.5.
[Disorganized Presentation] Building on the clarifications above, we have made each figure self-contained at its first reference in the introduction:
- Figure 1: The introduction now states that it reports AndroidWorld results from Table 2.
- Figure 2: The caption now describes the compared pairs and links to the full analysis procedure in Appendix B.2.5.
- Figure 4: The introduction now summarizes its key finding where it is first cited, and the caption links to the measurement protocol in Appendix B.2.5. See Q3 for the definition and measurement of the identity shortcut.
[1] Nah et al. NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study. CVPR Workshops, 2019.
[Q3] Clarification on Identity Shortcut
We thank the reviewer for this suggestion. We agree that the explanation was scattered across the introduction, Section 3.2.1, Figure 4, and Appendix B.2.5, and that the latent-space criterion was not explicitly stated.
[Verbal Example] An identity shortcut is a prediction that stays close to the current state even though the action substantially changes the next state. For example, when a click opens a new page, a pixel-space world model may still output a screen nearly identical to the current one (Figure 4(a)).
[Operational Definition] A prediction is an identity shortcut when (i) the true transition is larger than the predicted one, , and (ii) the prediction stays close to the current state, . Shortcut severity is the fraction of predictions meeting both conditions in each transition-magnitude group (Figure 4(b)).
- Pixel space: is the mean absolute difference between 64×64 grayscale screenshots, with .
- Latent space: we apply the same criterion to the target-encoder latents and and the predicted latent , with as the cosine distance and .
We have consolidated the example and definition in Section 3.2.1 and described the measurement protocol for both spaces in Appendix B.2.5.
[Q4] Number of Evaluated Tasks
We thank the reviewer for this careful observation. Upon checking, we found that four Retro Music tasks in AndroidWorld were excluded. A conda environment conflict prevented these tasks from running for all methods, so every method was evaluated on the same 112 tasks.
To check whether this affects our conclusions, we resolved the conflict and re-ran the four tasks. None of the methods, including the base agent, succeeds on them. Adding these tasks thus scales all SRs by the same factor and leaves every ranking unchanged. To stay consistent with the reviewed manuscript, we keep the 112-task results during the rebuttal and have clarified this in Appendices B.3 and B.4.2. In the final version, we will report results on all 116 tasks and update these appendices accordingly.
[Q5] Cost Justification and Repeated Evaluation
We thank the reviewer for this question. As discussed in Q1, the offline gains in Table 1 appear modest because offline evaluation accepts only a single GT route. This reflects a limitation of the benchmark rather than of our method. Even so, our method improves SR in seven of eight offline settings, whereas pixel-space WMs reduce SR in every setting by up to 8.6 points. Online evaluation, in contrast, runs in interactive environments and accepts any successful route, making it a better setting to assess whether the gains justify the cost. As step-level gains compound over the trajectory, our method improves online SR by +4.5 to +12.5pp across all backbones, with the largest gains on Hard tasks (+5.6 to +11.1pp; Table 2). This costs only 1.1–1.8× the run time of the base agent, compared with 2.1–4.1× for Code2World and 4.5–18.2× for Diffusion2Image. We have clarified the offline–online distinction in Section 4.
| Table R13 (Qwen3-VL-8B) | Method | SR (5 seeds) | SR (Table 2) |
|---|---|---|---|
| AndroidWorld | Latent (Ours) | 41.43 ± 1.02 | 40.18 |
To verify robustness, we evaluate our model with Qwen3-VL-8B on AndroidWorld over 5 seeds (Table R13). It achieves 41.43 ± 1.02 SR, comparable to the reported 40.18 and far above the base agent (27.68). We have added these results to Appendix C.5.
[Q1] Offline–Online Evaluation Gap
We thank the reviewer for this important question. Below, we address the offline gains relative to cost, then explain what drives the larger online gains and how we distinguish it from differences in datasets and gating.
[Offline Gains and Cost] Offline gains are modest but consistent: our method improves SR in seven of eight offline settings, whereas pixel-space world models reduce SR in every setting. The added cost is also small: for Qwen3-VL-8B, our method adds 0.13 s per step over the base agent, compared with 10.18 s for Code2World and 166.98 s for Diffusion2Image (Table 1).
[Route Diversity] The larger online gains largely stem from how success is evaluated. Offline evaluation compares each step with a single demonstrated route, whereas online evaluation accepts any successful route. To quantify route diversity, we define a route as the sequence of actions in a successful episode and run each AndroidWorld task 29 times, counting the distinct successful routes observed. On AndroidWorld, 91.5–100% of tasks solved by the base agents admit at least two successful routes, with 6.5–8.3 routes per task on average (Table R11), and the same holds across world models (7.9–8.3; Table R12). This matters most for our method, since it intervenes by replacing the base agent's top action with an alternative candidate. Offline evaluation penalizes such corrections whenever the alternative differs from the logged route, whereas online evaluation credits them if they complete the task.
[Gating Threshold] To rule out the threshold difference (0.95 online vs. 1.0 offline), we vary the online threshold chi in Table R7. With the offline threshold (chi = 1.0), our method still reaches 37.5 SR, a +9.8pp gain over the base agent (27.68), far larger than the offline gains. The large online gain thus persists under matched thresholds.
[Dataset Differences] Candidate coverage is high in all settings (85.7–94.5%; Table R3), so the online gain is not explained by a richer candidate pool.
We have added the route-diversity and threshold analyses to Appendix C.6 and clarified in Section 4.2 that the offline–online gap mainly reflects route diversity rather than the gating threshold.