acceptodds
Under review as a conference paper at ICLR 2027

Latent World Action Model for GUI Agents

Abstract

Pixel-space world models have recently emerged as a promising approach for GUI agents, predicting future screens prior to execution and subsequently re-perceiving them for action selection. However, this two-stage paradigm faces two key challenges in GUI environments: (L1) the high computational cost of pixel reconstruction and (L2) the need to reconstruct large and diverse appearance-level details beyond what is required for action selection. In practice, this screen-generation objective can induce an identity shortcut problem, where future-state prediction largely preserves the current state rather than capturing action-induced changes. In this work, we reformulate GUI world modeling by shifting the prediction target from full-screen reconstruction to action-induced future information in latent space. Concretely, we leverage a large multimodal model as the latent world model, whose shared representations support two dedicated heads for both action selection and predictive transition modeling. The action head directly translates these latent representations into action selection, while the transition head captures action-conditioned future information. To mitigate the identity shortcut, we further introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes. Extensive experiments across offline and online GUI settings show that our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models, respectively, while substantially reducing inference latency by up to 98.9%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agents, GUI World Models, Vision-Language-Action Models, Mobile Agents, Latent Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_DwCsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes a latent world-action model for GUI agents that combines candidate-action scoring with auxiliary next-state prediction and contrastive learning. It avoids explicit screenshot generation during inference and evaluates performance on offline datasets and AndroidWorld.

Critical strengths
  • Avoiding future-screen reconstruction is a reasonable approach to reducing the substantial inference cost of pixel-space world models.

  • The paper evaluates multiple agent backbones and reports consistent online improvements, supported by component ablations.

Critical weaknesses
  • Limited offline gains relative to added cost. In Table 1, base agents often achieve comparable performance with fewer tokens and lower latency. For example, Qwen3-VL-8B improves by only 1.53 and 1.10 SR points on AndroidControl and AITZ. These results leave the practical value of adding world modeling insufficiently established.

  • The larger online gains are insufficiently explained. Section 4.2 offers an intuitive trajectory-level explanation but no analysis establishing it. Since the benchmarks, metrics, and gating thresholds differ, it remains unclear what drives the much larger online improvement.

  • Insufficiently explained figures. Figure 1 does not clearly identify the implementations behind “Code2Image” and “Diffusion2Image,” and the naming is inconsistent with Code2World elsewhere. Figure 2 does not clearly specify the datasets, sampled pairs, or analysis procedure behind the similarity distributions.

  • Disorganized presentation. The introduction relies heavily on Figures 1, 2, and 4 without adequately explaining their experimental context. Figure 4 appears several pages after its first reference, making the central motivation difficult to follow.

  • Unclear treatment of identity shortcut. Although the paper briefly defines the term, its explanation and measurement are scattered across the introduction, method, and appendix. A clearer verbal example and operational definition are needed, particularly for the latent-space version.

Questions for authors
  • Given the modest gains in Table 1, when does the proposed method justify its additional token and latency costs over the base agent? Are these gains robust across repeated evaluations?

  • What explains the much larger online gains? Can the authors distinguish trajectory-level benefits from differences in datasets, evaluation criteria, and gating thresholds?

  • Section B.4.2 specifies 116 task instances, but Table 2 percentages appear consistent with 112 tasks—for example, 61.61% corresponds to 69/112. Were any tasks excluded?

I will consider raise my score if questions are addressed.

5 comments

[Q1] Offline–Online Evaluation Gap

We thank the reviewer for this important question. Below, we address the offline gains relative to cost, then explain what drives the larger online gains and how we distinguish it from differences in datasets and gating.

[Offline Gains and Cost] Offline gains are modest but consistent: our method improves SR in seven of eight offline settings, whereas pixel-space world models reduce SR in every setting. The added cost is also small: for Qwen3-VL-8B, our method adds 0.13 s per step over the base agent, compared with 10.18 s for Code2World and 166.98 s for Diffusion2Image (Table 1).

[Route Diversity] The larger online gains largely stem from how success is evaluated. Offline evaluation compares each step with a single demonstrated route, whereas online evaluation accepts any successful route. To quantify route diversity, we define a route as the sequence of actions in a successful episode and run each AndroidWorld task 29 times, counting the distinct successful routes observed. On AndroidWorld, 91.5–100% of tasks solved by the base agents admit at least two successful routes, with 6.5–8.3 routes per task on average (Table R11), and the same holds across world models (7.9–8.3; Table R12). This matters most for our method, since it intervenes by replacing the base agent's top action with an alternative candidate. Offline evaluation penalizes such corrections whenever the alternative differs from the logged route, whereas online evaluation credits them if they complete the task.

Table R11 (Base Agent)Avg. Successful RoutesTasks with ≥2 Routes (%)
GPT-5.66.892.2
Gemini-3.6-Flash6.591.5
GUI-Owl-7B8.0100.0
Qwen3-VL-8B8.3100.0
Qwen3-VL-32B7.997.9
Table R12 (Qwen3-VL-8B)Avg. Successful RoutesTasks with ≥2 Routes (%)
Base8.3100.0
Code2World7.996.9
Diffusion2Image7.994.7
Latent (Ours)8.195.1

[Gating Threshold] To rule out the threshold difference (0.95 online vs. 1.0 offline), we vary the online threshold chic_{\text{hi}} in Table R7. With the offline threshold (chic_{\text{hi}} = 1.0), our method still reaches 37.5 SR, a +9.8pp gain over the base agent (27.68), far larger than the offline gains. The large online gain thus persists under matched thresholds.

Table R7 ( chic_{\text{hi}} )Gating Ratio (%)AndroidWorld SRRun Time (s/task)
0.8025.938.4220.4
0.95 (Ours)9.240.2223.8
1.00 (= offline)73.637.5236.1

[Dataset Differences] Candidate coverage is high in all settings (85.7–94.5%; Table R3), so the online gain is not explained by a richer candidate pool.

Table R3 (Qwen3-VL-8B)(a) Coverage (%)(b) Gating Ratio (%)(c) No-Match Ratio (%)
AndroidControl85.74.339.4
AITZ86.04.851.3
GUI-Odyssey88.90.744.2
AndroidWorld94.59.2–

We have added the route-diversity and threshold analyses to Appendix C.6 and clarified in Section 4.2 that the offline–online gap mainly reflects route diversity rather than the gating threshold.

[Q2] Presentation Issues

We thank the reviewer for these helpful comments on presentation and apologize for the unclear parts.

[Insufficiently Explained Figures]

Figure 1. "Code2Image" was a typo for Code2World, and we have corrected it.

Figure 2. It compares three groups of image pairs:

  • Natural Video: consecutive frames sampled at 1-second intervals from 30 REDS videos [1], yielding 120 pairs.
  • GUI Transition: current and ground-truth next screenshots from 320 steps across 85 AndroidControl episodes.
  • GUI WorldModel: Diffusion2Image's predicted next screenshots and the ground-truth next screenshots on the same 320 steps.

Each image is encoded by DINOv2 (dinov2-base) and SigLIP2 (siglip2-base-patch16-224), and we compute the cosine similarity of the ℓ2\ell_2-normalized features for each pair. We have added these details to Appendix B.2.5.

[Disorganized Presentation] Building on the clarifications above, we have made each figure self-contained at its first reference in the introduction:

  • Figure 1: The introduction now states that it reports AndroidWorld results from Table 2.
  • Figure 2: The caption now describes the compared pairs and links to the full analysis procedure in Appendix B.2.5.
  • Figure 4: The introduction now summarizes its key finding where it is first cited, and the caption links to the measurement protocol in Appendix B.2.5. See Q3 for the definition and measurement of the identity shortcut.

[1] Nah et al. NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study. CVPR Workshops, 2019.

[Q3] Clarification on Identity Shortcut

We thank the reviewer for this suggestion. We agree that the explanation was scattered across the introduction, Section 3.2.1, Figure 4, and Appendix B.2.5, and that the latent-space criterion was not explicitly stated.

[Verbal Example] An identity shortcut is a prediction that stays close to the current state even though the action substantially changes the next state. For example, when a click opens a new page, a pixel-space world model may still output a screen nearly identical to the current one (Figure 4(a)).

[Operational Definition] A prediction s^t+1\hat{s}_{t+1} is an identity shortcut when (i) the true transition is larger than the predicted one, d(st,st+1)>d(st,s^t+1)d(s_t, s_{t+1}) > d(s_t, \hat{s}_{t+1}), and (ii) the prediction stays close to the current state, d(st,s^t+1)<δd(s_t, \hat{s}_{t+1}) < \delta. Shortcut severity is the fraction of predictions meeting both conditions in each transition-magnitude group (Figure 4(b)).

  • Pixel space: dd is the mean absolute difference between 64×64 grayscale screenshots, with δ=0.08\delta = 0.08.
  • Latent space: we apply the same criterion to the target-encoder latents ztz_t and zt+1z_{t+1} and the predicted latent z^t+1\hat{z}_{t+1}, with dd as the cosine distance and δ=0.03\delta = 0.03.

We have consolidated the example and definition in Section 3.2.1 and described the measurement protocol for both spaces in Appendix B.2.5.

[Q4] Number of Evaluated Tasks

We thank the reviewer for this careful observation. Upon checking, we found that four Retro Music tasks in AndroidWorld were excluded. A conda environment conflict prevented these tasks from running for all methods, so every method was evaluated on the same 112 tasks.

To check whether this affects our conclusions, we resolved the conflict and re-ran the four tasks. None of the methods, including the base agent, succeeds on them. Adding these tasks thus scales all SRs by the same factor and leaves every ranking unchanged. To stay consistent with the reviewed manuscript, we keep the 112-task results during the rebuttal and have clarified this in Appendices B.3 and B.4.2. In the final version, we will report results on all 116 tasks and update these appendices accordingly.

[Q5] Cost Justification and Repeated Evaluation

We thank the reviewer for this question. As discussed in Q1, the offline gains in Table 1 appear modest because offline evaluation accepts only a single GT route. This reflects a limitation of the benchmark rather than of our method. Even so, our method improves SR in seven of eight offline settings, whereas pixel-space WMs reduce SR in every setting by up to 8.6 points. Online evaluation, in contrast, runs in interactive environments and accepts any successful route, making it a better setting to assess whether the gains justify the cost. As step-level gains compound over the trajectory, our method improves online SR by +4.5 to +12.5pp across all backbones, with the largest gains on Hard tasks (+5.6 to +11.1pp; Table 2). This costs only 1.1–1.8× the run time of the base agent, compared with 2.1–4.1× for Code2World and 4.5–18.2× for Diffusion2Image. We have clarified the offline–online distinction in Section 4.

Table R13 (Qwen3-VL-8B)MethodSR (5 seeds)SR (Table 2)
AndroidWorldLatent (Ours)41.43 ± 1.0240.18

To verify robustness, we evaluate our model with Qwen3-VL-8B on AndroidWorld over 5 seeds (Table R13). It achieves 41.43 ± 1.02 SR, comparable to the reported 40.18 and far above the base agent (27.68). We have added these results to Appendix C.5.

Reviewer_sy8pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a latent world-action model for GUI agents that replaces explicit next-screen reconstruction with predictive supervision in a latent representation space. A shared LMM representation is used both to score candidate actions and to predict the latent representation of the resulting next state. To discourage an identity shortcut in which predictions preserve the current state instead of modeling action-induced changes, the method uses an action-aware contrastive objective with the actual next state as the positive and the current state and unrelated transitions as negatives. The transition head is used during training, while inference directly scores candidate actions from the learned latent representation. The approach is evaluated in offline, OOD, and interactive GUI benchmarks across several agent backbones.

Critical strengths
  1. The formulation is well motivated and practically consequential. Avoiding full future-screen generation removes a very expensive intermediate representation that is not obviously necessary for action selection. The proposed latent formulation is simple, efficient, and directly targeted at this mismatch.

  2. The empirical evaluation is broad and generally convincing. The method is tested across multiple GUI benchmarks and agent backbones, including online AndroidWorld rollouts, and demonstrates substantial latency advantages over pixel-space world models. The component ablation also shows incremental benefit from transition supervision and the contrastive objective.

Critical weaknesses

Conceptual overclaiming. At test time, the transition head is completely discarded, and no future state representations, rollouts, or latent dynamics are evaluated or simulated during inference. The decision-making process reduces strictly to a single-pass policy / candidate re-ranker fine-tuned with an auxiliary dynamics loss (LTransition\mathcal{L}_{\text{Transition}}). Framing a standard policy model regularized by auxiliary transition prediction as a "World Model" stretches the definition in the literature.

Questions for authors
  1. Can the authors report the action-head-only baseline across all main backbones and evaluation settings, rather than only in the module ablation? This would clarify how much of the improvement consistently comes from future-state supervision rather than improved candidate rescoring.

  2. How does the method compare against alternative auxiliary representation-learning objectives with the same action head and training budget but without explicit next-state prediction? This would help establish that modeling future transitions, rather than simply adding representation-level supervision, is responsible for the incremental gains.

6 comments

[Q1] Clarifying the "World Model" Terminology

We thank the reviewer for this insightful comment. As stated in the paper (Section 1, Figure 3, and Appendix B.2.2), we remove the transition head at inference, and our model performs no future-state rollout or simulation. It acts as a single-pass candidate scorer whose representations are shaped by future-state supervision during training. We agree, however, that some of our wording could still suggest otherwise.

We used "World-Action Model" to describe this training-time coupling of future prediction and action learning, following recent work where future prediction serves as an auxiliary objective removed at inference [1–3]. We compare against pixel-space world models because they play the same role in the pipeline, namely evaluating candidate actions with predictive knowledge. Our results show that this knowledge can be learned into the scorer during training rather than simulated at inference. Importantly, the gain does not come from generic auxiliary supervision: predicting the current state yields little improvement, whereas our contrastive objective targeting action-induced changes does (Table R10).

Table R10 (Qwen3-VL-8B)Transition TargetAndroidWorld SR
Group I–27.68
Group II–36.61
ControlCurrent state ztz_t (point-wise)37.50
Group IIINext state zt+1z_{t+1} (point-wise)38.39
Group IVNext state zt+1z_{t+1} (contrastive)40.18

Following the reviewer's suggestion, we have revised the wording that may imply inference-time simulation:

  • We relabeled the inference module in Figure 3(b) from "GUI World Model" to "GUI World-Action Model."
  • We replaced "world-model rescoring" with "candidate rescoring" (Figure 3 caption and Appendix B.2.3).
  • We replaced "world-model inference latency" with "inference latency" (Section 1).

[1] Zhang et al. From World Models to World Action Models: A Concise Tutorial for Robotics. arXiv:2607.00836, 2026.
[2] Soleymanzadeh et al. PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models. arXiv:2608.27609, 2026.
[3] Wu et al. FutureDuet: Decoupling Observation Access from Future Supervision in World Action Models. arXiv:2609.34362, 2026.

[Q2] Clarification on the Action Head

We thank the reviewer for raising this important point, which several reviewers shared (nbFF's Q1, CZ5M's Q1). The gain of the latent action head does not come from candidate rescoring alone. Rather, it builds up step by step through our two contributions: joint VLA training and latent representation learning.

Table R1 shows this decomposition directly. The conventional two-stage rescoring approach (Code2World) improves SR by 4.46pp over the base agent (R-I → R-II), accounting for about half of the total 8.93pp gain. For a fair comparison with our joint VLA training, R-III fine-tunes the Code2World scorer under the same supervision, adding another 3.57pp (R-II → R-III). Replacing it with our lightweight latent action head adds a further 0.89pp (R-III → R-IV), reaching the gain of Group II in Table 4.

Table R1 (Group)Action ScorerFTLatentSR
R-I. Base Agent✗✗✗27.68
R-II. Candidate Rescoring✓✗✗32.14
R-III. FT✓✓✗35.71
R-IV. Latent Action Head✓✓✓36.61

We also report the action-head-only baseline across additional benchmarks and backbones (Tables R8 and R9).

Table R8 (Qwen3-VL-8B)AndroidControl (ID)GUI-Odyssey (OOD)
Base55.652.1
Group II55.451.4
Group IV57.152.1
Table R9 (GUI-Owl-7B)AndroidWorld (Online)
Base15.2
Group II20.5
Group IV22.3

We have renamed the "action head" to the "latent action head," added the rows of Table R1 to Table 4, and discussed these results in Section 4.3.

[Q3] Action-Head-Only Baseline across Settings

We thank the reviewer for this suggestion. As detailed in our response to CZ5M's Q3, we report the action-head-only baseline (Table 4, Group II) in additional settings: offline ID (AndroidControl) and OOD (GUI-Odyssey) evaluation with Qwen3-VL-8B (Table R8), and online evaluation on AndroidWorld with GUI-Owl-7B (Table R9). Our full model consistently outperforms Group II, by +1.7, +0.7, and +1.8, respectively. Notably, on AndroidControl and GUI-Odyssey, Group II even falls below the base agent, whereas the full model recovers or surpasses it. This shows that future-state supervision consistently contributes beyond improved candidate rescoring.

We have added these results to Appendix C.3 and will include the remaining backbones and settings in the final version.

[Q4] Clarification on the Next-State Prediction

We thank the reviewer for this suggestion. To isolate the effect of future prediction, we add a control in Table R10. It uses the same transition head, loss, and training budget as Group III, but predicts the current state instead of the next state. Point-wise next-state prediction (Group III, 38.4) is only marginally above this control (37.5), suggesting that it captures little beyond current-state information. This reflects the identity shortcut discussed in Section 3.2.1. Only when our contrastive objective explicitly separates the predicted future from the current state does SR clearly improve, to 40.2 (Group IV).

Table R10 (Qwen3-VL-8B)Transition TargetAndroidWorld SR
Group I–27.68
Group II–36.61
ControlCurrent state ztz_t (point-wise)37.50
Group IIINext state zt+1z_{t+1} (point-wise)38.39
Group IVNext state zt+1z_{t+1} (contrastive)40.18
Table R13 (Qwen3-VL-8B)MethodSR (5 seeds)SR (Table 2)
AndroidWorldLatent (Ours)41.43 ± 1.0240.18

This gap over the control (2.7pp) is more than twice the seed standard deviation of our model (±1.02; Table R13). The gain thus comes from learning action-induced changes, not from extra supervision alone. We have added this control to Appendix C.4.

Reviewer_CZ5MOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a latent world-action model for GUI agents that predicts action-conditioned future representations in latent space rather than reconstructing future screenshots. A frozen GUI agent first generates candidate actions, which are jointly encoded with the current screenshot, task goal, and interaction history by a LoRA-adapted Qwen3-VL-8B-Instruct. The model is trained with an action-scoring objective supervised by annotated actions and an auxiliary contrastive objective that predicts the next-state embedding of the annotated action from this shared representation. At inference, the transition prediction head is discarded, and a confidence gate determines whether the base agent's action choices should be rescored. The key idea is to use latent transition prediction to improve action selection without requiring explicit future-screen generation and re-perception. The method is evaluated on AndroidControl, AITZ, GUI-Odyssey, and AndroidWorld, using GPT-5.6, Gemini 3.6 Flash, GUI-Owl-7B, Qwen3-VL-8B, and Qwen3-VL-32B, and compared against base agents, Code2World, and a diffusion-based screenshot world model. The authors report relative success-rate gains of up to 31.9% and 40.6% over pixel-space world models in offline and online settings, respectively, along with up to a 98.9% reduction in world-model inference latency.

Critical strengths
  1. Well-motivated critique of pixel-space GUI world models and substantial inference speedup. The paper identifies a clear practical bottleneck in the "Render-and-Re-Perceive" paradigm for GUI agents: generating full future screenshots or HTML code is computationally expensive and prone to failure when GUI transitions involve large, discontinuous visual changes (Figures A1–A2). By shifting future-state supervision to a training-time auxiliary objective in latent space and discarding the transition head at test time, the proposed method reduces online inference run time on AndroidWorld from 370.9–1453.6s (Code2World) and 789.9–3154.6s (Diffusion2Image) down to 141.4–960.0s—incurring only modest overhead over the base agents (125.8–703.6s) while consistently achieving higher task success rates than both pixel-space world models across all five backbones (Table 2).
  2. Consistent improvements across diverse agent backbones with modest training costs. The proposed method uses a single LoRA-adapted Qwen3-VL-8B latent model, with only 43.6M trainable parameters and 8 hours of training on a single A6000 GPU (Table A1), to rescore candidate actions from five diverse agent backbones, including proprietary API models (GPT-5.6, Gemini-3.6-Flash) and open-weight models (GUI-Owl-7B, Qwen3-VL-8B, Qwen3-VL-32B). On AndroidWorld (Table 2), the method consistently improves overall success rates across all five backbones by 4.46–12.50 percentage points, with gains of 5.56–11.11 points on hard tasks. These results demonstrate the potential of the proposed approach as a lightweight, plug-and-play enhancement for diverse GUI agents.
  3. A simple action-aware contrastive transition objective supported by broad empirical evaluation. To prevent latent predictions from collapsing to the current-state representation, the method uses an InfoNCE objective that treats the encoded next state zt+1z_{t+1} as the positive target and contrasts it against both the current state ztz_t and competing future states stored in a 192-transition FIFO memory bank. The ablation in Table 4 shows progressive improvements on AndroidWorld from adding the action head (27.68% to 36.36%), point-wise transition prediction (36.36% to 38.18%), and contrastive transition learning (38.18% to 40.18%). The evaluation further covers in-distribution offline datasets (AndroidControl and AITZ), out-of-distribution cross-app trajectories (GUI-Odyssey), and online emulator rollouts (AndroidWorld), providing evidence for the method’s effectiveness across different evaluation settings.
Critical weaknesses
  1. Limited ablation of the Action Head-only baseline (Group II) across benchmarks and backbones. In the module ablation on AndroidWorld with Qwen3-VL-8B (Table 4), adding the supervised Action Head alone (Group II, which rescores candidates without the transition head or contrastive loss) already improves success rate from 27.68% to 36.36% (+8.68 points), while adding the transition head (Group III, 38.18%) and contrastive learning (Group IV, 40.18%) contributes an additional +3.82 points. However, Group II is only evaluated on AndroidWorld with a single backbone (Qwen3-VL-8B). Reporting Group II on the offline datasets (Table 1), the OOD benchmark GUI-Odyssey (Table 3, where the overall SR gain for Qwen3-VL-8B is +0.06 points from 52.07% to 52.13%), or additional backbones would more clearly isolate the contribution of latent transition modeling from candidate reranking.
  2. Lack of sensitivity analysis on key hyperparameters and the confidence-based gating mechanism. The proposed framework relies on several key hyperparameters listed in Table A1, including the number of candidates (K=3K=3), transition loss weight (λ=1.0\lambda=1.0), contrastive memory bank size (192192), and the confidence gate thresholds (χ=0.95\chi = 0.95 for online evaluation and χ=1.0\chi = 1.0 for offline evaluation). At inference, the confidence gate (Algorithm 2) determines when the world model is invoked versus bypassed, which directly controls the trade-off between task success rate and inference latency. However, the paper does not report how frequently the gate is triggered across different backbones or provide sensitivity ablations for these hyperparameters.
Questions for authors
  1. Isolating the contribution of latent transition supervision (Group II vs. Group IV): How does Group II compare with the full model (Group IV) on the offline benchmarks (Table 1), the OOD benchmark GUI-Odyssey (Table 3), or on another backbone in AndroidWorld (Table 2)?
  2. Invocation rate of the confidence gate and hyperparameter sensitivity: Under the confidence thresholds χ=0.95\chi = 0.95 (online) and χ=1.0\chi = 1.0 (offline), what percentage of steps actually trigger the latent world-action model across the evaluated backbones in Tables 1 and 2? Additionally, how sensitive is the model's performance to key hyperparameters such as the transition loss weight λ\lambda, memory bank size, and confidence threshold χ\chi?
7 comments

[Q1] Clarification on the Action Head

We thank the reviewer for raising this issue, which several reviewers shared (nbFF's Q1, sy8p's Q2). To clarify where the gain comes from, we decompose the gain of the latent action head in Table R1. In the conventional two-stage paradigm (Code2World), candidate rescoring alone improves SR by 4.46pp over the base agent (R-I → R-II), accounting for about half of the total 8.93pp gain. Unlike this paradigm, our latent action head is trained within the VLA pipeline. For a fair comparison, we fine-tune the Code2World scorer under the same supervision, which adds another 3.57pp (R-II → R-III). Replacing it with our lightweight latent action head adds a further 0.89pp (R-III → R-IV).

Table R1 (Group)Action ScorerFTLatentSR
R-I. Base Agent✗✗✗27.68
R-II. Candidate Rescoring✓✗✗32.14
R-III. FT✓✓✗35.71
R-IV. Latent Action Head✓✓✓36.61

To further isolate the contribution of latent transition modeling, we also report Group II across additional benchmarks and backbones in our response to Q3 (Tables R8 and R9).

Table R8 (Qwen3-VL-8B)AndroidControl (ID)GUI-Odyssey (OOD)
Base55.652.1
Group II55.451.4
Group IV57.152.1
Table R9 (GUI-Owl-7B)AndroidWorld (Online)
Base15.2
Group II20.5
Group IV22.3

We have renamed the "action head" to the "latent action head," added the rows of Table R1 to Table 4, and discussed these results in Section 4.3.

[Q2] Additional Ablations

We thank the reviewer for this suggestion, which several reviewers shared. To address it, we conducted the requested ablations on our method, using Qwen3-VL-8B as the base agent.

[Number of Candidates] We first vary the number of candidates K from 2 to 5, as rescoring requires at least two candidates (Table R4). Offline SR remains stable across all settings (57.1–58.1). Online, reducing K to 2 lowers SR to 37.5, whereas K = 3 and K = 5 both reach 40.2. We therefore use K = 3, which provides enough alternatives without the longer prompt of K = 5.

Table R4 (K)AndroidControl (Offline)AndroidWorld (Online)
257.337.5
3 (Ours)57.140.2
558.140.2

[Transition Loss Weight] Next, we vary λ from 0.3 to 3 (Table R5). Offline SR changes by at most 0.3, while online SR peaks at λ = 1 (40.2) and drops at both ends (36.6 and 37.5). We attribute this to a trade-off between the two objectives: a small λ gives too weak a future signal, whereas a large λ overshadows action selection.

Table R5 (λ)AndroidControl (Offline)AndroidWorld (Online)
0.357.136.6
1 (Ours)57.140.2
356.837.5

[Memory Bank Size] Similarly, we vary the memory bank size from 48 to 768 (Table R6). Offline SR again stays within 0.4, and online SR peaks at 192 (40.2). A small bank provides too few competing futures as negatives. Conversely, a large bank contains more near-duplicate transitions that act as false negatives, since we apply no deduplication (Appendix B.2.2).

Table R6 (Memory Bank Size)AndroidControl (Offline)AndroidWorld (Online)
4857.439.3
192 (Ours)57.140.2
76857.037.5

[Confidence Gate] Finally, our gating follows the same setting as prior pixel-space WMs (Table A2). For Qwen3-VL-8B, our model is invoked in 0.7–4.8% of steps on the offline datasets and 9.2% on AndroidWorld (Table R3(b)). As the reviewer requested, Table R14 further reports the online gating ratio for each backbone. The ratio varies widely, from 9.2% for Qwen3-VL-8B to 70.6% for Qwen3-VL-32B, since it depends on how confident each agent is in its top candidate. For GUI-Owl-7B, the ratio is computed over the steps whose candidates could be parsed. Even for backbones with frequent gating, our method adds only 1.1–1.8× the run time of the base agent (Table 2), since each invocation is a single lightweight pass.

We further vary chic_{\text{hi}} on AndroidWorld (Table R7). Our default (0.95) performs best, while both 0.80 (38.4) and 1.00 (37.5) invoke the model more often (25.9% and 73.6%; a lower threshold lets multiple candidates exceed it, which also opens the gate). Frequent interventions thus tend to override correct confident decisions. Run time changes little across thresholds (220–236 s per task), consistent with the low cost of each invocation.

Table R3 (Qwen3-VL-8B)(a) Coverage (%)(b) Gating Ratio (%)(c) No-Match Ratio (%)
AndroidControl85.74.339.4
AITZ86.04.851.3
GUI-Odyssey88.90.744.2
AndroidWorld94.59.2–
Table R14 (Backbone)Online Gating Ratio (%)
GPT-5.634.1
Gemini-3.6-Flash41.6
GUI-Owl-7B56.2
Qwen3-VL-8B9.2
Qwen3-VL-32B70.6
Table R7 (chic_{\text{hi}})Gating Ratio (%)AndroidWorld SRRun Time (s/task)
0.8025.938.4220.4
0.95 (Ours)9.240.2223.8
1.0073.637.5236.1

Overall, our method is robust to these hyperparameters offline, and our default settings perform best or on par online. We have added these results to Appendix C.

[Q3] Group II vs. Group IV across Settings

We thank the reviewer for this question, which complements our response to Q1. Tables R8 and R9 compare Group II (latent action head only) and Group IV (full model) on the requested offline, OOD, and cross-backbone settings. Group IV consistently outperforms Group II: by +1.7 on AndroidControl, +0.7 on GUI-Odyssey, and +1.8 on AndroidWorld with GUI-Owl-7B. Notably, on AndroidControl and GUI-Odyssey, Group II even falls below the base agent (55.4 vs. 55.6 and 51.4 vs. 52.1; Tables 1 and 3), whereas Group IV recovers or surpasses it. This shows that latent transition supervision is what turns candidate rescoring into a consistent improvement over the base agent.

Table R8 (Qwen3-VL-8B)AndroidControl (ID)GUI-Odyssey (OOD)
Base55.652.1
Group II55.451.4
Group IV57.152.1
Table R9 (GUI-Owl-7B)AndroidWorld (Online)
Base15.2
Group II20.5
Group IV22.3

We have added these comparisons to Appendix C.3 and referred to them in the module ablation of Section 4.3.

Reviewer_nbFFOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a latent world action model for gui-agents. Instead of reconstructing future screens in pixel level and re-perceiving them, it learns shared latent representations with a LoRA-adapted LMM. An action head scores candidate actions, while a transition head predicts action-conditioned future latents. An action-aware contrastive objective uses the current state and a FIFO memory bank as negatives to avoid the identity shortcut. At inference, the transition head is discarded, and candidate actions are re-scored in a single pass, optionally gated by agent confidence. Experiments cover AndroidControl, AITZ, GUI-Odyssey, and AndroidWorld with multiple backbones.

Critical strengths

I think that, (1) the core insight of this paper is strong: gui action selection does not require full future screen reconstruction. latent action-induced transition information can be sufficient. (2) The identity-shortcut analysis is an empirical contribution, and the contrastive objective is a reasonable response. (3) Experiments are broad. (4) The inference design is practical: transition head removed, single-pass rescoring, confidence gate. (5) Figures are well-drawn for content delivery.

Critical weaknesses

(1) Attribution is unclear. Table 4 shows action head alone improves SR from 27.68 to 36.36, while transition head + contrastive add only +3.82 total. The paper should make clearer that the main gain comes from candidate rescoring with an extra trained judge, and isolate the incremental value of world modeling.

(2) Baseline fairness is not specified. The paper does not clearly state whether Code2World/Diffusion2Image selectors use the same scorer model, training data, prompt, and inference budget as the proposed judge.

(3) Candidate coverage and gating statistics are missing. The method only re-ranks K=3 candidates from a frozen agent. Without elaborating coverage rate, gate trigger rate, and no-match ratio, the upper bound and practical benefit are hard to judge.

Questions for authors

(1) Is the same Qwen3-VL-8B + LoRA judge used for all backbones? What extra parameters, training samples, and inference calls does Ours use compared with Base and pixel-space baselines?

(2) Are Code2World/Diffusion2Image selectors trained and sized comparably to the proposed judge?

(3) Please let me know more about the Candidate coverage and gating statistics I mentioned on weakness (2).

(4) I wonder whether the efficiency comparison is against base agents or only against pixel-space world models, and report absolute latency and average judge invocations.

5 comments

[Q1] Clarification on the Action Head

We thank the reviewers (nbFF, CZ5M, sy8p) for highlighting the need to clarify our contributions. The gain of the latent action head in Table 4 (+8.93pp) comes from three sources: (1) candidate rescoring itself, (2) joint training within our VLA pipeline, and (3) the learned latent space (Section 3.1.1). We isolate these sources in Table R1, starting from the action scorer of Code2World. To focus the comparison on action selection, this scorer does not take predicted future screenshots as input.

Table R1 (Group)Action ScorerFTLatentSR
R-I. Base Agent✗✗✗27.68
R-II. Candidate Rescoring✓✗✗32.14
R-III. FT✓✓✗35.71
R-IV. Latent Action Head✓✓✓36.61

[Candidate Rescoring] Rescoring alone improves SR by 4.46pp (R-I → R-II), recovering correct candidates that the base agent mistakenly ranked lower. As the reviewer notes, this accounts for about half of the total gain.

[Fine-Tuning] Unlike the original Code2World, our latent action head is trained within the VLA pipeline. For a fair comparison, we also fine-tune the Code2World scorer under the same supervision (FT). This adds another 3.57pp (R-II → R-III), as the scorer learns to judge the candidates that the agent actually proposes.

[Latent Representation] Finally, replacing the MLLM scorer with our latent action head adds a further 0.89pp (R-III → R-IV). This shows that a lightweight head on the policy's latent space can replace a separate MLLM scorer that re-reads raw screenshots.

[World Modeling] On top of the latent action head, the predictive components add a further 3.57pp (Table 4, Groups II → IV). This gain does not come from generic auxiliary supervision: a control predicting the current state yields little improvement, whereas our contrastive objective targeting action-induced changes does (Table R10).

Table R10 (Qwen3-VL-8B)Transition TargetAndroidWorld SR
Group I–27.68
Group II–36.61
ControlCurrent state ztz_t (point-wise)37.50
Group IIINext state zt+1z_{t+1} (point-wise)38.39
Group IVNext state zt+1z_{t+1} (contrastive)40.18

We have renamed the "action head" to the "latent action head," added the rows of Table R1 to Table 4, and discussed these results in Section 4.3.

[Q2] Clarification on Baseline Fairness

We thank the reviewer for raising this concern and summarizing all settings in Table R2. For each backbone, all methods share the same base agent, candidate prompt, candidate set (K = 3), and confidence gate. For the pixel-space baselines, we use the released checkpoints and inference pipelines, where an unfine-tuned base model selects among the candidates given the predicted screens.

Table R2BaseCode2WorldDiffusion2ImageOurs
Agent and candidatesQwen3-VL-8B, same candidate prompt, K = 3SameSameSame
Checkpoint / model—Code2World-8B (Qwen3-VL-8B)MobileWorld-Diffusion (Qwen-Image-Edit 20B)Qwen3-VL-8B with rank-16 LoRA judge
Candidate selectorBase agentUnfine-tuned base agent with Code2World's reward promptUnfine-tuned base Qwen pickerShared LoRA judge (43.6M trainable)
Training data—>80K AndroidCode screen-action pairs80K GUI transitionsJudge: 14,276 AndroidControl/AITZ transitions; transition head: AndroidControl only
Inference-time modulesBase agent onlyReleased world model and selectorReleased diffusion model and pickerLoRA judge (transition head discarded)
Additional work per gated stepNone3 generations, 3 renders, 3 scoring calls3 diffusion renders, 3 scoring calls1 judge pass
Additional latency per gated step—~10 s~140 s~0.5 s
Confidence gate—0.950.950.95

A single judge, Qwen3-VL-8B with a rank-16 LoRA (43.6M trainable parameters), is shared across all backbones. It is trained once on 14,276 offline AndroidControl and AITZ transitions, far fewer than the 80K+ transitions used to train each baseline world model. For the Qwen3-VL-8B backbone, the judge shares the backbone weights and adds only the LoRA. For other backbones, it runs as an additional 8B model at inference. Per gated step, our method adds a single judge pass (~0.5 s), whereas the pixel-space baselines require multiple generations and scoring calls (~10 s and ~140 s).

Since Table 2 compares the methods as released, Table R1 additionally provides a scorer-matched comparison, in which the Code2World scorer is fine-tuned under our supervision (R-III). Our latent action head still outperforms it. We have added Table R2 to Appendix B.2.1.

[Q3] Candidate Coverage and Gating Statistics

We thank the reviewer for suggesting these meaningful analyses. We report them in Table R3, using the same gating setting as prior GUI WM studies (K = 3; threshold 0.95 online and 1.0 offline; Table A2, Appendix B.2.3).

Table R3 (Qwen3-VL-8B)(a) Coverage (%)(b) Gating Ratio (%)(c) No-Match Ratio (%)
AndroidControl85.74.339.4
AITZ86.04.851.3
GUI-Odyssey88.90.744.2
AndroidWorld94.59.2–

[Candidate Coverage] Table R3(a) reports the coverage, where a step counts as covered only if all three candidates are distinct. The coverage exceeds 85% on all datasets, showing that the candidate pool rarely collapses to a few actions.

[Gating Ratio] Table R3(b) reports how often our model is invoked. As intended by the gating mechanism, it is invoked only when the base agent is not confident: for Qwen3-VL-8B, 0.7–4.8% of steps offline and 9.2% online. The higher online ratio reflects the harder conditions of interactive evaluation. Table R14 further reports the online gating ratio for each backbone. The ratio varies widely, from 9.2% to 70.6%, since it depends on how confident each agent is in its top candidate. For GUI-Owl-7B, the ratio is computed over the steps whose candidates could be parsed. Even for backbones with frequent gating, the overhead remains small, since each invocation is a single lightweight pass (see Q4).

Table R14 (Backbone)Online Gating Ratio (%)
GPT-5.634.1
Gemini-3.6-Flash41.6
GUI-Owl-7B56.2
Qwen3-VL-8B9.2
Qwen3-VL-32B70.6

[No-Match Ratio] Table R3(c) measures how often none of the three candidates matches the GT action. We measure this offline only, since the online setting has no step-wise GT annotation. No-match occurs in 39–51% of steps. As noted in Appendix A, our model cannot recover an action absent from the candidates, so these steps bound its offline gain. However, as discussed in our response to DwCs's Q1, this high ratio largely reflects a limitation of offline evaluation: GUI tasks often have multiple successful routes (6.5–8.3 on average; Tables R11 and R12), but offline evaluation accepts only one GT route. A high no-match ratio therefore does not imply low task success.

Table R11 (Base Agent)Avg. Successful RoutesTasks with ≥2 Routes (%)
GPT-5.66.892.2
Gemini-3.6-Flash6.591.5
GUI-Owl-7B8.0100.0
Qwen3-VL-8B8.3100.0
Qwen3-VL-32B7.997.9
Table R12 (Qwen3-VL-8B)Avg. Successful RoutesTasks with ≥2 Routes (%)
Base8.3100.0
Code2World7.996.9
Diffusion2Image7.994.7
Latent (Ours)8.195.1

We have added these results to a new appendix section (Appendix C.1) and revised the dataset description in Section 4 to distinguish offline and online benchmarks by whether they accept multiple successful routes.

[Q4] Clarification on Efficiency Comparison

[Efficiency Comparison] We apologize for the confusion. Our main efficiency comparison is against pixel-space WMs (Sections 4.1 and 4.2), not against the base agent. This follows the motivation of decoupling the WM from the base agent: as new GUI agents keep emerging [1], decoupled WMs serve as plug-in modules that attach to them. However, existing pixel-space WMs add substantial inference cost on top of the base agent. A natural goal is thus a plug-in WM that provides the benefits of world modeling at nearly no extra cost. In this view, the base agent serves as the efficiency reference that any plug-in WM can approach but not exceed.

[Absolute Latency & Judge Invocations] All run times in Tables 1 and 2 are absolute latencies: end-to-end wall-clock seconds per task online and per step offline (Appendix B.4.4), with each model served on a single RTX A6000 GPU. The judge corresponds to the action scorer of pixel-space WMs, whose counterpart in our method is the lightweight latent action head. Both are invoked only when the confidence gate opens, so the invocation rate equals the gating ratio in Tables R3(b) and R14. Since this ratio depends only on the base agent's confidence, it is identical for pixel-space WMs and ours, ranging from 9.2% to 70.6% of steps across backbones. The latency gap therefore comes from the cost of each invocation: our method adds only 1.1–1.8× the run time of the base agent, compared with 2.1–4.1× for Code2World and 4.5–18.2× for Diffusion2Image (Table 2).

As indicated by the gray shading in Tables 1 and 2, the base agent serves as a reference rather than a competitor. We have labeled the efficiency columns in both tables as absolute latency and stated in Sections 4.1 and 4.2 that the efficiency comparison is among plug-in WMs. We have also added a "Plug-in World Models" paragraph to Appendix B.2.1.

[1] OpenAI. GPT-6 Astra. September 2026.

Dear Reviewer nbFF,

We hope this message finds you well. As the discussion period ends in two days, we wanted to kindly check whether our responses have addressed your concerns. If there are any remaining questions or suggestions, we would be happy to address them. Otherwise, we would greatly appreciate it if you could consider updating your assessment.

Thank you again for your time and constructive feedback.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has substantially improved our paper. For the reviewers' and AC's convenience, we summarize the reviews and the additional experiments below. All revisions are highlighted in red in the updated PDF.

ReviewerRatingConfidenceResponded
nbFF3 → 32 → 3 ↑✓
CZ5M3 → 43 → 3✓
sy8p3 → 33 → 3Pending
DwCs2 → 3 ↑2 → 2✓

Summary of Reviews and Our Responses

We are encouraged that the reviewers found our work well-motivated (nbFF, CZ5M, sy8p, DwCs), recognized the substantial inference efficiency over pixel-space world models (CZ5M, sy8p, DwCs), and found the identity-shortcut analysis and contrastive objective meaningful (nbFF, CZ5M), supported by broad evaluation (nbFF, CZ5M, sy8p). We summarize the main concerns and our responses below.

ConcernReviewersOur Response
Attribution of the gainnbFF, CZ5M, sy8pDecomposed the gain and isolated the effect of future prediction (R1, R8–R10)
Baseline fairnessnbFFSummarized all configurations under a shared setup (R2)
Gating statistics and sensitivitynbFF, CZ5MReported gating statistics and hyperparameter ablations (R3–R7, R14)
"World model" terminologysy8pClarified inference behavior and revised the wording
Offline–online gap and costDwCsAnalyzed route diversity, matched thresholds, and repeated runs (R7, R11–R13, R15)
Presentation and identity shortcutDwCsMade figures self-contained and unified the definition
Number of evaluated tasksDwCsClarified the 112-task evaluation

Summary of Additional Experiments

TableQuestion AddressedKey FindingReviewersRevision
R1Where does the action-head gain come from?Rescoring explains about half; joint training and the latent space add the restnbFF, CZ5M, sy8pTable 4, Sec. 4.3
R2Are the baselines compared fairly?Same agent, candidates, and gate; ours uses the least training datanbFFApp. B.2.1
R3How often is the model invoked, and are candidates diverse?Coverage >85%; invoked in 9.2% of steps for Qwen3-VL-8BnbFF, CZ5MApp. C.1
R4–R7How sensitive is performance to hyperparameters?Robust offline; default settings best or on par onlineCZ5M, DwCsApp. C.2
R8–R9Does transition supervision help beyond one setting?Full model outperforms the action-head-only baseline in all settingsCZ5M, sy8pApp. C.3
R10Is the gain from future prediction or extra supervision?Current-state prediction gives little gain; contrastive future prediction doessy8pApp. C.4
R11–R12Why are online gains larger than offline?Tasks admit 6.5–8.3 successful routes, which offline evaluation does not creditnbFF, DwCsApp. C.6, Sec. 4.2
R13Are the gains robust across runs?41.43 ± 1.02 SR over five seedsDwCsApp. C.5
R14How often does the gate open across backbones?9.2–70.6% of steps; overhead stays small since each invocation is a single passnbFF, CZ5MApp. C.1
R15Are the alternative actions chosen by our model valid?Hit rate rises from 38.8% (single route) to 59.6% (any successful route)DwCsApp. C.6

We thank the reviewers again for their valuable feedback and would be happy to provide any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.