acceptodds
Withdrawn from ICLR 2027

Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A login screen that appears after a tap on *Sign in* is expected; the same screen after a tap on *View order* is an attack. For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one. Judging a transition requires an expectation of what should have followed the action. Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two. We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions. Verification reduces to a vector comparison, and the same signal reveals whether a mismatch is harmful. We evaluate on LGWM, a diagnostic where each credential screen appears under both a legitimate and a hijacked transition, so detectors that see only the screen are at chance by construction. The training-free score reaches 0.987 AUC at 17 ms per decision, on par with the strongest closed-source VLMs and about ten AUC points above generative GUI world models at over three orders of magnitude lower latency. The residual direction reaches 0.953 AUC at separating harmful from benign violations, where prompted VLMs are near chance. Further analyses show that the prediction is a usable future state rather than an anomaly score. World models have mostly served as simulators or planners; our results point to a third role, verification, for which predicting in representation space is the natural design.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
GUI, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GFzhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether GUI transitions can be checked efficiently by comparing a predicted future representation with the representation of the observed next screen. The proposed LGWM is an action-conditioned latent world model trained on approximately 1.85 million GUI transitions. It uses a DINOv2 encoder, structured action embeddings, and a Transformer predictor, with change-weighted prediction and auxiliary regularization. At inference, a cosine discrepancy provides an integrity score without attack-label supervision. A separately trained linear head uses the residual and action embedding to distinguish harmful from benign mismatches. The authors introduce RSWT-Bench, where each donor credential screen occurs as both a legitimate and a substituted outcome, making outcome-only detection chance-level on the primary metric. LGWM reports 0.987 RSWT AUC at 17.1 ms per decision on an A100. The supervised residual head achieves 0.953 Harm AUC. Additional experiments examine retrieval, action sensitivity, recursive prediction, and architectural ablations.

Critical strengths
  1. The decoder-free verification path is simple and potentially useful. Predicting and observing in a shared representation space avoids generating an entire screenshot or HTML page just to decide whether a transition looks consistent. The reported combination of 0.987 RSWT AUC and 17.1 ms latency is compelling for a per-step monitor. The fact that this integrity score requires no benchmark attack labels strengthens the result.

  2. The benchmark removes outcome-only shortcuts on its primary task, and the paper provides meaningful dynamics controls. Reusing identical donor screens under opposite labels makes the RSWT test more informative than a benchmark where attacks simply look suspicious. The action-counterfactual and nearest-training-successor comparisons further support the claim that LGWM learns transition-dependent predictions. Together, these provide evidence beyond a single detection score.

Critical weaknesses
  1. The harm experiment does not establish the claimed role of residual direction. Table 7 reports 0.975 Harm AUC for the outcome-only probe and 0.974 for the two-frame classifier, compared with 0.953 for the residual head. This is an important result: the primary RSWT task defeats outcome appearance, but the Harm task can apparently be solved very well from appearance alone. Since harmful replacements use credential screens and benign replacements use harmless donors, recognizing the screen category is a plausible explanation. The comparison with VLMs does not resolve this. Appendix F.1 explicitly says they were prompted about transition consistency, not harmfulness. Both classes in Harm are inconsistent transitions, so a consistency score being near chance does not demonstrate weak harm recognition. I appreciate that the appendix acknowledges this, but the abstract and main discussion still present the comparison as evidence for the residual’s interpretation capability. Also, Equation 5 uses an unnormalized residual concatenated with the action embedding. Its success does not isolate direction from residual scale, action information, or observed-screen features. This weakens one of the paper’s central contributions, even if the primary integrity result remains valid.

  2. The controlled diagnostic supports a narrower conclusion than the security-verifier framing, and the supporting ablations need reconciliation. Donor pairing proves that the outcome alone is insufficient on RSWT. It does not prove that successful detection requires an action-conditioned world model. The current-screen probe already achieves 0.796 RSWT AUC, indicating substantial label information in the preceding context. The 0.833 two-frame baseline is a useful control, but additional matched-context and action-ablation results would help establish what produces LGWM’s advantage. Appendix H appropriately limits the evidence to offline screen replacements; even the hard tier does not establish same-application execution. Detecting these replacements is useful, but it leaves uncertainty about legitimate authentication redirects and attacks that preserve the expected visual transition. The design evidence is also inconsistent. Table 2 reports inverse-dynamics copy scores of 0.244 versus 0.112, whereas Table 15 and Section 4.4 report 0.286 versus 0.193. In Table 15, removing inverse dynamics actually improves Overall AUC from 0.878 to 0.900. The coordinate-MLP alternative also slightly improves it. These results do not invalidate LGWM, but they do not clearly support the claimed necessity of those design choices.

Questions for authors
  1. How do you interpret the outcome-only probe’s 0.975 Harm AUC relative to the residual head’s 0.953? Could you report task-matched harm prompts for the VLMs, together with observed-only, action-only, and normalized-residual probes under matched supervision? Evidence that the residual adds information beyond screen category would strengthen this contribution.
  2. What happens to RSWT performance when the recorded action is removed or shuffled while keeping the two screens fixed? Can you also report the comparison against the paired safe transition sharing the same preceding screen and action? These controls would help distinguish learned action-conditioned expectation from context or cross-screen compatibility cues. 3.Which inverse-dynamics copy scores are correct: Table 2’s 0.244/0.112 or Table 15’s 0.286/0.193? Please also explain why removing inverse dynamics improves Overall AUC, and provide the corresponding RSWT results. This could change my assessment of how well the design claims are supported.
  3. The reproducibility statement points to Appendix F for exact prompts, the generative-model judge protocol, and latency measurement, but I could not find those details there. Could you provide them, including score extraction and the precise RSWT estimator—matched-pair concordance or conventional all-positive/all-negative ROC AUC? I would also like the concrete pretraining/benchmark leakage checks referenced in Appendix B.
Reviewer_hbW6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LGWM, an action-conditioned world model that predicts the next GUI screen in representation space and compares that prediction with the observed outcome. A cosine-based score detects unexpected transitions without attack labels, while a supervised residual classifier distinguishes harmful from benign mismatches. The authors also introduce RSWT-Bench, where identical outcome screens appear in both legitimate and hijacked transitions. LGWM achieves a reported 0.987 RSWT AUC at 17.1 ms per decision, offering a promising approach to efficient GUI transition monitoring.

Critical strengths
  1. The paper gives a clear motivation for evaluating transitions rather than isolated screenshots.
  2. LGWM achieves a strong accuracy–efficiency tradeoff: a reported 0.987 RSWT AUC at 17.1 ms per decision.
Critical weaknesses
  1. The harm experiment does not show that the predicted future helps identify harmful transitions. An outcome-only classifier achieves 0.975 Harm AUC, outperforming the residual head’s 0.953. Since harmful examples use credential screens and benign examples use harmless screens, the task may largely reduce to recognizing screen content.
  2. The evaluation does not yet establish practical security benefits. It uses offline screen replacement rather than realistic attacks within verified application workflows. The residual head’s 16.3% macro false-positive rate at 90% hijack recall also suggests that normal use could trigger frequent interruptions.
Questions for authors
  1. How do VLMs compare when explicitly prompted to assess harm?
  2. Can you evaluate realistic within-app redirects alongside legitimate reauthentication, and report detection performance at low false-positive rates?
Reviewer_bBEoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LGWM, an action-conditioned latent world model for detecting unexpected GUI transitions. It compares predicted and observed next-screen representations and introduces a supervised residual classifier for harm detection. Evaluation uses RSWT-BENCH, a controlled benchmark with pixel-identical outcomes under different transition labels.

Critical strengths

The transition-based verification formulation is well motivated. The donor-paired benchmark usefully prevents outcome appearance alone from solving the primary task. The latent interface enables inexpensive verification: approximately 17ms per decision. The paper provides extensive offline controls, baseline comparisons, and action-conditioning diagnostics.

Critical weaknesses

The methodological contribution appears incremental. The approach builds on established action-conditioned latent prediction and prediction-error detection. Its main contribution appears to be their adaptation to GUI verification. A sharper account of the technical advance and the necessity of the proposed design choices is needed.

Generalization is insufficiently established. Despite multi-source pretraining, security evaluation is limited to two Android-centric sources and controlled screen replacement, with 222 credential-screen donor pairs driving the primary metric. App-disjoint and web/desktop generalization are not demonstrated. Held-out episodes do not establish robustness to unfamiliar applications, layouts, or legitimate workflow variations.

Benefits to actual GUI agents remain unverified. The paper evaluates offline transition discrimination without reporting closed-loop attack prevention, normal-task completion, or recovery outcomes. High detection AUC alone does not establish that post-action monitoring improves an agent’s overall safety and usefulness.

The system-level cost–performance trade-off is unresolved. The reported inference cost is encouraging, but excludes the downstream consequences of alerts, such as VLM escalation, interruption, and recovery. Appendix F reports mFPR@90R=0.163 for the residual classifier on the diagnostic benchmark, motivating an evaluation of these consequences. No end-to-end operating curve establishes whether the complete monitoring system offers a favorable trade-off.

Questions for authors

Are evaluation applications disjoint from pretraining applications? Please distinguish episode-level separation from application-level generalization.

Are any closed-loop results available concerning attack prevention, normal-task success, or recovery?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.