acceptodds
Under review as a conference paper at ICLR 2027

Beyond Next-Screen Prediction: The Decision Value of GUI World Models

Abstract

GUI world models are increasingly used to improve action selection by predicting the consequences of candidate actions, but it is unclear how much of the resulting gain comes from prediction itself rather than from the evaluator that interprets those predictions. We study this attribution problem through controlled interventions on GUI action selection. With candidates and predictions held fixed, lookahead gains can change with how the prediction is presented to the evaluator, so end-to-end gains cannot be attributed to prediction alone. We then introduce Direct Action Ranking (DAR), which learns from reference actions to score candidates without generating future screens, and compare it with a prediction-augmented variant under matched supervision. Predicted successors add no detectable ranking signal once evaluation is learned directly. Without retraining, DAR improves AndroidWorld task success by 5.8–16.4 points across three proposers, outperforms world-model lookahead, and transfers zero-shot to unseen apps, where every prediction-conditioned configuration falls below proposer confidence. These results suggest that GUI world models should be evaluated by their decision value, the incremental action-selection value of their predictions beyond a strong direct evaluator, rather than by prediction fidelity or end-to-end lookahead gain alone.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
GUI Agent, GUI world model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3qPqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the decision value of GUI world models by examining whether predicted future observations provide additional benefits for action selection beyond direct action evaluation. The authors first analyze how different evaluators affect lookahead performance when predictions are fixed. They then propose Direct Action Ranking (DAR), which learns to rank candidate actions directly from reference-action supervision without generating future screens. Experiments on AndroidControl, AndroidWorld, MWMBench-KApps, and MiniWoB show that DAR improves action selection and often outperforms world-model-based lookahead. The results suggest that the contribution of world models should be evaluated separately from that of action evaluators.

Critical strengths
  1. The paper investigates an interesting question about the actual contribution of world models to GUI agent decision-making.
  2. The experimental design includes several controlled comparisons, particularly the fixed-prediction evaluator analysis and the comparison between DAR and prediction-augmented DAR under matched supervision. T
  3. The evaluation covers multiple benchmarks, candidate proposers, and world models. The experiments on AndroidWorld and unseen Korean applications provide useful evidence of the effectiveness and transferability of the proposed approach.
Critical weaknesses
  1. DAR essentially formulates action selection as a supervised ranking problem using a pretrained vision-language model, LoRA adaptation, and a combination of listwise and pointwise losses. These are well-established techniques, and the proposed architecture and training objective do not appear to introduce substantial methodological innovations.
  2. The matched comparison between DAR and prediction-augmented DAR in Section 5.3 is primarily based on offline reference-match AUROC. However, the authors also show in Section 6.4 that improvements in AUROC do not necessarily translate into better online task success. The online experiments compare trained DAR against prompted lookahead evaluators, which differ in both their inputs and training procedures.
  3. The comparison with existing action evaluation methods is insufficient. Most experiments focus on proposer confidence and world-model-based lookahead, while DAR is a separately trained action evaluator. Although the paper discusses learned evaluators in the related work, it does not provide direct experimental comparisons with alternative supervised action ranking, reward modeling, or preference-based evaluation methods.
  4. The reliance on reference-action matching introduces limitations in both training and evaluation. DAR is trained using labels derived from recorded reference actions, and reference-match AUROC is a central metric in the analysis. However, GUI tasks may have multiple valid action sequences, and actions that differ from the recorded reference may still lead to successful task completion. Although the authors use domain-specific matching rules, this does not fully address the ambiguity of reference-based supervision.
  5. The investigation of when world models are useful remains limited. The paper mainly studies one-step prediction with a relatively small candidate set (typically K ≤ 3). Although Appendix D includes experiments with two- and three-step lookahead, these experiments are limited to prompted evaluation and do not provide a matched analysis of decision value across different planning horizons.
Questions for authors
  1. Have the authors considered evaluating prediction-augmented DAR in the online AndroidWorld environment under the same training and candidate-generation settings as DAR?
  2. Could the authors provide more analysis of the tasks or action types where world-model lookahead outperforms DAR?
Reviewer_M3HEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that gains from GUI world-model lookahead cannot be attributed to prediction alone, because the pipeline couples a world model with an evaluator, and proposes decision value: DV_M(W_phi | E_theta) = M(E_psi; W_phi) - M(E_theta), where E_psi is trained identically to the direct evaluator E_theta and differs only in receiving the predicted successor. Section 4 holds candidates and cached Code2World-8B predictions fixed on 3,000 AndroidControl steps and varies only the prompted evaluator: Reference-Match Accuracy changes by -1.4 (self), +1.7 (Claude Sonnet 4.6), and +2.5 (Claude Opus 4.8) points over Policy Top-1, and re-encoding the same successors as an accessibility tree or plain text lowers the Opus estimate from +2.5 to +1.2. Override-Benefit AUROC shows that the signals identifying useful overrides all require the reference successor (SigLIP contrast 0.725/0.746, DOM contrast 0.711/0.671), while pre-execution signals are near chance (likelihood spread 0.503/0.502, policy confidence 0.428/0.430). The paper then introduces DAR, a LoRA-adapted Qwen3-VL-8B scorer with a combined listwise and pointwise objective over reference-action labels, which raises pooled reference-match AUROC from 0.775 (policy confidence) to 0.957 on AndroidControl and from 0.870/0.874/0.867 to 0.949/0.939/0.943 on the Mind2Web splits, and reference-match accuracy from 43.8% to 50.1%. Under matched training, adding predicted successors changes AUROC by -0.0003 and +0.0005 (95% intervals within +-0.004). Online, on 116 AndroidWorld tasks with three seeds and no retraining, DAR improves task success over Policy Top-1 by 16.4, 11.5, and 5.8 points for three proposers and over both lookahead configurations by 4.3-11.5 points (nine paired contrasts, all p < 0.05 uncorrected). On 14 unseen Korean apps DAR's AUROC exceeds policy confidence for all three proposers while every prediction-conditioned configuration falls below it; on MiniWoB the ordering depends on the proposer.

Critical strengths
  1. The separation of prediction from evaluation is the cleanest part of the paper, and it changes what a lookahead result can be taken to mean. Holding candidates and predictions fixed and varying only the prompted evaluator yields -1.4, +1.7, and +2.5 points for self, Sonnet 4.6, and Opus 4.8, and merely re-encoding the same predicted successor as an accessibility tree or plain text moves the Opus estimate from +2.5 to +1.2. The matched comparison is equally clean: prediction-augmented DAR shares backbone, candidates, labels, objective, and optimisation with DAR and differs only by the predicted successor, changing AUROC by -0.0003 and +0.0005 with 95% intervals inside +-0.004, against +0.17-0.18 from direct supervision. The authors also state the scope of the null explicitly — that it bounds offline ranking signal and does not by itself rule out online decision value — which is the right way to report a negative result.
  2. DAR is validated online without retraining across proposers, and the paper reports the cases where it loses. A single mobile DAR trained on Qwen2.5-VL-7B proposals is applied to Llama-4 Maverick, Qwen3-VL-8B, and Qwen3-VL-235B on 116 AndroidWorld tasks over three seeds, giving the highest mean success for every proposer with nine paired contrasts at p < 0.05. Swapping the world model (Code2World-8B to gWorld-8B) moves lookahead by -0.5 to +4.3 points depending on the proposer while DAR, which uses no world model, is unchanged — exactly the fixed reference point the paper argues for. The authors additionally report that lookahead beats DAR on MiniWoB with the weaker proposer (38.9% versus 32.0%), that DAR's gain shrinks as proposer quality rises (16.4 to 5.8 points), and that proposer SFT and candidate evaluation are complementary rather than competing.
Critical weaknesses
  1. The null is measured through one particular way of giving predictions to the evaluator, and the paper's own Section 4 shows this choice to be first-order. Section 5.3 injects the predicted successor as an additional input to an otherwise identical DAR, but does not test contrastive presentation, multi-step rollouts, explicit uncertainty about the prediction, or value estimates from world-model rollouts. Table 1 shows that the strongest override signals are precisely contrastive — the difference in agreement between the selected candidate and Top-1 (0.725/0.746) — rather than per-candidate scalars (0.608/0.590), and no contrastive training objective is evaluated. Since the gain from identical predictions already ranges from +2.5 to +1.2 with representation alone, concluding from two or three injection formats that "predicted successors add no detectable ranking signal once evaluation is learned" overstates what was tested; the conclusion section partially retreats to longer horizons and consequence-aware objectives, but the Section 5 takeaway and the abstract do not. This matters because the framing invites readers to generalise the null to GUI world models as a class.
  2. Decision value is measured with a metric the paper itself shows to be insensitive to what matters online, and the online comparison is missing. Section 6.4 reports that removing the pointwise loss lowers AUROC only from 0.957 to 0.934 but reduces conditional match accuracy from 0.911 to 0.890 and online task success by 6.0 points, and that MiMo has lower AUROC than GLM (0.879 versus 0.884) yet higher online success (30.5 versus 28.7). The paper explains this correctly — pooled AUROC ranks candidates within a set while execution depends on the winner — but Section 5.3 nevertheless uses pooled AUROC as M in the definition of decision value. A difference that is zero in AUROC need not be zero in task success, and no online comparison of DAR versus prediction-augmented DAR is reported. Relatedly, the AndroidWorld evidence is 116 tasks with three seeds, the nine paired tests are explicitly uncorrected, and the smallest contrast (+4.3 over External with Qwen3-VL-235B, p = 0.049) would not survive correction for nine comparisons; and since DAR is trained on Qwen2.5-VL-7B candidates but evaluated on Maverick and Qwen3-VL candidates, the gains conflate cross-proposer generalisation with correction of proposer confidence.
Questions for authors

Clarifications (answers could change my assessment):

  1. Section 5.3 reports only AUROC. Please report the online AndroidWorld task success of DAR versus prediction-augmented DAR under the same paired protocol and three seeds. Given Section 6.4 (near-equal AUROC, 6-point online gap), an AUROC-based null cannot settle whether the predictions have decision value online.
  2. Predictions are injected only as an extra input. Please add a contrastive variant, in which the evaluator compares candidates' predicted successors rather than scoring each in isolation — Table 1 identifies contrast as the informative form (0.725/0.746 versus 0.608/0.590) — and report whether the matched AUROC difference stays within +-0.004.
  3. Table 14 states that the nine McNemar tests are uncorrected and the smallest is p = 0.049. Please report Benjamini-Hochberg or Holm corrected values; does DAR still beat External for the Qwen3-VL-235B proposer?
  4. DAR is trained on Qwen2.5-VL-7B proposals and applied to three other proposers. Please report DAR trained and evaluated on the same proposer's candidates, to separate cross-proposer transfer from correction of proposer confidence (Table 2 shows 0.957 versus 0.946 offline; what is the online analogue?).

Suggestions for future work (not affecting my assessment): Decision value for multi-step lookahead and for consequence-aware objectives — Appendix D already shows depth 2-3 gains of +0.018/+0.019 versus +0.025 at one step, but not under matched evaluators; using world models for training-time rollouts or data augmentation, which the paper explicitly excludes; learning when to invoke lookahead at all, given that Table 1 finds no usable pre-execution signal today.

Reviewer_LbCTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how much of the benefit of world-model lookahead for GUI agents comes from the predicted next screen itself rather than from the evaluator that reads it, and defines a world model's decision value as the gain of a prediction-conditioned evaluator over an identically trained direct evaluator on the same candidates. With candidates and Code2World predictions held fixed on AndroidControl, lookahead gains over the proposer's top choice are shown to vary with the evaluator and with how the predictions are presented. The authors then introduce Direct Action Ranking (DAR), a LoRA-adapted Qwen3-VL-8B scorer trained with listwise and pointwise losses on reference-action matches, and find that adding predicted successors to DAR under matched training leaves reference-match AUROC unchanged within tight intervals. Applied without retraining, DAR improves AndroidWorld task success over the top choice and over prompted lookahead with two world models for three proposers, ranks candidates best on unseen Korean-language apps, and gives mixed results on MiniWoB.

Critical strengths
  • A clean offline control experiment. DAR is compared with a version that also receives the predicted next screen, under matched training. The AUROC difference is essentially zero, with tight intervals.

  • A practical selector with consistent gains on AndroidWorld. Without retraining, DAR beats the proposer's top choice and prompted lookahead on AndroidWorld for all three proposers and both world models. Even though this comparison does not isolate the role of prediction, it sets a strong and cheap baseline that any lookahead method should be compared against.

Critical weaknesses
  • The central decision-value result is offline only. It rests on one pooled AUROC comparison at one training budget, and the paper's own Section 6.4 shows that AUROC and task success can diverge. The prediction-augmented model is never run online, and the paper does not say which world model produced its predictions or how the predicted screen is fed in.

  • The online comparison is not a controlled test of prediction. DAR is trained, while the lookahead it beats is prompted zero-shot, so training and prediction change at the same time. Several baseline details are missing: which external evaluator is used, the rule for falling back to the top choice, and the world model on MiniWoB.

Questions for authors
  1. Could you run prediction-augmented DAR on AndroidWorld under the same protocol as DAR, and report its selection accuracy offline? Both models already exist, and this is the most direct test of the paper's main claim.

  2. Which world model produced the predictions for prediction-augmented DAR, and how is the predicted screen fed to the model? Does the null result still hold when both models are trained on less data, where the direct evaluator is weaker?

  3. Which model is the external evaluator, what rule triggers the fallback to the top choice, and how often does lookahead actually change the action online? Which world model is used on MiniWoB, and why do both lookahead variants score zero on date tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.