- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the decision value of GUI world models by examining whether predicted future observations provide additional benefits for action selection beyond direct action evaluation. The authors first analyze how different evaluators affect lookahead performance when predictions are fixed. They then propose Direct Action Ranking (DAR), which learns to rank candidate actions directly from reference-action supervision without generating future screens. Experiments on AndroidControl, AndroidWorld, MWMBench-KApps, and MiniWoB show that DAR improves action selection and often outperforms world-model-based lookahead. The results suggest that the contribution of world models should be evaluated separately from that of action evaluators.
Critical strengths
- The paper investigates an interesting question about the actual contribution of world models to GUI agent decision-making.
- The experimental design includes several controlled comparisons, particularly the fixed-prediction evaluator analysis and the comparison between DAR and prediction-augmented DAR under matched supervision. T
- The evaluation covers multiple benchmarks, candidate proposers, and world models. The experiments on AndroidWorld and unseen Korean applications provide useful evidence of the effectiveness and transferability of the proposed approach.
Critical weaknesses
- DAR essentially formulates action selection as a supervised ranking problem using a pretrained vision-language model, LoRA adaptation, and a combination of listwise and pointwise losses. These are well-established techniques, and the proposed architecture and training objective do not appear to introduce substantial methodological innovations.
- The matched comparison between DAR and prediction-augmented DAR in Section 5.3 is primarily based on offline reference-match AUROC. However, the authors also show in Section 6.4 that improvements in AUROC do not necessarily translate into better online task success. The online experiments compare trained DAR against prompted lookahead evaluators, which differ in both their inputs and training procedures.
- The comparison with existing action evaluation methods is insufficient. Most experiments focus on proposer confidence and world-model-based lookahead, while DAR is a separately trained action evaluator. Although the paper discusses learned evaluators in the related work, it does not provide direct experimental comparisons with alternative supervised action ranking, reward modeling, or preference-based evaluation methods.
- The reliance on reference-action matching introduces limitations in both training and evaluation. DAR is trained using labels derived from recorded reference actions, and reference-match AUROC is a central metric in the analysis. However, GUI tasks may have multiple valid action sequences, and actions that differ from the recorded reference may still lead to successful task completion. Although the authors use domain-specific matching rules, this does not fully address the ambiguity of reference-based supervision.
- The investigation of when world models are useful remains limited. The paper mainly studies one-step prediction with a relatively small candidate set (typically K ≤ 3). Although Appendix D includes experiments with two- and three-step lookahead, these experiments are limited to prompted evaluation and do not provide a matched analysis of decision value across different planning horizons.
Questions for authors
- Have the authors considered evaluating prediction-augmented DAR in the online AndroidWorld environment under the same training and candidate-generation settings as DAR?
- Could the authors provide more analysis of the tasks or action types where world-model lookahead outperforms DAR?