- Rating
- 2
- Confidence
- 2(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper compares two approaches to goal-free GUI step verification: a predictive verifier that forecasts the next-screen representation and compares it with the observed screen, and a discriminative verifier that jointly processes the previous screen, action, and observed successor. Under matched training conditions, the discriminative verifier outperforms the predictive approach by 6.53 AUC points, with a similar gap on an independent test set. Through controlled ablations, the authors show that the main bottleneck is late fusion of separately encoded screens, rather than predictive alignment losses. Further analyses examine label quality, cross-dataset generalization, and real-world interface anomalies.
Critical strengths
S1: The paper investigates a meaningful problem in GUI step verification. The authors' focus on action-effect compatibility is well motivated and comparing predictive and discriminative verifiers provides useful insight into the choice of verification architectures for GUI agents.
S2: This paper presents a thorough and well-controlled empirical study, which is supported by consistent results across multiple seeds, an independent test set, and label audits. And the controlled ablations further identify the architectural factors contributing to the performance gap.
Critical weaknesses
W1: The paper only focuses on JEPA-style latent prediction with Qwen3-VL backbones, without evaluating other predictive approaches, such as image- or language-based world models. While the findings clearly demonstrate the limitations of the tested predict-then-compare design, it remains unclear whether the conclusions generalize to other world-model families.
W2: The evaluation mainly uses synthetic negatives, which may not reliably reflect real-world execution failures. Although the authors audit these labels and confirm the model ranking, there is still a lack of real-world evaluation in practical performance. The real-anomaly analysis also remains offline, with no evaluation in an actual setting. So it remains unclear whether the reported 6.53-point AUC gain yields practical benefits for real-world end-to-end tasks.
Questions for authors
Q1: The advantage of early fusion is much larger on action-mismatch negatives than on no-op negatives (Section 5 and Appendix I). Could the authors discuss possible explanations for this difference, ideally with a few representative examples from the evaluation set? This would help clarify what the ablation reveals about GUI verification beyond the general advantage of early fusion.
Q2: The main AUC results use a normalized predictive mixture with statistics re-estimated for each corpus, while fixed-threshold transfer uses a single-component score (Section 6 and Appendix C). Please make the scoring protocols more explicit in the main text, including which score is used in each experiment, so I can better distinguish ranking performance from threshold transfer.
See questions in weaknesses also.
Urgent Reminder: Discussion Period Ends November 18
Dear Reviewers,
The discussion period will end on November 18, 2026. Please carefully review the authors' responses and actively engage in discussions.
If you have not yet initiated a discussion, please do so immediately. Reviewers are expected to address the authors' rebuttals, provide constructive follow-up comments, and update their evaluations when appropriate.
Please also ensure that your reviews reflect independent human judgment and that any AI assistance is properly documented in accordance with the review guidelines.
Active participation is an essential reviewer responsibility and is critical to ensuring a rigorous and fair review process.
Thank you for your cooperation.