acceptodds
Under review as a conference paper at ICLR 2027

Where Does A Predictive GUI Verifier Lose To A Discriminative One? A Controlled Diagnosis

Abstract

Step-level verification asks whether an observed GUI screen is compatible with the action claimed to have produced it, the goal-free check that retry and recovery rely on. A predictive verifier forecasts the successor with a latent world model and compares the forecast with the screen; a discriminative verifier reads the action and both screens jointly. We compare the two at matched data, negatives, backbone and training budget on a pre-registered benchmark of recorded transitions. The predictive verifier is worse by 6.53 AUC points, by 6.30 on a test set built and scored once after every model was frozen, and by a wider margin at low false-positive rates. The gap is not a labelling artefact: two independent annotators find a minority of constructed negatives still compatible, and on the negatives both confirm the gap is essentially unchanged. A ladder of controlled variants between the two verifiers locates it. The four losses that align the forecast with the observed screen move aggregate AUC by only 0.19 points. Comparing the screens after encoding them apart costs 4.93; in-distribution, neither keeping every patch nor matching the comparison head recovers more than a point of it, which leaves fusion timing. Gains that narrow the gap do not survive a change of distribution, and both verifiers refuse genuine interface anomalies more often than ordinary transitions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI agents, step-level verification, world models, JEPA, predict-then-compare, discriminative verifier, early fusion, benchmark label audit

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WKgmOpenReview ↗
Rating
2
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper compares two approaches to goal-free GUI step verification: a predictive verifier that forecasts the next-screen representation and compares it with the observed screen, and a discriminative verifier that jointly processes the previous screen, action, and observed successor. Under matched training conditions, the discriminative verifier outperforms the predictive approach by 6.53 AUC points, with a similar gap on an independent test set. Through controlled ablations, the authors show that the main bottleneck is late fusion of separately encoded screens, rather than predictive alignment losses. Further analyses examine label quality, cross-dataset generalization, and real-world interface anomalies.

Critical strengths

S1: The paper investigates a meaningful problem in GUI step verification. The authors' focus on action-effect compatibility is well motivated and comparing predictive and discriminative verifiers provides useful insight into the choice of verification architectures for GUI agents.

S2: This paper presents a thorough and well-controlled empirical study, which is supported by consistent results across multiple seeds, an independent test set, and label audits. And the controlled ablations further identify the architectural factors contributing to the performance gap.

Critical weaknesses

W1: The paper only focuses on JEPA-style latent prediction with Qwen3-VL backbones, without evaluating other predictive approaches, such as image- or language-based world models. While the findings clearly demonstrate the limitations of the tested predict-then-compare design, it remains unclear whether the conclusions generalize to other world-model families.

W2: The evaluation mainly uses synthetic negatives, which may not reliably reflect real-world execution failures. Although the authors audit these labels and confirm the model ranking, there is still a lack of real-world evaluation in practical performance. The real-anomaly analysis also remains offline, with no evaluation in an actual setting. So it remains unclear whether the reported 6.53-point AUC gain yields practical benefits for real-world end-to-end tasks.

Questions for authors

Q1: The advantage of early fusion is much larger on action-mismatch negatives than on no-op negatives (Section 5 and Appendix I). Could the authors discuss possible explanations for this difference, ideally with a few representative examples from the evaluation set? This would help clarify what the ablation reveals about GUI verification beyond the general advantage of early fusion.

Q2: The main AUC results use a normalized predictive mixture with statistics re-estimated for each corpus, while fixed-threshold transfer uses a single-component score (Section 6 and Appendix C). Please make the scoring protocols more explicit in the main text, including which score is used in each experiment, so I can better distinguish ranking performance from threshold transfer.

See questions in weaknesses also.

Reviewer_7HdoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper compares predictive and discriminative verifiers for GUI step-level verification and the authors show that discriminative verifiers outperform predictive ones by experiments, primarily due to joint encoding rather than late comparison. They further analyze the performance gap under distribution shifts and real interface anomalies.

Critical strengths
  1. The paper investigates the performance differences between predictive and discriminative approaches for GUI step-level verification.
  2. The controlled ablations and additional evaluations provide some insights into the factors contributing to the observed performance gap.
Critical weaknesses
  1. The finding that joint encoding outperforms separate encoding is not particularly surprising, as similar observations have been established in prior work.
  2. The experiments primarily rely on one backbone family and one predictive modeling approach, making it unclear whether the conclusions extend to other architectures.
  3. The paper does not demonstrate whether the observed improvements in verification AUC translate into better agent performance, recovery, or task success.
  4. The paper is overly dense and difficult to follow. The presentation of methods and experimental results(e.g., table 3) lacks clarity, making the main findings unnecessarily hard to understand.
Questions for authors
  1. Given that joint encoding has been shown to outperform separate encoding in prior work, the authors should explicitly articulate the key novel insights of this study beyond merely confirming this observation in the context of GUI verification.
  2. The manuscript would benefit from a deeper discussion regarding how representative the synthetic negative samples are of real GUI agent failures, along with an additional evaluation of the proposed approaches on more realistic failure cases.
Reviewer_LKyuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This main contribution from this paper is a comparison of two methods for verifying GUI transitions, which is a useful study of what a computer use agent might do after taking an action or clicking a link. They compare a latent predictive verifier vs. a joint discriminative verifier, where the predictive verifier tries to predict the next screen’s patch features compared against the actual next GUI transition, while the discriminative verifier gets to compare both screens jointly. They match the training sources, sample budget + backbone family, and find that the discriminative model is 6.53 AUC points better with a similar gap on a post-freeze test. They do ablation with intermediate architectures to test this and find much of the difference comes from the separate screen encodings (over joint encoding). For reproducibility, their paper includes training seeds, shortcut controls and external corpora, as well as small real-anomaly simulations (i.e. screen not changing) and pre-execution probes. I’d say that the main contribution is a controlled empirical diagnosis of these implementations.

Critical strengths
  • The model ordering is pretty strongly supported in their setting, as the main headline 6.53-point gap remains on both the fresh test and on negatives that are incompatible, found from the annotators (Sections 3 and 4, Tables 1-2). Therefore, these controls do make the comparison more robust/credible w.r.t. an ordinary comparison on generated corruptions. The independent-human-confirmed subset does keep a 6.60-point gap though, so the result is not just because of the generator's original labels.
  • The intermediate architectures are also a good, meaningful attempt at explanation. However, I believe the patch grids and matching the comparator still leaves a gap between joint and separate encoding (Section 5, Table 3). The paper also does disclose some limits in the claims so even though the experimental work is valuable, I don’t think it passes the threshold on significance for acceptance.
Critical weaknesses
  • The main weakness is that the scientific insight and reach is limited. The authors do discuss similar methods in Section 2, but their main recommendation (to let the action and both observed screens interact inside the encoder) remains very close to this Related Works discussion. This study also quantifies this effect in just one backbone family and one main latent predictive construction (with additional corpora/sizes)-- so the claims are not enough to establish a broader finding about predictive GUI verification. Therefore, I don’t believe the resulting design conclusion is sufficiently novel or consequential for acceptance in its current form. For example, Appendix J states there are different observed negative mixtures for the matched-head variants.
  • The practical significance of the gap they measure is not well established. Retry, recovery and process reward are important motivations for the work, but the main benchmark compares transitions with synthetic corruptions. The paper also doesn’t show that the AUC or low-FPR gap improves task completion, recovery, or invalid-retry decisions. See Table 19 and its discussion, i.e. the author audit says 80% of sampled no-op labels depend on successful execution.
Questions for authors
  • What result allows us to find a GUI-specific insight beyond the joint-versus-separate encoding distinction?
  • Does any existing eval connect the verifier differences to useful accept, retry, or recovery decisions with some frozen policy?

Comments on the paper

1 comment
Area_Chair_bftgOpenReview ↗

Urgent Reminder: Discussion Period Ends November 18

Dear Reviewers,

The discussion period will end on November 18, 2026. Please carefully review the authors' responses and actively engage in discussions.

If you have not yet initiated a discussion, please do so immediately. Reviewers are expected to address the authors' rebuttals, provide constructive follow-up comments, and update their evaluations when appropriate.

Please also ensure that your reviews reflect independent human judgment and that any AI assistance is properly documented in accordance with the review guidelines.

Active participation is an essential reviewer responsibility and is critical to ensuring a rigorous and fair review process.

Thank you for your cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.