acceptodds
Withdrawn from ICLR 2027

JustJudge: Better Evidence Beats More Machinery in Judging GUI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deciding from a recorded trajectory whether a GUI agent completed its task underpins how these agents are evaluated, how their training data is curated and how they are rewarded. To make that decision reliable, recent work adds machinery, such as critic pipelines and trained reward models. We show that the binding constraint is what the judge is shown. A trajectory is testimony as much as evidence: the agent under judgment writes part of the record, and a judge handed that record inherits the agent's verdict on itself. Prior judges warn the model about the record or discard the agent's text, and added machinery re-reads the same record. JustJudge instead sanitizes the record: it reduces each step to the action issued and a claim-free statement of what it was for, and masks recognized self-reported outcomes. The judge checks this step log against a fixed budget of screenshots anchored at the final state. The judge is a text-only extraction pass and one verification call to a vision-language model, with no training and no environment access. On OSReward, JustJudge with an open 27B model reaches 89.8% accuracy, level with the best proprietary judges the benchmark reports, and leads every judge on the hard split. The gain comes from the evidence, not the model: shown the sanitized record, a 9B model overtakes the reward model trained from it on that split. On the same backbone, published multi-call procedures cost up to 16 times as much and score 8 to 21 points lower. Downstream, the same judge produces stronger agents on OSWorld-Verified, as a filter on fine-tuning data and as the reward for online reinforcement learning. Reliable judging is a question of evidence, not of machinery.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI agents, LLM-as-a-judge, reward modeling, computer-use agents, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vqMuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JustJudge, a training-free framework for evaluating GUI-agent trajectories by sanitizing self-reported outcomes and combining claim-free action logs with selected screenshots. Experiments across three benchmarks demonstrate competitive judging accuracy at relatively low inference cost. The authors further show that JustJudge can improve GUI-agent training when used for trajectory filtering and reinforcement learning rewards.

Critical strengths
  1. Clear identification of an important evaluation bias. The paper highlights how self-reported success claims in agent trajectories can bias VLM-based judges. The controlled claim-flipping experiment provides convincing evidence of this issue and motivates the proposed evidence-sanitization strategy.

  2. Strong empirical evaluation and practical efficiency. JustJudge achieves competitive performance across three GUI judging benchmarks while requiring substantially fewer model calls than multi-stage evaluation frameworks. The controlled same-backbone comparisons and downstream SFT/RL experiments further demonstrate the practical value of improving evidence presentation.

Critical weaknesses
  1. Limited methodological novelty. The framework primarily combines rule-based masking, LLM-based intent summarization, and heuristic screenshot selection. Prior work has already recognized the unreliability of agent-generated reasoning and explored excluding or restructuring trajectory information [1, 2]. Although the particular combination is effective, the paper does not establish a substantial methodological advance beyond evidence preprocessing and prompt engineering.

  2. Limited isolation of the key design components. The proposed method jointly modifies the trajectory text and screenshot selection strategy. While the authors analyze their combined effects, the experiments do not sufficiently isolate the contributions of terminal-outcome masking, intent summarization, and frame selection. Moreover, the improvement over the raw-record baseline under a matched 16-frame setting is relatively modest (89.3% to 89.8% overall accuracy on OSReward). More fine-grained ablations are needed to establish which components are essential and whether the added extraction step is justified.

  3. Limited evidence of downstream generalization. Both downstream experiments train and evaluate agents on OSWorld-Verified, and the RL training tasks are drawn from the evaluation task pool. Although this controls the reward source across experimental conditions, it does not establish whether improved judging leads to better agent performance on unseen tasks. Consequently, the reported downstream gains support the usefulness of JustJudge within the evaluated task distribution but provide limited evidence of broader generalization.

References

[1] Xue, T., et al. (2025). An Illusion of Progress? Assessing the Current State of Web Agents. Conference on Language Modeling, 2025.

[2] Cui, C., et al. (2026). Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interaction. arXiv:2602.00575.

Questions for authors
  1. Can the authors provide separate ablations for terminal-outcome masking, intent extraction, and screenshot selection under identical evaluation conditions? How much of the improvement comes from removing explicit success claims alone?

  2. Given the relatively small overall accuracy gain over the raw-record baseline, can the authors provide statistical significance tests or confidence intervals across the evaluation benchmarks?

  3. Can the authors evaluate the downstream SFT and RL models on held-out OSWorld tasks or another GUI benchmark? This would clarify whether improvements from JustJudge generalize beyond the training task distribution.

Reviewer_HkhoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JustJudge, a training-free framework for evaluating GUI agent trajectories. The authors identify a key limitation of existing trajectory judges: recorded trajectories contain self-reported outcomes from the agents being evaluated, which may bias the judge toward accepting unsuccessful executions.To address this issue, JustJudge replaces the original trajectory narration with a claim-free step log that preserves actions and intentions while removing self-reported success claims. It also selects screenshots with an emphasis on the final state and uses a single VLM verification call to produce the judgment.Experiments on OSReward, OGRBench, and CUARewardBench demonstrate improvements over several existing judges. The authors further evaluate JustJudge as a trajectory filter for supervised fine-tuning and as a reward source for online reinforcement learning.

Critical strengths
  1. The paper is easy to understand, and the motivation is clear.
  2. Comprehensive evaluation with encouraging downstream results.
Critical weaknesses
  1. This work is more of a "trick" than a general-purpose algorithm.
  2. This work does not actually enhance the "judger" itself; in my view, it does not constitute a meaningful contribution to the field.
Questions for authors

Does each piece of reasoning data need to be pre-processed by an LLM to remove any expressions of confidence? It seems to me that this approach introduces significant unreliability, as well as additional costs and complexity.

Reviewer_9aQDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at improving verifiers for GUI agents that determine whether the task has been successfully achieved. It proposes an extremely simple method of sanitizing the GUI agent's logs, specifically including the steps taken and intent, but masking out claims about outcomes that could bias the judge. Results are evaluated on OSReward and OSWorld.

Critical strengths
  • The paper hinges on an interesting finding, if it can be thoroughly described and validated.
Critical weaknesses
  • The details of the sanitization, which is the main point of the method, is completely inadequate. How are the declarations that relate to intent versus claims differentiated? There is basically no method section to this paper.

  • Results are shown on limited settings (domain and model), which does not say anything about the generality of the findings. Further, compared to the state of art the model used is quite weak, leading to easier improvements.

  • Overall, there is no scientific contributions to this paper nor is there a strong empirical methodology to contreibute.

  • This paper is extremely poorly written with dense language that requires re-reading most sentences to even understand what is being said. There is very little staging, intuition, or transitions. This weakness is throughout the entire paper and so would require a complete re-write.

Questions for authors

How is the actual method proposed (sanitization) done?

How generalizable is this finding across models and domains?

Can this method improve current state of art models that achieve 80+% success rates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.