- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes JustJudge, a training-free framework for evaluating GUI-agent trajectories by sanitizing self-reported outcomes and combining claim-free action logs with selected screenshots. Experiments across three benchmarks demonstrate competitive judging accuracy at relatively low inference cost. The authors further show that JustJudge can improve GUI-agent training when used for trajectory filtering and reinforcement learning rewards.
Critical strengths
-
Clear identification of an important evaluation bias. The paper highlights how self-reported success claims in agent trajectories can bias VLM-based judges. The controlled claim-flipping experiment provides convincing evidence of this issue and motivates the proposed evidence-sanitization strategy.
-
Strong empirical evaluation and practical efficiency. JustJudge achieves competitive performance across three GUI judging benchmarks while requiring substantially fewer model calls than multi-stage evaluation frameworks. The controlled same-backbone comparisons and downstream SFT/RL experiments further demonstrate the practical value of improving evidence presentation.
Critical weaknesses
-
Limited methodological novelty. The framework primarily combines rule-based masking, LLM-based intent summarization, and heuristic screenshot selection. Prior work has already recognized the unreliability of agent-generated reasoning and explored excluding or restructuring trajectory information [1, 2]. Although the particular combination is effective, the paper does not establish a substantial methodological advance beyond evidence preprocessing and prompt engineering.
-
Limited isolation of the key design components. The proposed method jointly modifies the trajectory text and screenshot selection strategy. While the authors analyze their combined effects, the experiments do not sufficiently isolate the contributions of terminal-outcome masking, intent summarization, and frame selection. Moreover, the improvement over the raw-record baseline under a matched 16-frame setting is relatively modest (89.3% to 89.8% overall accuracy on OSReward). More fine-grained ablations are needed to establish which components are essential and whether the added extraction step is justified.
-
Limited evidence of downstream generalization. Both downstream experiments train and evaluate agents on OSWorld-Verified, and the RL training tasks are drawn from the evaluation task pool. Although this controls the reward source across experimental conditions, it does not establish whether improved judging leads to better agent performance on unseen tasks. Consequently, the reported downstream gains support the usefulness of JustJudge within the evaluated task distribution but provide limited evidence of broader generalization.
References
[1] Xue, T., et al. (2025). An Illusion of Progress? Assessing the Current State of Web Agents. Conference on Language Modeling, 2025.
[2] Cui, C., et al. (2026). Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interaction. arXiv:2602.00575.
Questions for authors
-
Can the authors provide separate ablations for terminal-outcome masking, intent extraction, and screenshot selection under identical evaluation conditions? How much of the improvement comes from removing explicit success claims alone?
-
Given the relatively small overall accuracy gain over the raw-record baseline, can the authors provide statistical significance tests or confidence intervals across the evaluation benchmarks?
-
Can the authors evaluate the downstream SFT and RL models on held-out OSWorld tasks or another GUI benchmark? This would clarify whether improvements from JustJudge generalize beyond the training task distribution.