- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SeekJudge, a model-based reward/judge framework for computer-use agents (CUAs). The authors decompose long-trajectory judging into localization and extraction, and implement it with four agents sharing a 9B backbone (Condense, Ground, Seek, Analyze): Condense and Ground compress the trajectory into a text timeline, and the Seek–Analyze loop queries screenshot evidence on demand. For training, they use seed-driven distillation, where teacher models such as DeepSeek-V3.2 and Gemini-3.0-Flash generate supervision data, followed by SFT into Qwen3.5-9B. The model outputs 7-dimensional trajectory scores and 9-class step labels, which are mapped by a regressor to trajectory scores and step scores for offline evaluation and online RL rewards. The authors also build CUAStepBench and propose a rollout-overlapped reward server. Experiments include GRPO on three OSWorld domains—Chrome, Impress, and OS—as well as three offline reward benchmarks.
Critical strengths
- The problem is important. Reward modeling for long CUA trajectories is indeed a key bottleneck in current RL training, and how to combine step-level and trajectory-level signals is worth studying.
- The system is relatively complete. The paper not only proposes a judge but also includes a benchmark, a reward server, online RL, and efficiency analysis, representing substantial engineering effort.
- The rollout overlap design has practical value. Table 5 shows that overall overhead drops from 18.3s to 7.9s, a reduction of about 56.8%, which is useful for RL training scheduling.
- The cost and context analysis is detailed. Although its assumptions still need verification, it at least provides a discussable engineering direction.
Critical weaknesses
1、 Limited novelty, the core is an engineering combination
Ideas such as multi-agent systems, tool use, PRMs, step-level rewards, and trajectory compression have already appeared extensively in prior work. The main contribution here looks more like combining these components into a system with teacher distillation. The 7-dimensional scoring and 9-class step taxonomy are largely heuristic definitions, lacking theoretical grounding or sufficient human-consistency validation.
2、 Experimental comparisons are not sufficiently fair
SeekJudge’s training relies on distillation from strong teachers such as DeepSeek-V3.2, Gemini-3.0-Flash, and Codex GPT-5.5, whereas baselines such as CUAJudge use only GPT-5-mini and OS-Themis uses Qwen3-VL-8B, so the resources and supervision signals are clearly unequal. When compared with rule-based reward, the rule reward requires no additional distillation, human annotation, or teacher calls, while SeekJudge uses a large amount of extra supervision. In Table 2, the average test success improvement is about 3.3 points with a standard error of about 1.15, i.e., about 2.9 SE, which is not overwhelming; moreover, on Qwen3-VL-8B in the OS domain, SeekJudge reaches 24.17, lower than rule-based reward’s 26.67, indicating that the improvement is unstable.
3、 Offline evaluation evidence is weak and carries bias and contamination risks
CUAStepBench is built by the authors. Although the paper claims leakage control, only about 20 trajectories are used for criteria prompt search, and no inter-annotator agreement (IAA) is reported. In Table 3, SeekJudge-9B’s step-level F1 is only 38.0, essentially on par with CUAJudge’s 37.9 and far below the human expert’s 82.2, showing that step-level judgments are not reliable. In Table 4, on OmniGUIRewardBench, the Rule baseline is 100.0 while SeekJudge-9B is 85.1, indicating that it still clearly underperforms rule-based rewards on tasks where rules can be defined. Yet the authors generalize the conclusion to replacing rule-based reward, which is not sufficiently supported. In addition, teacher models may have already seen public benchmarks, and the paper does not sufficiently perform n-gram, embedding, or task-level decontamination verification.
4、 RL experiments are small-scale, but the conclusions are too strong
Each domain has only 12–24 training/test tasks, and training is 75 steps, so both task quantity and diversity are limited. Table 11 shows that SeekJudge still has ΔJ−ΔG = 4.2 of unconfirmed improvement, indicating a risk of reward hacking. In Table 6, SeekJudge’s training success is 30.06, clearly lower than rule-based reward’s 43.00, showing that its reward signal is not fully aligned with true success. The authors claim to be “the first model-based reward to match or surpass native rule-based reward,” but this is verified only on three small-scale OSWorld domains, with degradation in some domains, and no cross-platform Android/Web RL results are given.
Questions for authors
1、Can you provide complete results, confidence intervals, and significance tests for each domain across multiple random seeds?
2、How do you prove that the teacher models have not seen CUAStepBench, AgentRewardBench, or OmniGUIRewardBench during training? Was task-level decontamination performed?
3、When comparing with rule-based reward, did you include SeekJudge’s distillation, teacher calls, and training costs? Was the comparison conducted under the same compute/data budget?
4、Why is SeekJudge lower than rule-based reward on Qwen3-VL-8B in the OS domain? How do you explain this drop?
3 comments
Response to Reviewer jgbD (Part 2/2)
RE3-Q3. Author-built CUAStepBench, about 20 anchor trajectories, and annotator agreement.
Through a multi-stage annotation check and a blind independent expert, we show that the labels of CUAStepBench are reliable, and the about 20 trajectories touch no benchmark label.
The about 20 human-labeled trajectories are anchors for the criteria prompt. They define what counts as success and touch no benchmark label (Appendix E.1).
During the annotation of CUAStepBench, the same annotator checked each trajectory twice, a Codex audit flagged 51 suspicious labels as candidates, and two human annotators reviewed every flagged label and made all corrections, correcting 12 and discarding 7. An independent expert who did not take part in the annotation blindly labeled all 278 trajectories and reaches a trajectory-level F1 of 82.2 (Appendix J).
RE3-Q4. The step-level F1 of 38.0 against the 82.2 of the human expert.
The comparison mixes two levels. 82.2 is a trajectory-level F1, and among step-level numbers, SeekJudge-9B has the best step-level F1 of all small models.
The step-level cells of the expert row in Table 3 are empty, and the paper reports no human step-level F1. 38.0 should be compared with other step-level numbers. Among small models, SeekJudge-9B has the highest step-level F1, 38.0 against 21.2 for CUAJudge on Qwen3-VL-8B and 15.5 for OS-Themis, and it matches CUAJudge on GPT-5-mini at 37.9.
RE3-Q5. Rule reaches 100.0 on OmniGUIRewardBench.
Rule reaches 100.0 by construction, because the labels of OmniGUIRewardBench are produced by its execution rules, and on human labels SeekJudge-9B is well above Rule.
The caption of Table 4 states this, and the entries of Rule are set in italics and excluded from the comparison. This column measures agreement with the rules and does not show that the rules are correct. Two results show that the rules and humans disagree.
- On the human-labeled AgentRewardBench, Rule reaches 67.1 and SeekJudge-9B reaches 76.6.
- In Figure 1(a), on the 71 trajectories where the rule and the judge disagree, blind human raters side with the judge 48 times.
RE3-Q6. Scale of the RL experiments, reward hacking, platforms and cost.
Through a bound on reward hacking and the cost of distillation, we show that SeekJudge has the most rule-confirmed gain among model-based rewards and needs only a one-time cost.
Scale and platforms. RL runs on OSWorld because the comparison with rule-based reward needs rules as the test metric, and the step limits follow ACuRL so that our results are comparable to prior CUA RL work. The offline evaluation covers Android, Windows, Ubuntu, macOS and the web. The training success of Table 6 is addressed in correction (3) above.
Reward hacking. ΔJ − ΔG treats the rules as ground truth, and the rules disagree with human judgment (RE3-Q5), so it is an upper bound on hacking. Under this bound, SeekJudge has the largest rule-confirmed gain ΔG among the three model-based rewards, 9.5 against 5.1 for CUAJudge and 5.0 for OS-Themis, and the smallest unconfirmed share of its J gain, 31% against 83% and 37%.
Cost. Distillation is a one-time cost. The teacher API costs below 6,000 RMB, about 850 USD, with a proxy model trained on the outputs of Gemini-3.0-Flash serving most Analyze, Condense and Ground calls. SFT takes 41.4 hours on eight RTX A6000 GPUs, 331 GPU hours in total. Every reward uses the same 75 RL steps and the same GPUs.
All new experiments and related changes in this response have been added to the revised paper.
Dear Reviewer jgbD,
We would appreciate it if you could let us know whether our clarifications address your concern. If there are any aspects that you feel would benefit from further explanation, we would be happy to elaborate.
Response to Reviewer jgbD (Part 1/2)
Response to Reviewer jgbD (RE3)
Before the point-by-point response, we point out the following two major misunderstandings in this review, each of which can be checked against a single table.
(1) The review compares the step-level F1 of SeekJudge-9B, 38.0, with the 82.2 of the human expert and concludes that step-level judgments are unreliable. In Table 3, 82.2 is the trajectory-level F1 of the expert, and the step-level cells of the expert row are empty.
(2) The review reads the 100.0 of Rule on OmniGUIRewardBench as evidence that SeekJudge underperforms rule-based reward. The caption of Table 4 states that the labels of OmniGUIRewardBench are produced by its execution rules, so Rule reaches 100.0 by construction, and its entries are set in italics and excluded from the comparison. On the human-labeled AgentRewardBench in the same table, Rule reaches 67.1 and SeekJudge-9B reaches 76.6.
Each of these errors takes a correct number from the paper and attaches it to the wrong quantity. We ask the reviewer to verify these two points against the paper and to reconsider the assessment in light of the corrections below.
RE3-Q1. Novelty, and the basis of the seven dimensions and nine step classes.
Through an RL ablation and the step-level F1 against human labels, we show that the seven dimensions and nine classes both improve RL and agree with human step labels, and the novelty of the paper lies in the two results below.
To our knowledge, this paper is the first in two respects.
We also separate localization from extraction in a CUA judge, after showing that a verdict degrades as more screenshots share one forward pass even when the decisive screenshot is present.
The review lists multi-agent systems, tool use, PRMs, step-level rewards and trajectory compression as prior ideas. Prior work has used each of them, and none of the prior work that we know of has produced a model-based reward that matches native rule-based reward in online RL.
The seven dimensions and nine classes are supported at two levels.
RE3-Q2. Is the comparison fair?
Yes. Through controlled comparisons on the same backbone, we show that the comparison is fair and that the SeekJudge framework leads without any distillation.
The paper already contains such comparisons, and we added more (R-Table 1 and R-Table 2, RE1-Q1).
Every entry that needs a fitted regressor follows the held-out protocol of RE1-Q3.