- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The verification of computer-use agents often requires the VLM models for judgement and therefore the paper proposes OSReward to evaluate existing VLMs on validating multi-platform (web, mobile, ubuntu, windows) benchmarks with human-gold labels. It also introduces the OSReward-Hard and OSReward-Multi for detailed analysis. In addition, the paper proposes OS-Shepherd 100K and trains the 9B and 35B model via SFT and GRPO for a stronger os reward judge. Results show a near-frontier accuracy at lower cost.
Critical strengths
- The problem formulation is clear and important.
- The end-to-end pipeline with labor work for annotation and meta review is incredible.
- The discussions are in-depth and extensive.
- The paper not only proposes a benchmark but also the instruction data with sft/rl experiments to transform the weak open-source model into stronger judges.
Critical weaknesses
- Certain details are not explained. For example, the annotators rely on the multi-modal replay for gold labeling while the agent judge only receives the last five images. It remains unknown whether the human labeler uses the same five images as agent judges.
- There exist no comparison with other CUA reward models (e.g., voting majority/voting-strict unanimous in CUARewardBench).
- The validity and generalizability of the OS-Shepherd instructions are not discussed with respect to the human-labeled instructions in OSReward bench.
Questions for authors
- Please add more explanations on the annotation details (see above).
- Please add cost analysis (including API cost).
- More details on the SFT and RL repos (open-sourced project would be nicer) are encouraged for reproduction.
3 comments
Response to Reviewer QZiv (2/2)
W3. Training-instruction validity and generalization
OSReward benchmark instructions undergo individual human validity checks and peer cross-checking (§3.2). The larger OS-Shepherd training corpus combines independently collected and reused trajectories, with source-specific instruction and trajectory screening (Figure 6; Appendices A and E.1). For example, the Ubuntu synthesis pipeline uses environment-grounded instruction generation, a second-model quality check, deduplication, and verifier compile/runtime checks.
Training instructions are also screened against OSReward instructions using embedding similarity. Appendix E.1 reports no matches above a cosine-similarity threshold of 0.8. This check complements the held-out benchmark construction. Separately, the 96/100 human-majority agreement audit assesses the retained outcome labels, providing evidence about supervision quality.
Finally, the new external evaluations directly assess model generalization: compared with the base model, OS-Shepherd improves CUARewardBench [1] accuracy by 4.04 points and AgentRewardBench [2] success precision by 12.44 points under matched inputs.
Table 2. AgentRewardBench success judgment.
| Judge | Success precision ↑ (primary) | Success recall ↑ | Failure recall ↑ |
|---|---|---|---|
| GPT-4o | 69.8 | 83.1 | 86.93 |
| Claude 3.7 Sonnet | 69.4 | 76.3 | 87.79 |
| WebJudge-7B | 75.7 | 58 | 93.22 |
| Qwen3.5-9B | 69.3 | 80.34 | 87.05 |
| OS-Shepherd-9B | 81.74 | 60.68 | 95.07 |
We check that the tasks are valid, verify the training labels through human review, and test the trained judge on external benchmarks to assess generalization.
Q2. Annotation, inference, and training costs
Appendix E.1 reports approximately $22K in API costs for corpus annotation. Under the pricing assumptions in the submission, Table 10 estimates the cost of judging all 1,019 benchmark trajectories at $1.36 for OS-Shepherd-9B, $45.44 for GPT-5.5, and $86.04 for Claude-Opus-4-8. These estimates show why a smaller specialized judge is useful when evaluation requires many repeated calls.
Section 6.3 illustrates this repeated-call setting using an agent-training scenario with 51,200 judge queries. These annotation and inference costs are separate from the compute cost of training OS-Shepherd.
Q3. Reproducing SFT and RL
Appendix E.2 and Table 15 describe the two-stage SFT and RL data construction, checkpoint selection, and GRPO setup. SFT runs for three epochs with the one-epoch checkpoint retained; RL uses about 2.9K training and 0.2K validation samples, batch size 16, eight rollouts per sample, learning rate 1e-6, and approximately 150 updates, implemented with verl and SGLang. Our anonymous repository (https://anonymous.4open.science/r/OSReward/) also includes trajectory-collection and evaluation code, along with illustrative cases. We will release the full project to support reproducibility and further research by the community.
References
[1] Lin et al. CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent. 2025. (https://arxiv.org/abs/2510.18596)
[2] Lù et al. AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories. 2025. (https://arxiv.org/abs/2504.08942)
Thank you for the response. I maintain my positive rating and recommend for acceptance. Please add the infomation above into your manuscript.
Response to Reviewer QZiv (1/2)
Thank you for your constructive review and suggestions.
Human annotators inspect the full step-by-step multimodal replay, including all screenshots, recorded reasoning, and actions. Three annotators label each trajectory independently, and disagreements receive senior review (§3.3; Appendix B.1). The standard procedure uses two senior reviewers; difficult Hard-selection cases may involve discussion among two to three senior reviewers.
Model judges receive the task, full textual history, and the last five screenshots under a fixed visual-input budget. Figure 15 supports that budget empirically, while the human labels use the fuller record to establish the reference verdict.
Thank you for suggesting comparisons with existing CUA reward models and voting strategies. We have added an evaluation on CUARewardBench, comparing OS-Shepherd-9B with its Qwen3.5-9B base and the published results of SE-WSM, GUI-OWL, and GLM-4.5V. Under matched inputs, OS-Shepherd improves accuracy by 4.04 percentage points, balanced accuracy by 4.43 points, and failure recall by 21.81 points over its base. It also exceeds the published GLM-4.5V-106B SE-WSM result in accuracy and success precision. OS-Shepherd and its Qwen3.5-9B base use the same last-five-screenshot setting. Published baselines retain their original evaluation settings.
Table 1. CUARewardBench outcome judging (272 trajectories).
Published baselines use SE-WSM and are from Table 4 of [1].
Section 5 also examines repeated sampling and voting as inference-time strategies for improving judge performance. Unanimous agreement serves a separate purpose in our framework: selecting training trajectories with consistent labels across teachers.