acceptodds
Under review as a conference paper at ICLR 2027

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

16 views

Abstract

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's states, reasoning, and actions. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across four platforms, and are then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge subset concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. A few reliable judges are too costly to scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. Based on it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, narrowing the gap to frontier judges with 30–60x lower inference cost. Extensive analyses further inform the design of reliable CUA reward at scale.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer-using Agents, VLM, Reward Model

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_QZivOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The verification of computer-use agents often requires the VLM models for judgement and therefore the paper proposes OSReward to evaluate existing VLMs on validating multi-platform (web, mobile, ubuntu, windows) benchmarks with human-gold labels. It also introduces the OSReward-Hard and OSReward-Multi for detailed analysis. In addition, the paper proposes OS-Shepherd 100K and trains the 9B and 35B model via SFT and GRPO for a stronger os reward judge. Results show a near-frontier accuracy at lower cost.

Critical strengths
  • The problem formulation is clear and important.
  • The end-to-end pipeline with labor work for annotation and meta review is incredible.
  • The discussions are in-depth and extensive.
  • The paper not only proposes a benchmark but also the instruction data with sft/rl experiments to transform the weak open-source model into stronger judges.
Critical weaknesses
  • Certain details are not explained. For example, the annotators rely on the multi-modal replay for gold labeling while the agent judge only receives the last five images. It remains unknown whether the human labeler uses the same five images as agent judges.
  • There exist no comparison with other CUA reward models (e.g., voting majority/voting-strict unanimous in CUARewardBench).
  • The validity and generalizability of the OS-Shepherd instructions are not discussed with respect to the human-labeled instructions in OSReward bench.
Questions for authors
  • Please add more explanations on the annotation details (see above).
  • Please add cost analysis (including API cost).
  • More details on the SFT and RL repos (open-sourced project would be nicer) are encouraged for reproduction.
3 comments
AuthorsOpenReview ↗

Response to Reviewer QZiv (1/2)

Thank you for your constructive review and suggestions.

W1 and Q1. Human annotation evidence and judge inputs

Human annotators inspect the full step-by-step multimodal replay, including all screenshots, recorded reasoning, and actions. Three annotators label each trajectory independently, and disagreements receive senior review (§3.3; Appendix B.1). The standard procedure uses two senior reviewers; difficult Hard-selection cases may involve discussion among two to three senior reviewers.

Model judges receive the task, full textual history, and the last five screenshots under a fixed visual-input budget. Figure 15 supports that budget empirically, while the human labels use the fuller record to establish the reference verdict.

W2. Other reward models and voting baselines

Thank you for suggesting comparisons with existing CUA reward models and voting strategies. We have added an evaluation on CUARewardBench, comparing OS-Shepherd-9B with its Qwen3.5-9B base and the published results of SE-WSM, GUI-OWL, and GLM-4.5V. Under matched inputs, OS-Shepherd improves accuracy by 4.04 percentage points, balanced accuracy by 4.43 points, and failure recall by 21.81 points over its base. It also exceeds the published GLM-4.5V-106B SE-WSM result in accuracy and success precision. OS-Shepherd and its Qwen3.5-9B base use the same last-five-screenshot setting. Published baselines retain their original evaluation settings.

Table 1. CUARewardBench outcome judging (272 trajectories).

Published baselines use SE-WSM and are from Table 4 of [1].

Judging modelAccuracy ↑BA ↑Success precision ↑Success recall ↑Failure recall ↑
Qwen2.5-VL-32B75.475.558069.182
GUI-OWL-32B73.273.27571.275.2
SE-WSM-7B54.855.557020.191
GLM-4.5V-106B80.180.2582.97783.5
Qwen3.5-9B76.4776.1571.1990.6561.65
OS-Shepherd-9B80.5180.5883.0877.783.46

Section 5 also examines repeated sampling and voting as inference-time strategies for improving judge performance. Unanimous agreement serves a separate purpose in our framework: selecting training trajectories with consistent labels across teachers.

AuthorsOpenReview ↗

Response to Reviewer QZiv (2/2)

W3. Training-instruction validity and generalization

OSReward benchmark instructions undergo individual human validity checks and peer cross-checking (§3.2). The larger OS-Shepherd training corpus combines independently collected and reused trajectories, with source-specific instruction and trajectory screening (Figure 6; Appendices A and E.1). For example, the Ubuntu synthesis pipeline uses environment-grounded instruction generation, a second-model quality check, deduplication, and verifier compile/runtime checks.

Training instructions are also screened against OSReward instructions using embedding similarity. Appendix E.1 reports no matches above a cosine-similarity threshold of 0.8. This check complements the held-out benchmark construction. Separately, the 96/100 human-majority agreement audit assesses the retained outcome labels, providing evidence about supervision quality.

Finally, the new external evaluations directly assess model generalization: compared with the base model, OS-Shepherd improves CUARewardBench [1] accuracy by 4.04 points and AgentRewardBench [2] success precision by 12.44 points under matched inputs.

Table 2. AgentRewardBench success judgment.

JudgeSuccess precision ↑ (primary)Success recall ↑Failure recall ↑
GPT-4o69.883.186.93
Claude 3.7 Sonnet69.476.387.79
WebJudge-7B75.75893.22
Qwen3.5-9B69.380.3487.05
OS-Shepherd-9B81.7460.6895.07

We check that the tasks are valid, verify the training labels through human review, and test the trained judge on external benchmarks to assess generalization.

Q2. Annotation, inference, and training costs

Appendix E.1 reports approximately $22K in API costs for corpus annotation. Under the pricing assumptions in the submission, Table 10 estimates the cost of judging all 1,019 benchmark trajectories at $1.36 for OS-Shepherd-9B, $45.44 for GPT-5.5, and $86.04 for Claude-Opus-4-8. These estimates show why a smaller specialized judge is useful when evaluation requires many repeated calls.

Section 6.3 illustrates this repeated-call setting using an agent-training scenario with 51,200 judge queries. These annotation and inference costs are separate from the compute cost of training OS-Shepherd.

Q3. Reproducing SFT and RL

Appendix E.2 and Table 15 describe the two-stage SFT and RL data construction, checkpoint selection, and GRPO setup. SFT runs for three epochs with the one-epoch checkpoint retained; RL uses about 2.9K training and 0.2K validation samples, batch size 16, eight rollouts per sample, learning rate 1e-6, and approximately 150 updates, implemented with verl and SGLang. Our anonymous repository (https://anonymous.4open.science/r/OSReward/) also includes trajectory-collection and evaluation code, along with illustrative cases. We will release the full project to support reproducibility and further research by the community.

References

[1] Lin et al. CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent. 2025. (https://arxiv.org/abs/2510.18596)

[2] Lù et al. AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories. 2025. (https://arxiv.org/abs/2504.08942)

Thank you for the response. I maintain my positive rating and recommend for acceptance. Please add the infomation above into your manuscript.

Reviewer_yd2oOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OSReward, a benchmark for evaluating reward models in computer-use domains, together with VLM-based reward models and corresponding training data. The benchmark contains 1,019 human-labeled trajectories collected across web, Android, Ubuntu, and Windows environments using agents from the Claude, Gemini, Kimi, and Qwen families. It includes binary success/failure labels and two additional evaluation tracks: OSReward-Hard, a challenging subset of 284 trajectories, and OSReward-Multi, which adds human ratings of alignment and efficiency on a three-level scale to 440 successful trajectories.

The authors evaluate 27 VLM judges across several closed and open-weight model families, examining the effects of visual and textual inputs, reasoning effort, and repeated sampling. They find that: (i) judges systematically over-accept failed trajectories; (ii) the best judge achieves 89.7% accuracy on the full benchmark and 69.7% on OSReward-Hard; (iii) smaller open-weight models offer lower costs but lower accuracy, particularly due to substantially lower failure detection; (iv) alignment and efficiency grading remains difficult, even for frontier models; and (v) text history strongly affects judgments, while additional screenshots and inference-time computation provide limited or inconsistent improvements.

The authors also introduce OS-Shepherd-100K, a corpus of approximately 100,000 training samples combining trajectories collected in their own environment suite and public computer-use data, labeled by unanimous agreement among strong VLM judges. They train 9B and 35B-A3B VLM judges through supervised fine-tuning followed by reinforcement learning targeted to reduce over-leniency. Evaluation on OSReward and against human-written programmatic verifiers on OSWorld, WebArena, and AndroidWorld shows that their models narrows the gap to frontier judges, particularly through improved failure detection, while reducing estimated inference costs by 30–60× for the 9B model.

Critical strengths
  • The paper is exceptionally well written and organized, with clear figures and tables that make the methodology and results easy to follow.
  • The evaluation is extensive, spanning 27 VLM judges from multiple model families, alongside several ablations including visual and textual inputs, reasoning effort, repeated sampling, and trajectory length.
  • The benchmark incorporates several design choices important for faithfully evaluating VLM judges [1]: human labels, diverse environments, a reasonably balanced success/failure split (43%/57%), filtering of trajectories affected by environment failures or external obstacles (e.g., anti-bot blocks, network failures, or app crashes), trajectories generated by agents from multiple model families, and fine-grained classification metrics.
  • This paper consolidates several computer-use domains under a shared evaluation protocol with thoughtful design choices. While other human-labeled CUA-reward benchmarks exist, they are often scattered across domains and use different protocols, making consistent evaluation difficult.
  • The paper proposes a method to address over-leniency in VLM-based judges/verifiers and demonstrates improved failure detection on OSReward and external benchmarks. This is relevant, since this is a significant failure mode that can adversely affect several applications of VLM-based judges in computer-use and other open-ended domains ([1]).
  • The analyses provide useful breakdowns of success and failure recall, judge error types, platform differences, and the trade-off between inference cost and performance.
  • The authors commit to open-source the code, training data, and model checkpoints, all of which could be highly valuable to the community.
Critical weaknesses

Major

W1. Limited evidence on the generalizability of the OS-Shepherd reward model

The evaluation on trajectories generated in OSWorld, WebArena, and AndroidWorld is useful evidence beyond OSReward. However, it measures agreement with programmatic verifiers, which have several limitations (§F.1, see also [1] and [2]). Moreover, there is limited detail about the trajectories used in these evaluations. For example, how capable are the generating agents? What is the success/failure balance? Are trajectories affected by environment failures filtered out?

This is relevant because:

  • Two training choices may affect generalization: (1) using a limited selection of screenshots, predominantly from the end of each trajectory, which can limit the model ability to evaluate tasks requiring visual reasoning across longer trajectories, and (2) targeting false successes during the second training stage.
  • The models and accompanying data are important contributions that would be strengthened by clearer evidence of generalization, particularly given some overlap with prior work (see W2)

Some options that could alleviate these issues:

  • Evaluation of the reward model on external human-labeled trajectories (see W2).
  • More details about the existing evaluation pipeline.

W2. Incomplete positioning relative to relevant prior work

Some novelty claims and findings would benefit from clearer positioning relative to prior work, including work published at this same venue. In particular:

  1. Other benchmarks for VLM judges in computer-use domains exist, including with human labels and annotations beyond binary outcome labels ([3], [4], [5]). Extending the comparison in Table 8 would help clarify how OSReward's differs and substantiate claims such as those in lines 136–137 and 238–240.

  2. Evaluations of VLM-based judges and their design choices across CUA and open-ended environments also exist, including the identification of leniency bias and its adverse effects ([1] at this same venue). Discussion of previous findings could help situate your contributions, and further substantiate some of the paper's choices and claims (e.g., lines 416–417 about why false positives can be particularly harmful and the motivation for targeted training).

A clearer discussion of prior work would clarify the paper's novelty and further substantiate its importance---for example, by highlighting its integration of diverse environments and its reproduction, extension, and mitigation of established issues with VLM judges.

Minor

W3. Clarification of OSReward-Hard trajectory selection

OSReward-Hard is an important component of the benchmark, given the relatively high full-set performance of frontier models and potential effects of evaluation choices discussed in W5. However, the selection procedure could be described more clearly. The authors explain this data comes from disagreement among human raters later revised by senior reviewers. More detail on how this review distinguishes genuinely challenging judgments from ambiguity in the task or available evidence would help clarify the nature of the subset.

W4. More fine-grained trajectory-length statistics

More detailed trajectory-length statistics, particularly for OSReward-Hard and individual platforms, would help interpret the results. This is relevant given the relationship between trajectory length and verifiability (§F.3) and the decision to provide judges with a limited number of trailing screenshots.

W5. Strengthening the evidence on VLM judge performance

  • Limiting visual input to the last five screenshots in the main setting may omit evidence needed to judge some trajectories, potentially reducing the accuracy of the evaluated models.
  • Some results suggest that screenshot selection warrants consideration alongside screenshot count: the first-plus-last-two screenshot configuration yields small improvements for some judges, and this configuration is included when training OS-Shepherd.
  • Although the authors vary the number of trailing screenshots from 1 to 16 and find no consistent improvement, this may reflect factors other than the models' ability to use additional visual information. For example, (i) trajectories may be shorter than the tested budgets, or (ii) benchmark tasks may rarely require visual reasoning across distant steps beyond what the textual history conveys (e.g., comparing a slide's initial and final appearance).
  • Moreover, requiring the judge to output a binary success/failure verdict and relying on textual history may affect failure detection ([1], [2], [6]; also lines 346–348), which is shown to be the most challenging feature in the full benchmark.

Additional analyses that could increase confidence in the reported results include:

  • More trajectory-length statistics and performance breakdowns by length (see W4)
  • Evaluation of the VLM judge implementation on other human-labeled benchmarks
  • Smarter selection of screenshot along the trajectory (see [2])

W6. Other minor comments on clarity, organization, and possible improvements

  • A release of the environments used for data collection would be greatly beneficial for the community.
  • Evaluating downstream applications of OS-Shepherd (e.g., best-of-N selection, RL training, filtered behavior cloning, or Reflexion-style pipelines) would further demonstrate its practical utility (see [1] for discussion about downstream evaluation options).
  • In lines "447-454", it may be beneficial to clarify that the cost is of repeatedly querying a reward model to train an agent. Given the surrounding discussion, readers may confuse this with the cost of training OS-Shepherd itself.
  • Additional breakdowns of sRec and fRec across the studied ablations may provide further insights (e.g., the impact of text/vision modalities on false positives/negatives)

References

[1] Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification. ICLR 2026.

[2] An Illusion of Progress? Assessing the Current State of Web Agents. COLM 2025.

[3] OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models. CVPR 2026.

[4] AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories. COLM 2025.

[5] WebArbiter: A Generative Reasoning Process Reward Model for Web Agents. ICLR 2026.

[6] RoboReward: General-Purpose Vision-Language Reward Models for Robotics. arXiv 2026.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer yd2o (1/4)

Thank you for your thoughtful review and helpful references. Your comments have helped us clarify OSReward’s contributions and better explain our evaluation design.

Major

W1. Whether OS-Shepherd generalizes beyond OSReward when evaluated against human labels rather than verifiers

The OSWorld evaluation uses the public OSWorld-Verified trajectory collection. The mobile evaluation uses AndroidWorld test trajectories, which are separate from the OpenMobile data used for training. We collected the WebArena trajectories with Gemini-3.5-Flash. Across the selected subsets, we screened out trajectories affected by identifiable environment failures as far as possible and maintained a roughly balanced mix of successful and failed executions.

We now evaluate OS-Shepherd-9B and its Qwen3.5-9B base on two external human-annotated benchmarks (CUARewardBench[1] and AgentRewardBench[2]), using the same last-five-screenshot setting for both models. Published baselines retain their original evaluation settings; their sources are given below.

Table 1. CUARewardBench outcome judging.

Judging modelAccuracy ↑BA ↑Success precision ↑Success recall ↑Failure recall ↑
Qwen2.5-VL-32B75.475.558069.182
GUI-OWL-32B73.273.27571.275.2
SE-WSM-7B54.855.557020.191
GLM-4.5V-106B80.180.2582.97783.5
Qwen3.5-9B76.4776.1571.1990.6561.65
OS-Shepherd-9B80.5180.5883.0877.783.46

On CUARewardBench, OS-Shepherd improves accuracy by 4.04 points, BA by 4.43 points, and failure recall by 21.81 points over its base. Its 80.51% accuracy and 83.08% success precision exceed the published GLM-4.5V-106B results under SE-WSM.

Table 2. AgentRewardBench success judgment. Our results use the official 1,106-trajectory test split. The benchmark prioritizes success precision because it measures how often a trajectory accepted as successful really succeeds, directly reflecting the quality of selected training demonstrations and limiting false-positive reward signals.

JudgeSuccess precision ↑ (primary)Success recall ↑Failure recall ↑
GPT-4o69.883.186.93
Claude 3.7 Sonnet69.476.387.79
WebJudge-7B75.75893.22
Qwen3.5-9B69.380.3487.05
OS-Shepherd-9B81.7460.6895.07

OS-Shepherd improves success precision by 12.44 points, and failure recall by 8.02 points. The proportion of failed trajectories among those accepted as successful falls from 30.70% to 18.26%, a 40.5% relative reduction.

Together, these results highlight the value of our diverse, cross-platform training data: OS-Shepherd’s gains extend beyond OSReward to external human-annotated desktop and web benchmarks. Its stronger failure detection and success precision demonstrate that a compact 9B judge can generalize across evaluation settings and more reliably distinguish successful executions from failures.

AuthorsOpenReview ↗

Response to Reviewer yd2o (2/4)

W2. Incomplete positioning relative to relevant prior work

Thank you for the references and for highlighting these distinctions. We have expanded Table 8 of the submission to clarify how our work compares with prior studies.

Table 3. Comparison with related CUA reward work.

Dataset / frameworkPlatformsActionInstr.Traj.GoldLabelsTraining corpusModel
OSReward (ours)W, M, DGUI+CLI✓✓✓Binary + fine-grained100K trajectory–response samples9B/35B
Web-Shepherd [4]WGUI~~~Checklist40K3B/8B
GUI-Shepherd [5]MGUI————52K7B
CUARewardBench [1]DGUI××✓Binary——
OS-Themis [6]W, M, DGUI×✓×Binary——
AgentRewardBench [2]WBrowser actions×✓✓Success, side effects, repetition——
OS-Oracle [7]W, M, DGUI××✓Binary action correctness—7B
WebArbiter [8]WBrowser actions×~~Action preference30K action-preference pairs3B/7B

OSReward is the first and largest human-annotated benchmark for computer-use judging spanning Web, Android, Ubuntu, and Windows under a common protocol. We write new instructions and collect fresh trajectories from diverse agents, expanding task coverage and reducing reliance on previously circulated traces. Three-way human annotation with meta-review establishes the evaluation labels independently of the original verifiers. OSReward-Hard targets difficult outcome judgments, while OSReward-Multi evaluates alignment and efficiency beyond task completion. Beyond the benchmark, we provide OS-Shepherd-100K, a large-scale cross-platform training corpus, together with open OS-Shepherd judges and an SFT–RL training recipe that connect evaluation to practical model improvement.

Our study provides the most comprehensive analysis across all four CUA platforms using human-annotated trajectories. It examines leniency and other judging errors, how input modalities and screenshot budgets affect verdicts, and the trade-off between accuracy and inference cost. We will add [9] to the related-work discussion and cite it alongside the discussion of false-success judgments in lines 416–417.

Minor

W3. Clarification of OSReward-Hard trajectory selection

Annotator disagreement triggers senior review of both outcome clarity and judgment difficulty. Two to three senior reviewers discuss each disputed candidate's instruction, recorded evidence, and proposed outcome. A trajectory enters Hard when they agree that the outcome is determinate but establishing it requires difficult evidence integration.

Appendix B.2 reports the review of 373 candidates: 284 are retained in Hard, 89 return to Full, and 18 outcome labels are corrected. Separately, 109 trajectories with residual quality problems are removed when constructing the complete gold set. The 89 trajectories returned to Full remain valid benchmark examples.

AuthorsOpenReview ↗

Response to Reviewer yd2o (3/4)

W4 and W5. Trajectory length and screenshot selection

Our default input combines the full recorded action and reasoning history with up to five trailing screenshots. Table 4 reports trajectory lengths by platform.

Table 4. Trajectory length (steps) by platform.

PlatformFull: nFull: meanFull: medianHard: nHard: meanHard: median
Web16615.3126015.312
Mobile25622.2216321.122
Desktop59737.63216135.424
Overall1,01930.1232842821.5

To examine performance on longer executions, Table 5 reports judging accuracy by trajectory length under the default last-five-screenshot setting.

Table 5. Judging accuracy (%) by trajectory length.

Length (steps)Full: nFull: Acc.Hard: nHard: Acc.
1–57789.21754.9
6–1630185.49058.1
17–3229886.610265.4
33–6422695.34080.8
65+11790.33567.6
Total1,01988.828464.9

Accuracy shows no consistent decline with trajectory length on either set. For example, the judges achieve 95.3% accuracy on Full and 80.8% on Hard for trajectories of 33–64 steps. These results describe performance across length groups; Figure 15 directly tests the screenshot budget and finds no consistent accuracy gain beyond five trailing screenshots.

Together, these findings support five trailing screenshots as a practical accuracy–cost trade-off on this benchmark. The same last-five-screenshot setting also supports effective judging on external human-annotated benchmarks. We agree that selecting screenshots according to the execution could further improve judging; all screenshots are released with the benchmark, allowing such strategies to be evaluated against the same human annotations.

AuthorsOpenReview ↗

Response to Reviewer yd2o (4/4)

W6. How the released resources and downstream results support practical use, and what the reported costs cover

The anonymous repository (https://anonymous.4open.science/r/OSReward/) contains our evaluation and cross-platform trajectory-collection code, together with examples. We are making the benchmark, training data, models, and code openly available to support reproducible research and further development by the community.

To demonstrate downstream utility, we evaluate OS-Shepherd on Best-of-3 trajectory selection, using publicly released OpenCUA-7B trajectories on OSWorld [10]. Each query has three independent rollouts. We construct a 100-query desktop subset that prioritizes queries containing both successful and unsuccessful rollouts, supplemented by randomly sampled queries whose rollouts are either all successful or all unsuccessful.

We select uniformly among candidates with the highest predicted binary reward and report the expected success rate under this rule, averaged across queries. Success is evaluated using the released verifier labels.

Table 6. Offline Best-of-3 trajectory selection on 100 OSWorld queries.

PlatformRandomQwen3.5-9BOS-Shepherd-9BOracle
Desktop33.67%39.83%47.67%56.00%

Oracle selects a successful rollout whenever one exists.

OS-Shepherd improves selection success over its base model by 7.83 percentage points on desktop. These results demonstrate that improved judging translates into more successful trajectory selection.

Finally, Section 6.3 estimates the cost of repeatedly querying a reward model during agent training: 200 updates × batch size 16 × 16 rollouts = 51,200 judge calls. Appendix E.1 separately reports approximately $22K in annotation API costs for constructing the training corpus. The former measures downstream judge usage and the latter measures training-data annotation cost.

References

[1] Lin et al. CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent. 2025.

[2] Lù et al. AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories. 2025.

[3] WebJudge-7B results: AgentRewardBench leaderboard (https://huggingface.co/spaces/McGill-NLP/agent-reward-bench-leaderboard); released predictions (https://github.com/OSU-NLP-Group/Online-Mind2Web/tree/main/data/evaluation_results/agent_reward_bench_evaluation_results).

[4] Chae et al. Web-Shepherd: Advancing PRMs for Reinforcing Web Agents. 2025.

[5] Chen et al. GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks. 2025.

[6] Li et al. OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards. 2026.

[7] Wu et al. OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models. 2025.

[8] Zhang et al. WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents. 2026.

[9] Andrade et al. Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification. 2025.

[10] XLANG Lab. OSWorld-Verified trajectories, OpenCUA-7B Ubuntu runs.

Reviewer_Yz1MOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of evaluating reward models for computer use agents, i.e., models that assign binary (or some more dense) rewards to trajectory traces of agents attempting to solve computer use/GUI tasks. Unlike problem where there are more verifiable rewards (i.e., math), the authors point out that for computer use tasks, the determination of the reward often itself requires intelligence beyond scripted rules, i.e., requires a LLM/VLM as a judge model, and so it is prudent to create a benchmark evaluating reward models for computer use agents. The benchmark the authors introduce is OSReward, which was constructed by sampling for a wide variety of computer use tasks across different operating system platforms (e.g., windows/linux), agent traces on these computer use tasks for a wide variety of different agents. Then human evaluators manually look at each trace and together assign a ground truth reward label for that trace, via a process where three humans look at each trace, and if there’s disagreement two “meta-reviewers” look at the trace again and make a final decision. Then open-weight and closed-source models are evaluated on this benchmark. The authors find that across the full OSReward benchmark, the best models can do quite well, but on a harder version of OSReward, called OSReward-Hard which samples traces where there was disagreement among the three humans, the best models only score 70% accuracy, meaning that even with SOTA models there is room for improvement. The authors also have a denser-reward variant of the benchmark (as opposed to binary) called OSReward-Multi, for which models generally to much worse, indicating that denser rewards are much harder to capture for these models accurately than binary rewards. A consistent trend the authors find across many models is that models tend to believe agent traces when the agent in these traces erroneously mention in their thinking that they have solved the task, when in reality they have not. As a second part of the paper, the authors set out to create a dataset for which better open-weight computer use reward models can be trained, as in their benchmark they found that open-weight models were much worse than closed-source ones. They release a dataset of 100k agent trajectories with their reward annotations, called OS-Shepherd-100k, where unlike with the benchmark, the reward annotations are not human generated, but instead generated using a combination of strong API models. Training 9B and 35B open-weight models on this dataset via SFT and RL leads to them getting much stronger.

Critical strengths

(1) The paper does a good job at what it sets out to do, creating a benchmark for computer use reward models. The way the authors sampled diverse computer use tasks, sampled trajectories from a diverse set of agents, and set up a human annotation pipeline all seem very sound to me and well designed. It is likely that this benchmark indeed is a good benchmark for evaluating computer use models. Also it is a good sign that at least on the hardest version of the benchmark, OSReward-Hard, the best models aren’t completely solving it yet, which means iterating on this benchmark might be useful for model developers. In the hard mode, the main failures that models find hard to catch are failures that are evident only after the model views the image from the trajectory, which the fact that this is the case shows that visuals are important, and so this is also a strength.

(2) The paper is fairly well-written and easy to follow.

(3) The experimentation is thorough: the authors do things like evaluate learned reward models against scripted verifiers on prior benchmarks whose reward models are scripted (section 7), and I thought experiments like this were well-thought-out and interesting.

Critical weaknesses

The main weakness in my opinion is with implementation details of the latter part of the authors’ paper, the OS-Shepherd-100k dataset release and subsequent open-weight model training on this dataset.

One major concern I have is with the use of an ensemble of strong closed API models as the source of reward model data for the 100k agent traces. The procedure the authors mentioned in the paper for obtaining reward supervision for the 100k agent traces is that using an ensemble of different reward models from I think different API providers (which models are used exactly?), only agent traces for which all reward models agree are kept in the 100k sized dataset. I think the motivation for doing this is that this procedure, at least intuitively, means that the reward annotations are likely correct if all models agreed on it. However this procedure is problematic for two reasons. (1) if all models share a failure mode, as outlined prior in the paper these models do — they all are too lenient, especially when the agent in its thinking trace seems to think that is succeeded even when it did not — then agreement can still result in the inclusion of this same failure mode in the final dataset. I.e., the bias of leniency will persist. Thus when later in the paper the authors say that training on this dataset reduced the model’s leniency problem, I don’t see why this will be the case (beyond chance). If the authors claim that this ensembling procedure removes this bias, then the authors should prove this by evaluating the ensemble of closed source models itself as a row on their benchmark (see final paragraph in weaknesses section). The second problem (and more major issue) is that the agent traces for which the API models disagree are exactly the ones one should want to keep, as it means these agent traces are particularly difficult to model rewards for. By not including them, the dataset will bias significantly towards very easy to assign reward agent traces.

Another concern related to the above: if API models are used as the reward labels, training open-weight models on this dataset can in theory result in the models doing no better than the closed-source models used for distillation. This may not be particularly bad if this is the intended behavior. Do the authors think there is a way to use API models to result in a dataset than if trained on could lead models (open weight or even hypothetically the same closed-source models) to do better than the closed-source API models? Again if not, this isn’t a major concern for me.

One more major concern/question: are the same OS-Shepherd-100k examples used for SFT also used for RL? If so, this is problematic, because new training examples should be used for RL, as otherwise the labels have already been seen and the model in the SFT phase has already overfit to these labels. Could the authors clarity more about the details of their SFT and RL setups? Are the performance curves that show performance after the SFT phase, and performance after RL, and how much each helps? Are multiple epochs/passes through the data taken in the RL phase? What about the SFT phase?

And one more weakness: even after SFT/RL, the open-weight models, while better than before, are not the perfect or even the best compared to closed source models. If such models are used as reward functions within an RL training run, can these models still learn despite the reward functions being erroneous a significant fraction of the time? To answer this question an RL training run using the learned reward models would be needed.

The paper is missing experimental evaluations of the ensemble of closed source models. This ensemble, which is used to create the 100k dataset, should also itself be evaluated as a “model” on the OS-Reward/OS-Reward-Hard/OS-Reward-Multi benchmarks. This feels like a needed experiment in the paper.

And one final (small) weakness: the paper only very briefly describes how it’s novel compared to prior work that also proposed reward model benchmarks for computer use agents. For me it wasn’t sufficiently justified/proven what this benchmark does better than prior ones, and why this one should be used.

Questions for authors

(1) There is an anonymous code link but nothing is posted there.

(2) Can you expand on how exactly the reward tasks in OSReward-Multi are “denser” than binary rewards? I.e., are there concrete examples of what exactly is being reward/evaluated in these scenarios that I can take a look at? Also, in frontier LLM training, binary/verifiable rewards are often used, but so is denser rewards. Is the type of dense reward in OSReward-Multi analogous to the types of dense rewards used in frontier LLM training? Are dense rewards needed do the authors think for computer use tasks (if so why?)? What is the implication if so given the fact that even the strongest API models seem bad at OSReward-Multi?

(3) “with trajectories ranging from roughly a dozen to 100 steps” —-> can you provide an example of a step?

(4) If trajectories can span up to 100 steps, is 5 screenshots sufficient? Why?

(5) Figure 5 shows that the reward models conditioning on reasoning is important for their accuracy. But isn’t this actually a really bad idea to do? For safety reasons, in frontier LLM training reward models do not condition on reasoning information (as far as I know) because doing so can lead the model to adjust its reasoning traces to be more opaque or even deceptive, meaning chain-of-thought-monitoring, a standard practice to assess safety and alignment of models, no longer works. If the reward models need to condition on reasoning traces, this seems like a significant weakness.

(6) The term ensembling is not the most correct. If multiple closed source models are used to only select examples for the dataset for which all models agree, then this is filtering, not ensembling, which usually means selecting the reward outcome that most models agree on.

(7) The authors mention the dataset they collected also contains judge reasoning traces. Does this mean they used an open-weight model? Which open-weight model or model(s) did they use?

(8) For what example trajectories do human-written verifiers not work? Why do we need learned models? I buy it, but I’d like to see examples. Perhaps this is in the appendix and I missed it (if so kindly point me to it).

5 comments
AuthorsOpenReview ↗

Response to Reviewer Yz1M (1/5)

Thank you for your review! We appreciate your recognizing our work as a timely addition to the community and that our evaluation results deliver insights.

W1, W5, and Q6. Whether teacher agreement yields reliable supervision despite shared biases and the exclusion of difficult examples

The labeling judges are GPT-5.5, Gemini-3.1-Pro, and Kimi-K2.5. We retain a trajectory only when all three agree on its outcome label. Following your clarification in Q6, we refer to this operation as cross-model agreement filtering; majority voting is a separate prediction-aggregation baseline. As the original GPT and Kimi models were unavailable to us during rebuttal, these experiments use GPT-5.6-Terra, Gemini-3.1-Pro, and Kimi-K3. Following W5, Table 1 evaluates each judge and their majority vote on OSReward and OSReward-Hard. We further examine the reliability of agreement-filtered labels on OSReward Full.

Table 1. Individual judges and unweighted majority voting.

JudgeOSReward Acc. (%)OSReward BA (%)OSReward-Hard Acc. (%)OSReward-Hard BA (%)
GPT-5.6-Terra88.7188.966.5567.46
Gemini-3.1-Pro87.9388.1262.6862.71
Kimi-K389.790.7165.4972.95
Majority vote90.1990.8267.2570.92

Label reliability. On Full, the three judges agree on 882/1,019 trajectories (86.56%), and their shared label is correct on 829/882 (93.99%). In contrast, majority voting is correct on only 90/137 (65.69%) of the disagreement cases. Filtering therefore retains most trajectories while selecting substantially more reliable automatic supervision. Appendix E.1 reports agreement with human majority labels on 96 of 100 audited trajectories (95% Wilson interval: 90.2% to 98.4%).

Shared errors and reduced student leniency. Agreement filtering cannot eliminate errors shared by the teachers. Our training nevertheless explicitly targets false-success judgments: data collection increases the supply of FAIL trajectories (§6.1), and targeted RL focuses on recoverable student errors (§6.2). Together, SFT and RL increase Hard failure recall from 14.1% to 57.6%(Figure 17). This improvement does not require agreement filtering to eliminate all shared teacher errors.

Difficult training examples. Teacher agreement does not mean that an example is easy for the student. Our RL stage specifically mines trajectories whose agreed labels the SFT model predicts incorrectly under greedy decoding but correctly in some sampled responses (§6.2; Appendix E.2). These examples remain challenging for the student despite consistent teacher supervision. Thus, excluding teacher-disagreement cases does not eliminate difficult examples from training.

W2. Student performance relative to its teachers

Our objective is an affordable open judge that approaches strong commercial judges. OS-Shepherd-9B reaches 86.1% accuracy on Full, compared with 89.5%–89.7% for the leading judges in the submission, at substantially lower estimated inference cost (Tables 9–10). This makes specialized judging accessible at the repeated-call scale needed for trajectory filtering and reward-model use.

The remaining gap may reflect differences in model capacity and the breadth of pretrained knowledge. Our evaluation spans Web, Android, Ubuntu, and Windows, requiring generalization across diverse applications, interface conventions, and task semantics. A compact judge must therefore learn both the judging procedure and how to apply it across substantially different environments. Task-specific training improves judging, but may not fully bridge these broader capability differences between a compact model and frontier models.

We have not demonstrated a data construction method that enables a student to surpass its teachers. However, voting and richer evaluation procedures suggest possible routes beyond an individual model’s single-pass performance. A richer evaluation harness could also help retrieve knowledge, inspect relevant intermediate screenshots, and combine requirement-level checks before producing a verdict. Better evidence and additional inference-time computation may help narrow or potentially overcome the gap to a standalone teacher call [1], although this remains a direction to investigate.

AuthorsOpenReview ↗

Response to Reviewer Yz1M (2/5)

W3. Reusing SFT examples for RL and training details

Thank you for raising this question. The RL set is mined from the SFT pool. Using the one-epoch SFT checkpoint, we identify recoverable errors: examples judged incorrectly under greedy decoding but correctly in some sampled responses. We mix in correctly judged examples and sample across platforms and success/failure labels to improve balance, producing approximately 3.1K examples.

The RL stage specifically targets false-success judgments that persist after SFT—failed executions that the model still accepts as successful (§6.2; Appendix E.2). These errors are particularly consequential for reward-model use because they assign positive rewards to unsuccessful behavior. We therefore focus on recoverable errors that the SFT checkpoint has encountered but still cannot judge reliably. The two stages also use different objectives: SFT learns to reproduce teacher responses, whereas RL evaluates the model’s own sampled responses for verdict correctness and output-format compliance. The aim is to reduce residual leniency and improve failure detection through targeted optimization of these errors.

We run SFT for three epochs and retain the one-epoch checkpoint. RL uses approximately 150 GRPO updates, batch size 16, eight rollouts per example, learning rate 1e-6, and a token-level KL penalty to the SFT reference with coefficient 0.001 (Appendix E.2; Table 15). Generalization is evaluated on held-out benchmarks.

Figure 17 compares base → SFT → SFT+RL. SFT provides most of the accuracy gain; RL chiefly improves failure recall, with reduced success recall. Across both stages, Hard failure recall increases from 14.1% to 57.6%, BA from 55.9% to 61.9% (Table 1).

W4. Whether an acting agent can improve through RL when its reward model makes errors

Thank you for this helpful question. We agree that an online agent-RL study is the most direct test of whether an acting agent can learn effectively from OS-Shepherd's rewards. Such a study requires more time and compute than were available during the rebuttal period. We therefore provide two complementary assessments.

Erroneous positive rewards. For success-based rewards, one relevant source of noise is failed trajectories being rewarded as successful. Using the success and failure recalls in Table 1 of the submission and OSReward Full's distribution of 440 successful and 579 failed trajectories, we estimate that 35% of trajectories accepted by the Qwen3.5-9B base are failures. This falls to 17.5% for OS-Shepherd-9B, approaching Claude-Opus-4-8's 14%. The same pattern holds on the external human-annotated AgentRewardBench test split [2], where success precision increases from 69.30% to 81.74% (+12.44 percentage points).

Reward-guided trajectory selection. Prior work studies both RL and best-of-N optimization against imperfect reward models, finding improvements before overoptimization degrades reference performance [3]. We therefore conduct an offline Best-of-3 experiment using publicly released OpenCUA-7B trajectories on OSWorld [4], with three independent rollouts per query.

We construct a 100-query subset that prioritizes queries containing both successful and unsuccessful rollouts, supplemented by randomly sampled queries whose rollouts are either all successful or all unsuccessful. Both judges evaluate identical candidates. We select the highest-reward candidate, breaking ties uniformly at random, and report the expected success rate under this rule, averaged over queries. Success is evaluated using the released verifier labels.

Table 2. Offline Best-of-3 trajectory selection on 100 OSWorld queries.

MetricRandomQwen3.5-9BOS-Shepherd-9BOracle
Success rate33.67%39.83%47.67%56.00%

Oracle selects a successful rollout whenever one exists.

OS-Shepherd raises selection success from 33.67% to 47.67% (+14.00 percentage points), recovering approximately 63% of the available improvement from random to oracle selection, compared with approximately 28% for its base model. It also exceeds the base model's selection success by approximately 7.8 percentage points.

These results show that, on this subset, favoring trajectories according to OS-Shepherd's predictions improves task outcomes despite its judgment errors. This supports its use for trajectory selection and provides an initial assessment of reward usefulness. Evaluating whether this signal supports sustained policy improvement as the trajectory distribution changes during online RL is an important direction for our future work.

AuthorsOpenReview ↗

Response to Reviewer Yz1M (3/5)

W6. What OSReward contributes beyond existing computer-use reward works

OSReward is the first human-annotated benchmark for complete-trajectory computer-use judging spanning Web, Android, Ubuntu, and Windows. Unlike benchmarks confined to individual platforms or focused on action-level criticism, OSReward evaluates both task completion and execution quality at the complete-trajectory level under a common protocol.

A key distinction is that we collect new execution trajectories rather than reusing existing public traces. Reused traces have three limitations: (1) their instructions are limited in diversity and number; (2) they may already be contaminated, and their labels inherit the errors of the original verifiers; and (3) most importantly, they lack human verification. On top of this, OSReward-Hard targets difficult outcome judgments and OSReward-Multi assesses alignment and efficiency beyond task success.

Beyond the benchmark, we release OS-Shepherd-100K, to our knowledge the largest open corpus of trajectory-level judgments to date, with more than 100K cross-platform trajectory-response samples, together with open OS-Shepherd judges and an SFT–RL training recipe that connect evaluation to practical model improvement. We also provide a detailed, systematic analysis of judge failure modes.

The following table highlights these distinctions by comparing OSReward with related work.

Table 3. Comparison with related CUA reward work.

Dataset / frameworkPlatformsActionInstr.Traj.GoldLabelsTraining corpusModel
OSReward (ours)W, M, DGUI+CLI✓✓✓Binary + fine-grained100K trajectory–response samples9B/35B
Web-Shepherd [5]WGUI~~~Checklist40K3B/8B
GUI-Shepherd [6]MGUI————52K7B
CUARewardBench [7]DGUI××✓Binary——
OS-Themis [8]W, M, DGUI×✓×Binary——
AgentRewardBench [2]WBrowser actions×✓✓Success, side effects, repetition——
OS-Oracle [9]W, M, DGUI××✓Binary action correctness—7B
WebArbiter [10]WBrowser actions×~~Action preference30K action-preference pairs3B/7B

Q1. Anonymous repository access

We have corrected the synchronization issue, and the anonymous repository (https://anonymous.4open.science/r/OSReward/) now contains evaluation and analysis code, cross-platform trajectory-collection code, and example benchmark and SFT trajectories. Its examples section also includes cases relevant to Q2–Q4, including a 100-step trajectory, so that the scoring definitions and long-trajectory setting can be inspected concretely.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.