- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OSWorld-Pro evaluates CUA trajectories through intermediate subgoals rather than final deliverables alone. It contains 305 tasks, 2,814 subgoals, and 67,264 step-level labels covering targeted subgoals, feasibility, progress, and completion. Independent human annotation is followed by review. The paper evaluates GPT5.6 judges against human labels, selects a whole-trajectory judge, and benchmarks 14 agent configurations. Results show difference in subgoal completion, action-level progress, and persistence on failed or infeasible subgoals.
Critical strengths
-
The central contribution is a rich annotation resource and a framework for diagnosing where long computer-use workflows break down.
-
The annotation effort is actually substantial and I appreciate it. Over 5,000 person-hours, independent reviewer passes, and strong reported human agreement. Labels distinguish intended targets, feasibility, advancement, and completion, providing diagnostic information unavailable from a final success bit, and from other existing benchmarks.
-
The action and subgoal-level analyses expose practical differences in agent efficiency and failure persistence.
Critical weaknesses
-
The targeting rules constrain successive labels to the same or next subgoal and explicitly exclude earlier subgoals. How these rules accommodate legitimate revisits or alternative execution orders is unclear, and no corresponding mislabeling audit is reported. I think this matters for evaluating open-ended workflows fairly.
-
Table 1 reports 94.1% subgoal accuracy and 93.0% task-level 1−MAE, but the latter measures the fraction of completed subgoals, not the all-subgoals success decision used in Table 2. Conditional feasibility/progress Macro-F1 are only 61.9/74.7. End-to-end validation of task success and close model rankings would be nice to have.
Questions for authors
-
How do the monotonic targeting rules interact with the restart/method-switch rule? Can you show how valid revisits and alternative orders are scored, and quantify errors on such trajectories?
-
Beyond Table 1’s subgoal accuracy and task-level 1−MAE, what is human agreement on the all-feasible-subgoals task decision for the Table 2 agents? (Related to 2nd weakness I listed earlier)
-
Were judge prompts selected using a separate development set? Please identify the trajectory-generating models and the task/model split used for judge development versus the Table 1 evaluation.
5 comments
Author Response to Reviewer S1Vv (Part 2 of 3)
Table 1 reports 94.1% subgoal accuracy and 93.0% task-level 1−MAE, but the latter measures the fraction of completed subgoals, not the all-subgoals success decision used in Table 2. Conditional feasibility/progress Macro-F1 are only 61.9/74.7. End-to-end validation of task success and close model rankings would be nice to have.
Beyond Table 1’s subgoal accuracy and task-level 1−MAE, what is human agreement on the all-feasible-subgoals task decision for the Table 2 agents? (Related to 2nd weakness I listed earlier)
We tabulate the binary agreement for tasks in Table 1 to derive a 80.7% binary agreement rate for the GPT-5.6-Sol Max judge that we later use for tabulating Table 2, versus the 91.4% binary agreement rate for Human Performance.
In addition, we agree that it will be useful to carry out a human-audit experiment on end-to-end task success. Specifically, we chose two representative models’ trajectories (Claude Opus 5 and Qwen 3.8 Flash Next - which are among top closed-source and open-weight models respectively) and for each model, sampled 15 near-miss trajectories (i.e. close to completing all feasible subgoals) and 16 successful trajectories. This represents a 10% sub-sample of the trajectories for each model, which is the limit for what human annotators can feasibly annotate during this discussion period without compromising on quality. Then, we ask three human annotators to indicate if the trajectories show that the task has been fully completed across all feasible subgoals. Comparing the majority vote across 3 human annotators (Cohen’s kappa=0.798, substantial agreement), the LLM-judge’s prediction agrees on 90.3% of Claude Opus 5 ratings and 93.5% of Qwen 3.8 Flash Next ratings.
Finally, we concur with the reviewer’s concern on conditional feasibility/progression only being 61.9/74.7 Macro-F1 and conduct an experiment to understand the trustworthiness of the LLM-Judge assessment of feasibility and progression. Specifically, we asked three human annotators to label step-wise feasibility/progress on two representative models’ trajectories (Claude Opus 5 and Qwen 3.8 Flash Next). Next, we identified 100 steps across 31 tasks for each model, stratified-sampled equally across feasibility and progression labels. Compared with the human majority label, llm-judge labelled feasibility agrees in 99% of labels while llm-judge labelled progression (when feasibility=Yes, following Section 6) agrees on 96%. Details on progression predictions across different agents and action categories are shown in the table below. Table B showing agreement proportion of step-level progression label between LLM Judge and majority vote among human judges.
| Model | Overall | Click | Drag+Move | Keyboard | Scroll | Execution Control | Others |
|---|---|---|---|---|---|---|---|
| Claude Opus 5 | 48/51 (94.1%) | 16/17 (94.1%) | 13/13 (100%) | 7/7 (100%) | 4/5 (80.0%) | 4/4 (100%) | 4/5 (80.0%) |
| Qwen 3.8 Flash Next | 48/49 (98.0%) | 13/13 (100%) | 9/9 (100%) | 13/13 (100%) | 3/3 (100%) | 6/7 (85.7%) | 4/4 (100%) |
| Combined | 96/100 (96.0%) | 29/30 (96.7%) | 22/22 (100%) | 20/20 (100%) | 7/8 (87.5%) | 10/11 (90.9%) | 8/9 (88.9%) |
Such high agreements across feasibility and progression indicate that the main diagnostic conclusions based on LLM-judge labels will likely remain consistent with human labels. It might be a little surprising that these agreement rates are so much higher than expected from the Macro-F1 scores of 61.9 and 74.7. This is because Macro-F1 averages across highly-imbalanced categories, which perceivably deflates the scores. The comparable agreement rate for the GPT-5.6-Sol Max judge is 86.5% and 82.3% for feasibility and progression respectively. In addition, human inter-annotator agreement is also excellent but not perfect at 0.820-0.869 Cohen’s kappa due to some annotator subjectivity. Table 1 compares the judge against a single annotator, whereas this audit uses the majority of three, which removes individual annotator noise, meaning that the 86.5/82.3% LLM Judge agreement is likely a deflated estimate of the true agreement compared to the “denoised” ground truth human annotation, which should be around 99/96%. Together, these results indicate that judge agreement on the headline metric is close to human–human agreement for the Table 2 agents.
Author Response to Reviewer S1Vv (Part 3 of 3)
Were judge prompts selected using a separate development set? Please identify the trajectory-generating models and the task/model split used for judge development versus the Table 1 evaluation.
Motivation: To the best of our knowledge, many literature that use LLM-Judges only report the final prompt template used, without disclosing how they were arrived at [1, 2, 3]. We sought to provide more transparent insights into the process through which LLM Judge prompt templates were derived so as to inspire others to build upon our efforts. At the same time, we believe that our LLM Judge prompt development strategy (detailed below) follows best practices in settings where human-annotated data is limited due to substantial annotator expertise/effort required and LLM-judges are off-the-shelf rather than purposefully trained.
Judge Prompt Development Details: We experimented with alternative judge prompt templates on a subset of the final evaluation dataset to identify the optimal prompt template. Specifically, judge prompts were selected based on ~20% of tasks delivered from initial batches, with trajectories generated with Kimi K2.5/K2.6 and the judge model being GPT-5.6-Luna Medium (different from the GPT-5.6-Sol Max we used for Table 2). This data was later included as part of the final dataset, used for experiments in Table 1 evaluation. We did not set aside a portion of the data for identifying the optimal judge prompt template given the limited data we have and the low risk of overfitting that we perceived due to prompt iterations having been made manually.
[1] NAACL 2025 Findings - RewardBench: Evaluating Reward Models for Language Modeling by Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
[2] NeurIPS 2023 - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica
[3] ArXiv/OpenAI 2025 - HealthBench: Evaluating Large Language Models Towards Improved Human Health by Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, Karan Singhal
Author Response to Reviewer S1Vv (Part 1 of 3)
We thank the reviewer for recognizing our work as a rich annotation resource, built with substantial human annotation effort (>5000 person-hours) and exposed practical differences in computer-use agent behavior in ways that existing benchmarks cannot.
We first describe our motivation behind establishing a rule to constrain successive labels to the same or next subgoal and then show evidence that the legitimate revisits or alternative execution orders only contribute a small difference (<4%) in overall task success rates.
Motivation: In early stages of our data annotation, we considered and experimented with different protocols for annotating subgoals targeted. Our initial protocol allowed annotators to select subgoals targeted at each step in an arbitrary manner to account for such behavior. Under that protocol, inter-annotator agreement on the targeted subgoal was unacceptably low, because "targeted" is subjective (e.g. acting on a subgoal vs. merely mentioning it in the reasoning). To make the human annotation more reliable, we updated the protocol asking annotators to only keep tasks that could be addressed in a singular sequence of monotonic subgoals. When the protocol is correctly adhered to, tasks that are kept after filtering cannot have subgoals tackled in an alternate sequence or to restart/method-switch earlier subgoals after finishing a later subgoal.
Table A comparing proportion of tasks with all subgoals completed with original step-level prompt (from Table 2) and new task-level prompt and their difference.
Experiments: We acknowledge that some annotation errors may exist, as expected of any human annotated dataset. Some tasks that admit an alternative order may have accidentally passed this filter. To assess their frequency, we conducted an experiment. Specifically, we modify the prompt to LLM-Judge to the below prompt so that we can understand the extent to which the computer-use agent completes each subgoal at a task level across all steps and therefore, whether all subgoals in a task are completed. This way, subgoals completed through legitimate revisits or alternative execution orders can be considered. Then, we compare it with the corresponding overall task completion score from Table 2. Models only show a small difference (below 4%), suggesting that there are only a tiny minority of tasks that allows models to address subgoals in an alternate order.