acceptodds
Under review as a conference paper at ICLR 2027

OSWorld-Pro: Process-based Evaluation for Computer Use Agents

Abstract

Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why agents fail in various tasks, obfuscating critical insight for subsequent improvement. For instance, agents that err during keyboard inputs would require a different mitigation strategy from those that fail to precisely provide click-based inputs on the graphical UI. We introduce OSWorld-Pro: a set of over 300 tasks containing over 2800 subgoals to enable the procedural evaluation of CUAs grounded in over 67,000 human annotations. We use robust human-aligned LLM-Judges to evaluate the fulfillment of OSWorld-Pro subgoals and thereby reveal the progress that models make throughout a series of sequentially dependent subgoals. Our findings reveal that OSWorld-Pro is challenging even for state-of-the-art LLMs, with top performers like Claude Opus 5 achieving only 75.7% vs. 83.4% on OSWorld. Furthermore, we identify critical process-focused failure modes of various models (e.g. subgoal-irrelevant actions and click-based mistakes) to provide insights to improve performance and efficiency of CUAs.

Primary area:
datasets and benchmarks
Keywords:
computer-use agents, process-based evaluation, human-annotated data, long-horizon tasks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_S1VvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OSWorld-Pro evaluates CUA trajectories through intermediate subgoals rather than final deliverables alone. It contains 305 tasks, 2,814 subgoals, and 67,264 step-level labels covering targeted subgoals, feasibility, progress, and completion. Independent human annotation is followed by review. The paper evaluates GPT5.6 judges against human labels, selects a whole-trajectory judge, and benchmarks 14 agent configurations. Results show difference in subgoal completion, action-level progress, and persistence on failed or infeasible subgoals.

Critical strengths
  • The central contribution is a rich annotation resource and a framework for diagnosing where long computer-use workflows break down.

  • The annotation effort is actually substantial and I appreciate it. Over 5,000 person-hours, independent reviewer passes, and strong reported human agreement. Labels distinguish intended targets, feasibility, advancement, and completion, providing diagnostic information unavailable from a final success bit, and from other existing benchmarks.

  • The action and subgoal-level analyses expose practical differences in agent efficiency and failure persistence.

Critical weaknesses
  • The targeting rules constrain successive labels to the same or next subgoal and explicitly exclude earlier subgoals. How these rules accommodate legitimate revisits or alternative execution orders is unclear, and no corresponding mislabeling audit is reported. I think this matters for evaluating open-ended workflows fairly.

  • Table 1 reports 94.1% subgoal accuracy and 93.0% task-level 1−MAE, but the latter measures the fraction of completed subgoals, not the all-subgoals success decision used in Table 2. Conditional feasibility/progress Macro-F1 are only 61.9/74.7. End-to-end validation of task success and close model rankings would be nice to have.

Questions for authors
  • How do the monotonic targeting rules interact with the restart/method-switch rule? Can you show how valid revisits and alternative orders are scored, and quantify errors on such trajectories?

  • Beyond Table 1’s subgoal accuracy and task-level 1−MAE, what is human agreement on the all-feasible-subgoals task decision for the Table 2 agents? (Related to 2nd weakness I listed earlier)

  • Were judge prompts selected using a separate development set? Please identify the trajectory-generating models and the task/model split used for judge development versus the Table 1 evaluation.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer S1Vv (Part 1 of 3)

We thank the reviewer for recognizing our work as a rich annotation resource, built with substantial human annotation effort (>5000 person-hours) and exposed practical differences in computer-use agent behavior in ways that existing benchmarks cannot.

The targeting rules constrain successive labels to the same or next subgoal and explicitly exclude earlier subgoals. How these rules accommodate legitimate revisits or alternative execution orders is unclear, and no corresponding mislabeling audit is reported. I think this matters for evaluating open-ended workflows fairly.

How do the monotonic targeting rules interact with the restart/method-switch rule? Can you show how valid revisits and alternative orders are scored, and quantify errors on such trajectories?

We first describe our motivation behind establishing a rule to constrain successive labels to the same or next subgoal and then show evidence that the legitimate revisits or alternative execution orders only contribute a small difference (<4%) in overall task success rates.

Motivation: In early stages of our data annotation, we considered and experimented with different protocols for annotating subgoals targeted. Our initial protocol allowed annotators to select subgoals targeted at each step in an arbitrary manner to account for such behavior. Under that protocol, inter-annotator agreement on the targeted subgoal was unacceptably low, because "targeted" is subjective (e.g. acting on a subgoal vs. merely mentioning it in the reasoning). To make the human annotation more reliable, we updated the protocol asking annotators to only keep tasks that could be addressed in a singular sequence of monotonic subgoals. When the protocol is correctly adhered to, tasks that are kept after filtering cannot have subgoals tackled in an alternate sequence or to restart/method-switch earlier subgoals after finishing a later subgoal.

Table A comparing proportion of tasks with all subgoals completed with original step-level prompt (from Table 2) and new task-level prompt and their difference.

ModelStep-level Prompt (%)Task-level Prompt (%)Difference (%)
Claude Opus 5 Max75.777.0+1.3
Claude Opus 4.8 Max77.777.4-0.3
Claude Opus 4.7 Max76.780.3+3.6
Claude Sonnet 5 Max76.477.7+1.3
GPT-5.6-Sol Max74.877.9+3.1
GPT-5.6-Terra Max70.873.9+3.1
GPT-5.6-Luna Max74.472.0-2.4
Gemini-3.8-Flash High59.059.3+0.3
Kimi K3 Max39.342.6+3.3
Minimax M3 Xhigh28.925.6-3.3
Qwen 3.8 Flash Next Xhigh55.159.0+3.9
Qwen 3.5 122B Xhigh16.719.0+2.3
Qwen 3.8 27B Xhigh32.133.1+1.0
Qwen 3.6 27B Xhigh10.28.5-1.7

Experiments: We acknowledge that some annotation errors may exist, as expected of any human annotated dataset. Some tasks that admit an alternative order may have accidentally passed this filter. To assess their frequency, we conducted an experiment. Specifically, we modify the prompt to LLM-Judge to the below prompt so that we can understand the extent to which the computer-use agent completes each subgoal at a task level across all steps and therefore, whether all subgoals in a task are completed. This way, subgoals completed through legitimate revisits or alternative execution orders can be considered. Then, we compare it with the corresponding overall task completion score from Table 2. Models only show a small difference (below 4%), suggesting that there are only a tiny minority of tasks that allows models to address subgoals in an alternate order.

Step 0. Reasoning: Action: Step 1. Reasoning: Action: ... Step k. Reasoning: Action: Which of the following subgoals are completed by the attached screenshots (one for each step in sequential order), reasoning, and action? Subgoals: 0. 1. ... n. Return only a nested JSON response e.g {“completed": [0, 1, 2]}.
AuthorsOpenReview ↗

Author Response to Reviewer S1Vv (Part 2 of 3)

Table 1 reports 94.1% subgoal accuracy and 93.0% task-level 1−MAE, but the latter measures the fraction of completed subgoals, not the all-subgoals success decision used in Table 2. Conditional feasibility/progress Macro-F1 are only 61.9/74.7. End-to-end validation of task success and close model rankings would be nice to have.

Beyond Table 1’s subgoal accuracy and task-level 1−MAE, what is human agreement on the all-feasible-subgoals task decision for the Table 2 agents? (Related to 2nd weakness I listed earlier)

We tabulate the binary agreement for tasks in Table 1 to derive a 80.7% binary agreement rate for the GPT-5.6-Sol Max judge that we later use for tabulating Table 2, versus the 91.4% binary agreement rate for Human Performance.

In addition, we agree that it will be useful to carry out a human-audit experiment on end-to-end task success. Specifically, we chose two representative models’ trajectories (Claude Opus 5 and Qwen 3.8 Flash Next - which are among top closed-source and open-weight models respectively) and for each model, sampled 15 near-miss trajectories (i.e. close to completing all feasible subgoals) and 16 successful trajectories. This represents a 10% sub-sample of the trajectories for each model, which is the limit for what human annotators can feasibly annotate during this discussion period without compromising on quality. Then, we ask three human annotators to indicate if the trajectories show that the task has been fully completed across all feasible subgoals. Comparing the majority vote across 3 human annotators (Cohen’s kappa=0.798, substantial agreement), the LLM-judge’s prediction agrees on 90.3% of Claude Opus 5 ratings and 93.5% of Qwen 3.8 Flash Next ratings.

Finally, we concur with the reviewer’s concern on conditional feasibility/progression only being 61.9/74.7 Macro-F1 and conduct an experiment to understand the trustworthiness of the LLM-Judge assessment of feasibility and progression. Specifically, we asked three human annotators to label step-wise feasibility/progress on two representative models’ trajectories (Claude Opus 5 and Qwen 3.8 Flash Next). Next, we identified 100 steps across 31 tasks for each model, stratified-sampled equally across feasibility and progression labels. Compared with the human majority label, llm-judge labelled feasibility agrees in 99% of labels while llm-judge labelled progression (when feasibility=Yes, following Section 6) agrees on 96%. Details on progression predictions across different agents and action categories are shown in the table below. Table B showing agreement proportion of step-level progression label between LLM Judge and majority vote among human judges.

ModelOverallClickDrag+MoveKeyboardScrollExecution ControlOthers
Claude Opus 548/51 (94.1%)16/17 (94.1%)13/13 (100%)7/7 (100%)4/5 (80.0%)4/4 (100%)4/5 (80.0%)
Qwen 3.8 Flash Next48/49 (98.0%)13/13 (100%)9/9 (100%)13/13 (100%)3/3 (100%)6/7 (85.7%)4/4 (100%)
Combined96/100 (96.0%)29/30 (96.7%)22/22 (100%)20/20 (100%)7/8 (87.5%)10/11 (90.9%)8/9 (88.9%)

Such high agreements across feasibility and progression indicate that the main diagnostic conclusions based on LLM-judge labels will likely remain consistent with human labels. It might be a little surprising that these agreement rates are so much higher than expected from the Macro-F1 scores of 61.9 and 74.7. This is because Macro-F1 averages across highly-imbalanced categories, which perceivably deflates the scores. The comparable agreement rate for the GPT-5.6-Sol Max judge is 86.5% and 82.3% for feasibility and progression respectively. In addition, human inter-annotator agreement is also excellent but not perfect at 0.820-0.869 Cohen’s kappa due to some annotator subjectivity. Table 1 compares the judge against a single annotator, whereas this audit uses the majority of three, which removes individual annotator noise, meaning that the 86.5/82.3% LLM Judge agreement is likely a deflated estimate of the true agreement compared to the “denoised” ground truth human annotation, which should be around 99/96%. Together, these results indicate that judge agreement on the headline metric is close to human–human agreement for the Table 2 agents.

AuthorsOpenReview ↗

Author Response to Reviewer S1Vv (Part 3 of 3)

Were judge prompts selected using a separate development set? Please identify the trajectory-generating models and the task/model split used for judge development versus the Table 1 evaluation.

Motivation: To the best of our knowledge, many literature that use LLM-Judges only report the final prompt template used, without disclosing how they were arrived at [1, 2, 3]. We sought to provide more transparent insights into the process through which LLM Judge prompt templates were derived so as to inspire others to build upon our efforts. At the same time, we believe that our LLM Judge prompt development strategy (detailed below) follows best practices in settings where human-annotated data is limited due to substantial annotator expertise/effort required and LLM-judges are off-the-shelf rather than purposefully trained.

Judge Prompt Development Details: We experimented with alternative judge prompt templates on a subset of the final evaluation dataset to identify the optimal prompt template. Specifically, judge prompts were selected based on ~20% of tasks delivered from initial batches, with trajectories generated with Kimi K2.5/K2.6 and the judge model being GPT-5.6-Luna Medium (different from the GPT-5.6-Sol Max we used for Table 2). This data was later included as part of the final dataset, used for experiments in Table 1 evaluation. We did not set aside a portion of the data for identifying the optimal judge prompt template given the limited data we have and the low risk of overfitting that we perceived due to prompt iterations having been made manually.

[1] NAACL 2025 Findings - RewardBench: Evaluating Reward Models for Language Modeling by Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi

[2] NeurIPS 2023 - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica

[3] ArXiv/OpenAI 2025 - HealthBench: Evaluating Large Language Models Towards Improved Human Health by Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, Karan Singhal

Reviewer_9n7XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OSWorld-Pro, a process-based benchmark for computer-use agents, comprising 305 tasks, 2,814 subgoals, and over 67,000 human annotation labels. It evaluates which subgoal each action targets, whether it is feasible, and whether the action makes progress or completes it. The authors validate LLM judges against human annotations and benchmark multiple agents, revealing failures and inefficiencies that final-outcome evaluation can overlook.

Critical strengths
  1. The process-based evaluation provides useful diagnostic information beyond task success, including where agents fail and how they waste actions. Its coverage of multiple applications, Linux distributions, and desktop interfaces also adds valuable evaluation settings beyond a single familiar environment.
  2. The benchmark involves substantial human annotation effort, with independent review and strong inter-annotator agreement. This provides a valuable reference for evaluating automated judges and supports the usefulness of the dataset.
Critical weaknesses
  1. The automated judge remains substantially weaker than humans on feasibility and progression, with Macro-F1 scores of 61.9 and 74.7, respectively (Table 1). These labels directly support the paper’s analysis of action failures and inefficient persistence. Although the authors acknowledge this limitation, further human validation is needed to establish whether judge errors affect the main diagnostic conclusions.
  2. The annotation guidelines and judge prompt impose a strict sequential ordering of subgoals, including restrictions on returning to earlier subgoals (Appendices D and F). This may penalize legitimate backtracking, repairs, or alternative execution orders. Selecting sequentially dependent tasks partially addresses this concern, but does not establish that all valid trajectories follow the prescribed order.
Questions for authors
  1. Have you manually checked feasibility and progression predictions across different agents and action categories? Do the main diagnostic conclusions remain consistent with human judgments?
  2. Clarification: How does the evaluation handle valid trajectories that revisit an earlier subgoal or complete subgoals in a different order? Can you provide evidence that the ordering restrictions do not incorrectly penalize such trajectories?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer 9n7X (Part 1 of 2)

We thank the reviewer for recognizing our work as useful, valuable and grounded on substantial human annotation effort.

The automated judge remains substantially weaker than humans on feasibility and progression, with Macro-F1 scores of 61.9 and 74.7, respectively (Table 1). These labels directly support the paper’s analysis of action failures and inefficient persistence. Although the authors acknowledge this limitation, further human validation is needed to establish whether judge errors affect the main diagnostic conclusions.

Have you manually checked feasibility and progression predictions across different agents and action categories? Do the main diagnostic conclusions remain consistent with human judgments?

To better understand the trustworthiness of the LLM-Judge assessment of feasibility and progression, we asked three human annotators to label step-wise feasibility/progress on two representative models’ trajectories (Claude Opus 5 and Qwen 3.8 Flash Next - which are among top closed-source and open-weight models respectively). Specifically, we identified 100 steps across 31 tasks for each model, stratified-sampled equally across feasibility and progression labels. This represents a 10% sub-sample of the tasks for each model, which is the limit for what human annotators can feasibly annotate during this discussion period without compromising on quality. Compared with the human majority label, llm-judge labelled feasibility agrees in 99% of labels while llm-judge labelled progression (when feasibility=Yes, following Section 6) agrees on 96%. Details on progression predictions across different agents and action categories are shown in the table below.

Table A showing agreement proportion of step-level progression label between LLM Judge and majority vote among human judges.

ModelOverallClickDrag+MoveKeyboardScrollExecution ControlOthers
Claude Opus 548/51 (94.1%)16/17 (94.1%)13/13 (100%)7/7 (100%)4/5 (80.0%)4/4 (100%)4/5 (80.0%)
Qwen 3.8 Flash Next48/49 (98.0%)13/13 (100%)9/9 (100%)13/13 (100%)3/3 (100%)6/7 (85.7%)4/4 (100%)
Combined96/100 (96.0%)29/30 (96.7%)22/22 (100%)20/20 (100%)7/8 (87.5%)10/11 (90.9%)8/9 (88.9%)

Such high agreements across feasibility and progression indicate that the main diagnostic conclusions based on LLM-judge labels will likely remain consistent with human labels. It might be a little surprising that these agreement rates are so much higher than expected from the Macro-F1 scores of 61.9 and 74.7. This is because Macro-F1 averages across highly-imbalanced categories, which perceivably deflates the scores. The comparable agreement rate for the GPT-5.6-Sol Max judge is 86.5% and 82.3% for feasibility and progression respectively. In addition, human inter-annotator agreement is also excellent but not perfect at 0.820-0.869 Cohen’s kappa due to some annotator subjectivity. Table 1 compares the judge against a single annotator, whereas this audit uses the majority of three, which removes individual annotator noise, meaning that the 86.5/82.3% LLM Judge agreement is likely a deflated estimate of the true agreement compared to the “denoised” ground truth human annotation, which should be around 99/96%.

AuthorsOpenReview ↗

Author Response to Reviewer 9n7X (Part 2 of 2)

The annotation guidelines and judge prompt impose a strict sequential ordering of subgoals, including restrictions on returning to earlier subgoals (Appendices D and F). This may penalize legitimate backtracking, repairs, or alternative execution orders. Selecting sequentially dependent tasks partially addresses this concern, but does not establish that all valid trajectories follow the prescribed order.

Clarification: How does the evaluation handle valid trajectories that revisit an earlier subgoal or complete subgoals in a different order? Can you provide evidence that the ordering restrictions do not incorrectly penalize such trajectories?

We first describe our motivation behind establishing a monotonic targeting rule and then perform experiments to evidence that the valid legitimate backtracking, repairs, or alternative execution orders only contribute a small difference (<4%) in overall task success rates.

Motivation: In early stages of our data annotation, we considered and experimented with different protocols for annotating subgoals targeted. Our initial protocol allowed annotators to select subgoals targeted at each step in an arbitrary manner to account for such behavior. Under that protocol, inter-annotator agreement on the targeted subgoal was unacceptably low, because "targeted" is subjective (e.g. acting on a subgoal vs. merely mentioning it in the reasoning). To make the human annotation more reliable, we updated the protocol asking annotators to only keep tasks that could be addressed in a singular sequence of dependent subgoals. When the protocol is correctly adhered to, tasks that are kept after filtering cannot have subgoals tackled in another sequence or recover from an uncompleted earlier subgoal after finishing a later subgoal.

Table B comparing proportion of tasks with all subgoals completed with original step-level prompt (from Table 2) and new task-level prompt and their difference

ModelStep-level Prompt (%)Task-level Prompt (%)Difference (%)
Claude Opus 5 Max75.777.0+1.3
Claude Opus 4.8 Max77.777.4-0.3
Claude Opus 4.7 Max76.780.3+3.6
Claude Sonnet 5 Max76.477.7+1.3
GPT-5.6-Sol Max74.877.9+3.1
GPT-5.6-Terra Max70.873.9+3.1
GPT-5.6-Luna Max74.472.0-2.4
Gemini-3.8-Flash High59.059.3+0.3
Kimi K3 Max39.342.6+3.3
Minimax M3 Xhigh28.925.6-3.3
Qwen 3.8 Flash Next Xhigh55.159.0+3.9
Qwen 3.5 122B Xhigh16.719.0+2.3
Qwen 3.8 27B Xhigh32.133.1+1.0
Qwen 3.6 27B Xhigh10.28.5-1.7

Experiments: We acknowledge that some annotation errors may exist, as expected of any human annotated dataset. Some tasks that admit an alternative order may have accidentally passed this filter. To assess their frequency, we conducted an experiment. Specifically, we modify the prompt to LLM-Judge to the below prompt so that we can understand the extent to which the computer-use agent completes each subgoal at a task level across all steps and therefore, whether all subgoals in a task are completed. This way, subgoals completed through legitimate backtracking, repairs, or alternative execution orders can be considered. Then, we compare it with the corresponding overall task completion score from Table 2. Models only show a small difference (below 4%), suggesting that there are only a tiny minority of tasks that allows models to address subgoals in an alternate order.

Step 0. Reasoning: Action: Step 1. Reasoning: Action: ... Step k. Reasoning: Action: Which of the following subgoals are completed by the attached screenshots (one for each step in sequential order), reasoning, and action? Subgoals: 0. 1. ... n. Return only a nested JSON response e.g {“completed": [0, 1, 2]}.

In addition, we carried out a human-audit experiment on binary task success on model trajectories from Table 2. Specifically, we chose two representative models’ trajectories (Claude Opus 5 and Qwen 3.8 Flash Next - which are among top closed-source and open-weight models respectively) and for each model, sampled 15 near-miss trajectories (i.e. close to completing all feasible subgoals) and 16 successful trajectories. Then, we ask three human annotators to indicate if the trajectories show that the task has been fully completed across all feasible subgoals. Comparing the majority vote across 3 human annotators (Cohen’s kappa=0.798, substantial agreement), the LLM-judge’s prediction agrees on 90.3% of Claude Opus 5 ratings and 93.5% of Qwen 3.8 Flash Next ratings.

Together, evidence from these two experiments suggests that while legitimate backtracking, repairs, or alternative execution orders might be possible for a tiny minority of tasks, they do not penalize such attempts for the majority of tasks, resulting in minimal difference on the overall task success rates reported in the paper.

Thank you for your response. I will maintain my positive rating.

Reviewer_EZvpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OSWorld-Pro evaluates computer-use agents through subgoal-level process annotations rather than only final artifacts. It comprises hundreds of tasks, thousands of subgoals, and 60k human labels across diverse applications and Linux environments. The authors validate trajectory-level LLM judging against human annotations, benchmark fourteen agents, and analyze action-specific progress and persistence on unsuccessful or infeasible subgoals. The proposed contribution is a benchmark and diagnostic evaluation framework, rather than a new agent-training method.

Critical strengths
  • The annotation effort is unusually thorough. Independent reviewer passes, subsequent reconciliation, explicit feasibility/progression/completion definitions, and reported pre-feedback agreement provide meaningful quality controls. The extension to 31 applications and multiple desktop environments also offers useful coverage beyond a single default desktop configuration.

  • The evaluation separates task completion from intermediate progress and wasted effort. The examples of incorrect clicks and prolonged irrelevant actions illustrate concrete diagnostic benefits. Reporting judge quality at step, subgoal, and fractional-task levels, together with reasoning-cost tradeoffs, is more informative than assuming an LLM judge is reliable.

Critical weaknesses
  • The evaluation may penalize valid trajectories. Appendix D.6.1 imposes monotonic subgoal targeting, and Appendix F explicitly forbids returning to earlier subgoals. Yet some tasks do not specify a unique order: the Kid3 example requests both album and year metadata without requiring one to be edited first. Recovery may also require revisiting earlier work. The paper does not establish that these restrictions preserve scores for valid alternative solutions. Agreement among annotators following the same rule cannot resolve this concern.

  • Judge validation does not directly cover the headline metric. Table 1 measures task-level agreement using 1-MAE of fractional subgoal completion, whereas Table 2 requires completion of every feasible subgoal for binary task success. High agreement on the former does not guarantee accurate binary outcomes or rankings. Step-level F1 is also conditioned on correct prerequisite predictions, so it does not capture the full error cascade. A human audit of binary task outcomes would make the benchmark's reliability more convincing.

Questions for authors
  1. How are valid reordered actions and recovery handled despite the monotonic targeting rule?
  2. What is the judge's agreement with humans on binary task success and model rankings, rather than fractional completion alone?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer EZvp (Part 1 of 2)

We thank the reviewer for recognizing our work as unusually thorough and informative, with a diverse range of applications and desktop environments as well as step-level analysis of LLM-Judge beyond task success.

The evaluation may penalize valid trajectories. Appendix D.6.1 imposes monotonic subgoal targeting, and Appendix F explicitly forbids returning to earlier subgoals. Yet some tasks do not specify a unique order: the Kid3 example requests both album and year metadata without requiring one to be edited first. Recovery may also require revisiting earlier work. The paper does not establish that these restrictions preserve scores for valid alternative solutions. Agreement among annotators following the same rule cannot resolve this concern.

How are valid reordered actions and recovery handled despite the monotonic targeting rule?

We first describe our motivation behind establishing a monotonic targeting rule and then perform experiments to evidence that the valid reordered actions/recovery behavior only contribute a small difference (<4%) in overall task success rates.

Motivation: In early stages of our data annotation, we considered and experimented with different protocols for annotating subgoals targeted. Our initial protocol allowed annotators to select subgoals targeted at each step in an arbitrary manner to account for such behavior. Under that protocol, inter-annotator agreement on the targeted subgoal was unacceptably low, because "targeted" is subjective (e.g. acting on a subgoal vs. merely mentioning it in the reasoning). To make the human annotation more reliable, we updated the protocol asking annotators to only keep tasks that could be addressed in a singular sequence of monotonic subgoals. When the protocol is correctly adhered to, tasks that are kept after filtering cannot have subgoals tackled in another sequence or recover from an uncompleted earlier subgoal after finishing a later subgoal.

Table A comparing proportion of tasks with all subgoals completed with original step-level prompt (from Table 2) and new task-level prompt and their difference

ModelStep-level Prompt (%)Task-level Prompt (%)Difference (%)
Claude Opus 5 Max75.777.0+1.3
Claude Opus 4.8 Max77.777.4-0.3
Claude Opus 4.7 Max76.780.3+3.6
Claude Sonnet 5 Max76.477.7+1.3
GPT-5.6-Sol Max74.877.9+3.1
GPT-5.6-Terra Max70.873.9+3.1
GPT-5.6-Luna Max74.472.0-2.4
Gemini-3.8-Flash High59.059.3+0.3
Kimi K3 Max39.342.6+3.3
Minimax M3 Xhigh28.925.6-3.3
Qwen 3.8 Flash Next Xhigh55.159.0+3.9
Qwen 3.5 122B Xhigh16.719.0+2.3
Qwen 3.8 27B Xhigh32.133.1+1.0
Qwen 3.6 27B Xhigh10.28.5-1.7

Experiments: We acknowledge that some annotation errors may exist, as expected of any human annotated dataset. Some tasks that admit an alternative order may have accidentally passed this filter - the Kid3 example the reviewer identifies is one. To assess their frequency, we conducted an experiment. Specifically, we modify the prompt to LLM-Judge to the below prompt so that we can understand the extent to which the computer-use agent completes each subgoal at a task level across all steps and therefore, whether all subgoals in a task are completed. This way, valid reordered subgoals and subgoals completed through recovery approaches are both considered. Then, we compare it with the corresponding overall task completion score from Table 2. Models only show a small difference (below 4%), suggesting that there are only a tiny minority of tasks that allows models to address subgoals in an alternate order.

Step 0. Reasoning: Action: Step 1. Reasoning: Action: ... Step k. Reasoning: Action: Which of the following subgoals are completed by the attached screenshots (one for each step in sequential order), reasoning, and action? Subgoals: 0. 1. ... n. Return only a nested JSON response e.g {“completed": [0, 1, 2]}.
AuthorsOpenReview ↗

Author Response to Reviewer EZvp (Part 2 of 2)

Judge validation does not directly cover the headline metric. Table 1 measures task-level agreement using 1-MAE of fractional subgoal completion, whereas Table 2 requires completion of every feasible subgoal for binary task success. High agreement on the former does not guarantee accurate binary outcomes or rankings. Step-level F1 is also conditioned on correct prerequisite predictions, so it does not capture the full error cascade. A human audit of binary task outcomes would make the benchmark's reliability more convincing.

What is the judge's agreement with humans on binary task success and model rankings, rather than fractional completion alone?

We agree with the recommendation and carried out a human-audit experiment on binary task success. Specifically, we chose two representative models’ trajectories (Claude Opus 5 and Qwen 3.8 Flash Next - which are among top closed-source and open-weight models respectively) and for each model, sampled 15 near-miss trajectories (i.e. close to completing all feasible subgoals) and 16 successful trajectories. This represents a 10% sub-sample of the trajectories for each model, which is the limit for what human annotators can feasibly annotate during this discussion period without compromising on quality. Then, we ask three human annotators to indicate if the trajectories show that the task has been fully completed across all feasible subgoals.

Comparing the majority vote across 3 human annotators (Cohen’s kappa=0.798, substantial agreement), the LLM-judge’s prediction agrees on 90.3% of Claude Opus 5 ratings and 93.5% of Qwen 3.8 Flash Next ratings. In addition, we also tabulate the binary agreement for tasks in Table 1 to derive a 80.7% binary agreement rate for the GPT-5.6-Sol Max judge that we later use for tabulating Table 2, versus the 91.4% binary agreement rate for Human Performance. Together, these results indicate that judge agreement on the headline metric is close to human–human agreement for the Table 2 agents.

We hope our response has alleviated the initial concerns of the reviewer and we would appreciate it for the reviewer to revisit their assessment of our paper if it has. Otherwise, we will be happy to respond to any follow up questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment from Authors

As we are approaching the end of the discussion period, we want to take the opportunity to show our appreciation for the reviewers’ detailed reviews and summarize the discussion thus far.

Reviewers generally recognize the strengths of our work in terms of:

  • Substantial and thorough human annotations spanning > 5000 person-hours spent across >60k human labels and validated with strong independent review (Reviewers EZvp, 9n7X, S1Vv)
  • Diverse coverage of multiple (>30) applications, Linux distributions, and desktop interfaces beyond the singular desktop environment that existing evaluation emphasize on. (Reviewers EZvp, 9n7X)
  • Insightful diagnostic framework highlighting intermediate progress, inefficient behavior and failure modes beyond final task success that existing benchmarks solely focus on. (Reviewers EZvp, 9n7X, S1Vv)

As we spearhead a fresh direction in the process-based evaluation of computer-use agents (beyond outcomes alone), there were also teething issues in our initial submission that have now been addressed in our updated submission:

  • Incomplete evaluation of LLM-Judge on end-to-end task success. (Reviewers EZvp, S1Vv).
    • This is now included as a new column within Table 1 and congruent with our earlier reported trend in partial task completion (1 - Mean Absolute Error).
  • Possibility for model trajectories to be penalized due to alternate valid orders for approaching and revisiting subgoals. (Reviewers EZvp, 9n7X, S1Vv)
    • We clarified that our annotation protocol sought to avoid such possibility (by filtering out tasks with such propensities) but the imperfect adherence by human annotators have led to a small proportion of such tasks to be accidentally included, leading to a minor penalty (<4%) observed through a new experiment, detailed in Section 6.
  • Further human audit of LLM-Judge on feasibility, progression and success. (Reviewers EZvp, 9n7X, S1Vv)
    • We found that the LLM-Judge very closely approximates human annotators in terms of accuracy as detailed in two additional human validation studies in Section 6.

Finally, we would like to encourage the reviewers to continue their engagement with our responses, pose final questions where helpful and reconsider their scoring in light of new evidence that has surfaced during this discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.