acceptodds
Under review as a conference paper at ICLR 2027

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

34 views

Abstract

Existing computer-use benchmarks fail to capture the realism, complexity, and long-horizon demands of real-world computer use, limiting their ability to reveal the limitations of frontier agents. We introduce OSWorld 2.0, a benchmark of 108 long-horizon computer-use workflows across everyday and professional tasks, designed to capture complex and challenging real-world phenomena. Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete and requires an average of 435 actions with Claude Opus 5 using maximum thinking, compared with about 30 in OSWorld 1.0. OSWorld 2.0 targets challenge phenomena that are common in real workflows yet underrepresented in prior benchmarks, spanning interaction-design challenges such as streaming interaction, proactive interaction and dynamic environments, as well as agent-pattern challenges such as cross-source reasoning, implicit-state inference, and visual-spatial precision. Tasks are grounded in authentic input artifacts and cross-referenced against realistic stateful user profile data, and include separate safety reports auditing safety-sensitive execution. Under our primary binary-completion metric at 500 steps, Claude Opus 5 with maximum thinking and batched tool calls scores best but still completes only 31.43% of tasks at a 68.31% partial score; GPT-5.6 Sol is far more token-efficient yet plateaus near 27.34%. These results show that current agents are still far from professional-level computer use: rather than stumbling on basic GUI control or coding, they lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification, struggling most when a task hinges on hidden state they must recover.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Computer Use, Long-horizon Tasks, Agents

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_cNeZOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

OSWorld 2.0 is a 108-task benchmark for long-horizon computer-use agents. It contributes tasks closer to real-world long-horizon computer work. Tasks have a reported median human duration of about 1.6 hours, span stateful web services and desktop applications. The environment includes 31 self-hosted websites, realistic artifacts and user state, a bounded simulated user, final-state functional evaluation, limited model-based judging, and explicit safety diagnostics.

Critical strengths
  • The benchmark includes long-horizon tasks that reflect real-world end-to-end computer workflows that go beyond simple GUI operations.
  • Detailed analysis on agent behavior and failure mode.
Critical weaknesses
  • The benchmark contains only 108 tasks, which is insufficient to reduce statistical variance and ensure reliable evaluation results.
Questions for authors
  • Why it's necessary to provide the agent internet access? It may allow agents to bypass intended GUI interactions.
  • How is evaluation randomness measured, and how much variance is observed across runs?
1 comment

Thank you for recognizing the realism of the tasks and the analysis of agent behavior. We respond to the two questions below:

  1. Why is Internet access needed? Is bypassing the GUI allowed?

    Some tasks require real online services or external resources. For example, Task 017 requires planning a walking route with multiple waypoints on Google Maps; Task 097 requires retrieving missing papers from arXiv, organizing them in Zotero, and completing the references. Our count shows that 82/108 tasks can be evaluated with public Internet access disabled while retaining access to self-hosted services and local resources; the remaining 26 tasks involved observed online information retrieval or publicly hosted resources required for setup.

    The main experiments provide only the computer-use tool, without separate Bash, code-execution, or API tools. However, agents are allowed to open a terminal through the GUI to enter commands, execute code, or call APIs, so these actions do not themselves violate the evaluation protocol. We will explicitly clarify this configuration.

  2. How is evaluation randomness measured? How much variability is observed across runs?

    We conducted three independent evaluations of Claude Opus 5 on v2026.06.24 using the batched-action configuration in Table 2. The Binary and Partial scores were 26.10 ± 1.52% and 66.01 ± 1.23%, respectively, where ± denotes the standard deviation across runs. Due to evaluation costs, the results for the other models are currently based on single runs, which we will explicitly indicate.

Reviewer_qCuHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OSWorld 2.0 introduces 108 extended desktop workflows spanning professional and everyday taks with controlled web services, dynamic updates, and simulated user interaction. Final-state checkpoints provide partial credit alongside strict completion, while separate audits identify unsafe side effects and interface bypasses. The v2026.08.08 evaluation reports 31.43% completion and 68.31% partial credit for Clude Opus 5, versus 27.34% and 62.72% for GPT5.6 Sol. Trajectory analyses attribute failures to state tracking, changing information, incomplete verification.

Critical strengths
  1. The benchmark combines substantial workflows, cross-application evidence, dynamic updates, and user clarification. These features test sustained coordination that short, isolated tasks miss, giving the resource clear evaluation value.

  2. Dense final-state checkpoints allow alternative solution paths and meaningful partial credit. Functional checks, model-judge validation, and separate safety audits provide a thoughtful foundation for evaluating both progress and execution quality.

Critical weaknesses
  1. The impact of evaluator/environment issues on reported completion needs clarification. Table 17 labels possible mismatches in 30.6–42.6% of the audited older-model trajectories; these are broad diagnostic flags, not confirmed scoring-error rates. Table 21 identifies a concrete canonicalization error on Task 024. Which issues were corrected in each release? It's better to quantify their effect on completion.

  2. Table 2 combines two benchmark releases. The paper provides no matched evaluation bridging releases, and the headline multi-run estimates lack uncertainty intervals. Comparisons across generations therefore need qualification, even though the two newest models share v2026.08.08, I guess.

Questions for authors
  1. Request for evidence: Which Table 17 mismatch flags were confirmed evaluator/environment errors, and which were fixed in each release? What is their impact on completion scores, including the Task 024 canonicalization issue?

  2. Clarification/request for evidence: What changed between v2026.06.24 and v2026.08.08? Can cross-generation comparisons use a shared release, and can the headline multi-run results include dispersion or confidence intervals?

  3. Clarification: Appendix H.4.2 reports a 1.0 task score for an interface bypass, while Section 2.2 describes anti-bypass checks. Given that safety reports are intentionally separate, which prohibited shortcuts invalidate task credit, and which are only audited?

3 comments
Reviewer_qCuHOpenReview ↗

Authors should engage in the Rebuttal - My questions are not resolved

I have posted genuine questions in my review, and I expect the authors to respect the time and the efforts of the reviewers to try to address their concerns and questions. My original score (3: Weak acceptance) means I like the paper, but my questions are also important, and if they are not addressed, in addition to the other reviewers' questions that are related to mine, I will lower my score. This is because the paper has some important inconsistencies in its report that both I and the reviewer nAzR have mentioned. Those need to be addressed before the paper is suitable for acceptance.

Thank you for your thoughtful feedback and for highlighting these issues. We provide clarifications below to address your concerns and will revise the paper accordingly.

Q1

  1. scoring/environment mismatch in Table 17 is a broad term, and 30.6%-42.6% is not the grader error rate. It includes agent-caused environment mismatches that agents failing to save or submit, leaving the state visible in screenshots inconsistent with the state persisted on disk or in the database. These are the agent's own errors, we will label this subcategory Perceived-recorded state divergence. For grader errors, we identified this error rate here:
Behavior labelMiniMax M3Claude Sonnet 4.6GPT-5.5Claude Opus 4.7
Identified grader-related under-scoring5/108 (4.6%)4/108 (3.7%)4/108 (3.7%)5/108 (4.6%)

The table reports the number of tasks involved. Since only a few checkpoints are involved in each task, the actual impact on scores is even smaller.

  1. We address grader issues through release updates. The fixing notes below cover grading corrections and robustness improvements. R = robustness improvement; G = grading fix.
Task IDRecorded fixType
004Normalize acceptable citation URLs.R
008Refine the attachment-evaluation prompt.R
009Relax abstract/author punctuation matching; correct author roles and affiliations.G
014Accept quotation punctuation variants; prevent deleting the essay to retain only answers; append optional evidence.R
016Accept valid email variants; add missing ICML papers/authors and update assets.G
025Remove ambiguity from the task PDF.R
027Specify population standard deviation for Open_STD.R
033Prevent simulation-log/success-state manipulation and bypassing keyboard operations.R
038Clarify the task JSON.R
039Address answer leakage through gt.json.R
041Remove a faulty 0.02-point checkpoint; clarify the instruction.G
049Use position rather than curve detection; remove an unstated page-2 check; retain per-page scoring.G
052Address JavaScript-based pop-up/UI bypasses.R
055Match tracks without fixed V1/V2/V3/V4 names; prevent reference-file bypasses.R
056Prevent submitting the reference file for credit.R
057Fix content extraction; install full TeX Live in self-hosted Overleaf.G + R
059Address image-processing shortcuts that violate GeoGebra/manual-point restrictions.R
061Use a darktable reference producible from the supplied JPG, rather than RAW/DNG.G
063Add rendering-based topology/occlusion checks and support valid bottom-only cropping.R
064Specify demonstration_data/tmp; prevent direct state setting that bypasses motion planning.R
065Address the reported task-information leakage/bypass.R
068Address game-script injection and DevTools/CDP state manipulation.R
069Fix the VM time zone to UTC.R
071Accept lyric punctuation variants.R
072Recognize valid analysis-section structures.G
074Prevent substituting a reference screenshot for a webpage implementation.R
079Prevent substituting reference screenshots for PPT content.R
080Remove hidden style/name requirements; check visible formatting, resolved chart ranges, and normalized Mid-Market capitalization.G
081Move the unstable local website online.R
082Remove the redundant password-reset requirement; require completed repairs.G
083Fix PDF filename matching; reject blank/fabricated PDFs through content checks.G + R
084Replace hidden JSFX/HTML/audio similarity checks with public behavioral criteria.G
085Replace hidden-waveform MFCC/DTW checks with public criteria; specify a ~0.05-second crossfade.G
086Remove leaked evaluation scripts from other tasks.R
088Resolve VBA/ActiveX evaluation incompatibilities after migration to Linux.G
089Check rendered page geometry instead of source strings.G
090Fix an evaluator bug and clarify the instruction.G
091Add pixel/boundary tolerances, UNKNOWN_NEUTRAL for missing reliable screenshots, and page markers to reject wrong/duplicate captures.G + R
092Check five frames (0.25, 1.25, 2.50, 3.75, 4.75 s); any passing frame earns 0.20 points.G
093Explicitly require annotated OpenBoard output as .ubz on the Desktop.G
101Address the reported bottle-rotation bypass.R

The releases also improve task design and stability. A task change does not itself establish a grading error; actual score impact depends on the checkpoints reached, their weights, and any task-wide gate.

Q2

  1. Both releases use the same underlying AWS AMI and Docker base image. Changes mainly concern tasks and graders, as listed above, with additional loading/setup/configuration changes.

  2. We additionally evaluated Opus 5 and GPT-5.6 Sol on v2026.06.24 to compare models on a shared release. Binary and Partial are percentages. The configuration used for the table below is the same as that of the batched-action version in Table 2.

ModelBinaryPartial
Claude Opus 526.10 ± 1.5266.01 ± 1.23
GPT-5.6 Sol21.5660.84
Claude Opus 4.820.654.8
Claude Opus 4.718.248.9
GPT-5.513.049.5

Given the budget, we ran Opus 5 three times independently and report mean ± standard deviation. The other entries are single-run results.

Q3

The anti-cheating checks in §2.2 target fabricated states, access to answers, or invalid deliverables, such as substituting static images for editable PPT objects in Task 058. These receive deductions or zero according to the rubric.

Bypassing the UI in Task 052 is a shortcut to achieving the real target state, rather than fabricating a false result: the agent uses the logged-in browser session to call the website's actual backend APIs, and the reservation state is created successfully in the server-side database. Therefore, awarding 1.0 points for the main task is reasonable.

Task scores and safety audits are separate to expose successful outcomes with unsafe execution, such as credential leakage in Task 026. Shortcuts that bypass the target challenge are Untested under the exposure-attribution rules (e.g., GPT-5.5 on Task 068), rather than evidence that the challenge was handled.

Reviewer_nAzROpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OSWorld 2.0, a benchmark of 108 realistic, long-horizon computer-use tasks involving cross-application coordination, dynamic updates, and user interaction.

The evaluation combines fine-grained checkpoints, full-completion metrics, and separate safety audits. Experiments show that the strongest evaluated configuration achieves only 31.43% full completion despite substantial partial progress.

The analysis highlights failures in maintaining constraints, tracking state, incorporating information, and verifying outcomes.

Overall, the paper presents a well-motivated benchmark and informative analyses, emphasizing state management and self-correction as important directions for improving the reliability of computer-use agents.

Critical strengths
  1. The benchmark addressed an important gap between short, isolated computer-use tasks and realistic workflows requiring sustained coordination across applicaitions.

  2. Stateful environments, fine-grained checkpoints, and validation through human completion and agent rollouts provide a strong foundation for evaluating both partial progress and full task completion.

  3. The analysis goes beyond aggregate scores to examine state tracking, dynamic updates, verification, and safety, offering useful insights into the limitations of current agents.

Critical weaknesses

No major weaknesses are found. A few questions about experiment consistency are listed below.

Questions for authors
  • Since Table 17 flags scoring/environment mismatches in 30.6%–42.6% of trajectories, the impact of evaluator and environment issues on the reported results is insufficiently addressed.
  • The consistency of the experimental results need clarification. Figure 7 and Table 18 report substantially different strategy distributions. For example, 14% versus 47.2% hybrid trajectories for Opus 4.7.
1 comment

Thank you for your positive assessment of our work and for carefully checking the consistency of the experimental results. We address your two concerns below.

Q1: Impact of evaluator and environment issues.

The scoring/environment mismatch label in Table 17 is a broad diagnostic category; the reported 30.6%–42.6% does not represent the grader error rate. It includes cases where agents failed to save or submit their work, leaving the state visible in screenshots inconsistent with the state persisted on disk or in the database. These are agent execution errors, and we will explicitly label this subcategory Perceived-recorded state divergence in the revision.

The identified grader-related under-scoring is summarized below:

Behavior labelMiniMax M3Claude Sonnet 4.6GPT-5.5Claude Opus 4.7
Identified grader-related under-scoring5/108 (4.6%)4/108 (3.7%)4/108 (3.7%)5/108 (4.6%)

These percentages indicate the proportion of tasks affected, rather than the magnitude of the reduction in aggregate scores. We address grading issues through release updates and will clarify these distinctions and document the relevant fixes in the revised paper.

Q2: Inconsistent strategy distributions in Figure 7 and Table 18.

Thank you for pointing out this inconsistency. Appendix Table 18 inadvertently retained strategy statistics from an earlier version of the analysis and was not updated alongside Figure 7. This accounts for the discrepancy between the 47.2% and 14% hybrid proportions reported for Claude Opus 4.7. The updated statistics in Figure 7 should be used. We will update the appendix table and associated text to ensure consistency throughout the paper. We apologize for this oversight and the resulting confusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.