- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
OSWorld 2.0 is a 108-task benchmark for long-horizon computer-use agents. It contributes tasks closer to real-world long-horizon computer work. Tasks have a reported median human duration of about 1.6 hours, span stateful web services and desktop applications. The environment includes 31 self-hosted websites, realistic artifacts and user state, a bounded simulated user, final-state functional evaluation, limited model-based judging, and explicit safety diagnostics.
Critical strengths
- The benchmark includes long-horizon tasks that reflect real-world end-to-end computer workflows that go beyond simple GUI operations.
- Detailed analysis on agent behavior and failure mode.
Critical weaknesses
- The benchmark contains only 108 tasks, which is insufficient to reduce statistical variance and ensure reliable evaluation results.
Questions for authors
- Why it's necessary to provide the agent internet access? It may allow agents to bypass intended GUI interactions.
- How is evaluation randomness measured, and how much variance is observed across runs?
Thank you for recognizing the realism of the tasks and the analysis of agent behavior. We respond to the two questions below:
Why is Internet access needed? Is bypassing the GUI allowed?
Some tasks require real online services or external resources. For example, Task 017 requires planning a walking route with multiple waypoints on Google Maps; Task 097 requires retrieving missing papers from arXiv, organizing them in Zotero, and completing the references. Our count shows that 82/108 tasks can be evaluated with public Internet access disabled while retaining access to self-hosted services and local resources; the remaining 26 tasks involved observed online information retrieval or publicly hosted resources required for setup.
The main experiments provide only the computer-use tool, without separate Bash, code-execution, or API tools. However, agents are allowed to open a terminal through the GUI to enter commands, execute code, or call APIs, so these actions do not themselves violate the evaluation protocol. We will explicitly clarify this configuration.
How is evaluation randomness measured? How much variability is observed across runs?
We conducted three independent evaluations of Claude Opus 5 on v2026.06.24 using the batched-action configuration in Table 2. The Binary and Partial scores were 26.10 ± 1.52% and 66.01 ± 1.23%, respectively, where ± denotes the standard deviation across runs. Due to evaluation costs, the results for the other models are currently based on single runs, which we will explicitly indicate.