- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents OpenComputer, a benchmark and construction infrastructure covering 33 real desktop applications and 1,000 tasks. The system automatically generates app-specific state verifiers, diagnoses and repairs verifier-side errors using execution-grounded feedback, and then synthesizes task descriptions, environment-initialization procedures, and machine-checkable success criteria. Evaluation computes task success rates and partial-credit rewards from the final application state. The authors evaluate multiple computer-use agents, analyze verifier reliability through human adjudication and ablation experiments, and compare GUI and CLI agents on a shared task subset. OpenComputer is positioned as extensible infrastructure for evaluating computer-use agents, collecting trajectories, and supporting training.
Critical strengths
-
The benchmark provides a desktop evaluation resource with practical infrastructure value. The paper covers 33 real desktop applications and 1,000 tasks, together with app-specific verifiers, input artifacts, and executable environment-initialization procedures. Integrating the construction of tasks, environments, and verification logic into an automated pipeline addresses the practical burden of preparing desktop environments and implementing application-specific evaluation logic. It also provides a reusable foundation for adding applications and tasks.
-
A human-annotated audit set held out from verifier evolution provides empirical support for verifier refinement. The paper reports that task-level human-checker agreement improves from 85.0% to 94.1% after self-evolution. Together with the concrete repair of a database-query bug in darktable described in the appendix, these results support the use of execution-grounded feedback to identify and repair actual defects in automatically generated verifiers.
Critical weaknesses
-
The benchmark's incremental evaluation value is insufficiently characterized. The paper emphasizes application coverage and task count, but provides limited systematic analysis of the finalized task suite. Beyond selected examples, the distribution of tasks across application feature areas, workflow types, and complexity levels remains unclear. The paper also does not sufficiently explain which important evaluation gaps these tasks address relative to existing benchmarks based on real desktop applications. Results are mostly aggregated, making it difficult to connect model performance to the expanded task coverage. Consequently, the paper does not sufficiently establish which additional capabilities or limitations this benchmark reveals, or why these findings are important for evaluation.
-
Agreement between verifiers and human judgments does not establish alignment with the complete user goal. The paper provides encouraging evidence for the reliability of programmatic verification, but the completeness of the automatically generated success criteria remains insufficiently validated. For example, the Blender task in the appendix specifies a cube, a UV sphere, and a cylinder, whereas the illustrated mesh object type checks alone cannot establish that the geometry is correct. Similarly, the MuseScore task specifies particular pitches and rhythms, which checks such as note counts cannot verify on their own. The paper does not adequately explain how coverage of all task requirements is validated. Moreover, the two automated evaluators in Section 5.1 use the same per-item checklist, while the human annotation protocol is insufficiently specified. If human annotators also primarily reviewed that checklist, high agreement could still conceal shared omissions of task requirements. This directly affects whether the evaluation scores accurately reflect completion of the user goal.
Questions for authors
-
For the 1,000 finalized tasks, how are the generated initial environments checked for consistency with the task specifications? Have these tasks been validated by executing reference solutions or otherwise checking that the target state is reachable?
-
Please provide further details on the agent harness, observation settings, action interface, and execution budget used for each model in Table 2, and explain how differences in the number of low-level actions per model step are accounted for in the budgets and efficiency comparisons. For the external OSWorld-Verified scores reported in Table 2, to what extent do differences in evaluation settings limit the conclusion about cross-benchmark generalization?
-
Section 3.5 states that OpenComputer has been released, and the conclusion also refers to "the released repository." However, the submitted paper provides neither a link to the project resources nor instructions for anonymous access. Could the authors provide an anonymous release corresponding to the reported experiments, including the task suite, environment-initialization artifacts, verifiers, and construction and evaluation configurations?
-
In the MuseScore and Blender examples in Appendix C (PDF pages 17 and 18), some file paths extend beyond the right edges of the text boxes and into the page margins. Please add line breaks.
3 comments
Thank you for the detailed response. I see the automated construction framework as the main contribution.
Table 9 reports that 356 proposals were filtered out for insufficient complexity, and Appendix E.2 describes the role of human review. Could you clarify how much human input was typically needed to finalize the tasks for an application, and how often review led to changes or reruns?
I am particularly interested in whether this effort differed for the more complex tasks. Based on your existing complexity criteria and evaluation results, could you characterize the more complex tasks that made it into the released suite and explain whether they typically required more human review or verifier extensions? A concrete example from the released suite would help illustrate what the pipeline can produce with limited human input.
Reply to reviewer ncWK
Thank you for the follow-up question. We would like to clarify that all 1,000 tasks in the final benchmark were generated automatically, without manual intervention on individual tasks.
Human review was primarily conducted during the development of the generation pipeline. For example, we identified the need for certain application-level configurations, such as enabling autosave, to ensure that verifiers evaluate the latest application state rather than outdated or unsaved changes. Based on these observations, we refined the pipeline to address such issues rather than manually correcting individual tasks. Once the pipeline was finalized, all 1,000 tasks were generated and validated automatically. We did not track the exact number of such refinements, as they were part of the pipeline development process rather than the construction of individual benchmark tasks.
Regarding task complexity, more complex tasks did not require additional task-specific human review or manual verifier extensions. They were generated using the same automated pipeline as simpler tasks.
For example, the Obsidian: Recipe Vault Construction task in Appendix C requires creating three folders and five Markdown notes, including four recipes with YAML frontmatter, tags, and internal links, as well as an index linking to the recipes. The corresponding verifier checks multiple aspects of the final state, including folder structure, frontmatter fields, cross-note links, tags, and the exact number of notes. Importantly, our pipeline first generates the verification endpoints and then automatically constructs tasks and mapping each task requirement to the corresponding endpoints, without human intervention. More complex tasks may involve a greater number of verification checks, but they do not require additional human effort.
Reply to reviewer ncWK
Thank you for this comment. Regarding the analysis of the task suite, the full specifications of all 1,000 tasks are available at our repo, including task instructions, seed files, and criterion-level verifier checks. We will add a systematic analysis of the finalized suite. This will cover the task distribution across the 33 applications and 6 application categories. The complexity statistics including the number of checks per task (6.9 on average), seed files per task (1.3 on average).
We would also like to clarify the positioning of our work. The main contribution of OpenComputer is not a new benchmark. It is a scalable framework that automatically constructs, tests, and improves programmatic verifiers across heterogeneous real applications, and then uses the resulting verification interfaces to synthesize executable tasks and environments. The 1,000-task suite is one instantiation of this pipeline, and its role is to show that the pipeline produces tasks that are reliably verifiable and challenging for current agents. The evaluation gap we address is therefore one of scale and cost. Building real-application benchmarks with reliable verifiers has so far required heavy manual effort: OSWorld reports about 400 man-hours of checks and two months of work from a roughly ten-person team. Synthetic-environment works either avoid real applications (GUI-Genesis[1], InfiniteWeb[2]) or fall back to LLM judges (Gym-Anything[3], whose generated verifiers reach only about 43.3% accuracy). OpenComputer makes it possible to extend verifiable evaluation to new real applications without this manual cost, and the same verifiers also provide execution-grounded rewards for training.
[1] Yuan Cao et al. 2026. GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training. arXiv:2602.14093.
[2] Ziyun Zhang et al. 2026. InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training. ACL 2026.
[3] Pranjal Aggarwal, Graham Neubig, and Sean Welleck. 2026. Gym-Anything: Turn any Software into an Agent Environment. arXiv:2604.06126.
Thank you for this comment. In OpenComputer, task generation is grounded in the verifier. A candidate goal is accepted only if each of its requirements maps to a verifier endpoint that reads the corresponding application state, and goals with requirements that cannot be checked programmatically are filtered out. The appendix shows only an abbreviated subset of checks. The full Blender verifier also checks the geometry of each mesh, classifying it as a cube, UV sphere, or cylinder from its vertices and faces, and the MuseScore verifier checks the complete pitch and duration sequence. We will include the complete checklists in the revision. Regarding the annotation protocol, each trajectory was evaluated by at least three annotators from a pool of five, all with practical experience in the corresponding application, through a interface showing the task instruction, the full trajectory, final state and the checklist.
Thank you for this question. Consistency is checked at two points. During generation, candidate tasks whose input artifacts cannot be instantiated coherently with the task description are filtered out. After generation, each task is initialized in a fresh sandbox, and we run the verifier on the untouched initial state; all 1,000 tasks fail this check, confirming that no task is trivially satisfied. For reachability, 914 of the 1,000 tasks were solved by at least one model in Table 2, which establishes empirically that their target states are reachable. We will describe this validation in the revision.
All models are run with the same agent implementation, prompt, action interface, and observation setting (1920×1080 screenshots) that OSWorld-Verified uses for each model, with a budget of 100 model steps. As in OSWorld, the budget counts model calls rather than low-level actions. Since the harness and action format match OSWorld, differences in evaluation settings cannot plausibly account for drops such as 52.3% → 5.7% for GUI-OWL-1.5-8B. The only exception is Gemini-3.7-Flash and GPT-5.6-Terra, which has no OSWorld-Verified score and is therefore excluded from this comparison.
We provide an anonymized repository at https://anonymous.4open.science/r/OpenComputer-03D3/.
Thank you for pointing this out. We will fix the line breaks in the revision.