acceptodds
Under review as a conference paper at ICLR 2027

OpenComputer: Verifiable Software Worlds for Computer-Use Agents

Abstract

We present OpenComputer, a verifier-grounded framework for constructing verifiable software worlds for computer-use agents. OpenComputer integrates four components: (1) app-specific state verifiers that expose structured inspection endpoints over real applications, (2) a self-evolving verification layer that improves verifier reliability using execution-grounded feedback, (3) a task-generation pipeline that synthesizes realistic and machine-checkable desktop tasks, and (4) an evaluation harness that records full trajectories and computes auditable partial-credit rewards. In its current form, OpenComputer covers 33 desktop applications and 1,000 finalized tasks spanning browsers, office tools, creative software, development environments, file managers, and communication applications. Experiments show that OpenComputer's hard-coded verifiers align more closely with human adjudication than LLM-as-judge evaluation, especially when success depends on fine-grained application state. Frontier agents struggle with end-to-end completion despite partial progress, and open-source models exhibit sharp drops from their OSWorld-Verified scores, exposing a persistent gap in robust computer automation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Computer-Use Agents, GUI Agents, Verifiable Environments, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ncWKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OpenComputer, a benchmark and construction infrastructure covering 33 real desktop applications and 1,000 tasks. The system automatically generates app-specific state verifiers, diagnoses and repairs verifier-side errors using execution-grounded feedback, and then synthesizes task descriptions, environment-initialization procedures, and machine-checkable success criteria. Evaluation computes task success rates and partial-credit rewards from the final application state. The authors evaluate multiple computer-use agents, analyze verifier reliability through human adjudication and ablation experiments, and compare GUI and CLI agents on a shared task subset. OpenComputer is positioned as extensible infrastructure for evaluating computer-use agents, collecting trajectories, and supporting training.

Critical strengths
  1. The benchmark provides a desktop evaluation resource with practical infrastructure value. The paper covers 33 real desktop applications and 1,000 tasks, together with app-specific verifiers, input artifacts, and executable environment-initialization procedures. Integrating the construction of tasks, environments, and verification logic into an automated pipeline addresses the practical burden of preparing desktop environments and implementing application-specific evaluation logic. It also provides a reusable foundation for adding applications and tasks.

  2. A human-annotated audit set held out from verifier evolution provides empirical support for verifier refinement. The paper reports that task-level human-checker agreement improves from 85.0% to 94.1% after self-evolution. Together with the concrete repair of a database-query bug in darktable described in the appendix, these results support the use of execution-grounded feedback to identify and repair actual defects in automatically generated verifiers.

Critical weaknesses
  1. The benchmark's incremental evaluation value is insufficiently characterized. The paper emphasizes application coverage and task count, but provides limited systematic analysis of the finalized task suite. Beyond selected examples, the distribution of tasks across application feature areas, workflow types, and complexity levels remains unclear. The paper also does not sufficiently explain which important evaluation gaps these tasks address relative to existing benchmarks based on real desktop applications. Results are mostly aggregated, making it difficult to connect model performance to the expanded task coverage. Consequently, the paper does not sufficiently establish which additional capabilities or limitations this benchmark reveals, or why these findings are important for evaluation.

  2. Agreement between verifiers and human judgments does not establish alignment with the complete user goal. The paper provides encouraging evidence for the reliability of programmatic verification, but the completeness of the automatically generated success criteria remains insufficiently validated. For example, the Blender task in the appendix specifies a cube, a UV sphere, and a cylinder, whereas the illustrated mesh object type checks alone cannot establish that the geometry is correct. Similarly, the MuseScore task specifies particular pitches and rhythms, which checks such as note counts cannot verify on their own. The paper does not adequately explain how coverage of all task requirements is validated. Moreover, the two automated evaluators in Section 5.1 use the same per-item checklist, while the human annotation protocol is insufficiently specified. If human annotators also primarily reviewed that checklist, high agreement could still conceal shared omissions of task requirements. This directly affects whether the evaluation scores accurately reflect completion of the user goal.

Questions for authors
  1. For the 1,000 finalized tasks, how are the generated initial environments checked for consistency with the task specifications? Have these tasks been validated by executing reference solutions or otherwise checking that the target state is reachable?

  2. Please provide further details on the agent harness, observation settings, action interface, and execution budget used for each model in Table 2, and explain how differences in the number of low-level actions per model step are accounted for in the budgets and efficiency comparisons. For the external OSWorld-Verified scores reported in Table 2, to what extent do differences in evaluation settings limit the conclusion about cross-benchmark generalization?

  3. Section 3.5 states that OpenComputer has been released, and the conclusion also refers to "the released repository." However, the submitted paper provides neither a link to the project resources nor instructions for anonymous access. Could the authors provide an anonymous release corresponding to the reported experiments, including the task suite, environment-initialization artifacts, verifiers, and construction and evaluation configurations?

  4. In the MuseScore and Blender examples in Appendix C (PDF pages 17 and 18), some file paths extend beyond the right edges of the text boxes and into the page margins. Please add line breaks.

3 comments
AuthorsOpenReview ↗

Reply to reviewer ncWK

W1: The benchmark's incremental evaluation value is insufficiently characterized.

Thank you for this comment. Regarding the analysis of the task suite, the full specifications of all 1,000 tasks are available at our repo, including task instructions, seed files, and criterion-level verifier checks. We will add a systematic analysis of the finalized suite. This will cover the task distribution across the 33 applications and 6 application categories. The complexity statistics including the number of checks per task (6.9 on average), seed files per task (1.3 on average).

We would also like to clarify the positioning of our work. The main contribution of OpenComputer is not a new benchmark. It is a scalable framework that automatically constructs, tests, and improves programmatic verifiers across heterogeneous real applications, and then uses the resulting verification interfaces to synthesize executable tasks and environments. The 1,000-task suite is one instantiation of this pipeline, and its role is to show that the pipeline produces tasks that are reliably verifiable and challenging for current agents. The evaluation gap we address is therefore one of scale and cost. Building real-application benchmarks with reliable verifiers has so far required heavy manual effort: OSWorld reports about 400 man-hours of checks and two months of work from a roughly ten-person team. Synthetic-environment works either avoid real applications (GUI-Genesis[1], InfiniteWeb[2]) or fall back to LLM judges (Gym-Anything[3], whose generated verifiers reach only about 43.3% accuracy). OpenComputer makes it possible to extend verifiable evaluation to new real applications without this manual cost, and the same verifiers also provide execution-grounded rewards for training.

[1] Yuan Cao et al. 2026. GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training. arXiv:2602.14093.

[2] Ziyun Zhang et al. 2026. InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training. ACL 2026.

[3] Pranjal Aggarwal, Graham Neubig, and Sean Welleck. 2026. Gym-Anything: Turn any Software into an Agent Environment. arXiv:2604.06126.

W2: Agreement between verifiers and human judgments does not establish alignment with the complete user goal.

Thank you for this comment. In OpenComputer, task generation is grounded in the verifier. A candidate goal is accepted only if each of its requirements maps to a verifier endpoint that reads the corresponding application state, and goals with requirements that cannot be checked programmatically are filtered out. The appendix shows only an abbreviated subset of checks. The full Blender verifier also checks the geometry of each mesh, classifying it as a cube, UV sphere, or cylinder from its vertices and faces, and the MuseScore verifier checks the complete pitch and duration sequence. We will include the complete checklists in the revision. Regarding the annotation protocol, each trajectory was evaluated by at least three annotators from a pool of five, all with practical experience in the corresponding application, through a interface showing the task instruction, the full trajectory, final state and the checklist.

Q1: How are the generated initial environments checked for consistency with the task specifications? Is the target state reachable?

Thank you for this question. Consistency is checked at two points. During generation, candidate tasks whose input artifacts cannot be instantiated coherently with the task description are filtered out. After generation, each task is initialized in a fresh sandbox, and we run the verifier on the untouched initial state; all 1,000 tasks fail this check, confirming that no task is trivially satisfied. For reachability, 914 of the 1,000 tasks were solved by at least one model in Table 2, which establishes empirically that their target states are reachable. We will describe this validation in the revision.

Q2: Details on harness, observation, action interface, and budget; low-level actions per step; OSWorld-Verified comparability.

All models are run with the same agent implementation, prompt, action interface, and observation setting (1920×1080 screenshots) that OSWorld-Verified uses for each model, with a budget of 100 model steps. As in OSWorld, the budget counts model calls rather than low-level actions. Since the harness and action format match OSWorld, differences in evaluation settings cannot plausibly account for drops such as 52.3% → 5.7% for GUI-OWL-1.5-8B. The only exception is Gemini-3.7-Flash and GPT-5.6-Terra, which has no OSWorld-Verified score and is therefore excluded from this comparison.

Q3: Anonymous release

We provide an anonymized repository at https://anonymous.4open.science/r/OpenComputer-03D3/.

Q4: File paths overflowing in Appendix C.

Thank you for pointing this out. We will fix the line breaks in the revision.

Thank you for the detailed response. I see the automated construction framework as the main contribution.

Table 9 reports that 356 proposals were filtered out for insufficient complexity, and Appendix E.2 describes the role of human review. Could you clarify how much human input was typically needed to finalize the tasks for an application, and how often review led to changes or reruns?

I am particularly interested in whether this effort differed for the more complex tasks. Based on your existing complexity criteria and evaluation results, could you characterize the more complex tasks that made it into the released suite and explain whether they typically required more human review or verifier extensions? A concrete example from the released suite would help illustrate what the pipeline can produce with limited human input.

AuthorsOpenReview ↗

Reply to reviewer ncWK

Thank you for the follow-up question. We would like to clarify that all 1,000 tasks in the final benchmark were generated automatically, without manual intervention on individual tasks.

Human review was primarily conducted during the development of the generation pipeline. For example, we identified the need for certain application-level configurations, such as enabling autosave, to ensure that verifiers evaluate the latest application state rather than outdated or unsaved changes. Based on these observations, we refined the pipeline to address such issues rather than manually correcting individual tasks. Once the pipeline was finalized, all 1,000 tasks were generated and validated automatically. We did not track the exact number of such refinements, as they were part of the pipeline development process rather than the construction of individual benchmark tasks.

Regarding task complexity, more complex tasks did not require additional task-specific human review or manual verifier extensions. They were generated using the same automated pipeline as simpler tasks.

For example, the Obsidian: Recipe Vault Construction task in Appendix C requires creating three folders and five Markdown notes, including four recipes with YAML frontmatter, tags, and internal links, as well as an index linking to the recipes. The corresponding verifier checks multiple aspects of the final state, including folder structure, frontmatter fields, cross-note links, tags, and the exact number of notes. Importantly, our pipeline first generates the verification endpoints and then automatically constructs tasks and mapping each task requirement to the corresponding endpoints, without human intervention. More complex tasks may involve a greater number of verification checks, but they do not require additional human effort.

Reviewer_pvQuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents OpenComputer, a pipeline in which LLM agents build verifiable computer-use tasks inside real desktop applications (Fig. 1, Sec. 3). App-specific Python verifiers read application state through native channels such as CDP, D-Bus, LibreOffice UNO and SQLite profile databases. A "self-evolving" layer runs a strong agent on calibration tasks, compares the verifier with a criterion-level LLM judgment, and repairs checker-side defects. A task generator proposes realistic goals, maps them to verifier endpoints, and synthesises the seed files, and a run counts as a success only when every check passes. The release covers 33 applications and 1,000 tasks (Table 1).

Eleven agents are evaluated (Table 2). The best reach about 76–77% success (Claude-Sonnet-5, GPT-5.6-Sol, Kimi-K3), while open-weight GUI models score far below their OSWorld-Verified numbers (e.g., GUI-OWL-1.5-8B: 5.7% vs. 52.3%). On a 120-task human-annotated audit set, the verifier agrees with the human task-level verdict on 113/120 tasks vs. 95/120 for a screenshot-based LLM judge (Fig. 3), and self-evolution raises human–checker agreement from 85.0% to 94.1% (Table 4). A GUI-vs-CLI comparison on 343 tasks is also reported (Table 3).

Critical strengths
  • The tasks are generated at scale in real desktop applications and verified against application state rather than screenshots (Sec. 3.2.1), across 33 apps including feature-rich ones such as Blender, darktable, MuseScore and Zotero (App. C). To my knowledge, the closest prior desktop benchmarks (OSWorld, Windows Agent Arena) also score by execution, but with hand-written evaluation scripts over fewer tasks. Automating that step while staying in real software rather than web replicas is the main contribution, and it matters for both evaluation and RL reward. I am not fully certain of the novelty relative to concurrent work such as Gym-Anything (Aggarwal et al., 2026), which Sec. 2 discusses in only one sentence.
  • There is evidence that state-based checks beat a screenshot-based judge and that the repair loop helps: 113/120 vs. 95/120 task-level agreement with humans (Fig. 3), and 85.0% to 94.1% agreement after self-evolution (Table 4). The darktable case study (App. A, Tables 5–7) shows the loop catching a real class of bug, schema drift.
Critical weaknesses
  • The validity of the benchmark itself is thinly established. The paper claims the tasks, environments and verifiers are "all automatically generated without relying on manual construction" (p. 2), but the 1,000 tasks are validated only by the 120-task audit (Sec. 5.1). The paper does not say who the annotators were, how many judged each trajectory, their agreement, or whether they could inspect the final application state or only saw the trajectory; if the latter, they share the LLM judge's limits. The 7 verifier disagreements are not split into false passes and false fails, and there is no estimate of how many tasks are infeasible or wrongly specified. The example checks in App. C suggest false passes are a real risk: the MuseScore checks do not appear to check the Viola or Violoncello pitches, and the Calc task spot-checks a few tiers rather than all 20 rows. Self-evolution cannot catch such cases, since it only surfaces errors where the LLM judge disagrees with the verifier. A residual error rate of about 6% is large compared with the 1.4-point spread among the top three agents (Table 2), so I cannot tell how often a "pass" is a real pass.
  • The headline comparisons lack uncertainty estimates and controls for confounds. Tables 2 and 3 report one run per model with no CIs; with n=1000 the binomial SE is about 1.3 pp, so the ranking of the top three agents is not supported, and the GUI-vs-CLI gap (72.9 vs. 67.1 on n=343) is borderline. The claim that open-weight models "exhibit sharp drops" from OSWorld-Verified is attributed to limited generalization, but the step budget, resolution, action space and prompt format are not reported, and GUI-OWL and EvoCUA average 73.6 and 67.0 steps, which suggests they often hit the budget. A harness mismatch is an equally plausible explanation. The paper also does not say which LLMs generated the tasks and verifiers (App. A uses kimi-k2.6), and since self-evolution tunes verifiers to accept that agent's solution paths, those model families may be favoured in Table 2.
  • Additionally, the benchmark is already almost solved, leaving little room for newer models (we are also missing evals the latest models like Opus 5 which was released before sonnet 5 and thus available to the authors).
Questions for authors
  1. (Clarification) Please describe the human audit in Sec. 5.1 and 5.3: how the 120 tasks were sampled and which agents produced the trajectories, how many annotators judged each trajectory and their agreement, whether they could inspect the final application state, and the false-pass/false-fail split of the 7 verifier and 25 LLM-judge disagreements. Please also confirm that no audit tasks, or tasks from the same templates, are among the 450 calibration tasks.
  2. (Clarification) Which models generated the task proposals, verifiers and seed environments, and which agents ran the calibration tasks? Do agents from those families score higher on OpenComputer than their OSWorld scores would predict?
  3. (Clarification) Please report the step budget, screen resolution, action space and prompt or harness for each model. Did GUI-OWL-1.5-8B and EvoCUA-8B use their native action formats, and what fraction of their runs ended at the step budget? Without this, I cannot tell a generalization gap from a harness mismatch.
  4. (Suggestion) Please add a manual audit of a random sample of the 1,000 released tasks that estimates the fraction of infeasible or wrongly specified tasks and the fraction where a clearly wrong final state still passes all checks. Please also add bootstrap CIs to Tables 2–3 and per-application success rates.
  5. (Clarification) How many proposals were rejected at each stage of the task pipeline (Sec. 3.3), and how much human effort went into reviewing each application's task set and finalizing the 1,000 tasks? This bears directly on the "without relying on manual construction" claim.
  6. (Suggestion) You spend a lot of time on the issues with LLM-as-a-Judge, but there are agentic verification systems like Agent-as-a-Judge (https://proceedings.mlr.press/v267/zhuge25a.html), Gandalf (https://joinhandshake.com/research/ai/gandalf-the-grader/), Archipelago (https://github.com/Mercor-Intelligence/archipelago). How do they compare against your verification system?

My score could increase if the authors (i) describe the audit protocol and the error direction of the verifier (Q1), (ii) show that the OSWorld drop is not a harness artifact (Q3), and (iii) report a false-pass rate on the released tasks together with CIs for the main tables (Q4).

4 comments
AuthorsOpenReview ↗

Reply to reviewer pvQu (part 1)

W1: The validity of the benchmark itself is thinly established.

Thank you for this detailed comment. Beyond the 120-task human audit, every task is validated automatically. We run the verifier on the untouched initial state of all 1,000 tasks and all of them fail, so no task is trivially satisfied. In addition, 914 tasks are solved by at least one model, which shows that their target states are reachable. In the human audit, each trajectory was judged by at least three annotators from a pool of five, all with practical experience in the corresponding application, and disagreements were resolved by discussion. They inspect the runs through a interface showing the task instruction, the full trajectory, final state and the checklist. Regarding App. C, the appendix shows abbreviated checks, we will include the complete checklists in the revision.

W2: The headline comparisons lack uncertainty estimates and controls for confounds.

Q2: Which models generated the task proposals, verifiers and seed environments, and which agents ran the calibration tasks? Do agents from those families score higher on OpenComputer than their OSWorld scores would predict?

Q3: Please report the step budget, screen resolution, action space and prompt or harness for each model. Did GUI-OWL-1.5-8B and EvoCUA-8B use their native action formats, and what fraction of their runs ended at the step budget? Without this, I cannot tell a generalization gap from a harness mismatch.

Thank you for this comment. We report 95% bootstrap confidence intervals (n = 1,000) for all models in the updated Table 2 and for Table 3.

ModelOSWorld-VerifiedSuccess rate95% bootstrap CI
Claude Sonnet 581.2%77.2%[74.6%, 79.8%]
GPT-5.6-Sol83.0%76.5%[73.8%, 79.1%]
Kimi-K384.8%75.8%[73.1%, 78.4%]
GPT-5.475.0%68.3%[65.4%, 71.2%]
Claude Sonnet 4.672.5%64.4%[61.4%, 67.4%]
GPT-5.6-Terra—63.3%[60.3%, 66.3%]
Gemini-3.7-Flash—36.7%[33.7%, 39.7%]
Qwen-3.5-27B56.2%32.3%[29.4%, 35.2%]
EvoCUA-8B46.1%10.9%[9.0%, 12.9%]
Qwen-3.5-9B41.8%7.8%[6.2%, 9.5%]
GUI-OWL-1.5-8B52.3%5.7%[4.3%, 7.2%]
SettingModelSuccess rate95% bootstrap CI
GUIGPT-5.475.2%[70.6%, 79.6%]
GUIClaude Sonnet 4.672.9%[68.2%, 77.6%]
CLIClaude Sonnet 4.667.1%[62.1%, 72.0%]

We present the first 3 model as a top tier rather than as a strict ranking. Regarding the harness, all models are run with the same agent implementation, prompt, action format, and observation setting (1920×1080 screenshots) that OSWorld-Verified uses for each model, with a budget of 100 model steps. The high step counts of GUI-OWL and EvoCUA are mainly because they frequently get stuck and repeatedly issue the same action until the budget is exhausted, 62.7% and 56.3% of their runs end at the 100-step budget because of the model capacity instead of harness. Our tasks can be completed in far fewer steps than the budget allows as demonstrated in other model step count. We used Kimi-K2.6 to generate the tasks and verifiers and to execute the calibration tasks. Self-evolution does not tune verifiers toward any agent’s solution paths: verifiers check only the final application state, and evolution operates at the application level on separate calibration tasks. More broadly, we see no consistent family advantage relative to OSWorld-Verified: the drops are −4.0 and −8.1 points for the two Claude models, −6.5 and −6.7 for the two GPT models, and −9.0 for Kimi-K3.

W3: Additionally, the benchmark is already almost solved, leaving little room for newer models

We would like to clarify that the main contribution of this paper is not the 1,000-task benchmark itself, but the pipeline that automatically constructs, tests, and improves programmatic verifiers across heterogeneous real applications, and then uses the resulting verification interfaces to synthesize executable tasks and environments. The current suite is one instantiation of this pipeline, not a fixed endpoint. Because tasks, environments, and verifiers are generated automatically, harder tasks and new applications can be added at low cost as models improve, so the benchmark does not saturate in the way a manually constructed one does. Even the current suite still leaves substantial headroom: the best model succeeds on 77.2% of tasks, leaving about a quarter unsolved, 86 tasks are not solved by any model, and the small open-source models all score below 33%. We will add results for the latest models in the revision.

AuthorsOpenReview ↗

Reply to reviewer pvQu (part 2)

Q1: Please describe the human audit in Sec. 5.1 and 5.3: how the 120 tasks were sampled and which agents produced the trajectories, how many annotators judged each trajectory and their agreement, whether they could inspect the final application state, and the false-pass/false-fail split of the 7 verifier and 25 LLM-judge disagreements. Please also confirm that no audit tasks, or tasks from the same templates, are among the 450 calibration tasks.

The 120 tasks were sampled uniformly at random from the 1,000 finalized tasks, and their trajectories were produced by Claude-Sonnet-5. Annotator details and the split of the 7 verifier disagreements are given in our response to Q4. Of the 25 LLM-judge disagreements, 15 were false passes and 10 were false fails. None of the 120 audit tasks is among the 450 calibration tasks, and the calibration tasks were generated separately, so no audit task shares a template with a calibration task.

Q4: Please add a manual audit of a random sample of the 1,000 released tasks that estimates the fraction of infeasible or wrongly specified tasks and the fraction where a clearly wrong final state still passes all checks. Please also add bootstrap CIs to Tables 2–3 and per-application success rates.

The human audit in Sec. 5.1 is such a random-sample audit: the 120 tasks were drawn uniformly at random from the 1,000 released tasks. We report the requested quantities below. Of the 7 verifier–human disagreements, 4 are false passes and 3 are false fails, giving a false-pass rate of 4/120 = 3.3% (95% Wilson CI [1.3%, 8.3%]). At the other extreme, the untouched initial state, which is the most clearly wrong final state, is rejected by all 1,000 tasks’ verifiers. Bootstrap CIs for Table 2 and Table 3 are given in our response to W2.

Q5:How many proposals were rejected at each stage of the task pipeline (Sec. 3.3), and how much human effort went into reviewing each application's task set and finalizing the 1,000 tasks? This bears directly on the "without relying on manual construction" claim.

Of the 1404 task proposals generated, 356 were rejected for insufficient complexity, 31 tasks because their input artifacts could not be instantiated coherently, 17 involve subjective or perceptual visual properties that cannot be reliably reduced to programmatic state checks, leaving 1,000 tasks. Human involvement was limited to review the tasks and improve the pipeline; no tasks, environments, or verifiers were written manually. We will report these numbers and clarify this claim in the revision.

Q6:You spend a lot of time on the issues with LLM-as-a-Judge, but there are agentic verification systems like Agent-as-a-Judge, Gandalf, Archipelago. How do they compare against your verification system?

Thank you for these pointers. These systems are agentic judges. Agent-as-a-Judge evaluates code-generation agents with an agent that provides feedback on the whole task-solving process. Our system uses the same idea at a different stage. During verifier construction, our comparator module is itself an agentic judge with access to the cached sandbox state, which diagnoses disagreements and repairs verifiers. At evaluation time, however, the result is compiled into deterministic programmatic checks, so scores are reproducible, cheap, and independent of any judge model, which also makes them suitable as RL rewards and benchmarking. These agentic judges are therefore complementary to our approach. We will discuss these works in the revision.

Because tasks, environments, and verifiers are generated automatically, harder tasks and new applications can be added at low cost as models improve, so the benchmark does not saturate in the way a manually constructed one does.

This need not be trivial, so you must demonstrate that your pipeline is able to generate difficult yet solvable tasks and that you have a good way to naturally control the difficulty of the tasks.

Even the current suite still leaves substantial headroom: the best model succeeds on 77.2% of tasks, leaving about a quarter unsolved, 86 tasks are not solved by any model, and the small open-source models all score below 33%. We will add results for the latest models in the revision.

Your best model tested is Sonnet 5 which is still far behind the frontier. From Artificial Analysis's Sonnet 5.5 launch tweet:

Anthropic has launched Claude Sonnet 5.5: it scores 56 on the Artificial Analysis Intelligence Index, just 2 points behind Opus 5.5 (max) [...]. With max effort, Sonnet 5.5 gains 18 points over Sonnet 5

AuthorsOpenReview ↗

Reply to reviewer pvQu

Thank you for the follow-up. OpenComputer focuses on automating the generation of desktop tasks, their environments, and executable state verifiers. We designed the dataset to assess the performance of models available at the time of submission. Claude Sonnet 5.5 was released on September 28, after the submission deadline. As models improve, future work can build on our pipeline to generate more challenging tasks.

Reviewer_8UbuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenComputer is a framework for building verifiable computer-use tasks inside real desktop applications. For each application, an agent generates a Python verifier module that exposes CLI endpoints reading application state through native inspection channels (Chrome DevTools Protocol, Marionette, LibreOffice UNO, D-Bus, SQLite profiles, saved-file parsing), with test plans and a debug-fix-retry loop. A "self-evolving" layer then runs a strong agent on about 15 calibration tasks per app, compares the programmatic verdicts with a criterion-level LLM judgment, and uses an attribution agent with access to the frozen sandbox to repair verifier-side defects. Tasks are proposed from realistic user goals, filtered for complexity, grounded in existing or newly added verifier endpoints, and materialized with seed files. A harness runs agents in fresh sandboxes and scores each run by the fraction of checks passed. The release covers 33 applications and 1,000 tasks. Frontier agents reach 75.8–77.2% success (Claude-Sonnet-5, GPT-5.6-Sol, Kimi-K3), while open-source CUA models drop sharply relative to their OSWorld-Verified scores (e.g., GUI-OWL-1.5-8B 52.3% → 5.7%). On a 120-task human audit, the programmatic verifier agrees with human task-level verdicts on 113/120 tasks versus 95/120 for an agentic screenshot-based LLM judge. Self-evolution repairs 68 of 76 checker-side errors found in 450 calibration tasks and raises human agreement from 85.0% to 94.1%. A GUI-vs-CLI comparison on a 343-task subset is also included.

Critical strengths
  1. Verification grounded in real application state, at meaningful breadth. Building app-specific verifiers over native inspection channels for 33 real desktop applications (including feature-rich ones such as Blender, darktable, MuseScore, and Zotero), and organizing task synthesis around what is actually inspectable, is a sound and scalable design that avoids the fidelity loss of web-reconstructed replicas. The 120-task human audit supports the design choice: 113/120 vs. 95/120 task-level agreement against an LLM judge. The appendix examples (cell-boundary errors, off-screen terminal evidence) explain clearly why state-based checking matters.

  2. A practical verifier-repair loop and an informative, unsaturated benchmark. The self-evolving layer restricts edits to the verifier stack, requires a concrete state-grounded attribution before repair, and records app-specific memories. It repairs 89% of identified checker bugs and raises human agreement by about 9 points, and the darktable schema-drift case study is convincing. The benchmark separates current models well (5.7%–77.2%) and exposes a large OSWorld-to-OpenComputer transfer gap for open-source CUA models, which is useful to the community.

Critical weaknesses
  1. The evaluation protocol is under-specified, which weakens the headline comparisons. The paper does not state the step budget, the agent harness/prompt/action space used for each model (native CUA tooling vs. a common harness), the screen resolution, or the number of runs and variance. The top three models differ by 1.4 points (14 of 1,000 tasks). The large drops for open-source models relative to OSWorld are attributed to poor cross-benchmark generalization, but could equally reflect harness or action-format mismatch, so this conclusion is not yet supported. The harness also performs an automatic final save, which changes task semantics for the many tasks that require saving. The LLM-judge comparison conflates observability with judging method: the judge sees only screenshots while the verifier reads state, so a judge with file/state access is needed to support the claim that "hard-coded verifiers align better than LLM-as-judge." Lastly, although the paper says the framework is released and supports training, no anonymous code or data is provided and no training experiment demonstrates its usefulness for training.
Questions for authors
  1. Please specify the step budget, harness, prompts, action space, screen resolution, and number of runs per model, and report confidence intervals.

  2. Have you used OpenComputer rewards for SFT or RL, and do they improve agents on OSWorld or held-out OpenComputer apps?

1 comment
AuthorsOpenReview ↗

Reply to reviewer 8Ubu

W1: The evaluation protocol is under-specified, which weakens the headline comparisons.

Q1: Please specify the step budget, harness, prompts, action space, screen resolution, and number of runs per model, and report confidence intervals.

Q2: Have you used OpenComputer rewards for SFT or RL, and do they improve agents on OSWorld or held-out OpenComputer apps?

Harness. We use no custom harness. Every model is run exactly as in OSWorld(-Verified), with the same agent implementation, prompt, action space, 1920×1080 screenshots. And we set the step budget to 100. Because harness and action format are identical across the two benchmarks, the open-source drop in Table 2 reflects the change in task distribution, not a format mismatch. The only exception is Gemini-3.7-Flash and GPT-5.6-Terra, which has no OSWorld entry and is excluded from that comparison. The final save is applied identically to all models, so it does not affect the comparison.

Runs and confidence intervals. We report 95% bootstrap confidence intervals over tasks (1,000 resamples) for all models in the updated Table 2 and for Table 3.

ModelOSWorld-VerifiedSuccess rate95% bootstrap CI
Claude Sonnet 581.2%77.2%[74.6%, 79.8%]
GPT-5.6-Sol83.0%76.5%[73.8%, 79.1%]
Kimi-K384.8%75.8%[73.1%, 78.4%]
GPT-5.475.0%68.3%[65.4%, 71.2%]
Claude Sonnet 4.672.5%64.4%[61.4%, 67.4%]
GPT-5.6-Terra—63.3%[60.3%, 66.3%]
Gemini-3.7-Flash—36.7%[33.7%, 39.7%]
Qwen-3.5-27B56.2%32.3%[29.4%, 35.2%]
EvoCUA-8B46.1%10.9%[9.0%, 12.9%]
Qwen-3.5-9B41.8%7.8%[6.2%, 9.5%]
GUI-OWL-1.5-8B52.3%5.7%[4.3%, 7.2%]
SettingModelSuccess rate95% bootstrap CI
GUIGPT-5.475.2%[70.6%, 79.6%]
GUIClaude Sonnet 4.672.9%[68.2%, 77.6%]
CLIClaude Sonnet 4.667.1%[62.1%, 72.0%]

LLM-as-judge. Our judge follows the standard trajectory-based LLM/VLM-as-judge protocol, used in WebVoyager[1], Pan et al [2], Online-Mind2Web[3], OSUniverse[4], and ZeroGUI[5]. We therefore compare against the judge as it is used in practice.

Release. The full framework (apps, tasks, verifiers, harness) is provided in the anonymized repository at https://anonymous.4open.science/r/OpenComputer-03D3/. And a training experiment will be added to the final version.

[1] He et al. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. ACL 2024, pages 6864–6890.

[2] Pan et al. Autonomous Evaluation and Refinement of Digital Agents. COLM 2024.

[3] Xue et al. An Illusion of Progress? Assessing the Current State of Web Agents. COLM 2025.

[4] Davydova et al. OSUniverse: Benchmark for Multimodal GUI-navigation AI Agents. arXiv:2505.03570, 2025.

[5] Yang et al. ZeroGUI: Automating Online GUI Learning at Zero Human Cost. arXiv:2505.23762, 2025

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.