acceptodds
Under review as a conference paper at ICLR 2027

OSWorld-Science: A Playground of Computer Use Agents for Learning and Using Scientific Software

16 views

Abstract

Scientific software presents a demanding test for computer-using agents based on visual language models (VLMs): completing a research workflow requires interpreting specialized interfaces, manipulating scientific objects, and producing verifiable results. We thus introduce OSWorld-Science, a benchmark and evaluation environment that combines scientifically meaningful tasks, artifact-based evaluation, and an efficient agent harness for studying computer use in the scientific domain. The benchmark contains 12 VLMs and 146 high-quality tasks across several scientific domains and software configurations, covering workflows such as molecular drawing and retrosynthesis, pathology image analysis, statistical computing, and physical simulation. Tasks are developed through expert proposals and iterative human–AI co-design, with selection guided by scientific value and difficulty. Task-specific execution-based evaluators inspect application states and generated artifacts, including molecular structures, segmentation masks, plots, and numerical results, and award partial credit for incomplete outcomes. Our special harness integrates model adapters, interaction-loop control, and trajectory logging to support comparisons of models and interaction strategies. This combination makes OSWorld-Science a testbed for examining how agents coordinate visual interpretation with graphical and command-line actions. Analysis of the task collection and results shows that current state-of-the-art VLMs with a strong harness still face challenges in addressing key questions in the scientific domains. We also analyze the benchmarking results across multi-linguistics, reasoning efforts, context length and other factors and derive several important conclusions and directions to assist future development. Overall, we provide an integrated framework connecting expert-defined scientific goals to verifiable software outcomes, enabling systematic evaluation of both agent capabilities and harness design in scientific workflows.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
computer-use agents, AI for Science, OSWorld, Multilingualism

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AAbNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OSWorld-Science, a benchmark for agents that use scientific software. It contains 146 tasks across seven scientific domains, with task-specific evaluators that check the resulting files and application states. The authors evaluate 12 vlms and analyze their failures.

Critical strengths

The scientific computer-using benchmark covers several scientific domains and provides detailed examples of failed workflows. The benchmark also includes task-specific evaluators that examine the agent's output and provide score for partially completed tasks.

Critical weaknesses

The contribution beyond existing benchmarks is not clear enough. OSWorld-Science follows the OSWorld setup and extends it to scientific applications and tasks. But the paper does not clearly establish which additional capabilities this benchmark measures or what new conclusions it supports.

Questions for authors
  1. Which scientific computer-use capabilities does OSWorld-Science evaluate that are not adequately covered by existing benchmarks
  2. How do the evaluators handle scientifically equivalent answers? For scientific tasks, maybe there can be different forms of correct answers.
5 comments

Thank you for your review.

W1 / Q1 OSWorld-Science reuses the execution interface of OSWorld (VMs, screenshots, and PyAutoGUI actions), but the capabilities it measures and the way it judges success are new.

  1. It measures scientific results, not application states. Our tasks follow real research scenarios and are proposed by domain experts or produced by expert-AI co-design. They require scientific decisions, for example setting up a CFD case and bringing it to convergence, or assigning the stereochemistry of a molecule from an image. OSWorld tasks usually ask for an operation, and success means reaching a target application state.

  2. It covers professional scientific software and real scientific data. The benchmark spans 17 software packages, several of which are commercially licensed, and its tasks are real, long-horizon research workflows. ScienceBoard targets six open-source applications with short tasks (9 steps on average). SciCode, ScienceAgentBench, and Terminal-Bench-Science evaluate scientific tasks through code generation or the terminal. Many scientific applications now offer a CLI, but researchers still operate much of this software through its GUI, which is more direct, easier to control, and easier to share with a human collaborator.

  3. Success is judged by rubrics that come from the scientific scenario of each task, not by a generic rule such as exact match or a loose ±5% threshold. In the OpenFOAM tasks, for example, the agent submits the full flow field, and the grader iterates the same solver once more from it in an isolated environment. A converged solution keeps residuals near zero, and a fabricated or unfinished field does not: for the k equation, legitimate runs stay below 3.7e-8 and cheating submissions are at least 2.4e-4. A mesh-convergence check and a comparison with DNS literature data follow. The grader therefore separates a computed result from an assembled one without looking at the operations.

In short, OSWorld-Science measures an end-to-end capability that existing benchmarks do not adequately cover: making scientific decisions, carrying them out in professional software, delivering a complete scientific artifact, and passing a scientifically grounded check of that artifact.

It also supports new conclusions. (i) Most failures deliver nothing: 60.6% of the runs without full credit end without the graded artifact (Sec. 4). (ii) Self-checks that share the agent's own assumption do not catch its errors, a pattern that recurs in our chemistry, astronomy, CFD, and geoscience case studies. (iii) The limiting capability shifts with model strength: the share of failures caused by scientific errors rises from 8% in the weakest tier of models to 25% in the strongest (Appendix G of the revised PDF).

Q2 Our evaluators compare the scientific content of an answer, not its surface form. They do so through five mechanisms.

  1. Canonical identity. Molecular structures are compared by InChIKey or canonical SMILES, so any representation of the same valid structure earns full credit.

  2. Numerical tolerances. Answers that are scientifically equivalent but reported slightly differently are accepted within a tolerance.

  3. Method replay and recomputation. The evaluator re-executes the agent's submission on held-out test inputs, so different implementations that compute the same scientific quantity pass. In structural biology, it recomputes the values directly from the submitted structure files, so the tool that produced them is not restricted. In GIS registration, it refits the transformation from the control points that the agent chose and scores it at 104 hidden check points.

  4. Geometric agreement. Geometric results may differ within a reasonable margin. A pathology gland contour passes when its Dice coefficient with the reference is at least 0.75. Contours drawn by a second pathologist reach 0.89 and receive full credit.

  5. Measured alternative routes. For the ANSYS tasks, the task authors computed about 170 plausible routes for extracting the results in advance. Routes that are physically equivalent agree within the 0.1% tolerance and pass. Routes that change the result fall outside it and fail: averaging over the wrong region, for example, differs by 0.21%.

In the revised PDF, Appendix J ("Comparison with existing benchmarks") states these distinguishing capabilities and conclusions, and Appendix I ("Evaluation protocol") describes the five mechanisms with these tests and examples.

Thank you for the response. It addresses most of my original concerns.

I further notice some distortions in the molecular structures in Figures 6(a) and 6(c), such as unclear bonds and stray marks. These make the figures look AI-generated, and this may make the corresponding results less convincing to readers. Could you explain how the figures were made and check the structures for errors?

I would be willing to raise my score if this is clarified or the figures are corrected.

Thank you for taking another look and for flagging the unclear bonds and stray marks in Figures 6(a) and 6(c). We agree that readers should be able to verify the molecular structures shown in both panels.

The panels were intended to illustrate how the agents worked through the two tasks. We used Claude Code to extract the steps from the agents’ outputs (trajectories), then drew the workflows ourselves based on those summaries. The low resolution of the submitted figures made some bonds and labels hard to read, so we are checking the underlying structures as well as improving the rendering.

We have asked PhD-trained chemists to verify the structures against the original outputs, including bond connectivity, bond orders, and atom labels, and to check that each workflow faithfully represents the steps the agents took (e.g. the main body of lenacapavir). We have addressed any discrepancies they identify, replace both panels with higher-resolution versions, and clarify in the caption how the figures were prepared.

We have updated the figures accordingly and will upload the editable version of this original figure in the pdf version after publication. Please let us know if you have any more questions.

Reviewer_ZUrNOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2(−1)
Soundness
2(−1)
Read review
Paper summary

This paper introduces OSWorld-Science, a benchmark for evaluating the computer-use capabilities of multimodal LLMs in professional scientific software. It consists of 146 tasks spanning 7 scientific domains, which were proposed by domain experts and refined through iterative human–AI co-design and selection. The authors also develop an agent harness that supports multiple models and design task-specific execution-based evaluators to assess task completion by inspecting application states and generated scientific artifacts, while allowing partial credit.

Critical strengths
  1. The benchmark covers multiple scientific domains, specialized software applications, and diverse scientific tasks, reflecting to some extent the complexity and practical value of operating scientific software.
  2. The authors design specialized evaluators for different types of scientific artifacts and allow partial credit, enabling task evaluation to accommodate the characteristics of different scientific workflows.
Critical weaknesses
  1. Although the paper acknowledges the uneven domain distribution and reports domain-macro-averaged results, macro-averaging does not address the statistical unreliability of domain-level estimates based on only two or five tasks. For example, the claim that GPT-6 Astra achieves 100% performance in linguistics is based on only two tasks and therefore provides limited evidence of domain-level capability or cross-domain generalization.
  2. The paper presents evidence-based evaluators for heterogeneous scientific outputs across disciplines as one of its main contributions. However, it remains unclear to what extent these evaluators constitute a unified evaluation framework rather than a collection of task-specific evaluators. Although the paper describes a common scoring range of 0–1, this alone does not establish a substantive shared evaluation methodology.
Questions for authors
  1. Could the authors expand the number of tasks in underrepresented domains to improve the task distribution and enable more reliable comparisons of agent performance across scientific domains?
  2. What substantive common methodology does the proposed evaluation framework provide across disciplines, beyond a common scoring interface and range?
7 comments

Thank you for your review.

W1 / Q1 We agree that two tasks cannot support a domain-level claim.

First, the small domains do not affect our conclusions. Excluding linguistics and geoscience (8 of 146 tasks) leaves the overall ranking of all 12 models unchanged and changes no overall score by more than 1.1 points. The models that lead overall also lead in these two domains (Fig. 10).

Second, we have added two linguistics tasks to the task set, and future versions of the benchmark will add more tasks and domains. They are not yet part of the reported results. Linguistics is included in this version partly to verify that the domain is feasible.

Both points are stated in Appendix B of the revised PDF ("Domains with few tasks"), which now describes the results of these two domains as pilot results rather than domain-level estimates. In Sec. 4, the example of rankings changing across domains no longer uses linguistics: it now contrasts chemistry (43 tasks, about 6%) with physics (31 tasks, 88%) for GPT-6 Astra.

W2 / Q2 Our common methodology is a shared evaluation protocol. The evaluators themselves are task-specific, because the software, the goal of the task, and the conditions under which a result is valid differ between scientific tasks. Other execution-based benchmarks work the same way: OSWorld uses 134 evaluation functions for 369 tasks, and Terminal-Bench 2.0 gives each task its own test suite. The protocol has four parts.

  1. A declarative evaluation contract per task, executed by one scorer. Each contract contains (i) the artifacts collected from the VM after DONE, (ii) the list of checks, taken from a registry, (iii) the weight of each check, (iv) the gates, that is, the checks that must pass for full credit: if a gate fails, the checks that depend on it score 0, and (v) the pass threshold.

  2. Grading is isolated from execution. Scoring never runs inside the VM. It always runs on the host, so the answers cannot leak.

  3. The rubrics come from the scientific scenario of each task, not from a generic rule such as exact match or a loose ±5% threshold. Two examples: ANSYS: the reference is a Fluent solution converged to residuals of 1e-10, bit-identical on rerun, and the tolerance is 0.1%. In one verification case, the textbook analytical solution differs from the discrete solution on the supplied mesh by 1.97% and averaging over the wrong region differs by 0.21%, so both fail, while physically equivalent routes pass. The check therefore tests whether the agent ran the simulation as specified, not whether the number looks right. OpenFOAM: the agent submits the full flow field, and the grader iterates the same solver once more from it in an isolated environment. A converged solution keeps residuals near zero, and a fabricated or unfinished field does not: for the k equation, legitimate runs stay below 3.7e-8 and cheating submissions are at least 2.4e-4, more than three orders of magnitude apart. A mesh-convergence check and a comparison with DNS literature data follow.

  4. Partial credit follows the scientific requirements, not the operation steps. In chemistry, a correct constitution earns 0.25, and full credit (1.0) requires the stereochemistry to be correct as well.

In the revised PDF, a new appendix (Appendix I, "Evaluation protocol") describes these components. We will also add one evaluator card per task, generated automatically from the task files.

Thank you for your responses to my concerns.

Regarding Q1, the additional analysis has partially alleviated my concerns about the uneven task distribution. However, labeling the underrepresented domains as "pilot results" does not fundamentally address this issue. Specifically, after excluding GIS and Linguistics, the remaining tasks nominally cover five domains. However, Medicine shares considerable overlap in research topics with Biology and Chemistry. Therefore, the actual breadth of disciplinary coverage may be narrower than the nominal count of five domains suggests.

Regarding Q2, I consider the first three points raised by the authors to be primarily engineering principles that should be followed when developing an evaluation framework. While they help ensure the consistency and reliability of the evaluation process, they do not yet demonstrate substantial methodological innovation. In contrast, the fourth point, concerning the assignment of partial credit based on the actual completion of scientific tasks rather than operational steps, offers an interesting insight in my view and deserves further discussion.

Overall, the authors' response has partially addressed my concerns, and I will accordingly raise my overall rating. In addition, the scores for presentation and soundness were mistakenly set to inappropriate values due to an oversight on my part, and I will correct them accordingly. I apologize for any inconvenience this may have caused.

AuthorsOpenReview ↗

Follow up discussion

Thank you for the thoughtful follow-up and for taking the time to revisit your assessment.

Q1. We agree that domain labels give only a rough indication of disciplinary breadth. Overlap is natural in science, and the medicine tasks illustrate why the workflows themselves matter. These tasks focus on extracting diagnostic information and processing clinical images, adding workflows that complement the molecular analysis and physical simulation tasks elsewhere in the benchmark. Even without GIS and linguistics questions, OSWorld-Science still covers a broad range of core scientific workflows. We will make that coverage and the overlap across domains explicit in the manuscript, and will add more domains (econ, material sciences, robotics, etc.) in the next version to keep it alive.

Q2. With OSWorld-Science as a benchmark study, we propose a new standard for evaluating how agents use scientific software. Full credit requires a scientifically valid result that meets the task’s requirements. Partial credit reflects which meaningful parts of that result the agent can demonstrate through its submitted artifacts. The first three components make this standard reliable to apply across tasks. For example, in the chemistry task, correct molecular connectivity earns 0.25, and full credit also requires correct stereochemistry. We will expand Appendix I to explain how we identify meaningful intermediate outcomes, what evidence is sufficient to verify them, and how they contribute to the score.

Thank you for your responses again.

Reviewer_QrSFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OSWorld-Science, a benchmark for evaluating computer-use agents (CUAs) on scientific tasks involving specialized software. The benchmark consists of 146 tasks across multiple scientific domains, including chemistry, physics, medicine, biology, and statistics. Tasks are constructed through expert proposals and human-AI collaborative design, with task-specific execution-based evaluators that assess scientific artifacts and support partial credit. The authors evaluate 12 frontier VLMs using a unified agent harness that supports both GUI and CLI interactions. Beyond overall performance comparisons, the paper provides extensive trajectory analyses and ablation studies on reasoning effort, interaction history, and multilingual instructions. The results reveal substantial performance gaps across scientific domains and highlight common failure modes, particularly the inability to produce valid artifacts and the limitations of self-consistency-based verification.

Critical strengths
  1. The paper extends general-purpose CUA evaluation to specialized scientific software, covering diverse domains and workflows. I particularly appreciate the emphasis on scientifically meaningful tasks and artifact-based evaluation, which provides a more reliable assessment of whether agents actually accomplish scientific objectives.
  2. The evaluation covers 12 VLMs and investigates several important factors, including reasoning effort, context history, and multilingual instructions. The detailed failure analysis and case studies are particularly valuable. For example, the distinction between self-consistency and independent verification offers useful insights into the challenges of developing reliable scientific agents.
Critical weaknesses

Overall, I find this a useful and well-motivated benchmark with solid empirical analyses. A few concerns:

  1. The benchmark evaluates end-to-end task completion, but it remains unclear whether failures primarily arise from insufficient scientific knowledge, visual understanding, software operation, or planning. A more systematic breakdown would help identify the actual bottlenecks of current scientific agents.

  2. Although the paper emphasizes its improved agent harness as a contribution, the experiments mainly investigate reasoning effort, history length, and prompt language. A direct comparison with the original OSWorld harness, or controlled GUI-only versus hybrid GUI/CLI experiments, would better demonstrate the effectiveness of the proposed harness design.

Questions for authors
  1. Have the authors considered categorizing failures according to their underlying causes, such as scientific reasoning errors, visual grounding failures, software operation errors, and planning failures? Qualitative analysis of representative trajectories may help clarify which capabilities are most limiting.
3 comments

Thank you for your review.

W1 / Q1 We carried out the failure attribution you suggested. We went through the trajectories of the failed runs, using agents (Claude Code with Opus 5.5) to read each trajectory and assign it one primary failure cause from the categories below. We then manually checked a sample of these labels to confirm there are no errors.

We start from your four categories: visual (perception and grounding), scientific knowledge/reasoning, software operation, and planning (incl. long-horizon execution). Two causes fall outside them and already appear in our per-domain cause tables: not following the instruction (IF) and submitting without checking (VF). The 12 models form three tiers of four by overall score (Fig. 3): Strong (Fable 5.1, Opus 5, GPT-6 Astra, GPT-5.6 sol), Middle (GPT-5.6 terra, Kimi K3, Sonnet 5, GPT-5.6 luna), and Weak (Gemini 3.1 Pro, Qwen-CUA, Qwen3.7-Plus, MiniMax M3).

Share of each tier's failed runs by primary cause (644 trajectories in total):

Primary causeStrongMiddleWeak
Visual47%19%54%
Software operation12%32%16%
Scientific25%10%8%
Planning16%28%17%
IF + VF1%10%5%
  1. The bottleneck depends on the tier. Strong models fail mainly on visual and scientific errors (72% combined), middle models on software operation and planning (60%), and weak models on visual errors (54%). For example, Sonnet 5 and Kimi K3 account for 10 of the 14 OpenFOAM runs in which the solver is never launched because the steps go to reading files (Table 24, cause C).
  2. Visual errors differ in kind. Strong models mostly misread what is on screen. Weak models mostly miss their click targets, largely through a coordinate-frame mismatch that follows the backbone (cause A in Tables 12, 20, 22). We therefore label perception (VP) and grounding (VG) separately.
  3. The share of scientific errors rises with capability (8% → 10% → 25%). These are shares of each tier's failures, and strong models fail far less often (Fig. 3), so this does not mean that they know less science. It is consistent with interface errors receding, so that the remaining failures increasingly concern the science itself.
  4. IF + VF are rarely the primary cause (1–10%) and are most frequent in the middle tier. Each run has a single primary cause, so these shares count only runs in which the ignored instruction or the missing check itself decides the outcome.

W2 We ran the requested direct comparison. Both arms use the same 27 tasks (a subset including tasks from all the software), the same three models (Claude Opus 5, Claude Sonnet 5 and GPT-5.6 luna, all at medium effort), and the same VMs, task text, 100-step budget and graders. The baseline is the original OSWorld PromptAgent and run loop:

Original OSWorldOursPairs: ours better / original better / tie
All (81 pairs)0.1590.41932 / 3 / 46
Opus 50.1810.71316 / 0 / 11
Sonnet 50.2150.3267 / 3 / 17
GPT-5.6 luna0.0810.2189 / 0 / 18

With our harness the number of fully solved runs rises from 11 to 27, and the API cost is 37% lower (253 → 159 USD), because the original's runs exhaust the step budget: 57 of 81 runs vs. 22 of 81 with ours. The trajectories trace the gap to the following mechanisms, each addressed by a specific change in our harness:

  1. Reply budget: The original 1,500-token reply limit also has to cover reasoning tokens. 32% of Opus calls hit it, and 29% of Opus steps produce no action (ours: none). Our long-horizon scientific tasks are complex and need a larger budget.

  2. Code parsing: OSWorld's harness splits the model reply at every ";", which breaks shell commands such as "a ; b" and makes many originally valid code blocks wrong. This turned 680 (Opus), 762 (luna) and 166 (Sonnet) executed code blocks into syntax errors.

  3. Coordinate frame: At the original's image setting, many models, such as GPT, receive a downscaled screenshot. We fix it in our new harness with a prompt modification.

  4. Loop design: We add an early-stop mechanism that ends the task when there is no output, or only "DONE" is output, for 5 steps, which avoids unnecessary cost.

  5. Both harnesses reach the GUI and the shell through the same pyautogui channel. However, the original makes shell work fragile, which pushes agents back to the GUI. Our harness has better computer-using capacities covering GUI and CLI operations.

In the revised PDF (changes in blue), Sec. 4 summarizes both analyses ("Failure attribution", "Harness comparison"). Appendix G gives the failure taxonomy with an illustration per category and the tier table (Tables 26–27). Appendix H gives the harness comparison: per-model scores, cost, the mechanisms above, and our changes to reply budget, code parsing, coordinate handling, and early stopping (Table 28).

Reviewer_QrSFOpenReview ↗

Response to Authors' Rebuttal

Thank you for the detailed rebuttal. The new failure attribution analysis and controlled harness comparison address my main concerns. I maintain my original rating.

Thank you for your comments!

Comments on the paper

2 comments
Area_Chair_vQmNOpenReview ↗

Requesting reviewer engagement

Dear Reviewers,

Thank you for your reviewing service. The authors have provided detailed rebuttals to the comments. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations. Thank you Reviewer ZUrN for your continued engagement; hope others will also engage.

Thank you,

AC

Thank you so much for your hard work. Please let us know what information we need to add and we would be more than happy to assist.

Sincerely, authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.