- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces OSWorld-Science, a benchmark for agents that use scientific software. It contains 146 tasks across seven scientific domains, with task-specific evaluators that check the resulting files and application states. The authors evaluate 12 vlms and analyze their failures.
Critical strengths
The scientific computer-using benchmark covers several scientific domains and provides detailed examples of failed workflows. The benchmark also includes task-specific evaluators that examine the agent's output and provide score for partially completed tasks.
Critical weaknesses
The contribution beyond existing benchmarks is not clear enough. OSWorld-Science follows the OSWorld setup and extends it to scientific applications and tasks. But the paper does not clearly establish which additional capabilities this benchmark measures or what new conclusions it supports.
Questions for authors
- Which scientific computer-use capabilities does OSWorld-Science evaluate that are not adequately covered by existing benchmarks
- How do the evaluators handle scientifically equivalent answers? For scientific tasks, maybe there can be different forms of correct answers.
5 comments
Thank you for the response. It addresses most of my original concerns.
I further notice some distortions in the molecular structures in Figures 6(a) and 6(c), such as unclear bonds and stray marks. These make the figures look AI-generated, and this may make the corresponding results less convincing to readers. Could you explain how the figures were made and check the structures for errors?
I would be willing to raise my score if this is clarified or the figures are corrected.
Thank you for taking another look and for flagging the unclear bonds and stray marks in Figures 6(a) and 6(c). We agree that readers should be able to verify the molecular structures shown in both panels.
The panels were intended to illustrate how the agents worked through the two tasks. We used Claude Code to extract the steps from the agents’ outputs (trajectories), then drew the workflows ourselves based on those summaries. The low resolution of the submitted figures made some bonds and labels hard to read, so we are checking the underlying structures as well as improving the rendering.
We have asked PhD-trained chemists to verify the structures against the original outputs, including bond connectivity, bond orders, and atom labels, and to check that each workflow faithfully represents the steps the agents took (e.g. the main body of lenacapavir). We have addressed any discrepancies they identify, replace both panels with higher-resolution versions, and clarify in the caption how the figures were prepared.
We have updated the figures accordingly and will upload the editable version of this original figure in the pdf version after publication. Please let us know if you have any more questions.
Thank you for your review.
W1 / Q1 OSWorld-Science reuses the execution interface of OSWorld (VMs, screenshots, and PyAutoGUI actions), but the capabilities it measures and the way it judges success are new.
It measures scientific results, not application states. Our tasks follow real research scenarios and are proposed by domain experts or produced by expert-AI co-design. They require scientific decisions, for example setting up a CFD case and bringing it to convergence, or assigning the stereochemistry of a molecule from an image. OSWorld tasks usually ask for an operation, and success means reaching a target application state.
It covers professional scientific software and real scientific data. The benchmark spans 17 software packages, several of which are commercially licensed, and its tasks are real, long-horizon research workflows. ScienceBoard targets six open-source applications with short tasks (9 steps on average). SciCode, ScienceAgentBench, and Terminal-Bench-Science evaluate scientific tasks through code generation or the terminal. Many scientific applications now offer a CLI, but researchers still operate much of this software through its GUI, which is more direct, easier to control, and easier to share with a human collaborator.
Success is judged by rubrics that come from the scientific scenario of each task, not by a generic rule such as exact match or a loose ±5% threshold. In the OpenFOAM tasks, for example, the agent submits the full flow field, and the grader iterates the same solver once more from it in an isolated environment. A converged solution keeps residuals near zero, and a fabricated or unfinished field does not: for the k equation, legitimate runs stay below 3.7e-8 and cheating submissions are at least 2.4e-4. A mesh-convergence check and a comparison with DNS literature data follow. The grader therefore separates a computed result from an assembled one without looking at the operations.
In short, OSWorld-Science measures an end-to-end capability that existing benchmarks do not adequately cover: making scientific decisions, carrying them out in professional software, delivering a complete scientific artifact, and passing a scientifically grounded check of that artifact.
It also supports new conclusions. (i) Most failures deliver nothing: 60.6% of the runs without full credit end without the graded artifact (Sec. 4). (ii) Self-checks that share the agent's own assumption do not catch its errors, a pattern that recurs in our chemistry, astronomy, CFD, and geoscience case studies. (iii) The limiting capability shifts with model strength: the share of failures caused by scientific errors rises from 8% in the weakest tier of models to 25% in the strongest (Appendix G of the revised PDF).
Q2 Our evaluators compare the scientific content of an answer, not its surface form. They do so through five mechanisms.
Canonical identity. Molecular structures are compared by InChIKey or canonical SMILES, so any representation of the same valid structure earns full credit.
Numerical tolerances. Answers that are scientifically equivalent but reported slightly differently are accepted within a tolerance.
Method replay and recomputation. The evaluator re-executes the agent's submission on held-out test inputs, so different implementations that compute the same scientific quantity pass. In structural biology, it recomputes the values directly from the submitted structure files, so the tool that produced them is not restricted. In GIS registration, it refits the transformation from the control points that the agent chose and scores it at 104 hidden check points.
Geometric agreement. Geometric results may differ within a reasonable margin. A pathology gland contour passes when its Dice coefficient with the reference is at least 0.75. Contours drawn by a second pathologist reach 0.89 and receive full credit.
Measured alternative routes. For the ANSYS tasks, the task authors computed about 170 plausible routes for extracting the results in advance. Routes that are physically equivalent agree within the 0.1% tolerance and pass. Routes that change the result fall outside it and fail: averaging over the wrong region, for example, differs by 0.21%.
In the revised PDF, Appendix J ("Comparison with existing benchmarks") states these distinguishing capabilities and conclusions, and Appendix I ("Evaluation protocol") describes the five mechanisms with these tests and examples.