acceptodds
Under review as a conference paper at ICLR 2027

SciWorkBench: A Benchmark for Real Scientific Workflows

Abstract

Language models are increasingly being used as agents capable of planning and executing projects, as well as refining their actions through feedback. This shift raises a question: can current agents complete projects like real scientists? To examine this question, we introduce SciWorkBench, a benchmark for evaluating agents' abilities on real science tasks across biology, chemistry, materials science, and physics. SciWorkBench is built from peer-reviewed literature or expert-contributed research scenarios. We also propose a unified scientific analysis framework validated by domain experts, which can categorize most tasks in SciWorkBench. Each task covers one or more stages of this framework. We evaluate up-to-date models within the Claude Code and Codex agent frameworks under a unified execution and grading protocol. The best-performing agent achieves a mean score of 57.38, indicating that current agents still struggle with realistic scientific workflows. Our failure analysis identifies three recurring weaknesses at the workflow level: difficulty translating scientific knowledge into task-specific procedures, weak integration of evidence across different modalities, and limited capacity for scientific self-correction. These findings highlight the importance of workflow-based evaluation for measuring progress toward reliable AI systems for scientific research.

Primary area:
datasets and benchmarks
Keywords:
scientific workflow, benchmark, agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ko2cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SciWorkBench, a benchmark. It includes 98 tasks covering biology, chemistry, materials science, and physics. Scientific agent evaluation is performed using different models with harness of codex and claude code. Experiments show that even the best-performing agent scores below 60.

Critical strengths
  1. The motivation behind this benchmark's construction is reasonable, as it considers real-world experimental workflows rather than simplified scenarios.
  2. The benchmark's construction involved collaboration among domain experts in the pipeline, ensuring the reliability of the data.
Critical weaknesses
  1. The dataset is relatively small. 98 tasks are insufficient for a benchmark. More data could be added.
  2. The authors state that there were 647 candidate tasks, with only 98 retained to represent criteria stringency. However, they don't explain why so many were eliminated for quality control, and have not provided specific reasons and statistics.
  3. A more balanced distribution of tasks across the disciplines would be preferable. Additionally, the article abstract divides the subject into four disciplines: biology, chemistry, materials science, and physics. However, Figure 5 group as only three, which is inconsistent. The number of tasks in physics is significantly smaller. A more balanced distribution across the disciplines would be better.
  4. The evaluator is Codex with GPT-5.6 Sol. It also performed best in the evaluation. This design is questionable because the evaluator may have implicit style biases, leading it to score its own output higher. It's recommended to use a different model or multiple models as evaluators to improve fairness.
  5. Since experts have already participated in benchmark construction, could the subsequent evaluation also involve expert scoring, rather than relying entirely on a single model? This would allow for consistency check between expert and model scores, further confirming the evaluator's reliability.
Questions for authors

Please address the weaknesses above. Figure 2: The text is too small. Figure 3: The text is too small and unclear.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Ko2c

Response to Critical Weakness 1: Benchmark size

Thank you for raising this concern. SciWorkBench evaluates executable scientific workflows rather than individual question–answer pairs. Each task requires an agent to inspect scientific inputs, select and implement appropriate analyses, evaluate intermediate results, and produce verifiable outputs and evidence-supported conclusions. A single task can therefore involve multiple interdependent decisions and cover several stages of scientific analysis.

The 98 retained tasks were selected from 647 candidates through automated checks, independent expert review, and pilot evaluation. Our priority was to ensure that each retained task provides a scientifically meaningful problem, sufficiently specified inputs, and assessable outputs. Figures 4 and 5 document coverage across workflow stages and scientific disciplines.

We agree that task complexity and stringent screening alone do not establish comprehensive coverage or sufficient statistical power. Our conclusions are therefore scoped to the current task collection, and comparisons within smaller disciplinary subsets require particular caution. We have made these limitations explicit in the revised manuscript and identify broader disciplinary and workflow coverage as an important direction for future releases.

AuthorsOpenReview ↗

Response to Reviewer Ko2c

Response to Critical Weakness 2: Transparency of task screening

Thank you for identifying this reporting gap. We agree that reporting the reduction from 647 candidates to 98 retained tasks is insufficient to explain the screening process or establish its quality.

Our screening criteria concern scientific grounding, input sufficiency, executability, and evaluability. Specifically, a retained task should be grounded in a real research scenario or peer-reviewed work, provide sufficient inputs and a clearly specified objective, support execution in the evaluation environment, and include reference materials and grading criteria consistent with the requested outputs. Candidates that did not meet these requirements were subject to revision or exclusion.

We will expand the description of these criteria and document exclusion reasons and representative rejected examples using the available screening records. We will also clarify how expert disagreements were handled. Where the records do not support a reliable retrospective count, we will state that limitation rather than infer a breakdown from the overall rejection rate.

We also acknowledge that filtering can introduce selection bias. Requiring executable inputs and assessable outputs may favor well-specified computational workflows and underrepresent tasks requiring physical experimentation, inaccessible data, or less standardized assessment. The retained collection should therefore be interpreted as a benchmark of scientific workflows that can be evaluated under our protocol, rather than a representative sample of all scientific research.

AuthorsOpenReview ↗

Response to Reviewer Ko2c

Response to Critical Weakness 3: Disciplinary distribution and reporting groups

Thank you for pointing out this ambiguity. SciWorkBench covers four disciplines: biology, chemistry, materials science, and physics. Figure 5 and Table 4 summarize them in three reporting groups, combining chemistry and materials science as “Chemistry & Materials” because of their overlapping research topics and analytical practices. This grouping does not change the benchmark’s disciplinary scope. We will explicitly clarify this distinction in Section 4.3 and the relevant captions.

Regarding the unequal task counts, SciWorkBench is primarily organized around the scientific analysis workflow, with coverage across workflow stages reported in Figure 4. We selected tasks based on scientific validity, input sufficiency, and evaluability rather than equal disciplinary quotas. The smaller physics subset reflects the eligible tasks retained through the current collection and screening process. We acknowledge that this limits the breadth of conclusions that can be drawn about physics.

Disciplinary imbalance also affects aggregate results: because the overall score averages available valid task scores, groups with more evaluated tasks contribute more to each agent’s overall mean. The overall score should therefore be interpreted as performance on the current task collection, rather than an equally weighted average across disciplines. We provide group-specific results in Table 4 and acknowledge unequal disciplinary coverage as a limitation. Expanding underrepresented disciplines, particularly physics, is a priority for future releases.

Reviewer_WjPrOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

SciWorkBench introduces 98 scientific workflow tasks across biology, chemistry, materials science, and physics. It evaluates 13 model–harness combinations on their ability to analyze experimental evidence and support scientific conclusions. The results highlight difficulties with choosing procedures, combining evidence, and correcting earlier mistakes.

Critical strengths
  1. The tasks capture meaningful parts of scientific work.
  2. The paired harness comparisons offer a useful practical finding.
Critical weaknesses
  1. The scoring needs stronger validation. Section 5.2 uses Codex with GPT-5.6 Sol as the sole grader, but the paper reports no agreement with domain experts or checks using another judge.
  2. The paper’s writing and formatting do not follow standard academic conventions. The figures are oversized and clustered together, and the conclusion spans five paragraphs, taking up nearly a full page.
Questions for authors
  1. Can you compare the grader’s scores with independent domain-expert assessments on a representative task subset, and check whether another judge gives similar rankings?
  2. Can you reconcile the overall and domain-level scores, and report the exact repetition counts and uncertainty estimates for the main comparisons?
10 comments
AuthorsOpenReview ↗

Response to Reviewer WjPr

Thank you for raising this concern. We have added an alternative-judge evaluation using Claude Code + DeepSeek-v4.1-Flash-1017, compared with the original Codex CLI + GPT-5.6-Sol judge. The analysis includes 1,226 paired task–agent scores across all 98 tasks and 13 model–harness configurations.

At the individual-score level, Pearson’s correlation is 0.900, Spearman’s correlation is 0.899, and Lin’s concordance correlation coefficient is 0.891. The alternative judge assigns scores 3.21 points higher on average, with a mean absolute difference of 5.84 points, indicating strong agreement alongside a systematic scoring shift.

Aggregate agent rankings are highly consistent across the two judges: Spearman’s ρ = 0.989, with 76 of 78 pairwise agent orderings preserved. To account for unequal task coverage, we also compare all configurations on the 55 tasks with complete paired scores. Ranking agreement remains high (ρ = 0.995), with 77 of 78 orderings preserved. These results support the robustness of aggregate rankings to the tested change in judge configuration.

We agree that cross-judge consistency does not establish agreement with domain experts or rule out shared evaluator biases. Independent expert scoring of submitted outputs remains an outstanding validation step, and we have explicitly acknowledged this limitation in the revised manuscript.

Dear Authors,

As the rebuttal period is nearing its end, I would like to follow up on the questions raised in my review,. I would appreciate any responses or clarifications you could provide, as these would help inform my final assessment.

Thank you for your time.

Thank you for the additional evaluation. The strong ranking agreement is helpful. Could you please clarify:

  1. Why are there 1,226 paired scores rather than 1,274 (98 × 13), and how were missing scores handled?
  2. Which agent pairs changed order? Did the top-ranked agent change?
  3. Could you provide independent expert scoring on a small, representative subset?
Reviewer_TXeYOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for AI Scientists/LLM agents. The paper says that it contrasts itself from other benchmarks by focussing on "workflow-level scientific judgment".

From my reading, it still reads more like a good workshop paper than a maintrack paper: There is little deep insight besides overall scores and some comparison of scaffolds on the overall scores.

Critical strengths
  • As far as I understand the motivation, it is well motivated. There is need to go beyond the outcome-focussed evals.

  • There is an evaluation with a decently wide array of models (and two scaffolds)

Critical weaknesses
  • There are certainly works that do deeper analysis (e.g. actually analyzing the workflow) that have not been referenced. More importantly, with those works in context, it is not clear what advance this benchmark brings: It is not clear what scientific insight SCIWORKBENCH brings that the existing evals could not bring:

  • "The challenge is not only to compute a result, but also to decide what should be analyzed, whether the data are reliable, which assumptions are reasonable, and what conclusions the evidence can support." is an important challenge. But I did not see how the paper addresses this.

I do not believe that those weaknesses can be fixed in the rebuttal as they target the core off what this work is about/what has been measured (before).

Questions for authors
  • Could you somehow share the corpus such that we can actually review the benchmark? It is very hard for me to assess the quality of the benchmark without seeing the items

  • How does the overall score you report (e.g. Figure 2) relate to the motivation to "decide what should be analyzed, whether the data are reliable, which assumptions are reasonable, and what conclusions the evidence can support." To my understanding it is only in there very indirectly via the rubrics you ask the judge to use

  • How did you validate the rubrics (that seem LLM-generated to a wide extent)

6 comments
AuthorsOpenReview ↗

Response to Reviewer TXeY

Thank you for recognizing the motivation and breadth of our evaluation. We address the concerns below.

  1. Corpus access

We will release the corpus and associated benchmark materials together and follow up here with access information once available.

  1. Related work and contribution

The distinction is not whether these studies analyze workflows, but how they organize the evaluation and what they assess. Our work proposes a unified scientific analysis framework validated by domain experts, and then organizes this cross-disciplinary scientific analysis into executable benchmark tasks with task-specific scientific requirements and rubrics. Our empirical contribution also includes model-dependent harness effects (Table 3), different leading configurations across disciplines (Table 4), and scientific failures despite successful code execution or artifact generation (§5.6). These are concrete benchmark findings, not a claim to have invented workflow analysis. A request for clearer positioning is reasonable; establishing redundancy requires comparing the framework, task coverage, scoring objects, and findings—not thematic overlap alone.

The cited studies are relevant, but their organizing focus differs:

WorkMain evaluation focus
Mandal et al. (2025), AILA/AFMBenchAFM-centered workflow automation, including experimental design, instrument operation, and results analysis.
Ríos-García et al. (2026), CorralTask performance and the epistemic structure of agent reasoning across scientific environments.
SciWorkBenchHeterogeneous experimental-analysis tasks organized by shared workflow stages, assessed through domain-specific scientific deliverables and rubrics.
  1. The motivation should be distinguished from the evaluation framework

First, a small correction: in our manuscript, Figure 2 shows construction and quality control; Figure 6 reports overall performance.

The four introductory questions explain why scientific analysis deserves evaluation; they do not define a four-dimensional evaluation taxonomy. Our framework is presented in Figure 1b and §3.1, its instantiation as scientific tasks in §4.2, and its workflow-stage coverage in Figure 4.

Tasks cover one or more stages of this framework and are assessed against task-specific scientific requirements. The overall score aggregates performance on these tasks; it is not intended as four separate measurements corresponding to the introductory sentence.

  1. Rubric construction and validation

Our rubrics are expert-authored (§5.2), and §4.1 and Figure 2 describe the expert-driven construction and quality-control pipeline. To supplement that description, we clarify the following procedures, which were applied during construction but not fully detailed in the manuscript:

  • Data provenance: Raw experimental inputs must come from genuine experiments, including expert-contributed research projects, or published, peer-reviewed studies. Fabricated or synthetically generated raw experimental data are excluded.

  • Consistency checks and independent review: Each task undergoes agent-as-a-judge checks and cross-review by at least two independent domain experts. They examine whether the inputs are sufficient, the reference answer is scientifically justified, and the rubric is consistent with both. Checks specifically address problems such as missing information needed to solve a task and scientifically correct responses failing to receive appropriate credit. Tasks that fail are revised or rejected.

  • Restrictions on LLM assistance: During task construction, LLM assistance is limited to data formatting, language polishing, and wording improvements. Task authors are prohibited from using LLMs to fabricate or generate complete tasks, including their rubrics. Tasks must originate from genuine research needs. At least two independent domain experts check that the tasks and scientific grading criteria are grounded in published studies or accepted disciplinary procedures and standards.

As already reported in §4.1, pilot agent runs additionally check executability and gradability, and 98 of 647 candidate tasks passed the full construction pipeline.

This rebuttal reads partially LLM authored to me.

Unfortunately, it does not convince me. The authors list the following findings as novel contributions:

  • model-dependent harness effects (Table 3) -> has been shown in previous works (e.g. https://arxiv.org/pdf/2605.08941)
  • different leading configurations across disciplines (Table 4) -> has been shown in previous work (e.g. the cited Rios-Garcia)
  • and scientific failures despite successful code execution or artifact generation (§5.6) -> in its very specific manifestation might not have been shown (but there are findings like that in https://arxiv.org/pdf/2605.08941)
AuthorsOpenReview ↗

Response to Reviewer TXeY

Two distinctions are central to assessing this review.

MDGym’s comparisons confound model and harness. Its Codex and Claude Code agents use different backbones, while open-source models run through OpenHands. Such comparisons cannot isolate harness effects for a fixed backbone. Our Table 3 evaluates the same four backbones under both harnesses and observes that the direction of the performance difference varies across models. MDGym’s cited comparison therefore does not establish the same finding.

The reviewer’s assessment overlooks the task-level workflow evaluation illustrated in Figure 3. As explained in §4.2, each task covers one or more workflow stages, with task-specific scientific requirements assessed through expert-authored rubrics (§5.2). Figure 6 aggregates these assessments; the aggregate score does not define the evaluation framework.

We acknowledge precedents for the broad failure patterns. Their recurrence does not establish redundancy of the benchmark’s task coverage and assessment requirements. These distinctions should inform the assessment of SCIWORKBENCH’s contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.