- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SciWorkBench, a benchmark. It includes 98 tasks covering biology, chemistry, materials science, and physics. Scientific agent evaluation is performed using different models with harness of codex and claude code. Experiments show that even the best-performing agent scores below 60.
Critical strengths
- The motivation behind this benchmark's construction is reasonable, as it considers real-world experimental workflows rather than simplified scenarios.
- The benchmark's construction involved collaboration among domain experts in the pipeline, ensuring the reliability of the data.
Critical weaknesses
- The dataset is relatively small. 98 tasks are insufficient for a benchmark. More data could be added.
- The authors state that there were 647 candidate tasks, with only 98 retained to represent criteria stringency. However, they don't explain why so many were eliminated for quality control, and have not provided specific reasons and statistics.
- A more balanced distribution of tasks across the disciplines would be preferable. Additionally, the article abstract divides the subject into four disciplines: biology, chemistry, materials science, and physics. However, Figure 5 group as only three, which is inconsistent. The number of tasks in physics is significantly smaller. A more balanced distribution across the disciplines would be better.
- The evaluator is Codex with GPT-5.6 Sol. It also performed best in the evaluation. This design is questionable because the evaluator may have implicit style biases, leading it to score its own output higher. It's recommended to use a different model or multiple models as evaluators to improve fairness.
- Since experts have already participated in benchmark construction, could the subsequent evaluation also involve expert scoring, rather than relying entirely on a single model? This would allow for consistency check between expert and model scores, further confirming the evaluator's reliability.
Questions for authors
Please address the weaknesses above. Figure 2: The text is too small. Figure 3: The text is too small and unclear.
6 comments
Response to Reviewer Ko2c
Response to Critical Weakness 2: Transparency of task screening
Thank you for identifying this reporting gap. We agree that reporting the reduction from 647 candidates to 98 retained tasks is insufficient to explain the screening process or establish its quality.
Our screening criteria concern scientific grounding, input sufficiency, executability, and evaluability. Specifically, a retained task should be grounded in a real research scenario or peer-reviewed work, provide sufficient inputs and a clearly specified objective, support execution in the evaluation environment, and include reference materials and grading criteria consistent with the requested outputs. Candidates that did not meet these requirements were subject to revision or exclusion.
We will expand the description of these criteria and document exclusion reasons and representative rejected examples using the available screening records. We will also clarify how expert disagreements were handled. Where the records do not support a reliable retrospective count, we will state that limitation rather than infer a breakdown from the overall rejection rate.
We also acknowledge that filtering can introduce selection bias. Requiring executable inputs and assessable outputs may favor well-specified computational workflows and underrepresent tasks requiring physical experimentation, inaccessible data, or less standardized assessment. The retained collection should therefore be interpreted as a benchmark of scientific workflows that can be evaluated under our protocol, rather than a representative sample of all scientific research.
Response to Reviewer Ko2c
Response to Critical Weakness 3: Disciplinary distribution and reporting groups
Thank you for pointing out this ambiguity. SciWorkBench covers four disciplines: biology, chemistry, materials science, and physics. Figure 5 and Table 4 summarize them in three reporting groups, combining chemistry and materials science as “Chemistry & Materials” because of their overlapping research topics and analytical practices. This grouping does not change the benchmark’s disciplinary scope. We will explicitly clarify this distinction in Section 4.3 and the relevant captions.
Regarding the unequal task counts, SciWorkBench is primarily organized around the scientific analysis workflow, with coverage across workflow stages reported in Figure 4. We selected tasks based on scientific validity, input sufficiency, and evaluability rather than equal disciplinary quotas. The smaller physics subset reflects the eligible tasks retained through the current collection and screening process. We acknowledge that this limits the breadth of conclusions that can be drawn about physics.
Disciplinary imbalance also affects aggregate results: because the overall score averages available valid task scores, groups with more evaluated tasks contribute more to each agent’s overall mean. The overall score should therefore be interpreted as performance on the current task collection, rather than an equally weighted average across disciplines. We provide group-specific results in Table 4 and acknowledge unequal disciplinary coverage as a limitation. Expanding underrepresented disciplines, particularly physics, is a priority for future releases.
Response to Reviewer Ko2c
Response to Critical Weakness 1: Benchmark size
Thank you for raising this concern. SciWorkBench evaluates executable scientific workflows rather than individual question–answer pairs. Each task requires an agent to inspect scientific inputs, select and implement appropriate analyses, evaluate intermediate results, and produce verifiable outputs and evidence-supported conclusions. A single task can therefore involve multiple interdependent decisions and cover several stages of scientific analysis.
The 98 retained tasks were selected from 647 candidates through automated checks, independent expert review, and pilot evaluation. Our priority was to ensure that each retained task provides a scientifically meaningful problem, sufficiently specified inputs, and assessable outputs. Figures 4 and 5 document coverage across workflow stages and scientific disciplines.
We agree that task complexity and stringent screening alone do not establish comprehensive coverage or sufficient statistical power. Our conclusions are therefore scoped to the current task collection, and comparisons within smaller disciplinary subsets require particular caution. We have made these limitations explicit in the revised manuscript and identify broader disciplinary and workflow coverage as an important direction for future releases.