acceptodds
Under review as a conference paper at ICLR 2027

SciProdBench: Evaluating and Improving Scientific Research by AI Agents

Abstract

AI agents can now carry out substantial parts of scientific research, from surveying the literature to running experiments and reporting conclusions. Whether those conclusions rest on synthesized evidence and correct computation, however, remains hard to assess, and automated research can improve only on what it can verify. We introduce \sciprod, a benchmark of 184 research tasks in Biology, Chemistry, Computer Science, and Physics that span the research cycle, from problem formulation to experiment design, analysis, and reporting. Each task pairs a research objective with required deliverables and a task-specific environment. Report rubrics assess evidence synthesis and scientific reasoning, and for tasks that require computation, hidden executable tests check the underlying artifacts. Tasks and their verifiers are derived from the same graph of scientific evidence, so that every requirement and scoring criterion can be traced to its scientific basis. Evaluating nine models and four harnesses, we find that agents meet 90–97% of instruction-following criteria but only 49–57% of evidence-synthesis criteria and that Codex and Claude Code raise scores on reasoning tasks by 10–23 points while lowering scores on execution tasks by 6–9 points. We further introduce \eulerflow, a multi-agent harness that checks reported findings against artifacts and execution records and returns unsupported findings for revision. \eulerflow improves both reasoning and execution tasks with two different underlying models, raising the overall score of Claude-Opus-4.8 from 44.7 to 53.1.

Primary area:
datasets and benchmarks
Keywords:
science agent benchmark hanress multi-agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xN4VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce SciProdBench, a benchmark to assess models' abilities to synthesize literature and execute code across 184 scientific tasks (biology, chemistry, ...). Tasks are constructed from existing research papers and repositories, with rubrics used to evaluate whether the model completed the task successfully and includes all deliverables/evidence for the models' claims. The authors' initial evaluations show coding agents excel in instruction-following but struggle with evidence synthesis, motivating a new workflow called EulerFlow: an agent that reports findings, checks them with existing evidence, and refines the unsupported ones. EulerFlow improves existing harnesses on SciProdBench.

Critical strengths
  1. The dataset would likely help other researchers who are interested in evaluating scientific execution and synthesis tasks. I like the split of tasks that require just literature search, or also literature search and code execution
  2. The main idea of EulerFlow is interesting: generate proposals, cross-check them with evidence, and revise unsupported ones. I think the harness contribution could be reframed, but I see how this could be useful in a variety of agentic tasks, not just AI4Science
Critical weaknesses
  1. I didn't see any examples of the benchmark tasks in the paper, so it's hard to understand what the dataset looks like. I would recommend adding a full example of the input task, rubrics for scoring, and full trajectory of a model completing such a task, even if it's in the appendix. Otherwise, I really can't tell whether SciProdBench is truly testing the capabilities that it claims to. Figure 1 describes tasks at a high level (i.e, the steps that are taken) but an actual example would be helpful
  2. The dataset lacks quality control, so it's hard to tell how rigorous the dataset construction is. It would be great if the authors could design a human annotation setup to quantify what proportion of the tasks are high-quality, and what should people look out for when using this dataset
  3. The proposed modeling contribution of EulerFlow is weak: its description in section 4 is sparse and implies that it is implemented as a skill file, so I'm not sure if it constitutes an entire harness. It's also unclear how much this harness is engineered towards the synthetic tasks that the authors generate. I would recommend testing generalization on another dataset, and tempering claims about developing a new harness in this paper. I also suspect this process is much more computationally expensive since the model needs to check and revise each generated output
Questions for authors
  1. The approach of representing tasks as graphs and then prompting LLMs to generate descriptions, test cases, etc. from the graphs seems unnecessarily complex. Is there any advantage to this approach that the authors could argue, theoretically or empirically?
  2. Can the authors provide quality control statistics of the dataset with human annotations?
  3. Is there any way to gauge the upper bound of this task? In theory it's 100%, but it might be useful to use the quality control statistics from (2) or discuss how humans could complete some of these tasks to establish a stronger ceiling
  4. Can the authors provide examples of the tasks in the dataset?
  5. What is the cost, generalization, and novelty of EulerFlow?
Reviewer_LSdSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SciProdBench, a benchmark of 184 scientific research tasks across multiple domains. It evaluates both evidence-based scientific reasoning and computational correctness using rubrics and executable tests derived from scientific evidence graphs. The authors also propose EulerFlow, a multi-agent research framework that verifies reported findings against computational artifacts. Experiments show that current agents struggle with evidence synthesis despite strong instruction-following performance, while EulerFlow improves performance over base agent configurations.

Critical strengths
  1. Jointly constructing research tasks and evaluation criteria from scientific evidence graphs is an interesting approach to improving evaluation traceability and consistency.
  2. The benchmark covers multiple scientific domains and includes substantial expert review, increasing confidence in task quality.
  3. The evaluation of multiple models and agent harnesses provides useful insights into the gap between scientific reasoning and executable correctness. EulerFlow also demonstrates promising improvements.
Critical weaknesses
  1. The benchmark tasks are derived from published scientific papers, including publicly available code repositories and datasets. Models may have encountered these materials during pretraining. Although reference results are withheld from task instructions, this does not eliminate contamination. The paper lacks a systematic analysis of how prior exposure might affect benchmark performance.
  2. Only 31 of 184 tasks include executable tests, while the remaining tasks rely on rubric-based evaluation of research reports. Given the paper's emphasis on verifying scientific computations, the relatively small proportion of execution-based tasks limits the strength of this contribution.
  3. EulerFlow combines multi-agent coordination, task decomposition, artifact verification, and iterative revision. Without detailed component ablations and comparisons under matched computational budgets, it is difficult to determine whether the improvements come specifically from evidence verification or from additional agents and computation.
Questions for authors
  1. How do the authors assess potential contamination from the published papers, repositories, and datasets used to construct SciProdBench?
  2. Since most tasks are grounded in existing scientific findings, to what extent does performance on SciProdBench reflect general scientific research ability rather than reproducing known results?
  3. Could the authors provide ablation studies of EulerFlow's components and control for token usage, execution attempts, and computational budgets to better isolate the contribution of artifact-level verification?
Reviewer_t4udOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SciProdBench contains 184 scientific research tasks across Biology, Chemistry, Computer Science, and Physics. Tasks and verifiers are constructed from shared scientific evidence graphs and reviewed by domain experts. All tasks receive report-based evaluation, while 31 additionally use hidden executable tests. Evaluations of nine models and several harnesses reveal gaps between instruction following, evidence synthesis, and computational performance. The paper also introduces EulerFlow, a multi-agent harness that reviews worker findings against artifacts and execution records, improving overall scores on two underlying models.

Critical strengths
  1. I found the joint evaluation of reports and computational artifacts valuable. A convincing scientific report does not necessarily establish that the underlying analysis was performed correctly. Scoring both on the same tasks makes this gap easier to examine. Constructing tasks and verifiers from shared evidence graphs, followed by expert review, also provides a useful basis for aligning evaluation with task requirements.

  2. The judge validation includes useful checks beyond a model leaderboard. It tests whether scientific content outweighs polished presentation, compares several judges, and examines repeatability. I think these checks provide some reassurance that report scores capture more than presentation quality.

Critical weaknesses
  1. I am not sufficiently convinced that executable failures consistently reflect incorrect computation. In Appendix D.2, the correct PK speedup is saved in mechanism_checks.csv, but a verifier rejects it because the corresponding rows are absent from operation_timing.csv. It is unclear whether this placement requirement was explicitly given to the agent. Since any failed required test sets the entire execution score to zero, such mismatches could materially affect the results. Reliable computational verification is central to the benchmark’s contribution, so this needs stronger validation.

  2. The evidence for EulerFlow’s claimed execution benefits is incomplete. RRE scores combine report quality with executable correctness, so higher RRE scores alone do not establish more reliable computation. Delegation, planning, and artifact review also change together, without an ablation identifying the contribution of verification. The results support gains for the full harness, but leave unclear whether its proposed verification mechanism addresses the computational failures motivating the paper.

Questions for authors
  1. Was the file-and-row requirement behind the PK verifier failure explicitly specified in the task? How do the verifiers handle scientifically correct outputs saved under alternative, otherwise permitted representations?

  2. Can you report report-only scores and executable pass rates separately on the same RRE tasks, including paired uncertainty for the harness comparisons?

  3. On a representative subset, how much of EulerFlow’s gain remains when artifact review is disabled while delegation and resource budgets remain fixed? Please also report actual resource use across harnesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.