acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking LLM abilities on postdoc-level natural science problems

Abstract

Scientific problems have no answer key, so evaluating an agent means deciding what counts as a better solution before any agent is run. Existing benchmarks either match a symbolic answer against a key or, where they require executable work, score it by unit tests, by a rubric, or by position on a competition leaderboard. None of these is the criterion by which a field decides that one published method beats another. STEMBench adopts that criterion directly. Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each is bounded by two measured anchors: a submission that uses no domain knowledge, and an expert reference reproducing a published method, both obtained by running code shipped with the benchmark. The interval between them states how much of a problem is crossed by scientific insight rather than by engineering. Across eight frontier models, task-level wins and domain-level leadership dissociate: the model with the most task wins leads no domain mean. The expert anchor is reached on eight of twelve tasks, while the tasks demanding calibrated uncertainty or causal latent structure resist. A large share of zero scores is structural: the episode ended without a valid submission at all, a failure mode that answer-matching benchmarks cannot see. We release the tasks, graders, anchors and harness.

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM agents, AI for science, scientific discovery, evaluation, code generation, agentic evaluation, tool use, reasoning, applications to physical sciences, neuroscience

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r6x8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the performance of different frontier LLM systems on 12 scientific tasks. They carefully curate a set of tasks for which both a modest and expert baseline solution exist, against which they can assess model performance.

They find significant heterogeneity in performance across tasks.

Critical strengths
  • Interesting approach, relying on the existence of both modest and expert solutions per task to gauge model performance is a valuable framing
  • These are real world scientific tasks, not contrived
  • Evaluations span a range of important models
Critical weaknesses
  • The methodology is insufficiently clarified. It is unclear to what extent soft-optimisations are employed (even if unintentionally by the authors) to improve model performance on the tasks. For example, it is widely known that model performance is highly sensitive to harness design & prompt phrasing. It is unclear when exactly the tasks became well-defined, and how divorced was the process of creating those tasks and running the evaluation procedure. For high quality model evaluation work, I would require authors pre-register their tasks before any evaluations are performed. Without a clear pre-registration, I am concerned that there have been many adjustments to the evaluation prompts and harness that are correlated with the test sets within each task.
  • while I very much appreciate the focus on real world scientific tasks, 12 tasks is insufficient for a main paper, especially without additional analysis to elucidate why models differ in performance across different tasks.
Questions for authors

What do you see as the main bottleneck(s) in scaling up your work to more than 12 tasks?

1 comment

We thank the reviewer for careful reading of the paper, please find our comments below.

The paper represents a methodology proposal that can later be used to update the benchmark with new problems. We decided to fix the amount of problems to 12 in order to test the approach in different domains with different amounts of data and information description, thus developing a unified setting for problem verification. Additional problems will be added once methodology is fixed. The increase of the number of problems at this stage is not needed, since the problems are carefully positioned and test different problem-specific peculiarities, for example, abundance of data, or the amount of description available in the internet. Tasks were not formally pre-registered, but tasks, prompts, harness and graders were frozen before all reported runs. Every model then ran under identical conditions: the same agent loop and system prompt, the same harness-side tools (code execution, data exploration, and a format-only validation check that never returns a score), the same isolated sandbox, and the same token and time limits, with no per-model tuning. Ground truth stayed on the host, so nothing could be adapted to the test labels.

The approach is two stage:

  • We develop a clear way of evaluating the Model performance. For this, experts make a preliminary run using older versions of the LLMs, in order to fix the possible flaws of the scoring system and implementations. The scoring system for the scientific tasks cannot be developed in advance, since it has to include potential pitfalls of the solution, as evaluated by experts. The contract is also updated in order to create a commonly usable prompt.
  • We run a massive test using new generation LLM, with traces evaluated by experts. At this point the contract, scoring, and checks had already been fixed. We check the performance and create a reasoning results description.

We hope that this answer clarifies methodology, We will add this description into the Appendix F.

Reviewer_UTe5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents STEMBench, comprising twelve computational scientific tasks across high-energy physics, astrophysics, and neuroscience. Agents implement methods and produce submissions evaluated on withheld data, with executable baseline and expert references. The authors evaluate eight frontier models and examine task performance, domain differences, and submission validity.

Critical strengths
  1. The tasks contain substantive scientific challenges, including uncertainty estimation, physical inversion, and causal-response prediction.
  2. Runnable baseline and expert implementations provide useful performance references and support reproducibility.
  3. The standardized task contracts, offline evaluation, detailed prompts, and transparent reporting constitute a valuable infrastructure effort.
Critical weaknesses
  1. The interpretation of the anchor gap is insufficiently justified. The difference between a deliberately weak baseline and a reproduced expert method does not isolate scientific insight from implementation, optimization, or computational effort. Stronger generic baselines are needed to support the claimed interpretation of scientific capability.

  2. The scientific meaning of the composite scores needs stronger validation. Several metrics introduce custom weights and gates, while agreement with blinded expert rankings remains untested. Consequently, exceeding an expert anchor does not clearly establish a scientifically superior solution across the intended requirements.

  3. High scores are not clearly connected to complete, reproducible submissions. Many runs have negative or unavailable submission-validity flags. The paper should clarify how these outcomes affect the above-expert results and demonstrate that the principal successes correspond to compliant, rerunnable artifacts.

Questions for authors

I'm willing to change my score if my concerns are well addressed.

2 comments

We thank the reviewer for careful reading of the paper, please find our comments below.

Q1: Two anchors are shipped with every task: a baseline, obtained by a submission that uses no domain knowledge with minimal viable solution, and an expert reference, obtained by reproducing a published method. Both are organiser-side artefacts. They are never staged into the agent sandbox and play no role in an episode beyond fixing the scale. While we believe that most of the human-made solutions will fall into the gap between the baseline and expert solution, we want to stress that the expert solution is not the maximum reachable score in this task, in addition, the straightforward solution without scientific reasoning might have a very good score, which is governed by a lucky coincidence with the problem statement.

We thus agree that the statement about the isolation of scientific reasoning can be softened to “The baseline-to-expert gap provides a task-specific performance reference for assessing progress toward expert-level computational solutions”, which we now put instead of last sentence in the Section 4.4

Q2: The compositeness of scores indeed concerns the specific problem we are solving in this paper. One of the key differences of the STEM scientific tasks with respect to Science olympiads is the absence of clear established and community accepted winning solution measures. A successful solution should not only be ranked first in one measure (for example, ROC-AUC), but also fulfill other implied constraints, which are needed for results to become scientifically useful. Given that the scientific results should have valuable properties, it was suggested that the score is composite, where the compositeness not only reflects the «best» numeric solution, but also the properties needed for this solution to be scientifically sound. The scores are introduced in such a way that the scientifically best solution will have the best score in all score components.

We will describe this methodology in Appendix F.

Q3: The solutions are evaluated by experts, in particular those, which attained highest scores and have the clearest problems. From this point of view, we remain certain that the ranking represents the runnable solutions. Zero-scoring runs are included in all reported means; results are not conditioned on successful or validated submissions. Section 5 defines retained runs as those whose model dialogue completed without an infrastructure interruption. In Table 7, the New-physics search result of 0.3 ± 0.5 at medium effort includes all three retained runs; the “4/9” zero count covers all three effort levels combined. Likewise, Figure 2 shows means over three retained runs, with bars indicating their minimum and maximum. Zeros therefore contribute to the averages rather than being excluded to inflate performance. The paper reports both the score achieved by each run and whether its final submission passed the submission-validation checks. Section 5 explicitly distinguishes them and reports 168 affirmative, 64 negative, and 56 unavailable validity flags across the 288 retained runs. Our central numerical finding - that at least one model’s mean exceeds the expert reference on seven of twelve tasks - includes runs with negative or unavailable flags, rather than selecting only successful submissions. These results establish above-reference scores under the benchmark’s task metrics. Submission-validity flags provide a separate assessment of compliance; independent reproducibility cannot be inferred from the scores alone.

Thank you to the authors for the rebuttal and the additional clarifications. While I appreciate the effort put into addressing my questions, the primary weaknesses raised in my review still persist. As a result, I am inclined to keep my current score.

Reviewer_WfGcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper introduces STEMBench, a benchmark of 12 computational research tasks in high-energy physics, astrophysics, and neuroscience. Each task includes a scoring rule, a simple baseline, and an executable expert reference. Agents work offline, and submissions are evaluated on withheld targets.
  2. The authors evaluate eight models with three runs per task. At least one model’s mean score exceeds the expert reference on seven tasks. The remaining tasks expose gaps in uncertainty estimation, clinical classification, and latent or causal inference.
Critical strengths
  1. Executable evaluation with explicit references. Each task includes a grader, a baseline, and an expert implementation. This makes performance easier to interpret and audit than a score without a concrete reference.
  2. Useful coverage beyond classification. The benchmark includes uncertainty estimation, physical inversion, and causal-response prediction. These tasks test capabilities that predictive accuracy alone would miss. Withheld targets and offline execution also help control evaluation leakage.
Critical weaknesses
  1. The baseline-to-expert gap does not isolate scientific reasoning. Sections 3 and 4.4 interpret this gap as progress due to domain insight rather than engineering. However, several baselines are constant, midpoint, or zero predictors. Beating them could reflect generic statistical learning or optimization. The paper does not compare against strong generic pipelines under matched budgets or ablate domain-specific components. The anchors are useful performance references, but their interpretation as a measure of scientific reasoning is not established.
  2. A high score does not necessarily establish a scientifically complete solution. In Appendix B.2, the simulated Stokes score is 0.80R+0.20RFeff0.80R+0.20\sqrt{RF_{\mathrm{eff}}}. A non-working forward model receives Feff=0F_{\mathrm{eff}}=0, while retaining recovery credit. Thus, R>0.78R>0.78 can exceed the expert score of 0.624 without a working forward model. This is a property of the scoring rule, not evidence that a reported run exploited it. Component-level results are needed to interpret the above-reference scores. The planned expert-ranking validation in Section 4.6 is also unfinished.
Questions for authors
  1. What evidence separates domain insight from generic modeling and optimization in the baseline-to-expert gap? Do strong generic methods under matched budgets remain substantially below the expert references?
  2. For simulated Stokes runs that exceed the expert reference, what are RR, FF, and PphysP_{\mathrm{phys}}? Do these submissions implement working, physically valid forward models?
2 comments

We thank the reviewer for careful reading of the paper, please find our comments below.

General: The paper describes an approach to validate the performance of the leading Reasoning models in the STEM field. This approach includes putting two anchors to estimate performance of the model relative to human-researchers. Two anchors are shipped with every task: a baseline, obtained by a submission that uses no domain knowledge with minimal solution possible, and an expert reference, obtained by reproducing a published method. Both are organiser-side artefacts. They are never staged into the agent sandbox and play no role in an episode beyond fixing the scale. While we believe that most of the human-made solutions will fall into the gap between the baseline and expert solution, we want to stress that the expert solution is not the maximum reachable score in this task.

We thus agree that the statement about the isolation of scientific reasoning can be softened to “The baseline-to-expert gap provides a task-specific performance reference for assessing progress toward expert-level computational solutions”, which we now put instead of last sentence in the Section 4.4

Q1: By construction, we have chosen three separate domains with different approaches to results public description. While the LLM traces are studied by domain experts we noticed that the models have different preferences depending on domain. In High-Energy Physics, where the information is abundant, many models tend to use quite a lot of scientific insights, while neuroscience has lower exposure and thus models tend to make less “physically” meaningful results. The budget constrained solutions mostly prefer “simpler” algorithms (for example, random forest is preferred to transformer), however, the model insights like constructing physics meaningful variable transformation or keeping an eye to the conservation laws, when needed, thus representing reasoning skills. More detailed information is now put in Appendix E.

We will put discussion on methodology in the Appendix F.

Q2: Yes the submissions are working solutions, as we mentioned the traces were checked by the experts in the field and the solution genuinely represents a working physics solution. 14 simulated-Stokes runs exceeding the reference (out of 99 re-scored submissions) submit an executable forward model, with (R = 0.65–0.88, F = 0.29–1.00, median 0.96). 13 of them pass all six physical-validity checks (P_phys = 1). Conversely, none of the 14 runs with a missing or non-functional forward model exceeds the reference (best score 0.54). The reference itself is an earlier, simple CNN trained on the public data and paired with the exact generating forward model, so exceeding it mainly reflects better parameter recovery, which we report separately from F and P_phys.

Thank you to the authors for the response and additional clarifications. While these address some of my questions, my main concerns about the methodology and validation remain. I therefore maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.