- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LEAP-Bench, a benchmark intended to evaluate whether AI agents can extract quantitative results from materials-science experimental data, reason about those results, and connect them to scientific claims. The benchmark consists of 64 tasks and 475 questions constructed from 12 public papers and their associated experimental data in condensed-matter physics, materials chemistry, and electrochemistry. Each task begins from a central claim in a source paper and evaluates the key quantities, reasoning steps, and physical interpretations supporting that claim. Agents do not receive the source paper's conclusions or analysis procedures: they first analyze the experimental files and research objective, and then answer evaluation questions from their saved analysis.
The authors evaluate six model configurations using Claude Code and Codex CLI over three rounds. Claude Code with Opus 5 achieves the highest mean strict task-success rate, 59.4% +/- 7.2%, while 12 tasks remain unsolved by every configuration in every round. The authors report that correct answers about key quantities and intermediate reasoning can coexist with incomplete scientific interpretations. They additionally analyze performance by question type, compare against a no-data condition, test sensitivity to the language-model judge used for free-text answers, and conduct expert review on selected tasks. The paper positions LEAP-Bench as a way to diagnose experimental-analysis agents and as a possible basis for studying future training methods.
Critical strengths
-
The paper integrates the full path from experimental data to a scientific claim. LEAP-Bench uses experimental data from published materials-science studies and asks agents to inspect the files, select and execute analyses, derive quantitative results, interpret them physically, and connect them to a scientific claim. Although the individual components have precedents, combining them into an end-to-end evaluation grounded in real experimental data is novel and useful. It addresses an important capability for assessing how much of a scientific workflow an AI agent can perform autonomously.
-
The evidence-chain decomposition makes the evaluation diagnostically useful. By separating key quantities, analytical reasoning, and claim interpretation, the benchmark can identify where an agent fails rather than reporting only a final-answer score. In particular, the finding that an agent can obtain correct numerical or intermediate results while still producing an incomplete scientific interpretation is informative for evaluating current scientific agents.
Critical weaknesses
-
The benchmark's limited scope does not support the breadth of the headline claims. The 64 tasks are derived from only 12 selected papers, and tasks from the same paper share data, domain context, conventions, and reference conclusions. Their effective scientific diversity is therefore closer to 12 research cases than to 64 independent cases. Moreover, the questions and reference answers are constructed from the source papers' claims and evidence chains, so the benchmark primarily measures the ability to reconstruct the source analyses and conclusions rather than open-ended scientific discovery from data. The paper nevertheless extends its conclusions to end-to-end experimental analysis across materials science and to a foundation for training AI co-scientists. The authors should either frame the current collection explicitly as a pilot benchmark, narrow the claimed scope, or expand the number of independent papers per modality and evaluate paper-level generalization.
-
The evaluation does not sufficiently isolate model limitations from benchmark, input, harness, and scoring limitations. Forty-eight of the 64 tasks require at least one gap resolution through folding requirements into the analysis, supplying derived information, or rescoping the target claim. Question and reference-answer construction and verification are also largely model-based, while comprehensive expert validation of the final benchmark and a human or specialist baseline are absent. In addition, absolute task-success rates vary substantially across language-model judges, and the compared agents differ in harness, reasoning effort, and subagent availability. These factors make it difficult to determine whether failures or performance differences arise from the underlying models' scientific reasoning, insufficient inputs, task construction, the agent environment, or the scoring procedure. Consequently, the broad conclusions about current AI's scientific-analysis capability are not yet sufficiently supported.
Questions for authors
-
Clarification: Is LEAP-Bench intended to evaluate reconstruction of the claims and evidence chains reported in the source papers, or the ability to derive any scientifically defensible conclusion from the supplied data, including conclusions that differ from the source paper? If the latter is in scope, how are alternative analysis procedures and scientifically valid interpretations that differ from the stored reference answer accepted during scoring?
-
Clarification: Can relevant domain experts solve the tasks and questions using exactly the same inputs provided to the agents? In particular, have experts been shown to solve the 12 tasks that remained unsolved by all model configurations and rounds, and how closely do their answers agree with the benchmark references? This would materially affect whether the observed failures can be attributed to the agents rather than to insufficient inputs or task construction.
-
Clarification: What supports treating 64 correlated tasks from 12 selected papers as representative of end-to-end experimental analysis across materials science? Do the main conclusions remain when results are aggregated at the paper or experimental-modality level? Would the authors instead consider positioning the current collection explicitly as an exploratory pilot benchmark?
Comprehensive expert baselines, comparisons under matched agent conditions, and paper-level generalization to unseen studies would materially reduce the concerns about benchmark validity and causal attribution.
7 comments
Response to Reviewer sJFK (Part 2/6)
(b) Regarding the scope of the claims (W1)
- Agreement with the reviewer. Tasks from the same paper are not independent. The independent unit is the paper, and LEAP-Bench comprises 12 research cases, one per paper, with 64 tasks.
- Our choice among the three options of the review. The reviewer offers three options: a pilot benchmark, a narrower claimed scope, or more independent papers with an evaluation at the level of the paper. We narrow the claimed scope, and we add an evaluation paper by paper with additional papers (section c).
- Reason for this scope. An agent unable to recover a known finding from the data is not ready for an unknown one.
- Revision of the paper. Abstract and introduction will state the scope: the recovery of established, peer-reviewed findings from the underlying measurements, on 12 research cases from condensed-matter physics, materials chemistry and electrochemistry. The training of AI co-scientists will be stated as our motivation.
(c) Regarding results at the level of the paper and generalization to new papers (W1, Q3)
The reviewer asks for the main conclusions with the results aggregated by paper or by experimental modality. RTable S1 recomputes the submitted results paper by paper, so that each paper counts once, in two ways: the average of the 12 per-paper success rates, and the result with one paper left out at a time. The confidence intervals come from a bootstrap that resamples whole papers, so they account for the correlation between tasks of the same paper. RTable S2 gives the results by type of measurement.
RTable S1: Task success of the six agents with the paper as the unit of analysis (submitted results, three rounds; 95% confidence intervals from a bootstrap over the 12 papers)
| Agent | Task success over all tasks | Average of the 12 per-paper success rates | Task success with one paper left out (lowest and highest of the 12 values) |
|---|---|---|---|
| Opus 5 | 59.4% [48.1, 70.4] | 58.7% [46.7, 69.9] | 56.3% to 62.3% |
| Sol | 38.0% [28.6, 49.3] | 41.7% [30.1, 55.7] | 34.5% to 41.2% |
| Sonnet 5 | 20.8% [13.1, 30.0] | 23.2% [13.1, 33.6] | 18.6% to 23.0% |
| Terra | 19.3% [9.4, 31.4] | 24.4% [11.4, 41.1] | 16.7% to 22.4% |
| Luna | 16.7% [8.4, 26.8] | 22.2% [12.1, 33.3] | 14.9% to 19.4% |
| Haiku 4.5 | 0.5% [0.0, 1.7] | 0.5% [0.0, 1.4] | 0.0% to 0.6% |
RTable S2: Task success by type of measurement (submitted results, three rounds; values in %; a task can involve several types)
| Type of measurement (papers, tasks) | Opus 5 | Sol | Sonnet 5 | Terra | Luna | Haiku 4.5 |
|---|---|---|---|---|---|---|
| Electrical transport (6, 24) | 68.1 | 27.8 | 25.0 | 9.7 | 12.5 | 0.0 |
| Microscopy and scanning probes (7, 15) | 62.2 | 51.1 | 22.2 | 24.4 | 20.0 | 2.2 |
| Spectroscopy (5, 11) | 42.4 | 30.3 | 15.2 | 21.2 | 18.2 | 0.0 |
| Scattering and diffraction (2, 6) | 72.2 | 44.4 | 11.1 | 16.7 | 5.6 | 0.0 |
| Bulk magnetic and thermal (2, 6) | 38.9 | 16.7 | 16.7 | 16.7 | 16.7 | 0.0 |
| Electrochemistry (2, 8) | 54.2 | 58.3 | 16.7 | 25.0 | 16.7 | 0.0 |
Conclusion for Q3.
- Opus 5 stays ahead of Sol with the paper as the unit. Opus 5 is ahead of Sol on 10 of the 12 papers, first in all 12 comparisons with one paper left out, and first in 5 of the 6 types of measurement. Its lead over Sol is 21.4 points over all tasks (95% confidence interval 10.9 to 30.4) and 17.1 points between the per-paper averages (interval 1.7 to 29.9).
- Every type of measurement leaves room for improvement. No agent exceeds 72.2% in any type.
Five additional papers outside the benchmark (W1). During the response period, we built tasks from five additional papers with the same, unchanged pipeline and evaluated the two strongest agents on them. These papers were chosen before any agent result on them was seen, and they are not added to the main results.
RTable S3: Task success on five additional papers outside the benchmark (three rounds; solved attempts / all attempts)
| Additional paper: material and measurement | Tasks / questions | Opus 5 | Sol |
|---|---|---|---|
| WSe2: scanning single-electron transistor | 5 / 41 | 2/15 (13.3%) | 0/15 (0.0%) |
| Organic kagome lattice: STM, STS, nc-AFM | 6 / 50 | 3/18 (16.7%) | 1/18 (5.6%) |
| MoS2: electron imaging | 3 / 20 | 6/9 (66.7%) | 4/9 (44.4%) |
| Oxide: XMCD, XANES, magnetization | 6 / 44 | 15/18 (83.3%) | 8/18 (44.4%) |
| Josephson-junction circuit: microwave spectroscopy | 5 / 29 | 11/15 (73.3%) | 5/15 (33.3%) |
| All five papers | 25 / 184 | 37/75 (49.3%) | 18/75 (24.0%) |
Conclusion for W1. The lead of Opus 5 and the large room for improvement carry over to papers outside the benchmark. Opus 5 is ahead of Sol on all five papers, and the room for improvement stays large, with 49.3% and 24.0% of the attempts solved.
Response to Reviewer sJFK (Part 3/6)
(d) Regarding construction with language models and validation by experts (W2)
The reviewer notes that questions and reference answers were built and verified largely by language models, and that a comprehensive expert validation of the final benchmark was absent. Domain experts have now reviewed the benchmark. Eight domain experts had the source paper, the published data files and the benchmark materials in front of them, and reviewed tasks and questions across all 12 papers. For each reviewed question, they judged the scientific validity of the reference answer and the suitability of the scoring criterion (the rule for accepting an agent's answer, such as the tolerance for a number or the required facts for a free-text answer). For each reviewed task, they also judged its clarity and the sufficiency of the information given to the agent.
RTable S4: Verdicts of the domain experts on the reference answers and on the scoring criteria, by type of question (number of questions; not shown: 3 reference answers and 11 scoring criteria without a clear verdict)
| Type of question | Reference answers reviewed | Reference answer valid | Reference answer needs correction | Scoring criteria reviewed | Criterion appropriate | Criterion too lenient | Criterion too strict |
|---|---|---|---|---|---|---|---|
| Key quantities | 170 | 162 (95.3%) | 7 | 170 | 143 (84.1%) | 21 | 3 |
| Reasoning steps | 149 | 145 (97.3%) | 4 | 144 | 137 (95.1%) | 0 | 0 |
| Claim interpretation | 56 | 52 (92.9%) | 2 | 55 | 52 (94.5%) | 0 | 2 |
| All reviewed questions | 375 | 359 (95.7%) | 13 (3.5%) | 369 | 332 (90.0%) | 21 (5.7%) | 5 (1.4%) |
Results without the items flagged by the experts. The experts flagged 24 questions: for each of them, the reference answer needs correction or could not be confirmed, a second answer is also valid, or the scoring criterion is too strict. They also found a necessary piece of information missing in 7 tasks. We removed these questions and tasks and recomputed the submitted results.
RTable S5: Task success of the six agents without the questions and tasks flagged by the experts (task success in %, three rounds)
| Items counted | Tasks | Opus 5 | Sol | Sonnet 5 | Terra | Luna | Haiku 4.5 |
|---|---|---|---|---|---|---|---|
| 1. All items, as submitted | 64 | 59.4 | 38.0 | 20.8 | 19.3 | 16.7 | 0.5 |
| 2. Without the 24 flagged questions and the 7 tasks with missing information | 57 | 63.2 | 42.1 | 24.6 | 22.8 | 21.1 | 0.6 |
| 3. Only the 23 tasks with a complete expert review and only positive verdicts | 23 | 72.5 | 52.2 | 29.0 | 21.7 | 24.6 | 1.4 |
Tasks with a gap resolution (W2). Some tasks need something absent from the published files. When we built such a task, we closed the gap in one of three ways: the missing analysis step became part of the agent's work, a missing constant or annotation was supplied, or the target claim was narrowed to the level supported by the data. The paper calls each such decision a gap resolution, and 48 of the 64 tasks have at least one. The reviewer's concern is that failures on these tasks may come from the construction and not from the agents. RTable S6 compares the two groups of tasks.
RTable S6: Task success on tasks with and without a gap resolution during construction (submitted results, three rounds; values in %)
| Tasks | Number of tasks | Opus 5 | Sol | Sonnet 5 | Terra | Luna | Haiku 4.5 | All six agents |
|---|---|---|---|---|---|---|---|---|
| With at least one gap resolution | 48 | 59.0 | 37.5 | 24.3 | 20.1 | 18.8 | 0.7 | 26.7 |
| Without gap resolution | 16 | 60.4 | 39.6 | 10.4 | 16.7 | 10.4 | 0.0 | 22.9 |
Conclusion for W2 (construction and validation).
- Domain experts have now reviewed the benchmark across all 12 papers. They find 95.7% of the 375 reviewed reference answers valid, and they judge none of the 53 reviewed tasks scientifically flawed. Their remarks, such as 13 reference answers to correct and 7 tasks with missing information, will go into the revision.
- The construction with language models starts from published claims and published data, and it extends to new papers. Every task rests on a peer-reviewed claim and on the data of the source authors. The same pipeline produced the tasks of five additional papers within the response period.
- The main result holds without the flagged items. Without them, Opus 5 moves from 59.4% to 63.2%, and the order of the six agents stays the same (RTable S5, row 2).
- Over the six agents together, tasks with a gap resolution are solved about as often as tasks without (26.7% and 22.9%). The failures are therefore not concentrated in the tasks with a gap resolution.
Response to Reviewer sJFK (Part 1/6)
We sincerely thank Reviewer sJFK for the rigorous and constructive review. We are encouraged by the reviewer's assessment: the paper "integrates the full path from experimental data to a scientific claim", an end-to-end evaluation "grounded in real experimental data is novel and useful", and the separation of key quantities, reasoning and interpretation makes the evaluation "diagnostically useful".
Regarding the timing of this response. We want to first apologize for the timing of this response and ask for the reviewer's kind understanding. The response required many new experiments and a review of the benchmark by eight domain experts across all 12 papers, and each paper requires deep knowledge of its own measurement technique.
Regarding our core vision. AI agents are entering the laboratory, and the field has no benchmark today that measures their ability to turn raw experimental measurements into defensible scientific claims. LEAP-Bench is built to close this gap.
The reviewer's closing sentence names three additions able to "materially reduce the concerns". We answer each of them with new evidence:
(a) Regarding the target of the evaluation and valid alternatives (Q1)