acceptodds
Under review as a conference paper at ICLR 2027

LEAP-Bench: Turning Raw Data into Scientific Claims

Abstract

Scientific discovery depends on turning raw experimental data into defensible claims through quantitative analysis and physical interpretation. Whether AI agents can carry out this process reliably remains unclear. We introduce LEAP-Bench, a benchmark of end-to-end experimental analysis in materials science. We construct 64 tasks from 12 papers spanning multiple instrument modalities, each task targeting a central scientific claim. Agents analyze raw experimental files without seeing the papers' conclusions or procedures. Using reference answers from the source papers, 475 questions assess whether agents extract the required quantities, correctly reason through the analysis, and justify their scientific conclusions. Across three evaluation rounds, Claude Code with Opus 5 achieves the highest task success rate among the six evaluated agents, at (mean standard deviation). Even when agents correctly answer questions about numerical results and intermediate reasoning, their scientific interpretations can remain incomplete. Twelve tasks remain unsolved by all six agents across all three rounds, including eight requiring model fitting or comparison across measurements. These findings highlight persistent difficulties in completing experimental analyses and connecting the results to scientific conclusions. By evaluating the reasoning from experimental data to scientific claims, LEAP-Bench provides a basis for training AI co-scientists with feedback on their numerical estimates, analytical reasoning, and physical interpretations.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
benchmark, coding agents, scientific reasoning, raw instrument data, materials science, agent benchmark, experimental analysis, autonomous discovery, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sJFKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LEAP-Bench, a benchmark intended to evaluate whether AI agents can extract quantitative results from materials-science experimental data, reason about those results, and connect them to scientific claims. The benchmark consists of 64 tasks and 475 questions constructed from 12 public papers and their associated experimental data in condensed-matter physics, materials chemistry, and electrochemistry. Each task begins from a central claim in a source paper and evaluates the key quantities, reasoning steps, and physical interpretations supporting that claim. Agents do not receive the source paper's conclusions or analysis procedures: they first analyze the experimental files and research objective, and then answer evaluation questions from their saved analysis.

The authors evaluate six model configurations using Claude Code and Codex CLI over three rounds. Claude Code with Opus 5 achieves the highest mean strict task-success rate, 59.4% +/- 7.2%, while 12 tasks remain unsolved by every configuration in every round. The authors report that correct answers about key quantities and intermediate reasoning can coexist with incomplete scientific interpretations. They additionally analyze performance by question type, compare against a no-data condition, test sensitivity to the language-model judge used for free-text answers, and conduct expert review on selected tasks. The paper positions LEAP-Bench as a way to diagnose experimental-analysis agents and as a possible basis for studying future training methods.

Critical strengths
  1. The paper integrates the full path from experimental data to a scientific claim. LEAP-Bench uses experimental data from published materials-science studies and asks agents to inspect the files, select and execute analyses, derive quantitative results, interpret them physically, and connect them to a scientific claim. Although the individual components have precedents, combining them into an end-to-end evaluation grounded in real experimental data is novel and useful. It addresses an important capability for assessing how much of a scientific workflow an AI agent can perform autonomously.

  2. The evidence-chain decomposition makes the evaluation diagnostically useful. By separating key quantities, analytical reasoning, and claim interpretation, the benchmark can identify where an agent fails rather than reporting only a final-answer score. In particular, the finding that an agent can obtain correct numerical or intermediate results while still producing an incomplete scientific interpretation is informative for evaluating current scientific agents.

Critical weaknesses
  1. The benchmark's limited scope does not support the breadth of the headline claims. The 64 tasks are derived from only 12 selected papers, and tasks from the same paper share data, domain context, conventions, and reference conclusions. Their effective scientific diversity is therefore closer to 12 research cases than to 64 independent cases. Moreover, the questions and reference answers are constructed from the source papers' claims and evidence chains, so the benchmark primarily measures the ability to reconstruct the source analyses and conclusions rather than open-ended scientific discovery from data. The paper nevertheless extends its conclusions to end-to-end experimental analysis across materials science and to a foundation for training AI co-scientists. The authors should either frame the current collection explicitly as a pilot benchmark, narrow the claimed scope, or expand the number of independent papers per modality and evaluate paper-level generalization.

  2. The evaluation does not sufficiently isolate model limitations from benchmark, input, harness, and scoring limitations. Forty-eight of the 64 tasks require at least one gap resolution through folding requirements into the analysis, supplying derived information, or rescoping the target claim. Question and reference-answer construction and verification are also largely model-based, while comprehensive expert validation of the final benchmark and a human or specialist baseline are absent. In addition, absolute task-success rates vary substantially across language-model judges, and the compared agents differ in harness, reasoning effort, and subagent availability. These factors make it difficult to determine whether failures or performance differences arise from the underlying models' scientific reasoning, insufficient inputs, task construction, the agent environment, or the scoring procedure. Consequently, the broad conclusions about current AI's scientific-analysis capability are not yet sufficiently supported.

Questions for authors
  1. Clarification: Is LEAP-Bench intended to evaluate reconstruction of the claims and evidence chains reported in the source papers, or the ability to derive any scientifically defensible conclusion from the supplied data, including conclusions that differ from the source paper? If the latter is in scope, how are alternative analysis procedures and scientifically valid interpretations that differ from the stored reference answer accepted during scoring?

  2. Clarification: Can relevant domain experts solve the tasks and questions using exactly the same inputs provided to the agents? In particular, have experts been shown to solve the 12 tasks that remained unsolved by all model configurations and rounds, and how closely do their answers agree with the benchmark references? This would materially affect whether the observed failures can be attributed to the agents rather than to insufficient inputs or task construction.

  3. Clarification: What supports treating 64 correlated tasks from 12 selected papers as representative of end-to-end experimental analysis across materials science? Do the main conclusions remain when results are aggregated at the paper or experimental-modality level? Would the authors instead consider positioning the current collection explicitly as an exploratory pilot benchmark?

Comprehensive expert baselines, comparisons under matched agent conditions, and paper-level generalization to unseen studies would materially reduce the concerns about benchmark validity and causal attribution.

7 comments
AuthorsOpenReview ↗

Response to Reviewer sJFK (Part 1/6)

We sincerely thank Reviewer sJFK for the rigorous and constructive review. We are encouraged by the reviewer's assessment: the paper "integrates the full path from experimental data to a scientific claim", an end-to-end evaluation "grounded in real experimental data is novel and useful", and the separation of key quantities, reasoning and interpretation makes the evaluation "diagnostically useful".

Regarding the timing of this response. We want to first apologize for the timing of this response and ask for the reviewer's kind understanding. The response required many new experiments and a review of the benchmark by eight domain experts across all 12 papers, and each paper requires deep knowledge of its own measurement technique.

Regarding our core vision. AI agents are entering the laboratory, and the field has no benchmark today that measures their ability to turn raw experimental measurements into defensible scientific claims. LEAP-Bench is built to close this gap.

The reviewer's closing sentence names three additions able to "materially reduce the concerns". We answer each of them with new evidence:

  • "Comprehensive expert baselines": eight domain experts reviewed the benchmark across all 12 papers, in place of a study with experts solving the tasks, and they confirm 95.7% of the 375 reviewed reference answers;
  • "comparisons under matched agent conditions": controlled comparisons inside each harness, with one change at a time, show changes of task success of at most 5.9 points for the tested settings, and we make no claim about models alone;
  • "paper-level generalization to unseen studies": Opus 5 stays ahead of Sol on 10 of the 12 papers and on all five additional papers.

(a) Regarding the target of the evaluation and valid alternatives (Q1)

  • Our answer to Q1. LEAP-Bench evaluates the recovery of the claims of the published paper from the underlying measurements. The source paper gives the conclusion, and the agent is free to choose the analysis.
  • Reason for this target. A peer-reviewed claim with published measurements is a correct answer independent of us, the benchmark builders. In our matched control without the data files, neither of the two strongest agents solves a single task (0 of 192 attempts each).
  • Alternative analysis procedures. The scoring checks only the answers and not the method, so any analysis procedure with correct answers is accepted.
  • Alternative valid answers. The domain experts marked a second valid answer for 5 of the 375 reviewed reference answers (1.3%) and a too-strict scoring criterion for 5 of 369 (1.4%). The revision will add the accepted alternatives for each of them.
AuthorsOpenReview ↗

Response to Reviewer sJFK (Part 2/6)

(b) Regarding the scope of the claims (W1)

  • Agreement with the reviewer. Tasks from the same paper are not independent. The independent unit is the paper, and LEAP-Bench comprises 12 research cases, one per paper, with 64 tasks.
  • Our choice among the three options of the review. The reviewer offers three options: a pilot benchmark, a narrower claimed scope, or more independent papers with an evaluation at the level of the paper. We narrow the claimed scope, and we add an evaluation paper by paper with additional papers (section c).
  • Reason for this scope. An agent unable to recover a known finding from the data is not ready for an unknown one.
  • Revision of the paper. Abstract and introduction will state the scope: the recovery of established, peer-reviewed findings from the underlying measurements, on 12 research cases from condensed-matter physics, materials chemistry and electrochemistry. The training of AI co-scientists will be stated as our motivation.

(c) Regarding results at the level of the paper and generalization to new papers (W1, Q3)

The reviewer asks for the main conclusions with the results aggregated by paper or by experimental modality. RTable S1 recomputes the submitted results paper by paper, so that each paper counts once, in two ways: the average of the 12 per-paper success rates, and the result with one paper left out at a time. The confidence intervals come from a bootstrap that resamples whole papers, so they account for the correlation between tasks of the same paper. RTable S2 gives the results by type of measurement.

RTable S1: Task success of the six agents with the paper as the unit of analysis (submitted results, three rounds; 95% confidence intervals from a bootstrap over the 12 papers)

AgentTask success over all tasksAverage of the 12 per-paper success ratesTask success with one paper left out (lowest and highest of the 12 values)
Opus 559.4% [48.1, 70.4]58.7% [46.7, 69.9]56.3% to 62.3%
Sol38.0% [28.6, 49.3]41.7% [30.1, 55.7]34.5% to 41.2%
Sonnet 520.8% [13.1, 30.0]23.2% [13.1, 33.6]18.6% to 23.0%
Terra19.3% [9.4, 31.4]24.4% [11.4, 41.1]16.7% to 22.4%
Luna16.7% [8.4, 26.8]22.2% [12.1, 33.3]14.9% to 19.4%
Haiku 4.50.5% [0.0, 1.7]0.5% [0.0, 1.4]0.0% to 0.6%

RTable S2: Task success by type of measurement (submitted results, three rounds; values in %; a task can involve several types)

Type of measurement (papers, tasks)Opus 5SolSonnet 5TerraLunaHaiku 4.5
Electrical transport (6, 24)68.127.825.09.712.50.0
Microscopy and scanning probes (7, 15)62.251.122.224.420.02.2
Spectroscopy (5, 11)42.430.315.221.218.20.0
Scattering and diffraction (2, 6)72.244.411.116.75.60.0
Bulk magnetic and thermal (2, 6)38.916.716.716.716.70.0
Electrochemistry (2, 8)54.258.316.725.016.70.0

Conclusion for Q3.

  • Opus 5 stays ahead of Sol with the paper as the unit. Opus 5 is ahead of Sol on 10 of the 12 papers, first in all 12 comparisons with one paper left out, and first in 5 of the 6 types of measurement. Its lead over Sol is 21.4 points over all tasks (95% confidence interval 10.9 to 30.4) and 17.1 points between the per-paper averages (interval 1.7 to 29.9).
  • Every type of measurement leaves room for improvement. No agent exceeds 72.2% in any type.

Five additional papers outside the benchmark (W1). During the response period, we built tasks from five additional papers with the same, unchanged pipeline and evaluated the two strongest agents on them. These papers were chosen before any agent result on them was seen, and they are not added to the main results.

RTable S3: Task success on five additional papers outside the benchmark (three rounds; solved attempts / all attempts)

Additional paper: material and measurementTasks / questionsOpus 5Sol
WSe2: scanning single-electron transistor5 / 412/15 (13.3%)0/15 (0.0%)
Organic kagome lattice: STM, STS, nc-AFM6 / 503/18 (16.7%)1/18 (5.6%)
MoS2: electron imaging3 / 206/9 (66.7%)4/9 (44.4%)
Oxide: XMCD, XANES, magnetization6 / 4415/18 (83.3%)8/18 (44.4%)
Josephson-junction circuit: microwave spectroscopy5 / 2911/15 (73.3%)5/15 (33.3%)
All five papers25 / 18437/75 (49.3%)18/75 (24.0%)

Conclusion for W1. The lead of Opus 5 and the large room for improvement carry over to papers outside the benchmark. Opus 5 is ahead of Sol on all five papers, and the room for improvement stays large, with 49.3% and 24.0% of the attempts solved.

AuthorsOpenReview ↗

Response to Reviewer sJFK (Part 3/6)

(d) Regarding construction with language models and validation by experts (W2)

The reviewer notes that questions and reference answers were built and verified largely by language models, and that a comprehensive expert validation of the final benchmark was absent. Domain experts have now reviewed the benchmark. Eight domain experts had the source paper, the published data files and the benchmark materials in front of them, and reviewed tasks and questions across all 12 papers. For each reviewed question, they judged the scientific validity of the reference answer and the suitability of the scoring criterion (the rule for accepting an agent's answer, such as the tolerance for a number or the required facts for a free-text answer). For each reviewed task, they also judged its clarity and the sufficiency of the information given to the agent.

RTable S4: Verdicts of the domain experts on the reference answers and on the scoring criteria, by type of question (number of questions; not shown: 3 reference answers and 11 scoring criteria without a clear verdict)

Type of questionReference answers reviewedReference answer validReference answer needs correctionScoring criteria reviewedCriterion appropriateCriterion too lenientCriterion too strict
Key quantities170162 (95.3%)7170143 (84.1%)213
Reasoning steps149145 (97.3%)4144137 (95.1%)00
Claim interpretation5652 (92.9%)25552 (94.5%)02
All reviewed questions375359 (95.7%)13 (3.5%)369332 (90.0%)21 (5.7%)5 (1.4%)

Results without the items flagged by the experts. The experts flagged 24 questions: for each of them, the reference answer needs correction or could not be confirmed, a second answer is also valid, or the scoring criterion is too strict. They also found a necessary piece of information missing in 7 tasks. We removed these questions and tasks and recomputed the submitted results.

RTable S5: Task success of the six agents without the questions and tasks flagged by the experts (task success in %, three rounds)

Items countedTasksOpus 5SolSonnet 5TerraLunaHaiku 4.5
1. All items, as submitted6459.438.020.819.316.70.5
2. Without the 24 flagged questions and the 7 tasks with missing information5763.242.124.622.821.10.6
3. Only the 23 tasks with a complete expert review and only positive verdicts2372.552.229.021.724.61.4

Tasks with a gap resolution (W2). Some tasks need something absent from the published files. When we built such a task, we closed the gap in one of three ways: the missing analysis step became part of the agent's work, a missing constant or annotation was supplied, or the target claim was narrowed to the level supported by the data. The paper calls each such decision a gap resolution, and 48 of the 64 tasks have at least one. The reviewer's concern is that failures on these tasks may come from the construction and not from the agents. RTable S6 compares the two groups of tasks.

RTable S6: Task success on tasks with and without a gap resolution during construction (submitted results, three rounds; values in %)

TasksNumber of tasksOpus 5SolSonnet 5TerraLunaHaiku 4.5All six agents
With at least one gap resolution4859.037.524.320.118.80.726.7
Without gap resolution1660.439.610.416.710.40.022.9

Conclusion for W2 (construction and validation).

  • Domain experts have now reviewed the benchmark across all 12 papers. They find 95.7% of the 375 reviewed reference answers valid, and they judge none of the 53 reviewed tasks scientifically flawed. Their remarks, such as 13 reference answers to correct and 7 tasks with missing information, will go into the revision.
  • The construction with language models starts from published claims and published data, and it extends to new papers. Every task rests on a peer-reviewed claim and on the data of the source authors. The same pipeline produced the tasks of five additional papers within the response period.
  • The main result holds without the flagged items. Without them, Opus 5 moves from 59.4% to 63.2%, and the order of the six agents stays the same (RTable S5, row 2).
  • Over the six agents together, tasks with a gap resolution are solved about as often as tasks without (26.7% and 22.9%). The failures are therefore not concentrated in the tasks with a gap resolution.
Reviewer_FchtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on creating a benchmark for end-to-end experimental analysis in materials science. Specifically, the benchmark consists of 64 tasks and 475 questions from 12 papers. Each task focuses on a scientific claim, and agents need to use experimental files to reason through the data and reach the same scientific conclusions. The benchmark uses a two-phase evaluation design, where agents first analyze the experimental data and then answer evaluation questions. The evaluation uses both deterministic scoring and an LLM judge for interpreting responses that cannot be evaluated using deterministic scoring. Results show that Claude Code with Opus 5 achieves the highest performance, with a mean task success rate of 59.4%, while 12 tasks remain unsolved across all evaluated models and runs. The findings show that agents generally struggle with scientific interpretation, even when they correctly answer questions about numerical results and intermediate reasoning.

Critical strengths
  • An insightful benchmark that highlights an important problem that existing benchmarks have not fully addressed. This could provide a better understanding of how useful agents are in the materials science domain and how models reason in this domain.

  • Good benchmark design and useful diagnostics. Specifically, I like the two-phase evaluation design, where agents analyze the experimental data before receiving the evaluation questions. I also like the combination of deterministic scoring and an LLM judge for evaluating open-ended answers.

  • Good, thorough analysis of what the models are lacking, along with useful robustness evaluations and ablations. I particularly appreciate that the authors evaluate multiple runs, investigate different scoring criteria, and analyze the consistency of LLM judges.

  • I also like that the benchmark covers different experimental techniques in materials science rather than focusing on only one type of experiment. This could help researchers understand how agents perform across different scientific analysis tasks.

Critical weaknesses
  • This benchmark is definitely insightful and helpful, but I feel like the tasks are relatively curated compared to real-life scenarios for a benchmark claiming to evaluate end-to-end experimental analysis in materials science. In real-life situations, there would be a lot of experimental files and logs, and it's important for models to navigate the messiness of the data, identify the relevant measurements, and determine the correct scientific conclusions. I'm not sure whether the current benchmark fully captures this part of experimental analysis.
  • I also have concerns about whether the reference answers can actually be derived from the given input data and whether each reference answer has only one scientifically valid conclusion. Since the reference answers are derived from published scientific claims, it is possible that some experimental data could support multiple reasonable interpretations. If the data is insufficient or supports multiple interpretations, some questions might be impossible to answer correctly, or scientifically reasonable answers might be marked as wrong. This makes me question whether the benchmark always evaluates scientific correctness or sometimes just agreement with the original paper's conclusions.
  • Another concern is that only a limited number of models were tested. The authors only evaluated two model families (Claude and GPT), which may limit how well the findings generalize to other models. Additionally, the two model families use different agent harnesses and configurations, which makes it difficult to determine whether the performance differences come from the models themselves or their execution environments.
  • Although the authors state that the benchmark, construction pipeline, and per-system evaluation harness will be released upon publication, I feel that without access to these materials during the review process, I cannot independently verify whether the evaluation setup is valid or reproduce the reported results.
  • I also have some concerns about the reliability of the LLM-as-a-judge evaluation. Although the authors investigate judge robustness and report that the benchmark judge agrees with the panel's median verdict on 89.8% of answers, the task success rates still vary significantly depending on which LLM judge is used. Looking at Table 24, Opus 5's first-round task success rate ranges from 37.5% to 73.4% across different judges. This seems like a large difference when evaluating the same agent responses. While the high judge agreement is encouraging, it's still unclear which judge provides a more scientifically accurate evaluation without independent human-expert validation.
Questions for authors
  1. What criteria were used to choose the 12 papers to represent different materials science domains? How did the authors ensure that the selected papers cover different levels of difficulty? Given the relatively small number of source papers, how representative is this benchmark of materials science research overall?

  2. Is it possible that the raw experimental datasets are missing files or necessary information, making it impossible for the models to reach the same final conclusions claimed in the papers? Have human experts independently verified that the tasks can be solved using only the provided experimental data?

  3. Is it possible that some claims are inconclusive based on the given raw experimental data, but an agent's answer is still marked as wrong? How does the evaluation handle cases where the agent reaches a scientifically valid conclusion that differs from the reference answer?

  4. Have the authors validated the LLM judge's decisions against human domain experts? Given the large differences in task success rates across judges in Table 24, how can we determine which judge provides the most scientifically accurate evaluation?

  5. In materials science, is it possible that the agent harness doesn't provide all the necessary tools, since the paper mainly describes image inspection and code execution? Were all agents provided with the specialized scientific libraries and instrument-file readers needed for the different experimental techniques?

  6. Have the authors considered evaluating human domain experts on a subset of the benchmark? This could help establish whether the tasks are reasonably solvable and provide a useful reference for interpreting agent performance.

  7. Beyond the comparison in Table 1, what additional insights does LEAP-Bench provide compared to existing scientific reasoning and analysis benchmarks? Are there specific limitations or failure modes of AI agents that LEAP-Bench can identify but existing benchmarks cannot?

  8. Can the authors share the exact agent and judge prompts, benchmark tasks, and evaluation harness during the review process rather than only upon publication? This would help reviewers independently verify the experimental setup and reported results.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Fcht (Part 1/6)

We sincerely thank Reviewer Fcht for the thorough and constructive review. We are encouraged that the reviewer sees LEAP-Bench as an "insightful benchmark that highlights an important problem that existing benchmarks have not fully addressed". We also value the appreciation of "the two-phase evaluation design", of "the combination of deterministic scoring and an LLM judge" and of the coverage of "different experimental techniques in materials science".

Regarding the timing of this response. We want to first apologize for the timing of this response and ask for the reviewer's kind understanding. The response required many new experiments and a review of the benchmark by eight domain experts across all 12 papers, and each paper requires deep knowledge of its own measurement technique.

Regarding our core vision. AI agents are entering the laboratory, and the field has no benchmark today that measures their ability to turn raw experimental measurements into defensible scientific claims. LEAP-Bench is built to close this gap. We answer the five weaknesses (W1 to W5) and the eight questions (Q1 to Q8) of the review in the reviewer's own order.


(a) Regarding the curation of the inputs (W1)

The reviewer points out that a real laboratory has many files and logs, and that finding the relevant measurements is part of the work. In LEAP-Bench, the agent receives the data folder of the paper, with the files of all tasks of this paper and with files of no use for any task. Over the 12 papers, these folders hold 1,237 data files in about 20 formats. A task needs a median of 4 of the 39 files (median) in the folder of its paper, and in some tasks one listed file is a measurement database with many runs. The submitted benchmark adds one form of help, a list of the files needed for each task. To come closer to the reviewer's scenario, we ran the same tasks without this list, so the agent has to find its files among all files of the paper.

RTable F1: Task success with and without the list of relevant files, by size of the data package (new runs for this response; Sol three rounds, Opus 5 one round; the split by size was fixed before the runs)

PapersAgentTask attemptsWith the file list (new run)Without the file listDifference [95% interval]
All 12 papersOpus 56439 (60.9%)37 (57.8%)−3.1 points [−15.2, +8.2]
All 12 papersSol19265 (33.9%)63 (32.8%)−1.0 points [−10.7, +8.3]
5 papers with a large data package (88 to 425 files)Opus 52416 (66.7%)11 (45.8%)−20.8 points [−40.0, 0.0]
5 papers with a large data package (88 to 425 files)Sol7228 (38.9%)19 (26.4%)−12.5 points [−24.1, −5.8]
7 papers with a smaller data package (5 to 46 files)Opus 54023 (57.5%)26 (65.0%)+7.5 points [−3.2, +21.2]
7 papers with a smaller data package (5 to 46 files)Sol12037 (30.8%)44 (36.7%)+5.8 points [−7.1, +20.8]

Conclusion for W1.

  • The reviewer's concern is confirmed on large data packages. Without the list, task success on the 5 papers with the most files falls by 12.5 points for Sol and by 20.8 points for Opus 5 (one round, with an interval that reaches 0.0). On the 7 papers with smaller packages, task success does not fall.
  • The revised benchmark will add this second version without the file list, closer to the laboratory situation described by the reviewer. On large data packages it is harder, so the paper understates the remaining difficulty there.
AuthorsOpenReview ↗

Response to Reviewer Fcht (Part 2/6)

(b) Regarding the derivation of the reference answers from the supplied data, and alternative valid conclusions (W2, Q2, Q3)

The reviewer raises three related points: a reference answer may not be derivable from the files given to the agent (W2, Q2), a claim may be inconclusive on these data, and a scientifically valid answer different from the reference answer may be marked wrong (Q3). We tested these points in three ways.

Review by domain experts. Eight domain experts had the source paper, the published data files and the benchmark materials in front of them. For each reviewed question, they judged the scientific validity of the reference answer and the suitability of the scoring criterion (the rule for accepting an agent's answer, such as the tolerance for a number or the required facts for a free-text answer). For each reviewed task, they also judged the sufficiency of the information given to the agent.

RTable F2: Verdicts of the domain experts on the reference answers and on the scoring criteria, by type of question (number of questions; not shown: 3 reference answers and 11 scoring criteria without a clear verdict)

Type of questionReference answers reviewedReference answer validReference answer needs correctionScoring criteria reviewedCriterion appropriateCriterion too lenientCriterion too strict
Key quantities170162 (95.3%)7170143 (84.1%)213
Reasoning steps149145 (97.3%)4144137 (95.1%)00
Claim interpretation5652 (92.9%)25552 (94.5%)02
All reviewed questions375359 (95.7%)13 (3.5%)369332 (90.0%)21 (5.7%)5 (1.4%)

Count of the questions answered correctly from the supplied files. A correct answer from the supplied files indicates that the needed information is available, in particular for numerical and free-text questions, which are hard to guess. We therefore counted, for each of the 475 questions, the attempts with a correct answer.

RTable F3: Number of the 475 questions answered correctly from the supplied files (submitted results; 18 attempts per question: six agents, three rounds)

QuestionsAll 475Numerical and free-text (264)Yes/no, categorical and counting (211)
Answered correctly in at least one attempt458 (96.4%)249 (94.3%)209 (99.1%)
Answered correctly by agents of both model families (Claude and GPT)435 (91.6%)229 (86.7%)206 (97.6%)
Never answered correctly17 (3.6%)15 (5.7%)2 (0.9%)

Results without the items flagged by the experts. The experts flagged 24 questions: for each of them, the reference answer needs correction or could not be confirmed, a second answer is also valid, or the scoring criterion is too strict. They also found a necessary piece of information missing in 7 tasks. We removed these questions and tasks step by step and recomputed the submitted results.

RTable F4: Task success of the six agents without the questions and tasks flagged by the experts (task success in %, three rounds)

Items countedTasksOpus 5SolSonnet 5TerraLunaHaiku 4.5
1. All items, as submitted6459.438.020.819.316.70.5
2. Without the 24 flagged questions6462.542.225.522.921.40.5
3. Also without the 7 tasks with missing information5763.242.124.622.821.10.6
4. Only the 23 tasks with a complete expert review and only positive verdicts2372.552.229.021.724.61.4

Conclusion for W2, Q2 and Q3.

  • 95.7% of the reviewed reference answers are scientifically valid (W2). The experts confirm 359 of 375, and the revision will correct the 13 marked for correction.
  • Missing information affects a minority of the tasks (Q2). 96.4% of the questions are answered correctly from the supplied files in at least one attempt. The experts find the inputs sufficient for 39 of the 52 tasks reviewed on this point and a missing piece of information in 7, to be completed in the revision.
  • Valid alternative answers are rare, and the revision will accept each of them (Q3). The experts find a second valid answer for 5 reference answers and a too-strict scoring criterion for 5 of 369. The submission already handles such cases in three ways: the LLM judge accepts different wordings of the same physical meaning, four questions list accepted alternative answers, and in 30 recorded cases we narrowed the requested conclusion to the level supported by the data.
  • The main result holds without the flagged items. Without them, Opus 5 moves from 59.4% to 63.2%, and the order of the six agents is the same in rows 1 to 3. On the 23 tasks of row 4, Opus 5 still fails 27.5% of its attempts.
AuthorsOpenReview ↗

Response to Reviewer Fcht (Part 3/6)

(c) Regarding two model families, different harnesses and the available tools (W3, Q5)

  • Our view on W3. LEAP-Bench evaluates each agent in the form in which people use it: a model together with its harness and its reasoning-effort setting (the amount of reasoning of the model before it answers). Agent leaderboards such as SWE-bench and Terminal-Bench follow the same convention. We agree with the reviewer that a difference between a Claude agent and a GPT agent cannot be attributed to the model alone. The revised paper will call each such combination an "agent configuration" and make no such claim.
  • Our answer to Q5. Both harnesses let the agent inspect files and images and execute code in a Python environment with 184 installed packages. These include the general scientific libraries (numpy, scipy, pandas, matplotlib, scikit-image, h5py) and libraries for the Igor, TDMS and Gwyddion instrument formats (igor2, npTDMS, gwyfile). Libraries for the Nanonis, Bruker and QCoDeS formats were not pre-installed.
  • New analysis. RTable F5 groups the six agents by harness, so the effect of the model inside one harness can be read directly. Its confidence intervals come from a bootstrap that resamples whole papers, so they account for the correlation between tasks of the same paper.
  • New experiments. RTable F6 reports controlled experiments with one change of the setup at a time. For rows 1 and 2, we installed the missing libraries (pySPM, QCoDeS, OpenCV with imagecodecs), verified them on the real files of the four papers concerned, and ran these papers again. Row 3 lowers the reasoning effort of Sol. Rows 4 and 5 enforce the separation of the two phases in code; in the submission, this separation was an instruction to the agent.

RTable F5: Task success of the six agents, grouped by harness, with the uncertainty across papers (submitted results, three rounds; 95% confidence intervals from a bootstrap over the 12 papers)

HarnessModelTask success over all tasksAverage of the 12 per-paper success ratesTask success with one paper left out (lowest and highest of the 12 values)
Claude CodeOpus 559.4% [48.1, 70.4]58.7% [46.7, 69.9]56.3% to 62.3%
Claude CodeSonnet 520.8% [13.1, 30.0]23.2% [13.1, 33.6]18.6% to 23.0%
Claude CodeHaiku 4.50.5% [0.0, 1.7]0.5% [0.0, 1.4]0.0% to 0.6%
Codex CLIGPT-5.6 Sol38.0% [28.6, 49.3]41.7% [30.1, 55.7]34.5% to 41.2%
Codex CLIGPT-5.6 Terra19.3% [9.4, 31.4]24.4% [11.4, 41.1]16.7% to 22.4%
Codex CLIGPT-5.6 Luna16.7% [8.4, 26.8]22.2% [12.1, 33.3]14.9% to 19.4%

RTable F6: Controlled experiments on tools, reasoning effort and protocol: task success with and without one change (new runs for this response; Opus 5 one round, Sol three rounds)

Change of the setupAgentTask attemptsWithout the change (new run)With the changeDifference [95% interval]
1. Missing libraries for instrument file formats installed (4 papers)Opus 51710 (58.8%)10 (58.8%)0.0 points [−13.6, +15.8]
2. Missing libraries for instrument file formats installed (4 papers)Sol5120 (39.2%)17 (33.3%)−5.9 points [−13.3, +6.7]
3. Reasoning effort of the model lowered from "xhigh" to "high"Sol19265 (33.9%)54 (28.1%)−5.7 points [−11.9, −1.0]
4. Separation of the two phases enforced in codeOpus 56438 (59.4%)39 (60.9%)+1.6 points [−9.6, +13.3]
5. Separation of the two phases enforced in codeSol19264 (33.3%)65 (33.9%)+0.5 points [−9.7, +11.8]

In rows 1 to 3, both columns use the enforced separation of the phases. In rows 4 and 5, the column without the change uses the instructions of the submission.

Conclusion for W3 and Q5.

  • The installed libraries bring no gain (Q5). On these four papers, the missing libraries were not holding the agents back.
  • Inside one harness, the model matters far more than the three tested settings (W3). Inside Codex CLI, task success goes from 16.7% (Luna) to 38.0% (Sol), a difference of 21 points. Inside Claude Code, it goes from 0.5% (Haiku 4.5) to 59.4% (Opus 5), a difference of 59 points (RTable F5). Each of these three changes of the setup changes it by at most 5.9 points. Sol ran at "xhigh", the better of its two tested effort settings.
  • The same failure pattern appears in both harnesses. The best agent of each harness fails at least 40% of its task attempts. Most of these failed attempts have correct key quantities and reasoning steps and an incomplete or wrong interpretation: 22.4 of the 40.6 points of failed attempts for Opus 5, and 40.6 of 62.0 for Sol.
  • Further model families can be added. We will release the code connecting each harness to the benchmark for this purpose.
Reviewer_VTRHOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

LEAP-Bench contains 64 materials-science analysis tasks and 475 questions derived from 12 papers. Agents receive measurement files and a research objective, perform an analysis, and then answer questions about quantities, reasoning, and interpretation. Numerical rules and a language-model judge score the answers. Task success requires all reasoning and interpretation questions to pass, with at most one quantity miss.

Critical strengths
  1. The evaluation target is interesting and meaningful. The paper asks agents to independently choose analysis methods based on raw scientific instrument data and connect the results to scientific claims. This is closer to real scientific research than simply answering predefined questions. The two-phase evaluation design also prevents agents from seeing the full task checklist before starting their analysis.

  2. The experimental analysis is fairly comprehensive, and the results are reported objectively. The paper not only compares average scores but also examines whether models can consistently complete the same tasks across multiple runs. It also includes a no-data control experiment, compares different scoring methods, and uses multiple models as judges.

Critical weaknesses
  1. Scoring based only on answer agreement does not guarantee that the model has performed a correct scientific analysis. The interpretive judge can see the expected facts and reference answers, but not the actual code, figures, measurements, or papers (Appendix L). Thus, a model may receive a high score for a convincing conclusion even if its analysis is incorrect. Similar issues are also discussed in Section 4.5 and Appendix K.

  2. Some tasks may lack necessary information, and the input conditions are not fully consistent across tasks. For example, Section 4.5 states that the Bismuth Q10 task requires extracting data from figures in earlier papers, but the agents have no access to paper PDFs or the internet. It is unclear whether this information is provided elsewhere. If not, task failures may be caused by missing information rather than poor scientific analysis ability.

Questions for authors

Q1. How many benchmark tasks require analyzing raw experimental data, and how many use already processed or fitted results? Do the main failure patterns remain the same when these two types of tasks are evaluated separately?

Q2. Are the restrictions of the two-phase evaluation strictly enforced by code or system design, or are agents only instructed to follow them through prompts?

Q3. For Paper 24 Task 04 Q10, which files provided to the agent contain the earlier experimental data needed for quantitative comparison? Can a domain expert solve this task using only the files available to the agent?

5 comments
AuthorsOpenReview ↗

Response to Reviewer VTRH (Part 1/5)

We sincerely thank Reviewer VTRH for the careful and fair review. We are encouraged that the reviewer finds the evaluation target "interesting and meaningful" and "closer to real scientific research than simply answering predefined questions", and the experimental analysis "fairly comprehensive" with results "reported objectively".

We address all concerns in the following parts.

Regarding the timing of this response. We want to first apologize for the timing of this response and ask for the reviewer's kind understanding. The response required many new experiments and a review of the benchmark by eight domain experts across all 12 papers, and each paper requires deep knowledge of its own measurement technique.

Regarding our core vision. AI agents are entering the real laboratory, and the field has no benchmark today that measures their ability to turn raw experimental measurements into defensible scientific claims. LEAP-Bench is built to close this gap. Sections (a) to (d) answer the two weaknesses (W1, W2) and the three questions (Q1 to Q3) of the review with new experiments and ten result tables: the meaning of a solved task (W1), missing information and the bismuth task (W2, Q3), the form of the input files, raw or reduced (Q1), and the enforcement of the two phases (Q2).


(a) Regarding the meaning of a solved task (W1)

The reviewer notes that agreement of the answers does not guarantee a correct analysis. The LLM judge sees the reference answers but not the code, the figures or the measurements, so a convincing conclusion may be accepted on top of a wrong analysis.

  • Our view. We agree with the reviewer. A solved task shows that the agent's answers agree with the reference answers. It does not prove that the analysis behind them is correct. The revised paper will state this explicitly.
  • A safeguard already in the scoring. A convincing conclusion alone does not solve a task, because the key quantities and the reasoning steps must be correct as well. Of the 1,152 task attempts of the submission, 391 have every claim-interpretation question correct, and 94 of these (24.0%) still fail the task for this reason.
  • Three new experiments. We examined the analyses behind the answers in three ways.

New experiment 1: inspection of the agents' analyses by domain experts. For 12 task attempts (one per paper, six by Opus 5 and six by Sol), a domain expert inspected the figures and analysis files saved by the agent and judged the choice of data, the methods and the conclusions. Four of these attempts were solved in the benchmark and eight failed.

RTable V1: Verdicts of the domain experts on the saved analyses of 12 task attempts (number of attempts)

Verdict of the expert4 attempts solved in the benchmark8 attempts failed in the benchmark
Choice of data, samples and conditions appropriate, or usable with limits48
Methods and calculations appropriate, or valid with limits45
Error in a method or calculation03
Conclusions supported by the results of the analysis44

Conclusion of experiment 1. In the inspected attempts, a solved task rests on a real analysis. The experts find valid methods and supported conclusions in all 4 solved attempts, and all 3 attempts with a method or calculation error are failed attempts in the benchmark.

AuthorsOpenReview ↗

Response to Reviewer VTRH (Part 2/5)

(a) Regarding the meaning of a solved task (W1), continued

New experiment 2: the same tasks without the data. An agent able to solve a task by recalling the published paper, or by writing a plausible conclusion, would still solve it without the data. The no-data condition of the paper also changed the task descriptions and the answer policy. We therefore ran a matched control: all tasks again for the two strongest agents, with all data files removed and everything else unchanged.

RTable V2: Task success and correct answers of the two strongest agents with and without the data files (new runs for this response, phases enforced in code, three rounds; same task descriptions and questions)

AgentData filesTask attempts solvedKey-quantity answers correctReasoning-step answers correctClaim-interpretation answers correct
Opus 5present109/192 (56.8%)85.3%96.2%69.4%
Opus 5removed0/192 (0.0%)7.9%39.9%8.2%
Solpresent65/192 (33.9%)76.8%92.8%47.5%
Solremoved0/192 (0.0%)0.0%0.0%0.0%

New experiment 3: check of the passing answers against the agent's own files. A passing answer contradicted by the agent's own analysis would be a case of the reviewer's concern. Two LLM auditors (language models, not human experts) received the code, figures and outputs saved by the agent. They checked each of the 750 answers of the 104 task attempts solved in round 1 of the submission and labeled it as supported by these files, contradicted by them, or not found in the files given to the auditor.

RTable V3: Check of the 750 answers of the 104 solved task attempts of round 1 against the agent's own saved code, figures and outputs (number of answers)

Label given by the auditorLLM auditor 1LLM auditor 2
Answer supported by the saved files563 (75.1%)537 (71.6%)
Answer contradicted by the saved files21 (2.8%)32 (4.3%)
Answer not found in the files given to the auditor (neither supported nor contradicted)166 (22.1%)181 (24.1%)

Conclusion for W1.

  • In the inspected attempts, a solved task rests on a real analysis. The experts find valid methods and supported conclusions in all 4 solved attempts. All 3 attempts with a method or calculation error are failed attempts in the benchmark.
  • In the matched control, no task is solved without the data. Sol then answers "unknown" to 1,336 of its 1,425 questions, and Opus 5 to 914. Opus 5 answers 7.9% of the key-quantity questions and 8.2% of the claim-interpretation questions correctly, against 85.3% and 69.4% with the files.
  • In a separate test of the LLM judges, a scientifically reversed answer never passes. We prepared 74 test answers with one required fact reversed, and all six LLM judges of this test reject all 74.
  • Few of the traced answers are contradicted by the agent's own files. The two auditors find 2.8% and 4.3% of the 750 answers contradicted, and they could not find 22.1% and 24.1% in the files they received. A contradicted answer can only raise the score. This risk therefore cannot explain the main result of the paper: even the best agent fails about 40% of its task attempts.

Revision of the paper. The revised paper will state explicitly what a solved task means, and a new appendix will contain RTables V1 to V3. We will release the files saved by the agents and tighten the tolerances of the 21 scoring criteria judged too lenient by the experts.

(b) Regarding missing information and the bismuth task (W2, Q3)

The reviewer points to the bismuth task (Paper 24, Task 04). Its question 10 asks the agent to compare its result with earlier published measurements on Bi(111). The reviewer asks which supplied files contain these earlier data, and whether a domain expert can solve the task from the supplied files alone (Q3).

  • Our answer to Q3. None of the files given to the agent contains the earlier Bi(111) data. We searched all 51 files given to the agent for Paper 24 to confirm this. Neither an agent nor a domain expert can therefore make this comparison from the supplied files alone. We agree with the reviewer that, for this part of the question, a wrong answer reflects missing information rather than poor analysis.
  • The expert's assessment. The domain expert for this task judged all 10 reference answers acceptable and wrote about question 10 that the reference answer "is reasonable, although maybe high standard. Cross-checking past published results to confirm the finding is one crucial part of scientific research." We agree with the expert on the science and with the reviewer on the protocol, because an agent without access to the literature cannot make this comparison.
AuthorsOpenReview ↗

Response to Reviewer VTRH (Part 3/5)

(b) Regarding missing information and the bismuth task (W2, Q3), continued

Question 10 of the bismuth task (Paper 24, Task 04) asks for a comparison with earlier published measurements, and the files given to the agent do not contain them. We agree with the reviewer on this point and changed the scoring.

Our fix for the questions with outside literature (Q3). We screened the reference answers of all 475 questions for comparisons with outside literature. The screen flagged 2 questions, both in Paper 24 (question 9 of Task 03 and question 10 of Task 04). In both, the comparison with earlier literature is no longer required for a correct answer, and the 36 affected answers were graded again.

RTable V4: Submitted scores with the comparison with earlier literature required and no longer required (18 attempts per task, from six agents and three rounds)

ResultComparison required (as submitted)Comparison no longer required
Paper 24, Task 03, question 9: correct answers0/184/18
Paper 24, Task 04, question 10: correct answers0/186/18
Paper 24, Task 03: attempts solved0/184/18
Paper 24, Task 04: attempts solved0/182/18
Opus 5: task success114/192 (59.4%)118/192 (61.5%)
Sol: task success73/192 (38.0%)75/192 (39.1%)
All six agents: task success297/1,152 (25.8%)303/1,152 (26.3%)
Tasks never solved by any agent1210

The same concern across the 475 questions (W2). The bismuth example raises the broader concern that other tasks may lack necessary information as well. We tested this in two ways.

  • Count of the questions answered correctly from the supplied files (RTable V5). A correct answer from the supplied files indicates that the needed information is available, in particular for numerical and free-text questions, which are hard to guess.
  • Results without the flagged items (RTable V6). We recomputed task success without the items flagged by the domain experts and, as an extreme case, without every question never answered correctly.

RTable V5: Number of the 475 questions answered correctly from the supplied files (submitted results; 18 attempts per question: six agents, three rounds)

QuestionsAll 475Numerical and free-text (264)Yes/no, categorical and counting (211)
Answered correctly in at least one attempt458 (96.4%)249 (94.3%)209 (99.1%)
Answered correctly by agents of both model families (Claude and GPT)435 (91.6%)229 (86.7%)206 (97.6%)
Answered correctly in at least 9 of the 18 attempts353 (74.3%)163 (61.7%)190 (90.0%)
Never answered correctly17 (3.6%)15 (5.7%)2 (0.9%)

RTable V6: Task success of the six agents without the items flagged by the experts, and without the questions never answered correctly (in %, three rounds; submitted results recomputed)

Items countedTasksOpus 5SolSonnet 5TerraLunaHaiku 4.5
All items, as submitted6459.438.020.819.316.70.5
Without the 24 questions and 7 tasks flagged by the experts5763.242.124.622.821.10.6
The same, comparison with earlier literature no longer required5765.543.324.622.821.10.6
Extreme case: all 64 tasks, without the 17 questions never answered correctly6471.447.426.025.521.40.5

The experts flagged 24 questions: for each of them, the reference answer needs correction or could not be confirmed, a second answer is also valid, or the scoring criterion (the rule for accepting an answer) is too strict. They also flagged 7 tasks with a necessary piece of information missing from the inputs.

Conclusion for W2 and Q3.

  • The reviewer's diagnosis is correct for two questions. Their effect on the task success of all six agents is 0.5 points (25.8% to 26.3%), and the revision will remove this requirement.
  • Missing information affects a minority of the tasks. 96.4% of the questions are answered correctly from the supplied files in at least one attempt. The experts find the inputs sufficient for 39 of the 52 reviewed tasks and a missing piece of information in 7, and the revision will complete the descriptions of these 7 tasks.
  • Missing information does not explain the main result. Without the 24 questions and 7 tasks, the best agent still fails 36.8% of its attempts, and in the extreme case of RTable V6 still 28.6%. The ranking of the six agents is the same in every row.

Revision of the paper. The comparisons with earlier literature will be removed from the scoring, the descriptions of the 7 tasks will be completed, and RTables V4 to V6 will be added.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Global Response (Part 1/6): our vision and the strengths recognized by the reviewers

We sincerely thank Reviewers VTRH, Fcht and sJFK for their careful and constructive reviews, and the area chair for handling our submission.

Regarding the timing of this response. First, we sincerely apologize for the timing of this response, and we ask for the reviewers' kind understanding. Two efforts took most of the time. First, eight domain experts reviewed the benchmark question by question across all 12 papers, and each paper requires deep knowledge of its own measurement technique. Second, we ran 19 new computational experiments with about 390 new agent runs, so that every major concern of the reviewers is answered with data.

Regarding our core vision. AI agents are entering the real scientific laboratory. They already help scientists plan and run experiments, and the analysis of the measured data is the natural next step. This analysis decides which claims enter the scientific record and which experiment comes next. An agent that analyzes data fluently but wrongly is therefore a risk for science, while an agent that analyzes data reliably can become a true co-scientist. Today the field has no benchmark that tells these two apart, and LEAP-Bench is built to close this gap. Among the 16 benchmarks compared in Table 1 of the paper, it is the only one that combines raw experimental data, several measurement modalities, the discovery of a claim with the conclusion withheld, scientific interpretation, and expert verification.

Strengths recognized by the reviewers

  • (1) An important and open problem. LEAP-Bench "highlights an important problem that existing benchmarks have not fully addressed" (Fcht) and "addresses an important capability for assessing how much of a scientific workflow an AI agent can perform autonomously" (sJFK). Its target is "closer to real scientific research than simply answering predefined questions" (VTRH).
  • (2) A novel idea: the full path from experimental data to a scientific claim. The paper "integrates the full path from experimental data to a scientific claim", and an end-to-end evaluation "grounded in real experimental data is novel and useful" (sJFK).
  • (3) A sound benchmark design. "Good benchmark design and useful diagnostics" (Fcht). The two-phase design "prevents agents from seeing the full task checklist before starting their analysis" (VTRH), and Reviewer Fcht likes "the combination of deterministic scoring and an LLM judge".
  • (4) Thorough and objective experiments. "The experimental analysis is fairly comprehensive, and the results are reported objectively" (VTRH). Reviewer Fcht sees a "good, thorough analysis of what the models are lacking, along with useful robustness evaluations and ablations".
  • (5) Diagnostic insight into current agents. The benchmark "can identify where an agent fails rather than reporting only a final-answer score", and its central finding is "informative for evaluating current scientific agents" (sJFK).

Summary of this response

  • Major concerns of the reviewers. First, the benchmark was built with the help of language models, so human experts should confirm the tasks, the reference answers and the scoring (sJFK, Fcht). Second, the failures may come from missing inputs, the protocol, the tools or the scoring, and not from the agents (sJFK, VTRH, Fcht). Third, 12 papers may be too few for the conclusions (sJFK, Fcht).
  • Our rebuttal in summary. Eight domain experts reviewed the benchmark, and they confirm 95.7% of the 375 reviewed reference answers. Without the questions and tasks flagged by the experts, the best agent moves only from 59.4% to 63.2% of solved task attempts. In new controlled runs, the two strongest agents solve no task without the data, and changes of protocol, tools and reasoning effort move task success by at most 5.9 percentage points. The best agent stays ahead of the second on 10 of the 12 papers and on five additional papers.
AuthorsOpenReview ↗

Global Response (Part 2/6): the benchmark and its central finding

We summarize the core messages of our benchmark in brief

  • Tasks. LEAP-Bench contains 64 analysis tasks built from 12 published experimental studies, the source papers. Each task targets one scientific claim of its source paper.
  • Two phases. In the analysis phase, the agent receives the data files and a research objective and carries out its own analysis, without seeing any question. In the question phase, it receives the questions and answers them from its saved analysis.
  • Three types of questions, 475 in all. Key quantities are the numbers the analysis must produce. Reasoning steps are the intermediate decisions of the analysis. Claim interpretation is the scientific conclusion.
  • Scoring. Each answer is compared with a reference answer taken from the source paper. Numbers and categories are scored by fixed rules, such as a tolerance. Free-text answers are graded by a language model, called the LLM judge below. A task is solved only if every reasoning-step and claim-interpretation answer is correct and at most one key quantity is wrong.
  • Agents and attempts. An agent is a model inside its agent software, also called harness: Opus 5, Sonnet 5 and Haiku 4.5 in Claude Code, and GPT-5.6 Sol, Terra and Luna in Codex CLI. Each agent attempts each task in three rounds (192 task attempts per agent), and task success is the share of solved attempts. Opus 5 and Sol are the two strongest agents.

RTable G1: Share of task attempts solved by each of the six agents, and share with correct key quantities and reasoning steps but an incomplete or wrong interpretation (submitted results; share of the 192 task attempts of each agent, in %; in columns 3 and 4, the key quantities count as correct with at most one wrong value, as in the rule for a solved task)

Agent (agent software, model)Task attempts solvedKey quantities and reasoning steps correctKey quantities and reasoning steps correct, but interpretation incomplete or wrong
Claude Code, Opus 559.481.822.4
Codex CLI, GPT-5.6 Sol38.078.640.6
Claude Code, Sonnet 520.851.030.2
Codex CLI, GPT-5.6 Terra19.356.237.0
Codex CLI, GPT-5.6 Luna16.749.532.8
Claude Code, Haiku 4.50.57.87.3

Conclusion: the central finding of the paper.

  • Even the best of the six agents solves only 59.4% of its task attempts.
  • The agents reach correct numbers far more often than a correct scientific conclusion. Sol has correct key quantities and reasoning steps in 78.6% of its attempts, but solves the task in only 38.0%.
  • A benchmark scoring only the final numbers would count these attempts as successes. LEAP-Bench counts them as failures, because the interpretation is incomplete or wrong.
AuthorsOpenReview ↗

Global Response (Part 3/6): the new experiments

RTable G2: The main new experiments of this response, in order of importance (Opus 5 and Sol: the two strongest agents; task success: share of task attempts solved; points: percentage points; in italics: pointer to the full results)

Concern of the reviewersOur new experimentKey resultConclusion
1. Built largely by language models, not validated by experts (sJFK, Fcht)Eight domain experts reviewed reference answers, scoring criteria (rules for accepting answers), tasks and inputs in all 12 papers. RTables G3–G4, Global Response (Part 4/6)95.7% of 375 reviewed reference answers valid; 90.0% of 369 reviewed scoring criteria appropriate; no scientific flaw in 53 reviewed tasksThe answers and rules we score against are sound
2. Failures may come from benchmark errors, not the agents (sJFK, VTRH, Fcht)Results recomputed without the 24 questions and 7 tasks flagged by the experts. RTable G5, Global Response (Part 5/6)Best agent: 59.4% → 63.2% of attempts solved; ranking of the six agents unchangedThe main result holds without the flagged items
3. 12 papers may be too few (sJFK, Fcht)Results recomputed per paper; 25 tasks built from 5 additional papers and tested. RTables S1–S3, Response to Reviewer sJFK (Part 2/6)Opus 5 ahead of Sol on 10 of 12 papers and on all 5 new ones; on the new ones, only 49.3% and 24.0% of attempts solvedSame leader and large room for improvement on new papers
4. A solved task may hide a wrong analysis (VTRH)Experts inspected the saved analyses of 12 attempts; all tasks rerun by the two strongest agents without the data files. RTables V1–V2, Response to Reviewer VTRH (Parts 1/5–2/5)All 4 inspected solved attempts rest on a sound analysis; without the data, neither agent solves a single task (0 of 192 attempts each)Success needs the data; inspected successes rest on real analyses
5. Unclear source of failure: numbers or interpretation (sJFK, Fcht)Sol given the reference values of all 208 key quantities (the numbers to compute) in advance. RTable F11, Response to Reviewer Fcht (Part 6/6)Even with every key quantity given, Sol still gets a reasoning or interpretation question wrong in half of its attemptsCorrect numbers alone do not solve the tasks
6. The LLM judge (an AI grader) may be unreliable (Fcht, sJFK)Six LLM judges graded 272 test answers with an intended grade (148 complete, 74 with a fact reversed, 50 with a fact left out); experts graded 13 real agent answers. RTable F8, Response to Reviewer Fcht (Part 4/6)All judges accept the 148 complete and reject the 74 reversed answers, differing only on omissions; the benchmark's judge matches experts on 9 of 13 real answers, stricter on 3, more lenient on 1The judges agree on clear cases; against experts, ours leans strict
7. Results may reflect tools and settings, not the model (Fcht, sJFK)Missing libraries for instrument files installed; reasoning effort lowered. RTable G6, Global Response (Part 5/6)Task success moves by at most 5.9 points; with another model in the same agent software, by up to 59 pointsThe model matters far more than the tested settings
8. Two phases (analysis, then questions) not enforced (VTRH)Logs of the 216 main runs audited; phases enforced in code and agents rerun. RTables V9–V10, Response to Reviewer VTRH (Part 5/5)Access rules broken in only 5 of 216 runs, none with a solved task; enforcement moves task success by +1.6 (Opus 5) and +0.5 points (Sol)Enforcing the two phases makes no clear difference
9. Inputs tidier than in a real laboratory (Fcht)List of relevant files removed; the agent searches all files of the paper. RTable F1, Response to Reviewer Fcht (Part 1/6)5 largest data packages: task success drops by 20.8 (Opus 5) and 12.5 points (Sol); no drop on smaller onesLarge packages are harder without the list; the revised benchmark will add this version
10. Reviewers cannot check the results (Fcht)Anonymized package with all prompts, tasks, code, saved answers and grades. RTable F7, Response to Reviewer Fcht (Part 4/6)All 360 submitted runs and all 14,250 answer grades reproduced exactly from the package aloneEvery submitted score can be checked independently

Conclusion.

  • Domain experts confirm the reference answers and the scoring (95.7% and 90.0% of the reviewed items).
  • The failures persist. Without the flagged items, the best agent still fails 36.8% of its attempts, and protocol, tools and effort move task success by at most 5.9 points.
  • The gap is real, and the field needs to close it. AI agents are entering laboratories, and the best of six still fails four task attempts in ten on these 12 research cases. LEAP-Bench makes this gap measurable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.