acceptodds
Under review as a conference paper at ICLR 2027

CompMat-Bench: Benchmarking AI Agents for Computational Materials Science

Abstract

Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations. In computational materials research, repeating the same expensive simulations across agents and trials can make evaluation impractical. We introduce CompMat-Bench, a benchmark of 94 tasks derived from recently published computational materials studies, each asking agents to complete a step toward achieving the study's scientific goal. We reproduce the research steps in advance and assess agents on preparing inputs and analyzing outputs for expensive simulations, so expensive simulations can be avoided during evaluation. The reproduced inputs and results serve as ground truth for grading agents with fixed rules, without an LLM judge. The benchmark supports four evaluation conditions: single tasks and workflows composed of related tasks, each with full or reduced methodological guidance. With full guidance on single tasks, agents based on three LLMs demonstrate the ability to complete individual materials research steps, with pass rates of 66.0–90.4% across 94 tasks. Both longer workflows and reduced guidance can limit agent performance, but in different ways for different agents: they lower the pass rates of the weaker agents, whereas the strongest agent falls only when a long workflow is combined with reduced guidance. Failure analysis attributes most failures to scientific errors rather than to errors in software usage. CompMat-Bench provides a basis for comparing agents on the steps of real materials research and for analyzing agent failure modes.

Primary area:
datasets and benchmarks
Keywords:
AI agents, computational materials science, scientific benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yuGmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CompMat-Bench, a benchmark with 94 tasks drawn from 15 published computational materials studies. It evaluates agents on single tasks and complete workflows made up of related research steps. The authors evaluate three LLMs with different harnesses and analyse their errors.

Critical strengths

The motivation of the paper to evaluate LLM agents on time-consuming computational materials research is good. The analysis of agent trajectory is also meaningful.

Critical weaknesses

The main experiments test only three LLMs. With full guidance, stronger models get very high scores on many single tasks. The strongest model also passes 92% of the complete workflows. These results suggest that the fully guided tasks may be still too easy for current LLMs. Testing more models and workflows would help show whether the benchmark can reliably separate agents with different levels of ability.

Questions for authors
  1. Can the authors provide more results for other strong LLMs to show whether the benchmark can distinguish their performance?
  2. Do the authors have estimates of how long a materials researcher would take to complete these tasks, compared with the agents? For time-consuming research tasks, reporting completion times would help show whether agents can save researchers time.
1 comment
AuthorsOpenReview ↗

Response to Reviewer yuGm

Thank you for the review.

Weakness and Q1. More strong models and workflows; can the benchmark distinguish them?

We evaluated six more models under the same settings, bringing the total to nine (see the table in item 2 of our response to all reviewers): two open-weight models (Qwen3.8-Max and GLM-5.3), two strong closed models (Gemini 4 Argon and GPT-6.1 Sol), and the two strongest models (GPT-6.2 Astra and Gemini 4 Carbon). (Claude models are not included: the regulations of our funding do not allow us to evaluate them.)

We apologize for not completing the workflow runs for GPT-6.2 Astra and Gemini 4 Carbon, and for not building more workflows within the time available for discussion. Thanks to the reviewer's suggestion, we added an analysis of the seven models with workflow results and found that the single tasks can predict the workflow ranking (item 3 of our response to all reviewers).

We agree with the reviewer about the results: SF still distinguishes the models (GPT-6.2 Astra > GPT-6.1 Sol > GPT-5.6 Sol, 92 > 91 > 85 of 94), but their scores are close and near saturation. There are two reasons. (1) LLMs improve rapidly: GPT-6.1 Sol, released less than three months after GPT-5.6 Sol, passes 91 tasks compared with 85 for GPT-5.6 Sol. (2) The difficulty of our tasks also plays a role. To control the difficulty and avoid saturation, we can add more difficult tasks and remove easy ones, which should rescale the scores but preserve the ranking.

We added the results of the six models (Section 4, Fig. 4 and Appendix F) and the ranking analysis (last paragraph of Section 5.2), and identified score saturation as a limitation in Section 6.

Q2. Time for a researcher compared with the agents.

We thank the reviewer for this suggestion, which strengthens the significance of the benchmark. We can provide some estimates and data, but the discussion period did not give us enough time to have researchers repeat the tasks, so we cannot provide the most direct evidence. We report the following.

(1) Agent time. On single tasks (one-hour limit; the expensive simulations are precomputed), the median time for a passing run ranges from 1.5 to 18 minutes across the seven models. On workflows with full guidance (five-hour limit; the agent runs all simulations itself), the corresponding range is 25 to 195 minutes.

(2) Reproduction time. Our own reproduction of one study took about two weeks and yielded about six tasks. This time includes trial and error and the computation itself. AI tools were inevitably used in the reproduction, and a researcher reviewed and accepted the results, so it is hard to isolate the researcher's effort.

(3) Researcher time, estimated from our experience. It varies. An average task (e.g. preparing the input files of one calculation, or analyzing its output files) takes 0.5 to 1 hour. A complex task can take a researcher working without AI several hours to several days, including time to learn the code from its manual (the LAMMPS manual alone exceeds 1,000 pages). Some tasks (e.g. a saddle-point search for a transition state, or phonon scattering at an interface) represent the main work of an entire paper: the difficulty is in the theoretical derivation and algorithm development, and the computation itself takes little time. We estimate that these tasks take weeks.

We added the agents' completion times to Appendix I (Table I1).

Reviewer_mGoGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents a benchmark of 94 tasks in computational material science, and evaluate some LLM-based science-agents on them. The benchmarks are derived from real experiments run in 15 published papers. (this is a common scientific-task benchmark construction method by this point).

The core selling point of the benchmark, as I understand it, is that the benchmark manages to avoid "running expensive simulations" which other benchmarks include. The way to avoid the simulation-running cost seems to be pre-computing the gold-standard simulation as run in the original experiment, checking only if the agents managed to provide the exact same simulation arguments, and providing the pre-computed results for the following steps.

This is either trivial (memoizing results of long-running computations) or not sufficient. It is not clear to me that there is only a single way to run the simulation, and that agents that provide a slightly different parameters indeed fail the task. It is also important for the agent to realize its simulation may have been wrong, and recover from that. This is not discussed at all in the paper: there is an implicit assumption that there is a single gold way to run a simulation, that all small variations are invalid, and that agents do not need to be able to identify their own mistakes. If this is indeed the case in material science, then the paper should at least state this assumption (but then the cost-saving method becomes trivial, it is just caching the outputs). If it is not the case, then this should also be discussed, and the caching method should be evaluated for sufficiency.

Beyond the cost-saving benefit (which is itself questinable), I do not think yet-another-scientific-agentic-tasks-benchmark is a sufficient contribution for an ICLR paper.

Critical strengths

Somewhat useful benchmark of 94 material design tasks, based on experiments from recent published papers.

Critical weaknesses
  • Main claim (cost saving due to caching) is either trivial or not well supported.
  • Limited novelty and contribution for ICLR paper.
Questions for authors

Is there really only one way to specify simulation parameters, and no variation is acceptable?

Don't you care about the agent's ability to see its simulation results, conclude they might be wrong, and correct it's simulation? (isn't this a core part of the scientific task?)

1 comment
AuthorsOpenReview ↗

Response to Reviewer mGoG

Thank you for the review.

Weakness 1 and Q1. Only one way to specify the parameters?

No. Different inputs can pass the same task. We apologize for not making this clear in the paper.

We reproduce each calculation to obtain a complete set of inputs and outputs (most materials papers do not report all of these inputs). We use our reproduction to write grading rules that also accept other valid answers. Each task is reviewed by a computational materials researcher with a PhD in a related field, explicitly on the questions "Does the prompt clearly define the physical task and its valid answer set?" and "Could an essentially correct answer fail a stated condition?", to ensure that the task description and the grading rules correspond to each other.

Domain experts decide which variations are valid. The 94 tasks have 1191 checks. Apart from exact matches (e.g. integers, signs, booleans), 1072 of them (90%) are of three kinds:

(1) Range. We accept a range of values. For example, once the energy cutoff is high enough, raising it further has little effect on the result. We therefore accept any cutoff at or above the recommended minimum. For calculated results, we normally accept values within 10% of our own result.

(2) Subset. We check only the settings that matter for the result. For example, our CrN task depends on the magnetic arrangement, so we check the settings for a magnetic calculation (e.g. ISPIN and MAGMOM in VASP). For nonmagnetic hBN, we accept input files with or without these settings.

(3) Physical equivalence. We accept different answers that describe the same physical result. For example, shifting all atoms together changes their coordinates but leaves the crystal structure unchanged. We wrote a program to test whether the submitted structure matches our reference after such a shift.

The reviewer also requests that the precomputation method be evaluated for sufficiency. We added an analysis of seven models in total. We compare the single tasks (SF, the precomputation approach) with the workflows (WF, in which the agent runs all simulations itself). Both give almost the same ranking of the seven models. This indicates that the single tasks predict how the models rank on the workflows, at about 2% of their compute cost (item 3 of our response to all reviewers).

Section 3.1 now explains the grading rules, and Appendix G (Table G1) gives the number of checks of each kind. The last paragraph of Section 5.2 reports the ranking comparison.

Q2. Can the agent correct a wrong simulation?

Yes, we care about this ability. However, we have to balance it against the cost of failed agent runs. In computational materials science, an expensive simulation can take hours to days (the largest step in our reproduction used about 9,500 core-hours). A wrong input file can waste all of it. Therefore, we let the agent run the calculation and correct itself when the cost is affordable. In this paper, we set the threshold at one hour on a single machine (4 CPU cores) and build three kinds of single tasks based on it:

(1) Run-and-analyze tasks (35 of 94) fall below this threshold. They include tasks that require substantial algorithm design but little computation (e.g. a saddle-point search for a transition state, or phonon scattering at an interface).

(2) Input-preparation (30) and output-analysis (29) tasks cover calculations above this threshold. We use precomputation and grade the input files directly. The agent can consult the offline software manuals to check its inputs.

We also have workflows (WF, WR), in which the agent runs all simulations itself. The ranking analysis above shows that the WF ranking is consistent with the ranking from the precomputation approach.

To state this more clearly, we revised the abstract, the introduction, the caption of Fig. 1 and Section 3.2.

Weakness 2. Novelty and contribution.

We believe that CompMat-Bench is more than another benchmark: it addresses a class of research tasks that is hard to include in agent benchmarks because the key step is expensive. Here it is an HPC simulation; in experimental fields it is a lab experiment, as Reviewer Rzip notes. As AI agents take on more complex and long-running scientific tasks, we expect the verification of scientific results through expensive simulations or laboratory experiments to become a bottleneck in evaluating these agents. We hope CompMat-Bench can be a stepping stone: precomputation is one option, and our analysis shows that it predicts how the models rank on the workflows.

Reviewer_RzipOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark to measure LLM agents' performance on computational material science research tasks. These tasks typically involve computationally-heavy simulations, so to make the benchmark tractable, the authors propose pre-defining research task workflows and pre-computing simulation outputs for heavy steps in these workflows. In this setting, the LLM agent cannot execute the workflow open-endedly, but instead proposes inputs to each step, which are then graded. Similarly, given the pre-computed outputs, the LLM's analysis of the outputs is graded. It is found that models/harnesses can solve the majority of individual tasks (with e.g. GPT 5.6 Sol getting perfect accuracy on output analysis), though performance is far from perfect. Performance drops substantially when the agent is asked to perform end to end workflows without additional guidance. Finally, failure cases are analyzed, pointing primarily at challenges with long workflows and making scientific mistakes over operational ones.

Critical strengths

The idea to pre-compute simulation outputs is clever and allows the benchmark to be computationally-tractable while also simulating real research tasks. This approach could plausibly applied to other domains as well, including potentially non-computational research (e.g. where lab experiments are prohibitive).

Critical weaknesses

My key concern is that I struggle to understand how the workflow setup works with pre-computed steps. When chaining multiple steps together, I assume step (b) depends on step (a) being executed correctly. What do you do if (a) is incorrect? Do you inject the correct version of (a), do you proceed as-is, or do you terminate the episode and mark (b, c, ...) as incorrect? This is a key implementation detail that is missing from the paper (apologies if I missed it) and substantially changes how a reader interprets the results.

Questions for authors
  1. I don't quite understand the performance drop from single task to workflow. Rephrasing my comment above: If the agent gets the inputs to task (a) wrong, how do you propagate this to task (b) if (a) is an amortized calculation? Do you correct the agent input to task (a) or let it remain incorrect but return the pre-calculated outputs? If the former, it seems to me that the workflow setting should have similar performance to the single-task setting (modulo different macro averaging).
  2. You note that for GPT-5.6 Sol, WF=SR=SF, and performance drops in the WR setting, and suggest this is because the agent has to plan itself. However from Figure 5, it seems like the jump in error rate is driven by 'wrong implementation' - can you reconcile these things?
  3. To understand how much the pre-defined workflow affects the observed results, are you able to run a subset of the LLM agents on a subset of the 94 tasks in an open-ended setting? The sizes of these subsets could be chosen so that the amount of simulation required is comparable to what was required to construct CompMat-Bench, and would give greater confidence in the pre-computation approach if results correlate.
  4. Do you see a publication time-dependence in the accuracy of models across tasks? I imagine older papers are in pretraining corpora and may also propose workflows that are now field-standard.
1 comment
AuthorsOpenReview ↗

Response to Reviewer Rzip

Thank you for the careful reading.

Weakness and Q1. What happens when (a) is wrong, and why workflows score lower.

We proceed as-is. In a workflow the agent runs all simulations itself (item 1 of our response to all reviewers), so (b) starts from the agent's own output of (a). We do not inject our output of (a), and we do not stop the run. Only the final quantities are graded. An error in (a) can therefore carry into (b) and lower the workflow score.

We apologize for not stating this prominently in the paper. In the workflow variant, we test whether the agent can run the whole chain itself, which costs more than evaluating single tasks. Our precomputation ensures that the whole chain is reproducible and provides the reference for grading. To make this clear, we revised the abstract, the introduction, the caption of Fig. 1 and Section 3.2.

Q2. Planning and "wrong implementation" in Fig. 5.

We thank the reviewer for noticing this detail. Under SF, the agent works on one task at a time with full guidance. Under WR, the agent has to "plan and carry out the multi-step calculation itself", which leaves room for both "wrong physics" and "wrong implementation". For GPT-5.6 Sol, WF = SR = SF, and performance drops only under WR. The agent was more likely to fail when it had to both run a longer workflow (vs. SR) and plan the calculation itself (vs. WF).

For example, in one WR run of the CrN workflow of Fig. 2, GPT-5.6 Sol set up the correct model and equations and designed the algorithm. But it failed in the implementation: it set the Wannier90 option translate_home_cell incorrectly, which led to incorrect magnon frequencies. Since the model and equations are correct, we classified this run as wrong implementation. This is what we see in Fig. 5. Under WR, GPT-5.6 Sol carries out its own plan. The number of its failed runs with wrong physics is the same as under WF (2 and 2), and the new failures are in the implementation.

It is therefore difficult to attribute the additional failures to planning alone or to the longer workflow alone: they appear only when the two are combined. We added this explanation to Section 5.2.

Q3. Open-ended runs on a subset, to validate the precomputation.

We thank the reviewer for suggesting this comparison. Our eight workflows already provide part of this comparison. All tasks of the workflows are from CompMat-Bench, and they cover 31 of the 94 single tasks, from eight of the 15 studies. Following the reviewer's suggestion, we compared how the seven models ranked on single tasks and workflows under full guidance (SF and WF). Our analysis shows that, ranked by single tasks or by workflows, the seven models come in almost the same order (item 3 of our response to all reviewers). This supports using precomputation to predict how models rank on workflows at much lower cost.

We added this analysis to the paper: the last paragraph of Section 5.2.

Q4. Dependence on publication time.

We see no clear dependence. We split the 94 tasks by publication date into an earlier and a later half of 47 tasks each. For six of the seven models, the pass counts differed by at most two between the two halves. For Qwen3.8-Max, the difference was six. We think the short publication window may help explain this result. Fourteen of the 15 studies were published between January and March 2026, and one in September 2025.

We added this analysis to the paper: Appendix H (Table H1).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers

We thank the three reviewers for their time. Rzip notes that precomputation makes the benchmark "computationally-tractable while also simulating real research tasks" and suggests that the approach could also apply "where lab experiments are prohibitive". mGoG notes that the tasks come from recently published papers, and yuGm highlights the motivation for the benchmark and the analysis of agent trajectories.

1. How single tasks and workflows relate (Rzip Weakness, Q1, Q3; mGoG Summary, Q2)

We apologize for not explaining this distinction more clearly. The agent receives precomputed outputs only in single tasks. In a workflow, the agent runs all simulations itself.

CompMat-Bench consists of 94 single tasks. For expensive simulations, we assess input preparation and output analysis separately, so agents can be evaluated without rerunning the simulation. We grade both tasks using rules based on our own reproduction.

We built the workflow variant by linking single tasks from the same study into a chain, e.g. A -> B -> C. We give the agent the starting files for A and ask it for the final quantities from C. The agent runs all simulations itself on one 128-core node for up to five hours, so B uses the output the agent produced in A. Our precomputation ensures that the whole chain is reproducible and provides the reference for grading the final quantities from C.

Precomputing the expensive simulations gave us a low-cost way to predict how models rank on workflows. For the seven models we evaluated on workflows, single-task scores predicted the workflow ranking under both levels of guidance (see item 3 below). For a typical model, the 94 single tasks take about 75 core-hours in total, and the 3x8 workflow runs take about 3,300 core-hours in total.

We clarified this distinction in the abstract, the introduction, the caption of Fig. 1, and Section 3.2.

2. Results for six more models (yuGm Weakness, Q1)

We evaluated six more models under the same settings. The main finding of the paper holds for them: workflows with reduced guidance (WR) were the hardest of the four conditions for each new model evaluated on workflows. The benchmark also separates models of different ability: under reduced guidance, the seven models passed between 8 and 21 of the 24 workflow runs. Passes per condition ("-": not run):

ModelSF, 94 tasksSF, 52SR, 52WF, 24 runsWR, 24 runs
GPT-6.2 Astra (new)9250---
GPT-6.1 Sol (new)9150492421
Gemini 4 Argon (new)9049492421
GPT-5.6 Sol8546462218
Gemini 3.8 Flash8447431913
Qwen3.8-Max (new)643231138
GLM-5.3 (new)6234311613
DeepSeek V4.1 Flash6233281012
Gemini 4 Carbon (new)59 (86*)31 (46*)---

*Gemini 4 Carbon did not finish 34 tasks within one hour, although the agent sees the remaining time after every step. The numbers in parentheses show the results when we extended the time limit for these 34 tasks to 2.5 hours.

GPT-6.2 Astra and Gemini 4 Carbon were run on the fully guided single tasks only. Because of provider rate limits, we could not run workflows for these two models during the discussion period.

In the revised PDF, the nine models and their settings are in Section 4, Fig. 4 and Section 5.2 include three of the added models, and Appendix F gives all results (Table F1) and the failure analysis of the added models (Table F2).

3. Single tasks predict the workflow ranking (Rzip Q3; mGoG Weakness 1)

We tested how well single-task scores predict the workflow ranking of the seven models evaluated in both settings. We compare the two at the same level of guidance: SF with WF, and SR with WR. With full guidance, the ranking is GPT-6.1 Sol > Gemini 4 Argon > GPT-5.6 Sol > Gemini 3.8 Flash > Qwen3.8-Max > GLM-5.3 = DeepSeek V4.1 Flash in SF, and only Qwen3.8-Max and GLM-5.3 appear in the opposite order in WF (Spearman rank correlation 0.93, n = 7). With reduced guidance, the rank correlation between SR and WR is 0.92. The single tasks thus predict how the models rank on the workflows, at about 2% of the compute cost (item 1).

In the revised PDF, this analysis is the last paragraph of Section 5.2.

AuthorsOpenReview ↗

Summary of revisions

The revised PDF contains the changes below, marked in blue. Details are in our responses to each reviewer.

  1. Single tasks and workflows (Rzip Weakness, Q1; mGoG Q2). The abstract, the introduction, the caption of Fig. 1 and Section 3.2 state that the agent runs all simulations itself in a workflow. Section 3.2 also explains that the agent does not receive our precomputed outputs.
  2. Six more models (yuGm Weakness, Q1). Section 4 lists the nine models and their references. Table B1 gives the settings. The second row of Fig. 4 shows results for three of the added models, and Section 5.2 includes these results in its two conclusions. Appendix F gives the results of all nine models (Table F1) and the failure analysis of the six added models (Table F2). The abstract, the introduction, Sections 4, 5.1, 5.3 and 6, and the conclusion now refer to nine models. We also revised the statements in the abstract and the conclusion about the stronger agents and the causes of failure. The abstract reports that scientific errors accounted for more failures than software usage errors. The conclusion identifies scientific errors as the most frequent cause of failure.
  3. Single tasks predict the workflow ranking (Rzip Q3; mGoG Weakness 1). We revised the title of Section 5.2 and added the ranking analysis in its final paragraph.
  4. Grading (mGoG Weakness 1, Q1). Section 3.1 explains how three kinds of grading checks allow answers to differ from our reference. Appendix G (Table G1) gives the number of checks of each kind.
  5. Failures of GPT-5.6 Sol under WR (Rzip Q2). Section 5.2 states that the additional failures cannot be attributed to planning alone or to the longer workflow alone.
  6. Publication date (Rzip Q4). Appendix H (Table H1) compares pass counts for tasks from earlier and later publications. Section 5.1 refers to this analysis.
  7. Completion times (yuGm Q2). Appendix I (Table I1) reports the median completion times of passing runs. Section 4 refers to this table.
  8. The AI Use Statement now specifies that AI tools were used to reproduce the calculations. Appendix C clarifies that its figure gives the individual runs behind the top row of Fig. 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.