- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CompMat-Bench, a benchmark with 94 tasks drawn from 15 published computational materials studies. It evaluates agents on single tasks and complete workflows made up of related research steps. The authors evaluate three LLMs with different harnesses and analyse their errors.
Critical strengths
The motivation of the paper to evaluate LLM agents on time-consuming computational materials research is good. The analysis of agent trajectory is also meaningful.
Critical weaknesses
The main experiments test only three LLMs. With full guidance, stronger models get very high scores on many single tasks. The strongest model also passes 92% of the complete workflows. These results suggest that the fully guided tasks may be still too easy for current LLMs. Testing more models and workflows would help show whether the benchmark can reliably separate agents with different levels of ability.
Questions for authors
- Can the authors provide more results for other strong LLMs to show whether the benchmark can distinguish their performance?
- Do the authors have estimates of how long a materials researcher would take to complete these tasks, compared with the agents? For time-consuming research tasks, reporting completion times would help show whether agents can save researchers time.
Response to Reviewer yuGm
Thank you for the review.
Weakness and Q1. More strong models and workflows; can the benchmark distinguish them?
We evaluated six more models under the same settings, bringing the total to nine (see the table in item 2 of our response to all reviewers): two open-weight models (Qwen3.8-Max and GLM-5.3), two strong closed models (Gemini 4 Argon and GPT-6.1 Sol), and the two strongest models (GPT-6.2 Astra and Gemini 4 Carbon). (Claude models are not included: the regulations of our funding do not allow us to evaluate them.)
We apologize for not completing the workflow runs for GPT-6.2 Astra and Gemini 4 Carbon, and for not building more workflows within the time available for discussion. Thanks to the reviewer's suggestion, we added an analysis of the seven models with workflow results and found that the single tasks can predict the workflow ranking (item 3 of our response to all reviewers).
We agree with the reviewer about the results: SF still distinguishes the models (GPT-6.2 Astra > GPT-6.1 Sol > GPT-5.6 Sol, 92 > 91 > 85 of 94), but their scores are close and near saturation. There are two reasons. (1) LLMs improve rapidly: GPT-6.1 Sol, released less than three months after GPT-5.6 Sol, passes 91 tasks compared with 85 for GPT-5.6 Sol. (2) The difficulty of our tasks also plays a role. To control the difficulty and avoid saturation, we can add more difficult tasks and remove easy ones, which should rescale the scores but preserve the ranking.
We added the results of the six models (Section 4, Fig. 4 and Appendix F) and the ranking analysis (last paragraph of Section 5.2), and identified score saturation as a limitation in Section 6.
Q2. Time for a researcher compared with the agents.
We thank the reviewer for this suggestion, which strengthens the significance of the benchmark. We can provide some estimates and data, but the discussion period did not give us enough time to have researchers repeat the tasks, so we cannot provide the most direct evidence. We report the following.
(1) Agent time. On single tasks (one-hour limit; the expensive simulations are precomputed), the median time for a passing run ranges from 1.5 to 18 minutes across the seven models. On workflows with full guidance (five-hour limit; the agent runs all simulations itself), the corresponding range is 25 to 195 minutes.
(2) Reproduction time. Our own reproduction of one study took about two weeks and yielded about six tasks. This time includes trial and error and the computation itself. AI tools were inevitably used in the reproduction, and a researcher reviewed and accepted the results, so it is hard to isolate the researcher's effort.
(3) Researcher time, estimated from our experience. It varies. An average task (e.g. preparing the input files of one calculation, or analyzing its output files) takes 0.5 to 1 hour. A complex task can take a researcher working without AI several hours to several days, including time to learn the code from its manual (the LAMMPS manual alone exceeds 1,000 pages). Some tasks (e.g. a saddle-point search for a transition state, or phonon scattering at an interface) represent the main work of an entire paper: the difficulty is in the theoretical derivation and algorithm development, and the computation itself takes little time. We estimate that these tasks take weeks.
We added the agents' completion times to Appendix I (Table I1).