acceptodds
Desk rejected at ICLR 2027

MatEval: A Benchmark for LLMs from Knowing to Doing in the Materials Preparation Research Loop

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) hold promise for materials science, yet existing benchmarks fall short of covering the full research loop of real-world materials preparation. To bridge this gap, we present MatEval, a comprehensive benchmark assessing LLMs across the materials preparation research loop: from foundational competencies (Materials Knowledge QA and Instruction Understanding) through five stages (Literature Understanding, Data Retrieval, Experimental Planning, Experimental Execution, and Experimental Result Analysis). MatEval contains 24 tasks with a total of 2,135 expert-curated instances and stage-specific scoring protocols for evaluating model capabilities from knowing to doing. We evaluate 24 general-purpose and materials-specialized LLMs. Results reveal systematic performance differences across stages, with no model performing consistently across the full loop. Notably, materials-specialized models show no consistent advantage over general-purpose models. MatEval redefines evaluation from answering materials questions to operating across the materials preparation research loop, providing evidence that current LLMs remain unable to complete the full research loop.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Materials Science, Large Language Models, AI4Science

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FvfyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MatEval is a benchmark with 24 tasks and 2,135 expert-validated instances, covering five stages of materials research from literature understanding to result analysis. It evaluates 24 LLMs using rule-based and LLM-based evaluation. Results show large performance gaps across tasks, for example, the best score is 95 on MP Database Call but only 41 on AFLOW. Materials knowledge correlates strongly with literature understanding but weakly with data retrieval and experimental planning. No model performs best across all stages, and materials-specialized models show no consistent advantage over general-purpose models.

Critical strengths

1. Practical task design with reproducible evaluation. MatEval evaluates actual research outputs, such as database queries, experiment configurations, and instrument API calls, rather than simple question answering. Its offline evaluators produce reproducible results, and the released response records allow others to verify the scores and analyze model errors. This makes the benchmark useful beyond comparing model rankings.

2. Useful evidence that performance depends heavily on the query interface. Models perform much better on Materials Project queries than on AFLOW, despite both tasks using rule-based evaluation. Seven strong models average 90.3 on MP but only 17.3 on AFLOW. The paper also identifies concrete error patterns and acknowledges that offline scores do not necessarily reflect real-world execution.

Critical weaknesses

1. The tasks behind the main findings do not measure what the paper claims.

Plan Recommendation measures how closely a response reproduces one paper, not planning ability. Scoring uses only the source excerpt and ignores knowledge beyond it, so a valid alternative route can earn little or no credit. The authors acknowledge this limit in D.4. Scores split into one group at 3.5-6.9 and another at 28.6-35.8. GPT-6 Astra, the top model on knowledge QA, falls in the low group. In the high group, 2-11 responses per model are truncated and scored as zero. The paper's weak knowledge-to-planning result, r = 0.17, depends heavily on this task.

Parameter Recommendation uses loose matching rules, and its showcased reference contains an error. A precursor counts as correct if the reference string appears anywhere in the prediction, so a reference of "Zn" accepts any zinc compound. Carrier gas gets credit when half of the reference gases are listed, so listing more gases helps. Case 3 also puts zinc acetate in the "non-metal precursor" field.

The result-analysis tasks have no simple baselines. XRD is a yes/no question about whether a phase is present, yet two models score 38.75%, below chance. The class balance is not reported. Microscopy MAE leaves out failed parses, and the authors state it is not meant for comparing models. Section 4.1 still says Innovator-VL leads this task.

2. The conclusions go further than the evidence supports.

TThe Abstract is inconsistent with the Conclusion. The Abstract presents the results as evidence that LLMs cannot complete the research loop. The Conclusion and Limitations state that the loop was never run end to end.

The claim that specialized models offer no advantage rests on a single model. The only true materials model in the pool is MatChat, an 8B-scale model from 2023. It beats the same-size general model Qwen3-VL-8B on 17 of the 22 non-vision tasks, including 31.95 vs 6.85 on plan recommendation and 18 vs 1 on AFLOW.

The judge and the rankings have not been validated. The judge was chosen by comparing only eight task means, and every calibration response came from Claude. GPT-5.6 Sol, the chosen judge, then grades its own model family. With no confidence intervals, rankings based on 1-2 point gaps over 40-60 items cannot be interpreted.

Questions for authors

Questions

1. In Plan Recommendation, does the evaluated model see the source excerpt? If it does not, the task tests recall of a specific paper rather than planning. How many instances have empty excerpts, and why does an unsupported fabrication on those instances score 50? What causes responses to be truncated before they reach the judge, and what max_tokens value applied to each model? Could experts re-rate a sample of the low group's tier-E responses for feasibility rather than fidelity to the source?

2. Why is zinc acetate the reference for the "non-metal precursor" in Case 3? How many of the 50 references were re-checked? Do the rankings change if precursors are matched by normalized formula instead of substring, and if the carrier-gas criterion also accounts for precision?

3. What is the parse rate for each model on Microscopy Image Analysis? Please also report the MAE of a baseline that always predicts the mean vector. How did MatChat receive image input?

4. What are the class balance and majority-class baseline for XRD, Safety Assessment, and Abnormal-input Detection? Did anyone check that the diagnostic XRD peaks survive the 10x downsampling?

5. How much schema documentation do the AFLOW and MatKG prompts include? Please give a character count so it can be compared with the roughly 57,000 characters in the MP prompt.

6. Can you report item-level judge agreement from the 665 responses you already have? Specifically, please report per-task ICC or kappa, expert agreement before adjudication, and a re-score of GPT-family outputs by a different judge.

7. How were the domain scores built for the knowledge-artifact correlations? Please describe the task weighting, how the lower-is-better MAE task was handled, and how missing entries were treated.

8. Which LLM expanded the seeds in Stage 2? Is it, or another model from the same family, in the evaluated pool? What share of the final 2,135 instances are original expert seeds?

Suggestions

Report bootstrap confidence intervals for each task. On tasks with 40–60 items, a 1–2 point gap is enough to change the ranking.

Evaluate a materials-adapted model alongside its base model. For example, the LLaMat models from Ahlawat et al., 2026, which the paper cites, could be compared with the Llama models they were adapted from.

Align the Abstract with the Conclusion. The sentence presenting the results as evidence that LLMs cannot complete the loop goes beyond what was tested.

Minor

Table 1 lists Lai et al., 2025 as "Chart QA" instead of the benchmark's actual name. The same task is called "Result Image Analysis" in Figure 2 and "Experimental Image Analysis" in Table 2.

4 comments

We thank the reviewer for recognizing MatEval’s evaluation of research outputs across the materials preparation research loop, its stage-specific scoring protocols, and its released response records. We present revised task evaluations, baselines, and agreement analyses below; we will incorporate these results into the revised manuscript.

Response to Concern 1: Stage-Specific Task Validity

Experimental Plan Recommendation. We thank the reviewer for the task-validity critique. The revised task measures experimental planning given explicit evidence, resources, and constraints. Models see the source excerpt, objective, candidate operations, resources, parameter options, ordering clues, and verification options. All 45 revised instances have nonempty excerpts. The LLM-based judge deducts for semantic errors while accepting equivalent plans and valid reorderings of independent steps. The revised evaluation covers 24 models and 45 instances per model, yielding 1,080/1,080 valid scores, a mean of 96.84, and model means ranging from 93.67 to 99.80.

Experimental Parameter Recommendation. We appreciate your careful scrutiny of chemical reference annotations and the validity of precursor and carrier-gas matching. We corrected the misplaced zinc acetate field in the rescoring reference and rechecked all 50 task references. We replaced substring and carrier-gas recall rules with GPT-5.6 Sol-based evaluation of precursor identity and role, temperatures, substrate, hold time, and carrier-gas identities and flows. The rubric accepts chemically equivalent names and formulas and penalizes unsupported extra components. The paper’s 24 models have 50 valid revised scores each. Comparing the paper’s reported rule-based scores with the revised LLM scores across these models gives a Spearman rank correlation of 0.948.

Experimental Result Analysis. For Microscopy Image Analysis, we will report the constant mean-vector baseline (MAE 1.7181) and present each model’s MAE alongside its parse rate. We will revise the microscopy discussion accordingly.

Response to Concern 2: Scope of the Conclusions

Materials preparation research loop. We understand this concern as emphasizing the distinction between task-level evidence and end-to-end validation of the research loop. MatEval evaluates LLM capabilities from Knowing to Doing through 24 independently evaluated tasks across five stages of the materials preparation research loop and two foundational competency domains. As defined in the Introduction, Doing means producing the research output required at each stage. These stage-specific diagnostics reveal systematic performance differences across the loop and identify the types of research outputs that models produce reliably. We will align the Abstract and Conclusion with this interpretation.

Materials-specialized systems. We thank the reviewer for noting that MatChat scored higher than Qwen3-VL-8B on 17 of 22 text tasks. To our knowledge, MatChat is a materials-specialized system, evaluated through its official web interface. We also examined two additional materials-focused models with distinct intended uses: LLaMat-3-Chat, a materials-research copilot for literature analysis and information extraction, and Darwin(Xie T, Wan Y, Liu Y, et al. Darwin 1.5: Large language models as materials science adapted learners[J]. arXiv preprint arXiv:2412.11970, 2024.), which emphasizes natural-language materials property prediction and discovery. In our supplementary run, LLaMat-3-Chat scored below the tested Meta-Llama-3-8B-Instruct checkpoint on 20 of 22 text tasks (with two ties); Darwin returned empty answers on 758/1,995 items and scored zero on 15 of 22 tasks. Given these low task-level results, we did not include either model in the paper’s 24-model main comparison. We report their full task scores below and will state the conclusion at the scope of the systems and tasks evaluated in MatEval: materials specialization did not yield a consistent advantage across these research outputs.

Judge and rankings. Judge selection used task means from eight tasks, with calibration responses from Claude. We will state both facts and distinguish task-mean calibration from item-level agreement. Among the 665 responses independently scored by two human experts before adjudication, both assigned identical scores to 406 (61.1%); their absolute score difference was at most 0.05 for 505 (75.9%) and at most 0.10 for 602 (90.5%). Overall inter-expert ICC(2,1)= 0.946 and quadratic-weighted Kappa = 0.946. Q6 separately reports per-task inter-expert agreement, automatic-judge calibration against the two-expert mean, and Claude Opus 4.8 rescoring of GPT-family outputs. We will report uncertainty alongside close model scores.

Answers to the Questions

Q1. Source excerpts, truncation, and feasibility. In the original task, models received the experimental objective but not the source-paper excerpt, which appeared in the reference. In the revised task, all 45 model prompts include a nonempty source excerpt together with candidate operations, resources, parameter options, ordering clues, and verification options. The GPT-5.6 Sol judge deducts semantic errors from evidence fidelity (35 points), plan quality (40), and decision and verification quality (25); its experiment_executable field is diagnostic and does not separately change the score. The revised evaluation yielded 1,080/1,080 valid scores across 24 models × 45 items (mean 96.84). In the original runs, max_tokens was set to the maximum supported by each model interface.

Q2. Case 3 and ranking sensitivity. We thank the reviewer for identifying the zinc acetate field error. We corrected its role in the rescoring reference and rechecked all 50 Experimental Parameter Recommendation references. The revised LLM rubric evaluates precursor identity and role and considers the correctness of both listed and extra carrier gases. Across the paper’s 24 models, the published rule-based and revised LLM rankings have Spearman (ρ = 0.948).

Q3. Microscopy parse rates and MatChat input. The microscopy analysis covers 20 models with image outputs. Gemini 3.6 Flash and Kimi K2.6 each parse 37/40; GLM 5.3 Flash parses 39/40; Kimi K3 has 39/40 task coverage, with all 39 recorded outputs parsed; the other 16 models parse 40/40. Models outside this microscopy cohort are excluded from parse-rate denominators. The mean-vector baseline error is 1.7181. MatChat received images through manual upload to its official website.

Q4. Class balance and XRD sampling. Thank you for highlighting majority-class baselines and the need to verify diagnostic XRD peaks after downsampling. Majority-class baselines are 66.25% for X-ray Diffraction Analysis (53 False, 27 True), 56% for Experimental Safety Assessment (28 No, 22 Yes), and 61% for Abnormal-input Detection (61 Valid, 20 Irrelevant, 19 Ambiguous). Yes. We checked the spectra after applying the same every-tenth-point sampling used in the evaluation. In the positive spectra, the diagnostic AlN peaks near 36° and 37.9° remain identifiable after downsampling, supporting the qualitative phase-presence decision required by this task. We will clarify this task-specific check in the revised manuscript.

Q5. Schema documentation. Excluding each item-specific query, the fixed task-input material (schema guidance and instructions) contains 57,027 characters for MP Database Call, 80,551 for AFLOW Database Call, and 4,723 for MatKG Call.

Q6. Item-level agreement and GPT-family rescoring. Thank you for emphasizing item-level agreement and independent rescoring of GPT-family outputs when assessing judge reliability. Before adjudication, the two experts gave identical scores on 406/665 responses (61.1%); their absolute score difference was at most 0.05 on 505/665 (75.9%) and at most 0.10 on 602/665 (90.5%). Overall pre-adjudication ICC(2,1)= 0.946 and quadratic-weighted Kappa = 0.946. Across the eight tasks, ICC(2,1) ranged from 0.544 to 0.891; the lowest was for Literature Abstract Comparison. We will include the per-task coefficients in the revised manuscript.

On the eight task means from Claude Opus 4.8 responses, the automatic judges compare with the mean score of these two experts as follows (mean absolute difference on the 0–100 scale):

JudgeICC(2,1)Pearson (r)Mean absolute difference
Claude Opus 4.80.9910.9952.05
Gemini 3.6 Flash0.9930.9971.72
GPT-5.6 Sol0.9980.9981.04

Separately, Claude Opus 4.8 rescored the GPT-6 Astra, GPT-5.6 Sol, and GPT-5.6 Terra outputs. For each model, the normalized item-level mean absolute difference from GPT-5.6 Sol’s scores was 0.016, with ICCs of 0.996–0.997. GPT-5.6 Sol served as both a judge and an evaluated model.

Q7. Domain construction, MAE, and missing results. We thank the reviewer for identifying the labeling error. The domain score labeled “Experimental Result Analysis” was computed from X-ray Diffraction Analysis alone, and we will correct the label. Microscopy Image Analysis MAE did not enter that domain score or its correlation. Missing image-task outputs were not imputed as zero. For each included task, we average item scores; we then weight tasks equally within each domain and compute Pearson correlations across models with both domain scores. Using the revised scores for Experimental Plan Recommendation and Experimental Parameter Recommendation, together with the Experimental Safety Assessment scores, the Materials Knowledge QA–Experimental Planning correlation across the paper’s 24 models is (r = 0.636), (p < 0.001). We will update the reported correlation and its interpretation in the revised manuscript.

Q8. Seed expansion and original expert share. Thank you for examining the expansion models, their overlap with evaluated systems, and the proportion of expert-authored seeds. Stage 2 used Claude Opus 4.8, GPT-5.6, DeepSeek V4, and other models to generate candidates from expert-authored seeds; some of these models or model families also appear in the evaluated pool. This expansion broadened candidate coverage across the 24 tasks, while expert review determined which candidates met each task’s scientific and reference requirements: 2,135 of 3,849 candidates were retained. All 214 original expert seeds remain in the final 2,135 instances (10.02%). We will add the expansion model identities, their overlap with the evaluated pool, seed counts, and expert-review procedure to the revised manuscript.

Suggestions

Bootstrap intervals. We thank the reviewer for highlighting uncertainty in close comparisons. The reported scores describe performance on MatEval’s fixed, expert-curated test set. We will report task sample sizes alongside the scores and revise the discussion so that 1–2 point differences on 45–50 item tasks are treated as descriptive, rather than as evidence of a reliable model ordering. Our main conclusions concern performance differences across tasks and stages, rather than narrow adjacent ranks.

Supplementary materials-model comparison. We appreciate your suggestion to compare materials-adapted models with their base models to clarify the role of specialization. We evaluate LLaMat-3-Chat alongside Llama-3-8B-Instruct of the same size; Darwin in the materials domain is reported separately rather than treated as a paired Llama-3 baseline. The same task-wise scoring protocols were applied to all 22 text tasks.

Table S1. Supplementary text-task scores (0–100), tasks 1–11.

ModelObjective QASubjective QAAbnormal InputInstruction Class.Entity Recog.Entity Norm.Property Ext.Action RetrievalDevice Ext.Parameter Ext.Plan Ext.
n100100100100200100150100506060
Llama-3-8B-Instruct49.0042.4054.0072.0053.425.9233.3472.9655.6340.0313.20
LLaMat-3-Chat38.0024.0026.001.005.090.002.527.2510.534.842.36
Darwin0.000.000.000.000.000.000.162.002.902.960.90

Table S2. Supplementary text-task scores (0–100), tasks 12–22.

ModelLiterature QAAbstract Cmp.MP CallAFLOW CallMatKG CallParameter Rec.Plan Rec.SafetyConfig SettingInstrument CallXRD Analysis
n1001001001001005045505010080
Llama-3-8B-Instruct48.2069.430.001.000.0032.5447.6472.0061.0064.0097.50
LLaMat-3-Chat15.701.310.000.000.0012.5040.3848.0021.200.0068.75
Darwin0.600.000.000.000.000.0046.440.000.000.000.00

Across these 22 tasks, the equal-task macro-average scores (0–100) are 44.78 for Llama-3-8B-Instruct, 14.97 for LLaMat-3-Chat, and 2.54 for Darwin.

Abstract–Conclusion alignment. We thank the reviewer for this suggestion. MatEval evaluates LLM capabilities from Knowing to Doing across the materials preparation research loop. As defined in the Introduction, Doing means producing the research output required at a given stage, including structured literature extractions, database queries, experimental plans, process configurations, instrument calls, and result interpretations. Section 3.1 explains why each task is evaluated independently: stage-specific output requirements identify which research artifacts a model produces reliably. We will revise the Abstract and Conclusion to state the supported finding: systematic performance differences across the stages of the materials preparation research loop. Connected-workflow validation can build on these stage-specific diagnostics.

Minor Corrections

We thank the reviewer for noting these presentation errors. We will correct the Lai et al. (2025) benchmark name in Table 1 to MatCha and use Experimental Image Analysis consistently in Figure 2, Table 2, and the text.

Reviewer_PzgjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MatEval, a benchmark containing 2,135 instances across 24 tasks covering different stages of materials-preparation research. The benchmark evaluates 24 LLMs on literature understanding, data retrieval, experimental planning, configuration and instrument-call generation, and experimental result analysis.

The results reveal substantial performance differences across tasks and interfaces, with no consistent advantage for materials-specialized models. The main contribution is a unified collection of research-oriented tasks and task-specific evaluation protocols.

My main assessment is that the benchmark covers a useful range of materials research tasks, but its contribution is largely descriptive. I would like to see deeper quantitative analysis explaining why models fail across different tasks, rather than primarily reporting performance differences. I am also concerned about the validity of some evaluation metrics, particularly whether semantic similarity scores can distinguish scientifically meaningful errors from equivalent expressions. Additional baselines and targeted validation would make the benchmark more informative and convincing.

Critical strengths

Practical coverage of research-oriented tasks. MatEval extends materials-science evaluation beyond conventional knowledge QA to structured research outputs, including database queries, experimental configurations, and instrument calls. The expert-curated task collection provides a useful foundation for evaluating LLM capabilities across different stages of materials research.

Critical weaknesses
  1. Limited diagnostic insight beyond task-level performance comparisons.

The paper evaluates a broad collection of tasks, but the analysis does not sufficiently explain why models fail to translate scientific knowledge into correct research outputs. The observed performance differences may arise from missing scientific knowledge, difficulty using provided evidence, or failures to follow interface constraints.

A controlled study on the same problems, varying scientific evidence and interface assistance, could quantify these factors. Reporting scientific correctness, interface compliance, and joint success would provide a clearer understanding of the observed failures. Such findings could also guide the design of more informative benchmark tasks. Without this analysis, the central "knowing-to-doing" contribution remains primarily descriptive.

  1. Insufficient validation of scientific correctness in the scoring protocols.

Experimental Image Analysis relies on embedding similarity and BERTScore, which may fail to distinguish scientifically important errors, such as changes in numerical values, units, or trend directions. The paper does not validate whether these metrics reliably penalize such errors while preserving scores for scientifically equivalent expressions.

Experimental Plan Recommendation raises a similar concern: scoring fidelity to a reference procedure does not necessarily measure the feasibility of alternative experimental plans.

These issues directly affect the interpretation of the reported results. A targeted evaluation using correct paraphrases, controlled scientific errors, and expert-verified alternative plans would substantially strengthen the benchmark's validity.

Questions for authors
  1. Can the authors provide controlled evidence distinguishing scientific knowledge limitations from interface-related failures? For example, how does performance change when additional scientific evidence or interface assistance is provided for the same problems?

  2. How reliably do the evaluation metrics distinguish scientifically correct paraphrases from answers containing numerical, unit, or trend-direction errors? How are feasible experimental plans that differ from the reference procedure scored?

  3. How stable are the main model comparisons after accounting for evaluator uncertainty, missing parsing coverage in microscopy, and the truncated planning responses reported in Appendix B?

1 comment

We thank the reviewer for the constructive comments. We provide a controlled evidence ablation, scientific-correctness scoring, evaluator sensitivity analysis, and microscopy parse coverage. We will incorporate these results into the revised manuscript.

Q1: How do scientific evidence and interface conditions affect performance?

A1: We appreciate your emphasis on distinguishing knowledge limitations from difficulties using evidence or following interface constraints. We created an item-matched question-only control for Literature-Based QA by removing the evidence passage from the same 100 instances while retaining each question, reference answer, item order, generation settings, and evaluator. Across the paper’s 24 models, the macro-average rises from 52.11 without the passage to 64.18 with it (+12.07 points); 23/24 models improve. Innovator-VL-8B is the sole exception, decreasing from 51.6 to 51.2 ( − 0.4 points). Overall, this brings benefits to nearly all models.

Q2: Do the evaluators distinguish scientific errors from paraphrases and feasible plans from reference wording?

A2: Thank you for emphasizing that scoring should penalize scientific errors while accepting equivalent expressions and feasible alternative plans. For Experimental Image Analysis, we replaced similarity-based scoring with an LLM-based scientific-correctness evaluator. The judge first assigns points for scientific correctness (0–35), quantitative fidelity (0–25), trend and comparison fidelity (0–20), coverage of salient evidence (0–15), and grounding and clarity (0–5). Incorrect claims and omissions reduce the relevant dimension scores; equivalent paraphrases, reordered statements, and valid unit conversions are not penalized. The five dimension scores are summed, then the lowest applicable cap is applied: 40/100 for a central trend or sign reversal, an incorrect central material or condition, or a conclusion that contradicts the reference; 55/100 for a major numerical, unit, dimensional, or order-of-magnitude error that changes the scientific interpretation; and 70/100 when the core conclusion is correct but a major variable, extremum, comparison, or causal explanation is wrong. Empty, non-responsive, or unrelated answers score 0.

We also redesigned Experimental Plan Recommendation as planning under explicit evidence, resources, and constraints. Each item supplies an objective, source evidence, candidate operations and resources, parameter options, dependencies, and verification choices. The evaluator scores evidence fidelity (35%), plan quality (40%), and decision and verification quality (25%). It accepts semantically equivalent steps, merged or split operations, and valid reorderings of independent steps. Across 24 models × 45 items, all 1,080/1,080 responses have valid scores. The overall mean is 96.84, with dimension means of 97.48, 97.56, and 94.77, respectively. We will update the manuscript’s task description, protocol, and results.

Q3: How robust are comparisons to evaluator choice, microscopy parsing, and planning-response truncation?

A3 — Evaluator choice. We appreciate your careful examination of comparison stability under evaluator uncertainty, incomplete parsing, and truncated responses. Section 4.2 compares the scores assigned by our automatic judge with those of two materials-science experts on 665 responses across eight tasks. We will also incorporate the updated evaluation results for the two tasks into the revised manuscript. At the task-mean level, GPT-5.6 Sol differs from expert consensus by 1.04 points on average in absolute value (ICC(2,1) 0.998, Pearson (r = 0.998); maximum deviation 2.19; signed bias − 0.75 points). For Experimental Image Analysis, scores for models valid under both scoring protocols have Pearson (r = 0.690) and Spearman (ρ = 0.332); the largest model-level change is 8.86 points. We will report this sensitivity with the revised scores and avoid treating small gaps as decisive rankings.

A3 — Microscopy parse coverage. We will report parse rate alongside MAE. Gemini 3.6 Flash and Kimi K2.6 each parse 37/40 (92.5%) aligned responses; GLM 5.3 Flash parses 39/40 (97.5%); Kimi K3 parses all 39 recorded responses (39/40 task coverage); the other evaluated models parse 40/40. Microscopy MAE is reported at the task level rather than pooled into the cross-model aggregate.

A3 — Planning responses. We reran the revised 45-item Experimental Plan Recommendation task across 24 models and obtained 1,080/1,080 valid scores. We will replace the corresponding results and Appendix B description with this revised protocol and its scores.

Reviewer_AupTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a benchmark focusing on the issue: In material science, LLM can answer knowledge questions, but it doesn't mean it can be used in the research loop. The later is more about extracting experiment setup from papers, database searching, designing experiment with constraints, analyze experiment results and so on. The proposed benchmark mostly focusing on those topics.

The benchmark construction pipeline includes: expert sourced research materials, expert designed seed experiment setups + LLM extension. Expert filtered and reviewed final question set. Expert justified evaluation rubrics.

Critical strengths
  • The benchmark includes many human annotations, which is always a valuable contribution the community.

  • The topic about science agent is well motivated.

  • The task coverage is broad, including different stage of science research lifecycle.

Critical weaknesses
  • Benchmark for research agent is not a completely new concept, there are previous benchmark doing so, e.g. MatTools and ScienceAgentBench.

  • The conclusion about material domain model is not always better than general model is not novel neither, which has been discussed in previous work such as MatTools.

  • Although the benchmark includes all steps in a research cycle, but there is no a end-to-end experiment validation.

Questions for authors

N/a

1 comment

We thank the reviewer for the thoughtful assessment and for recognizing the value of our expert annotations, the motivation of the benchmark, and its coverage of the materials preparation research loop. We address the three concerns below.

Q1: Novelty compared with existing benchmarks.

We appreciate your emphasis on clarifying the contribution of MatEval relative to MatTools and ScienceAgentBench. We agree that prior benchmarks evaluate LLMs on scientific research tasks. MatTools evaluates LLMs’ ability to answer materials science questions by generating and safely executing code using physics-based computational packages. ScienceAgentBench evaluates performance on data-driven scientific discovery tasks across four disciplines, assessing generated Python programs, execution results, and costs. Our benchmark, MatEval, assesses LLMs across the full materials preparation research loop—from literature understanding and data retrieval to experimental planning, execution, and result analysis—while also evaluating their capabilities in materials knowledge QA and instruction understanding. It provides 24 independently evaluated tasks and 2,135 expert-curated instances, with stage-specific scoring protocols to assess capabilities from knowing to doing.

Q2: The finding about specialized models is not new.

Thank you for asking how our specialization analysis extends the observation already reported in MatTools. We agree that MatTools has reported a similar finding. However, to our knowledge, MatEval is the first to examine whether materials specialization provides consistent benefits across all five stages of the materials preparation research loop. Our evaluation of 24 general-purpose and materials-specialized LLMs reveals systematic performance differences across the loop, with specialized models showing no consistent advantage over general-purpose models. This extends the prior observation to whether domain specialization supports the transition from answering materials questions to producing the research outputs required throughout materials preparation.

Q3: End-to-end experimental validation.

We appreciate this suggestion. We understand this concern as asking whether stage-level capabilities hold in an end-to-end experimental workflow. MatEval is designed to assess LLM capabilities across the full materials preparation research loop through stage-specific diagnostics. As defined in the Introduction, “Doing” refers to producing the research artifacts required at each stage, including structured literature extractions, database queries, experimental plans, process configurations, instrument calls, and result interpretations.

Independent evaluation is central to this diagnostic objective. As explained in Section 3.1, anchoring each task to a specific stage and output requirement allows us to identify where a model falls short and which type of research output it struggles to produce. An end-to-end outcome alone would not provide the same diagnostic resolution.

End-to-end validation can additionally examine how these capabilities interact in a connected workflow, including cross-stage error propagation. We will align the Abstract with the Introduction and Section 3.1: MatEval provides stage-specific diagnostic evidence about the research outputs required across the materials preparation research loop. Connected-workflow validation can build on this foundation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.