- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MatEval is a benchmark with 24 tasks and 2,135 expert-validated instances, covering five stages of materials research from literature understanding to result analysis. It evaluates 24 LLMs using rule-based and LLM-based evaluation. Results show large performance gaps across tasks, for example, the best score is 95 on MP Database Call but only 41 on AFLOW. Materials knowledge correlates strongly with literature understanding but weakly with data retrieval and experimental planning. No model performs best across all stages, and materials-specialized models show no consistent advantage over general-purpose models.
Critical strengths
1. Practical task design with reproducible evaluation. MatEval evaluates actual research outputs, such as database queries, experiment configurations, and instrument API calls, rather than simple question answering. Its offline evaluators produce reproducible results, and the released response records allow others to verify the scores and analyze model errors. This makes the benchmark useful beyond comparing model rankings.
2. Useful evidence that performance depends heavily on the query interface. Models perform much better on Materials Project queries than on AFLOW, despite both tasks using rule-based evaluation. Seven strong models average 90.3 on MP but only 17.3 on AFLOW. The paper also identifies concrete error patterns and acknowledges that offline scores do not necessarily reflect real-world execution.
Critical weaknesses
1. The tasks behind the main findings do not measure what the paper claims.
Plan Recommendation measures how closely a response reproduces one paper, not planning ability. Scoring uses only the source excerpt and ignores knowledge beyond it, so a valid alternative route can earn little or no credit. The authors acknowledge this limit in D.4. Scores split into one group at 3.5-6.9 and another at 28.6-35.8. GPT-6 Astra, the top model on knowledge QA, falls in the low group. In the high group, 2-11 responses per model are truncated and scored as zero. The paper's weak knowledge-to-planning result, r = 0.17, depends heavily on this task.
Parameter Recommendation uses loose matching rules, and its showcased reference contains an error. A precursor counts as correct if the reference string appears anywhere in the prediction, so a reference of "Zn" accepts any zinc compound. Carrier gas gets credit when half of the reference gases are listed, so listing more gases helps. Case 3 also puts zinc acetate in the "non-metal precursor" field.
The result-analysis tasks have no simple baselines. XRD is a yes/no question about whether a phase is present, yet two models score 38.75%, below chance. The class balance is not reported. Microscopy MAE leaves out failed parses, and the authors state it is not meant for comparing models. Section 4.1 still says Innovator-VL leads this task.
2. The conclusions go further than the evidence supports.
TThe Abstract is inconsistent with the Conclusion. The Abstract presents the results as evidence that LLMs cannot complete the research loop. The Conclusion and Limitations state that the loop was never run end to end.
The claim that specialized models offer no advantage rests on a single model. The only true materials model in the pool is MatChat, an 8B-scale model from 2023. It beats the same-size general model Qwen3-VL-8B on 17 of the 22 non-vision tasks, including 31.95 vs 6.85 on plan recommendation and 18 vs 1 on AFLOW.
The judge and the rankings have not been validated. The judge was chosen by comparing only eight task means, and every calibration response came from Claude. GPT-5.6 Sol, the chosen judge, then grades its own model family. With no confidence intervals, rankings based on 1-2 point gaps over 40-60 items cannot be interpreted.
Questions for authors
Questions
1. In Plan Recommendation, does the evaluated model see the source excerpt? If it does not, the task tests recall of a specific paper rather than planning. How many instances have empty excerpts, and why does an unsupported fabrication on those instances score 50? What causes responses to be truncated before they reach the judge, and what max_tokens value applied to each model? Could experts re-rate a sample of the low group's tier-E responses for feasibility rather than fidelity to the source?
2. Why is zinc acetate the reference for the "non-metal precursor" in Case 3? How many of the 50 references were re-checked? Do the rankings change if precursors are matched by normalized formula instead of substring, and if the carrier-gas criterion also accounts for precision?
3. What is the parse rate for each model on Microscopy Image Analysis? Please also report the MAE of a baseline that always predicts the mean vector. How did MatChat receive image input?
4. What are the class balance and majority-class baseline for XRD, Safety Assessment, and Abnormal-input Detection? Did anyone check that the diagnostic XRD peaks survive the 10x downsampling?
5. How much schema documentation do the AFLOW and MatKG prompts include? Please give a character count so it can be compared with the roughly 57,000 characters in the MP prompt.
6. Can you report item-level judge agreement from the 665 responses you already have? Specifically, please report per-task ICC or kappa, expert agreement before adjudication, and a re-score of GPT-family outputs by a different judge.
7. How were the domain scores built for the knowledge-artifact correlations? Please describe the task weighting, how the lower-is-better MAE task was handled, and how missing entries were treated.
8. Which LLM expanded the seeds in Stage 2? Is it, or another model from the same family, in the evaluated pool? What share of the final 2,135 instances are original expert seeds?
Suggestions
Report bootstrap confidence intervals for each task. On tasks with 40–60 items, a 1–2 point gap is enough to change the ranking.
Evaluate a materials-adapted model alongside its base model. For example, the LLaMat models from Ahlawat et al., 2026, which the paper cites, could be compared with the Llama models they were adapted from.
Align the Abstract with the Conclusion. The sentence presenting the results as evidence that LLMs cannot complete the loop goes beyond what was tested.
Minor
Table 1 lists Lai et al., 2025 as "Chart QA" instead of the benchmark's actual name. The same task is called "Result Image Analysis" in Figure 2 and "Experimental Image Analysis" in Table 2.
4 comments
Answers to the Questions
Q1. Source excerpts, truncation, and feasibility. In the original task, models received the experimental objective but not the source-paper excerpt, which appeared in the reference. In the revised task, all 45 model prompts include a nonempty source excerpt together with candidate operations, resources, parameter options, ordering clues, and verification options. The GPT-5.6 Sol judge deducts semantic errors from evidence fidelity (35 points), plan quality (40), and decision and verification quality (25); its experiment_executable field is diagnostic and does not separately change the score. The revised evaluation yielded 1,080/1,080 valid scores across 24 models × 45 items (mean 96.84). In the original runs, max_tokens was set to the maximum supported by each model interface.
Q2. Case 3 and ranking sensitivity. We thank the reviewer for identifying the zinc acetate field error. We corrected its role in the rescoring reference and rechecked all 50 Experimental Parameter Recommendation references. The revised LLM rubric evaluates precursor identity and role and considers the correctness of both listed and extra carrier gases. Across the paper’s 24 models, the published rule-based and revised LLM rankings have Spearman (ρ = 0.948).
Q3. Microscopy parse rates and MatChat input. The microscopy analysis covers 20 models with image outputs. Gemini 3.6 Flash and Kimi K2.6 each parse 37/40; GLM 5.3 Flash parses 39/40; Kimi K3 has 39/40 task coverage, with all 39 recorded outputs parsed; the other 16 models parse 40/40. Models outside this microscopy cohort are excluded from parse-rate denominators. The mean-vector baseline error is 1.7181. MatChat received images through manual upload to its official website.
Q4. Class balance and XRD sampling. Thank you for highlighting majority-class baselines and the need to verify diagnostic XRD peaks after downsampling. Majority-class baselines are 66.25% for X-ray Diffraction Analysis (53 False, 27 True), 56% for Experimental Safety Assessment (28 No, 22 Yes), and 61% for Abnormal-input Detection (61 Valid, 20 Irrelevant, 19 Ambiguous). Yes. We checked the spectra after applying the same every-tenth-point sampling used in the evaluation. In the positive spectra, the diagnostic AlN peaks near 36° and 37.9° remain identifiable after downsampling, supporting the qualitative phase-presence decision required by this task. We will clarify this task-specific check in the revised manuscript.
Q5. Schema documentation. Excluding each item-specific query, the fixed task-input material (schema guidance and instructions) contains 57,027 characters for MP Database Call, 80,551 for AFLOW Database Call, and 4,723 for MatKG Call.
Q6. Item-level agreement and GPT-family rescoring. Thank you for emphasizing item-level agreement and independent rescoring of GPT-family outputs when assessing judge reliability. Before adjudication, the two experts gave identical scores on 406/665 responses (61.1%); their absolute score difference was at most 0.05 on 505/665 (75.9%) and at most 0.10 on 602/665 (90.5%). Overall pre-adjudication ICC(2,1)= 0.946 and quadratic-weighted Kappa = 0.946. Across the eight tasks, ICC(2,1) ranged from 0.544 to 0.891; the lowest was for Literature Abstract Comparison. We will include the per-task coefficients in the revised manuscript.
On the eight task means from Claude Opus 4.8 responses, the automatic judges compare with the mean score of these two experts as follows (mean absolute difference on the 0–100 scale):
| Judge | ICC(2,1) | Pearson (r) | Mean absolute difference |
|---|---|---|---|
| Claude Opus 4.8 | 0.991 | 0.995 | 2.05 |
| Gemini 3.6 Flash | 0.993 | 0.997 | 1.72 |
| GPT-5.6 Sol | 0.998 | 0.998 | 1.04 |
Separately, Claude Opus 4.8 rescored the GPT-6 Astra, GPT-5.6 Sol, and GPT-5.6 Terra outputs. For each model, the normalized item-level mean absolute difference from GPT-5.6 Sol’s scores was 0.016, with ICCs of 0.996–0.997. GPT-5.6 Sol served as both a judge and an evaluated model.
Q7. Domain construction, MAE, and missing results. We thank the reviewer for identifying the labeling error. The domain score labeled “Experimental Result Analysis” was computed from X-ray Diffraction Analysis alone, and we will correct the label. Microscopy Image Analysis MAE did not enter that domain score or its correlation. Missing image-task outputs were not imputed as zero. For each included task, we average item scores; we then weight tasks equally within each domain and compute Pearson correlations across models with both domain scores. Using the revised scores for Experimental Plan Recommendation and Experimental Parameter Recommendation, together with the Experimental Safety Assessment scores, the Materials Knowledge QA–Experimental Planning correlation across the paper’s 24 models is (r = 0.636), (p < 0.001). We will update the reported correlation and its interpretation in the revised manuscript.
Q8. Seed expansion and original expert share. Thank you for examining the expansion models, their overlap with evaluated systems, and the proportion of expert-authored seeds. Stage 2 used Claude Opus 4.8, GPT-5.6, DeepSeek V4, and other models to generate candidates from expert-authored seeds; some of these models or model families also appear in the evaluated pool. This expansion broadened candidate coverage across the 24 tasks, while expert review determined which candidates met each task’s scientific and reference requirements: 2,135 of 3,849 candidates were retained. All 214 original expert seeds remain in the final 2,135 instances (10.02%). We will add the expansion model identities, their overlap with the evaluated pool, seed counts, and expert-review procedure to the revised manuscript.
Suggestions
Bootstrap intervals. We thank the reviewer for highlighting uncertainty in close comparisons. The reported scores describe performance on MatEval’s fixed, expert-curated test set. We will report task sample sizes alongside the scores and revise the discussion so that 1–2 point differences on 45–50 item tasks are treated as descriptive, rather than as evidence of a reliable model ordering. Our main conclusions concern performance differences across tasks and stages, rather than narrow adjacent ranks.
Supplementary materials-model comparison. We appreciate your suggestion to compare materials-adapted models with their base models to clarify the role of specialization. We evaluate LLaMat-3-Chat alongside Llama-3-8B-Instruct of the same size; Darwin in the materials domain is reported separately rather than treated as a paired Llama-3 baseline. The same task-wise scoring protocols were applied to all 22 text tasks.
Table S1. Supplementary text-task scores (0–100), tasks 1–11.
| Model | Objective QA | Subjective QA | Abnormal Input | Instruction Class. | Entity Recog. | Entity Norm. | Property Ext. | Action Retrieval | Device Ext. | Parameter Ext. | Plan Ext. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| n | 100 | 100 | 100 | 100 | 200 | 100 | 150 | 100 | 50 | 60 | 60 |
| Llama-3-8B-Instruct | 49.00 | 42.40 | 54.00 | 72.00 | 53.42 | 5.92 | 33.34 | 72.96 | 55.63 | 40.03 | 13.20 |
| LLaMat-3-Chat | 38.00 | 24.00 | 26.00 | 1.00 | 5.09 | 0.00 | 2.52 | 7.25 | 10.53 | 4.84 | 2.36 |
| Darwin | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.16 | 2.00 | 2.90 | 2.96 | 0.90 |
Table S2. Supplementary text-task scores (0–100), tasks 12–22.
| Model | Literature QA | Abstract Cmp. | MP Call | AFLOW Call | MatKG Call | Parameter Rec. | Plan Rec. | Safety | Config Setting | Instrument Call | XRD Analysis |
|---|---|---|---|---|---|---|---|---|---|---|---|
| n | 100 | 100 | 100 | 100 | 100 | 50 | 45 | 50 | 50 | 100 | 80 |
| Llama-3-8B-Instruct | 48.20 | 69.43 | 0.00 | 1.00 | 0.00 | 32.54 | 47.64 | 72.00 | 61.00 | 64.00 | 97.50 |
| LLaMat-3-Chat | 15.70 | 1.31 | 0.00 | 0.00 | 0.00 | 12.50 | 40.38 | 48.00 | 21.20 | 0.00 | 68.75 |
| Darwin | 0.60 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 46.44 | 0.00 | 0.00 | 0.00 | 0.00 |
Across these 22 tasks, the equal-task macro-average scores (0–100) are 44.78 for Llama-3-8B-Instruct, 14.97 for LLaMat-3-Chat, and 2.54 for Darwin.
Abstract–Conclusion alignment. We thank the reviewer for this suggestion. MatEval evaluates LLM capabilities from Knowing to Doing across the materials preparation research loop. As defined in the Introduction, Doing means producing the research output required at a given stage, including structured literature extractions, database queries, experimental plans, process configurations, instrument calls, and result interpretations. Section 3.1 explains why each task is evaluated independently: stage-specific output requirements identify which research artifacts a model produces reliably. We will revise the Abstract and Conclusion to state the supported finding: systematic performance differences across the stages of the materials preparation research loop. Connected-workflow validation can build on these stage-specific diagnostics.
Minor Corrections
We thank the reviewer for noting these presentation errors. We will correct the Lai et al. (2025) benchmark name in Table 1 to MatCha and use Experimental Image Analysis consistently in Figure 2, Table 2, and the text.
We thank the reviewer for recognizing MatEval’s evaluation of research outputs across the materials preparation research loop, its stage-specific scoring protocols, and its released response records. We present revised task evaluations, baselines, and agreement analyses below; we will incorporate these results into the revised manuscript.
Response to Concern 1: Stage-Specific Task Validity
Experimental Plan Recommendation. We thank the reviewer for the task-validity critique. The revised task measures experimental planning given explicit evidence, resources, and constraints. Models see the source excerpt, objective, candidate operations, resources, parameter options, ordering clues, and verification options. All 45 revised instances have nonempty excerpts. The LLM-based judge deducts for semantic errors while accepting equivalent plans and valid reorderings of independent steps. The revised evaluation covers 24 models and 45 instances per model, yielding 1,080/1,080 valid scores, a mean of 96.84, and model means ranging from 93.67 to 99.80.
Experimental Parameter Recommendation. We appreciate your careful scrutiny of chemical reference annotations and the validity of precursor and carrier-gas matching. We corrected the misplaced zinc acetate field in the rescoring reference and rechecked all 50 task references. We replaced substring and carrier-gas recall rules with GPT-5.6 Sol-based evaluation of precursor identity and role, temperatures, substrate, hold time, and carrier-gas identities and flows. The rubric accepts chemically equivalent names and formulas and penalizes unsupported extra components. The paper’s 24 models have 50 valid revised scores each. Comparing the paper’s reported rule-based scores with the revised LLM scores across these models gives a Spearman rank correlation of 0.948.
Experimental Result Analysis. For Microscopy Image Analysis, we will report the constant mean-vector baseline (MAE 1.7181) and present each model’s MAE alongside its parse rate. We will revise the microscopy discussion accordingly.
Response to Concern 2: Scope of the Conclusions
Materials preparation research loop. We understand this concern as emphasizing the distinction between task-level evidence and end-to-end validation of the research loop. MatEval evaluates LLM capabilities from Knowing to Doing through 24 independently evaluated tasks across five stages of the materials preparation research loop and two foundational competency domains. As defined in the Introduction, Doing means producing the research output required at each stage. These stage-specific diagnostics reveal systematic performance differences across the loop and identify the types of research outputs that models produce reliably. We will align the Abstract and Conclusion with this interpretation.
Materials-specialized systems. We thank the reviewer for noting that MatChat scored higher than Qwen3-VL-8B on 17 of 22 text tasks. To our knowledge, MatChat is a materials-specialized system, evaluated through its official web interface. We also examined two additional materials-focused models with distinct intended uses: LLaMat-3-Chat, a materials-research copilot for literature analysis and information extraction, and Darwin(Xie T, Wan Y, Liu Y, et al. Darwin 1.5: Large language models as materials science adapted learners[J]. arXiv preprint arXiv:2412.11970, 2024.), which emphasizes natural-language materials property prediction and discovery. In our supplementary run, LLaMat-3-Chat scored below the tested Meta-Llama-3-8B-Instruct checkpoint on 20 of 22 text tasks (with two ties); Darwin returned empty answers on 758/1,995 items and scored zero on 15 of 22 tasks. Given these low task-level results, we did not include either model in the paper’s 24-model main comparison. We report their full task scores below and will state the conclusion at the scope of the systems and tasks evaluated in MatEval: materials specialization did not yield a consistent advantage across these research outputs.
Judge and rankings. Judge selection used task means from eight tasks, with calibration responses from Claude. We will state both facts and distinguish task-mean calibration from item-level agreement. Among the 665 responses independently scored by two human experts before adjudication, both assigned identical scores to 406 (61.1%); their absolute score difference was at most 0.05 for 505 (75.9%) and at most 0.10 for 602 (90.5%). Overall inter-expert ICC(2,1)= 0.946 and quadratic-weighted Kappa = 0.946. Q6 separately reports per-task inter-expert agreement, automatic-judge calibration against the two-expert mean, and Claude Opus 4.8 rescoring of GPT-family outputs. We will report uncertainty alongside close model scores.