- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a dataset to test whether an agent can deliver reproducible materials. It must ensure that the delivered sample recipe meets the standard, matches the actual production records, and can be reproduced in new batches. The authors tested Claude and Codex and provided related analyses.
Critical strengths
-
The authors propose a dataset focused on critical scientific domains, examining whether AI can deliver reproducible materials.
-
This benchmark features an insightful design that combines materials discovery with reproducible synthesis, ensuring the delivery of a reliable process.
-
The authors evaluated strong coding agents and conducted experimental analysis.
Critical weaknesses
-
This is a dataset paper, but the authors didn't include the dataset in the supplementary materials.
-
The paper only tested two models, which is too limited. The authors didn't test any open-source models to see the gap between open-source and closed-source models.
-
The analysis of the tested models isn't insightful enough. I would expect a deep expert analysis or case study to point out how current models can be improved further.
Questions for authors
What is the reasoning effort that you were using during testing the performance of Claude Code and Codex?
5 comments
Response to Reviewer hZHq
W3. Deeper expert analysis and case studies are needed to explain model failures and identify areas for improvement.
Following your suggestion for a deeper analysis of failure causes, we examined all 312 Sol and Opus campaigns and replayed the 124 unsuccessful ones. We identified difficulties in applying mechanisms under the relevant conditions, satisfying multiple requirements together, and retaining promising preparations through further processing and final selection. The specific decisions and their outcomes are summarized below.
| Problem identified | Evidence from trajectories and replay |
|---|---|
| Mechanism interpretation and joint constraints | In a polymer case, the agent invoked crystallinity at 60°C, where that mechanism was inactive. Gentler drying increased conductivity but violated stiffness and residual-solvent requirements. |
| Material and process choices remain incompatible | In 52/124 failures, replacing both material and process with the reference passed, while neither replacement alone did. In the ALD case, repeated tuning of one chemistry left the leakage requirement unresolved. |
| Further processing or final selection loses a qualifying preparation | Replay identifies 18/124 failures with a lost qualifying state or an undelivered qualifying alternative. In the alloy case, additional aging reduced hardness below the target. |
| Observed success does not persist across fresh batches | In 14/124 failures, only fresh-batch acceptance failed. The alloy case had two observed samples above the targets, but the submitted recipe passed only one of three fresh batches. |
These counts come from replay diagnostics of the 124 failed campaigns; categories may overlap. Reference replacements and qualifying states are evaluated in replay.
These cases clarify the decisions that agents struggle with: which material or process conditions to change, when to stop processing, and which sample–recipe pair to deliver. In the polymer and ALD trajectories examined, continued tuning did not satisfy all constraints. The alloy case shows how successful observations on existing samples can still be followed by unreliable fresh-batch delivery. We summarize these findings in Section 4.3 and provide the replay procedures and corresponding trajectories in Appendix D.
Q1. Which reasoning-effort settings were used for Claude Code and Codex?
We explicitly set the reasoning effort to high for both Codex and Claude Code. To assess recent model progress, we updated their underlying models to GPT-6.1 Sol and Claude Opus 5.5, and added evaluations of Qwen3.6-35B-A3B and Frontis-MA1-35B using their default thinking settings. The configurations are summarized below.
| Model | Reasoning configuration |
|---|---|
| GPT-6.1 Sol | High, explicitly set |
| Claude Opus 5.5 | High, explicitly set |
| Qwen3.6-35B-A3B | Thinking enabled, model default |
| Frontis-MA1-35B | Thinking enabled, model default |
We also examined whether increasing reasoning effort improves task completion by comparing high and xhigh on ten prespecified tasks, with three attempts per task. Sol succeeds in 17/30 attempts with high and 14/30 with xhigh, while Opus succeeds in 17/30 and 16/30, respectively. On these tasks, neither model achieves a higher success rate with xhigh. We describe the configurations and comparison results in Section 4.1 and Appendix C.2.
References
[1] Junlin Yang et al. Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering. arXiv preprint, arXiv:2607.28568, 30 July 2026.
Official Comment by Authors
Dear Reviewer hZHq,
Thank you again for your valuable feedback. We have carefully addressed your comments and provided additional analyses and clarifications in our revised manuscript.
We would greatly appreciate it if you could take another look when you have a chance. Please let us know if you have any remaining concerns.
Thank you for your time and consideration!
Response to Reviewer hZHq
Thank you for your careful review and helpful comments.
We have added an anonymized MatGym package to the supplementary materials, including the benchmark tasks, simulators, evaluation code, and reproduction instructions. We will publicly release the benchmark and open-source the code upon acceptance.
To address your concern about model coverage, we added two open-weight models, Qwen3.6-35B-A3B and Frontis-MA1-35B. We selected Qwen3.6 as a general-purpose comparison. Frontis was post-trained for machine-learning engineering using execution feedback and iterative revision [1]. We include Frontis as an engineering-oriented comparison, since materials experiments also require iterative decisions based on tool feedback.
To assess recent model progress, we also updated the proprietary systems to GPT-6.1 Sol with Codex and Claude Opus 5.5 with Claude Code. All four systems are evaluated on the same 52 tasks, with three attempts per task and the same laboratory budgets and acceptance criteria. Success rates (%) are shown below.
We find that Sol and Opus achieve higher success rates than both open-weight systems in each of the five laboratories. Qwen3.6 succeeds in all five, while Frontis succeeds only on alloy tasks. Examining Frontis’s trajectories reveals difficulties in completing experimental campaigns, including repeated rejected tool calls, budget exhaustion and delivered samples that miss several service requirements. For example, in one liquid-electrolyte run, the system repeatedly supplies formulation identifiers where the tool requires a salt name. These observations show that successful delivery requires reliable laboratory-tool use and campaign management alongside suitable material and process choices.
We report performance and acceptance-stage results in Table 3 and Figure 2, and discuss the observed failures in Sections 4.2–4.3 and Appendix D.