acceptodds
Under review as a conference paper at ICLR 2027

MatGym: An Agent-Centric Environment Bridging Materials Discovery and Reproducible Synthesis

Abstract

Materials discovery requires exploring how composition and processing affect performance when these relationships are only partly known. We introduce MatGym, an agent-centric environment for evaluating autonomous exploration and reproducible delivery across five simulated materials laboratories. Given a material specification and a resource budget, agents conduct open-ended exploration: they select candidates, control preparation steps, and choose computations and measurements to inform subsequent actions. Each task ends with a sample and a recipe, evaluated for service performance, agreement with the sample's preparation history, and replication on fresh batches. Across 52 tasks using the largest implemented candidate libraries, with three attempts per task, Codex and Claude Code achieve success rates of 43.6% and 41.0%, respectively. Similar overall rates mask different strengths across laboratories, and each system solves fewer than 30% of tasks successfully in all three attempts. Separate delivery checks distinguish material performance, process traceability, and batch reproducibility. In auxiliary transfer tests with an earlier Codex configuration, success on new ALD tasks increases from 30.6% without memory to 55.6% with prior campaign memory, while gains across material systems vary. MatGym provides a controlled setting for evaluating how agents explore unfamiliar material processes, deliver reproducible outcomes, and reuse experience on new tasks.

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM agents, materials discovery, autonomous experimentation, reproducible synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hZHqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a dataset to test whether an agent can deliver reproducible materials. It must ensure that the delivered sample recipe meets the standard, matches the actual production records, and can be reproduced in new batches. The authors tested Claude and Codex and provided related analyses.

Critical strengths
  1. The authors propose a dataset focused on critical scientific domains, examining whether AI can deliver reproducible materials.

  2. This benchmark features an insightful design that combines materials discovery with reproducible synthesis, ensuring the delivery of a reliable process.

  3. The authors evaluated strong coding agents and conducted experimental analysis.

Critical weaknesses
  1. This is a dataset paper, but the authors didn't include the dataset in the supplementary materials.

  2. The paper only tested two models, which is too limited. The authors didn't test any open-source models to see the gap between open-source and closed-source models.

  3. The analysis of the tested models isn't insightful enough. I would expect a deep expert analysis or case study to point out how current models can be improved further.

Questions for authors

What is the reasoning effort that you were using during testing the performance of Claude Code and Codex?

5 comments
AuthorsOpenReview ↗

Response to Reviewer hZHq

Thank you for your careful review and helpful comments.

W1. The benchmark dataset was not included in the supplementary materials.

We have added an anonymized MatGym package to the supplementary materials, including the benchmark tasks, simulators, evaluation code, and reproduction instructions. We will publicly release the benchmark and open-source the code upon acceptance.

W2. Evaluation covers only two models and lacks open-weight comparisons.

To address your concern about model coverage, we added two open-weight models, Qwen3.6-35B-A3B and Frontis-MA1-35B. We selected Qwen3.6 as a general-purpose comparison. Frontis was post-trained for machine-learning engineering using execution feedback and iterative revision [1]. We include Frontis as an engineering-oriented comparison, since materials experiments also require iterative decisions based on tool feedback.

To assess recent model progress, we also updated the proprietary systems to GPT-6.1 Sol with Codex and Claude Opus 5.5 with Claude Code. All four systems are evaluated on the same 52 tasks, with three attempts per task and the same laboratory budgets and acceptance criteria. Success rates (%) are shown below.

Agent systemAlloyGarnetLiquidPolymerALDOverall
Frontis-MA1-35B20.00.00.00.00.03.8
Qwen3.6-35B-A3B30.03.330.63.310.016.0
Claude Opus 5.560.023.388.946.766.758.3
GPT-6.1 Sol80.013.391.750.070.062.2

We find that Sol and Opus achieve higher success rates than both open-weight systems in each of the five laboratories. Qwen3.6 succeeds in all five, while Frontis succeeds only on alloy tasks. Examining Frontis’s trajectories reveals difficulties in completing experimental campaigns, including repeated rejected tool calls, budget exhaustion and delivered samples that miss several service requirements. For example, in one liquid-electrolyte run, the system repeatedly supplies formulation identifiers where the tool requires a salt name. These observations show that successful delivery requires reliable laboratory-tool use and campaign management alongside suitable material and process choices.

We report performance and acceptance-stage results in Table 3 and Figure 2, and discuss the observed failures in Sections 4.2–4.3 and Appendix D.

AuthorsOpenReview ↗

Response to Reviewer hZHq

W3. Deeper expert analysis and case studies are needed to explain model failures and identify areas for improvement.

Following your suggestion for a deeper analysis of failure causes, we examined all 312 Sol and Opus campaigns and replayed the 124 unsuccessful ones. We identified difficulties in applying mechanisms under the relevant conditions, satisfying multiple requirements together, and retaining promising preparations through further processing and final selection. The specific decisions and their outcomes are summarized below.

Problem identifiedEvidence from trajectories and replay
Mechanism interpretation and joint constraintsIn a polymer case, the agent invoked crystallinity at 60°C, where that mechanism was inactive. Gentler drying increased conductivity but violated stiffness and residual-solvent requirements.
Material and process choices remain incompatibleIn 52/124 failures, replacing both material and process with the reference passed, while neither replacement alone did. In the ALD case, repeated tuning of one chemistry left the leakage requirement unresolved.
Further processing or final selection loses a qualifying preparationReplay identifies 18/124 failures with a lost qualifying state or an undelivered qualifying alternative. In the alloy case, additional aging reduced hardness below the target.
Observed success does not persist across fresh batchesIn 14/124 failures, only fresh-batch acceptance failed. The alloy case had two observed samples above the targets, but the submitted recipe passed only one of three fresh batches.

These counts come from replay diagnostics of the 124 failed campaigns; categories may overlap. Reference replacements and qualifying states are evaluated in replay.

These cases clarify the decisions that agents struggle with: which material or process conditions to change, when to stop processing, and which sample–recipe pair to deliver. In the polymer and ALD trajectories examined, continued tuning did not satisfy all constraints. The alloy case shows how successful observations on existing samples can still be followed by unreliable fresh-batch delivery. We summarize these findings in Section 4.3 and provide the replay procedures and corresponding trajectories in Appendix D.

Q1. Which reasoning-effort settings were used for Claude Code and Codex?

We explicitly set the reasoning effort to high for both Codex and Claude Code. To assess recent model progress, we updated their underlying models to GPT-6.1 Sol and Claude Opus 5.5, and added evaluations of Qwen3.6-35B-A3B and Frontis-MA1-35B using their default thinking settings. The configurations are summarized below.

ModelReasoning configuration
GPT-6.1 SolHigh, explicitly set
Claude Opus 5.5High, explicitly set
Qwen3.6-35B-A3BThinking enabled, model default
Frontis-MA1-35BThinking enabled, model default

We also examined whether increasing reasoning effort improves task completion by comparing high and xhigh on ten prespecified tasks, with three attempts per task. Sol succeeds in 17/30 attempts with high and 14/30 with xhigh, while Opus succeeds in 17/30 and 16/30, respectively. On these tasks, neither model achieves a higher success rate with xhigh. We describe the configurations and comparison results in Section 4.1 and Appendix C.2.

References

[1] Junlin Yang et al. Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering. arXiv preprint, arXiv:2607.28568, 30 July 2026.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer hZHq,

Thank you again for your valuable feedback. We have carefully addressed your comments and provided additional analyses and clarifications in our revised manuscript.

We would greatly appreciate it if you could take another look when you have a chance. Please let us know if you have any remaining concerns.

Thank you for your time and consideration!

Reviewer_MUu4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces MatGym, an agent-centric benchmark that connects materials discovery with reproducible synthesis across five simulated laboratories (ALD dielectrics, Al-Mg-Si alloys, garnet, liquid, and polymer electrolytes). Tasks are open-ended: agents select candidates, plan and execute synthesis and measurements under budgets, and must deliver both a working sample and a matching synthesis recipe that reproduces on fresh batches. The authors provide a rigorous acceptance protocol that simultaneously checks service performance, recipe–history consistency, and 2-out-of-3 fresh-batch replication, and evaluate two coding-agent systems and several scripted baselines, with additional analyses on tool access, feedback relevance, repeated-attempt consistency, library scaling, and cross-task memory transfer.

Critical strengths

Introduces a unified acceptance protocol that jointly assesses material performance, process traceability (recipe–history matching), and fresh-batch replication capabilities largely missing from prior benchmarks. Models persistent samples, coupled properties, destructive service tests, and batch variability, forcing agents to plan realistically under uncertainty and costs. Spans five distinct laboratories with different physics and process steps, including mechanisms such as precipitation and coarsening, densification, and ALD exposure dynamics. Careful task generation with hidden process parameters, privileged reference searches to set achievable targets, and admission gates ensuring feasibility while defeating a nominal open-loop recipe. Quantitative breakdowns of failure modes (sample targets, recipe matching, replication) and clear accounting of resource budgets and delivery validity. Auxiliary controls show the importance of tool access and the value of task-relevant feedback. Memory-transfer experiments probe reuse of experience on new tasks. Clear formalization of the acceptance protocol and task interface. Figures and tables convey the workflow, results, and ablations effectively. Separation of concerns (sample success vs traceability vs replication) helps interpret where agents fail. Addresses a key gap between computational discovery and deployable synthesis by making reproducibility and procedural fidelity first-class evaluation targets.

Critical weaknesses

Simulators necessarily simplify kinetics and processing. While validation checks are described, the extent to which simulator idiosyncrasies influence agent strategies remains a concern. Recipe-history strict matching may penalize practically equivalent, reordered, or merged/split step variants. Although some normalization is allowed, the rigidity may underrepresent realistic protocol flexibility. Algorithmic baselines are limited: BO and random search operate on a fixed “public recipe pool,” which handicaps them relative to agents that can propose new parameterizations. Stronger baselines using the same open action space (e.g., BO/RL over continuous step parameters) would better contextualize agent performance. Only two proprietary LLM-based agent systems are evaluated, and three repeats per task may be thin for robust statistics given stochasticity and batch variability. Memory-transfer experiments use an earlier configuration and a single source-target pair with the most pronounced gains in ALD. Broader, systematic transfer studies and negative controls would strengthen claims. Some details on the magnitude and distribution of batch variability and measurement noise are deferred to appendices. A concise main-text summary would help assess difficulty. Quantitative characterization of simulator calibration vs. real data is described at a high level. Explicit main-text examples of fit quality and residuals per lab would improve trust. While related work is broad and well-cited, comparisons to strong algorithmic planners that interact with the same tool interface (beyond scripted heuristics and pool-based BO) are missing. Discussion of how MatGym complements recent agentic allocation or planning layers in MOBO/self-driving labs could be expanded with more direct contrasts.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer MUu4

Thank you for your careful review and helpful comments.

W1. Simulator-specific assumptions may influence agent strategies.

We ground the environments in measured data and physical mechanisms, and supplement this with a measured-feedback comparison to examine whether agents can complete tasks beyond the original simulated feedback.

Our liquid- and polymer-electrolyte conductivity models are fitted to 8,549 and 622 public experimental records, respectively, with residuals reported in Table 5. For the three liquid-electrolyte salts, the median absolute relative residual on the fitting records ranges from 4.8% to 5.8%. On 125 external Clio measurements, the unchanged LiPF₆ model has a median absolute relative error of 9.0% [5]. The other laboratories combine database inputs with literature-informed process mechanisms to describe how processing changes sample states and properties.

To address your concern about dependence on simulator-specific feedback, we report a supplementary liquid-electrolyte comparison using the original simulator and archived measured conductivity [6]. Sol ran three attempts at each of three temperatures within one task world. Both conditions used the same conductivity targets, budgets and sample–recipe delivery workflow.

Conductivity feedbackAccepted attempts
Original simulator9/9
Archived measured conductivity9/9

Sol completed all nine attempts in both environments. It also completed formulation selection, sample preparation and recipe delivery when conductivity observations and acceptance used archived measurements. The replay uses fixed conductivity medians for 103 recorded formulations without the original conductivity noise, batch corrections or freezing criterion; other processes remain simulated.

We designed MatGym to assess how agents investigate initially unknown material and process responses. In physical experiments, these specific relationships often need to be established through experimentation. Our tasks likewise require agents to combine prior knowledge with new observations, choose subsequent experiments, revise preparation conditions and find solutions satisfying multiple requirements. We use controlled tasks to evaluate this aspect of exploration, then examine campaign trajectories to identify bottlenecks in interpreting feedback, handling constraints and completing delivery.

We have also provided a common laboratory interface for incorporating additional experimental data, separating agent actions from the underlying process and measurement models. We plan to expand the benchmark through this interface with further experimental datasets and physical models, covering a wider range of material responses and experimental conditions.

W2. Strict history matching may penalize equivalent procedures, including reordered, merged or split steps.

To check whether history matching rejects equivalent procedures, we examined runs with history mismatches and replayed the submitted recipes alongside the procedures actually executed. We identified two Qwen3.6 runs in which the complete and shortened procedures produced nearly identical properties and the same fresh-batch pass/fail outcomes. Removing only the history-matching check raises Qwen3.6 acceptance from 25/156 to 27/156 (+1.28 percentage points); Sol and Opus acceptance is unchanged in this audit.

We use history matching to assess whether the agent accurately documents how the delivered sample was prepared. For example, an agent may process the same sample several times but include only the final treatment in its submitted recipe, omitting earlier operations that affected the sample’s state. Even if the shortened recipe also passes fresh-batch tests, it does not fully document the preparation of the delivered sample. We therefore check both consistency with the sample’s preparation history and performance across fresh batches, assessing whether agents can deliver qualifying samples together with traceable preparation recipes. Section 3.4 explains these requirements, and Appendix A.3 reports the equivalent-operation examples and their effect on acceptance.

AuthorsOpenReview ↗

Response to Reviewer MUu4

W3. Fixed recipe pools disadvantage search baselines; stronger comparisons should allow the same open action space as the agents.

To address the recipe-pool restriction you identified, we added open-parameter random search and GP-BO, allowing both to propose continuous process parameters and categorical choices beyond the fixed pool. Both use public laboratory tools, predefined workflow variants and the same budgets as the agents. We also compare GP-BO with and without computational screening to examine whether public-tool predictions help the search. All six conditions use the same 52 tasks and three matched repeats.

Search methodFixed recipe poolOpen parameters
Random search2/156 (1.3%)1/156 (0.6%)
GP-BO3/156 (1.9%)1/156 (0.6%)
GP-BO + screening24/156 (15.4%)19/156 (12.2%)

Screened open-parameter BO tests recipes outside the pool in 600 of 912 trials and achieves a task success rate of 12.2%. We find that computational screening improves BO in both search spaces, while opening the parameters alone does not outperform the corresponding pool-based methods. On the same tasks, Sol and Opus achieve success rates of 62.2% and 58.3%, respectively. Thus, for these baselines, the performance gap with the agents remains when the search can propose recipes beyond the pool.

We report all six conditions in Table 3 and describe the search variables, workflow variants and screening rules in Appendix C.1.

W4. Evaluation covers only two proprietary systems, and three attempts per task may be insufficient given stochasticity and batch variation.

W4.1. Open-weight model comparison

To address your concern about model coverage, we added two open-weight models, Qwen3.6-35B-A3B and Frontis-MA1-35B. We selected Qwen3.6 as a general-purpose comparison. Frontis was post-trained for machine-learning engineering using execution feedback and iterative revision [7]. We include Frontis as an engineering-oriented comparison, since materials experiments also require iterative decisions based on tool feedback.

To assess recent model progress, we also updated the proprietary systems to GPT-6.1 Sol with Codex and Claude Opus 5.5 with Claude Code. All four systems are evaluated on the same 52 tasks, with three attempts per task and the same laboratory budgets and acceptance criteria. Success rates (%) are shown below.

Agent systemAlloyGarnetLiquidPolymerALDOverall
Frontis-MA1-35B20.00.00.00.00.03.8
Qwen3.6-35B-A3B30.03.330.63.310.016.0
Claude Opus 5.560.023.388.946.766.758.3
GPT-6.1 Sol80.013.391.750.070.062.2

We find that Sol and Opus achieve higher success rates than both open-weight systems in each of the five laboratories. Qwen3.6 succeeds in all five, while Frontis succeeds only on alloy tasks. Examining Frontis’s trajectories reveals difficulties in completing experimental campaigns, including repeated rejected tool calls, budget exhaustion and delivered samples that miss several service requirements. For example, in one liquid-electrolyte run, the system repeatedly supplies formulation identifiers where the tool requires a salt name. These observations show that successful delivery requires reliable laboratory-tool use and campaign management alongside suitable material and process choices.

We report performance and acceptance-stage results in Table 3 and Figure 2, and discuss the observed failures in Sections 4.2–4.3 and Appendix D.

AuthorsOpenReview ↗

Response to Reviewer MUu4

W4.2. Repeated evaluation and run-to-run variation

We examined the results of individual repeats and whether our main comparisons depend on a particularly favorable repeat.

Our evaluation comprises three complete repeats, each covering all 52 tasks with a different laboratory random seed, for a total of 156 campaigns per system. Table 3 reports the mean and sample standard deviation of the three benchmark-level success rates. ScienceAgentBench [4] also evaluates agents with three attempts per task. The results for the two primary agents are:

SystemRepeat 1Repeat 2Repeat 3Mean ± SD
Codex / GPT-6.1 Sol (high)67.3%63.5%55.8%62.2% ± 5.9 pp
Claude Code / Opus 5.5 (high)65.4%55.8%53.8%58.3% ± 6.2 pp

Although the overall success rates vary across repeats, Sol achieves a higher success rate than Opus in all three. To further examine the screening and memory results, we omitted each repeat in turn and recomputed the comparisons using the remaining two. In every two-repeat summary, screening improves BO in both search spaces, and same-domain memory outperforms both no memory and generic guidance.

Thus, across the available repeats, the ordering of the two agents remains consistent, and the gains from screening and same-domain memory do not depend on any single repeat. Section 4.1 describes the aggregation procedure, and Appendix C.4 presents this sensitivity analysis.

W5. Transfer evidence uses an earlier configuration and one source–target pair, with gains concentrated in ALD; broader studies and negative controls are needed.

To address your questions about transfer scope and controls, we expanded the memory study to all five laboratories and added a generic-guidance control, testing whether the gains extend beyond ALD and whether recorded experimental experience provides an additional benefit. Across the 52 targets, acceptance is higher with same-domain cases than with either no memory or generic guidance. Relative to no memory, the gains appear in alloy, polymer and ALD.

The study covers 52 targets across five laboratories, with five same-domain transfers to independent task worlds and five cross-domain directions. All conditions use GPT-6.1 Sol (high), matched laboratory seeds and budgets, and three attempts per target. N denotes no memory; G denotes generic guidance; S and X add six same-domain or cross-domain cases to that guidance. We extracted the cases from 90 independent source campaigns and fixed them before target evaluation. N reuses matched runs under the same configuration; G/S/X were interleaved.

LaboratoryNo memory NGeneric guidance GSame-domain SCross-domain X
Alloy24/30 (80.0%)26/30 (86.7%)29/30 (96.7%)26/30 (86.7%)
Garnet4/30 (13.3%)2/30 (6.7%)2/30 (6.7%)3/30 (10.0%)
Liquid33/36 (91.7%)32/36 (88.9%)32/36 (88.9%)34/36 (94.4%)
Polymer15/30 (50.0%)20/30 (66.7%)17/30 (56.7%)18/30 (60.0%)
ALD21/30 (70.0%)18/30 (60.0%)25/30 (83.3%)21/30 (70.0%)
Overall97/156 (62.2%)98/156 (62.8%)105/156 (67.3%)102/156 (65.4%)

Overall acceptance with same-domain cases is 4.5 percentage points higher than with generic guidance. We also observe higher overall acceptance with cross-domain cases, while the ordering of S and X varies across laboratories and source banks. The study records gains beyond ALD, while the changing S–X ordering shows that domain labels alone do not identify the more useful source cases. We report the main results in Section 4.3 and Figure 2, with laboratory-level results and the protocol in Table 9 and Appendices C.3–C.4.

W6. The main text should summarize the distributions and magnitudes of batch variation and measurement noise.

We have added the distributions and representative scales of batch variation and measurement noise to Section 3.2.

We model batch variation during preparation separately from measurement noise during observation: the former causes the same recipe to produce different outcomes across batches, while the latter causes the measurements available to the agent to differ from the sample’s underlying properties. Temperature offsets are Gaussian, with standard deviations of 1.5–6 K; dose, water-content and inhibitor multipliers are lognormal. Representative measurement-noise standard deviations are 2.5 HV for alloy hardness, 0.1 nm for ALD thickness and 2–6% for relative errors in total electrolyte conductivity.

These settings require agents to select and refine recipes using noisy feedback and test whether those recipes continue to meet the task requirements in fresh batches.

Reviewer_jVXsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MatGym, an agent-centric benchmark designed to evaluate how autonomous AI agents explore partially known materials processes and deliver reproducible synthesis recipes under strict resource constraints. Operating across five simulated materials laboratories—atomic layer deposition (ALD) dielectrics, an Al–Mg–Si age-hardening alloy, garnet solid electrolytes, liquid electrolytes, and polymer electrolytes—MatGym presents open-ended tasks where agents select candidates, execute preparation steps, and perform computations and measurements within explicit campaign budgets. To connect agent decision-making with practical reproducibility, the framework enforces a three-part acceptance protocol requiring that (1) the delivered sample meets specified service targets under realistic usage conditions, (2) the submitted recipe strictly matches the sample's recorded preparation history, and (3) the recipe successfully reproduces performance on at least two out of three fresh batches. Evaluating LLM agent systems (Codex and Claude Code) alongside six non-privileged scripted baselines across 52 primary maximum-library tasks demonstrates that coding agents substantially outperform rule-based baselines (43.6% and 41.0% overall success rates vs. a maximum of 5.1% for scripted baselines). However, aggregate success rates mask distinct laboratory-specific strengths, and both agent systems struggle with consistency across attempts, solving fewer than 30% of tasks successfully in all three evaluation runs. Diagnostic controls further highlight the necessity of active tool access and task-relevant feedback, while memory-transfer experiments show that providing prior campaign experience can significantly boost success on new task domains.

Critical strengths

S1. Interesting Problem Formulation MatGym evaluates agents across five simulated materials laboratories under explicit campaign budgets, bridging candidate search with synthesis traceability. Its key contribution is a three-part acceptance protocol that enforces service performance targets, recipe-history alignment, and multi-batch reproducibility—directly addressing a major gap in evaluating AI agents for real-world scientific workflows.

S2. Methodological Rigor and Diagnostic Depth The benchmark goes beyond aggregate performance by evaluating coding agents against six scripted baselines across 52 tasks and running thorough diagnostic controls. Analyzing multi-attempt consistency (pass@3 vs. pass3), delivery failure modes, tool access ablations, and cross-domain memory transfer provides actionable structural insights into agent behavior rather than simple scalar accuracy scores.

Critical weaknesses
  • W1. Reliance on Simplified Synthetic Simulators Although MatGym incorporates literature-informed physics (such as CALPHAD equilibrium tables, pymatgen stability, and kinetic precipitation equations), the five simulated laboratories rely heavily on simplified kinetic approximations and pre-computed offline database lookups (e.g., pre-calculated OBELiX conductivity records) to maintain computational tractability. While well-suited for high-throughput digital evaluation, these simplified models omit key physical realities present in real-world automated labs—such as multi-scale microstructural defects, spatial temperature gradients, and unpredictable mechanical/hardware failures—leaving an unquantified gap between benchmark success and physical laboratory deployment.

  • W2. Simulation-Based Evaluator: For the evaluator in Section 3.4, it remains uncertain how faithfully this simulated evaluator reflects true real-world synthesis success and physical reproducibility.

Questions for authors

Please see Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jVXs

Thank you for your careful review and helpful comments.

W1. Simplified simulators omit physical factors such as microstructural defects, spatial temperature gradients and hardware failures, leaving the gap to physical deployment unquantified.

Our liquid- and polymer-electrolyte conductivity models are fitted to public experimental data, with errors on the fitting records reported in Table 5. For the three liquid-electrolyte salts, the median absolute relative residual ranges from 4.8% to 5.8%. We also report an additional external comparison: the unchanged LiPF₆ model has a median absolute relative error of 9.0% and an RMSE of 1.58 mS/cm on 125 Clio measurements [1]. These results quantify agreement between modeled conductivity and experimental records. Processing is examined separately through mechanism checks such as overaging, lithium loss and ALD saturation.

For the three physical factors you identify, we clarify their representation below:

Physical factorCurrent representationNot currently included
Microstructure and defectsGarnet grain size, porosity, lithium vacancies, phase contributions and grain-boundary resistance; alloy precipitate size, fraction and coarseningExplicit spatial defect distributions and multiscale morphology
Temperature variationBatch furnace offsets and temperature-dependent process kineticsSpatial temperature fields and heat-transfer gradients
Mechanical and hardware failuresNo component-failure modelPump, valve and other equipment failures, and recovery actions

Offline tables supply inputs to the process models, while processing operations continue to change sample states. For example, alloy aging updates precipitation and consequently hardness and conductivity; garnet sintering changes density, lithium retention and grain size. Section 3.2 now describes these state changes and the physical effects represented in each laboratory.

We also supplemented the evaluation with a test that incorporates archived experimental data through the existing laboratory interface. In supplementary liquid-electrolyte tasks, Sol used either the original simulator or measured-conductivity replay [2], with three attempts at each of three temperatures within one task world. Both conditions achieved 9/9 accepted attempts. The replay restricts choices to 103 recorded formulations and uses fixed conductivity medians without the original conductivity noise, batch corrections or freezing criterion; other processes remain simulated. Under these conditions, the agent completed formulation selection, sample preparation and recipe delivery using archived measured conductivity.

We built MatGym to evaluate how agents investigate initially unknown composition–process responses. As in physical experiments where specific material responses must be established progressively, agents need to combine prior knowledge with observations, choose subsequent experiments, revise preparations and find solutions satisfying multiple requirements. The current tasks assess these material-exploration and delivery capabilities; spatial heat transfer and equipment-failure recovery are further experimental situations to incorporate. We have separated the action interface from the process and measurement models, and plan to introduce additional laboratory measurements and more detailed physical models to progressively expand the benchmark’s coverage.

AuthorsOpenReview ↗

Response to Reviewer jVXs

W2. It is unclear whether simulated acceptance reflects successful synthesis and reproducibility in physical experiments.

Our current evaluator checks whether the delivered sample meets the requirements, whether the submitted recipe records its actual preparation, and whether that recipe passes at least two of three fresh simulated batches. We have not yet physically synthesized the full recipes submitted by agents; the reported success rate refers to these simulated delivery requirements.

To examine whether successful simulated compositions also have measured-property evidence, we additionally checked the deliveries from a supplementary liquid-electrolyte study. Using the original simulator, Sol completed nine deliveries and selected three distinct formulations. The corresponding historical conductivity medians also met the diagnostic targets [2]. One delivery at −20°C is shown below.

ItemDelivered composition and measured comparison
CompositionLiPF₆ at approximately 0.591 mol/kg solvent; EC/PC/EMC mass fractions of 0.3002/0.2003/0.4995; no additive
Conductivity target2.18 mS/cm
Simulated delivered conductivity2.479 mS/cm
Historical measured conductivityMedian 2.526 mS/cm; range 2.461–2.568 mS/cm across four readings

After formulation normalization, the delivered composition matches the experimental record, and both simulated and measured conductivity meet the target. This supplementary diagnostic concerns composition and conductivity and is separate from the 52-task evaluation.

We built MatGym to assess whether agents can investigate unknown material and process responses through experimentation, find solutions satisfying multiple requirements, and turn exploration into traceable deliveries with executable recipes. To understand where these capabilities break down, we examined campaign trajectories and simulator replays for the 124 unsuccessful Sol and Opus campaigns in the primary evaluation. The delivery-related findings include:

Delivery-related findingFailed campaigns
Earlier qualifying state lost or qualifying alternative left undelivered18/124
All other acceptance checks passed; only fresh-batch replication failed14/124

For example, in one alloy campaign, further aging reduced hardness below the target, and another qualifying sample was not selected for delivery. For the different recipe ultimately submitted, the agent prepared two samples whose measurements met the targets and considered batch uncertainty, yet only one of three fresh batches passed. The difficulties in this case were retaining and selecting a suitable preparation, and assessing batch reliability from limited observations. We therefore include the final sample, its actual preparation history and fresh-batch performance in acceptance.

In future work, we plan to physically synthesize representative agent-delivered recipes and measure their properties across independent batches to examine the correspondence between simulated acceptance and physical reproducibility.

References

[1] Dave et al. Autonomous optimization of non-aqueous Li-ion battery electrolytes via robotic experimentation and machine learning coupling. Nat. Commun., 2022.

[2] Rahmanian et al. Conductivity experiments for electrolyte formulations and their automated analysis. Sci. Data, 2023.

Dear Reviewer jVXs,

Thank you again for your valuable feedback. We have carefully addressed your comments and provided additional analyses and clarifications in our revised manuscript.

We would greatly appreciate it if you could take another look when you have a chance. Please let us know if you have any remaining concerns.

Thank you for your time and consideration!

Comments on the paper

2 comments
AuthorsOpenReview ↗

We have updated the revised PDF and responded to every reviewer

Dear Reviewers,

We deeply appreciate the time and effort you have dedicated to providing detailed and insightful feedback on our work. Your comments have been invaluable in improving the quality of our paper. Detailed responses to each point are provided under the corresponding reviewer’s comments.

Dear reviewers,

Thank you very much for your time and effort in reviewing this submission. The authors have now submitted their rebuttal, and the discussion period is nearing its end. We would greatly appreciate it if you could take some time to carefully review the authors’ responses and share your assessment of whether your concerns have been adequately addressed. Thank you again for your valuable time and contributions to the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.