acceptodds
Under review as a conference paper at ICLR 2027

The Materials Scientist Agent: Reinforcement Learning for Crystal Discovery

Abstract

Discovering new inorganic crystals can be seen as a search problem, where the evaluation reveals if a candidate is stable, but not how to improve it. For this purpose, we develop an LLM-based agent and train a 4B model to do targeted materials search. We give the model computational tools, have it propose candidates in a Wyckoff representation, check their novelty, and evaluate their stability and properties over multiple turns. To stop the agent from resubmitting one good material, we score a group of attempts together and split credit between repeated structures. We evaluate on in-domain compositions from MP-20, out-of-domain from WBM, and band-gap targets, judging every structure with MLIPs not used in training. Reinforcement learning nearly doubles the rate of stable, unique, and novel (S.U.N.) structures over supervised fine-tuning, on known and unknown compositions, and the resulting 4B agent outperforms GLM-5.3 Flash and nearly matches Gemini 3.1 Pro, the teacher model. For band-gap targets, it finds up to more S.U.N. wide-gap materials than a state-of-the-art language model for crystal generation. We also show how the agent can be used to search for lead-free perovskites for solar cells.

Primary area:
reinforcement learning
Keywords:
LLM agents, reinforcement learning, tool use, materials discovery, crystal structure generation, AI for science

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_a9qrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a language model agent for inorganic crystal discovery that integrates materials retrieval, structure generation based on Wyckoff representations, elemental substitution, novelty checks, stability assessment, and band gap prediction into a multi-turn interactive environment. The authors first perform supervised fine-tuning of Qwen3.5-4B using approximately 1,100 trajectories generated by Gemini 3.1 Pro, then optimize its search policy through GRPO-style reinforcement learning. The reward accounts for validity, stability, novelty, property constraints, and duplicates both within each group and among previous submissions.

The experiments cover composition-conditioned generation on MP-20 and WBM, unconditional generation, band gap-conditioned generation, and lead-free perovskite search. Under a stability threshold that allows metastable structures, RL increases the S.U.N. rate on composition-conditioned tasks from approximately 11% to approximately 20%, approaching the teacher model’s performance. The authors also evaluate candidates using an MLIP that was not used during training and perform DFT calculations on selected candidates.

This work demonstrates the potential of training small models to carry out materials search workflows. However, the current evidence more strongly supports the conclusion that RL improves how the tool budget is used; it remains insufficient to establish that the agent has learned a more effective materials discovery strategy at the same computational cost. The numerical specification of the reward function, the comparability of the DFT validation, and the evaluation protocol for property-conditioned generation also require further clarification.

Critical strengths
  1. The authors formulate crystal generation as a sequential decision-making process in which the agent reads computational feedback, modifies candidates, and decides when to stop. The tool interfaces cover key operations in materials search, and the Wyckoff representation is well suited to language models’ handling of crystal symmetry. Training a 4B model to carry out this workflow has practical value.

  2. Table 4 shows that, under the threshold (Ehull<0.1E_{\mathrm{hull}}<0.1) eV/atom, the S.U.N. rate increases from 11.4% to 20.5% on ID tasks and from 11.5% to 20.1% on OOD tasks. These results support the conclusion that RL improves the agent’s overall behavior, with gains extending beyond the training compositions. However, this strength is primarily evident under an evaluation threshold that permits metastable structures.

Critical weaknesses
  1. Section 7 shows that RL increases the number of stability evaluations per episode from 2.8 to 6.8. Figure 5 further shows that, under the same evaluation budget, SFT with repeated sampling and filtering can match RL’s metastable S.U.N. rate and performs better at (Ehull≤0E_{\mathrm{hull}}\leq0). The “nearly twofold” improvement at the episode level therefore cannot be directly interpreted as an improvement in search efficiency. Integrating search into a single episode has engineering value. However, the paper does not report end-to-end latency, generated token counts, or computational cost per unique discovery, and it lacks comparisons with simple search controllers under a fixed budget. The stopping-behavior analysis in Figure 6 primarily provides correlational evidence and does not establish that the model has learned a more effective strategy for revising candidates.

  2. The paper introduces within-group credit sharing, a structural uniqueness reward, a penalty for duplicates of previous submissions, and a penalty for known stable compositions, without separately testing their necessity. In the unconditional task, in particular, 3,237 stable samples correspond to only 318 distinct structures, yielding a final S.U.N. rate of 3.0%, below PLaID++’s 7.7%. This result does not imply that the diversity rewards are entirely ineffective, but it leaves their effectiveness and scope unresolved. Moreover, using “chemical formula + space group” as the identifier for some duplicate penalties may incorrectly classify distinct polymorphs with the same composition and space group as duplicates. Within-group diversity, diversity across episodes, and actual structural diversity should be distinguished.

Questions for authors

Could you separately remove (rdisc)r_{\mathrm{disc}}), (rdivr_{\mathrm{div}}), the historical duplicate penalty, and the known-composition penalty, and report the cumulative number of unique discoveries in both conditional and unconditional tasks? How do you prevent the “chemical formula + space group” identifier from incorrectly penalizing distinct polymorphs?

3 comments

Thank you for your comments. In this rebuttal, we ran the two additional experiments, a fixed-budget comparison and a reward ablation, and answer each point below. Additionally, we retrained the agent on episodes from Gemini 4 Argon instead of Gemini 3.1 Pro, following the same procedure, and the results below use the retrained SFT and RL models. We use this model hereafter.


Budget vs. better search

We capped the RL agent's stability evaluations at 3, 4 and 6 on the out-of-domain task, and compared it with SFT sampled repeatedly until it has used the same number of evaluations. The table gives the distinct stable and novel structures found per prompt (E_hull < 0.1 under both Orb-v3 and eSEN).

RL capEvaluations per promptSFTRL
3192.192.55
4262.752.89
6383.643.62

With few evaluations RL finds more than SFT (+0.37 ± 0.17 at a cap of 3), and the two are within noise at 4 and 6. Candidate quality (13.9% of evaluated candidates are stable and novel) and submission accuracy (about 72%) are the same for both. The gain comes from a larger search that the agent has learned to direct: it keeps evaluating while its candidates are unstable and submits once it holds a stable one (Figure 4c in the revised paper). Uncapped, RL spends about 12% fewer tokens per discovery (259k against 293k).


Reward ablation

One RL run takes four to five days on our 8 A100 GPUs, so on the retrained agent we removed all our diversity additions in a single run instead of one at a time. Each episode is scored on its own, with no diversity term, shared credit, history, or repeat notice. The known-composition penalty is kept in both runs. The table gives the distinct S.U.N. structures found over 800 episodes per task under the consensus of Orb-v3 and eSEN (rate in parentheses).

TaskDistinct S.U.N.Plain RLOurs
In-domainE_hull ≤ 033 (4.1%)35 (4.4%)
E_hull < 0.1183 (22.9%)207 (25.9%)
Out-of-domainE_hull ≤ 059 (7.4%)57 (7.1%)
E_hull < 0.1155 (19.4%)200 (25.0%)
UnconditionalE_hull ≤ 064 (8.0%)160 (20.0%)
E_hull < 0.189 (11.1%)420 (52.5%)

Our reward finds a few more distinct structures when the composition is fixed, and 2.5 to 4.7 times as many on the unconditional task, where the plain agent keeps resubmitting the same ones. The unconditional rates here are over 800 samples and are not comparable with Table 1, where uniqueness is counted over 10,000. That comparison is running and will go in the paper.


Polymorphs.

Our reported uniqueness and novelty compare full structures with StructureMatcher. Formula + space group is only used during training, to penalize repeated structures. To see how often it wrongly merges polymorphs, we re-checked every pair it flags as a duplicate with StructureMatcher.

  • During training. 38% of pairs flagged within a group, and up to 21% of pairs flagged against earlier submissions, are actually different structures, mostly on prompts with a fixed composition (1.2% on unconditional prompts). StructureMatcher avoids this, but it is much slower inside the training loop.
  • In our reported results. In the paper's 10,000 unconditional samples, stable structures sharing a formula and space group are always the same structure (no exceptions across 150 such sets under eSEN and 120 under Orb-v3). The low uniqueness therefore reflects true repeats, not merged polymorphs.
  • Clarifications. Appendix A.1 gives the value, range and weight of every reward term (Table 4). The DFT validation and its scope are in Appendix C, and the band-gap protocol, which follows PLaID++, is in Section 6.3.

As the discussion period is coming to an end, we wanted to check if our response has addressed your concerns. In particular, we added a fixed-budget comparison between SFT and RL, a reward ablation, and a StructureMatcher check of the formula + space group duplicate identifier. Let us know if you would like any other clarification.

The responses addressed my main concerns and I have raised the scores accordingly.

Reviewer_LARuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents an LLM agent for crystal discovery that iteratively learns from and searches over candidate structures. The agent is first trained with supervised fine-tuning (SFT) on trajectories generated by Gemini 3.1 Pro, then further optimized with GRPO-style reinforcement learning to learn tool-calling strategies that yield novel, stable, and property-aligned candidates. Across the evaluated tasks, the authors show that RL improves the rate of discovering stable and unique candidates compared with both the base model and its SFT-trained counterpart. The paper thus demonstrates an effective application of tool use and reinforcement learning to crystal discovery.

Critical strengths
  • The paper is well written and easy to follow and addresses an important problem of crystal discovery.

  • The approach follows prior works to move beyond LLM-only crystal discovery by integrating it with different tools.

  • The reward seems to be well-thought of combining several objectives such as validity, stability, novelty, diversity, target properties, etc. This is well-thought and covers a large portion of metrics. Further, the use of teacher model to generate trajectories for the smaller model in open-ended discovery setting like this is interesting.

  • The experiments cover different aspects such as conditional, unconditional and application-specific generations.

Critical weaknesses
  • Missing Related Work and Baselines: The paper should provide a more rigorous discussion with recent LLM-based materials-discovery methods, particularly LLEMA, MatExpert, MatLLMSearch. This omission is particularly important because these methods directly challenge the paper’s motivation as this evolutionary framework tends to provide the signal to guide the search better. The authors should therefore explain more clearly what reinforcement learning provides beyond evolutionary memory and selection. Furthermore, MatExpert is also relevant because it decomposes materials discovery into expert-like stages involving retrieval, candidate proposal or modification, and subsequent evaluation, which is missing.

  • Multi-objective discovery: Works such as LLEMA/SparksMatter emphasizes that practical materials discovery is inherently multi-objective. The paper on the other hand, only discusses a single property-conditioned tasks such as band-gap target. The evaluation would therefore be stronger if it included tasks requiring simultaneous satisfaction of multiple competing objectives.

  • Reproducibility: The paper mentions the data curation process via Gemini, but lacks specific details. The lack of a public code and dataset release further limits reproducibility.

  • The paper needs to do a deeper dive into recent papers and do a thorough comparison. The related work section is not comprehensive enough. The additions such as reinforcement learning are good, but, this lack of comparison is crucial as it tends to over-claim the paper's strengths.


References

  • Ghafarollahi.et al, Autonomous Inorganic Materials Discovery via Multi-Agent Physics-Aware Scientific Reasoning (SparksMatter)

  • Ding, et. al, MatExpert: Decomposing Materials Discovery by Mimicking Human Experts

  • Gan, et al, MatLLMSearch: Crystal Structure Discovery with Evolution-Guided Large Language Models

  • Abhyankar, et.al, LLEMA: Evolutionary Search with LLMs for Multi-Objective Materials Discovery

Questions for authors
  • Did the authors evaluate robustness to noisy tool outputs or changes in the MLIP? How does performance change when the MLIP is replaced at test time or when responses are noisy?

  • A lot of the tools are not at the disposal of some the baselines. What happens when LLM-based baselines with the feedback from such tools are compared with the proposed method? What are the gains from the efficient calling of tools vs simply the availability?

  • How important is retrieval to the reported performance? Have the authors conducted tool-removal ablations, particularly by disabling retrieval? It would also be useful to evaluate how the agent performs when other tools are unavailable or fail.

  • Did the authors observe any cases of reward hacking?

  • The dataset-curation process is particularly interesting for an open-ended discovery problem. Could the authors provide more details on how the Gemini trajectories were selected and what principles should guide the creation of dedicated datasets for tuning LLM-based scientific agents?

5 comments

Thank you for the comments and for the pointers to recent work. We have added these works to the related work and a new appendix on how the SFT data are collected and filtered (Appendix A.2), and we answer your questions below.

AuthorsOpenReview ↗

Weaknesses

Related Work. We thank the reviewer for pointing us to these works and have added them to the related work. None of them trains the LLM to run the search, which is our core contribution:

  • MatLLMSearch: off-the-shelf LLMs in an evolutionary loop with MLIP stability feedback, no fine-tuning or RL.
  • LLEMA: off-the-shelf LLMs in an evolutionary loop, with surrogate property models and success/failure memories in the prompt.
  • MatExpert: fine-tuned LLMs for a retrieve–modify–generate pipeline, run in a single pass with no evaluation feedback and no RL.
  • SparksMatter: GPT-4-family agents orchestrating pretrained tools, without training; RL is listed as future work.

In contrast, we use RL to teach a 4B model the search itself (when to retrieve, evaluate, revise and submit), so that the strategy lives in the model's weights rather than in prompts or an external loop.

Multi-Objective. Our tasks target a single property (band gap) together with stability, novelty and composition constraints. The perovskite application task, with five conditions at once, is the closest to a multi-objective setting. The framework is not limited to one property, since each target is simply a reward term and a tool. Evaluating several competing property targets (e.g., band gap and bulk modulus, as in LLEMA) is a natural next step.

Reproducibility. The data curation is described in Section 4.4. The SFT data are collected by running the teacher (Gemini 3.1 Pro) in our own environment, with the same tools, on training prompts disjoint from the evaluation prompts. We keep only positive trajectories, those that end with an explicit submission, pass the validity and novelty filters, and have a successfully computed stability score (about 1,100 episodes). The full pipeline, including the data collection and filtering code, is provided in the supplementary material, and we will open-source the code, the curated SFT dataset, the trained models and the generated structures upon de-anonymization.

  1. Robustness to MLIPs and tools. This is a central part of our design. The agent uses one MLIP during its episodes (EquFlashV2, also the reward MLIP), but our main results are judged by two different MLIPs that the agent never sees during training or inference: Orb-v3 and eSEN. A structure counts as stable only if both agree. So these results already measure how well the agent's discoveries hold up under MLIPs other than its own. S.U.N. rates under the reward MLIP are close to those under the held-out consensus (Appendix B), and DFT confirms the held-out judgment: of the 24 structures both held-out MLIPs call stable, 23 lie within 0.1 eV/atom of the hull.

  2. LLM baselines with tools. Our LLM baselines, Gemini 3.1 Pro and GLM-5.3 Flash, run in our environment with exactly the same tools, budget and prompts as our agent. The comparison between Qwen3.5-4B base, SFT and RL also separates having the tools from using them well: all three have identical tools, yet in-domain S.U.N. (E_hull < 0.1 eV/atom) rises from 1.6% (base) to 11.4% (SFT) and 20.5% (RL) (Table 6 in the revised paper). The gain therefore comes from how the tools are used, not from their availability.

  3. Importance of retrieval. Retrieval is a deliberate part of the design, not a shortcut. It mirrors how materials scientists work: start from known compounds close to the target and modify them. It also cannot inflate our results by handing back known materials, since any structure that matches a known one fails the novelty check and does not count as S.U.N. And retrieval alone does not explain the performance: the base, SFT and RL models all have the same retrieval tool, yet S.U.N. goes from 1.6% to 11.4% to 20.5% (Table 6 in the revised paper). What makes the difference is how the agent uses what it retrieves.

  4. Reward hacking. We saw a few cases during development and changed the reward to stop them: drifting off the prompted composition gets zero reward, repeated submissions are penalized, and we have since added a check for relaxations that return an unphysical energy. To check the final model, we took the submissions the reward MLIP calls stable and counted how many both held-out MLIPs reject. This is about 3% in-domain and 2% out-of-domain, and it does not grow from SFT to RL.

  5. SFT Data curation. We added the full details in a new Appendix A.2. We run Gemini 3.1 Pro on 2,000 training prompts in the same environment and with the same tools as our agent, and keep only episodes where it submits a valid, novel structure with a stability score (and hits the target for property prompts). This leaves 1,183 episodes, which we balance across tasks to 1,123. The main principles are to collect the data in the same environment the student will use, and to keep only episodes with verified outcomes.

Reviewer_5vdjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an LLM-based agent for crystal discovery that uses computational tools to iteratively generate, evaluate, and refine crystal structures. Using Qwen3.5 4B as the base LLM, the authors perform supervised fine-tuning (SFT) on trajectories generated by the Gemini 3.1 Pro teacher model, followed by reinforcement learning (RL) with a reward combining multiple components. The method is evaluated on composition-conditioned, unconditional, and property-conditioned crystal generation tasks, also further applied to lead-free perovskite discovery. The results show improvements over the SFT baseline and competitive performance against larger teacher LLMs and specialized crystal generative models. The authors also validate selected discoveries using held-out MLIPs and DFT calculations.

Critical strengths
  1. Paper is well written. The experimental settings, training process, reward design, and implementation detail are described clearly. Evaluation includes multiple episode per prompt with boostrap standard errors. The analysis of tool usage and search behavior helps explain the effects of RL.

  2. Different crystal discovery tasks are comprehensively evaluated. Using two held-out MLIPs and further validating selected structures with DFT provide stronger evidence that the improvements are not simply due to exploiting the reward model during RL training.

Critical weaknesses
  1. Although the authors discuss that RL's improvement appears to be associated with a larger search budget, Figure 5 shows that SFT with best-of-k selection matches or even outperforms RL under the same evaluation budget. This leaves it unclear whether RL improves the proposals' quality or mainly learns to use more evaluations and make better submission decisions.

  2. The method's S.U.N. rate is only 3.0% in unconditional crystal generation due to a low uniqueness ratio. As the reward explicitly includes diversity and repetition penalties, it suggests that these mechanisms may be insufficient for unconstrained exploration.

Questions for authors
  1. The authors compare Base, SFT, and SFT+RL models in Figure 2. Have the authors considered an RL-only baseline with the pretrained model? This could further clarify the role of SFT in this training pipeline.

  2. Given the matched-budget results in Figure 5, could the authors further distinguish improvements in candidate quality from improvements in tool-use and submission behavior?

  3. Section 3 introduces the standard GRPO advantage normalized by the group standard deviation, but Section 4.4 defines the advantage using mean-centered rewards. Could the authors clarify which formulation was used during training? Besides, the method drops "tokens whose probabilities have changed too much," how is this threshold defined?

  4. Given the duplication in unconditional generation, have the authors considered ablating the diversity-related reward components, like the group-diversity bonus and historical repetition penalties? This may help explain why the proposed reward does not sufficiently encourage unique discoveries.

  5. Some minor presentation comments: (1) the titles of section 6.1 and 6.2 could be more descriptive of the corresponding experimental tasks. (2) In Figure 2, the Qwen3.5-4B grouping could be made clearer by labeling the three settings as Base, SFT, and SFT+RL under the same model, for instance, Qwen3.5-4B Base, Qwen3.5-4B SFT, Qwen3.5-4B SFT+RL. Also, the hatching of the out-of-domain bars could be made more visible.

2 comments

Thank you for reviewing the paper, we found your comments useful. During the rebuttal, we ran the reward ablation and the matched-budget analysis you asked for, clarified the GRPO formulation in Section 4.4, and made the presentation changes. For the revision, we retrained the agent on episodes from Gemini 4 Argon instead of Gemini 3.1 Pro, following the same procedure, and the results below use the retrained SFT and RL models.


RL-only baseline SFT followed by RL is common practice for training small models. Given the very low results of the base model, starting RL from the SFT model is the more sensible choice to prevent the cold-start problem.


Matched-budget comparison. We capped the RL agent's stability evaluations at 3, 4 and 6 on the out-of-domain task, and compared it with SFT sampled repeatedly until it has used the same number of evaluations. The table gives the distinct stable and novel structures found per prompt (E_hull < 0.1 under both Orb-v3 and eSEN).

RL capEvaluations per promptSFTRL
3192.192.55
4262.752.89
6383.643.62

With few evaluations RL finds more than SFT (+0.37 ± 0.17 at a cap of 3), and the two are within noise at 4 and 6. Candidate quality (13.9% of evaluated candidates are stable and novel) and submission accuracy (about 72%) are the same for both. The gain comes from a larger search that the agent has learned to direct. It keeps evaluating while its candidates are unstable and submits once it holds a stable one (Figure 4c in the revised paper). Uncapped, RL spends about 12% fewer tokens per discovery (259k against 293k).


GRPO Clarification. We believe there might be a misunderstanding with the way it is presented. Section 3 gives standard GRPO as background, and Section 4.4 describes our variant. We use the mean-centered advantage without standard-deviation normalization, as in Dr. GRPO [1]: Ai=Ri−RˉA_i = R_i - \bar{R}. For the loss, we use prime-rl's default, which follows DPPO [2]. Instead of clipping the ratio, it drops a token when its probability has already changed by more than 0.2 from the policy that generated the episode, in the direction of the advantage. A small KL penalty (weight 10−310^{-3}) toward that policy is also applied. This keeps updates stable during asynchronous training, where episodes may come from a slightly older policy. We now state these choices and values in Section 4.4 and in Appendix A.3 of the revised paper.


Reward ablation On the retrained agent, we repeated the RL stage with each episode scored on its own, without the diversity term, the shared credit, the history, or the tool's notice that a candidate repeats an earlier one. Everything else is the same. Numbers are over 800 episodes per task under the consensus of Orb-v3 and eSEN.

TaskMetric (%)Plain RLOurs
In-domainSubmission78.894.1
S.U.N. (E_hull ≤ 0)4.14.4
S.U.N. (E_hull < 0.1)22.925.9
Out-of-domainSubmission78.595.6
S.U.N. (E_hull ≤ 0)7.47.1
S.U.N. (E_hull < 0.1)19.425.0
UnconditionalSubmission78.394.8
S.U.N. (E_hull ≤ 0)8.020.0
S.U.N. (E_hull < 0.1)11.152.5

The reward matters most when the composition is free: without it, the agent keeps resubmitting the same stable structures on the unconditional task. The unconditional rates here are over 800 samples and are not comparable with Table 1, where uniqueness is counted over 10,000. We are extending this comparison to that protocol and will add it to the paper.


Presentation Comments: Thank you for your suggestions, we've included them

  1. Title changes: Section 6.1 "In-Domain/Out-of-Domain" → "Composition-Conditioned Generation: In-Domain and Out-of-Domain", Section 6.2 "Unconstrained" → "Unconditional Generation".
  2. Figures updated.

[1] Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., ... & Lin, M. (2025). Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.
[2] Qi, P., Zhou, X., Liu, Z., Pang, T., Du, C., Lin, M., & Lee, W. S. (2026). Rethinking the trust region in llm reinforcement learning. arXiv preprint arXiv:2602.04879.

Thank you again for your review. The discussion period ends soon, we’d love to hear whether our response addresses your concerns. We’re happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.