acceptodds
Under review as a conference paper at ICLR 2027

CrysTune: Crystal Generation via Fine-Tuning of Large Language Models on Wyckoff Representations

18 views

Abstract

The discovery of novel materials is essential for driving scientific and technological breakthroughs. Recent work has explored fine-tuning large language models (LLMs) for autoregressive crystal generation, but the ideal representation and training strategies for symmetry-based inductive biases remain unclear. We propose CrysTune, a class of LLMs fine-tuned on Wyckoff representations of crystals with two auxiliary tasks: canonicalization and template prediction. CrysTune shows competitive performance and improved stability-related metrics relative to LLMs trained on standard string-encoded representations. We further use these models as initial policies for reinforcement learning (RL) fine-tuning to optimize stability, validity, uniqueness, novelty, and diversity. RL-trained policies produce more valid and metastable crystals, while introducing novelty and diversity trade-offs. We also explore crystal system conditioning, showing that RL-trained policies produce a higher proportion of crystals matching the target condition.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
materials discovery, generative modeling, large language models, reinforcement learning, Wyckoff positions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CMZyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CrysTune, a framework for autoregressive inorganic crystal generation by fine-tuning the general-purpose large language model Qwen‑3.5‑9B on canonical Wyckoff string representations. Training incorporates two auxiliary tasks, Wyckoff lookup and canonicalization of symmetry-equivalent representations, followed by GRPO fine-tuning with rewards for stability, novelty, uniqueness, and diversity.

Experiments show that, compared with a CIF-style representation using the same base model, the Wyckoff representation improves validity and several stability-related metrics. The auxiliary tasks improve Wyckoff template recall accuracy but do not yield a clear improvement in generation quality. RL improves validity and the number of MetaSUN candidates, at the cost of some novelty and space-group coverage. The paper also analyzes the effects of mode collapse, decoding settings, and evaluation protocols on the results.

Critical strengths

1.The experiments reveal limitations in pretrained LLMs’ recall of Wyckoff templates. The paper evaluates Qwen‑3.5 models of several sizes and finds that their ability to accurately produce Wyckoff coordinate templates in the specified standard setting is limited. This provides empirical motivation for explicitly incorporating crystallographic knowledge and offers useful insights for LLM-based materials generation.

2.The research questions are clear, and controlled comparisons of representations support the effectiveness of the approach. CrysTxtLLM and CrysTune use the same base model, data, and main training settings, facilitating assessment of the effect of the representation. Together with the ablation experiments, the results support the practical value of the Wyckoff representation.

3.The paper provides a valuable empirical diagnosis of mode collapse during RL. It shows that improvements in reward and validity can coincide with concentration on a small number of prototypes and losses in novelty and diversity. It also investigates mitigation strategies, including a cross-batch uniqueness buffer, an expanded novelty reference set, a bounded creativity reward, and KL and entropy regularization. These findings offer practical insights for applying RL to crystal generation.

4.Implementation details are relatively comprehensive, and the paper is clearly written and easy to follow. The paper provides task proportions, prompts, parser correction rules, reward definitions, training hyperparameters, and computational costs, offering a solid basis for understanding and reproducing the method. The presentation is well structured and fluent.

Critical weaknesses

1.The downstream value of the auxiliary tasks, a major new contribution, has not yet been demonstrated. In Table 11, the generation-only model achieves validity, stability, and MetaSUN rates of 86.44%, 1.99%, and 3.61%, respectively, all higher than the full SFT model’s 84.08%, 1.62%, and 2.81%; both produce eight SUN candidates. Although the full model improves novelty, the current evidence more strongly supports a trade-off among metrics than a clear connection between learning symmetry knowledge and improving crystal generation.

2.Conditional generation remains weak for low-symmetry crystal systems, and the causes of failure are unclear. After conditional RL, the fractions of generations that are both valid and on-target are only 44.9% and 16.6% for monoclinic and triclinic systems, respectively. The paper does not distinguish whether these failures arise from invalid structures, failure to match the requested condition, or both, nor does it sufficiently analyze the effects of the training distribution and structural complexity.

3.The training dataset is limited, and generalization to more complex crystals remains unverified. Generative model training and the main experiments are restricted to MP‑20, leaving it unclear whether the conclusions extend to crystals with more symmetry-inequivalent sites or more complex compositions. Particularly for low-symmetry structures, increasing the number of sites lengthens the generated sequence and may make constraint satisfaction more difficult. The current experiments do not test this scalability.

4.The necessity of the curriculum is unclear. The paper does not compare dynamic task proportions with fixed proportions, so it is unclear whether progressively reducing the lookup share and increasing the generation share outperforms a simple fixed mixture of tasks. A controlled comparison with matched total numbers of examples for each task would be helpful.

Questions for authors

1.Both SFT and RL produce eight SUN candidates, whereas the number of MetaSUN candidates increases from 59 to 80. How do the authors explain these different effects of RL on the two metrics? The number of strictly stable structures actually increases from 34 to 38; is the lack of improvement in SUN primarily due to limitations in novelty or to duplicate structures? In addition, the energy reward assigns the same score to all structures on or below the convex hull, and training and evaluation use different interatomic potentials. Could these choices affect the gains in strictly stable candidates? Could repeated independent sampling or targeted analyses establish whether this observation is robust?

2.Appendix H.4 shows that, even after incorporating all mitigation strategies, space-group coverage decreases from 0.309 for SFT to 0.257 for RL. How can the authors distinguish whether this change reflects a trade-off in multi-objective optimization or residual mode collapse? Under a fixed sampling budget, could the authors report how the entropy of the space-group distribution, the proportions of dominant space groups, and the cross-batch structure duplication rate evolve during training? These measurements could help determine whether the distribution becomes increasingly concentrated and whether such metrics can inform checkpoint selection.

4 comments
AuthorsOpenReview ↗

Author's Response to Reviewer CMZy (1/3)

We sincerely thank Reviewer CMZy for the detailed, constructive review and for recognizing the motivation, controlled comparisons, and reproducible implementation. We have expanded the study with independent DFT checks and controlled experiments, with manuscript additions marked in blue. The main generation tables now report mean ± sample SD over four random, disjoint 2,500-crystal batches.

W1: The evidence supports a trade-off rather than a demonstrated link from symmetry knowledge to improved generation.

The expanded experiments show a clear trade-off: auxiliary supervision improves explicit symmetry recall, novelty, and uniqueness, while generation-only SFT gives higher validity and predicted stability.

Table 1 uses identical non-thinking decoding, with five samples for each of 1,729 held-out lookup positions:

Qwen-3.5-9BLookup mean@5Lookup pass@5
Pretrained9.74%17.87%
Generation-only SFT10.09%16.19%
Full CrysTune SFT64.01%78.08%

Generation-only training leaves explicit recall close to its pretrained level. The auxiliary tasks substantially improve this capability. The generation-only comparison is presented in (Appendix H.3, Table 12):

Metric (%)Generation onlyFull SFT
Validity86.5 ± 0.982.9 ± 0.8
Uniqueness95.0 ± 0.496.4 ± 0.4
Novelty29.5 ± 0.436.5 ± 0.3
Stable1.68 ± 0.061.39 ± 0.24
Metastable60.1 ± 1.650.4 ± 1.3
SUN0.35 ± 0.100.21 ± 0.16
MetaSUN3.22 ± 0.293.27 ± 0.62

Full SFT raises novelty by 7.0 percentage points and uniqueness by 1.4 percentage points. Generation-only SFT has higher validity, stability, and mean SUN, while mean MetaSUN is similar.

Applying the same GRPO RL settings to both SFT initializations preserves this trade-off (Appendix H.3, Table 13). Full-SFT initialization gives higher novelty (35.1% versus 30.2%) and uniqueness (96.4% versus 95.2%). Generation-only initialization gives higher stability (1.62% versus 1.16%) and near-hull fraction (57.1% versus 47.6%). The parser audit (Appendix H.2, Table 11) also shows mixed effects: full SFT reduces invalid Wyckoff letters, but not every type of rule violation.

These experiments show that auxiliary supervision substantially improves explicit symmetry recall and increases generation novelty and uniqueness. They also identify a trade-off with validity and predicted stability, rather than a uniform generation improvement. Improved recall alone does not establish that the acquired symmetry knowledge causes the gains in novelty and uniqueness. We have revised Section 4 to distinguish these contributions from the benefits of the Wyckoff representation itself.

W4: Does the decreasing task schedule outperform a fixed mixture at matched task exposure?

We added a four-epoch fixed-mixture control (Appendix H.4, Table 14): 20% lookup, 15% canonicalization, and 65% generation. The base model, optimizer settings, seed, and total training budget (3,392 updates) match the decreasing schedule.

MetricDecreasing lookupConstant 20%
Generation validation loss0.459280.46566
Teacher-forced lookup exact match81.86%99.94%
Rejected outputs / first 2,500 raw attempts15 (0.60%)24 (0.96%)
Physical validity84.08%82.20%
Novelty36.58%38.98%
SUN count810
MetaSUN count6777

Each schedule has one 2,500-structure evaluation. Constant lookup produces more SUN and MetaSUN structures (10 versus 8 and 77 versus 67). We retain decreasing lookup for its lower generation validation loss, higher physical validity, and fewer parser rejections. The control therefore identifies a trade-off between generation reliability and observed discovery yield. Appendix H.4, Table 14 and Appendix H.2, Table 11 report both outcomes.

The control matches total training budget, not per-task exposure: constant lookup doubles lookup examples (21,708 versus 10,855), leaving fewer generation examples. It tests task allocation, while isolating temporal scheduling would require equal cumulative exposure to each task. Appendix H.4 details this distinction.

AuthorsOpenReview ↗

Author's Response to Reviewer CMZy (2/3)

W2: Why do low-symmetry requests fail?

We now separate condition matching from physical validity for the final conditional policy in Figure 2(b):

Requested systemOn targetValidValid and on targetDistance-check failure
Cubic100.0%96.4%96.4%0.12%
Monoclinic96.8%50.4%49.2%47.0%
Triclinic92.9%20.6%19.6%78.6%

Most outputs match the requested crystal system. The main bottleneck is physical validity, especially interatomic-distance violations, affecting 47.0% of monoclinic and 78.6% of triclinic outputs.

Structural complexity and training coverage are plausible contributors. Triclinic outputs average 7.61 Wyckoff sites, versus 2.99 for cubic outputs. Within triclinic outputs, validity falls from 79.6% for 1–3 sites to 11.4% for 10–13 sites, consistent with greater difficulty placing more sites without overlaps. Triclinic structures also constitute only 4.2% of MP-20, compared with 22.9% cubic. These diagnostics identify geometric validity as the immediate failure mode, while suggesting complexity and data imbalance as contributing factors.

We hope this clarifies the source of the low-symmetry failures and would be happy to address any further questions.

W3: Does this extend beyond the small MP-20 setting?

We added two separate scaling comparisons against the common 9B MP-20 SFT reference (updated Table 2 in the revised version):

Metric (%)9B MP-2027B MP-209B Alex-MP-20
Validity82.9 ± 0.885.9 ± 0.894.4 ± 0.4
Novelty36.5 ± 0.331.5 ± 0.720.7 ± 0.7
SUN0.21 ± 0.160.30 ± 0.200.16 ± 0.11
MetaSUN3.27 ± 0.623.31 ± 0.5711.72 ± 0.36

Alex-MP-20 substantially improves validity and MetaSUN in this experiment, while reducing novelty. It also reduces raw Yb to 0.44%, although broader f-block prevalence remains high. This changes the corpus composition and tuned hyperparameters as well as its size. Moreover, Alex-MP-20 retains the 20-atom restriction. It therefore extends the data-distribution test but does not answer generalization to larger cells or more symmetry-inequivalent sites. The 27B experiment also does not establish a discovery improvement from model size.

AuthorsOpenReview ↗

Author's Response to Reviewer CMZy (3/3)

Q1: Why does RL affect SUN and MetaSUN differently?

How variable are the discovery results? Table 2 now reports mean ± sample SD across four 2,500-crystal batches from each 10k evaluation. RL increases mean SUN from 0.21 ± 0.16% to 0.27 ± 0.06% and MetaSUN from 3.27 ± 0.62% to 3.76 ± 0.47%. Mean SUN count rises from 4.3 to 6.3. These results quantify variability beyond the original single-batch comparison.

Is SUN limited by novelty or duplication? The pooled 10k analysis separates the filters:

Filtering stageSFTRL
Valid8,2859,180
Stable115106
Stable and unique10188
Stable, unique and novel1421

Among stable structures, deduplication removes 14 SFT and 18 RL entries. Novelty filtering then removes 87 and 67, respectively. Thus novelty is the larger additional bottleneck within the already small stable set. RL retains more SUN candidates despite having fewer stable candidates overall.

Why clip energy reward? Clipping limits the incentive to exploit overly favorable MLIP predictions and helps guard against reward hacking. Chemeleon2 uses the same clipping strategy. Structures on or below the hull therefore receive equal energy reward, while the creativity term continues to reward novelty and uniqueness, providing an incentive to explore distinct structures rather than pursue increasingly negative predicted energies.

Does the choice of potential matter? We compared MACE-MPA-0 and MACE-OMAT-0 as training reward potentials under the same validity-gated RL recipe (Table 18). Both belong to the MACE architecture family, but are pretrained on different datasets (MPTrj + sAlex and OMAT, respectively) and use different reference hulls in our reward calculation. Both policies are evaluated with the same ORB model and benchmark settings at 320 updates.

MetricMACE-MPAMACE-OMAT
Generated crystals2,5002,500
Valid crystals2,401 (96.04%)2,400 (96.00%)
Stable crystals22 (0.92%)57 (2.38%)
MetaSUN crystals67 (2.79%)82 (3.42%)

These are pre-relaxation results, with stability and MetaSUN rates over valid crystals. MACE-OMAT improves stability while maintaining validity. RL training was limited to these MACE variants. Additional MatterSim and CHGNet evaluations appear in the appendix. Across 60,769 structures, unrelaxed MACE-OMAT and relaxed ORB hull energies show strong agreement (Appendix H.13, Figure 7).

What do targeted analyses show? With novelty-only reward, LeMat-Bulk novelty rises from 40.2% to 60.8%, while the near-hull fraction falls from 49.0% to 22.5% and MetaSUN counts from 32 to 17 (Appendix H.6, Table 21). Novelty is learnable, but jointly achieving novelty and low energy remains challenging. The Yb audit and separate DFT analysis are discussed in our responses to TSt1 (W2/Q2) and Lk52 (Q3). The DFT study supports selected non-f-block near-hull candidates, without directly validating the Yb-rich SUN subset.

Q2: Is the reduced space-group coverage residual collapse?

We analyzed nine checkpoints of the main RL run using 2,500 raw generation attempts per checkpoint, with the same unconditional prompt, temperature 0.7 and top-p 1.0.

RL stepSG entropySG225 (%)SG194 (%)SG166 (%)Cross-batch duplicates (%)
0 (SFT)3.11023.110.17.55.31
403.08023.311.06.84.35
803.06124.310.97.43.69
1202.98326.010.77.76.04
1602.85828.612.17.15.58
2002.85029.211.97.64.86
2402.83828.013.17.86.07
2802.81228.314.27.14.98
3202.72931.713.76.25.18

Cross-batch duplication is the fraction of parseable crystals with a StructureMatcher-equivalent crystal in another of five fixed 500-attempt batches at the same checkpoint.

Key observations. Entropy decreases at every checkpoint and SG225 rises from 23.1% to 31.7%, showing symmetry concentration. Structural duplication shows no sustained increase (5.31% to 5.18%). This separates symmetry concentration from repeated-crystal generation, but does not rule out loss of symmetry modes or establish a causal optimization trade-off.

Checkpoint selection. Diversity alone would favor earlier checkpoints in this trajectory, but checkpoint selection should also consider validity, stability, and discovery metrics. We report the final checkpoint and have not used these diagnostics to select an alternative, because of the inherent trade-offs. The analysis shows why diversity should be considered alongside generation quality, rather than optimized in isolation.

We thank the reviewer for suggesting these diagnostics. We hope the added evidence addresses the central concerns and supports reconsideration of our work.

Thank you for the detailed responses and additional experiments, which strengthen the empirical analysis and clarify several of my concerns. While I appreciate these efforts, some concerns regarding the strength of the evidence for the main contributions and the scope of the validation remain, so I maintain my original score.

Reviewer_Lk52OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CrysTune, a method for generating crystal structures by fine-tuning pretrained LLMs on Wyckoff representations that explicitly encode crystal symmetry. CrysTune trains Qwen-3.5-9B with generation, Wyckoff lookup, and canonicalization tasks, then applies GRPO with rewards for stability, uniqueness, novelty, and diversity. Experiments show improved validity and predicted stability over a CIF-based LLM using the same backbone. And RL further improves validity and the yield of metastable, unique, and novel structures at some cost to novelty and symmetry coverage.

Critical strengths
  1. The authors compare CIF-style and Wyckoff representations using the same LLM backbone, training data, and fine-tuning configuration. The generation-only ablation helps distinguish the contribution of the representation from that of the auxiliary tasks. The results show the improvement in predicted stability over the CIF baseline persists under two evaluation protocols. These comparisons provide convincing evidence that the choice of representation matters for LLM-based crystal generation.

  2. Informative analysis of RL failure modes and evaluation sensitivity. The paper demonstrates that improvements in reward and validity can conceal severe mode collapse, including a concrete example where within-batch uniqueness fails to capture repetition across batches. It also shows how the choice of novelty reference and stability evaluation protocol can change model rankings. These analyses provide useful guidance for interpreting apparent progress in crystal generation and assessing whether RL improves the quality of the generated candidates.

Critical weaknesses
  1. The lookup and canonicalization tasks are newly proposed, but their benefit for crystal generation is not clear. In Table 11, adding these tasks improves novelty but lowers validity, predicted stability, and MetaSUN, while leaving the SUN count unchanged. Though the authors acknowledge this trade-off, the results do not establish a clear advantage over simpler generation-only training. I think this weakens the significance of one of the paper’s main contributions, particularly because the gains over the CIF baseline appear to come primarily from the existing Wyckoff representation.

  2. This paper lacks controlled ablations of the RL improvements. The experiments do not isolate the interventions credited with preventing mode collapse. In Table 12, adding the replay buffer also reduces the learning rate, while the final config changes the energy model, novelty reference, and evaluation checkpoint. So the observed recovery cannot be attributed to the buffer or individual reward modifications. Since explaining how these changes stabilize RL is an important part of the paper’s contribution, I think these limit the strength and generalizability of its conclusions.

Questions for authors
  1. How are parser failures accounted for in the evaluation? Section 3.3 states that rejected strings lower the reported validity, whereas Appendix K says they are resampled until 2,500 parseable structures are obtained. Could you explain these descriptions and clarify whether the reported validity is conditional on successful parsing?

  2. Are metastable structures counted consistently across Tables 2 and 15? For CrysTuneSFT, the metastability rate in Table 2 corresponds to approximately 1,061 structures, matching the 1,027 metastable plus 34 stable structures in Table 15. For CrysTuneRL, however, the rate corresponds to approximately 1,093 structures, already equal to the reported metastable count before adding the 38 stable structures. Could the authors clarify whether stable structures are included in each table’s metastability definition and explain the difference between these counts?

  3. Given that all reported SUN candidates contain lanthanides, and most or all contain Yb, can the authors provide independent DFT validation using consistent reference phases to establish whether these candidates are thermodynamically stable? How would the validation results affect the paper’s conclusions about its utility for materials discovery?

4 comments
AuthorsOpenReview ↗

Author's Response to Reviewer Lk52 (1/3)

We sincerely thank Reviewer Lk52 for the careful assessment and for recognizing the controlled representation comparison and our examination of how evaluation protocols affect the results. We have strengthened the evaluation with DFT checks and controlled experiments, with manuscript revisions marked in blue. The principal generation tables now report mean ± sample SD across four disjoint batches of 2,500 . All metrics are recomputed separately in each batch.

W1: What do the auxiliary tasks contribute beyond generation-only training?

The expanded experiments show a clear trade-off: auxiliary supervision improves explicit symmetry recall, novelty, and uniqueness, while generation-only SFT gives higher validity and predicted stability.

Table 1 uses identical non-thinking decoding, with five samples for each of 1,729 held-out lookup positions:

Qwen-3.5-9BLookup mean@5Lookup pass@5
Pretrained9.74%17.87%
Generation-only SFT10.09%16.19%
Full CrysTune SFT64.01%78.08%

Generation-only training leaves explicit recall close to its pretrained level. The auxiliary tasks substantially improve this capability. The generation comparison (Appendix H.3, Table 12) reports mean ± sample SD across four disjoint 2,500-crystal batches:

Metric (%)Generation onlyFull SFT
Validity86.5 ± 0.982.9 ± 0.8
Uniqueness95.0 ± 0.496.4 ± 0.4
Novelty29.5 ± 0.436.5 ± 0.3
Stable1.68 ± 0.061.39 ± 0.24
Metastable60.1 ± 1.650.4 ± 1.3
SUN0.35 ± 0.100.21 ± 0.16
MetaSUN3.22 ± 0.293.27 ± 0.62

Full SFT raises novelty by 7.0 percentage points and uniqueness by 1.4 percentage points. Generation-only SFT has higher validity, stability, and mean SUN, while mean MetaSUN is similar.

Applying the same GRPO RL to both SFT initializations preserves this trade-off (Appendix H.3, Table 13). Full-SFT initialization gives higher novelty (35.1% versus 30.2%) and uniqueness (96.4% versus 95.2%). Generation-only initialization gives higher stability (1.62% versus 1.16%) and near-hull fraction (57.1% versus 47.6%). The parser audit (Appendix H.2, Table 11) also shows mixed effects: full SFT reduces invalid Wyckoff letters, but not every type of rule violation.

We acknowledge this limitation: better symmetry recall may not establish an overall improvement in crystal generation. We consider the gap between learning symmetry rules and applying them during generation an insightful empirical finding. Section 4 now makes this distinction explicit, and we welcome feedback on framing the paper more clearly around this contribution, including presenting generation-only CrysTune as the simpler generation baseline.

W2: The earlier RL comparison changes several factors at once.

Thank you for the suggestion. We added a matched-step controlled study in the revised version (Appendix H.6, Tables 16–20). We also corrected the reference used to judge crystal stability during RL training (Appendix H.6, with energy-agreement checks in H.13). The observations are summarized below.

  • Learning rate and buffer: we compare buffer on/off at 3e-7, 6e-7, and 1e-6. At the main rate (6e-7), uniqueness changes only slightly, from 93.0% to 93.5%. At 1e-6, it improves from 36.2% to 70.2%, but space-group coverage remains low (0.026), and changing elements within similar structural templates can make outputs appear novel. The buffer helps reduce repetition at the higher rate. We have also highlighted this in our response to Reviewer TSt1.
  • Energy model and reference hull: keeping the other training settings, training steps, and evaluation protocol fixed, we compare MACE-OMAT with its corrected reference hull against MACE-MPA-0 with its Materials Project reference hull. Both give 96.0% validity. MetaSUN is 3.25% versus 2.67%, and stability is 2.38% versus 0.92%, respectively. This isolates the choice of energy model together with its reference hull.
  • Novelty reference: with the evaluation reference held fixed, using MP-20, Alexandria+MP, or LeMat-Bulk in the reward yields novelty of 13.4%, 24.9%, and 36.3%. LeMat-Bulk is also the evaluation reference, so that variant optimizes against the benchmark's novelty database and is reported as such.
  • Other factors: matched-step KL, entropy, and LoRA ablations are included. Larger-group and larger-batch experiments are reported with their unequal rollout budgets, shorter effective buffer histories, and changed diversity-term scaling. They are not clean evidence that batch size alone improves discovery.

The revised interpretation is that learning rate and reward/reference alignment matter substantially.

AuthorsOpenReview ↗

Author's Response to Reviewer Lk52 (2/3)

Q1: Are parser failures included in reported validity?

GenBench validity is conditional on obtaining an accepted CIF. A rejected raw string is logged and replaced until the requested number of accepted structures is obtained. GenBench then checks charge neutrality, minimum interatomic distance, and physical plausibility. Parser rejections are not added to that denominator.

We acknowledge the inconsistency in the earlier description. The revised Section 3.3 explicitly separates parser rejection from GenBench validity. Parser correction and rejection counts are reported in Appendix Table 11. The CrysTune_SFT evaluation rejects only 15 of its first 2,500 attempts (0.60%). This is distinct from its physical-validity score among accepted structures.

Q2: Do the metastability counts include stable structures consistently?

We acknowledge the inconsistency in our earlier reporting and have corrected it so that Table 2 and the appendix controlled-ablation tables use the same definitions. Stable means hull energy at or below zero. Metast. includes all structures at or below 0.1 eV/atom, including stable structures. MetaSUN measures uniqueness and novelty within this full near-hull set.

Updated discovery results. With the definitions and evaluation budget made explicit, RL improves mean SUN from 0.21 ± 0.16% to 0.27 ± 0.06% and mean MetaSUN from 3.27 ± 0.62% to 3.76 ± 0.47%. Validity also rises from 82.9% to 91.8%. The revised tables report these gains with uncertainty estimates.

AuthorsOpenReview ↗

Author's Response to Reviewer Lk52 (3/3)

Q3: Independent DFT validation and the lanthanide-rich SUN candidates.

For complementary DFT validation, we evaluated 30 selected ORB-relaxed near-hull candidates per model, excluding structures containing lanthanides or actinides, using Quantum ESPRESSO and consistent MC3D PBE-v1 references. The results below follow DFT relaxation. All counts use the original 30 selected candidates, and unresolved calculations are not counted as unstable.

ModelAcceptedUnresolvedBelow available hullNear hull, including below
Chemeleon21614215
PLaID++2010320
CrysTextLLM219221
CrysTune-gen228322
CrysTune SFT1911319
CrysTune RL273527

CrysTune RL has the largest verified near-hull count in this selected cohort, including five candidates below the available hull. Table 3 and Appendix H.11 provide the full protocol.

These results support near-hull energetics for selected CrysTune candidates under DFT. Candidate selection, unequal completion, and reference-hull coverage limit broader comparisons. Lanthanides and actinides were excluded because their reliable treatment exceeded the available computational budget, so this study does not directly validate the Yb-rich SUN candidates.

To address the Yb concern directly, Appendix H.10 separates Yb occurrence in all outputs from its concentration among SUN structures:

Please see our response to Reviewer TSt1 (W2/Q2) for the model-by-model Yb table.

Yb occurs in 2.23% of MP-20. The MP-20-fine-tuned LLMs evaluated here, including PLaID++, generate it more often than the non-LLM baselines. This tendency is not specific to CrysTune. RL increases Yb occurrence in all outputs, although its Yb share within SUN is lower than SFT's. Both CrysTune SUN sets remain strongly Yb-enriched.

MLIP prediction errors for particular compositions and missing competing phases in the reference hull could further favor Yb-containing structures (and potentially other lanthanides too) during stability screening. Separating these effects from genuine stability is difficult with the available data and would require a dedicated validation of Yb-containing candidates. The Alex-MP-20 run reduces raw Yb to 0.44%, suggesting a practical route to mitigate this specific enrichment. Yb occurs in less than 1% of both the Alex-MP-20 training corpus (0.68%) and its generated outputs (0.44%). Broader lanthanide and actinide prevalence remains higher.

We thank the reviewer for the constructive comments and suggestions. We hope the additional experiments and new insights addresses the central concerns and supports reconsideration of our work.

Thank you for the detailed response and substantial additional experiments. I appreciate the authors' efforts. The new controlled experiments clarify the contribution of the auxiliary tasks. And the matched-learning-rate and matched-step ablations substantially address my concern about confounded RL experiments.

My view of the submission is no longer negative, but as the paper involves strongly material-related content, which falls outside my area of expertise, I cannot increase my confidence rating or recommend acceptance/rejection. If the AC finds that some of my concerns are off the mark, please feel free to set them aside.

Reviewer_TSt1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies LLM-based crystal generation using canonical Wyckoff strings. CrysTune combines supervised generation with Wyckoff lookup and canonicalization tasks, followed by GRPO. Experiments on MP-20 show improvements over a CIF-style baseline using the same backbone. The full model improves Wyckoff recall over the pretrained model, while RL mainly improves validity and MetaSUN. The paper also examines mode collapse and how evaluation choices affect model rankings.

Critical strengths
  1. The comparison with CIF-style generation uses the same backbone, dataset, and tuning configuration. Together with the generation-only ablation, this provides good evidence that the Wyckoff representation helps LLM-based crystal generation.

  2. The paper looks carefully at cases where the metrics can be misleading. Improved validity and energy can hide mode collapse, and changing the evaluation protocol can reverse model rankings. These observations are useful for interpreting results in this area.

Critical weaknesses
  1. The experiments do not establish a clear overall generation benefit from the auxiliary tasks. In Table 11, generation-only training produces the same eight SUN structures and has higher validity, stability, and MetaSUN. The full model improves novelty, but the practical benefit of explicitly teaching the symmetry rules remains unclear. The authors acknowledge this, though it limits the contribution beyond the representation itself.

  2. The SUN comparison is based on only eight candidates per CrysTune model, most containing Yb. The authors acknowledge this limitation, but without repeated sampling or uncertainty estimates, it is difficult to judge whether the small differences between models are meaningful. This limits the conclusions that can be drawn from the SUN ranking.

Questions for authors
  1. What is the lookup accuracy after generation-only fine-tuning? This would help distinguish what the auxiliary tasks teach from what is already learned through generation.

  2. How much do the SUN and MetaSUN results vary across repeated sampling runs? Are the differences between CrysTune and the baselines consistent across runs?

  3. Is there an experiment isolating the replay buffer while keeping the learning rate, other reward components, and training steps fixed? Table 12 changes several factors together, so it is difficult to tell whether the buffer is responsible for avoiding collapse.

3 comments
AuthorsOpenReview ↗

Author's Response to Reviewer TSt1 (1/2)

We sincerely thank Reviewer TSt1 for the detailed, constructive assessment and for recognizing the representation comparison and evaluation analysis. We have added DFT checks and controlled experiments in response, with manuscript revisions marked in blue. The main generation comparisons now use four disjoint batches of 2,500 crystals, with mean ± sample SD. All metrics are recomputed separately for each evaluation.

W1 and Q1: Do the auxiliary tasks improve generation, and what does generation-only SFT learn?

The expanded experiments show a clear trade-off: auxiliary supervision improves explicit symmetry recall, novelty, and uniqueness, while generation-only SFT gives higher validity and predicted stability.

Table 1 uses identical non-thinking decoding, with five samples for each of 1,729 held-out lookup positions:

Qwen-3.5-9BLookup mean@5Lookup pass@5
Pretrained9.74%17.87%
Generation-only SFT10.09%16.19%
Full CrysTune SFT64.01%78.08%

Generation-only training leaves explicit recall close to its pretrained level. The auxiliary tasks substantially improve this capability. The generation-only comparison is presented in (Appendix H.3, Table 12):

Metric (%)Generation onlyFull SFT
Validity86.5 ± 0.982.9 ± 0.8
Uniqueness95.0 ± 0.496.4 ± 0.4
Novelty29.5 ± 0.436.5 ± 0.3
Stable1.68 ± 0.061.39 ± 0.24
Metastable60.1 ± 1.650.4 ± 1.3
SUN0.35 ± 0.100.21 ± 0.16
MetaSUN3.22 ± 0.293.27 ± 0.62

Full SFT raises novelty by 7.0 percentage points and uniqueness by 1.4 percentage points. Generation-only SFT has higher validity, stability, and mean SUN, while mean MetaSUN is similar.

Applying the same GRPO RL settings to both SFT initializations preserves this trade-off (Appendix H.3, Table 13). Full-SFT initialization gives higher novelty (35.1% versus 30.2%) and uniqueness (96.4% versus 95.2%). Generation-only initialization gives higher stability (1.62% versus 1.16%) and near-hull fraction (57.1% versus 47.6%). The parser audit (Appendix H.2, Table 11) also shows mixed effects: full SFT reduces invalid Wyckoff letters, but not every type of rule violation.

We acknowledge this limitation: better symmetry recall may not establish an overall improvement in crystal generation. We consider the gap between learning symmetry rules and applying them during generation an insightful empirical finding. Section 4 now makes this distinction explicit, and we welcome feedback on framing the paper more clearly around this contribution, including presenting generation-only CrysTune as the simpler generation baseline.

AuthorsOpenReview ↗

Author's Response to Reviewer TSt1 (2/2)

W2 and Q2: How robust are SUN and MetaSUN, particularly given Yb enrichment?

Table 2 now reports mean ± sample SD for every model. RL has SUN 0.27 ± 0.06% and MetaSUN 3.76 ± 0.47%, compared with SFT at 0.21 ± 0.16% and 3.27 ± 0.62%. RL produces 5–8 SUN structures per batch, compared with 2–3 for PLaID++ and 0–1 for CrysTextLLM. Its counts overlap with SFT, DiffCSP, and MatterGen.

To address the Yb concern directly, Appendix H.10 separates Yb occurrence in all outputs from its concentration among SUN structures:

ModelYb in all outputs (%)Yb among SUN (% and count)
Chemeleon20.020.0% (0/1)
DiffCSP2.2017.6% (3/17)
MatterGen MP-201.885.9% (1/17)
CrystalFormer2.4528.6% (2/7)
WyFormer2.40Undefined (0 SUN)
PLaID++3.2316.7% (1/6)
CrysTextLLM2.86100.0% (2/2)
CrysTune-gen3.5653.6% (15/28)
CrysTune SFT3.6685.7% (12/14)
CrysTune RL4.8371.4% (15/21)
CrysTune SFT, Alex-MP-200.440.0% (0/14)

These are pooled 10k counts. SUN totals differ from summed batch counts because uniqueness is recomputed at each sample budget. Yb occurs in 2.23% of MP-20. The MP-20-fine-tuned LLMs evaluated here, including PLaID++, generate it more often than the non-LLM baselines. RL increases Yb occurrence in all outputs, although its Yb share within SUN is lower than SFT's. Both CrysTune SUN sets remain Yb-enriched.

MLIP prediction errors for particular compositions and missing competing phases in the reference hull could further favor Yb-containing structures during stability screening. Separating these effects from genuine stability is difficult with the available data and would require a dedicated validation of Yb-containing candidates. The Alex-MP-20 run reduces raw Yb to 0.44%, suggesting a practical route to mitigate this specific enrichment. Yb occurs in less than 1% of both the Alex-MP-20 training corpus (0.68%) and its generated outputs (0.44%). Broader lanthanide and actinide prevalence remains high.

As a separate check, we tested whether candidates outside the Yb-rich subset remain near the hull under DFT calculations. We selected 30 generated structures per model that ORB predicted to be metastable and ran Quantum ESPRESSO calculations under the same chemistry restrictions. Table 3 (provided below) currently shows the results following DFT evaluations.

ModelAcceptedUnresolvedBelow hullNear hull
PLaID++2010320
CrysTextLLM219221
CrysTune-gen228322
CrysTune SFT1911319
CrysTune RL273527

All counts use the original 30 selected candidates. Near hull includes below hull. Unresolved calculations are not classified as unstable. RL has the largest verified near-hull count in this selected cohort, although selection and unequal completion limit broader comparisons. Full discovery counts and protocol are in Table 3 and Appendix H.11.

Q3: Is the replay buffer isolated from learning rate and checkpoint?

Yes. The new controlled study crosses buffer on/off with three learning rates, holding the initialization, other reward settings, and final step 320 fixed. The study uses a validity gate (i.e., minimum reward for invalid crystals), unlike the main RL policy, but we also show a separate gate-on/off experiment.

Learning rateBufferUnique (%)Novel (%)SG coverage
6e-7Off93.023.60.187
6e-7On93.524.90.196
1e-6Off36.216.60.017
1e-6On70.236.90.026

At the main learning rate (6e-7), adding the buffer changes uniqueness only slightly, from 93.0% to 93.5%. At the higher rate (1e-6), it improves uniqueness from 36.2% to 70.2%. However, the model still generates a narrow range of space groups and can appear novel by changing elements within similar structural templates. The buffer therefore helps reduce repetition at the higher learning rate.

We appreciate that your questions led to a more precise contribution: evidence about representation, learned symmetry knowledge, and the limits of successful RL optimization. We hope these additions address the central concerns and support reconsideration of the paper.

Thanks for addressing my concern. I've increased my rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Official Comment by Authors - Part 1

Dear Reviewers,

We sincerely thank you for your constructive assessments and for recognizing the controlled representation comparison, presentation, and analysis of RL failure modes. In response, we added extensive experiments, including model-size and training data scaling, DFT validation, and controlled ablations. Manuscript additions are marked in blue. The principal generation tables now report mean ± sample SD over four random, disjoint batches of 2,500 crystals from each model's 10,000 generated structures, with all metrics evaluated separately in each batch.

1. What auxiliary symmetry supervision teaches. Table 1 shows that lookup mean@5 rises from 9.74% before fine-tuning to 10.09% with generation-only training and 64.01% with the full curriculum. The auxiliary tasks improve generation novelty and uniqueness, while generation-only training gives higher validity and predicted stability. Applying the same RL recipe to both models preserves this pattern (Appendix H.3, Tables 12–13). We fully acknowledge this limitation: better recall of symmetry rules does not establish an overall improvement in crystal generation. We consider this gap an interesting empirical finding about how LLMs use crystallographic knowledge, and Section 4 now makes it explicit. We welcome the reviewers' feedback on how to frame the paper more clearly around this empirical contribution.

2. Controlled RL experiments. Appendix H.6 isolates learning rate and replay buffer (Table 16), novelty reference (Table 17), and the energy-model/hull pairing, KL, entropy, and fine-tuning method (Table 18). The buffer helps most at an aggressive learning rate. A validity gate-on/off comparison connects the gated ablations to the main policy (Table 15). Larger-batch and novelty-only diagnostics (Tables 19–21) further clarify the difficulty of improving stability and novelty together. These experiments replace the earlier interpretation based on changing several settings at once.

3. Chemistry and independent DFT checks. The 10k chemistry audit confirms Yb enrichment in CrysTune outputs and also in PLaID++, pointing to a broader challenge for LLM-based generation whose cause remains unresolved (Appendix H.10). To test performance beyond this chemistry, we performed DFT calculations with Quantum Espresso on 30 selected candidates per model, excluding lanthanides and actinides. RL achieves the largest currently verified near-hull count in this cohort: 27/30, including five below the available DFT reference hull, with three unresolved cases (Table 3, Appendix H.11). This provides independent evidence for promising RL-generated candidates outside the Yb-rich subset. The comparison retains failed and incomplete calculations and accounts for unequal completion.

4. Conditional generation. We extended conditional training to 320 steps. Valid-and-on-target success reaches 74.0% overall, including 49.2% for monoclinic and 19.6% for triclinic requests (Figure 2b). The failure analysis identifies interatomic-distance violations as the main obstacle for low-symmetry requests (Appendix H.14).

5. Task scheduling. We added a constant-lookup control with the same total training budget. The decreasing schedule has lower generation validation loss and fewer audited symmetry errors, while the fixed mixture improves lookup and has higher discovery counts in its 2,500-sample evaluation. Appendix H.4 and Table 14 explain this practical trade-off and the difference in cumulative task exposure.

6. Model and data scaling. Table 2 separately compares the 27B model and the 9B Alex-MP-20 model with the common 9B MP-20 baseline. The larger model improves validity and stability. Training on Alex-MP-20 raises validity to 94.4% and MetaSUN to 11.72%, while reducing novelty. Appendix H.15 details the data and hyperparameter changes accompanying this comparison.

The revised contribution is more extensive: a controlled study of representation and symmetry knowledge in crystal-generating LLMs, together with evidence about where multi-objective RL succeeds, fails, and can mislead evaluation. We hope the additional experiments and clearer limitations provide a stronger basis for reconsidering the work's significance.

AuthorsOpenReview ↗

Official Comment by Authors - Part 2

Dear Reviewers,

Thank you for your thoughtful reviews, which helped us investigate further and develop a more insightful analysis. We have highlighted key findings in bold to make our detailed responses easier to navigate. Before the discussion period ends, we would like to share a few final observations.

  1. LLM fine-tuning for crystal generation remains an active research area. Understanding the strengths and limitations of current fine-tuning strategies, including how learned structural and symmetry knowledge translates into candidate quality, is important for further progress. This is the central question our study seeks to address.

  2. With multiple evaluation metrics, identifying a single best model is difficult, particularly when optimizing competing objectives. We hope the revised paper, including a small-scale DFT study using open-source software, strengthens our contribution by showing that some CrysTune candidates lie below or near the available DFT reference hull.

  3. CrysTune Gen, trained only on generation, achieves higher predicted stability and the highest mean SUN score among the baselines in our MP-20 comparison, but lower novelty, uniqueness, and explicit symmetry recall than CrysTune SFT, trained with both auxiliary tasks. Applying the same GRPO procedure to both initializations preserves this trade-off. These experiments identify that stronger explicit symmetry knowledge may not always automatically translate into more stable generated crystals.

We have conducted extensive experiments to examine these outcomes from several perspectives. We sincerely thank Reviewer TSt1 for raising their score and all reviewers for the questions that motivated these additional analyses. In the final revision, we plan to focus on the following improvements.

  1. Refine the paper's organization to present CrysTune Gen as a strong baseline and more clearly explain why CrysTune SFT's substantial gains in Wyckoff knowledge do not yet translate into higher predicted stability. This will make the contributions and limitations of the auxiliary tasks easier to assess.

  2. Work toward extending the DFT analysis to f-block candidates, which remain challenging within our available computational resources and simulation capabilities. We will explore feasible options while clearly distinguishing completed calculations from future work. We believe the current DFT results already provide useful complementary evidence for the selected candidates studied.

Overall, we believe this work makes a valuable contribution to understanding LLM-based crystal generation, providing empirical evidence about both its practical promise and the limitations that future methods must address. We hope the expanded experiments and clarified findings address your concerns and support reconsideration of your scores. We remain happy to answer further questions before the discussion deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.