- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CrysTune, a framework for autoregressive inorganic crystal generation by fine-tuning the general-purpose large language model Qwen‑3.5‑9B on canonical Wyckoff string representations. Training incorporates two auxiliary tasks, Wyckoff lookup and canonicalization of symmetry-equivalent representations, followed by GRPO fine-tuning with rewards for stability, novelty, uniqueness, and diversity.
Experiments show that, compared with a CIF-style representation using the same base model, the Wyckoff representation improves validity and several stability-related metrics. The auxiliary tasks improve Wyckoff template recall accuracy but do not yield a clear improvement in generation quality. RL improves validity and the number of MetaSUN candidates, at the cost of some novelty and space-group coverage. The paper also analyzes the effects of mode collapse, decoding settings, and evaluation protocols on the results.
Critical strengths
1.The experiments reveal limitations in pretrained LLMs’ recall of Wyckoff templates. The paper evaluates Qwen‑3.5 models of several sizes and finds that their ability to accurately produce Wyckoff coordinate templates in the specified standard setting is limited. This provides empirical motivation for explicitly incorporating crystallographic knowledge and offers useful insights for LLM-based materials generation.
2.The research questions are clear, and controlled comparisons of representations support the effectiveness of the approach. CrysTxtLLM and CrysTune use the same base model, data, and main training settings, facilitating assessment of the effect of the representation. Together with the ablation experiments, the results support the practical value of the Wyckoff representation.
3.The paper provides a valuable empirical diagnosis of mode collapse during RL. It shows that improvements in reward and validity can coincide with concentration on a small number of prototypes and losses in novelty and diversity. It also investigates mitigation strategies, including a cross-batch uniqueness buffer, an expanded novelty reference set, a bounded creativity reward, and KL and entropy regularization. These findings offer practical insights for applying RL to crystal generation.
4.Implementation details are relatively comprehensive, and the paper is clearly written and easy to follow. The paper provides task proportions, prompts, parser correction rules, reward definitions, training hyperparameters, and computational costs, offering a solid basis for understanding and reproducing the method. The presentation is well structured and fluent.
Critical weaknesses
1.The downstream value of the auxiliary tasks, a major new contribution, has not yet been demonstrated. In Table 11, the generation-only model achieves validity, stability, and MetaSUN rates of 86.44%, 1.99%, and 3.61%, respectively, all higher than the full SFT model’s 84.08%, 1.62%, and 2.81%; both produce eight SUN candidates. Although the full model improves novelty, the current evidence more strongly supports a trade-off among metrics than a clear connection between learning symmetry knowledge and improving crystal generation.
2.Conditional generation remains weak for low-symmetry crystal systems, and the causes of failure are unclear. After conditional RL, the fractions of generations that are both valid and on-target are only 44.9% and 16.6% for monoclinic and triclinic systems, respectively. The paper does not distinguish whether these failures arise from invalid structures, failure to match the requested condition, or both, nor does it sufficiently analyze the effects of the training distribution and structural complexity.
3.The training dataset is limited, and generalization to more complex crystals remains unverified. Generative model training and the main experiments are restricted to MP‑20, leaving it unclear whether the conclusions extend to crystals with more symmetry-inequivalent sites or more complex compositions. Particularly for low-symmetry structures, increasing the number of sites lengthens the generated sequence and may make constraint satisfaction more difficult. The current experiments do not test this scalability.
4.The necessity of the curriculum is unclear. The paper does not compare dynamic task proportions with fixed proportions, so it is unclear whether progressively reducing the lookup share and increasing the generation share outperforms a simple fixed mixture of tasks. A controlled comparison with matched total numbers of examples for each task would be helpful.
Questions for authors
1.Both SFT and RL produce eight SUN candidates, whereas the number of MetaSUN candidates increases from 59 to 80. How do the authors explain these different effects of RL on the two metrics? The number of strictly stable structures actually increases from 34 to 38; is the lack of improvement in SUN primarily due to limitations in novelty or to duplicate structures? In addition, the energy reward assigns the same score to all structures on or below the convex hull, and training and evaluation use different interatomic potentials. Could these choices affect the gains in strictly stable candidates? Could repeated independent sampling or targeted analyses establish whether this observation is robust?
2.Appendix H.4 shows that, even after incorporating all mitigation strategies, space-group coverage decreases from 0.309 for SFT to 0.257 for RL. How can the authors distinguish whether this change reflects a trade-off in multi-objective optimization or residual mode collapse? Under a fixed sampling budget, could the authors report how the entropy of the space-group distribution, the proportions of dominant space groups, and the cross-batch structure duplication rate evolve during training? These measurements could help determine whether the distribution becomes increasingly concentrated and whether such metrics can inform checkpoint selection.
4 comments
Author's Response to Reviewer CMZy (2/3)
W2: Why do low-symmetry requests fail?
We now separate condition matching from physical validity for the final conditional policy in Figure 2(b):
| Requested system | On target | Valid | Valid and on target | Distance-check failure |
|---|---|---|---|---|
| Cubic | 100.0% | 96.4% | 96.4% | 0.12% |
| Monoclinic | 96.8% | 50.4% | 49.2% | 47.0% |
| Triclinic | 92.9% | 20.6% | 19.6% | 78.6% |
Most outputs match the requested crystal system. The main bottleneck is physical validity, especially interatomic-distance violations, affecting 47.0% of monoclinic and 78.6% of triclinic outputs.
Structural complexity and training coverage are plausible contributors. Triclinic outputs average 7.61 Wyckoff sites, versus 2.99 for cubic outputs. Within triclinic outputs, validity falls from 79.6% for 1–3 sites to 11.4% for 10–13 sites, consistent with greater difficulty placing more sites without overlaps. Triclinic structures also constitute only 4.2% of MP-20, compared with 22.9% cubic. These diagnostics identify geometric validity as the immediate failure mode, while suggesting complexity and data imbalance as contributing factors.
We hope this clarifies the source of the low-symmetry failures and would be happy to address any further questions.
W3: Does this extend beyond the small MP-20 setting?
We added two separate scaling comparisons against the common 9B MP-20 SFT reference (updated Table 2 in the revised version):
| Metric (%) | 9B MP-20 | 27B MP-20 | 9B Alex-MP-20 |
|---|---|---|---|
| Validity | 82.9 ± 0.8 | 85.9 ± 0.8 | 94.4 ± 0.4 |
| Novelty | 36.5 ± 0.3 | 31.5 ± 0.7 | 20.7 ± 0.7 |
| SUN | 0.21 ± 0.16 | 0.30 ± 0.20 | 0.16 ± 0.11 |
| MetaSUN | 3.27 ± 0.62 | 3.31 ± 0.57 | 11.72 ± 0.36 |
Alex-MP-20 substantially improves validity and MetaSUN in this experiment, while reducing novelty. It also reduces raw Yb to 0.44%, although broader f-block prevalence remains high. This changes the corpus composition and tuned hyperparameters as well as its size. Moreover, Alex-MP-20 retains the 20-atom restriction. It therefore extends the data-distribution test but does not answer generalization to larger cells or more symmetry-inequivalent sites. The 27B experiment also does not establish a discovery improvement from model size.
Author's Response to Reviewer CMZy (3/3)
Q1: Why does RL affect SUN and MetaSUN differently?
How variable are the discovery results? Table 2 now reports mean ± sample SD across four 2,500-crystal batches from each 10k evaluation. RL increases mean SUN from 0.21 ± 0.16% to 0.27 ± 0.06% and MetaSUN from 3.27 ± 0.62% to 3.76 ± 0.47%. Mean SUN count rises from 4.3 to 6.3. These results quantify variability beyond the original single-batch comparison.
Is SUN limited by novelty or duplication? The pooled 10k analysis separates the filters:
| Filtering stage | SFT | RL |
|---|---|---|
| Valid | 8,285 | 9,180 |
| Stable | 115 | 106 |
| Stable and unique | 101 | 88 |
| Stable, unique and novel | 14 | 21 |
Among stable structures, deduplication removes 14 SFT and 18 RL entries. Novelty filtering then removes 87 and 67, respectively. Thus novelty is the larger additional bottleneck within the already small stable set. RL retains more SUN candidates despite having fewer stable candidates overall.
Why clip energy reward? Clipping limits the incentive to exploit overly favorable MLIP predictions and helps guard against reward hacking. Chemeleon2 uses the same clipping strategy. Structures on or below the hull therefore receive equal energy reward, while the creativity term continues to reward novelty and uniqueness, providing an incentive to explore distinct structures rather than pursue increasingly negative predicted energies.
Does the choice of potential matter? We compared MACE-MPA-0 and MACE-OMAT-0 as training reward potentials under the same validity-gated RL recipe (Table 18). Both belong to the MACE architecture family, but are pretrained on different datasets (MPTrj + sAlex and OMAT, respectively) and use different reference hulls in our reward calculation. Both policies are evaluated with the same ORB model and benchmark settings at 320 updates.
| Metric | MACE-MPA | MACE-OMAT |
|---|---|---|
| Generated crystals | 2,500 | 2,500 |
| Valid crystals | 2,401 (96.04%) | 2,400 (96.00%) |
| Stable crystals | 22 (0.92%) | 57 (2.38%) |
| MetaSUN crystals | 67 (2.79%) | 82 (3.42%) |
These are pre-relaxation results, with stability and MetaSUN rates over valid crystals. MACE-OMAT improves stability while maintaining validity. RL training was limited to these MACE variants. Additional MatterSim and CHGNet evaluations appear in the appendix. Across 60,769 structures, unrelaxed MACE-OMAT and relaxed ORB hull energies show strong agreement (Appendix H.13, Figure 7).
What do targeted analyses show? With novelty-only reward, LeMat-Bulk novelty rises from 40.2% to 60.8%, while the near-hull fraction falls from 49.0% to 22.5% and MetaSUN counts from 32 to 17 (Appendix H.6, Table 21). Novelty is learnable, but jointly achieving novelty and low energy remains challenging. The Yb audit and separate DFT analysis are discussed in our responses to TSt1 (W2/Q2) and Lk52 (Q3). The DFT study supports selected non-f-block near-hull candidates, without directly validating the Yb-rich SUN subset.
Q2: Is the reduced space-group coverage residual collapse?
We analyzed nine checkpoints of the main RL run using 2,500 raw generation attempts per checkpoint, with the same unconditional prompt, temperature 0.7 and top-p 1.0.
| RL step | SG entropy | SG225 (%) | SG194 (%) | SG166 (%) | Cross-batch duplicates (%) |
|---|---|---|---|---|---|
| 0 (SFT) | 3.110 | 23.1 | 10.1 | 7.5 | 5.31 |
| 40 | 3.080 | 23.3 | 11.0 | 6.8 | 4.35 |
| 80 | 3.061 | 24.3 | 10.9 | 7.4 | 3.69 |
| 120 | 2.983 | 26.0 | 10.7 | 7.7 | 6.04 |
| 160 | 2.858 | 28.6 | 12.1 | 7.1 | 5.58 |
| 200 | 2.850 | 29.2 | 11.9 | 7.6 | 4.86 |
| 240 | 2.838 | 28.0 | 13.1 | 7.8 | 6.07 |
| 280 | 2.812 | 28.3 | 14.2 | 7.1 | 4.98 |
| 320 | 2.729 | 31.7 | 13.7 | 6.2 | 5.18 |
Cross-batch duplication is the fraction of parseable crystals with a StructureMatcher-equivalent crystal in another of five fixed 500-attempt batches at the same checkpoint.
Key observations. Entropy decreases at every checkpoint and SG225 rises from 23.1% to 31.7%, showing symmetry concentration. Structural duplication shows no sustained increase (5.31% to 5.18%). This separates symmetry concentration from repeated-crystal generation, but does not rule out loss of symmetry modes or establish a causal optimization trade-off.
Checkpoint selection. Diversity alone would favor earlier checkpoints in this trajectory, but checkpoint selection should also consider validity, stability, and discovery metrics. We report the final checkpoint and have not used these diagnostics to select an alternative, because of the inherent trade-offs. The analysis shows why diversity should be considered alongside generation quality, rather than optimized in isolation.
We thank the reviewer for suggesting these diagnostics. We hope the added evidence addresses the central concerns and supports reconsideration of our work.
Thank you for the detailed responses and additional experiments, which strengthen the empirical analysis and clarify several of my concerns. While I appreciate these efforts, some concerns regarding the strength of the evidence for the main contributions and the scope of the validation remain, so I maintain my original score.
Author's Response to Reviewer CMZy (1/3)
We sincerely thank Reviewer CMZy for the detailed, constructive review and for recognizing the motivation, controlled comparisons, and reproducible implementation. We have expanded the study with independent DFT checks and controlled experiments, with manuscript additions marked in blue. The main generation tables now report mean ± sample SD over four random, disjoint 2,500-crystal batches.
W1: The evidence supports a trade-off rather than a demonstrated link from symmetry knowledge to improved generation.
The expanded experiments show a clear trade-off: auxiliary supervision improves explicit symmetry recall, novelty, and uniqueness, while generation-only SFT gives higher validity and predicted stability.
Table 1 uses identical non-thinking decoding, with five samples for each of 1,729 held-out lookup positions:
Generation-only training leaves explicit recall close to its pretrained level. The auxiliary tasks substantially improve this capability. The generation-only comparison is presented in (Appendix H.3, Table 12):
Full SFT raises novelty by 7.0 percentage points and uniqueness by 1.4 percentage points. Generation-only SFT has higher validity, stability, and mean SUN, while mean MetaSUN is similar.
Applying the same GRPO RL settings to both SFT initializations preserves this trade-off (Appendix H.3, Table 13). Full-SFT initialization gives higher novelty (35.1% versus 30.2%) and uniqueness (96.4% versus 95.2%). Generation-only initialization gives higher stability (1.62% versus 1.16%) and near-hull fraction (57.1% versus 47.6%). The parser audit (Appendix H.2, Table 11) also shows mixed effects: full SFT reduces invalid Wyckoff letters, but not every type of rule violation.
These experiments show that auxiliary supervision substantially improves explicit symmetry recall and increases generation novelty and uniqueness. They also identify a trade-off with validity and predicted stability, rather than a uniform generation improvement. Improved recall alone does not establish that the acquired symmetry knowledge causes the gains in novelty and uniqueness. We have revised Section 4 to distinguish these contributions from the benefits of the Wyckoff representation itself.
W4: Does the decreasing task schedule outperform a fixed mixture at matched task exposure?
We added a four-epoch fixed-mixture control (Appendix H.4, Table 14): 20% lookup, 15% canonicalization, and 65% generation. The base model, optimizer settings, seed, and total training budget (3,392 updates) match the decreasing schedule.
Each schedule has one 2,500-structure evaluation. Constant lookup produces more SUN and MetaSUN structures (10 versus 8 and 77 versus 67). We retain decreasing lookup for its lower generation validation loss, higher physical validity, and fewer parser rejections. The control therefore identifies a trade-off between generation reliability and observed discovery yield. Appendix H.4, Table 14 and Appendix H.2, Table 11 report both outcomes.
The control matches total training budget, not per-task exposure: constant lookup doubles lookup examples (21,708 versus 10,855), leaving fewer generation examples. It tests task allocation, while isolating temporal scheduling would require equal cumulative exposure to each task. Appendix H.4 details this distinction.