- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CrystalGPD, a benchmark for crystal LLMs that covers de novo generation, structure prediction, design with a given formula and space group, and design with a given space group only. Each model keeps its own prompt and output format, and all outputs are scored with a shared pipeline. Six LLM families are tested, together with DiffCSP, OMatG and GPT-5.6-Luna. The main findings are that stable and novel structures are rare, performance drops a lot on the larger MP52 structures, symmetry is harder to control than composition, and rare space groups are much harder than common ones. The authors also build CrystalReAct, a GPT-5.6-Luna agent with retrieval tools and a verifier, which reaches 93.6% joint formula and space group match on MP20.
Critical strengths
-
A shared evaluation of crystal LLMs across several tasks is useful for the community, and the code is released. Many models were retrained with their released code, which is a lot of work.
-
The analysis goes beyond a ranking. The step by step breakdown of constraint satisfaction, the paired high and low frequency space groups and the subgroup analysis help explain where models fail. The size controlled comparison in Finding 6 is also well done.
Critical weaknesses
-
Because each model keeps its native interface, the comparisons are confounded. Some models get the full unit cell formula and others only the reduced formula, token limits range from 500 to 3,584, model sizes range from GPT-2 to 7B, and some checkpoints were trained on data other than MP20. MP20 also only contains structures with up to 20 atoms, so part of the drop on MP52 is expected. The findings support conclusions about these specific systems, but the paper presents them as general limits of crystal LLMs. In addition, MP52 is de-duplicated against MP20 only by Materials Project ID, which may leave duplicate structures.
-
CrystalReAct is not compared on equal terms. It can retrieve similar structures from the MP20 training set, and its verifier checks much of what the benchmark scores, with up to fifteen steps to revise. The baselines get neither. To support the claim that external knowledge helps, the baselines should at least get the same verifier and a similar sampling budget.
-
All results come from single runs. SUN counts are between 0 and 8 out of 2,500, so many differences are within noise.
-
In Table 4, CrystalReAct has a higher formula match (84.30) than validity (84.20) on MP20, which is not possible. Table 12 gives 82.30.
Questions for authors
-
Which checkpoints and training data were used for the MP52 results, especially for DiffCSP and OMatG?
-
How many MP52 targets remain duplicates of MP20 training structures if structure matching is used instead of IDs?
-
How does CrystalReAct compare with baselines that use the same verifier and a matched budget, and what does it cost per structure?
-
Do the space group findings hold for other symprec values?
6 comments
Response to Reviewer Nt7e (2/4)
Weakness 3 -- All results come from single runs. SUN counts are between 0 and 8 out of 2,500, so many differences are within noise.
Response: Our main conclusions do not rely on fine-grained rankings based on small differences in SUN discoveries. Our central objective is to identify recurring characteristics of current crystal-generation paradigms, particularly LLM post-training with SFT and RL. Specifically, our conclusions concern shared bottlenecks in physical validity, exact symmetry control, and out-of-distribution generation, rather than marginal performance differences of one or two SUN discoveries. Across the evaluated DNG sets, the consistently small SUN counts support our descriptive finding that jointly stable, unique, and novel material discovery remains rare.
Weakness 4 -- In Table 4, CrystalReAct has a higher formula match (84.30) than validity (84.20) on MP20, which is not possible. Table 12 gives 82.30.
Response: Thank you for identifying this transcription error in Table 4, which we have corrected and synchronized throughout the manuscript. The preserved evaluation records show that CrystalReAct produced 823 valid formula matches and 842 valid outputs out of 1,000 targets on the MP20 CSD task. Therefore, the correct FM is 82.30%, rather than 84.30%, consistent with the 84.20% Valid rate and the results in Table 12.
Question 1 -- Which checkpoints and training data were used for the MP52 results, especially for DiffCSP and OMatG?
Response: All MP52 results are obtained using the same MP20-trained checkpoints, without additional training or adaptation on MP52. As described in Appendix C.2, we train the models using the MP20 training set and evaluate them on the test sets of both MP20 and MP52. Specifically, the MP52 results are obtained by directly transferring the checkpoints used for MP20 evaluation. For DiffCSP and OMatG, we use their publicly released checkpoints trained on MP20 and evaluate them under the same protocol. Thus, the MP52 experiments assess cross-dataset generalization from MP20 to MP52.
Question 2 -- How many MP52 targets remain duplicates of MP20 training structures if structure matching is used instead of IDs?
Response: After the initial filtering based on Materials Project IDs and exact CIF overlap, StructureMatcher identified 81 additional duplicate structures (1.29%) among the remaining 6,298 MP52 targets. To assess the impact of these overlaps, we conducted supplementary evaluations on the deduplicated MP52 set. For CSP, we recalculated metrics over all 6,217 remaining targets; for CSD, we replaced the 10 overlapping targets in the original 1,000-target subset with non-overlapping targets. The results are reported below.
| CSP | CSD | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Valid | FM | MR@1 | RMSD | Valid | FM | SGM | FSGM |
| CrystaLLM | 43.64 | 40.15 | 14.06 | 0.070 | 33.60 | 27.80 | 33.20 | 27.50 |
| LLaMat-2-CIF | 47.29 | 35.27 | 11.13 | 0.132 | 47.40 | 35.00 | 6.20 | 5.80 |
| LLaMat-3-CIF | 57.13 | 42.37 | 17.53 | 0.099 | 54.70 | 43.00 | 8.80 | 8.60 |
| CrystalTextLLM | 25.45 | 22.87 | 2.62 | 0.228 | 25.10 | 22.20 | 2.40 | 2.30 |
| PLaID++-SFT | 48.45 | 41.27 | 7.14 | 0.098 | 40.70 | 34.60 | 34.20 | 29.70 |
| PLaID++-DPO | 45.36 | 39.01 | 6.64 | 0.096 | 35.10 | 28.70 | 31.20 | 26.50 |
| CrysText-SFT | 24.40 | 22.60 | 1.80 | 0.294 | 32.00 | 30.40 | 18.20 | 17.60 |
| CrysText-GRPO | 25.64 | 23.79 | 1.50 | 0.304 | 32.90 | 31.10 | 18.90 | 18.30 |
| CrysText++-SFT | 37.80 | 37.00 | 8.53 | 0.115 | 32.70 | 31.90 | 14.00 | 14.00 |
| CrysText++-GRPO | 39.33 | 38.46 | 8.51 | 0.113 | 39.40 | 37.70 | 15.60 | 15.60 |
| CrystalReasoner-SFT | 40.95 | 39.01 | 4.95 | 0.118 | 33.70 | 31.10 | 20.70 | 19.30 |
| CrystalReasoner-GRPO | 50.06 | 47.40 | 5.44 | 0.102 | 39.80 | 35.00 | 23.20 | 21.30 |
| CrystalReasoner++-SFT | 41.53 | 39.54 | 5.15 | 0.115 | 34.40 | 31.40 | 22.30 | 20.90 |
| CrystalReasoner++-GRPO | 49.01 | 46.60 | 5.52 | 0.104 | 39.20 | 35.80 | 24.10 | 22.50 |
| DiffCSP | 78.70 | 78.70 | 11.11 | 0.097 | ||||
| OMatG | 79.73 | 79.73 | 14.88 | 0.097 | ||||
| GPT-5.6-Luna | 72.67 | 68.65 | 2.48 | 0.100 | 32.50 | 30.40 | 29.80 | 27.70 |
| CrystalReAct | 65.64 | 63.29 | 9.62 | 0.050 | 47.10 | 40.50 | 45.80 | 39.50 |
| CrystalReAct-verify | 82.21 | 81.26 | 9.70 | 0.051 | 83.10 | 80.10 | 82.60 | 79.70 |
These results remain consistent with our original findings and preserve the overall performance trends across the evaluated methods. Thus, our key conclusions on MP52 remain valid under this stricter structural deduplication criterion. The structure-level overlap check is described in Appendix B.2, and the resulting CSP and CSD evaluations are reported in Appendix D.6.
Response to Reviewer Nt7e (3/4)
Question 3 -- How does CrystalReAct compare with baselines that use the same verifier and a matched budget, and what does it cost per structure?
Response: We conducted additional experiments comparing GPT-5.6-Luna and CrystalReAct with and without the same verifier. We report the generation performance and average token usage per structure under each setting in the tables below.
| DNG | CSP | CSD | SG | |||
|---|---|---|---|---|---|---|
| Model | MSUN Rate | MP20 MR@1 | MP52 MR@1 | MP20 FSGM | MP52 FSGM | SGM |
| GPT-5.6-Luna | 0.72 | 14.09 | 2.78 | 47.60 | 27.80 | 43.66 |
| GPT-5.6-Luna w/verify | 0.16 | 17.90 | 3.18 | 88.10 | 85.40 | 86.37 |
| CrystalReAct | 1.67 | 45.84 | 9.96 | 80.10 | 39.50 | 58.93 |
| CrystalReAct w/verify | 2.02 | 46.88 | 10.18 | 93.30 | 79.50 | 89.41 |
| Model | DNG | CSP MP20 | CSP MP52 | CSD MP20 | CSD MP52 | SG |
|---|---|---|---|---|---|---|
| GPT-5.6-Luna | 4885.75 | 1567.59 | 2064.98 | 2294.34 | 3128.64 | 2164.07 |
| GPT-5.6-Luna w/verify | 5797.17 | 3499.76 | 4858.32 | 5993.13 | 8922.86 | 5112.10 |
| CrystalReAct | 10394.15 | 4209.73 | 5244.48 | 3991.25 | 4259.85 | 2409.56 |
| CrystalReAct w/verify | 10945.13 | 7233.93 | 9639.96 | 7210.99 | 9478.21 | 6639.14 |
Question 4 -- Do the space group findings hold for other symprec values?
Response: Our main findings remain consistent across different symprec settings. While our main evaluation uses symprec = 0.1 Å, we additionally evaluate four alternative settings (0.01, 0.05, 0.5, and 1.0 Å) to examine the sensitivity of space-group identification to positional tolerance. Smaller symprec values impose stricter symmetry constraints and tend to yield lower-symmetry assignments, whereas larger values allow greater positional deviations and tend to yield higher-symmetry assignments.
| Model | Valid | SGM | High | Low | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| symprec | 0.01 | 0.05 | 0.1 | 0.5 | 1.0 | 0.01 | 0.05 | 0.1 | 0.5 | 1.0 | 0.01 | 0.05 | 0.1 | 0.5 | 1.0 | |
| LLaMat-2-CIF | 85.50 | 4.30 | 4.03 | 6.77 | 7.86 | 7.89 | 6.71 | 6.60 | 12.74 | 15.31 | 15.46 | 1.89 | 1.46 | 0.80 | 0.40 | 0.31 |
| LLaMat-3-CIF | 90.79 | 4.74 | 4.64 | 5.59 | 6.04 | 6.00 | 7.17 | 7.17 | 10.03 | 11.77 | 11.71 | 2.31 | 2.11 | 1.14 | 0.31 | 0.29 |
| CrystalTextLLM | 44.23 | 2.20 | 2.17 | 9.49 | 11.33 | 11.79 | 2.71 | 2.80 | 17.89 | 21.97 | 23.11 | 1.69 | 1.54 | 1.09 | 0.69 | 0.46 |
| PLaID++-SFT | 47.93 | 23.81 | 23.57 | 23.43 | 22.60 | 21.86 | 41.60 | 41.34 | 41.11 | 39.86 | 38.57 | 6.03 | 5.80 | 5.74 | 5.34 | 5.14 |
| PLaID++-DPO | 51.50 | 28.01 | 27.91 | 27.83 | 27.27 | 25.71 | 49.11 | 49.11 | 49.09 | 48.31 | 45.57 | 6.91 | 6.71 | 6.57 | 6.23 | 5.86 |
| CrysText-SFT | 52.67 | 21.34 | 21.50 | 21.14 | 20.09 | 19.34 | 37.74 | 38.86 | 38.66 | 37.77 | 36.54 | 4.94 | 4.14 | 3.63 | 2.40 | 2.14 |
| CrysText-GRPO | 56.96 | 22.69 | 22.76 | 22.56 | 21.50 | 20.60 | 40.51 | 41.06 | 41.09 | 39.97 | 38.69 | 4.86 | 4.46 | 4.03 | 3.03 | 2.51 |
| CrysText++-SFT | 63.79 | 10.63 | 10.14 | 9.90 | 8.91 | 8.66 | 20.29 | 19.54 | 19.20 | 17.34 | 17.00 | 0.97 | 0.74 | 0.60 | 0.49 | 0.31 |
| CrysText++-GRPO | 65.57 | 10.99 | 10.66 | 10.53 | 9.79 | 9.43 | 20.63 | 20.14 | 20.11 | 18.94 | 18.29 | 1.34 | 1.17 | 0.94 | 0.63 | 0.57 |
| CrystalReasoner-SFT | 58.36 | 19.34 | 19.51 | 19.89 | 20.80 | 19.91 | 33.63 | 35.03 | 36.23 | 38.60 | 37.37 | 5.06 | 4.00 | 3.54 | 3.00 | 2.46 |
| CrystalReasoner-GRPO | 67.01 | 24.36 | 23.86 | 23.94 | 23.86 | 23.06 | 40.69 | 41.69 | 42.60 | 43.66 | 42.69 | 8.03 | 6.03 | 5.29 | 4.06 | 3.43 |
| CrystalReasoner++-SFT | 44.34 | 18.03 | 18.10 | 18.06 | 17.57 | 16.83 | 31.83 | 33.06 | 33.83 | 33.86 | 32.57 | 4.23 | 3.14 | 2.29 | 1.29 | 1.09 |
| CrystalReasoner++-GRPO | 60.47 | 26.03 | 25.33 | 25.43 | 25.29 | 24.17 | 47.29 | 47.49 | 48.29 | 48.63 | 46.74 | 4.77 | 3.17 | 2.57 | 1.94 | 1.60 |
| GPT-5.6-Luna | 60.23 | 45.80 | 45.04 | 43.66 | 39.03 | 34.71 | 72.66 | 72.49 | 71.31 | 66.97 | 62.31 | 18.94 | 17.60 | 16.00 | 11.09 | 7.11 |
| CrystalReAct | 75.76 | 60.27 | 59.96 | 58.93 | 52.86 | 44.79 | 78.71 | 78.57 | 78.09 | 74.37 | 69.49 | 41.83 | 41.34 | 39.77 | 31.34 | 20.09 |
| CrystalReAct w/verify | 93.16 | 91.06 | 90.54 | 89.41 | 78.34 | 64.61 | 97.49 | 97.23 | 96.43 | 92.14 | 83.06 | 84.63 | 83.86 | 82.40 | 64.54 | 46.17 |
Response to Reviewer Nt7e (1/4)
We would like to address the weaknesses and questions you pointed out one by one below.
Weakness 1 -- Because each model keeps its native interface, the comparisons are confounded. Some models get the full unit cell formula and others only the reduced formula, token limits range from 500 to 3,584, model sizes range from GPT-2 to 7B, and some checkpoints were trained on data other than MP20. MP20 also only contains structures with up to 20 atoms, so part of the drop on MP52 is expected. The findings support conclusions about these specific systems, but the paper presents them as general limits of crystal LLMs. In addition, MP52 is de-duplicated against MP20 only by Materials Project ID, which may leave duplicate structures.
Response: We thank the reviewer for highlighting these important comparability and generalization concerns. We address them from three aspects: (1) clarifying the scope of cross-model comparisons under native interfaces, (2) examining the contribution of structural size to the MP20-MP52 performance gap, and (3) performing additional structure-level deduplication of MP52 against the MP20 training set. These analyses help establish the appropriate scope of our findings and strengthen the reliability of the benchmark results.
(1) Comparability across models. CrystalGPD is designed as a system-level benchmark for evaluating representative crystal-generation systems under their native interfaces, rather than as a strictly controlled comparison of model architectures or parameter scales. While preserving each model's native input representations, output formats, and generation budgets, we standardize the task instances, evaluation protocols, number of evaluated candidates, and downstream metrics. Consequently, differences in model configurations may affect absolute performance, and our cross-model comparisons characterize system-level capabilities rather than isolate individual model factors.
(2) Structural size and the MP20-MP52 performance gap. We agree that the larger structural sizes in MP52 contribute to the observed performance degradation. However, structural size alone does not explain this gap. As analyzed in Finding 6, even when restricting MP52 to structures with at most 20 atoms, the median Valid and FSGM rates decrease by 11.29 and 16.96 percentage points, respectively, relative to MP20.
(3) Structure-level deduplication. To address potential structural overlaps beyond identical Materials Project IDs, we performed additional structure-level deduplication using StructureMatcher against the MP20 training set. With ltol = 0.3, stol = 0.5, and angle_tol = 10°, we identified 81 additional overlapping structures among the 6,298 remaining MP52 targets, resulting in a final deduplicated test set of 6,217 structures. We conducted additional CSP and CSD evaluations on the deduplicated MP52 test set. The additional evaluation results and detailed deduplication procedure are provided in our response to Question 2.
Weakness 2 -- CrystalReAct is not compared on equal terms. It can retrieve similar structures from the MP20 training set, and its verifier checks much of what the benchmark scores, with up to fifteen steps to revise. The baselines get neither. To support the claim that external knowledge helps, the baselines should at least get the same verifier and a similar sampling budget.
Response: We would like to clarify that the reviewer conflates two variants of our method. The original CrystalReAct does not employ a verifier or verification-guided iterative refinement; these components are introduced only in the additional CrystalReAct w/verify variant. Therefore, the reported performance of CrystalReAct is achieved without the additional verification and revision opportunities described in the review.
Our experimental comparisons serve two complementary purposes. First, we compare CrystalReAct with conventional crystal-generation models to investigate the capabilities of the agentic generation paradigm for crystallographic tasks. Second, we compare CrystalReAct with its backbone, GPT-5.6-Luna, to examine the contribution of incorporating external crystallographic knowledge into the generation process. CrystalReAct w/verify is an additional extension designed to investigate whether verification feedback can further improve generation performance, rather than a component required by the original method.