acceptodds
Under review as a conference paper at ICLR 2027

CrystalGPD: Benchmarking the Capability Boundaries of Crystal Large Language Models

Abstract

Crystal large language models (LLMs) have emerged as a flexible paradigm for crystal structure generation, yet their capabilities remain difficult to characterize due to heterogeneous tasks, interfaces, representations, datasets, and evaluation protocols. To bridge this gap, we introduce CrystalGPD (Crystal Generation, Prediction, and Design), a unified benchmark for characterizing crystal LLM capability boundaries across de novo generation, crystal structure prediction, composition- and symmetry-constrained design, and symmetry-constrained design. CrystalGPD preserves native model interfaces while standardizing task instances, generation protocols, representation-specific parsing, and downstream evaluation. Our evaluation reveals recurring limitations in jointly achieving physical feasibility and novelty, maintaining validity and structural recovery as complexity increases, satisfying exact crystallographic symmetry, and generalizing from MP20 to MP52. In particular, composition is preserved more reliably than exact symmetry, and symmetry controllability degrades for lower-frequency space groups. Motivated by these findings, we further study CrystalReAct, a benchmark-driven agentic case study that provides on-demand crystallographic knowledge and verification during generation. CrystalReAct substantially improves physical validity and crystallographic constraint satisfaction, demonstrating that external knowledge can alleviate several limitations exposed by CrystalGPD. Together, CrystalGPD provides a unified framework for evaluating crystal LLMs and, more importantly, a systematic characterization of where their current capabilities succeed and break down, providing actionable guidance for developing more reliable crystal-generation systems. Anonymous code is available at https://anonymous.4open.science/r/CrystalGPD.

Primary area:
datasets and benchmarks
Keywords:
Crystal Structure Generation, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Nt7eOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CrystalGPD, a benchmark for crystal LLMs that covers de novo generation, structure prediction, design with a given formula and space group, and design with a given space group only. Each model keeps its own prompt and output format, and all outputs are scored with a shared pipeline. Six LLM families are tested, together with DiffCSP, OMatG and GPT-5.6-Luna. The main findings are that stable and novel structures are rare, performance drops a lot on the larger MP52 structures, symmetry is harder to control than composition, and rare space groups are much harder than common ones. The authors also build CrystalReAct, a GPT-5.6-Luna agent with retrieval tools and a verifier, which reaches 93.6% joint formula and space group match on MP20.

Critical strengths
  1. A shared evaluation of crystal LLMs across several tasks is useful for the community, and the code is released. Many models were retrained with their released code, which is a lot of work.

  2. The analysis goes beyond a ranking. The step by step breakdown of constraint satisfaction, the paired high and low frequency space groups and the subgroup analysis help explain where models fail. The size controlled comparison in Finding 6 is also well done.

Critical weaknesses
  1. Because each model keeps its native interface, the comparisons are confounded. Some models get the full unit cell formula and others only the reduced formula, token limits range from 500 to 3,584, model sizes range from GPT-2 to 7B, and some checkpoints were trained on data other than MP20. MP20 also only contains structures with up to 20 atoms, so part of the drop on MP52 is expected. The findings support conclusions about these specific systems, but the paper presents them as general limits of crystal LLMs. In addition, MP52 is de-duplicated against MP20 only by Materials Project ID, which may leave duplicate structures.

  2. CrystalReAct is not compared on equal terms. It can retrieve similar structures from the MP20 training set, and its verifier checks much of what the benchmark scores, with up to fifteen steps to revise. The baselines get neither. To support the claim that external knowledge helps, the baselines should at least get the same verifier and a similar sampling budget.

  3. All results come from single runs. SUN counts are between 0 and 8 out of 2,500, so many differences are within noise.

  4. In Table 4, CrystalReAct has a higher formula match (84.30) than validity (84.20) on MP20, which is not possible. Table 12 gives 82.30.

Questions for authors
  1. Which checkpoints and training data were used for the MP52 results, especially for DiffCSP and OMatG?

  2. How many MP52 targets remain duplicates of MP20 training structures if structure matching is used instead of IDs?

  3. How does CrystalReAct compare with baselines that use the same verifier and a matched budget, and what does it cost per structure?

  4. Do the space group findings hold for other symprec values?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Nt7e (1/4)

We would like to address the weaknesses and questions you pointed out one by one below.

Weakness 1 -- Because each model keeps its native interface, the comparisons are confounded. Some models get the full unit cell formula and others only the reduced formula, token limits range from 500 to 3,584, model sizes range from GPT-2 to 7B, and some checkpoints were trained on data other than MP20. MP20 also only contains structures with up to 20 atoms, so part of the drop on MP52 is expected. The findings support conclusions about these specific systems, but the paper presents them as general limits of crystal LLMs. In addition, MP52 is de-duplicated against MP20 only by Materials Project ID, which may leave duplicate structures.

Response: We thank the reviewer for highlighting these important comparability and generalization concerns. We address them from three aspects: (1) clarifying the scope of cross-model comparisons under native interfaces, (2) examining the contribution of structural size to the MP20-MP52 performance gap, and (3) performing additional structure-level deduplication of MP52 against the MP20 training set. These analyses help establish the appropriate scope of our findings and strengthen the reliability of the benchmark results.

(1) Comparability across models. CrystalGPD is designed as a system-level benchmark for evaluating representative crystal-generation systems under their native interfaces, rather than as a strictly controlled comparison of model architectures or parameter scales. While preserving each model's native input representations, output formats, and generation budgets, we standardize the task instances, evaluation protocols, number of evaluated candidates, and downstream metrics. Consequently, differences in model configurations may affect absolute performance, and our cross-model comparisons characterize system-level capabilities rather than isolate individual model factors.

(2) Structural size and the MP20-MP52 performance gap. We agree that the larger structural sizes in MP52 contribute to the observed performance degradation. However, structural size alone does not explain this gap. As analyzed in Finding 6, even when restricting MP52 to structures with at most 20 atoms, the median Valid and FSGM rates decrease by 11.29 and 16.96 percentage points, respectively, relative to MP20.

(3) Structure-level deduplication. To address potential structural overlaps beyond identical Materials Project IDs, we performed additional structure-level deduplication using StructureMatcher against the MP20 training set. With ltol = 0.3, stol = 0.5, and angle_tol = 10°, we identified 81 additional overlapping structures among the 6,298 remaining MP52 targets, resulting in a final deduplicated test set of 6,217 structures. We conducted additional CSP and CSD evaluations on the deduplicated MP52 test set. The additional evaluation results and detailed deduplication procedure are provided in our response to Question 2.

Weakness 2 -- CrystalReAct is not compared on equal terms. It can retrieve similar structures from the MP20 training set, and its verifier checks much of what the benchmark scores, with up to fifteen steps to revise. The baselines get neither. To support the claim that external knowledge helps, the baselines should at least get the same verifier and a similar sampling budget.

Response: We would like to clarify that the reviewer conflates two variants of our method. The original CrystalReAct does not employ a verifier or verification-guided iterative refinement; these components are introduced only in the additional CrystalReAct w/verify variant. Therefore, the reported performance of CrystalReAct is achieved without the additional verification and revision opportunities described in the review.

Our experimental comparisons serve two complementary purposes. First, we compare CrystalReAct with conventional crystal-generation models to investigate the capabilities of the agentic generation paradigm for crystallographic tasks. Second, we compare CrystalReAct with its backbone, GPT-5.6-Luna, to examine the contribution of incorporating external crystallographic knowledge into the generation process. CrystalReAct w/verify is an additional extension designed to investigate whether verification feedback can further improve generation performance, rather than a component required by the original method.

AuthorsOpenReview ↗

Response to Reviewer Nt7e (2/4)

Weakness 3 -- All results come from single runs. SUN counts are between 0 and 8 out of 2,500, so many differences are within noise.

Response: Our main conclusions do not rely on fine-grained rankings based on small differences in SUN discoveries. Our central objective is to identify recurring characteristics of current crystal-generation paradigms, particularly LLM post-training with SFT and RL. Specifically, our conclusions concern shared bottlenecks in physical validity, exact symmetry control, and out-of-distribution generation, rather than marginal performance differences of one or two SUN discoveries. Across the evaluated DNG sets, the consistently small SUN counts support our descriptive finding that jointly stable, unique, and novel material discovery remains rare.

Weakness 4 -- In Table 4, CrystalReAct has a higher formula match (84.30) than validity (84.20) on MP20, which is not possible. Table 12 gives 82.30.

Response: Thank you for identifying this transcription error in Table 4, which we have corrected and synchronized throughout the manuscript. The preserved evaluation records show that CrystalReAct produced 823 valid formula matches and 842 valid outputs out of 1,000 targets on the MP20 CSD task. Therefore, the correct FM is 82.30%, rather than 84.30%, consistent with the 84.20% Valid rate and the results in Table 12.

Question 1 -- Which checkpoints and training data were used for the MP52 results, especially for DiffCSP and OMatG?

Response: All MP52 results are obtained using the same MP20-trained checkpoints, without additional training or adaptation on MP52. As described in Appendix C.2, we train the models using the MP20 training set and evaluate them on the test sets of both MP20 and MP52. Specifically, the MP52 results are obtained by directly transferring the checkpoints used for MP20 evaluation. For DiffCSP and OMatG, we use their publicly released checkpoints trained on MP20 and evaluate them under the same protocol. Thus, the MP52 experiments assess cross-dataset generalization from MP20 to MP52.

Question 2 -- How many MP52 targets remain duplicates of MP20 training structures if structure matching is used instead of IDs?

Response: After the initial filtering based on Materials Project IDs and exact CIF overlap, StructureMatcher identified 81 additional duplicate structures (1.29%) among the remaining 6,298 MP52 targets. To assess the impact of these overlaps, we conducted supplementary evaluations on the deduplicated MP52 set. For CSP, we recalculated metrics over all 6,217 remaining targets; for CSD, we replaced the 10 overlapping targets in the original 1,000-target subset with non-overlapping targets. The results are reported below.

CSPCSD
ModelValidFMMR@1RMSDValidFMSGMFSGM
CrystaLLM43.6440.1514.060.07033.6027.8033.2027.50
LLaMat-2-CIF47.2935.2711.130.13247.4035.006.205.80
LLaMat-3-CIF57.1342.3717.530.09954.7043.008.808.60
CrystalTextLLM25.4522.872.620.22825.1022.202.402.30
PLaID++-SFT48.4541.277.140.09840.7034.6034.2029.70
PLaID++-DPO45.3639.016.640.09635.1028.7031.2026.50
CrysText-SFT24.4022.601.800.29432.0030.4018.2017.60
CrysText-GRPO25.6423.791.500.30432.9031.1018.9018.30
CrysText++-SFT37.8037.008.530.11532.7031.9014.0014.00
CrysText++-GRPO39.3338.468.510.11339.4037.7015.6015.60
CrystalReasoner-SFT40.9539.014.950.11833.7031.1020.7019.30
CrystalReasoner-GRPO50.0647.405.440.10239.8035.0023.2021.30
CrystalReasoner++-SFT41.5339.545.150.11534.4031.4022.3020.90
CrystalReasoner++-GRPO49.0146.605.520.10439.2035.8024.1022.50
DiffCSP78.7078.7011.110.097
OMatG79.7379.7314.880.097
GPT-5.6-Luna72.6768.652.480.10032.5030.4029.8027.70
CrystalReAct65.6463.299.620.05047.1040.5045.8039.50
CrystalReAct-verify82.2181.269.700.05183.1080.1082.6079.70

These results remain consistent with our original findings and preserve the overall performance trends across the evaluated methods. Thus, our key conclusions on MP52 remain valid under this stricter structural deduplication criterion. The structure-level overlap check is described in Appendix B.2, and the resulting CSP and CSD evaluations are reported in Appendix D.6.

AuthorsOpenReview ↗

Response to Reviewer Nt7e (3/4)

Question 3 -- How does CrystalReAct compare with baselines that use the same verifier and a matched budget, and what does it cost per structure?

Response: We conducted additional experiments comparing GPT-5.6-Luna and CrystalReAct with and without the same verifier. We report the generation performance and average token usage per structure under each setting in the tables below.

DNGCSPCSDSG
ModelMSUN RateMP20 MR@1MP52 MR@1MP20 FSGMMP52 FSGMSGM
GPT-5.6-Luna0.7214.092.7847.6027.8043.66
GPT-5.6-Luna w/verify0.1617.903.1888.1085.4086.37
CrystalReAct1.6745.849.9680.1039.5058.93
CrystalReAct w/verify2.0246.8810.1893.3079.5089.41
ModelDNGCSP MP20CSP MP52CSD MP20CSD MP52SG
GPT-5.6-Luna4885.751567.592064.982294.343128.642164.07
GPT-5.6-Luna w/verify5797.173499.764858.325993.138922.865112.10
CrystalReAct10394.154209.735244.483991.254259.852409.56
CrystalReAct w/verify10945.137233.939639.967210.999478.216639.14

Question 4 -- Do the space group findings hold for other symprec values?

Response: Our main findings remain consistent across different symprec settings. While our main evaluation uses symprec = 0.1 Å, we additionally evaluate four alternative settings (0.01, 0.05, 0.5, and 1.0 Å) to examine the sensitivity of space-group identification to positional tolerance. Smaller symprec values impose stricter symmetry constraints and tend to yield lower-symmetry assignments, whereas larger values allow greater positional deviations and tend to yield higher-symmetry assignments.

ModelValidSGMHighLow
symprec0.010.050.10.51.00.010.050.10.51.00.010.050.10.51.0
LLaMat-2-CIF85.504.304.036.777.867.896.716.6012.7415.3115.461.891.460.800.400.31
LLaMat-3-CIF90.794.744.645.596.046.007.177.1710.0311.7711.712.312.111.140.310.29
CrystalTextLLM44.232.202.179.4911.3311.792.712.8017.8921.9723.111.691.541.090.690.46
PLaID++-SFT47.9323.8123.5723.4322.6021.8641.6041.3441.1139.8638.576.035.805.745.345.14
PLaID++-DPO51.5028.0127.9127.8327.2725.7149.1149.1149.0948.3145.576.916.716.576.235.86
CrysText-SFT52.6721.3421.5021.1420.0919.3437.7438.8638.6637.7736.544.944.143.632.402.14
CrysText-GRPO56.9622.6922.7622.5621.5020.6040.5141.0641.0939.9738.694.864.464.033.032.51
CrysText++-SFT63.7910.6310.149.908.918.6620.2919.5419.2017.3417.000.970.740.600.490.31
CrysText++-GRPO65.5710.9910.6610.539.799.4320.6320.1420.1118.9418.291.341.170.940.630.57
CrystalReasoner-SFT58.3619.3419.5119.8920.8019.9133.6335.0336.2338.6037.375.064.003.543.002.46
CrystalReasoner-GRPO67.0124.3623.8623.9423.8623.0640.6941.6942.6043.6642.698.036.035.294.063.43
CrystalReasoner++-SFT44.3418.0318.1018.0617.5716.8331.8333.0633.8333.8632.574.233.142.291.291.09
CrystalReasoner++-GRPO60.4726.0325.3325.4325.2924.1747.2947.4948.2948.6346.744.773.172.571.941.60
GPT-5.6-Luna60.2345.8045.0443.6639.0334.7172.6672.4971.3166.9762.3118.9417.6016.0011.097.11
CrystalReAct75.7660.2759.9658.9352.8644.7978.7178.5778.0974.3769.4941.8341.3439.7731.3420.09
CrystalReAct w/verify93.1691.0690.5489.4178.3464.6197.4997.2396.4392.1483.0684.6383.8682.4064.5446.17
Reviewer_cTfwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CrystalGPD, a benchmark for comprehensively evaluating the capability boundaries of crystal LLMs on four tasks: de novo crystal generation (DNG), crystal structure prediction (CSP), composition- and symmetry-constrained crystal structure design (CSD), and symmetry-constrained CSD. The authors report that their evaluation reveals a number of limitations of existing crystal LLMs, and they further propose CrystalReAct to illustrate model performance when crystallographic knowledge and verification are accessible.

During evaluation, the benchmark preserves each model's native prompts and output formats, converts all outputs into crystal structures in a unified manner, and applies only minimal post-processing to the generated structures.

The paper does not appear to introduce new datasets or evaluation metrics; instead, it provides a comprehensive evaluation built on existing datasets and metrics.

Critical strengths
  1. The paper evaluates crystal LLMs across multiple tasks under a unified and comprehensive protocol, and identifies the limitations of these methods on crystal-structure tasks.
  2. The paper proposes CrystalReAct, which demonstrates the performance of the closed-source model GPT-5.6-Luna when crystallographic knowledge and verification are accessible.
Critical weaknesses
  1. The reported metrics contain internal contradictions. This is consequential because the evaluation pipeline, rather than a new model, is the primary contribution.

    • Appendix A defines FM as valid, formula-matching outputs. Nevertheless, Table 4 gives CrystalReAct 84.30% FM and 84.20% Valid on MP20. Table 12 instead reports 82.30% FM for the same configuration. The first pair violates the stated subset relation; the second establishes a main-text/appendix discrepancy.
    • Overall validity is defined as passing CN, MinDist, and PhysPlaus. In Table 11, however, DiffCSP has 99.30% Valid versus 98.47% CN on MP20, and OMatG has 98.81% Valid versus 98.14% CN. Either a different validity convention is used for these baselines or the reported values are inconsistent with the definition. Either explanation affects the cross-family comparison.
    • Figure 6 reports CrystaLLM's MP20 MinDist validity as 74.5%, whereas Table 11 reports 85.07%. Its reported MP20 MR/Valid of 34.9% also cannot be reconciled with Table 3's 65.95% MR@1 and 85.02% Valid under the stated common evaluation population.

    Until these discrepancies and configurations are reconciled, the benchmark's central numerical evidence is not sufficiently trustworthy.

  2. The protocol supports useful descriptive system comparisons, but not all of the stronger interpretations of capability boundaries or intervention effects. Several relevant factors remain entangled. CrystaLLM, DiffCSP, and OMatG receive unit-cell compositions with explicit counts, while other systems receive reduced formulas. Conditioning on validity does not remove this information advantage. The paper acknowledges the distinction, but does not quantify its contribution to the recovery gap. Likewise, DNG rejection sampling collects 2,500 parseable outputs and normalizes discovery rates by valid outputs, without reporting the total attempts and parsing failures. These are conditional quality estimates, not end-to-end yields under a fixed generation budget.

Questions for authors
  1. Which values are authoritative for the discrepancies between Tables 4 and 12, between Valid and CN in Table 11, and between Figure 6 and the CSP tables? Can the authors provide corrected results from one evaluation artifact and explain whether the same validity predicate and denominator are applied to every model?
  2. How many raw attempts, parsing failures, internal candidate revisions, and tokens were required for each task and model? Can the authors report DNG yield per raw attempt?
5 comments
AuthorsOpenReview ↗

Response to Reviewer cTfw (1/3)

We would like to address the weaknesses and questions you pointed out one by one below.

Weakness 1 -- The reported metrics contain internal contradictions. This is consequential because the evaluation pipeline, rather than a new model, is the primary contribution.

Response: Thank you for identifying these inconsistencies. We traced them to three distinct issues and have corrected the corresponding results and descriptions in the revised manuscript.

(1) Formula match discrepancy. This inconsistency resulted from a transcription error in Table 4, which we have corrected and synchronized throughout the manuscript. The preserved evaluation records show that CrystalReAct produced 823 valid formula matches and 842 valid outputs out of 1,000 targets on the MP20 CSD task. Therefore, the correct FM is 82.30%, rather than 84.30%, consistent with the 84.20% Valid rate and the results in Table 12.

(2) Validity and charge neutrality (CN). In the CSP and CSD tasks, the Valid rate can exceed the standalone CN pass rate because we apply a composition-matching exemption to the CN requirement. We have clarified this exemption in the revised evaluation metrics section. Specifically, the conditioning formulas in these tasks are taken from reference structures in the MP20/MP52 test sets. However, some reference compositions, such as RbO2\text{RbO}_2 (mp-12105) and Cs3O\text{Cs}_3\text{O} (mp-510262), do not pass our implemented CN check. To avoid penalizing generated structures for reproducing these reference compositions, we waive the CN requirement when the generated formula matches the conditioning formula. Thus, Valid requires passing both MinDist and PhysPlaus, together with either formula matching or CN passing. $\mathrm{Valid} = \mathrm{MinDist}\land\mathrm{PhysPlaus} \land(\mathrm{FormulaMatch}\lor\mathrm{CN}).$ The CN column reports the standalone CN pass rate without this exemption, explaining why Valid can exceed CN. The composition-matching exemption and the definition of MR/Valid are explicitly described in Appendix A.2.

(3) CrystaLLM's discrepancy. This inconsistency arose because Figure 6 retained results from an earlier run that constructed input prefixes using composition counts from the original test cell representation, rather than the standardized cell required by the official preprocessing procedure used for the CSP tables. We have updated Figure 6 using the corrected run to ensure consistency throughout the manuscript. The corrected MP20 results are MinDist = 85.07%(7,695/9,046), Valid = 85.02%(7,691/9,046), and MR@1 = 65.95%(5,966/9,046). Among the 5,966 matched structures, 5,817 also satisfy the validity criteria, yielding a MR/Valid rate of 75.63% (5,817/7,691).

AuthorsOpenReview ↗

Response to Reviewer cTfw (2/3)

Weakness 2 -- The protocol supports useful descriptive system comparisons, but not all of the stronger interpretations of capability boundaries or intervention effects.

Response: We thank the reviewer for raising two concerns: (1) differences in composition inputs may confound cross-family comparisons, and (2) DNG rejection sampling does not measure end-to-end yield. We address the first through the controlled experiments below and the second in our response to Question 2 (Q2).

We agree that differences in compositional information may confound cross-family comparisons, and thus conducted additional experiments using explicit unit-cell formulas. Since DiffCSP and OMatG preserve the number and types of atoms during generation, their inputs cannot be replaced with reduced formulas without altering the generated structures. We therefore adapted the inputs of PLaID++ and CrystalReasoner to use explicit unit-cell formulas and retrained both models accordingly. The results show that providing unit-cell compositions improves structural recovery, while Findings 3 and 4 remain supported. We report the overall CSP performance and size-stratified MinDist validity and MR/Valid, with DiffCSP and OMatG as references.

TypeModelMP20MP52
ValidFMMR@1RMSDValidFMMR@1RMSD
Reduced FormulaPLaID++76.4174.6239.190.04845.8139.527.130.098
Full Cell FormulaPLaID++81.2680.2357.680.05241.2436.9511.530.101
Reduced FormulaCrystalReasoner79.7379.3635.290.05550.4647.825.870.104
Full Cell FormulaCrystalReasoner73.8073.7044.880.06523.9922.428.370.126
Full Cell FormulaDiffCSP99.3099.3056.920.06078.9578.9511.750.097
Full Cell FormulaOMatG98.8198.8168.410.07079.9379.9315.670.097

The results confirm that providing explicit unit-cell compositions improves MR@1, explaining part of the original recovery gap. Nevertheless, Findings 3 and 4 remain supported: physical validity still deteriorates with increasing structural complexity, and structural recovery remains challenging even for valid generations. To further assess the robustness of these findings under explicit composition inputs, we compare MinDist validity and MR/Valid across reference-structure size bins below.

MinDist ValidMP20MP52
TypeModel1-56-1011-201-56-1011-2021-3030+
Reduced FormulaPLaID++93.3582.9162.1782.4476.3662.1443.2633.83
Full Cell FormulaPLaID++98.3989.2365.4796.9585.2060.2436.7924.88
Reduced FormulaCrystalReasoner96.8886.6064.8891.6085.3764.6447.0238.16
Full Cell FormulaCrystalReasoner98.5386.3350.8597.7187.2446.8515.374.09
Full Cell FormulaDiffCSP100.0099.9399.15100.0098.8196.1091.6557.33
Full Cell FormulaOMatG99.8299.5697.81100.0098.6494.0188.5562.88
MR/ValidMP20MP52
TypeModel1-56-1011-201-56-1011-2021-3030+
Reduced FormulaPLaID++62.0347.5642.2730.5636.8334.030.721.15
Full Cell FormulaPLaID++82.5969.2758.1957.4856.0951.501.252.01
Reduced FormulaCrystalReasoner66.0443.1722.2435.0034.8618.890.330.70
Full Cell FormulaCrystalReasoner76.7458.2437.6359.2046.2036.031.000.00
Full Cell FormulaDiffCSP79.7063.5039.3061.0751.1232.951.940.40
Full Cell FormulaOMatG81.9972.8858.3261.5463.4552.601.320.67

Even with unit-cell formula conditioning, MinDist validity declines as the number of atoms increases, with the evaluated crystal LLMs exhibiting more pronounced deterioration than the diffusion- and flow-based references. This supports Finding 3: physical validity becomes a scaling bottleneck as structural complexity increases. Furthermore, MR/Valid also deteriorates substantially with increasing structure size, indicating that the difficulty of structural recovery persists even when considering only valid generations. This supports Finding 4 and shows that the recovery gap cannot be attributed solely to invalid outputs. Overall, these experiments quantify the impact of unit-cell formula inputs while confirming that both findings persist. These additional experiments, including the overall CSP results and size-stratified MinDist validity and MR/Valid, are included in Appendix D.5.

AuthorsOpenReview ↗

Response to Reviewer cTfw (3/3)

Question 1 -- Which values are authoritative for the discrepancies between Tables 4 and 12, between Valid and CN in Table 11, and between Figure 6 and the CSP tables? Can the authors provide corrected results from one evaluation artifact and explain whether the same validity predicate and denominator are applied to every model?

Response: Thank you for raising this concern. As detailed in our response to Weakness 1, we have corrected the transcription error, clarified the composition-matching exemption in the validity criterion, and standardized CrystaLLM's evaluation configuration. The corrected results reported in the revised manuscript are authoritative. All models follow the same metric definitions and task-specific denominator rules, with the composition-matching exemption applied uniformly across CSP and CSD. We have also updated the corresponding tables and figures to ensure consistency throughout the manuscript.

Question 2 -- How many raw attempts, parsing failures, internal candidate revisions, and tokens were required for each task and model? Can the authors report DNG yield per raw attempt?

Response: Thank you for raising this question. We provide raw generation attempts, parsing failure rates, MSUN yield per raw attempt, and average output tokens for the evaluated models. The DNG sampling-efficiency results and output-token statistics are also reported in Appendices D.7.1 and D.7.2, respectively. Regarding internal candidate revisions, our evaluation pipeline does not include a separately defined revision process, and we would appreciate clarification on what the reviewer means by this term.

ModelRaw AttemptParse Failure RateMSUNMSUN Rate
CrystaLLM25240.95%371.47%
LLaMat-2-CIF25050.20%381.52%
LLaMat-3-CIF25100.40%471.87%
CrystalTextLLM26856.89%90.34%
PLaID++-SFT26856.89%692.57%
PLaID++-DPO25702.72%411.60%
CrysText-SFT27609.42%120.43%
CrysText-GRPO25361.42%70.28%
CrysText++-SFT25481.88%160.63%
CrysText++-GRPO25150.60%210.83%
CrystalReasoner-SFT278010.07%20.07%
CrystalReasoner-GRPO292714.59%50.17%
CrystalReasoner++-SFT25883.40%662.55%
CrystalReasoner++-GRPO25361.42%722.84%
GPT-5.6-Luna341226.73%120.35%
CrystalReAct26084.14%321.23%
CrystalReAct w/verify25923.55%401.54%

We further report the average output token counts for each model across DNG, CSP, CSD, and SG. The results are summarized below to facilitate comparison of token consumption across models and tasks.

ModelDNGCSPCSDSG
MP20MP52MP20MP52
CrystaLLM389.47363.99438.79275.34350.02-
LLaMat-2-CIF145.87195.28328.74195.00338.83171.98
LLaMat-3-CIF117.78163.47294.14157.01291.07135.58
CrystalTextLLM161.81182.50269.05189.47289.22217.73
PLaID++-SFT164.58168.60230.23162.47233.29190.93
PLaID++-DPO139.78168.31229.94163.68233.54171.40
CrysText-SFT881.62941.571177.56772.121183.19788.13
CrysText-GRPO881.08937.531162.20781.791153.58781.83
CrysText++-SFT808.80799.101080.30770.321113.10817.89
CrysText++-GRPO801.96788.251065.22768.941082.37817.50
CrystalReasoner-SFT780.411045.061441.791039.881562.59919.53
CrystalReasoner-GRPO737.761006.141394.001010.171507.40850.81
CrystalReasoner++-SFT992.141049.601450.551023.421557.341136.73
CrystalReasoner++-GRPO821.301021.421418.651014.691500.23929.97
GPT-5.6-Luna3052.791187.981684.371900.322737.451729.06
CrystalReAct2253.20661.951043.60738.191248.10693.70
CrystalReAct w/verify2187.77884.461501.911043.642077.481362.05

Regarding internal candidate revisions, our evaluation pipeline does not include a separately defined revision procedure. We would appreciate clarification on what the reviewer means by this term so that we can respond accordingly.

Reviewer_K9hiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on evaluating crystal LLMs for crystal structure generation and proposes CrystalGPD, a unified benchmark covering four crystal generation tasks:

1). De novo generation (DNG): The model is asked to generate a crystal structure without any specified chemical constraints.

2). Crystal structure prediction (CSP): The model is given a chemical formula and is asked to generate the corresponding crystal structure, including the lattice parameters and atomic coordinates.

3). Composition- and symmetry-constrained crystal structure design (CSD): Besides the chemical formula, the model is also given a target space group and is required to generate a crystal structure satisfying both constraints.

4). Symmetry-constrained crystal structure design: The model is given only a target space group and is asked to generate a crystal structure satisfying the symmetry constraint.

The benchmark data are mainly curated from existing crystal datasets. The paper then evaluates multiple existing crystal LLMs in terms of the generated structures' physical validity, stability, novelty, structure prediction accuracy, and symmetry constraint satisfaction. The paper also introduces an agentic crystal generation framework, CrystalReAct, which can access external crystallographic knowledge and verification tools during generation.

The benchmark results show that existing crystal LLMs struggle to generate structures that are both stable and novel. The paper also reports other findings, such as decreased performance for larger crystal structures. In addition, the authors show that CrystalReAct improves physical validity and symmetry constraint satisfaction.

Critical strengths
  1. The paper is well-organized and well-written. The figures and tables are also very neat and clear. Overall, it is a pleasure to read this paper.

  2. The paper provides extensive experiments across multiple models, including crystal LLMs, traditional diffusion/flow-based models, and some smaller general-purpose LLMs.

Critical weaknesses
  1. I feel the contribution of curating existing datasets from the literature and databases is quite limited. Also, the tasks are not novel. As mentioned by the authors, these tasks already exist in the literature, and this benchmark mainly combines them to systematically evaluate the models. I respect the work the authors have done, but I still feel that these contributions are limited.

  2. I also feel that CrystalReAct is not a particularly novel idea, as it almost follows the existing ReAct framework, in my opinion. The main new contribution seems to be the introduction of crystallographic tools. I do not consider this a major contribution of the paper.

  3. The paper focuses too much on specialized crystal LLMs (only evaluating GPT-5.6-Luna and some smaller Qwen models in the appendix). Why not evaluate more general-purpose LLMs? Are they good at these tasks, or do they perform very poorly? And why? I feel that evaluating only GPT-5.6-Luna and small Qwen models is not sufficient.

  4. Another concern I have is why it is necessary to use LLMs for these generation tasks. I think this concern is not limited to this paper, but also applies to this line of research. To be honest, I feel that these generation tasks could also be handled without involving language. This is also supported by the results in Table 3, where the conventional flow/diffusion models perform quite well. I understand that aligning crystal generation with LLMs can enable reasoning and benefit broader design tasks, such as property-conditioned generation. However, such tasks could also be addressed by using an LLM for reasoning and then using a specialized crystal generation model to generate the desired structures. Therefore, I feel that directly involving LLMs in crystal generation is not that necessary.

  5. The provided anonymous repositories were not accessible when I reviewed the paper, so I could not inspect the code or evaluate the reproducibility of the paper. I feel that this is particularly important for a dataset/benchmark paper, as reviewers need to evaluate the quality of the benchmark data. I would like to further evaluate this aspect if the authors can fix the repository access issues during the discussion period.

Questions for authors
  1. Could the authors justify why they do not focus on or evaluate more general-purpose LLMs? If these models do not perform well on these tasks, could the authors explain why?

  2. Could the authors also justify why language needs to be involved in these tasks, instead of directly using conventional machine learning models for crystal generation?

6 comments
AuthorsOpenReview ↗

Response to Reviewer K9hi (1/4)

We sincerely thank you for your appreciation of our presentation. Below, we address the weaknesses and questions you raised point by point.

Weakness 1 -- I feel the contribution of curating existing datasets from the literature and databases is quite limited. Also, the tasks are not novel. As mentioned by the authors, these tasks already exist in the literature, and this benchmark mainly combines them to systematically evaluate the models. I respect the work the authors have done, but I still feel that these contributions are limited.

Response: We appreciate the reviewer's concern regarding the novelty of the benchmark. The motivation for CrystalGPD stems from a broader need in this rapidly developing field: although existing studies demonstrate promising results on individual tasks, the lack of a unified evaluation makes it difficult to compare different approaches across tasks, identify priorities for future research, and assess their suitability for practical materials discovery.

Accordingly, our contribution lies not in introducing new datasets or tasks, but in systematically characterizing the capabilities and limitations of existing crystal LLMs across diverse crystallographic tasks. Rather than simply aggregating existing tasks, CrystalGPD provides a unified evaluation framework to investigate how different generation paradigms perform under diverse crystallographic requirements and where they systematically fail.

Our evaluation reveals several previously under-characterized limitations, including difficulties in physical validity, reference structure recovery, and exact space-group control, particularly for complex structures and rare space groups. Beyond reporting performance, we identify systematic failure patterns, and characterize how these limitations vary with structural complexity. These findings provide empirical insights into the capabilities and failure modes of current crystal LLMs, offering concrete directions for future model development.

Weakness 2 -- I also feel that CrystalReAct is not a particularly novel idea, as it almost follows the existing ReAct framework, in my opinion. The main new contribution seems to be the introduction of crystallographic tools. I do not consider this a major contribution of the paper.

Response: We agree that CrystalReAct builds on the existing ReAct framework and do not claim a new general-purpose agent architecture. Since existing crystal-generation methods have largely focused on non-agentic paradigms, we adapt ReAct to crystal generation to explore an agentic approach and provide a more comprehensive comparison of generation paradigms.

CrystalReAct goes beyond simply adding crystallographic retrieval tools. It integrates knowledge acquisition with crystallographic decision-making and direct structure construction. Specifically, it retrieves reference structures, space-group information, and elemental properties, and uses this knowledge to guide decisions on elemental composition, space groups, Wyckoff positions, and atomic coordinates. Our experiments demonstrate improvements in physical validity and crystallographic constraint satisfaction on the evaluated tasks. Thus, our contribution lies in the domain-specific adaptation and empirical evaluation of agentic crystal generation, rather than a new general-purpose agent framework.

AuthorsOpenReview ↗

Response to Reviewer K9hi (2/4)

Weakness 3 -- The paper focuses too much on specialized crystal LLMs (only evaluating GPT-5.6-Luna and some smaller Qwen models in the appendix). Why not evaluate more general-purpose LLMs? Are they good at these tasks, or do they perform very poorly? And why? I feel that evaluating only GPT-5.6-Luna and small Qwen models is not sufficient.

Response: Our study primarily focuses on specialized crystal LLMs, including CrystaLLM, LLaMat, CrysText, PLaID++, and CrystalReasoner, because our primary objective is to systematically characterize their capabilities and limitations in crystal generation, prediction, and design following domain-specific training or adaptation, rather than to comprehensively benchmark general-purpose LLMs. Nevertheless, we agree that evaluating general-purpose LLMs is important for understanding whether general language and reasoning capabilities can be transferred to crystal-related tasks.

To address the concern about the limited coverage of general-purpose LLMs, we have extended our evaluation to include Qwen3.8-Flash, DeepSeek V4.1 Flash, Opus 5.5, and GPT-6.1-Sol, in addition to GPT-5.6-Luna. For these additional experiments, DNG follows the standard evaluation configuration, while CSP and SG use reduced sampling budgets of 1,000 fixed samples and 100 samples per space group across 14 space groups, respectively.

DNGCSPCSDSG
ModelMSUN RateMP20 MR@1MP52 MR@1MP20 FSGMMP52 FSGMSGM
Qwen3.8-Flash0.073.600.109.201.805.57
DeepSeek v4.1 Flash0.209.200.6013.301.5011.29
Opus 5.50.1122.904.1050.5022.3042.57
GPT-5.6-Luna0.7212.902.3047.6027.8044.57
GPT-6.1-Sol4.3226.106.7069.0047.0048.86

As shown in the results, GPT-6.1-Sol exceeds domain-trained crystal LLMs in the MSUN rate for DNG and in space-group conditioned generation performance, while its crystal structure prediction capability remains limited. GPT-5.6-Luna also demonstrates strong space-group control, whereas models such as DeepSeek v4.1 Flash and Qwen-3.8-Flash obtain lower overall scores.

We distinguish failures in parsing, physical validity, condition satisfaction, and structure recovery. DeepSeek v4.1 Flash and Qwen-3.8-Flash are primarily limited by parsing, with DeepSeek v4.1 Flash additionally exhibiting substantial physical-validity failures. For GPT-5.6-Luna and GPT-6.1-Sol, structure recovery dominates CSP failures even after generating valid, composition-matched structures. GPT-5.6-Luna's CSD failures are concentrated at the parsing and physical-validity stages. The complete results and task-specific analyses for these general-purpose LLMs are provided in Appendix E.6.

AuthorsOpenReview ↗

Response to Reviewer K9hi (3/4)

Weakness 4 -- Another concern I have is why it is necessary to use LLMs for these generation tasks. I think this concern is not limited to this paper, but also applies to this line of research. To be honest, I feel that these generation tasks could also be handled without involving language. This is also supported by the results in Table 3, where the conventional flow/diffusion models perform quite well. I understand that aligning crystal generation with LLMs can enable reasoning and benefit broader design tasks, such as property-conditioned generation. However, such tasks could also be addressed by using an LLM for reasoning and then using a specialized crystal generation model to generate the desired structures. Therefore, I feel that directly involving LLMs in crystal generation is not that necessary.

Response: We argue that incorporating LLMs into crystal generation is valuable for three reasons: (1) offering an alternative paradigm for incorporating crystallographic symmetry; (2) achieving competitive crystal generation performance; and (3) leveraging reasoning capabilities for knowledge-guided structural decisions and integration with specialized models and computational tools.

First, LLMs offer an alternative paradigm for incorporating crystallographic symmetry into crystal generation. SymmCD shows that symmetry-agnostic models such as DiffCSP and FlowMM tend to generate more low-symmetry structures, failing to capture the space-group distribution of real crystals. In contrast, methods such as DiffCSP++ and SGFM explicitly incorporate crystallographic symmetry into generation, but require the target space group and the Wyckoff position of each atom to be specified beforehand. LLMs, on the other hand, can autoregressively model discrete crystallographic decisions, including space-group selection, Wyckoff-position selection, and atomic species specification, providing a flexible way to incorporate crystallographic knowledge into generation.

Second, crystal LLMs have demonstrated competitive generation capabilities. LeMat-GenBench reports that PLaID++ achieves the best SUN performance on DNG, outperforming approaches such as MatterGen. For CSP, we further addressed the input discrepancy in Table 3 by retraining PLaID++ and CrystalReasoner with unit-cell formulas, as detailed in our response to Reviewer cTfw (Weakness 2). The results further show that PLaID++ achieves an MR on par with DiffCSP, while its MR among valid generated structures is comparable to that of OMatG.

Third, LLMs can leverage their reasoning capabilities to determine key structural components, such as space groups, Wyckoff positions, and atomic species, during crystal generation. Crystal generation involves not only producing atomic coordinates but also determining these discrete structural components. Recent studies, including SGEquiDiff and CrysLLMGen, have explored combining autoregressive generation of discrete structural decisions with diffusion-based coordinate generation and refinement. Building on this direction, our agentic paradigm further leverages the reasoning capabilities of LLMs to make and refine structural decisions based on crystallographic knowledge retrieved through external tools. Moreover, this paradigm can be readily extended to incorporate specialized models (e.g., DiffCSP++) and computational tools (e.g., CHGNet and M3GNet) for structure generation and optimization.

Weakness 5 -- The provided anonymous repositories were not accessible when I reviewed the paper, so I could not inspect the code or evaluate the reproducibility of the paper. I feel that this is particularly important for a dataset/benchmark paper, as reviewers need to evaluate the quality of the benchmark data. I would like to further evaluate this aspect if the authors can fix the repository access issues during the discussion period.

Response: We thank the reviewer for pointing out this issue and apologize for the inconvenience it caused during the review process. On October 25, we identified an access issue caused by the permission settings of the anonymous code repository. We have since resolved the issue, and access to the repository has been restored.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all the reviewers for their valuable and constructive feedback. Your comments have helped us improve the consistency, experimental coverage, and presentation of CrystalGPD, and clarify the scope of our findings.

To the best of our efforts, we have provided detailed responses to address the concerns raised by each reviewer in the following. Meanwhile, we have carefully revised the paper according to the reviewers' comments. All feedback has been incorporated into the revised version of the manuscript, and all changes and additions are highlighted in blue for easy reference.

Specifically, the main modifications are as follows.

  • We corrected the CrystalReAct formula-match entry and the CrystaLLM figure, clarified the composition-matching exemption in the validity criterion, and resolved the access issues with the anonymous code repositories to improve consistency and reproducibility.
  • We added controlled experiments by retraining PLaID++ and CrystalReasoner with explicit unit-cell formulas, together with size-stratified analyses, to examine how composition information affects physical validity and structure recovery.
  • We expanded the evaluation to include Qwen3.8-Flash, DeepSeek V4.1 Flash, Opus 5.5, and GPT-6.1-Sol, and provided additional analyses of their performance and failure modes across tasks.
  • We performed structure-level deduplication of MP52 against the MP20 training set and sensitivity analyses across five symprec settings to assess the robustness of the generalization and space-group findings.
  • We added comparisons of GPT-5.6-Luna and CrystalReAct with and without the verifier, and reported their token consumption to examine the benefits and costs of knowledge access and verification.
  • We supplemented the DNG evaluation with raw generation attempts, parsing failure rates, discovery yield per raw attempt, and output-token statistics.

We hope these revisions address your concerns and strengthen the clarity, reliability, and empirical support of the manuscript. As the author-reviewer discussion period draws to a close, please feel free to raise any remaining questions or concerns. We would be happy to provide further clarification. Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.