- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors introduce an autoregressive framework that hierarchically generates crystals from discrete symmetry configurations to three-dimensional geometry. The main contribution is an iterative coordinate refinement, achieving very competitive results in crystal generation benchmarks.
Critical strengths
Very promising benchmarking results. Interesting and highly relevant new contributions regarding the iterative refinement.
Critical weaknesses
The discussion regarding what is actually new here should be improved (representation, model, refinement, ...).
The experiments section requires improvement and clarification:
- Most important point: The results of the SV model on LeMat-GenBench should also be included in Table 1. The non-SV version gets a really strong prior with the information on (G, M, N, K). The tuple restricts the possible templates (sometimes only one is possible), which makes the non-SV Version not completely template-free.
- The metrics in Tables 1 and 2 are only explained in the appendix (which is not mentioned in the main text).
- Novelty metrics should also be compared with respect to the training data, not only LeMat-Bulk
- The bold and underline labels in Table 1 are misleading (bold value should be the best in the whole column)
- The comparison in the CSP task is not fair, as some models (including theirs) get the space group as input, which is an advantage, and others don't
- CrysBFN should be included in the CSP, as they have better results than the mentioned models in CSP on MP-20
Questions for authors
Please add a more detailed conceptual comparison to other autoregressive models, e.g. WyFormer and CrystalFormer, to discuss the novel contributions. The approach seems to be quite expensive. Would it be possible to discuss a runtime comparison with other models? What is the performance for CSP and DNG with ancestral sampling (no beam search)? What is the performance for CSP and DNG without the geometric consistency term (lambda = 0 in Eq. 7), and additionally without pruning at d_prune? What are the values for d_min and d_prune, and where do they come from?
6 comments
Further Replies to Reviewer Qgx5
W5 Fair Comparison and Space-Group Consistency in CSP
(1) Clarification of CSP settings We thank the reviewer for raising this important question regarding the fairness of CSP comparisons. Previous symmetry-native generative models either cannot directly perform CSP due to limitations in their probabilistic factorization or rely on external template retrieval methods, such as CSPML, to obtain the required crystallographic information. In the latter case, retrieving a crystal template effectively provides both the space group and the Wyckoff configuration , along with additional structural priors. In contrast, CrystalAR addresses a more challenging template-free, space-group-conditioned CSP setting, where only the chemical composition and space group are provided, while the Wyckoff configuration and crystal geometry must be predicted without external template retrieval. We consider conditioning on a well-motivated setting, as space-group information is a fundamental conditioning variable commonly adopted by symmetry-native generative models. In our original comparison, symmetry-native approaches were highlighted in red to distinguish them from generic crystal generation models.
(2) Fair comparison under unified conditions & space-group consistency evaluation. To provide a more comprehensive and fair evaluation, we have conducted additional experiments in which generic CSP baselines are retrained with space-group conditioning, following the soft-conditioning strategy introduced by MatterGen. This establishes a more unified comparison in which all evaluated models receive the same chemical composition and ground-truth space group, allowing us to directly assess their ability to exploit crystallographic symmetry information. Furthermore, we introduce Space-Group Accuracy (SG Acc.), which measures the proportion of generated structures whose crystallographic space groups agree with those of the ground-truth structures, as identified by symmetry analysis. Related space-group consistency evaluations have also been adopted in previous studies, including CrystaLLM~\cite[1].
Importantly, SG Acc. and the conventional CSP Match Rate evaluate two distinct and complementary aspects of structure prediction. Match Rate measures geometric agreement between predicted and reference structures under predefined tolerances, whereas SG Acc. evaluates whether the predicted structures preserve the target crystallographic symmetry. The relatively permissive tolerances commonly used in CSP benchmarks may identify two geometrically similar structures as matching even when they belong to different space groups (See Page 32 Appendix Figure 10 in the newly revised manuscript). This discrepancy can persist even when relatively generous symmetry-detection tolerances are applied. Such differences are physically meaningful: symmetry breaking can fundamentally alter the allowed physical properties of a material~\cite[2], as exemplified by structural phase transitions associated with the emergence of ferroelectric polarization. Therefore, a high Match Rate alone does not necessarily guarantee the correct recovery of crystallographic symmetry, motivating the inclusion of SG Acc. as an independent evaluation metric.
The updated CSP results demonstrate a distinctive advantage of CrystalAR: our model achieves nearly perfect space-group accuracy through its symmetry-native autoregressive generation mechanism, while maintaining a Match Rate comparable to state-of-the-art CSP models. In contrast, the competing methods do not simultaneously achieve these two properties, even when explicitly conditioned on the same ground-truth space groups.
Composition & SG
| Method | SC-MR, 0.1 Å | SG Acc, 0.1 Å |
|---|---|---|
| CrystalAR | 62.47 | 98.12 |
| DiffCSP-SG-CFG | 54.11 | 63.30 |
| DiffCSP-SG | 53.16 | 62.99 |
| FlowMM-SG | 34.50 | 41.43 |
| CrystalFormer | 27.54 | 87.78 |
| CDVAE-SG | 15.02 | 20.10 |
| WyFormer (no geometry for 7,611/9,046 targets in slot 0) | 7.63 | 15.68 |
| OMatG-SG | 1.35 | 3.55 |
| CrystalFlow-SG | 1.23 | 3.59 |
SG-MR measures the proportion of predicted structures that both match the ground-truth structures under the MR criterion and preserve their ground-truth space-group symmetry.
Composition Only
| Method | SC-MR, 0.1 Å | SG Acc, 0.1 Å |
|---|---|---|
| Uni-3DAR (released checkpoint, rescored) | 52.14 | 59.22 |
| CrystalAR, 1 of 20 by a frozen rule | 51.46 | 65.11 |
| CrystalAR, first candidate | 49.20 | 64.66 |
| DiffCSP | 41.83 | 49.03 |
| FlowMM | 34.57 | 40.50 |
| CDVAE | 10.07 | 13.94 |
| CrystalFlow | 1.37 | 3.53 |
| OMatG | 1.28 | 3.35 |
| CrysBFN | 35.67 | 41.66 |
We have updated CSP section in the manuscript accordingly.
[1] Crystal structure generation with autoregressive large language modeling. Nature Communications 2024
[2] Physical Properties of Crystals: Their Representation by Tensors and Matrices 1957
Further Replies to Reviewer Qgx5
W6 CrysBFN
We have included CrysBFN as an additional baseline in the revised manuscript. Please refer to Table 3 for the corresponding experimental results.
Q1 Conceptual Comparison to Prior AR approaches
We have added an additional page to the revised manuscript (page 9-10) to provide a detailed comparison of our contributions with prior autoregressive approaches. We believe this analysis clarifies the fundamental differences between CrystalAR and existing methods, highlighting its key methodological innovations and advantages. We appreciate the reviewer for raising this important point and will further refine the discussion in the camera-ready version.
Q2 Runtime Comparison
| Method / Setting | Wall Time (s/attempt) | Wall Time (s/generated structure) | Generated / Attempts | Batch Size |
|---|---|---|---|---|
| CrystalAR, Ancestral Sampling (No Search) | 0.069 | 0.069 | 2,500 / 2,500 | 32 |
| CrystalAR, SGBS Q = 64 (Including Ancestral Stage) | 0.404 | 0.768 | 202 / 384 | 32 |
| DiffCSP, 1,000 Timesteps (2,000 Network Evaluations) | 0.679 | 0.679 | 1,500 / 1,500 | 500 |
| SymmCD, 1,000 Steps | 0.618 | 0.619 | 1,497 / 1,500 | 500 |
| CrystalFormer, K = 1, T = 1 | 0.036 | 0.036 | 2,400 / 2,400 | 100 |
As shown in the table, CrystalAR is computationally efficient and does not necessarily incur substantial inference overhead. With pure ancestral sampling, CrystalAR requires only 0.069 seconds per structure, substantially faster than DiffCSP (0.679 s) and SymmCD (0.619 s). Even with our simulation-guided beam search (SGBS, Q = 64), CrystalAR requires only 0.404 seconds per attempt, with an average of 0.768 seconds per successfully generated structure, which remains comparable to diffusion-based approaches. These results demonstrate that CrystalAR offers a flexible trade-off between generation efficiency and structural quality, allowing users to choose between fast ancestral sampling and more extensive geometry-guided search depending on their computational budget.
Q3 CSP and DNG Performance with Ancestral Sampling/ Q4 Without geometric consistency & d_prune
We thank the reviewer for this important question. We have conducted additional experiments to evaluate CrystalAR under pure ancestral sampling without beam search, together with detailed ablations of our search strategy.
As shown below, CrystalAR achieves 54.12% MR@1 with pure ancestral sampling, demonstrating that the autoregressive model itself possesses strong structure prediction capability without relying on search. Incorporating our hierarchical search strategy further improves MR@1 to 62.92%, yielding a substantial 8.80 percentage-point improvement ().
Importantly, our search mechanism goes beyond simply increasing the beam size. When coordinate candidates are ranked solely by autoregressive likelihood (), MR@1 decreases to 60.92%, demonstrating the additional value of geometry-aware ranking. Meanwhile, reducing the coordinate beam size to 1 or 4 produces relatively small performance differences, suggesting that the improvement is not merely attributable to larger coordinate beams.
These results demonstrate that hierarchical search and geometry-aware candidate selection provide complementary benefits to our autoregressive modeling, enabling more effective exploration of plausible crystal configurations rather than relying solely on ancestral sampling.
| Setting | MR@1 (%) | Δ vs. Default | p-value |
|---|---|---|---|
| Default (Ours) | 62.92 | — | — |
| Ancestral sampling (no search) | 54.12 | −8.80 | 7.2e-127 |
| Likelihood-only ranking () | 60.92 | −2.00 | 4.1e-23 |
| Without 0.5 Å contact mask | 62.96 | +0.03 | 0.508 |
| Coordinate beam size = 1 | 62.61 | −0.31 | 0.010 |
| Coordinate beam size = 4 | 62.79 | −0.13 | 0.201 |
| Arm | MR@1 % | Δ vs default | 95% CI (paired) | p |
|---|---|---|---|---|
| Default | 62.92 | — | — | — |
| Ancestral (no search anywhere) | 54.12 | −8.80 | — | 7.2e-127 |
| λ = 0 (coordinates ranked by log p only) | 60.92 | −2.00 | — | 4.1e-23 |
λ = 0 and no d_prune (mask and cap removed) | 59.40 | −3.53 | [−3.98, −3.07] | 1.9e-56 |
No d_prune mask only (cap kept) | 62.96 | +0.03 | — | 0.508 |
No clearance cap only (d_min counterpart removed) | 62.68 | −0.24 | [−0.53, +0.05] | 0.117 |
| Coordinate beam 1 | 62.61 | −0.31 | — | 0.010 |
| Coordinate beam 4 | 62.79 | −0.13 | — | 0.201 |
| Sampling strategy | Valid (%) | Unique (%) | Novel (%) | E_f (eV/atom) | E_hull (eV/atom) | RMSD (Å) | Stable (% (n)) | S.U.N. (% (n)) | Meta (% (n)) | M.S.U.N. (% (n)) |
|---|---|---|---|---|---|---|---|---|---|---|
| Ancestral sampling (no search) | 64.40 | 62.24 | 18.76 | −0.45 ± 1.39 | 0.46 ± 1.14 | 0.38 ± 0.68 | 5.44 (136) | 0.20 (5) | 36.68 (917) | 2.12 (53) |
| SGBS (default) | 91.44 | 89.08 | 42.76 | −0.23 ± 1.27 | 0.71 ± 1.08 | 0.77 ± 1.25 | 6.00 (150) | 0.28 (7) | 39.24 (981) | 2.72 (68) |
Replying to Reviewer Qgx5
Thank you for your positive feedback and constructive comments. We believe that all the concerns raised can be fully addressed. Please find our detailed responses below:
W1 SV-variant Model Issue
Note that the results in this table are obtained using ancestral sampling without any search.
To investigate whether our non-SV variant benefits from excessive size priors, we conducted an additional experiment in which CrystalAR is conditioned only on the number of symmetry-inequivalent sites (M), without explicitly specifying the other size variables. As shown above, the results remain highly consistent with our original setting across validity (64.08% vs. 64.40%), uniqueness (62.60% vs. 62.24%), novelty (19.16% vs. 18.76%), and stability (4.88% vs. 5.44%). This demonstrates that the additional size priors are not essential to CrystalAR's generation performance.
Importantly, conditioning on the number of sites is conceptually analogous to conditioning on the total atom count (N), as commonly practiced in generic crystal generative models. In symmetry-native generation, (M) specifies only the number of symmetry-inequivalent sites, not their Wyckoff assignments, elemental identities, or geometric configurations, all of which must still be generated by the model. Therefore, providing (M) does not amount to supplying a crystallographic template, and CrystalAR remains template-free in the sense that no specific Wyckoff configuration is given or retrieved.
Overall, these additional results alleviate the concern that CrystalAR's performance relies heavily on informative size priors and further support the effectiveness of our symmetry-native generative formulation.
W2: Clarification of Evaluation Metrics
Thank you for pointing out these issues. Due to space constraints, we originally provided detailed explanations of the evaluation metrics in the appendix. To improve clarity and readability, we have revised the manuscript to include an additional paragraph introducing the key evaluation metrics in the main text. We have also expanded the table captions to clarify the relevant metric definitions, making the results easier to interpret without referring to the appendix.
W3 Novelty Evaluation Against the Training Data
The results reveal several important observations. First, the relative trends in structural novelty remain consistent across the two reference datasets, indicating that our original evaluation against LeMat-Bulk provides a meaningful assessment of structural novelty. Second, while some baseline methods achieve higher raw novelty percentages, higher novelty does not necessarily translate into better stable material discovery. Notably, CrystalAR achieves the highest S.U.N. discovery rate against the MP-20 training set (32 structures, 1.28%), outperforming DiffCSP (25, 1.00%) and FlowMM (31, 1.24%), despite their higher raw novelty rates. This highlights the importance of jointly considering stability, uniqueness, and novelty rather than optimizing structural novelty in isolation.
Importantly, CrystalAR does not aim to maximize novelty at the expense of physical plausibility. Instead, our symmetry-native formulation seeks to generate crystallographically consistent structures while maintaining competitive discovery performance. The additional results demonstrate that CrystalAR achieves a favorable balance between structural exploration and stable material discovery, rather than merely producing a larger fraction of structures distant from the reference dataset. Note that here reported S.U.N. results are evaluated with respect to the MP-20 training set, which serves as the reference dataset for assessing structural novelty.