- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents CrystalReasoner, an LLM-based crystal structure generation framework combining explicit reasoning traces with GRPO reinforcement learning. The method incorporates structure-aware textual descriptions and physical property supervision for condition-guided generation and is evaluated on MP-20 for validity, stability, and condition consistency.
Critical strengths
-
Explicit thinking traces with crystallographic information provide a promising direction for enabling LLMs to incorporate structural knowledge during crystal generation.
-
The method combines thinking-trace supervision with reinforcement learning and extends to generation conditioned on different physical properties. Comparisons among Base, Thinking, RL, and the full model provide insights into the contributions of reasoning traces and GRPO.
Critical weaknesses
The evaluation protocol is not presented clearly enough to interpret performance across generation settings or identify the sources of improvement. Section 4.1 refers to “textual specifications” (Line 259) and evaluates 1,024 test compositions (Lines 292-295), but does not explicitly describe the evaluated input conditions. Lines 962-968 describe prompts that always provide the formula and select additional conditions from formation energy, band gap, energy above the hull, and space group, with 20% containing only the formula. Together with Lines 989-992, this suggests that Table 1 uses these mixed conditions. However, the actual distribution of evaluation conditions and results for each setting are not reported, preventing separate assessment of formula-only, space-group-conditioned, and energy/property-conditioned generation. Moreover, the interpretation of Chemical Validity, Novelty, Uniqueness, S.U.N., and S.S.U.N. under formula-conditioned evaluation requires clarification. Chemical Validity is largely determined by the supplied composition when the model follows the formula. Generating one structure per composition per seed and computing Uniqueness only within reduced-formula groups (Lines 1159-1169) does not adequately measure structural diversity for a given composition. Novelty also depends substantially on the overlap between test and training compositions. Consequently, it remains unclear how much S.U.N. and S.S.U.N. reflect actual structure exploration rather than the supplied compositions and evaluation protocol.
Questions for authors
- What specifically constitutes the “atomic precision” problem? Lines 14-15 and 198-200 identify insufficient low-level atomic precision as a key motivation. Which existing methods exhibit this limitation, and does it concern coordinate errors, unreasonable interatomic distances, symmetry inconsistencies, or other issues? Could the authors provide direct quantitative evidence and clarify how reasoning traces and GRPO individually improve atomic precision?
- What information is encoded in the thinking traces, and how does it constrain generation? Lines 161-184 state that the traces contain Wyckoff letters and multiplicities, whereas the case study in Appendix D (Lines 886-889) does not appear to include Wyckoff position information. Please reconcile these descriptions and explain how the traces implement the “progressively shrinking” crystallographic solution space described in Section 3.2.
- Could results be reported separately by input setting? Is the condition selection strategy in Lines 962-968 used in Table 1? Please separate formula-only, formula with space-group conditions, and formula with additional property conditions, and clarify the interpretation of Chemical Validity, Novelty, Uniqueness, S.U.N., and S.S.U.N. in these settings.
- Why is PLAID++’s DPO stage omitted? Table 1 uses only its Wyckoff representation without DPO (Lines 218-219), whereas the proposed method includes RL alignment. Could the authors include complete PLAID++ or clarify which conclusions about superiority over existing methods the current comparison supports?
- How were the data filtered, and how representative is the evaluation subset? The standard MP-20 train/validation/test split contains 27,136 / 9,047 / 9,046 structures, but the paper reports 24,231 training and 8,141 test structures (Lines 704-707), then selects 1,024 test compositions (Lines 755-759). Please explain both reductions and the resulting subset’s representativeness. Evaluation on the full test set would strengthen the reliability of the reported improvements.
4 comments
Author Rebuttal to Reviewer jDwD (Part 2)
W & Q3. Formula-only, space-group-conditioned, and property-conditioned results, and interpretation of validity, novelty, and diversity.
We evaluate the models on the full CDVAE test set now. The full evaluation uses the mixed-condition prompt protocol, with the following fixed population shared by all models:
| Input setting | Prompts |
|---|---|
| Formula only | 1,660 |
| Formula + space group | 416 |
| Formula + properties | 2,408 |
| Formula + space group + properties | 3,657 |
| Total | 8,141 |
Each prompt has five candidates. Table 1 evaluates structural validity, chemical validity, and composition consistency on the 1,660 formula-only prompts; space-group consistency and S.S.U.N. on the 4,073 prompts explicitly requesting a space group; and stability, energy, uniqueness, and S.U.N. on all 8,141 prompts. Condition-specific results are also available. Failed attempts remain in rate denominators; mean energy uses finite hull distances.
We agree that Chemical Validity is a composition-based check and is largely determined by the supplied formula when the model follows it. Every evaluation prompt specifies a formula, so we interpret this metric together with Composition Consistency and Structural Validity. Novelty also depends on the reference set: 88.5% of test compositions are not in the training set. We therefore repeat novelty matching against the larger MP+Alexandria reference set. The table reports the resulting rates on the same 8,141 prompts.
| Reference set | Metric (%) | Wyckoff+DPO | Wyckoff+GRPO | CrysReas |
|---|---|---|---|---|
| SFT training structures | Novelty | 78.50 | 81.88 | 82.32 |
| SFT training structures | S.U.N. | 3.33 | 3.49 | 4.58 |
| MP+Alexandria | Novelty | 45.14 | 46.20 | 41.38 |
| MP+Alexandria | S.U.N. | 0.14 | 0.16 | 0.21 |
The larger reference set substantially lowers novelty and S.U.N. for every model. CrysReas has lower standalone novelty against MP+Alexandria than either Wyckoff baseline, while retaining the highest S.U.N. under that reference. This means RL of CrysReas is able to improve structure quality (hence improve S.U.N.), but since the reward does not contain novelty, the novelty over MP+Alexandria is the lowest.
We report the uniqueness over the same formula. We select 256 distinct formula-only prompts from the full test set and generate 32 outputs per prompt. We find this number is sensitive to sample count, indicating a reasonable evaluation requires more prompts, making it hard to evaluate for multiple conditions. It is also unfair for us to compare our uniqueness (always condition on formula) and prior works' uniqueness (unconditional), hence we retain uniqueness in Table 1 as the uniqueness over 8,141 test prompts, averaged by 5 seeds.
| Outputs per prompt | Mean within-formula Uniqueness (%) |
|---|---|
| 1 | 100.00 |
| 8 | 53.20 |
| 16 | 44.99 |
| 32 | 38.07 |
Q4. PLAID++'s DPO stage and comparison with post-trained Wyckoff baselines.
We include Wyckoff+DPO, incorporating PLAID++-style offline preference optimization, and Wyckoff+GRPO. Both start from the same Wyckoff SFT checkpoint, use the same reward definition and 8,000 training prompts, and generate eight candidates per prompt. DPO uses these candidates to construct offline preference pairs, while GRPO samples candidates online during training. All models are evaluated on the same 8,141 test prompts with five candidates per prompt.
| Metric | Wyckoff+DPO | Wyckoff+GRPO | CrysReas |
|---|---|---|---|
| Structural validity (%) | 90.40 | 94.77 | 95.58 |
| Space-group consistency (%) | 61.55 | 62.70 | 68.51 |
| Stability (%) | 3.66 | 3.92 | 5.13 |
| Mean hull distance (eV/atom) | 0.527 | 0.480 | 0.434 |
| S.U.N. (%) | 3.33 | 3.49 | 4.58 |
CrysReas retains an advantage over both post-trained Wyckoff baselines on these metrics. Thus, the comparison remains favorable after including the preference-optimization stage omitted from the original evaluation, and when the Wyckoff generator also receives GRPO. Within the shared Wyckoff representation and SFT initialization, GRPO also achieves higher performance than DPO on these metrics.
Author Rebuttal to Reviewer jDwD (Part 3)
Q5. MP-20 filtering, the 1,024-prompt subset, and evaluation on the full test set.
The difference in dataset size comes from two filtering steps: we remove MP-20 entries unavailable in the Materials Project data used for this study, then exclude structures containing elements outside MatterGen's fixed 76-element vocabulary. The following counts account for the reductions in all three splits:
| Split | Original MP-20 | Unavailable MP entries | Excluded by element vocabulary | Retained |
|---|---|---|---|---|
| Train | 27,136 | 1,098 | 1,807 | 24,231 |
| Validation | 9,047 | 370 | 582 | 8,095 |
| Test | 9,046 | 345 | 560 | 8,141 |
Following the reviewer's suggestion, the main evaluation now covers all 8,141 retained test prompts, with five candidates per prompt, compared with the earlier 1,024-prompt subset. Every model is evaluated on the same prompts and candidate count. The updated main results therefore cover the entire filtered test set, removing dependence on the representativeness of the earlier subset.
Dear Reviewer jDwD, We posted a response to your concerns on Nov 15, and we wanted to check whether our clarifications address the issues you raised. If any point remains unresolved, we are happy to provide further evidence. Thank you.
Author Rebuttal to Reviewer jDwD
We thank the reviewer for recognizing the value of crystallographic thinking traces and the CrysReas-Base/CrysReas-Thinking/CrysReas-RL/CrysReas ablations. The additional experiments address the missing DPO baseline, evaluation by input setting, structural diversity for a fixed prompt, and coverage of the full test set. We also clarify the trace contents, the meaning of "atomic precision" and the filtering from the original MP-20 splits.
We agree that the term "atomic precision" was insufficiently specific. Our intended meaning is the ability to generate coordinates that yield physically plausible structures and satisfy the requested constraints, including reasonable interatomic distances and consistent symmetry. The number of decimal places follows the training representation; the quality of the resulting structures is assessed through structural validity, symmetry consistency, energy, and property-target satisfaction. We therefore correct it to "structural validity and target-property satisfaction" to state the objective in terms of the quantities directly evaluated.
We agree that the construction program computes Wyckoff letters for grouping and ordering sites, but does not explicitly print those letters in the trace. We correct the wording to match these actual outputs in the revision.
Three parts of thinking traces describe symmetries, local coordinations, and physical properties. We add an experiment to prove that they progressively shrink the crystal solution space. We select 32 distinct formula-only prompts from the held-out generation split, testing the generated crystal when the thinking traces are truncated at different positions. The observed cluster count decreases at every prefix stage, with the largest drop after the symmetry and site-count segment.