- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces a multimodal benchmark to evaluate LLMs' ability to reason about crystal structures, symmetry changes, relaxation, and dynamics. Experiments show that current models still struggle with these tasks, even with additional reasoning.
Critical strengths
-
The benchmark goes beyond from static materials structure-related benchmarks into a dynamic region. This is closer to real-life computational materials science studies, and I really like this direction.
-
The benchmark uses a simulation-based pipeline to programmatically generate ground-truth labels, which makes the evaluation more systematic, reproducible, and scalable.
Critical weaknesses
-
One biggest problem I think in this paper is that the using of 'stability' is not so clear and not correctly related to the generally used ones in real computations. And some concepts are not clearly defined, especially 'instability', 'thermal stability', and 'fragility'. These quantities may be related to physical stability, but they are not necessarily equivalent. In real case analysis, we generally use things like 'phonon spectrum', 'free energy', 'Born stability criteria', etc. The paper also lacks mathematical definitions and physical justification for these metrics.
-
I am not fully convinced that some tasks actually evaluate the claimed physical reasoning abilities. For example, random defects or atomic displacements can easily reduce the global space group to P1, but such changes may have limited physical meaning compared to local symmetry changes around defects. Also, the MD simulations seem to be independent from the structural perturbations, making the connection between symmetry changes, relaxation, and subsequent dynamics less clear.
Questions for authors
-
For the stability-related tasks, could the authors provide clearer mathematical definitions and physical justification for the labels? For example, large atomic displacement or slow optimization convergence does not necessarily mean a structure is physically unstable. These quantities also strongly depend on the potential, optimizer, and simulation settings. I think the authors should either provide more evidence supporting their connection to physical stability, or use more precise terms instead of directly calling them 'stability' or 'instability', which could be misleading.
-
I am a bit concerned about the symmetry-related tasks. Random defects or atomic displacements can easily reduce the detected space group to P1, which may not reflect meaningful physical symmetry breaking. In defect studies, local point-group symmetry is often more important, especially for understanding defect states and energy-level splitting, while changes in the global space group caused by random defects or displacements are usually not the main focus. Could the authors provide the distribution of space-group transitions, especially the fraction of structures becoming P1? Also, would local symmetry be more physically meaningful for the defect-related tasks?
4 comments
Response to RRhu
The fraction ending in P1 is 31/218 = 14.22%, whereas the fraction becoming P1 through a non-P1 → P1 transition is 1.93%.
| Initial → final space group | Base | Large | Total | Fraction of total |
|---|---|---|---|---|
| P1 → P1 | 1 | 4 | 5 | 2.29% |
| P1 → non-P1 | 57 | 22 | 79 | 36.24% |
| non-P1 → P1 | 12 | 14 | 26 | 11.93% |
| Same non-P1 group | 11 | 2 | 13 | 5.96% |
| Different non-P1 groups | 61 | 34 | 95 | 43.58% |
| Total | 142 | 76 | 218 | 100.00% |
Separating the two simulation branches reveals an important difference. Percentages in the following table use the respective branch totals as denominators.
| Initial → final space group | Perturbation/relaxation | Temperature MD |
|---|---|---|
| P1 → P1 | 4 | 1 |
| P1 → non-P1 | 79 | 0 |
| non-P1 → P1 | 2 | 24 |
| Same non-P1 group | 13 | 0 |
| Different non-P1 groups | 94 | 1 |
| Total | 192 | 26 |
Thus, the retained task as a whole is not dominated by transitions to P1, and the relaxation branch contains substantial symmetry recovery and transitions between distinct non-P1 groups.
Q2-2: Local symmetry around defects
We agree that local point-group symmetry is more directly relevant to many defect-specific questions. To address this concern, we constructed a 100-question local symmetry-transition supplement, reusing all 40 defect-related structure pairs from the retained task and adding 60 pairs from existing simulations:
| Defect type | Questions |
|---|---|
| Vacancy | 36 |
| Substitution | 29 |
| Interstitial | 35 |
| Total | 100 |
Each question specifies a defect-associated center and a tracked local neighborhood, and asks for its point-group transition before and after relaxation. Labels are recomputed from species-preserving local symmetry operations, rather than inherited from global space groups. Neighborhood definitions, atom correspondence, and positional tolerances are explicitly documented; 14 tolerance-sensitive questions are flagged.
On this supplement, Gemma-4-31B-it achieves 75% without thinking and 88% with native thinking, both at temperature 0, using free generation and the same answer-normalization rules. Output budgets are 2,048 and 32,768 tokens, respectively; Two truncated non-thinking responses are counted incorrectly.
Our benchmark primarily targets global structural changes. This experiment adds local structural reasoning complementary to our benchmark’s global focus.
Response to W1: Ambiguous Stability Terminology and Missing Physical Definitions
As discussed in Q1, our intended physical focus is thermal stability and finite-temperature structural response. Nevertheless, we agree that the current terminology conflates several distinct quantities: observed thermal displacement, perturbation-induced structural change, and simulation convergence or anomaly flags.
We will revise the task names and descriptions, provide explicit mathematical definitions and threshold rules, and explain the physical interpretation and limitations of each label. In particular, we will not equate large displacement or slow optimization convergence with physical instability. These revisions will align the benchmark’s claims with the quantities actually computed.
Response to W2: Physical Meaning of Symmetry Tasks and Their Connection to MD
W2-1
As discussed in Q2, the symmetry task evaluates global crystallographic changes under a specified detection tolerance. Our audit shows that 26 of 218 retained questions involve non-P1 → P1 transitions, with 24 of these belonging to the MD branch. We will report this concentration explicitly and acknowledge that global space-group recognition does not substitute for defect-centered local-symmetry analysis or establish a thermodynamic phase transition.
W2-2
In the current pipeline, structural perturbations followed by relaxation and temperature-driven MD constitute separate simulation branches. The temperature-MD branch starts from the source structure; it does not systematically propagate every defect-modified, relaxed structure into a subsequent MD trajectory.
We will revise the abstract, main text, and pipeline figure accordingly to clearly distinguish the two branches:
Branch 1: Original Structure → Perturbation → Relaxation
Branch 2: Original Structure → Temperature Conditions → MD
I appreciate the authors' clarification of the stability-related terminology and the physical scope of the benchmark. The proposed revisions address my main concerns, and I am happy to maintain my original positive rating.
Thank you very much for your positive feedback and continued support. We sincerely appreciate your constructive comments, which have helped us improve the paper. We are always happy to engage in further discussion and welcome any additional feedback.
Response to RRhu
We sincerely thank the reviewer for their thoughtful and constructive feedback. We appreciate the recognition of the benchmark’s extension beyond static crystal structures toward relaxation and finite-temperature behavior, as well as the value of its simulation-based label-generation pipeline. The reviewer’s comments identify two aspects that require clearer presentation: the physical interpretation of our stability-related terminology and the scope of the symmetry and dynamics tasks. We address the questions first and then respond to each critical weakness.
Response to Q1: Definitions and Physical Interpretation of Stability-Related Tasks
Thank you for highlighting this distinction. We agree that “stability” was used too broadly. Our tasks characterize structural responses and simulation outcomes under a specified protocol, rather than establish thermodynamic, mechanical, or harmonic dynamical stability through free energies, Born criteria, or phonon spectra. We will clarify the terminology and definitions accordingly.
Q1-1: Finite-temperature response.
The MD RMSD task compares the observed endpoint displacement,
$D(T)=\sqrt{\frac{1}{N}\sum_{i=1}^{N} \left|\Delta\mathbf r_i(T)\right|^2},$
where displacements are computed by wrapping fractional-coordinate differences using the initial lattice and preserving atom correspondence. This measures structural displacement over the simulated interval, not a definitive loss of stability. The task currently named “Thermal Stability” instead uses ascending simulation temperature as its ground truth. We will rename it “Temperature Ordering from Trajectory Images”, since it does not independently rank physical stability.
Q1-2: Simulation-defined instability.
This label flags recorded failures or anomalies, including relaxation non-convergence, approaching the optimizer step limit, invalid energies, or excessive MD temperatures, forces, or energy changes:
$y_{\mathrm{flag}} =\mathbf{1}!\left[\bigvee_k C_k(\text{simulation record})\right],$
where Ck denotes a specified failure or threshold condition. These flags can reflect numerical or protocol limitations and do not independently establish physical instability. We will report the exact thresholds and distinguish relaxation-based flags from MD-based flags.
Q1-3: Structural “fragility.”
This task selects the atom-preserving perturbation producing the largest geometric change after relaxation:
$S_p=D_p+\bar{\ell}_0|F_p-I|_F, \qquad p^*=\arg\max_p S_p,$
where Dp is the atomic RMSD from the original structure, ℓˉ0 is its mean lattice-vector length, and Fp is the implemented lattice-deformation matrix. The score combines atomic rearrangement and cell deformation in units of length. It is a heuristic measure of response to the tested perturbations, not an established measure of physical stability or mechanical brittleness. We will describe it as “Perturbation-Induced Structural Change.”
Q1-4: Dependence on simulation settings
We agree that these outcomes depend on the interatomic potential, optimizer, convergence thresholds, temperature, simulation duration, thermostat, and initialization. However, we conducted an additional small-scale calculation to assess the reliability of our results. We randomly sampled 25 complete four-option questions, covering different system sizes, crystal systems, chemical compositions, and perturbation types. We performed 100 DFT single-point calculations using VASP and compared the DFT-computed energies with the corresponding MACE-MP-0 predictions.
The results show that DFT and MACE-MP-0 selected the same lowest-energy option in 23 out of 25 questions, achieving 92.0% agreement.These results provide independent ab initio evidence supporting the reliability of MACE-MP-0-generated energy-based labels.
Response to Q2: Space-Group Transitions, P1 Cases, and Local Symmetry
Thank you for this insightful comment. We agree that a reduction of the detected global space group to P1 does not, by itself, establish a physically significant symmetry-breaking mechanism, particularly for defective structures or instantaneous configurations with atomic displacements.
Q2-1: Distribution of transitions and the fraction becoming P1
We agree that the transition distribution is necessary to assess whether the task is dominated by trivial P1 outcomes. We audited all 218 retained symmetry-transition questions, comprising 142 base and 76 large-subset questions. The following counts are calculated from their saved initial and final space-group labels.