acceptodds
Under review as a conference paper at ICLR 2027

CrystalBench Dynamics: A Multimodal Benchmark for dynamic reasoning in crystal systems

Abstract

Scientific reasoning over crystals requires connecting structural perturbations to changes in symmetry, relaxation response, and finite-temperature evolution. Existing materials-science benchmarks evaluate domain knowledge, property prediction, multimodal interpretation, or structure manipulation, but rarely connect controlled perturbations with their simulated physical consequences. We introduce CrystalBench Dynamics, a simulation-grounded benchmark containing 4,016 questions across 18 text and multimodal tasks derived from 1,500 real crystal structures. Our generation pipeline applies five structural perturbations, including lattice strain, vacancy creation, atomic substitution, interstitial insertion, and atomic displacement, followed by geometry relaxation and finite-temperature molecular dynamics. The resulting tasks are organized into four capability families: Perception, Symmetry Reasoning, Symmetry–Response Reasoning, and Dynamics Prediction. All reference answers are computed programmatically from crystal structures, optimization records, and simulation trajectories rather than generated by language models. Across contemporary language and vision-language models, the strongest system achieves only 51.96% task-macro accuracy on the Full Track, remaining more than 20 percentage points below the expert performance. Enabling thinking raises Gemma-4-31B-IT from 31.26% to 43.04%, demonstrating the value of inference-time reasoning while leaving substantial headroom. These results show that recognizing crystal structures does not imply the ability to predict how their symmetry and physical state evolve under perturbation.

Primary area:
datasets and benchmarks
Keywords:
Crystal structures, materials science, multimodal reasoning, foundation models, atomistic simulation, molecular dynamics, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RRhuOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a multimodal benchmark to evaluate LLMs' ability to reason about crystal structures, symmetry changes, relaxation, and dynamics. Experiments show that current models still struggle with these tasks, even with additional reasoning.

Critical strengths
  1. The benchmark goes beyond from static materials structure-related benchmarks into a dynamic region. This is closer to real-life computational materials science studies, and I really like this direction.

  2. The benchmark uses a simulation-based pipeline to programmatically generate ground-truth labels, which makes the evaluation more systematic, reproducible, and scalable.

Critical weaknesses
  1. One biggest problem I think in this paper is that the using of 'stability' is not so clear and not correctly related to the generally used ones in real computations. And some concepts are not clearly defined, especially 'instability', 'thermal stability', and 'fragility'. These quantities may be related to physical stability, but they are not necessarily equivalent. In real case analysis, we generally use things like 'phonon spectrum', 'free energy', 'Born stability criteria', etc. The paper also lacks mathematical definitions and physical justification for these metrics.

  2. I am not fully convinced that some tasks actually evaluate the claimed physical reasoning abilities. For example, random defects or atomic displacements can easily reduce the global space group to P1, but such changes may have limited physical meaning compared to local symmetry changes around defects. Also, the MD simulations seem to be independent from the structural perturbations, making the connection between symmetry changes, relaxation, and subsequent dynamics less clear.

Questions for authors
  1. For the stability-related tasks, could the authors provide clearer mathematical definitions and physical justification for the labels? For example, large atomic displacement or slow optimization convergence does not necessarily mean a structure is physically unstable. These quantities also strongly depend on the potential, optimizer, and simulation settings. I think the authors should either provide more evidence supporting their connection to physical stability, or use more precise terms instead of directly calling them 'stability' or 'instability', which could be misleading.

  2. I am a bit concerned about the symmetry-related tasks. Random defects or atomic displacements can easily reduce the detected space group to P1, which may not reflect meaningful physical symmetry breaking. In defect studies, local point-group symmetry is often more important, especially for understanding defect states and energy-level splitting, while changes in the global space group caused by random defects or displacements are usually not the main focus. Could the authors provide the distribution of space-group transitions, especially the fraction of structures becoming P1? Also, would local symmetry be more physically meaningful for the defect-related tasks?

4 comments
AuthorsOpenReview ↗

Response to RRhu

We sincerely thank the reviewer for their thoughtful and constructive feedback. We appreciate the recognition of the benchmark’s extension beyond static crystal structures toward relaxation and finite-temperature behavior, as well as the value of its simulation-based label-generation pipeline. The reviewer’s comments identify two aspects that require clearer presentation: the physical interpretation of our stability-related terminology and the scope of the symmetry and dynamics tasks. We address the questions first and then respond to each critical weakness.

Response to Q1: Definitions and Physical Interpretation of Stability-Related Tasks

Thank you for highlighting this distinction. We agree that “stability” was used too broadly. Our tasks characterize structural responses and simulation outcomes under a specified protocol, rather than establish thermodynamic, mechanical, or harmonic dynamical stability through free energies, Born criteria, or phonon spectra. We will clarify the terminology and definitions accordingly.

Q1-1: Finite-temperature response.

The MD RMSD task compares the observed endpoint displacement,

$D(T)=\sqrt{\frac{1}{N}\sum_{i=1}^{N} \left|\Delta\mathbf r_i(T)\right|^2},$

where displacements are computed by wrapping fractional-coordinate differences using the initial lattice and preserving atom correspondence. This measures structural displacement over the simulated interval, not a definitive loss of stability. The task currently named “Thermal Stability” instead uses ascending simulation temperature as its ground truth. We will rename it “Temperature Ordering from Trajectory Images”, since it does not independently rank physical stability.

Q1-2: Simulation-defined instability.

This label flags recorded failures or anomalies, including relaxation non-convergence, approaching the optimizer step limit, invalid energies, or excessive MD temperatures, forces, or energy changes:

$y_{\mathrm{flag}} =\mathbf{1}!\left[\bigvee_k C_k(\text{simulation record})\right],$

where CkC_k denotes a specified failure or threshold condition. These flags can reflect numerical or protocol limitations and do not independently establish physical instability. We will report the exact thresholds and distinguish relaxation-based flags from MD-based flags.

Q1-3: Structural “fragility.”

This task selects the atom-preserving perturbation producing the largest geometric change after relaxation:

$S_p=D_p+\bar{\ell}_0|F_p-I|_F, \qquad p^*=\arg\max_p S_p,$

where DpD_p is the atomic RMSD from the original structure, ℓˉ0\bar{\ell}_0 is its mean lattice-vector length, and FpF_p is the implemented lattice-deformation matrix. The score combines atomic rearrangement and cell deformation in units of length. It is a heuristic measure of response to the tested perturbations, not an established measure of physical stability or mechanical brittleness. We will describe it as “Perturbation-Induced Structural Change.”

Q1-4: Dependence on simulation settings

We agree that these outcomes depend on the interatomic potential, optimizer, convergence thresholds, temperature, simulation duration, thermostat, and initialization. However, we conducted an additional small-scale calculation to assess the reliability of our results. We randomly sampled 25 complete four-option questions, covering different system sizes, crystal systems, chemical compositions, and perturbation types. We performed 100 DFT single-point calculations using VASP and compared the DFT-computed energies with the corresponding MACE-MP-0 predictions.

The results show that DFT and MACE-MP-0 selected the same lowest-energy option in 23 out of 25 questions, achieving 92.0% agreement.These results provide independent ab initio evidence supporting the reliability of MACE-MP-0-generated energy-based labels.


Response to Q2: Space-Group Transitions, P1 Cases, and Local Symmetry

Thank you for this insightful comment. We agree that a reduction of the detected global space group to P1 does not, by itself, establish a physically significant symmetry-breaking mechanism, particularly for defective structures or instantaneous configurations with atomic displacements.

Q2-1: Distribution of transitions and the fraction becoming P1

We agree that the transition distribution is necessary to assess whether the task is dominated by trivial P1 outcomes. We audited all 218 retained symmetry-transition questions, comprising 142 base and 76 large-subset questions. The following counts are calculated from their saved initial and final space-group labels.

AuthorsOpenReview ↗

Response to RRhu

The fraction ending in P1 is 31/218 = 14.22%, whereas the fraction becoming P1 through a non-P1 → P1 transition is 1.93%.

Initial → final space groupBase n=142n = 142Large n=76n = 76Total n=218n = 218Fraction of total
P1 → P11452.29%
P1 → non-P157227936.24%
non-P1 → P112142611.93%
Same non-P1 group112135.96%
Different non-P1 groups61349543.58%
Total14276218100.00%

Separating the two simulation branches reveals an important difference. Percentages in the following table use the respective branch totals as denominators.

Initial → final space groupPerturbation/relaxation n=192n = 192Temperature MD n=26n = 26
P1 → P14 20˙82\.08%1 38˙53\.85%
P1 → non-P179 411˙541\.15%0 00˙00\.00%
non-P1 → P12 10˙41\.04%24 923˙192\.31%
Same non-P1 group13 67˙76\.77%0 00˙00\.00%
Different non-P1 groups94 489˙648\.96%1 38˙53\.85%
Total192 1000˙0100\.00%26 1000˙0100\.00%

Thus, the retained task as a whole is not dominated by transitions to P1, and the relaxation branch contains substantial symmetry recovery and transitions between distinct non-P1 groups.

Q2-2: Local symmetry around defects

We agree that local point-group symmetry is more directly relevant to many defect-specific questions. To address this concern, we constructed a 100-question local symmetry-transition supplement, reusing all 40 defect-related structure pairs from the retained task and adding 60 pairs from existing simulations:

Defect typeQuestions
Vacancy36
Substitution29
Interstitial35
Total100

Each question specifies a defect-associated center and a tracked local neighborhood, and asks for its point-group transition before and after relaxation. Labels are recomputed from species-preserving local symmetry operations, rather than inherited from global space groups. Neighborhood definitions, atom correspondence, and positional tolerances are explicitly documented; 14 tolerance-sensitive questions are flagged.

On this supplement, Gemma-4-31B-it achieves 75% without thinking and 88% with native thinking, both at temperature 0, using free generation and the same answer-normalization rules. Output budgets are 2,048 and 32,768 tokens, respectively; Two truncated non-thinking responses are counted incorrectly.

Our benchmark primarily targets global structural changes. This experiment adds local structural reasoning complementary to our benchmark’s global focus.

Response to W1: Ambiguous Stability Terminology and Missing Physical Definitions

As discussed in Q1, our intended physical focus is thermal stability and finite-temperature structural response. Nevertheless, we agree that the current terminology conflates several distinct quantities: observed thermal displacement, perturbation-induced structural change, and simulation convergence or anomaly flags.

We will revise the task names and descriptions, provide explicit mathematical definitions and threshold rules, and explain the physical interpretation and limitations of each label. In particular, we will not equate large displacement or slow optimization convergence with physical instability. These revisions will align the benchmark’s claims with the quantities actually computed.

Response to W2: Physical Meaning of Symmetry Tasks and Their Connection to MD

W2-1

As discussed in Q2, the symmetry task evaluates global crystallographic changes under a specified detection tolerance. Our audit shows that 26 of 218 retained questions involve non-P1 → P1 transitions, with 24 of these belonging to the MD branch. We will report this concentration explicitly and acknowledge that global space-group recognition does not substitute for defect-centered local-symmetry analysis or establish a thermodynamic phase transition.

W2-2

In the current pipeline, structural perturbations followed by relaxation and temperature-driven MD constitute separate simulation branches. The temperature-MD branch starts from the source structure; it does not systematically propagate every defect-modified, relaxed structure into a subsequent MD trajectory.

We will revise the abstract, main text, and pipeline figure accordingly to clearly distinguish the two branches:

Branch 1: Original Structure → Perturbation → Relaxation

Branch 2: Original Structure → Temperature Conditions → MD

I appreciate the authors' clarification of the stability-related terminology and the physical scope of the benchmark. The proposed revisions address my main concerns, and I am happy to maintain my original positive rating.

Thank you very much for your positive feedback and continued support. We sincerely appreciate your constructive comments, which have helped us improve the paper. We are always happy to engage in further discussion and welcome any additional feedback.

Reviewer_QR9TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CrystalBench Dynamics, a text-only and multimodal benchmark for reasoning about perturbed crystal structures and their subsequent physical responses. Starting from 1,500 structures sampled from Materials Project, the authors apply five types of structural perturbation and use MACE-based geometry relaxation and finite-temperature molecular dynamics to derive simulation-grounded labels. The resulting benchmark contains 4,016 multiple-choice questions across 18 tasks, organized into four capability families: Perception, Symmetry Reasoning, Symmetry-Response Reasoning, and Dynamics Prediction.

Critical strengths

The main strength of the paper is its attempt to connect controlled structural perturbations with relaxation and finite-temperature responses within a unified benchmark. This goes beyond conventional materials-science question answering, static property prediction, or image interpretation. The pipeline produces labels related to symmetry, energy, displacement, convergence, and thermal response, providing a potentially useful resource for studying whether models can reason about the consequences of structural changes.

Critical weaknesses

The validity of the task design and benchmark-generation pipeline is not sufficiently established. Although the benchmark contains 18 tasks, the paper provides only one brief case study and four representative examples, while the complete question set is not available for inspection. The question-construction procedure is described only at a high level as validity checking, ambiguity filtering, and answer balancing. It is therefore difficult to assess whether the model-visible inputs contain sufficient information to determine the answers, whether the distractors are scientifically meaningful, and how ambiguous questions are identified and removed. In addition, all simulation-dependent labels are generated using MACE-MP-0 medium, but the paper does not verify whether other representative machine-learned interatomic potentials would produce the same task labels and hence the same correct options. Without a cross-MLIP consistency analysis, it remains unclear whether these tasks capture robust physical responses or outcomes specific to MACE-MP-0.

Questions for authors
  1. Could the authors provide sufficient task-level details to evaluate the design and validity of all 18 tasks? Table 2 lists the 18 tasks in Lines 163–215, whereas Appendix D provides only four representative examples in Lines 754–858, and the two CIF-based examples omit the actual CIF inputs. Moreover, the question-construction procedure is described only in broad terms as validity checking, ambiguity filtering, and answer balancing in Lines 297–301. Please release the full benchmark through an anonymous link or provide several complete examples for each task, including all model-visible inputs, answer options, the ground-truth construction rule, distractor construction, filtering criteria, and the number of questions removed at each stage.
  2. How are perturbation intensities defined for small unit cells? The benchmark contains structures with as few as three atomic sites (Lines 248–255), but site-based perturbations use n=max⁡(1,round⁡(cN))n=\max(1,\operatorname{round}(cN)) for nominal fractions of 5%, 10%, and 15% (Lines 662–670). In small cells, these levels can collapse to the same operation count or produce realized fractions far above the nominal values. Please report the realized perturbation-fraction distribution and the proportion of structures with collapsed intensity levels. Please also clarify whether the structures are primitive, conventional, or supercells, how representation dependence is controlled, and how many samples are generated and retained for each operation and intensity.
  3. What is the exact relationship between the perturbed structures and the MD simulations? The abstract and main text state that perturbed structures undergo both relaxation and finite-temperature MD in Lines 20–22 and 78–79, whereas Lines 679–682 state that temperature-only MD starts from the original Materials Project structure. Please clarify whether MD is performed on the perturbed structures or separately on the original structures, and revise the abstract, main text, and figures to present a consistent data-generation process.
  4. How sensitive are the benchmark labels to the simulation settings? Lines 266–286 and 679–682 show that all labels derived from geometry relaxation and molecular dynamics are generated using a single fixed simulation setup based on MACE-MP-0 medium. The paper also acknowledges that the benchmark is limited by its simulation conditions and machine-learned potential in Lines 465–470. Please rerun the benchmark-generation simulations with several representative MLIPs and, for each task, report the agreement between the labels obtained under each alternative MLIP and those obtained under MACE-MP-0. In particular, please report the proportion of questions for which changing the MLIP changes the correct answer option, and explain how such MLIP-sensitive questions are handled. Without this validation, it is unclear whether the benchmark captures robust physical responses or results specific to the selected MLIP and simulation setup.
  5. Why is the expert evaluation restricted to a model-conditioned subset of only 90 questions? Lines 719–726 state that the subset was selected so that GPT-5.5 achieved 42.22% accuracy, within a predefined range of 26%–45%. Please explain why expert performance was not evaluated on the full 4,016-question benchmark, or on a model-independent stratified sample, and discuss how this selection affects whether the reported 73.33% is representative of human performance on the benchmark. Nevertheless, Lines 417–420 compare the experts’ question-level accuracy on this subset with GPT-5.5’s task-macro accuracy on the full benchmark, which does not provide a direct estimate of the overall human–model gap. Moreover, Lines 743–749 state that three experts participated but report only a single score of 66/90. Please clarify what this score represents and report the accuracy of each expert and their inter-expert agreement, since the current presentation provides no information about performance variation across the three experts.
  6. Could the authors provide the evaluation details and per-task results promised in the main text? Lines 333–337 state that the Appendix documents each model’s context limit, output budget, and decoding parameters, while Lines 346–351 state that it provides per-task correct counts and accuracies for all evaluated models. These materials are absent from the submitted version. Please provide the complete model-specific inference settings and the per-task correct counts and accuracies for all evaluated models.
9 comments

As the author–reviewer discussion period draws to a close, the authors have yet to respond to my concerns. Given the substantial weaknesses identified in my review, which remain unaddressed, I am lowering my score from 2 to 1.

AuthorsOpenReview ↗

Response to QR9T

We sincerely thank the reviewer for their thoughtful and constructive feedback. We appreciate the recognition of CrystalBench Dynamics' contribution to connecting controlled structural perturbations with geometry relaxation and finite-temperature responses within a unified benchmark, moving beyond conventional static materials-science evaluations. We are also grateful for the acknowledgment of its simulation-grounded labels and potential value in evaluating models' ability to reason about structural changes and their physical consequences. We carefully address the reviewer's concerns below.

Response to Q1 : Task Design and Data Transparency

Thank you for this suggestion. To improve transparency, we have prepared supplementary documentation covering all 18 tasks:

Anonymous task documentation and complete examples

Q1-1: Complete task examples. We provide 54 complete examples (three per task), including model-visible prompts, answer options, CIF files or images, and separate ground-truth annotations. Each task is accompanied by its ground-truth construction rules, distractor generation methods, and filtering criteria.

Q1-2: Validity and ambiguity filtering. We document structural validity checks, including minimum interatomic distance 07˙5A˚0\.75 Å, positive cell volume, and lattice condition number ≤100≤100, together with task-specific eligibility and ambiguity filtering rules. We also clarify that answer balancing adjusts option positions without changing question content.

Q1-3: Question counts. We report per-task question counts, totaling 2,861 base and 1,155 large questions (4,016 overall). However, complete historical records of exclusions at each filtering stage are unavailable, which we explicitly acknowledge.

The supplement addresses the request for complete task examples and construction details, although it does not constitute a release of the full benchmark.


Response to Q2 : Perturbation Intensity and Small-Cell Effects

Thank you for raising this important concern. We agree that integer rounding can cause nominal perturbation levels to collapse in small cells and can produce realized fractions substantially above the nominal targets. We have audited the stored perturbation records and the structures referenced by the final benchmark to quantify this effect.

Q2-1 Intensity definition and realized fractions. For substitution, vacancy, interstitial insertion, and atomic displacement, the implementation uses k = max(1, round(p × N)), where p ∈ {0.05, 0.10, 0.15} and N is the number of sites before the operation. The realized fraction is recorded as k/N, including for interstitial insertion. Thus, for a three-site cell, all three nominal levels yield one operation and a realized fraction of 33.33%.

Our audit covers 18,000 stored site-based perturbation records from 1,000 base-set structures and 500 large-set structures. The realized-fraction distributions are summarized below.

DatasetNominal fractionMean realized fractionMedianRange
Base5%10.04%6.67%3.45–33.33%
Base10%12.75%10.71%7.14–33.33%
Base15%16.71%15.63%11.11–33.33%
Large5%4.92%4.90%4.00–5.88%
Large10%9.96%10.00%8.89–10.91%
Large15%15.01%15.00%13.95–16.00%

Q2-2 Collapsed intensity levels. Among the 1,000 base-set structures, 564 564˙056\.40% have at least two nominal levels producing the same operation count, and 286 286˙028\.60% have all three levels producing the same count. Neither type of collapse occurs in the 500 large-set structures. Across the complete source set, these proportions are therefore 37.60% and 19.07%, respectively. Among the 1,283 source structures represented by retained site-based perturbations, the corresponding proportions are 38.97% and 19.49%.

Generated and retained samples. Each source material has one stored sample per operation and intensity level, giving 1,500 generated records per operation–level combination. The retained counts below represent unique perturbation records referenced by final questions, including candidate structures and deduplicated across tasks.

OperationGenerated: low / medium / highRetained: low / medium / high
Substitution1,500 / 1,500 / 1,500238 / 223 / 240
Vacancy1,500 / 1,500 / 1,500229 / 227 / 292
Interstitial insertion1,500 / 1,500 / 1,500232 / 172 / 215
Atomic displacement1,500 / 1,500 / 1,500657 / 629 / 644
Lattice strain1,500 / 1,500 / 1,500786 / 789 / 794

For lattice strain, the three levels represent strain amplitudes.

Q2-3 Primitive-cell representation. The benchmark predominantly uses primitive cells supplied by Materials Project, without deliberate supercell expansion for perturbation generation. All perturbations for a given material use the same fixed source CIF, and both nominal and realized fractions are recorded.

We appreciate this comment and will revise the manuscript to distinguish nominal sampling targets from realized fractions, report the distributions and retention counts above.


Response to Q3 : Relationship Between Perturbations and MD

We thank the reviewer for pointing out the inconsistency between the main text and the appendix. The temperature MD is performed separately on the original structures.

We will revise the abstract, main text, and pipeline figure accordingly to clearly distinguish the two branches:

Branch 1: Original Structure → Perturbation → Relaxation

Branch 2: Original Structure → Temperature Conditions → MD


Response to Q4 : Label consistency across different MLIPs

We thank the reviewer for raising the important question of whether the benchmark labels are sensitive to the simulation settings and the interatomic potential used during benchmark generation.

As an additional validation, we randomly sampled 25 complete questions, each containing four candidate structures, and performed a total of 100 DFT single-point calculations using VASP. The sampled questions cover different system sizes, crystal systems, chemical compositions, and perturbation types.

For each candidate structure, we calculated its DFT single-point energy and compared the lowest-energy option with the corresponding ground-truth answer generated using MACE-MP-0 medium.

The results show that DFT and MACE-MP-0 selected the same lowest-energy option in 23 out of 25 questions, achieving an agreement rate of 92.0%:

These results demonstrate strong consistency between MACE-MP-0 and DFT in identifying the lowest-energy structures within the randomly sampled questions. This additional validation provides independent ab initio evidence supporting the reliability of the energy-based ground-truth labels generated using MACE-MP-0.


Response to Q5: Expert Evaluation Fairness and Representativeness

Thank you for this constructive comment. Evaluating all 4,016 questions would require substantial effort from domain experts. Since the benchmark answers are automatically derived from simulation trajectories and structural records, we adopted a sampled expert evaluation. Initially, we selected 90 questions based on the performance of the strongest evaluated model, GPT-5.5, targeting an accuracy range of 26%–45% to focus on challenging questions. we acknowledge that this model-conditioned selection introduces potential bias and that the reported 73.33% cannot be regarded as representative of human performance across the full benchmark.

Following your suggestion, we conducted an additional model-independent evaluation on 150 randomly sampled questions. Each of the three experts independently answered 50 different questions, with each expert's subset covering all 18 tasks. The results are as follows:

ExpertQuestionsCorrectAccuracy
Expert 1503264.00%
Expert 2503672.00%
Expert 3503876.00%
Pooled15010670.67%

For clarification, the original 66/90 733˙373\.33% represents the pooled accuracy of three experts answering 30 different questions each, rather than a consensus score. Their individual results were:

ExpertQuestionsCorrectAccuracy
Expert 1301860.00%
Expert 2302376.67%
Expert 3302583.33%
Pooled906673.33%

We will use the new 150-question evaluation as the primary human evaluation . The randomly sampled evaluation provides a more appropriate basis for comparing human and model performance, with experts achieving 70.67% question-level accuracy and GPT-5.5 achieving 51.96% task-macro accuracy on the full benchmark.


Response to Q6: Inference Settings and Per-Task Results

Thank you for pointing out this omission. We apologize for the missing evaluation settings and per-task results in the original submission.

We have now included three tables in Appendix E of the revised manuscript, covering:

  1. Inference settings for all evaluated models, including context windows, output budgets, and decoding parameters.

  2. Per-task correct counts and accuracies for commercial API models.

  3. Per-task correct counts and accuracies for open-weight models.

We also clarify which inference settings were not recorded in the original evaluation protocols to ensure transparent reporting.

These additions address the missing information and improve the reproducibility of our evaluation.

Reviewer_jaY3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CrystalBench Dynamics, a simulation-grounded benchmark comprising 4,016 questions across 18 tasks in text-only and multimodal settings, designed to evaluate large language models (LLMs) and multimodal large language models (MLLMs) on dynamic reasoning in crystal systems. Starting from 1,500 real crystal structures sourced from the Materials Project, the generation pipeline applies five structural perturbations (lattice strain, vacancy creation, atomic substitution, interstitial insertion, and atomic displacement), followed by geometry relaxation and finite-temperature molecular dynamics using the MACE-MP-0 machine-learned potential. The benchmark organizes tasks into four capability families: Perception, Symmetry Reasoning, Symmetry-Response Reasoning, and Dynamics Prediction. Evaluation results show that the strongest model (GPT-5.5) achieves 51.96% task-macro accuracy on the Multimodel Track, trailing human experts (73.33%) by over 20 percentage points, though inference-time reasoning (thinking mode) substantially improves performance for models like Gemma-4-31B-IT.

Critical strengths

Introduces CrystalBench Dynamics, a comprehensive simulation-grounded benchmark spanning 4,016 questions across 18 tasks and four capability families, leveraging perturbed crystal structures and molecular dynamics simulations to systematically evaluate LLMs’ and MLLMs’ dynamic reasoning in crystal systems.

Critical weaknesses

While CrystalBench Dynamics addresses an important gap in evaluating physical and structural reasoning in materials science, several critical conceptual, methodological, and experimental concerns must be addressed:

  1. Significance of LLMs' Structural Perturbation Capabilities

The paper evaluates models on predicting the outcomes of structural perturbations, but it remains unclear whether this setup assesses genuine spatial and physical reasoning capacity or merely tests memorized scientific knowledge and pattern matching from pretraining data. The authors should clarify whether these tasks reflect core multimodal/language reasoning capacities (such as spatial tracking and symmetry transformation tracking) or simply serve as an advanced scientific knowledge retrieval test.

  1. Rigor and Potential Artifacts of Simulation Settings

The benchmark relies heavily on data generated via the MACE-MP-0 machine-learned potential (medium model in float64 precision with dispersion disabled) combined with ASE BFGS optimization and fixed-cell NVT molecular dynamics. It is vital to examine whether these specific simulation settings introduce systematic errors or artifacts. For instance, does the 1,000-step optimization limit or the 0.03 eV/Å force threshold create premature convergence, and how do these settings affect the generalization and ground-truth correctness of the generated data across diverse chemical systems?

  1. Simulation vs. Real-World Discrepancies

As all reference answers are derived programmatically from machine-learned potential simulations (MACE-MP-0), the benchmark inherits any limitations or biases of the underlying potential. The paper lacks a direct comparison or discussion regarding how these simulation results deviate from real-world experimental crystal dynamics. Providing empirical comparisons or validation against experimental trajectories or DFT (Density Functional Theory) ground truth would significantly strengthen the credibility of the benchmark targets.

  1. Framing Perturbations as Causal Inference (Intervention Level)

A controlled structural perturbation applied to a crystal system is fundamentally a special form of causal intervention (i.e., an intervention-level query within structural causal models or do-calculus). The paper currently frames the benchmark around descriptive multi-step reasoning and dynamics prediction. The authors should explicitly position their contributions above standard observational evaluations by framing these tasks as rigorous causal inference challenges, clarifying how models reason about cause-and-effect interventions versus static structural correlations.

  1. Impact of Prior Domain Knowledge Prompting on the Human-Model Gap

Human experts achieved an accuracy of 73.33% on the expert-evaluation subset compared to GPT-5.5's 51.96%. However, the evaluation protocol strictly provides raw CIF files and images without general crystallographic guidelines or contextual principles. An important question is whether providing foundational crystal knowledge, symmetry rules, or domain-specific prompting upfront would compensate for the performance gap between LLM/VLMs and human experts, or if the bottleneck lies strictly in spatial and physical inference rather than knowledge access.

Questions for authors
  1. Did you test whether supplying general crystallographic principles or prompt-based scaffolding closes the gap between models and human experts?
  2. What is the innovation of CrystalBench Dynamics against common causal intervention benchmark?
  3. How many of the excluded unconverted relaxations (190 total out of 22,500 planned) exhibited physical instability versus numerical failure in the MACE potential?
  4. Are there subsets of structures where DFT-computed reference data was compared against MACE-MP-0 to quantify potential trajectory drift?
  5. In the multimodal track, how much of the performance drop in Dynamics Prediction is attributable to visual information loss in 2D rendered projections versus actual reasoning failure?
  6. What is the standard deviation across multiple inference runs for models like GPT-5.5 and Gemma-4-31B-IT on the text and multimodal tracks?
  7. Why are tasks like Instability Prediction exhibiting near-random performance across almost all evaluated models?
4 comments
AuthorsOpenReview ↗

Reponse to jaY3

We sincerely thank the reviewer for their thoughtful and constructive feedback. We appreciate the recognition of CrystalBench Dynamics as a simulation-grounded benchmark for reasoning about structural perturbations, relaxation, and finite-temperature behavior. The comments identify important distinctions between structural reasoning and knowledge retrieval, simulation-defined reference outcomes and physical ground truth, and intervention-based prediction and formal causal inference. We address the seven questions first, followed by the remaining conceptual and methodological concerns.

Response to Q1: Effect of Crystallographic Knowledge Prompting

Thank you for this suggestion. We conducted a paired system-prompt ablation using Gemma-4-31B-IT on 180 questions, with 10 questions per task. We appended a fixed crystallographic knowledge passage covering periodic geometry, symmetry, defects, relaxation, and molecular dynamics, drawing on IUCr, MIT OpenCourseWare, and ASE documentation. The same passage was used across all tasks, without question-specific answers. All other inputs and settings were unchanged within each mode, with temperature = 0.

Capability familyNo-thinking: OriginalNo-thinking: + KnowledgeThinking: OriginalThinking: + Knowledge
Perception37.50%32.50%42.50%40.00%
Symmetry Reasoning30.00%25.00%55.00%57.50%
Symmetry–Response Reasoning30.00%24.00%56.00%54.00%
Dynamics Prediction34.00%36.00%50.00%44.00%

Scores are task-macro accuracies on the sampled subset; invalid answers count as incorrect.

The knowledge prompt helps in some settings but hurts in others. Dynamics Prediction improves without thinking, whereas Symmetry Reasoning improves with thinking. Thus, the benefit depends on both the capability and the inference mode.

One possible explanation is that general knowledge can encourage broad heuristics over instance-specific evidence. In an optimizer-step question, the thinking model changed a correct answer to an incorrect one by assuming that lattice strain required slower variable-cell optimization. In an MD displacement question, it favored a 600 K trajectory over a 300 K trajectory based partly on temperature, although the question concerned their actual finite-trajectory displacements. These explanations suggest possible failure modes, rather than establishing the model’s internal mechanism.

These mixed results highlight the value of the benchmark’s capability-based design. Although the tasks share a crystallographic domain, they require distinct forms of perception, symmetry reasoning, and response prediction that the tested uniform knowledge prompt does not consistently improve. This makes CrystalBench Dynamics useful for studying how models apply general scientific principles to specific structural evidence. The small-scale results motivate further research, without ruling out improvements from other prompting strategies.


Response to Q2: Relationship to Causal Intervention Benchmarks

We thank the reviewer for this question. While existing causal reasoning benchmarks primarily focus on abstract causal relations or macroscopic physical events, CrystalBench Dynamics extends intervention-based reasoning to atomistic crystal structures.

BenchmarkFocusKey Difference
CLadder 20232023Causal graphs and counterfactual reasoningAbstract causal relations
CLEVRER 20202020Object collisions and physical eventsMacroscopic physical interactions
CausalPhys 20262026Visual physical and causal reasoningPhysical scenes rather than crystal structures
CrystalBench Dynamics (Ours)Crystal perturbations and physical responsesAtomic relaxation, symmetry changes, and finite-temperature dynamics

Our benchmark offers three key innovations:

  1. Atomistic interventions: Evaluating how controlled crystal perturbations affect structural relaxation and symmetry.

  2. Simulation-grounded outcomes: Using atomistic simulations to generate quantitative physical response labels rather than relying on static correlations.

  3. Comprehensive evaluation: Providing 18 text-only and multimodal tasks covering perception, symmetry, structural responses, and dynamics.

Thus, our contribution is extending intervention-based physical reasoning to crystal structures through a unified, simulation-grounded benchmark


AuthorsOpenReview ↗

Reponse to jaY3

Response to Q3: Physical Instability versus Numerical Failure in Excluded Relaxations

Thank you for raising this distinction. We audited the optimization logs and final trajectory frames of all 190 completed but unconverged relaxations, and we rerun 190 cases. . All reached the 1,000-step limit . These results demonstrate sensitivity to the optimization protocol. They do not uniquely attribute failure to the MACE potential or establish genuine physical instability. We therefore report 126 numerically pathological runs and 64 unresolved non-converged runs.

Numerical pathology versus unresolved non-convergence. Among these runs, 126 663˙66\.3% exhibited clearly nonphysical outcomes: final energies below −4,309 eV/atom, maximum atomic forces above 2.68 × 10⁷ eV/Å, and minimum interatomic distances below 0.348 Å. The remaining 64 337˙33\.7% did not exhibit these extreme final-state signatures but failed to converge within the step limit.

Additional optimization controls. We reran all 190 cases using four settings: extended BFGS, smaller-step BFGS, smaller-step FIRE, and fixed-cell optimization followed by joint atomic/cell optimization, totaling 760 runs. Of the 190 cases, 116 reached the optimizer’s convergence criterion under at least one setting, including 55 of the 126 pathological cases.


Response to Q4: DFT Validation and Simulation-to-Reality Discrepancies

We thank the reviewer for raising this important question regarding potential trajectory drift introduced by the MACE-MP-0 interatomic potential.

To assess the consistency between MACE-MP-0 and DFT, we randomly sampled 25 complete four-option questions, covering different system sizes, crystal systems, chemical compositions, and perturbation types. We performed 100 DFT single-point calculations using VASP and compared the DFT-computed energies with the corresponding MACE-MP-0 predictions.

The results show that DFT and MACE-MP-0 selected the same lowest-energy option in 23 out of 25 questions, achieving 92.0% agreement.

These results provide independent ab initio evidence supporting the reliability of MACE-MP-0-generated energy-based labels.


Response to Q5: Visual Information Loss versus Reasoning Failure

Thank you for raising this important question. To investigate it, we conducted a paired experiment using Gemma-4-31B-IT with and without thinking, both at temperature 0, on all 339 questions from the two multimodal Dynamics Prediction tasks. We compared four input conditions while retaining the same questions, candidates, and labels:

  • A: Original single-view images.

  • B: Three orthogonal views of the same trajectory frames.

  • C: Exact geometric measurements for the two visible coordinate axes.

  • D: Exact measurements for all three coordinate axes.

Our ablation suggests that visual information loss and extraction difficulties are substantial contributors.

TaskModeA: Single viewB: Three viewsC: 2D measurementsD: 3D measurements
Envelope-volume change (180 questions)No-thinking34.44%38.33%70.00%100.00%
Envelope-volume change (180 questions)Thinking26.67%37.78%70.00%99.44%
Temperature ordering (159 questions)No-thinking22.64%27.04%84.28%85.53%
Temperature ordering (159 questions)Thinking29.56%40.88%92.45%98.11%

The effects differ by task. Adding views improves accuracy by 3.89–11.32 percentage points. Under the numerical interface, adding the third coordinate improves envelope-volume accuracy by 29.44–30.00 points, compared with 1.26–5.66 points for temperature ordering. This supports a stronger depth-information bottleneck for the volume task. The substantial improvements from images to exact measurements also indicate difficulty extracting and tracking geometric information from rendered trajectories. With complete measurements, thinking-mode accuracy reaches 99.44% and 98.11%, suggesting that downstream computation and inference are largely successful under this assisted setting.


Response to Q6: Variability across Repeated Inference Runs

Thank you for asking about inference reproducibility. In our latest evaluation, all models were run with the decoding temperature set to 0, including GPT-5.5 and Gemma-4-31B-IT, across both the text and multimodal tracks. This setting was used to minimize variability from stochastic token sampling and keep decoding settings consistent across evaluations. We will state this explicitly in the evaluation protocol.


Response to Q7: Near-Random Performance on Instability Prediction

Thank you for raising this question. We believe the main reason for the relatively low accuracy is that this task remains challenging for current models, as it requires predicting complex structural responses and simulation outcomes rather than simply recognizing static crystal properties.

Nevertheless, GPT-5.5 achieves 35.01% accuracy and Gemma-4-31B-IT (thinking) achieves 35.31%, both exceeding the 25% random baseline for four-option questions. These results suggest that models demonstrate some predictive capability, although substantial room for improvement remains.


Responses to Critical Weaknesses

W1: Structural reasoning versus scientific knowledge retrieval.

We agree that benchmark accuracy alone cannot establish whether a model performs genuine physical reasoning. Our tasks require interpreting instance-specific coordinates, structural changes, and trajectories, but may also involve pretrained knowledge and heuristics. The knowledge-prompting ablation in Q1 yielded no significant improvement, while the representation ablation in Q5 revealed substantial sensitivity to geometric information access. These findings support evaluating multiple contributing capabilities, without claiming to isolate reasoning from memorization.

W2: Simulation settings and numerical artifacts.

We acknowledge that reference outcomes depend on the potential and numerical protocol. As detailed in Q3, alternative optimization settings allowed 116 of 190 previously unconverged cases to reach the convergence criterion, demonstrating protocol sensitivity. Some converged endpoints nevertheless remained suspicious. We will distinguish numerical convergence from physical validity and clarify that these controls do not establish the universal adequacy of the force threshold or simulation settings across chemical systems.

W3: Simulation-to-reality discrepancies.

The DFT validation in Q4 provides a targeted independent check: DFT and MACE selected the same lowest-energy candidate in 23/25 questions 920˙92\.0%, based on 100 single-point calculations. This supports the sampled energy-ranking labels.

W4: Causal-intervention framing.

We thank the reviewer for this insightful suggestion. We agree that controlled crystal perturbations can be viewed as intervention-level causal reasoning, rather than merely descriptive structural prediction.

Unlike observational benchmarks that evaluate static structural correlations, CrystalBench Dynamics applies explicit interventions, including atomic substitution, vacancies, interstitials, displacements, and lattice strain, and evaluates their resulting structural responses through atomistic simulations. This requires models to reason about how controlled changes affect relaxation outcomes and crystallographic symmetry.

We will revise the Introduction and Related Work to explicitly frame our benchmark as simulation-grounded, intervention-based causal reasoning at the atomic scale. However, we clarify that our benchmark evaluates reasoning about intervention outcomes rather than formal causal identification through structural causal models or do-calculus.

W5: Domain knowledge and the human–model comparison.

The paired ablation in Q1 shows that foundational crystallographic guidance has mixed effects: it improves Dynamics Prediction without thinking and Symmetry Reasoning with thinking, while other capability-level scores decrease. These results highlight the distinct reasoning demands within a shared scientific domain and the value of evaluating them separately. The tested uniform knowledge prompt does not consistently improve performance, but this does not establish that knowledge access is irrelevant or that the bottleneck lies exclusively in spatial and physical inference.

Thank you very much for your thoughtful and constructive feedback.

We have provided detailed responses to your questions, conducted additional experiments, and revised the manuscript accordingly.

May we kindly ask whether our responses have addressed your concerns and if you have any further comments or suggestions? We would be happy to provide additional clarification if needed.

Thank you again for your time and consideration.

Comments on the paper

1 comment

Hi reviewers,

Thank you for your reviewing service. The authors have provided fairly extensive rebuttals. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.