acceptodds
Under review as a conference paper at ICLR 2027

Beyond In-Domain Evaluation: Benchmarking Experimental Generalization in Disordered Crystal Generation

Abstract

Crystal generation models are commonly trained and evaluated on open databases that combine experimentally measured and computationally generated structures. This creates a distribution-shift blind spot: strong in-domain performance may reflect database fit rather than generalization to laboratory materials. We introduce Disorder-Bench, a multi-source benchmark for measuring experimental generalization in disordered crystal generation. It contains 91,704 structures from COD, ICSD, AMCSD, and Springer Materials, organized into positional, site, and mixed positional–site disorder. The benchmark provides a unified evaluation protocol covering structural and compositional validity, distributional fidelity, occupancy statistics, Wyckoff-site diversity, and space-group preservation. Across representative diffusion, flow-matching, equivariant, and language-model generators, models trained on COD degrade substantially when evaluated on experimentally curated sources. MatterGen reaches 69.5% overall validity, whereas CrystaLLM drops to 0.0%; mixed positional–site disorder is consistently the most challenging regime. Failure analysis identifies occupancy handling, multiplicity and space-group inconsistencies, and CIF representation errors as major sources of failure. Disorder-Bench provides a reproducible testbed for separating database fit from empirical fidelity in crystal generation.

Primary area:
datasets and benchmarks
Keywords:
disorderd crystal structure generation, materials generation, benchmark, cross-domain evaluation, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DE4BOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper attempts to introduce a multi-source benchmark for measuring experimental generalization in disordered crystal generation based on structures from COD, ICSD, AMCSD, and Springer Materials, but because it bears all signs of a fully AI-generated paper (ranging from hallucinated references to hallucinated numbers in other publications to unfinished experiments) it fails to do so.

Critical strengths

None.

Critical weaknesses

This paper has a series of wrong claims and hallucinated facts. In the following, I will only list the most serious problems:

  • The authors claim that the "vast majority of crystal generation models are trained and evaluated on the Crystallography Open Database (COD)." This is just wrong. DiffCSP, MatterGen, CrystalLLM, FlowMM, and CrysLLMGen, all models that were considered in this paper, are not trained on COD.

  • Table 2 claims to take COD results from the original publications that are just not present in the original publications. Also, the work is unfinished ("FlowMM deferred due to PyTorch manifold operator compability; CrysLLMGen deferred due to upstream matgl realization dependencies and inference latency.").

  • Most of the considered generative models were not designed with disordered materials in mind. The authors never explain how they adapted, if at all, these models to the disordered setting.

  • Several references are hallucinated. The very first Dis-CSP reference is claimed to be by "Anonymous". The EquiCSP reference misses several authors. The review by De Breuck et al. reference has the wrong article number.

Since the authors explicitly state that they take "responsibility for the final content," I have to be blunt: Submitting a paper with such fundamental errors and apparently fabricated results is disrespectful to the research community and to the reviewers who volunteer their time to evaluate it. The authors should refrain from submitting work of this quality in the future.

Questions for authors

None.

Reviewer_c256OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Disorder-Bench. It evaluate the generalization of crystal structure generative models from conventional crystallographic databases to experimentally curated disordered crystal structures. The benchmark comprises 91,704 structures from four sources and categorizes them as positional disorder (PD), site disorder (SD), and mixed positional-site disorder (SPD). The authors propose a unified evaluation framework that covers structural and compositional validity, distributional fidelity, occupancy statistics, and preservation of crystallographic symmetry.

Critical strengths
  1. The paper addresses a meaningful limitation of existing crystal-generation benchmarks. Evaluating cross-domain generalization is particularly relevant for assessing the practical utility of generative models in materials discovery.
  2. The dataset contains a large number of structures and multiple sources. The three types of disorder also provide a useful feature to analyze model performance under different structural complexities.
Critical weaknesses
  1. The evaluation metrics is inconsistent. In table 5, it seems the experimental generalization gap is calculated by difference between COD compositional validity and cross-domain overall validity, which reflects two difference metrics. However, as defined in eqn 3, the metric f should be the same.
  2. Also in table 5, it seems DiffCSP reports match rate but other methods report validity.
  3. The cross-domain structure validity score reported in table 5 seems inconsistent with Fig 5.
Questions for authors
  1. Can the authors clarify this inconsistency mentioned in W1 and recompute the generalization gaps using the same metric definitions?
  2. What exactly is measured in Figure 5, and how is that different from the cross-domain structure validity score reported in table 2?
Reviewer_7BeiOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Disorder-Bench, a benchmark of 91,704 disordered crystal structures from Springer Materials, COD, AMCSD and ICSD. The structures are labeled as positional, site, or mixed positional–site disorder. It proposes a three-tier evaluation strategy namely validity, distributional fidelity, and disorder-specific metrics. Then it measures an "experimental generalization gap" Δexp\Delta_{\text{exp}} between COD-trained performance and performance on the curated sources. Evaluating DiffCSP, DMFlow, MatterGen and CrystaLLM, the authors report large cross-domain drops, from 69.5% validity for MatterGen to 0.0% for CrystaLLM. They find mixed disorder hardest and attribute CrystaLLM's failure mainly to how CIF files represent occupancy.

Critical strengths

An important, under-served problem. Nearly all crystal generators and benchmarks assume fully occupied sites. Yet partial occupancy and site mixing are central to solid electrolytes, high-entropy materials and thermoelectrics. A large disordered-structure corpus with provenance labels and a disorder-aware evaluation protocol would be valuable to the community. The right framing. Stratifying by source database, to separate fitting one database from generalizing across curated sources, is a sensible design. The CrystaLLM failure analysis points to a concrete, actionable issue: CIF-level representation of occupancies.

Critical weaknesses

1. Inconsistencies in the metrics and generalization claims that lack sufficient support. Experimental differences in generalization cannot be compared across model families as DiffCSP has been measured in terms of match rate while others have been measured in terms of validity metrics. As for Table 2, it provides the results for DiffCSP, DMFlow, and CrystaLLM that are Bench ID scores from the original papers without measuring it on Bench ID. It is important for the authors to explain the origins of all the entries. Numerical inconsistencies can also be seen in the DiffCSP (COD-all) experiment. As mentioned by the authors, the match rates are 10.4% for PD, 17.7% for SD, and 0.0% for SPD. However, applying weights according to the distribution of disorders in the non-COD benchmark sources will result in about 5.8%, which is inconsistent with the match rate of 16.5%. 2. Lack of adequate validation of disorder taxonomy and incomplete assessment of disorder fidelity. The threshold for positional disorder criteria is τU=0.01,A˚2\tau_U=0.01,\mathrm{\mathring{A}}^2. Since these parameters can reflect both thermal motion and static disorder, the authors must explain how the differences are understood within the criterion. The expert agreement and sensitivity analysis of the threshold are helpful, but additional information on the methods used is necessary for evaluation. Disorder classification distributions differ across datasets. No structures are classified as SD in Springer Materials, while AMCSD contains 4170 SD structures and 243 PD structures. This may pose some problems regarding possible confounding between disorder categories and database specific trends of disorder description. Finally, the paper describes occupancy-distribution fidelity, Wyckoff-position diversity, and space-group preservation as important measures for disorder evaluation. But it does not provide any results for the same. It is necessary to describe how the evaluation procedure will work for models which cannot explicitly handle partial occupancies and disorder targets.

Questions for authors

which checkpoint, and under which metric? Please provide a version of Table 2 where every model is trained and evaluated under the same protocol and the same metric.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.