acceptodds
Under review as a conference paper at ICLR 2027

GEMBench: Separating Distribution Matching from Discovery in Crystal Generation

Abstract

Generative models for crystals can serve two distinct aims: reproducing the distribution embodied in the training dataset, or discovering new stable materials beyond it. Classical generative metrics (Fr\'echet distances, kernel MMD, and related feature-space distances) evaluate only the first aim, while widely used S.U.N.-style metrics (fraction of stable, unique, and novel structures) are often reported as discovery progress without declaring the discovery aim. We make this split explicit as the *generative discovery dilemma*: the two aims share the same notion of stability, yet prescribe different optimal output distributions, so evaluation must declare the aim. We then introduce GEMBench (Generative Evaluation for Materials), which scores the two aims separately as and under explicit choices of stability probe and crystal-similarity measure. A shared multi-probe stability panel (energy above hull, phonon, and finite-temperature checks) replaces the single energy-above-hull bit used by S.U.N.-style metrics. Both scores exclude near-duplicates of training-set crystals to avoid rewarding memorization. Beyond that, rewards dataset-like distributional shapes, while rewards a large and diverse set of stable structures. On public generators trained on MP-20, , , and CHGNet S.U.N. rank the same models differently. Which generator leads depends on the declared aim and on how crystal similarity is defined. In particular, the CHGNet S.U.N. leader ranks last on , because it concentrates on a few modes of the crystal space rather than discovering many distinct crystals.

Primary area:
datasets and benchmarks
Keywords:
crystal generation, materials discovery, benchmarks, generative model evaluation, distribution matching, novelty, structural stability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_h1W4OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes GEMBench to evaluate crystal generation from two different goals: distribution matching and new material discovery. The authors show that existing metrics such as S.U.N. may not fully reflect discovery ability, and different evaluation metrics can give different model rankings.

Critical strengths
  1. The paper proposes new evaluation metrics to better assess crystal generation.
  2. The authors introduce a comprehensive evaluation protocol for crystal generation models.
Critical weaknesses

see my question

Questions for authors
  1. For a new benchmark, shouldn't there be new datasets or OOD tests? Is proposing new evaluation metrics enough?
  2. How well do these metrics correlate with real material discovery results, such as experimentally validated structures?
  3. How can the authors prove that these metrics are more meaningful than existing ones like S.U.N.? Different model rankings do not necessarily mean better evaluation.
3 comments

Thank you for the three clarifying questions. We answer each below.

Q1. Is this a benchmark without a new dataset?

Yes: GEMBench does not release a new crystal dataset. The generators and training corpora we score are already public (MP-20, Alex-MP-20, and published checkpoints). What we propose is a way to score those generations under two explicit aims, matching and discovery.

We still call the result a benchmark because the object is larger than a closed-form metric formula. In our logic, a usable score for crystal generation needs two non-trivial pieces of infrastructure before any equation can be trusted:

  1. A stability panel. Defining which generated crystals count as “stable” is not a free one-bit choice. The panel has to be designed (which probes and pass criteria), tested against experimentally known solids, optimized for wall-clock cost, and sized so that ranking noise stays small. That work is what turns an informal MLIP check into a reproducible filter that decides which materials enter the distribution-level scores.
  2. Crystal-similarity kernels. Distribution matching and discovery both need a notion of how similar two crystals are. Before using any kernel on real models, we calibrate it on controlled mode-collapse suites. Only after that calibration do we plug the kernel into form agreement (matching) and Vendi (discovery).

As a result we do obtain closed-form expressions for GEM-match and GEM-disc, plus a concrete algorithm to compute them. Because that algorithm rests on the stability panel and calibrated kernels above, we treated the whole package as a benchmark. If the missing new dataset makes the word benchmark misleading, we are happy to rename the contribution to an evaluation protocol or metric suite. The contribution is an evaluation protocol built around a stability panel and similarity kernels, and metrics that use them for aim-oriented quality measurement.

Q2. How well do these metrics correlate with experimentally validated discovery?

GEM-match and GEM-disc cannot themselves be verified as experimental discovery rates. They score generators as distributions under a fixed panel and kernel, rather than as wet-lab outcomes. What is experimentally checkable, and what we already report in the paper, is the stability panel that selects which materials enter those scores.

Appendix A.7 reports how accurate the stability tests are on real, experimentally known solids. On a hold-out catalog of 100 known solids (disjoint from tuning), the NPT screen reaches overall accuracies of 0.840.84 / 0.880.88 / 0.970.97 at 5050 / 300300 / 600600~K (tuning: 0.930.93 / 0.900.90 / 0.990.99). GEM-match and GEM-disc then measure generative quality only for crystals that pass these tests. A model therefore cannot reach high scores without finding materials that pass a filter already checked on experimentally known solids.

Q3. How can you prove these metrics are more meaningful than S.U.N.?

Different rankings alone would not prove a better metric. The argument is that S.U.N. leaves an important quality axis unmeasured. Some public models already climb higher S.U.N. rates than others by a behaviour S.U.N. does not reveal. That behaviour looks questionable once diversity of the stable set is measured.

S.U.N. is essentially a yield of crystals that pass a hull-style stability gate (plus discrete uniqueness/novelty bits). It does not account for the shape of the distribution of those stable materials. A generator can therefore raise S.U.N. by concentrating mass on a thin slice of crystal space and emitting many near-duplicates that still clear the hull cut and a StructureMatcher uniqueness bit. Under S.U.N. that looks like success, and for a developer oriented on S.U.N. the mode collapse is invisible.

Example from paper.

On the same MP-20 generation batches:

  • MiAD leads CHGNet S.U.N.\ (13.213.2) and has a high on-hull rate under our panel (γEaH0=0.139\gamma_{\mathrm{EaH}_0}=0.139).
  • DiffCSP++ has a lower on-hull rate (γEaH0=0.081\gamma_{\mathrm{EaH}_0}=0.081) but a higher EaH$_0$ discovery term ρ⋅VS/n\rho\cdot\mathrm{VS}/n under the physical kernel (0.0240.024 vs MiAD’s 0.0180.018), and it leads overall physical GEM-disc (0.1210.121).
  • Globally, MiAD is last on GEM-disc (Rank 5.55.5) despite leading S.U.N.

That pattern is exactly the failure mode above: more stable crystals does not imply more really different stable crystals. Where S.U.N. tracks a rate of stable and unique materials, GEM-disc uses a Vendi score on the accepted set to count how many modes in materials space are covered by stable materials, at fixed budget. Collapse among accepted stables is therefore punished. GEM-match separately asks whether the accepted set agrees with the training-dataset law in rate and shape, which S.U.N. does not ask at all.

Modern crystal generators are already optimizing quantities like S.U.N. If that scalar ignores distributional collapse of the stable part, leaderboards can reward models that look productive while covering only a thin slice of crystal space. GEMBench makes that failure visible and gives developers scores that speak not only to a model's ability to find stable crystals, but to two practical desires. GEM-match tracks finding more materials similar to the dataset, without exact copies. GEM-disc tracks covering as much of the materials domain as possible. Different rankings alone would not prove progress. The point is that these scores measure a quality axis that S.U.N. leaves out, and that axis already reorders public checkpoints.

We hope the answers above resolve your three questions on contribution scope, experimental grounding of the panel, and why GEM-match/GEM-disc go beyond S.U.N. If so, we would be grateful if you would consider raising your score.

Thank you for your clear explanations. I appreciate your efforts in addressing my concerns, and I will raise my score accordingly.

Reviewer_uKK7OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In the manuscript titled "GEMBench: Separating Distribution Matching From Discovery in Crystal Generation", authors have raised a fundamental issue in evaluating the material generating models vis-a-vis the difference between the training task (matching the distribution), and inference task (sampling out of distribution). It correctly identifies the limitation of current S.U.N metric, and provides adequate, robust and tractable solution in terms of multi property stability panel, and kernel based structure matching. . The GEM-match, and GEM-disc provides comprehensive overview and categorization of the generated structures. The models selection for the demo is exhaustive, and convincing enough.

Critical strengths
  • Comprehensive model selection for demo
  • Robust and tractable mathematical foundation for the the scores (GEM-match and GEM-disc)
  • Manuscript is well organized, and easy to follow.
Critical weaknesses
  • For the embedding kernels, the authors used the MACE-MP model, while the solutions is acceptable, a more robust solution will be to use some form of first-principle structural embedding like a descriptor, or some structure factor etc.
  • Paper fails to address one of the key strength of the SUN metrics, it provides a single scalar to compare and rank models. GEM-Bench is less clear on how will it rank based on two scalar values.
Questions for authors

Authors should address the two weaknesses mentioned:

  • Can they use some form of first-principle structural embedding like a descriptor, or some structure factor etc for structure kernel? will it be better or worse?
  • Paper fails to address one of the key strength of the SUN metrics, it provides a single scalar to compare and rank models. GEM-Bench is less clear on how will it rank based on two scalar values.
1 comment

Thank you for the review. We address the two questions below.

W1 / Q1. First-principles structural embeddings instead of MACE-MP for the embedding kernel

Yes. Testing first-principles structural embeddings (descriptors, structure factors, and related classical fingerprints) is a reasonable alternative and should be checked. Our physical kernel κ_phys is already in a similar direction: a hand-crafted structural descriptor built from size, symmetry, packing, cell geometry, and composition, rather than an MLIP embedding, though it is not a structure-factor representation. We are continuing work on kernels and will include the suggested descriptors among the design choices we evaluate.

W2 / Q2. S.U.N. gives one scalar, GEMBench gives two

This is the main design difference from S.U.N.

S.U.N. is a reasonable desire for a generator: raise the yield of stable, unique, and novel crystals, and get one number that is easy to rank and tune against. It does not correspond to the matching aim, because it never compares a representation of the generated distribution to the training distribution. It is also only a weak proxy for the discovery aim, because it tracks a rate of stable and unique materials, whereas discovery needs coverage of distinct modes in materials space by the accepted stables. That is why GEMBench reports two scores instead of one S.U.N.-style yield: GEM-match for agreement with the training-dataset law, and GEM-disc for Vendi mode coverage of the accepted set under a fixed budget (Section 1).

We treat some quality axes as combinable into one score and others as not. Inside matching, rate agreement and form agreement are multiplied because both must hold, and the six stability probes are averaged without weights because the bench does not declare one stability notion primary: those axes belong to one aim, so we combine them. In our work we propose that matching and discovery are non-combinable aims that should not be pursued simultaneously. Empirically that already shows up on MP-20 under the Physical GEM-disc kernel: DiffCSP++ scores 0.121 while the training-set row is 0.103, so being close to the dataset means a lower GEM-disc (Section 4).

After the aim is declared, the developer again has a single metric to chase and can forget about the other. Matching-oriented work tracks GEM-match. Discovery-oriented work tracks GEM-disc. Building a model for only one of those aims is the intended pipeline. Two reported scalars are therefore how the protocol keeps the aims separate until that choice is made, rather than a request to optimise both at once.

Reviewer_hU4MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new benchmark for material generation that goes beyond the SUN metric and tries to evaluate stability in a more robust way than previous works with MLIP approaches. The authors also try to get a better measurement of novelty based on relaxation of generated structures.

Critical strengths
  • generative models performance formulated as a dual objective: generating stable and novel structures
  • GEM-match: Similarity between features from generated structures and known stable structures
  • GEM-disc: Diverse and stable structures
  • results are not limited to energy above hull at 0K for stability.
  • tentative to improve previous methods' novelty metrics.
Critical weaknesses
  • To the best of my knowledge, MLIP methods are not precise enough when energy above hull is small. The authors were honest about it but didn't bring experiments to better characterize this limitation. To the credit of the authors, DFT is expensive but from my perspective, assessment of material generation without DFT is always fairly limited.

  • The authors might have dug deeper into validating MLIP precision without DFT, for example, we might add small random noise and check if MLIP relaxation brings the structures back to their original positions or a more stable configuration.

  • Section A.7 is informative but limited to binary structures, meaning simple and well-known compositions. Precision is expected to be high as these systems are well known, and most interesting new materials are composed of more than 2 types of atoms. This limitation is not discussed.

  • Novelty measurement is limited: most criticism of generative models centers on substitution approaches being able to make similar discoveries to generative models. Your measurement of novelty doesn't take this limitation of diffusion models into account.

  • The most interesting novel structures might be considered out of distribution for pre-trained MLIPs, and I don't think enough evidence has been given to trust MLIP evaluation for generative models.

Questions for authors
  • MACE relaxation is not always precise enough for low energy above hull structures. Personal experimentation led me to take MACE relaxation with a grain of salt. Section "A.7 Catalogs and accuracy" is great to help us understand the precision of your approach but in my opinion 100 structures is not a lot, and why haven't you tested your approach for larger structures?

  • Why don't you compare MLIP with DFT calculations? I understand DFT is expensive but a small number of DFT calculations to validate your results could be helpful.

3 comments

Thank you for a careful and constructive review. The weaknesses you raised named concrete gaps that can make an MLIP stability protocol look more reliable than it is, and they pushed us toward rebuttal experiments that we think make the paper stronger. We address each point below.

W1. DFT verification of MLIP-based tests

Fair point that MLIP-based stability checks are imperfect, especially near the hull, and that generator crystals can sit outside the effective support of a pretrained potential. GEMBench does not try to replace first-principles validation of individual candidates. We rely on the MLIP community and on colleagues who build universal potentials and the protocols used to evaluate them (e.g. CHGNet, Deng et al., 2023; MACE foundation models, Batatia et al., 2023). Modern MLIPs are typically judged by agreement with DFT on energies, forces, and stresses on held-out structures, plus transfer tests such as phonons, elastic constants, formation or hull energetics, and short MD integrity (Loew et al., 2025). Public leaderboards such as Matbench Discovery further stress hull ranking on large crystal sets (Riebesell et al., 2025). Those evaluations are why we treat a strong universal MLIP as a practical panel oracle for a fixed, reproducible filter, not as a substitute for DFT discovery claims. The reviewer’s OOD concern for generated structures is still real: catalog solids and standard MLIP test sets do not fully cover generator output. We therefore treat panel choice as an ongoing design decision and will continue searching for an MLIP (or checkpoint) that better matches our OOD setup of generator-emitted crystals under the GEMBench probes. Under the present MACE panel, GEM-match and GEM-disc should be read as scores with the MLIP limitation stated in the paper.

Full DFT molecular dynamics or a full DFT re-run of the panel on every generated crystal remains beyond our compute budget. As a complementary check we compare MACE to DFT on generator geometries from the GEMBench pipeline, without claiming DFT pass/fail of the full dynamics.

Rebuttal experiment.

To test this concern directly, we compared MACE-MP-0 small, the checkpoint used in GEMBench, with DFT for 17 generated crystals, including 12 compositions with three to five elements. We ran DFT relaxation of the atoms and cell from MACE minima and compared both methods on identical geometries. Fifteen candidates have electronically converged static DFT evaluations at both geometries with matching settings. Only two satisfy the final force/stress criteria (0.02 eV/Å and 0.2 GPa), so the other comparisons remain provisional.

For these 15 pairs, the median of the mean atomic displacements is 0.053 Å and the MAE of force components is 0.111 eV/Å. Comparing each method's per-atom energy change between the MACE minimum and latest DFT geometry removes constant offsets. The median and mean absolute discrepancies are 10.9 and 51.4 meV/atom, respectively.

The largest DFT energy decreases (44-253 meV/atom) occur for Pr₂CNClO, Bi₂Sb₂Cl₂O₃, Cs₂LaNdC₂N₉, and YbP₂, where MACE instead predicts energy increases. All four also fail GEMBench's phonon and 600 K MD probes. Among the paired cases, candidates passing the MD probe (n=10) show smaller median mean displacements than failing candidates (n=5): 0.032 versus 0.239 Å.

This association supports the panel's qualitative screening role within this subset. It does not establish DFT agreement on finite-temperature or hull stability. The sample was selected randomly and contains only 3-15 atoms per cell. Generalization to larger cells and other OOD candidates remains for future tests. Near-hull validation additionally requires consistent DFT energies for competing phases.

W2. MLIP consistency under structural noise

We agree that this is a useful check, and we carried it out. Recovery under noise checks whether the MLIP stays consistent in a small neighborhood of a prerelaxed generated structure. It is best read as a range-of-applicability check: does the panel MLIP behave reliably on structures that generators actually emit? However, it does not by itself give an error estimate for GEM-match or GEM-disc, and it complements the DFT fidelity study in W1.

Rebuttal experiment.

Setup. 100 prerelaxed MatterGen crystals from the GEMBench evaluated subset. Panel MLIP: MACE-MP-0 small, TorchSim FIRE, cell free. Re-anchor each structure, add isotropic Cartesian Gaussian noise at thermal-scale amplitudes (σ_1d = 0.030 / 0.0735 / 0.1039 Å for nominal 50 / 300 / 600 K), then re-FIRE with the same settings (3 draws per crystal). Same-basin: StructureMatcher finds the same basin as the reference minimum. Inclusive recovery: Same-basin, or energy per atom no higher than the reference by more than 1 meV/atom.

Noise scaleSame-basinInclusive recovery
50 K0.9701.000
300 K0.9531.000
600 K0.9431.000

Reading. Same-basin recovery stays high. The remaining trials hop to a different, lower-energy basin (allowed by the inclusive criterion). This supports local consistency of the GEMBench MLIP near generator geometries, as a consistency check rather than a calibration of the panel.

W3. Stability catalogs limited to elemental and binary solids

Also a fair point. Appendix A.7 tunes and holds out the NPT screen on two disjoint 100-crystal catalogs of known elemental and binary solids (paper accuracies 0.93 / 0.90 / 0.99 on tuning and 0.84 / 0.88 / 0.97 on hold-out at 50 / 300 / 600 K). Those systems are better studied and often easier for universal MLIPs.

Rebuttal experiment.

Setup. Keep the original catalogs unchanged and only expand each into a matched pack of 227 known solids: the same 100 cores plus 127 multi-element extensions with N≥3 distinct elements (93 ternaries and 34 quaternaries per pack), labeled by experimental melting temperatures only (handbook / Wikidata, no predicted Tm). Near-ladder-boundary compositions were excluded so labels are not sitting on the 50 / 300 / 600 K decision edges. The GEMBench stability tests are unchanged (frozen NPT screen, no retuning). Rates below are accuracies on N=227 per pack (N=454 combined).

Pack50 K300 K600 K
Tuning (core + extensions)0.9340.8280.943
Hold-out (core + extensions)0.8990.8590.912
Combined0.9160.8440.927

Reading. Combined accuracy stays high at 50 K and 600 K. The 300 K rung is weakest, with residual false positives mostly on soft / near-Tm mid-band multi-element solids, including some molecular mid-low cases, the same error family noted for cryogens and vdW systems in Appendix A.

W4. Novelty does not punish near-duplicates of the training set

Yes. Novelty in GEMBench is a StructureMatcher filter against the training set, as in S.U.N., so exact copies are removed, including those that become matches only after prerelaxation, while soft near-copies of training crystals, including many element substitutions, can still pass.

A natural response is to put a kernel into novelty and punish any proximity to the training set, not only exact copies. We think that changes the discovery question rather than repairing the current one. Element substitution is a legitimate materials-building strategy, so a soft novelty gate would discard many chemically valid candidates before the stability panel. A model that can only produce near-substitutions of training crystals can still be hypothetically strong on matching the training distribution (GEM-match), but it should not reach the highest discovery levels (GEM-disc). We already see partial separation in the tables: on MP-20, DiffCSP++ exceeds the dataset row on physical GEM-disc: 0.121 vs 0.103, so the present discovery score already distinguishes, at least in part, models that cover more distinct stable modes than resampling the training set.

If the goal is extreme novelty relative to training support, a kernelised novelty term or a separate extreme-novelty score is appropriate, and we will think about converting that into a closed-form aim. We would first need to check carefully whether kernels calibrated to catch mode collapse, and therefore to measure domain coverage at the set level, are the right tool for deciding whether a particular crystal is novel from the point of view of materials-science experts.

W5. MLIP trust on out-of-distribution generated structures

Agreed that generated crystals can sit outside the effective support of a pretrained MLIP, which is a real risk for any MLIP panel. Catalog solids are a weak test of that risk, because they lie closer to the MLIP training support. W1 therefore both states our reliance on community MLIP evaluation and reports a DFT comparison on 17 generated crystals (15 paired comparisons) under the panel MACE checkpoint. That check targets MLIP accuracy outside the common catalog support on generator geometries, including multicomponent cells.

Q1. Catalog size and larger structures

Agreed that 100 per split was modest. Scaling further is not just a matter of pulling more CIFs. When we tried expanding with predicted melting temperatures (composition-level GBR), the frozen NPT screen disagreed with those labels much more often than with experimental melting temperatures, especially at 600 K (about 0.69 accuracy on predicted-Tm extensions versus about 0.90 on experimental-Tm extensions in the same protocol). That gap is large enough that we treat predicted Tm as unreliable supervision for this screen. We therefore keep the strategy of tuning and validating NPT dynamics only on experimentally known melting points, and the W3 packs grow each split from 100 to 227 under that constraint (N=3 and N=4).

Q2. Direct MLIP vs DFT comparison

Yes. This is the same experiment as in W1: we cannot afford full DFT dynamics for the panel, so we compare DFT to the panel MLIP on generator geometries from MACE minima. Setup and results are under W1.

We hope the responses above — especially the completed checks under W1-W3, the novelty discussion under W4, and the MLIP-evaluation framing under W1 — address your concerns. If so, we would be grateful if you would consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.