GEMBench: Separating Distribution Matching from Discovery in Crystal Generation

Generative models for crystals can serve two distinct aims: reproducing the distribution embodied in the training dataset, or discovering new stable materials beyond it. Classical generative metrics (Fr\'echet distances, kernel MMD, and related feature-space distances) evaluate only the first aim, while widely used S.U.N.-style metrics (fraction of stable, unique, and novel structures) are often reported as discovery progress without declaring the discovery aim. We make this split explicit as the *generative discovery dilemma*: the two aims share the same notion of stability, yet prescribe different optimal output distributions, so evaluation must declare the aim. We then introduce GEMBench (Generative Evaluation for Materials), which scores the two aims separately as $\mathrm{GEM}\text{-}\mathrm{match}$ and $\mathrm{GEM}\text{-}\mathrm{disc}$ under explicit choices of stability probe and crystal-similarity measure. A shared multi-probe stability panel (energy above hull, phonon, and finite-temperature checks) replaces the single energy-above-hull bit used by S.U.N.-style metrics. Both scores exclude near-duplicates of training-set crystals to avoid rewarding memorization. Beyond that, $\mathrm{GEM}\text{-}\mathrm{match}$ rewards dataset-like distributional shapes, while $\mathrm{GEM}\text{-}\mathrm{disc}$ rewards a large and diverse set of stable structures. On public generators trained on MP-20, $\mathrm{GEM}\text{-}\mathrm{match}$, $\mathrm{GEM}\text{-}\mathrm{disc}$, and CHGNet S.U.N. rank the same models differently. Which generator leads depends on the declared aim and on how crystal similarity is defined. In particular, the CHGNet S.U.N. leader ranks last on $\mathrm{GEM}\text{-}\mathrm{disc}$, because it concentrates on a few modes of the crystal space rather than discovering many distinct crystals.