- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes GEMBench to evaluate crystal generation from two different goals: distribution matching and new material discovery. The authors show that existing metrics such as S.U.N. may not fully reflect discovery ability, and different evaluation metrics can give different model rankings.
Critical strengths
- The paper proposes new evaluation metrics to better assess crystal generation.
- The authors introduce a comprehensive evaluation protocol for crystal generation models.
Critical weaknesses
see my question
Questions for authors
- For a new benchmark, shouldn't there be new datasets or OOD tests? Is proposing new evaluation metrics enough?
- How well do these metrics correlate with real material discovery results, such as experimentally validated structures?
- How can the authors prove that these metrics are more meaningful than existing ones like S.U.N.? Different model rankings do not necessarily mean better evaluation.
3 comments
Q3. How can you prove these metrics are more meaningful than S.U.N.?
Different rankings alone would not prove a better metric. The argument is that S.U.N. leaves an important quality axis unmeasured. Some public models already climb higher S.U.N. rates than others by a behaviour S.U.N. does not reveal. That behaviour looks questionable once diversity of the stable set is measured.
S.U.N. is essentially a yield of crystals that pass a hull-style stability gate (plus discrete uniqueness/novelty bits). It does not account for the shape of the distribution of those stable materials. A generator can therefore raise S.U.N. by concentrating mass on a thin slice of crystal space and emitting many near-duplicates that still clear the hull cut and a StructureMatcher uniqueness bit. Under S.U.N. that looks like success, and for a developer oriented on S.U.N. the mode collapse is invisible.
Example from paper.
On the same MP-20 generation batches:
- MiAD leads CHGNet S.U.N.\ () and has a high on-hull rate under our panel ().
- DiffCSP++ has a lower on-hull rate () but a higher EaH$_0$ discovery term under the physical kernel ( vs MiAD’s ), and it leads overall physical GEM-disc ().
- Globally, MiAD is last on GEM-disc (Rank ) despite leading S.U.N.
That pattern is exactly the failure mode above: more stable crystals does not imply more really different stable crystals. Where S.U.N. tracks a rate of stable and unique materials, GEM-disc uses a Vendi score on the accepted set to count how many modes in materials space are covered by stable materials, at fixed budget. Collapse among accepted stables is therefore punished. GEM-match separately asks whether the accepted set agrees with the training-dataset law in rate and shape, which S.U.N. does not ask at all.
Modern crystal generators are already optimizing quantities like S.U.N. If that scalar ignores distributional collapse of the stable part, leaderboards can reward models that look productive while covering only a thin slice of crystal space. GEMBench makes that failure visible and gives developers scores that speak not only to a model's ability to find stable crystals, but to two practical desires. GEM-match tracks finding more materials similar to the dataset, without exact copies. GEM-disc tracks covering as much of the materials domain as possible. Different rankings alone would not prove progress. The point is that these scores measure a quality axis that S.U.N. leaves out, and that axis already reorders public checkpoints.
We hope the answers above resolve your three questions on contribution scope, experimental grounding of the panel, and why GEM-match/GEM-disc go beyond S.U.N. If so, we would be grateful if you would consider raising your score.
Thank you for your clear explanations. I appreciate your efforts in addressing my concerns, and I will raise my score accordingly.
Thank you for the three clarifying questions. We answer each below.
Q1. Is this a benchmark without a new dataset?
Yes: GEMBench does not release a new crystal dataset. The generators and training corpora we score are already public (MP-20, Alex-MP-20, and published checkpoints). What we propose is a way to score those generations under two explicit aims, matching and discovery.
We still call the result a benchmark because the object is larger than a closed-form metric formula. In our logic, a usable score for crystal generation needs two non-trivial pieces of infrastructure before any equation can be trusted:
As a result we do obtain closed-form expressions for GEM-match and GEM-disc, plus a concrete algorithm to compute them. Because that algorithm rests on the stability panel and calibrated kernels above, we treated the whole package as a benchmark. If the missing new dataset makes the word benchmark misleading, we are happy to rename the contribution to an evaluation protocol or metric suite. The contribution is an evaluation protocol built around a stability panel and similarity kernels, and metrics that use them for aim-oriented quality measurement.
Q2. How well do these metrics correlate with experimentally validated discovery?
GEM-match and GEM-disc cannot themselves be verified as experimental discovery rates. They score generators as distributions under a fixed panel and kernel, rather than as wet-lab outcomes. What is experimentally checkable, and what we already report in the paper, is the stability panel that selects which materials enter those scores.
Appendix A.7 reports how accurate the stability tests are on real, experimentally known solids. On a hold-out catalog of 100 known solids (disjoint from tuning), the NPT screen reaches overall accuracies of 0.84 / 0.88 / 0.97 at 50 / 300 / 600~K (tuning: 0.93 / 0.90 / 0.99). GEM-match and GEM-disc then measure generative quality only for crystals that pass these tests. A model therefore cannot reach high scores without finding materials that pass a filter already checked on experimentally known solids.