What Defines a High-Quality Generated Material? Stress-Testing Generative Models on Amorphous Matter

Generative models for materials excel at producing crystals and molecules with targeted properties. However, most technologically relevant materials are metastable, defected, or disordered systems that challenge the definitions of uniqueness, stability, and novelty on which typical metrics and benchmarks rest. Here, we introduce AmorphBench, a protocol that evaluates how well generative models sample inorganic amorphous structures according to physical principles. We define 6 metric families and 21 quality metrics covering short- and medium-range order, topology, spectroscopic observables, mechanics, energetics, and phase-space coverage. By scoring properties and experimental observables rather than structural agreement alone, we expose multiple limitations of the learned distributions, where even single outlying atoms lead to outright unphysical behavior. Applying the protocol to 8 generative and structure-fitting methods across 5 systems and 14 density and composition conditions, we show that generative model quality for materials is far from saturated. Scaling tests indicate that performance improves with training data, a promising signal for amorphous systems, where reference datasets are expensive to obtain. We release the benchmark and its reference datasets to guide the development of generative models that meet the bars for data efficiency and sample quality required by real-world applications.