acceptodds
Under review as a conference paper at ICLR 2027

What Defines a High-Quality Generated Material? Stress-Testing Generative Models on Amorphous Matter

61 views

Abstract

Generative models for materials excel at producing crystals and molecules with targeted properties. However, most technologically relevant materials are metastable, defected, or disordered systems that challenge the definitions of uniqueness, stability, and novelty on which typical metrics and benchmarks rest. Here, we introduce AmorphBench, a protocol that evaluates how well generative models sample inorganic amorphous structures according to physical principles. We define 6 metric families and 21 quality metrics covering short- and medium-range order, topology, spectroscopic observables, mechanics, energetics, and phase-space coverage. By scoring properties and experimental observables rather than structural agreement alone, we expose multiple limitations of the learned distributions, where even single outlying atoms lead to outright unphysical behavior. Applying the protocol to 8 generative and structure-fitting methods across 5 systems and 14 density and composition conditions, we show that generative model quality for materials is far from saturated. Scaling tests indicate that performance improves with training data, a promising signal for amorphous systems, where reference datasets are expensive to obtain. We release the benchmark and its reference datasets to guide the development of generative models that meet the bars for data efficiency and sample quality required by real-world applications.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
generative models, sampling, physical sciences, materials science, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_NqRbOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors introduce new metrics to evaluate generative models for amorphous materials. The community so far has focused mainly on crystalline systems, while amorphous materials also have high technological relevance. The metrics and results indicate that generative models are not yet capable of sampling in-distribution configurations of given amorphous materials.

Critical strengths

The topic is relevant and timely. The metrics introduced are interesting for the community.

Critical weaknesses

The setting of generative models used in this paper is very different from how generative models are used for molecules or inorganic crystalline systems. The generative models here seem to be focused on a specific material (e.g., a-Si), rather than being trained on a large amount of known materials with a CSP or de novo generation task. This does not really become clear in the beginning, but it is rather mentioned indirectly towards the end of Section 2.1.

The actual task discussed in this paper does not become clear. If the models are trained on a given dataset of MD snapshots of a specific amorphous material and then used to sample more configurations that follow the same distributions (defined by the metrics introduced here), then what can be done with those models? It seems the relevant related work is more in the sampling community (e.g., Boltzmann generator literature) rather than in the generative models community, where tasks are CSP and de novo design of new materials. Both tasks, by definition, mean that structures of novel materials should be generated, which is not the case here.

Questions for authors

The metrics are discussed in Section 2.2, but it does not always become clear if the definition makes sense. For some distributions, EMD metrics are used, but also "relative error of mean CN". Is it guaranteed that CN follows a Gaussian distribution, rather than a skewed or even multimodal distribution, which would remove the meaning of "mean" and make a relative error of a mean meaningless?

The metrics table has a line "Overall (x MD)" which averages over the previous metrics. How do you justify this average? To me, it seems like the metrics are normalized to the leave-one-out noise floor, i.e., all values <1.0 indicate that the generative models generate structures that are under the expected variance, which seems non-optimal to me. Fluctuations are expected at finite temperatures, and this should be reflected in generated configurations. Regarding the averaging, it seems not clear why metrics <1.0 together with metrics >1.0 should "cancel out" to an overall value of e.g. 1.0 for CDVAE.

See the second paragraph in weaknesses. What is the task here and what is the relevance and use case for this task? It is clearly not a CSP-equivalent for amorphous materials (e.g., given the elements and a density, generate amorphous structures) or a de novo design task (generate novel amorphous structures incl. their composition).

6 comments
AuthorsOpenReview ↗

Response to Reviewer NqRb: weaknesses

We thank the reviewer for the positive evaluation and comment. The modifications discussed here have been added to the manuscript. Questions are addressed in a separate comment.

The setting of generative models used in this paper is very different from how generative models are used for molecules or inorganic crystalline systems. The generative models here seem to be focused on a specific material (e.g., a-Si), rather than being trained on a large amount of known materials with a CSP or de novo generation task. This does not really become clear in the beginning, but it is rather mentioned indirectly towards the end of Section 2.1.

We agree, and the revision states the setting in the first paragraph of the introduction. Generative models are trained from scratch on MD structures of one material system and sample that system's amorphous configurations at a given density or composition. This is the setting of amorphous structure modeling, where data-fitting and network-construction algorithms such as RMC, HRMC, and CRN have been the standard route to atomistic models for decades, yet continue to fail in several real-world characterization tasks. It is not the setting of crystal structure prediction (CSP) or de novo design, which are tasks focused on ordered crystals or molecules.

The actual task discussed in this paper does not become clear. If the models are trained on a given dataset of MD snapshots of a specific amorphous material and then used to sample more configurations that follow the same distributions (defined by the metrics introduced here), then what can be done with those models? It seems the relevant related work is more in the sampling community (e.g., Boltzmann generator literature) rather than in the generative models community, where tasks are CSP and de novo design of new materials. Both tasks, by definition, mean that structures of novel materials should be generated, which is not the case here.

Task. Given a material system (its elements and composition) and a processing condition (here density or composition), generate configurations that are statistically indistinguishable from reference MD samples at that condition. Novelty is configurational. Following Section 2.2, a "novel" structure is one whose QUESTS overlap and entropy error are as close to MD as possible. This task has been quite limited in disordered systems. In contrast with inorganic crystal structures, where one can easily propose periodic arrangements that look sensible and use them for property calculations, in amorphous and disordered systems, proposing an arrangement of atoms is nearly impossible. This severely hinders computational studies of amorphous matter, which exhibits extremely important applications, yet remains challenging to simulate/analyze.

Examples on what the models are used for.

  • Structure determination. Atomistic models consistent with measured data are the standard product of amorphous structure determination, and RMC and HRMC exist to produce them. Our conditional tasks (suffix c) add a gradient toward a target PDF at each denoising step, testing whether an experimental observable can steer a learned prior. This is the main bottleneck of a large number of materials characterization tasks, including in top-tier synchrotron facilities across the world: data interpretation is the slow step that has not been automated yet.
  • Cost at scale. Simulations cost grows with cell size and inverse cooling rate (i.e., long trajectories), whereas a trained sampler's cost per structure can be easily parallelized across atoms and are prepared. Samplers can therefore bypass costly simulations for realistic systems - large cells and slow cooling rates - where MD is most expensive.
  • Transfer across compositions. Samplers trained on many compositions can generate at new ones, which our new NCBS experiment tests (response to Question 3).

Related work. We agree that ensemble sampling is closely related, including Boltzmann generators (Noé et al., 2019) and conformer-ensemble generation for molecules (e.g., Jing et al., 2022). The revision adds it. One difference is important to highlight: a glass is kinetically arrested rather than thermodynamically stable, so sampling from Boltzmann distributions is not necessarily guaranteed to lead to amorphous structures. In many cases in amorphous materials, there is no target probability density to reweight against, so the reference must be samples of the protocol itself. For this reason, AmorphBench scores against MD ensembles instead of relying only on an energy function, which can favor the formation of crystalline structures that are incompatible with the desired distribution.

AuthorsOpenReview ↗

Response to Reviewer NqRb: questions

The metrics are discussed in Section 2.2, but it does not always become clear if the definition makes sense. For some distributions, EMD metrics are used, but also "relative error of mean CN". Is it guaranteed that CN follows a Gaussian distribution, rather than a skewed or even multimodal distribution, which would remove the meaning of "mean" and make a relative error of a mean meaningless?

No distributional assumption is made. CN is compared in three ways (Section 2.2):

  • the Jensen-Shannon distance (JSD) between CN distributions, which is sensitive to skewed and multimodal shapes
  • the relative error of mean CN
  • the fraction of atoms whose CN lies outside the range of the ground-truth MD histogram The mean of a discrete distribution is defined whatever its shape, and in these systems it is the physically meaningful moment. a-C mixes sp² and sp³ bonding in a ratio set by density, so its CN distribution is split between threefold and fourfold atoms, and its mean CN is about 3 plus the sp³ fraction. In network glasses, the mean coordination produces a rigidity in constraint counting. The three comparisons are complementary. The JSD separates two distributions that share a mean but differ in shape, and the outlier fraction reveals defects too rare to shift the mean or the distribution distance.

The metrics table has a line "Overall (x MD)" which averages over the previous metrics. How do you justify this average? To me, it seems like the metrics are normalized to the leave-one-out noise floor, i.e., all values <1.0 indicate that the generative models generate structures that are under the expected variance, which seems non-optimal to me. Fluctuations are expected at finite temperatures, and this should be reflected in generated configurations. Regarding the averaging, it seems not clear why metrics <1.0 together with metrics >1.0 should "cancel out" to an overall value of e.g. 1.0 for CDVAE.

Thanks for bringing this up. We clarify with two observations: first, Overall×MD is the median deviation over the rows a method carries, not their mean, as stated in the caption of the per-metric profile figure for a-Si in the Appendix. Rows below and above 1.0 therefore do not cancel, and a median of 1.0 indicates that half of a method's rows lie within normal MD variation. The standings are the mean ranks, assigned from pooled per-structure ranks, not Overall×MD. The revision also states the median in the main table caption.

Second, each row is normalized by MD's leave-one-out noise floor, the deviation of one MD structure from the mean of the others. The reviewer is right that per-structure rows reward proximity to the reference, so a method that mode-collapsed toward the mean could score below 1.0. Phase-space coverage is therefore scored separately (Section 2.2). The entropy error |H_src − H_MD| increases whether a method's atomic environments are too narrow or too broad. The fraction of environments outside the MD manifold (1 − overlap) flags configurations unlike any in the reference.

Nevertheless, "beating MD" can be a feature. MD is produced without conditioning, whereas PDF-conditioned generation using a model typically reduces the error in PDF and other spectroscopic quantities. In this case, a metric where the deviation is below the MD variance is positive, and in fact a sign that the generative model is producing structures as desired.

See the second paragraph in weaknesses. What is the task here and what is the relevance and use case for this task? It is clearly not a CSP-equivalent for amorphous materials (e.g., given the elements and a density, generate amorphous structures) or a de novo design task (generate novel amorphous structures incl. their composition).

We responded to Weakness 2 above, but in summary, the task is to sample the amorphous configurations of a given material system at a given processing condition. Of the reviewer's two candidates, the CSP-equivalent (given the elements and a density, generate amorphous structures) is what every benchmark condition asks, at fixed composition and density. Our new NCBS experiment extends it to compositions without training structures. Models trained on 100 or 1,000 structures spanning 68 or 112 other compositions generate structures at five unseen compositions, and the order of the methods is the same as in the benchmark.

Importantly, it is dangerous to refer to "crystal structure prediction" for a system that, by definition, is not crystalline, which is why we did not refer to that as such in our work. Design given new compositions is a task that can only be solved if sampling is consistent and, thus, is strong motivation for this benchmark. The main relevance of our work lies in the use cases listed in our response to Weakness 2: structure determination against measured data; sampling configurations at realistic cell sizes and cooling rates where MD is most expensive; and creating samplers that transfer across compositions.

Setting: Thank you for adding this to the introduction.

Task: I do not agree with your task definition "Given elements and composition and a density, generate configurations". This would imply generalization across different elements and compositions, which is not the case. Your model does not have elements and composition as input. It is trained on one specific set of elements and a composition and samples amorphous configurations. The only input or condition for the model might actually be processing conditions or density, so I would phrase it "Trained on a specific set of elements and a specific composition, given the processing conditions (density or composition), generate configurations."

Cost at scale: You mention "slow cooling rates - where MD is most expensive". Please explain this in more detail? How can a "rate" or any dynamic process be modelled with your approach? Sampling is purely thermodynamic and there is no notion of dynamics, no barriers, no time scale. How can a trained generative model be used to evaluate e.g. phase changes or any other dynamic processes e.g. at slow cooling rates?

Related work: Thank you for pointing this out, this helps. Your systems are far from thermodynamic equilibrium, which I did not consider. However, data-based training of Boltzmann generators with data that does not contain the low-energy crystalline modes would still lead to a generative model similar to yours. Energy-based training and active learning would eventually discover lower-energy ordered phases and move away from amorphous samples.

Distributions: I agree that JSD is a good measure. I just don't understand what the meaning of "relative error of mean CD" would be in case of non-Gaussian distribitions.

Thank you for clarifying the MD comparisons. Please make it more clear that the median is used and add a discussion of what <1.0 for individual metrics means from your perspective.

Reviewer_7n9eOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper focuses on the problem of inverse design of amorphous materials with target properties, and proposes a benchmark of existing conventional and generative methods. Five amorphous material families are covered at 14 density or composition conditions. With the benchmark results, the paper presents insights into the gap of existing methods, as well as promising directions for improving the performance.

Critical strengths
  1. The paper presents a quite comprehensive benchmark covering 5 amorphous material families and 14 condition types.
  2. The benchmark protocol is described in detail, enhancing the reproducibility of the benchmark and allowing easy future expansion.
Critical weaknesses
  1. The scale of the dataset used in the proposed benchmark seems relatively small, especially compared to the training and evaluation datasets originally used in many of the compared methods.
Questions for authors
  1. The authors could reiterate what they observed during their benchmark, where a relatively small scale dataset is used to train and evaluate the comparison methods.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 7n9e

We thank the reviewer for the positive evaluation and comment. The modifications discussed here have been added to the manuscript.

The scale of the dataset used in the proposed benchmark seems relatively small, especially compared to the training and evaluation datasets originally used in many of the compared methods.

The benchmark works in the low-data regime on purpose to emulate real-world problems in materials characterization. Unfortunately, thirty-five training structures per condition matches the current reality of computational studies in amorphous materials, where each reference structure requires an expensive simulation (Section 3.5). Each structure has 216 to 500 atoms whose environments are learned individually, so a training set holds up to 25,000 distinct atomic environments. To test whether the dataset size biases the comparison, we added an ablation on our NCBS dataset. We trained models on 100 or 1,000 structures of about 3,000 atoms each (up to 3M atoms) and generated structures at five compositions absent from training. Separately, we trained models in-distribution on 35 structures per composition. The order is unchanged in every setting: GLASS-u, CDVAE, DM²-u, AMDEN-u. More data alone does not close the gap for the models developed on larger datasets. Overall×MD is the median normalized deviation over the 13 rows, averaged over the five compositions. From 100 to 1,000 training structures, at an equal sample budget, the performance changes as follows:

  • CDVAE: increases from 2.53 to 4.13
  • DM²-u: decreases from 28.5 to 25.0
  • AMDEN-u: decreases from 27.2 to 25.5
  • GLASS-u: stays at 2.73 and 2.68

CDVAE's increase is likely not an artifact. We regenerated it starting from its lowest-validation checkpoint instead of its last one and both values, to 1.67 and 2.83, thus still exhibiting the increase.

The authors could reiterate what they observed during their benchmark, where a relatively small scale dataset is used to train and evaluate the comparison methods.

The benchmark revealed three failure modes at this scale, and the ablation shows that they are not artifacts of the dataset size.

  • First, structural metrics alone suggest the problem is nearly solved. Energy, mechanical properties, and topology show that most models still miss the distribution of amorphous configurations.
  • Second, sample quality and validity are decoupled. CDVAE ties for second while only 73% of its structures are valid, and every invalid metallic structure fails on a foreign element.
  • Third, model performance does not transfer across bonding types. Covalent networks fail through coordination defects, bond-angle errors, and overlapping atoms, whereas metallic glasses match packing but fail on energetics and plastic response. On NCBS, ten times more training data leaves the order unchanged and does not let the models developed on larger datasets catch up (response to Weakness 1). The one fine-tuned model, MatterGen-c, places fourth (3.96 ± 0.08), behind three models trained from scratch on 35 structures per condition.

All these comments were clarified on the manuscript.

I would like to thank the authors for their detailed response, which solved the concerns raised in my original review. I will maintain my positive rating.

Reviewer_Z4bgOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce AmorphBench, a novel benchmark for amorphous material samplers. Motivated by the growing development of generative AI for materials and the challenges in evaluating these tools with respect to one another, AmorphBench identifies 21 metrics for comparing these proposal mechanisms in a way which is grounded in computationally obtained simulation data. The authors focus special attention on validity and yield as well as median performance in an attempt to quantify how often different models produce usable samples and reveal the striking reality that, despite reasonable replication of reference statistics, many generative models do struggle to sample valid amorphous materials despite strong median scores and there appears to be much room for improvement.

The benchmarking framework is the main contribution of the work. The authors take care to gather testing MD data judiciously via independent melt-quench simulations as opposed to taking correlated frames from a single trajectory. Overall, AmorphBench serves as a novel, albeit data-scarce, benchmarking platform for generative amorphous materials. Overall a very strong piece of scientific work.

Critical strengths

A particular strength of this work is the thoughtful construction of which suite of metrics would most accurately score amorphous material sampling in a broad sense. Unlike inorganic crystals, amorphous materials are much more complex and disordered. Each aspect of AmorphBench is backed by solid reasoning lending credibility to the proposed benchmark and I believe that assessing energetics, local environments, ring statistics, and mechanical properties is crucial for scaling generative models to the amorphous setting since, as demonstrated for CDVAE, it can be the case that a particular sampler demonstrates strong performance in some aspects while failing in others. Even more complicated is the fact that different models perform at varying levels on different systems and AmorphBench proposes a diverse set of test systems to address this which I think is a major strength.

I also appreciate that the work is clearly documented and thorough enough to be replicated with clearly stated training details for each model as well as descriptions on the budgets and checkpoint selection for each model. I also appreciate that the authors have linked the code for the benchmark.

Critical weaknesses

The primary weakness of AmorphBench is its data-scarce nature. Scoring on ten structures per condition as described in the paper is quite limiting. I understand that amorphous materials is a data-scarce field in general and this is clearly stated as a limitation but I find it somewhat concerning especially when considering that deviation from MD is explicitly penalized. It seems that a big assumption which is cooked into this benchmark is that the 10 chosen MD references are representative of the entire ensemble. Ideally I would like to see an ablation for at least a single test system to demonstrate that, upon the introduction of more data, the rankings are largely unchanged.

Questions for authors
  • How come it was decided to evenly weight each rank? Would a practitioner weight each of these normally when deciding between generative algorithms? It seems that the decision of how to weight each component was taken to be uniform for simplicity.
  • Why is there no report on the computational cost of each method? I think that this would be a useful addition to GenAI for materials leaderboards. It seems that GLASS is the most promising neural sampler for amorphous materials but is it efficient in terms of inference time compared to other generative models or MC? I think this is especially relevant with respect to section 3.4 where a discussion on yield is given.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Z4bg

We thank the reviewer for the positive evaluation and comment. The modifications discussed here have been added to the manuscript.

The primary weakness of AmorphBench is its data-scarce nature. Scoring on ten structures per condition as described in the paper is quite limiting. [...] Ideally I would like to see an ablation for at least a single test system to demonstrate that, upon the introduction of more data, the rankings are largely unchanged.

We ran the requested ablation on a separate dataset, sodium-calcium borosilicate (NCBS, Na₂O–CaO–B₂O₃–SiO₂; Section 3.5), with about 3,000 atoms per structure. We trained the four unconditional generative models (GLASS-u, CDVAE, DM²-u, and AMDEN-u) from scratch and scored them with the benchmark's metrics and ranking. The ablation tests both parts of the concern.

More MD reference structures. In-distribution, we trained on 35 structures per composition at three compositions, as in the benchmark, and scored against 10 MD test structures and again against 60. At held-out compositions, we scored against 10 MD structures per composition and again against 100.

More training data. We trained on 100 or 1,000 NCBS structures (up to 3M atoms) and generated at five compositions absent from training.

The results show that, at large dataset sizes, the order of the main benchmark remain entirely unaltered, with the performance score nearly the same for all methods. The revision adds Appendix E.7 fully describing these results and providing all details of the benchmark, including the results.

How come it was decided to evenly weight each rank? Would a practitioner weight each of these normally when deciding between generative algorithms? It seems that the decision of how to weight each component was taken to be uniform for simplicity.

Correct - the uniform weighting is the most general choice that we could have made. Any non-uniform weighting encodes a single use case: a practitioner refining a structure against diffraction data would weight the diffraction rows, one computing mechanical response would weight the mechanics rows, but our benchmark has no need to encode that preference. In a way, weighting each metric row equally is also the conservative choice, because it favors the standings toward the rows on which methods agree. Eleven of the 21 metric rows are structural (short-range order, topology, and diffraction), where the leading models reproduce the reference within the MD variability. Only two are energetic, where they remain furthest from it (Section 4). This is the reason why we report per-row deviations alongside the mean rank, so that practitioners can compare methods on the rows relevant to their task.

Why is there no report on the computational cost of each method? I think that this would be a useful addition to GenAI for materials leaderboards. It seems that GLASS is the most promising neural sampler for amorphous materials but is it efficient in terms of inference time compared to other generative models or MC? I think this is especially relevant with respect to section 3.4 where a discussion on yield is given.

We agree, and the revision adds a computational-cost table (Appendix C.10, Table 9). For every method, it reports the one-time training cost, the cost of one structure per system, and the total for the benchmark, with the hardware each step ran on. Times come from our production runs on a single NVIDIA RTX A6000 GPU per model.

Per sampling run, every generative model samples in seconds to minutes of GPU time:

  • GLASS: 5-10 s unconditional, 14-38 s conditional
  • AMDEN: 4-8 s
  • CDVAE: 38-89 s
  • DM²: 1.7-4.8 min
  • MatterGen: 5-14 min

RMC and HRMC are Monte Carlo methods, and were run at up to 71 h and 103 to 504 h of CPU time (Intel Xeon 6736P) on each structure. Both ran at fixed budgets.

Training costs 10 to 74 GPU-hours per model family, summed over the five systems. GLASS is the most expensive at 74 GPU-hours. MatterGen's pretraining cost was not included.

On yield (Section 3.4), a usable structure costs the per-draw time divided by the yield. GLASS fixes composition and density and passes every validity criterion, so it pays no such factor. CDVAE's lower yield (73% valid overall) increases the effective sampling time by 1.4 times. MatterGen's density filter accepted 104 of 734 conditional draws, so sampling an accepted structure takes 9 to 94 GPU-min.

These discussions were added to the manuscript PDF (revised text in blue colors)

Reviewer_Z4bgOpenReview ↗

Response to Authors

I thank the authors for their work to adequately answer my questions and concerns as well as for running additional experiments to a) report the computational cost of each method and b) demonstrate that 10 reference configurations is sufficient to rank generative engines for amorphous systems. I maintain my recommendation to clearly accept this work to the conference.

We once again thank the reviewer for the discussion and suggestions that improved the manuscript!

Comments on the paper

3 comments
~Lei_Jiafeng1OpenReview ↗

Some questions about this work.

We are really interested in your work! After reading the paper, we have three questions and would love to hear your thoughts:

  1. Test-set contamination. We noticed that Appendices A.7 and C.1 mention that the PDF-guidance hyperparameters for MatterGen on a-C, a-Si, and SiO₂ were selected using test-split references. Could this introduce evaluation leakage and potentially affect the reported results? Have the authors considered rerunning these experiments with validation-only hyperparameter selection?

  2. Fairness of model comparisons. GLASS pools training structures across conditions (105–140 structures per model), whereas DM² and CDVAE are trained separately on 35 structures per condition. Appendix C.6.1 also suggests that DM² may be undertrained. Could these differences influence the reported rankings? Would the conclusions remain robust under matched training information and sufficiently converged baselines?

  3. Novelty and incremental value. We noticed that many of the 21 metrics are established physical observables, and prior works such as Dismai-Bench (2024) and A Generative Diffusion Model for Amorphous Materials (2026) have already employed similar structural, mechanical, and diversity-based evaluations. We are curious what additional diagnostic value the full 21-metric framework provides beyond these existing approaches. Have the authors considered a metric-family ablation or redundancy analysis to demonstrate which metrics reveal previously overlooked failure modes or meaningfully affect the model rankings?

We appreciate the work and would be grateful for any clarification. Thank you!

Thank you for the careful reading.

  1. We reran MatterGen's guidance selection with validation references. As before, we swept all 28 (ρ, σ) settings per condition against validation structures and required ten accepted structures per setting. Two of the seven conditions kept their optimal settings and four changed (a-C 1.9 and 2.4, a-Si 1.8 and 2.3). During generation (200 draws to fill 10 accepted structures), the new settings improve some metrics and worsen others. For example, a-C 2.4's XRD error falls from 3.5 to 1.4 and g(r) from 3.6 to 2.7, while energy per atom rises from 15 to 26.

  2. Pooling does not explain the ranking, as AMDEN also trains on each system's pooled conditions and ranks sixth. To provide additional evidence, we also trained GLASS-u, CDVAE, DM²-u and AMDEN-u on identical borosilicate training sets of 100 and 1,000 structures (~3,000 atoms each; Sec. 3.5), plus an in-distribution version that mirrors the main benchmark. The order matched the main benchmark in every setting. Mean ranks at 100/1,000 structures were GLASS-u 1.48/1.28, CDVAE 1.58/1.89, DM²-u 3.35/3.17 and AMDEN-u 3.58/3.66. We have not trained it longer, but ten times more data moves it only from 3.35 to 3.17, and DM²-c and CDVAE are already reported as tied.

  3. The observables are established by design. What is new is the protocol:

  • every row is normalized by MD's own spread
  • energies and relaxation are scored on the raw draws
  • Metrics in Dismai-Bench have very few overlap with ours, which are rooted in typical conventions in amorphous materials.
  • The DM² paper has some properties and information metrics, but tends to rely on more system-specific metrics.

In addition to more comprehensive analysis of potential energies, spectra, properties, and structure, we also:

  • provide one frozen scorer covers eight methods, including baseline algorithms such as RMC, HRMC and CRN;
  • rank with bootstrap uncertainties, and validity is reported separately.

Following your comment, we ran a metric-family ablation. Dropping any one family continues to leave GLASS-c first. The clearest redundancies are shear vs Young's modulus and XRD vs ND, as expected from their interplay.

We added this analysis to the appendix.

Thank you very much for the detailed response and the additional experiments! We really appreciate the effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.