- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors introduce new metrics to evaluate generative models for amorphous materials. The community so far has focused mainly on crystalline systems, while amorphous materials also have high technological relevance. The metrics and results indicate that generative models are not yet capable of sampling in-distribution configurations of given amorphous materials.
Critical strengths
The topic is relevant and timely. The metrics introduced are interesting for the community.
Critical weaknesses
The setting of generative models used in this paper is very different from how generative models are used for molecules or inorganic crystalline systems. The generative models here seem to be focused on a specific material (e.g., a-Si), rather than being trained on a large amount of known materials with a CSP or de novo generation task. This does not really become clear in the beginning, but it is rather mentioned indirectly towards the end of Section 2.1.
The actual task discussed in this paper does not become clear. If the models are trained on a given dataset of MD snapshots of a specific amorphous material and then used to sample more configurations that follow the same distributions (defined by the metrics introduced here), then what can be done with those models? It seems the relevant related work is more in the sampling community (e.g., Boltzmann generator literature) rather than in the generative models community, where tasks are CSP and de novo design of new materials. Both tasks, by definition, mean that structures of novel materials should be generated, which is not the case here.
Questions for authors
The metrics are discussed in Section 2.2, but it does not always become clear if the definition makes sense. For some distributions, EMD metrics are used, but also "relative error of mean CN". Is it guaranteed that CN follows a Gaussian distribution, rather than a skewed or even multimodal distribution, which would remove the meaning of "mean" and make a relative error of a mean meaningless?
The metrics table has a line "Overall (x MD)" which averages over the previous metrics. How do you justify this average? To me, it seems like the metrics are normalized to the leave-one-out noise floor, i.e., all values <1.0 indicate that the generative models generate structures that are under the expected variance, which seems non-optimal to me. Fluctuations are expected at finite temperatures, and this should be reflected in generated configurations. Regarding the averaging, it seems not clear why metrics <1.0 together with metrics >1.0 should "cancel out" to an overall value of e.g. 1.0 for CDVAE.
See the second paragraph in weaknesses. What is the task here and what is the relevance and use case for this task? It is clearly not a CSP-equivalent for amorphous materials (e.g., given the elements and a density, generate amorphous structures) or a de novo design task (generate novel amorphous structures incl. their composition).
6 comments
Response to Reviewer NqRb: questions
The metrics are discussed in Section 2.2, but it does not always become clear if the definition makes sense. For some distributions, EMD metrics are used, but also "relative error of mean CN". Is it guaranteed that CN follows a Gaussian distribution, rather than a skewed or even multimodal distribution, which would remove the meaning of "mean" and make a relative error of a mean meaningless?
No distributional assumption is made. CN is compared in three ways (Section 2.2):
- the Jensen-Shannon distance (JSD) between CN distributions, which is sensitive to skewed and multimodal shapes
- the relative error of mean CN
- the fraction of atoms whose CN lies outside the range of the ground-truth MD histogram The mean of a discrete distribution is defined whatever its shape, and in these systems it is the physically meaningful moment. a-C mixes sp² and sp³ bonding in a ratio set by density, so its CN distribution is split between threefold and fourfold atoms, and its mean CN is about 3 plus the sp³ fraction. In network glasses, the mean coordination produces a rigidity in constraint counting. The three comparisons are complementary. The JSD separates two distributions that share a mean but differ in shape, and the outlier fraction reveals defects too rare to shift the mean or the distribution distance.
The metrics table has a line "Overall (x MD)" which averages over the previous metrics. How do you justify this average? To me, it seems like the metrics are normalized to the leave-one-out noise floor, i.e., all values <1.0 indicate that the generative models generate structures that are under the expected variance, which seems non-optimal to me. Fluctuations are expected at finite temperatures, and this should be reflected in generated configurations. Regarding the averaging, it seems not clear why metrics <1.0 together with metrics >1.0 should "cancel out" to an overall value of e.g. 1.0 for CDVAE.
Thanks for bringing this up. We clarify with two observations: first, Overall×MD is the median deviation over the rows a method carries, not their mean, as stated in the caption of the per-metric profile figure for a-Si in the Appendix. Rows below and above 1.0 therefore do not cancel, and a median of 1.0 indicates that half of a method's rows lie within normal MD variation. The standings are the mean ranks, assigned from pooled per-structure ranks, not Overall×MD. The revision also states the median in the main table caption.
Second, each row is normalized by MD's leave-one-out noise floor, the deviation of one MD structure from the mean of the others. The reviewer is right that per-structure rows reward proximity to the reference, so a method that mode-collapsed toward the mean could score below 1.0. Phase-space coverage is therefore scored separately (Section 2.2). The entropy error |H_src − H_MD| increases whether a method's atomic environments are too narrow or too broad. The fraction of environments outside the MD manifold (1 − overlap) flags configurations unlike any in the reference.
Nevertheless, "beating MD" can be a feature. MD is produced without conditioning, whereas PDF-conditioned generation using a model typically reduces the error in PDF and other spectroscopic quantities. In this case, a metric where the deviation is below the MD variance is positive, and in fact a sign that the generative model is producing structures as desired.
See the second paragraph in weaknesses. What is the task here and what is the relevance and use case for this task? It is clearly not a CSP-equivalent for amorphous materials (e.g., given the elements and a density, generate amorphous structures) or a de novo design task (generate novel amorphous structures incl. their composition).
We responded to Weakness 2 above, but in summary, the task is to sample the amorphous configurations of a given material system at a given processing condition. Of the reviewer's two candidates, the CSP-equivalent (given the elements and a density, generate amorphous structures) is what every benchmark condition asks, at fixed composition and density. Our new NCBS experiment extends it to compositions without training structures. Models trained on 100 or 1,000 structures spanning 68 or 112 other compositions generate structures at five unseen compositions, and the order of the methods is the same as in the benchmark.
Importantly, it is dangerous to refer to "crystal structure prediction" for a system that, by definition, is not crystalline, which is why we did not refer to that as such in our work. Design given new compositions is a task that can only be solved if sampling is consistent and, thus, is strong motivation for this benchmark. The main relevance of our work lies in the use cases listed in our response to Weakness 2: structure determination against measured data; sampling configurations at realistic cell sizes and cooling rates where MD is most expensive; and creating samplers that transfer across compositions.
Setting: Thank you for adding this to the introduction.
Task: I do not agree with your task definition "Given elements and composition and a density, generate configurations". This would imply generalization across different elements and compositions, which is not the case. Your model does not have elements and composition as input. It is trained on one specific set of elements and a composition and samples amorphous configurations. The only input or condition for the model might actually be processing conditions or density, so I would phrase it "Trained on a specific set of elements and a specific composition, given the processing conditions (density or composition), generate configurations."
Cost at scale: You mention "slow cooling rates - where MD is most expensive". Please explain this in more detail? How can a "rate" or any dynamic process be modelled with your approach? Sampling is purely thermodynamic and there is no notion of dynamics, no barriers, no time scale. How can a trained generative model be used to evaluate e.g. phase changes or any other dynamic processes e.g. at slow cooling rates?
Related work: Thank you for pointing this out, this helps. Your systems are far from thermodynamic equilibrium, which I did not consider. However, data-based training of Boltzmann generators with data that does not contain the low-energy crystalline modes would still lead to a generative model similar to yours. Energy-based training and active learning would eventually discover lower-energy ordered phases and move away from amorphous samples.
Distributions: I agree that JSD is a good measure. I just don't understand what the meaning of "relative error of mean CD" would be in case of non-Gaussian distribitions.
Thank you for clarifying the MD comparisons. Please make it more clear that the median is used and add a discussion of what <1.0 for individual metrics means from your perspective.
Response to Reviewer NqRb: weaknesses
We thank the reviewer for the positive evaluation and comment. The modifications discussed here have been added to the manuscript. Questions are addressed in a separate comment.
We agree, and the revision states the setting in the first paragraph of the introduction. Generative models are trained from scratch on MD structures of one material system and sample that system's amorphous configurations at a given density or composition. This is the setting of amorphous structure modeling, where data-fitting and network-construction algorithms such as RMC, HRMC, and CRN have been the standard route to atomistic models for decades, yet continue to fail in several real-world characterization tasks. It is not the setting of crystal structure prediction (CSP) or de novo design, which are tasks focused on ordered crystals or molecules.
Task. Given a material system (its elements and composition) and a processing condition (here density or composition), generate configurations that are statistically indistinguishable from reference MD samples at that condition. Novelty is configurational. Following Section 2.2, a "novel" structure is one whose QUESTS overlap and entropy error are as close to MD as possible. This task has been quite limited in disordered systems. In contrast with inorganic crystal structures, where one can easily propose periodic arrangements that look sensible and use them for property calculations, in amorphous and disordered systems, proposing an arrangement of atoms is nearly impossible. This severely hinders computational studies of amorphous matter, which exhibits extremely important applications, yet remains challenging to simulate/analyze.
Examples on what the models are used for.
Related work. We agree that ensemble sampling is closely related, including Boltzmann generators (Noé et al., 2019) and conformer-ensemble generation for molecules (e.g., Jing et al., 2022). The revision adds it. One difference is important to highlight: a glass is kinetically arrested rather than thermodynamically stable, so sampling from Boltzmann distributions is not necessarily guaranteed to lead to amorphous structures. In many cases in amorphous materials, there is no target probability density to reweight against, so the reference must be samples of the protocol itself. For this reason, AmorphBench scores against MD ensembles instead of relying only on an energy function, which can favor the formation of crystalline structures that are incompatible with the desired distribution.