acceptodds
Under review as a conference paper at ICLR 2027

Representations from Pretrained Machine-Learning Interatomic Potentials as Coarse Coordinates for Material Generation and Evaluation

Abstract

Generative machine learning is increasingly tested for inorganic materials generation. Most models and the corresponding evaluation approaches thereby rely on simple forms of crystal structure representations. In this paper, we showcase the power of integrating atom-averaged features from pretrained Machine-Learning Interatomic Potentials (MLIPs), such as MACE, into these tasks. We first introduce the Coarse-Fine Transport Distance (CFTD), a distance measure that captures both quality and novelty of materials created with generative models in a single distribution-based evaluation framework. It uses two different featurizers, directly deriving the quality component from coarse MACE representations. We showcase CFTD's versatility in capturing crystal structure quality while also detecting memorization and compare it with the recently introduced continuous SUN metrics. We further illustrate the power of MLIP representations by directly adding them to the generation process itself and show that coarse MACE features can be used as helpful guidance for a material generative model.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Machine Learning Interatomic Potentials, Materials Generation, Generative Models, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_66ANOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CFTD, a metric for crystal generation evaluation. The metric has two views. A fine-grained one flags crystals too close to training data, and thus evaluates novelty using a contrastive GNN. A second, coarse view uses embeddings from MACE, a foundation model trained on DFT data from the Materials Project, to flag crystals too far from training data, and thus measures quality. Both views are combined into a single score using optimal transport: crystals in a “safe zone” between the two thresholds are not penalized, so the metric avoids the artificial quality-novelty trade-off caused by using a single feature space. Using toy experiments, the authors show that CFTD catches mode collapse better than cSUN and broken crystals better than TNovD, and that it detects memorization. They also rank 8 existing generators with CFTD and show that relaxation improves quality but increases memorization, and that some rankings differ strongly from cSUN (e.g., Chemeleon2). They also show that feeding MACE embeddings of training crystals into a flow-matching generator gives more stable but less novel crystals.

Critical strengths
  • The proposed CFTD metric detects simulated mode collapse (distribution shifts) that cSUN misses. It also penalizes broken or unrealistic crystals (lattice strain, noise, bad substitutions) more strongly than TNovD.
  • It combines two competing goals, novelty and quality, in a single distributional metric without a forced trade-off between them, thanks to the separate feature spaces and the “safe zone”.
  • It is computationally cheaper than DFT-based evaluation: it reuses pretrained MACE features, with no costly DFT or energy-above-hull computations, much as FID reuses a pretrained classifier.
  • It also detects memorization and overfitting, which makes it a potential checkpoint-selection signal.
Critical weaknesses
  • Limited novelty. CFTD builds directly on TNovD and reuses its optimal transport framework and contrastive GNN. The main additions are a second feature space based on MACE and a zone with no penalty. It is mostly a combination of existing methods rather than a new approach.
  • Limited experiments. The comparisons with TNovD and cSUN rely mostly on toy experiments, and other metrics (FID-style material metrics, precision/recall) are not considered. Also, CFTD ranks some generators very differently from cSUN (e.g., Chemeleon2), but the authors do not check what causes these differences.
  • Quality is only measured indirectly. CFTD uses MACE features as a proxy for stability, but it is never compared to DFT energy above hull on generated crystals. So it is unclear how well it reflects real stability.
  • Generalization and scalability are not fully shown. CFTD is only evaluated on MP20 and Perov-5, with no larger structures. The MP20-trained contrastive GNN is reused on Perov-5, but only for toy tests, so transfer across datasets is not studied systematically. Exact optimal transport scales as n3n^3; the authors mention faster alternatives but do not test them or report runtimes.
  • Only one feature representation. The coarse view only uses MACE-MP-0b3 with mean pooling. It is not compared to other pretrained MLIPs or classical structural descriptors, and the authors themselves say the pooling choice is arbitrary. So it is hard to know if the gain comes from MACE itself or just from adding a second, physics-based feature space.
  • Possible evaluation bias. MACE is used to evaluate, relax, and condition the generated crystals, so a MACE-based metric could favor the MACE-conditioned generator. LeMat-GenBench uses other MLIPs, which helps, but an independent check (another representation or DFT) would be more convincing.
Questions for authors
  • Does CFTD correlate with DFT energy above hull on generated crystals, even on a small subset?
  • How does CFTD compare to other distributional or memorization-aware metrics, such as FID-style material metrics or precision/recall?
  • Did you try other pretrained MLIPs, classical descriptors, or other pooling choices for the coarse view? Does the gain come from MACE specifically, or from any physics-based feature space?
  • How do CFTD and its runtime behave on larger datasets and larger unit cells (exp: MPTS-52)? How well does the contrastive GNN generalize to these datasets?
  • Since MACE is used to evaluate, relax, and condition the crystals, could this introduce a bias in favor of the MACE-conditioned generator? Would the CFTD results hold with a different pretrained representation, beyond the LeMat-GenBench evaluation?
4 comments

We thank the reviewer for their constructive and helpful review. We answer point by point.

  • Limited Novelty In contrast to TNovD, we added a stability-aware featurizer, changed the training of the contrastive GNN (Pettifor augmentation), proved coverage-novelty bounds, performed extensive analysis on the MACE features, ablated this to other MLIPs, and used these MACE features for generative modeling. Note that for TNovD, the score barely changes as we perturb the lattice or add Gaussian fractional noise. Further, every quality decrease of TNovD corresponds to a memorization increase, which is not the case for CFTD.

    With this, we have several new findings, namely that CFTD spots Chemeleon2, the RL fine-tuned model, as off-distribution, we found that MACE features serve as good coarse coordinates, and found that FID-style metrics do not detect memorization. We spotted an insufficiency in instance-level metrics (cSUN) and found that feeding in empirical MACE features increases stability but lowers novelty, and that naively learned ones can be used to get a more novel sampler.

  • Limited experiments: Good suggestion. We added a distributional metric, FID from LeMatGen, to argue that such metrics would not penalize the training set. This can be seen in Fig. 17 and Appendix L. For TNovD, you can see from their figures or relative differences that TNovD with the contrastive featurizer is not responsive to large perturbations in the same way our CFTD is (see Section 3). For Chemeleon2, we believe we have found the reason: It mode collapses with respect to composition (see Appendix K), which explains, in our opinion, why distributional metrics detect this.

  • Quality of CFTD/Question regarding correlation Good question. We first want to clarify that we see our metric as complementary to classical thermodynamic stability measures. We checked in Fig. 18 how the MACE features relate to the energetic differences and, similarly, how the CFTD relates. The MACE features are very responsive, the CFTD reacts from a perturbation of 0.1 eV/atom onwards. This was tested with a different MLIP, so there is no MACE overlap here. We would say it can tell the difference between 0.2 and 0.05 eV/atom, however we need to remark that it is not easy to extract this effect since CFTD measures strictly more than stability. Further, we also computed MACE and DFT convex hulls for 6 generated samples and observed that the MACE metastability assessment is accurate for 5 out of 6 generated samples. You can find these DFT results in Appendix O and P.

  • Generalization and Scalability The main scaling bottleneck is not large crystals but a large number of generated crystals. We therefore checked how the run time of featurization and CFTD behaves when we scale up the number of generated samples to 50k. In that case, it is still feasible in roughly 15 mins (Appendix M), which is still cheaper than relaxation or thermodynamic stability measures. We still think this is an important area of future research, and speeding it up substantially is important.

  • Only MACE We added CFTD-UMA, where we mirrored our MACE feature construction with the UMA MLIP. We see that they mostly have the same ranking, with the exception of Chemeleon2 in the raw case, but in the relaxed case UMA also identifies the Chemeleon2-produced samples as out of distribution.

    We also compared the prediction of the lowest structure in polymorphs (using DFT energies) with a purely structural featurizer AMD, where we show that MACE features strongly outperform AMD at predicting the minimum energy structure, see Appendix O.

  • Evaluation bias We split up the LeMatGen into the MACE+ORB and ORB score (UMA energy hull does not seem to be reliable in the current LeMatGen version, see for instance this commit), so we see that the improved stability holds up under independent MLIP evaluations. Further, we added independent DFT assessments for a small number of structures in Appendix P.

  • Other distributional metrics Good point. There are indeed a few, but none of them has emerged as standard. They all share, however, a flaw: a model that would generate only training crystals would score extremely well. We exemplify this for the FID as provided in LeMatGen and showcase the results in Appendix L and Figure 17, where we indeed see that the copier scores best! This will be the case for any other distributional metric on the space of probability measures (e.g., Wasserstein, MMD) as well. Similarly, Precision and recall measures would be maximized with a purely memorizing model, so they suffer from the same problem.

We thank the reviewer for their helpful evaluation of our manuscript and hope that we addressed the concerns!

Thank you for the detailed rebuttal and the new experiments. They address most of my concerns. Two questions remain:

  • Regarding stability: As far as I can tell, the DFT results validate MACE, not CFTD itself. Also, CFTD only reacts from about 0.1 eV/atom in the perturbation experiments. How should this sensitivity be interpreted when comparing models that generate materials with smaller energy differences, especially within the metastable window (0–0.1 eV/atom above hull)? Could this limitation be discussed explicitly in the paper?

  • Regarding the penalty rule: Algorithm 1 adds both penalties, but Appendix F, which is used in the experiments, keeps only the memorization penalty when both apply. Do the guarantees in Appendix B still hold for the Appendix F rule, and is the proof included in the revision?

Dear reviewer,

thank you for your thoughtful response.

  1. Indeed, the experiment in Appendix N suggests that CFTD reacts to perturbations resulting in energetic differences larger than 0.1 eV/atom. We referenced this accordingly in the paper and stated more precisely that we view CFTD related to stability information. However CFTD measures more, for instance the bad rating of Chemeleon2 is mainly related to compositional/elemental collapse. Therefore, a precise experiment only identifying the relation of thermodynamic information and CFTD is hard. We state this limitation more clearly. In particular, that for precise stability information classical measures are still needed. We also recommend to use CFTD together with relaxation (as can be seen in the overfitting experiment) so that structures close to energetic equilibrium move to that equilibrium and minor energetic differences do not matter as much anymore. Indeed, the DFT experiments mainly referred to checking the content of the MACE features as suggested by other reviewers.

  2. Regarding the CFTD penalty, we now included the proof of the precise notion we use in the code, called \mathrm{CFTD^{mem}. Only the coverage part needs adaptation and you can find it in Theorem F.1.

We thank you again for your questions, and hope that these additional points resolve your concerns.

I thank the authors for their detailed responses. As most of my concerns have been addressed, I will raise my score to Weak Acceptance, provided that the final version includes the stated revisions and clarifications.

Reviewer_mxMvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose a new metric that captures stability and novelty of generated structures. "Coarse" part of the metric evaluates stability of the structures and the "Fine" part evaluates novelty. Both parts use MACE as a pre-trained feature extractor. The fine part builds a contrastive loss to evaluate the novelty of structures, and optimal transport is used to evaluate the coverage or novelty of the generated structures.

Critical strengths
  • The experiments show show sensitivity to structural corruption and training-set leakage on MP20 and Perov-5.
  • Restricting samples to small or oxygen-free crystals increases CFTD while keeping the change in continuous SUN small
  • The MACE conditioning is somehow supported by ablation analysis
Critical weaknesses

The paper gives two different rules for computing CFTD. Algorithm 1 adds the memorization and quality penalties while in appendix F the quality penalty is removed. The coverage proof in theorem B.2 uses the first rule.

  • The novelty guarantee also requires close training neighbours to receive enough weight in the matching. The paper does not report checking this condition.

  • Results show that MACE features can be used to predict formation energy. However, they do not directly show how well the CFTD quality score measures stability across diverse generated crystals.

  • On figure 2.a and 2.b we can see that small noise didn't impact the metric at all. The problem is that small variations on a stable material can dramatically increase energy level and make a structure unstable. Indeed, the usual threshold is 0.1 eV/atom and it's difficult to get how random noise impacts energy above hull based on the scale of the noise. It's possible that CFTD cannot capture the difference between a structure at 0.2 eV/atom and one at 0.05 eV/atom above hull but it's not really possible to know with only the results reported in your article. I must highlight that many metastable materials (materials under 0.1 eV/atom) can exist with same composition but different conformations and high precision is really required to assess SOTA generative models.

  • MACE is used for relaxing structures and then to assess stability, but evaluation is not complete ; for example, use DFT calculation to check if MACE relaxation actually improves stability when applied on structures close to the metastable area by computing the delta of total energy before and after relaxation.

Questions for authors
  • Does the coupling assumption in Theorem B.2 hold empirically for the benchmarked generators?
  • Which penalty rule produced the results: Algorithm 1 or Appendix F?
  • Why are you comparing only on MACE?
  • Can you use DFT to check precision of your approach for structures with small eV/atom above hull?
2 comments

We thank the reviewer for their constructive and helpful comments.

  • Coupling assumption Good question! We checked empirically how often violations occur and found that: The number of generated points for which there exists a training point but receive no mass within r=0.5τmemr = 0.5 \tau_{mem} is extremely small: Typically around 1.79% for raw material generative models and 2.83 % for relaxed ones, mainly driven by ADiT. We added this to the paper at the end of appendix B.

  • MACE and stability We added multiple new experiments to investigate the relationship between MACE features and independent stability measures (other MLIPs or DFT). We compared MACE features and purely structural AMD features to see how well they can predict thermodynamic DFT minima. Appendix O shows that MACE features strongly outperform purely structural AMD features, identifying the thermodynamic DFT minima among polymorphs about twice as often as AMD features. Appendix N relates CFTD directly to energy changes measured with an independent MLIP (see the next point), and Appendix P reports DFT calculations on generated structures, where the MACE (meta)stability assessment is confirmed in 5 of 6 systems. For the discussion on CFTD versus small energetic scales, see the next point.

  • MACE/CFTD for small energetic shifts Good question. We want to clarify that we see our metric as complementary to classical thermodynamic stability measures. We checked in Fig. 18 how the MACE features relate to the energetic differences and similarly how the CFTD relates. The MACE features seem to be very responsive, the CFTD reacts from a perturbation of 0.1 eV/atom onwards. This was tested with a different MLIP, so no MACE overlap here. We would say it can tell apart the difference between 0.2 and 0.05 eV/atom, however we need to remark that it is not easy to extract this effect since CFTD measures strictly more than stability.

  • Different penalty rules We thank the reviewer for spotting this inconsistency. We left the main paper's penalty in a simplified form so that the paper is easier to read, Appendix F specifies the version used for the results. In practice, it does not matter for the theory or for the results. The same theory can be proved under the assumption that the set of bad quality and bad memorization is small, e.g., that the two featurizers agree (or rather: do not completely disagree). In practice, this is rare and does not influence rankings. To quantify it, we put this in a table for all the models in our default setting.

Percentage of coupling mass assigned to pairs with Cqual>τqualC_{\mathrm{qual}}>\tau_{\mathrm{qual}} and Cid≤τmemC_{\mathrm{id}}\leq\tau_{\mathrm{mem}}, for β=0.6\beta=0.6.

ModelRaw (%)Relaxed (%)
ADiT2.9260.639
CDVAE0.2990.264
Chemeleon0.8390.679
Chemeleon20.1440.130
Crystalite1.1260.824
DiffCSP0.8420.635
DiffCSP++0.6920.587
MatterGen0.6860.615
Average0.9440.546
  • DFT experiments Thanks for the suggestion. We added several DFT studies as requested. We checked whether MACE relaxation improves DFT energies for generated compounds close to the metastable limit, which it did in all 6 cases (see Appendix P). Further, we computed DFT and MACE convex hulls for several generated samples and found agreement between MACE and DFT in (meta-)stability classification in 5 out of 6 cases (see also Appendix P). Generally, many papers on generative models for crystals only rely on MLIP evaluations, it is a good sanity check that these indeed agree.

  • Only MACE We added CFTD-UMA where we mirrored our MACE feature construction with the UMA MLIP. We see that they mostly have the same ranking, with the exception of Chemeleon2 in the raw case, but in the relaxed case, UMA also identifies the Chemeleon2-produced samples as out-of-distribution.

    Further, we split up the LeMatGen into the MACE+ORB and ORB score (UMA energy hull does not seem to be reliable in the current LeMatGen version), so we see that the improved stability holds up under independent MLIP evaluations.

We thank the reviewer for their helpful review. We hope these additions address your concerns.

Dear reviewer,

thanks for your review. Since we are close to the end of the discussion period, we wanted to follow up whether we addressed your concerns! We would appreciate your feedback on the updated version. Thanks!

Reviewer_vQcgOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The central contribution of this paper is the introduction of CFTD, a metric that combines MACE representations with contrastively learned features to evaluate distributional fidelity and novelty in materials generation. Specifically, MACE, trained on energies, is used to assess the physical plausibility of generated structures, with generation quality controlled through MLIP-TD. Contrastively learned features, which can effectively distinguish different structures, are used to discourage the generation of overly similar structures and thereby promote novelty.

In my view, the more valuable insight is the integration of two established evaluation approaches: (1) metrics based on distributional fidelity and (2) evaluations of the plausibility and novelty of individual generated structures. Overall, I find this combination interesting and practically valuable. The experiments are detailed and extensive, and they are well grounded in domain knowledge, which I appreciate. However, distribution-based evaluation approaches are not uncommon in materials generation, and the lack of comparisons with these approaches weakens the paper’s persuasiveness.

After establishing the effectiveness of MACE representations for evaluation, the authors use these features to guide a generative model. They find that this improves the validity and stability of generated structures while reducing novelty, which is consistent with expectations. The authors also demonstrate that CFTD can detect overfitting.

Critical strengths

An insightful perspective on evaluating plausibility and novelty. The authors offer a new perspective by evaluating these properties at the distribution level rather than solely at the individual-sample level. Using MLIP representations to assess plausibility and contrastively learned features to assess novelty are both sound methodological choices. As a minor suggestion regarding presentation, I found it somewhat difficult to understand how these two components relate to the terminology of “coarse” and “fine.”

Extensive and informative analytical experiments. The analyses of lattice perturbations, coordinate noise, and elemental substitutions are interesting and convincing. The appendix also provides substantial evidence of the structural information encoded in MACE features, supporting the rationale behind the proposed approach. I particularly appreciate analytical experiments that investigate the underlying mechanisms rather than merely report performance improvements. The authors have done this well.

A promising and practically valuable approach to guiding generation. Using MACE features to guide a generative model and improve the validity of its outputs is an interesting idea with practical value. To date, the connections between MLIPs—which are often used as backbones for property prediction—and generative models remain relatively underexplored.

Utility for detecting memorization and overfitting. The proposed metric can identify memorization and overfitting, providing an additional use beyond evaluating the final outputs of a generative model.

Critical weaknesses

Missing comparisons with existing distribution-based metrics. The absence of comparisons with existing distribution-based evaluation metrics for materials generation is surprising.

Heuristic choices in the metric design. The design of the core metric relies on heuristic choices, particularly in the selection of thresholds.

Questions for authors

Overall, I consider the paper sufficiently complete and substantial to warrant acceptance. Nevertheless, I have a few additional questions that I would like to discuss with the authors. These are intended to invite broader discussion rather than serve as conditions for acceptance: my view that this work merits acceptance would remain unchanged even if the authors choose not to address them.

  1. The relationship between distributional fidelity and task-oriented generation. Distribution-based evaluation metrics are relatively uncommon in materials generation, which motivates the following question. Distributional fidelity can often effectively assess a generative model’s modeling capacity. However, matching the data distribution is frequently not the ultimate objective of a generation task. Examples include the targeted, RL-driven generation of novel materials, as in Chemeleon2, and the generation of synthetic data for LLM training. Such outputs may occupy only a small region of the overall distribution—a region containing higher-quality samples that better satisfy our requirements. Could there therefore be a mismatch between a model’s ability to generate these high-quality samples and its ability to generate samples that match the original overall data distribution? I would be interested in hearing the authors’ perspective.
  2. The physical meaning of MLIP representations beyond energy prediction. Although the potential of MLIPs as foundation models has been demonstrated by work such as JMP, they are ultimately supervised models trained using energy targets. How should we understand their potential for assessing overall physical quality? In particular, why can two structures with similar energies have different representations in an MLIP feature space?
2 comments

We thank the reviewer for their insightful review! We are glad that you deem the paper sufficiently complete and substantial.

We will first respond to your weaknesses.

  • Missing distributional baselines Good point. There are indeed a few, but none of them has emerged as standard. They all share, however, a flaw: a model that would generate only training crystals would score extremely well. We exemplify this for the FID as provided in LeMatGen and showcase the results in Appendix L and Figure 17, where we indeed see that the copier scores best! This will be the case for any other distributional metric on the space of probability measures (e.g., Wasserstein, MMD) as well.

  • Heuristic selections We agree with the reviewer that the design of the core metric relies on heuristic choices. However, this is why we have plenty of ablations to quantify their impact. We observe that the rankings of the evaluated models are pretty stable to the thresholds, but MM can be understood as a tuning knob.

  • Relationship between discovery and distributional Great question. This is one of the biggest problems in the evaluation of material generators. Usually, they are trained to approximate a distribution, but the metrics to test (or checkpoint) them on are often discovery-based, e.g., MSUN. A relatively simple thought experiment is: what is the optimal (M)SUN? If we drew again from the same data distribution, say e.g., the validation set, we would not get SUN=1SUN = 1. Hence, especially if we RL finetune, these models are incentivized to leave the "full" data manifold to reach certain goals (e.g., higher MSUN), which is what we see with Chemeleon2 in Fig.16. Overall, we think that the trade off between distributional and discovery metrics is not well understood and both have their place.

  • MLIP representations This depends: if you only use the readout layer, we expect that there will be more "energetic" information or stability information encoded. However, for our representations we also use intermediate representations of these crystals. Then in earlier layers it is to be expected that two structures with (almost) the same energy but different composition could have different MACE features.

We again thank the reviewer for their assessment and time!

Dear reviewer,

thanks for your review. Since we are close to the end of the discussion period, we wanted to follow up whether we addressed your concerns! We would appreciate your feedback on the updated version. Thanks!

Reviewer_CijnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends TNovD and introduces CFTD (Coarse-fine Transport distance) to address stability and mode collapse issues that metrics like cSUN suffer from. It proposes dual representations for evaluating crystal generation. For the coarse features, it uses mean pooled and projected MACE features, and a contrastive GNN for fine features. It also introduces a Goldilocks zone, with penalties for anything lying outside it so as to curb both memorization and low-quality structures. Paper also goes on to do MACE-conditioned generation.

Critical strengths
  1. Moving from an instance-wise evaluation to a distribution-based evaluation metric is well motivated and relevant.

  2. They also show improvements over the previous single featurizer metric TNovD, and testing on established models they found that Chemeleon2 ranked poorly under CFTD but high under cSUN.

Critical weaknesses
  1. My primary concern is the connection between "implicit stability" and actual physical stability. Independent stability calculations of the generated structures, for instance calculating DFT-relaxed energies, could be useful.

  2. There seems to be some inconsistency between how the two penalties are implemented together. Algorithm 1 simply adds the two penalties, whereas Appendix F says that these regions are mutually exclusive and memorization takes precedence.

  3. On page 9, we also see that MACE conditioning increases the LeMat stability and the SUN score, while novelty and MSUN decrease. The trade-off and the novelty claim need to be addressed a little more explicitly.

  4. Figure 5 shows high conditioning cosine similarities, but in Fig. 9 in the appendix, we see that even random structure pairs can have high MACE cosine similarity. The authors should address this confound when interpreting the conditioning cosine similarities.

  5. The paper claims that it is a necessity to have two-feature geometries if one wants to capture quality and novelty. I believe this should be qualified, because one could imagine having a single representation with an upper and lower threshold that could create a similar Goldilocks zone.

In the current form, I lean towards rejection; however, the direction is promising and these questions need to be addressed for me to reconsider my decision.

Questions for authors
  1. Does the penalty for quality actually reflect physical stability?

  2. Could the authors please clarify how the two penalties are actually combined in the implementation?

2 comments

We thank the reviewer for their constructive and helpful review! We will reply to these points one by one.

  • Implicit Stability We ran several related DFT experiments and independent MLIP verifications. In particular, in LeMatGen we split up between ORB and ORB+MACE stability to showcase that the MACE conditioning generalizes beyond this specific MLIP. Please see the new improved results in Table 1. Further, we want to draw your attention to sections N, O and P, where we investigate the relation between MACE features, physical energies (as measured either by an independent MLIP or by DFT) and the CFTD. We find that there is a strong relation between MACE features and physical stability. Further, we performed some DFT calculations on a small subset. DFT is very costly, but we confirmed that, for 5/6 structures, MACE's metastability assessment is consistent with DFT. Overall, since CFTD measures also other qualities, such as distributional fidelity, we only refer to CFTD as a coarse proxy for stability and changed some of the wording accordingly.

  • Penalty We thank the reviewer for catching this inconsistency. We left the penalty in the main paper in a simplified form so that the paper is easier to read. Appendix F specifies the version used for the results. In practice, it does not matter for the theory or for the results. The same theory can be proved under the assumption that the set of bad quality and bad memorization is small, e.g., that the two featurizers agree (or rather: do not completely disagree). In practice, this is rare and does not influence rankings. To quantify it, we put this in a table for all the models in our default setting.

Percentage of coupling mass assigned to pairs with Dqual>τqualD_{\mathrm{qual}}>\tau_{\mathrm{qual}} and Did≤τmemD_{\mathrm{id}}\leq\tau_{\mathrm{mem}}, for β=0.6\beta=0.6.

ModelRaw (%)Relaxed (%)
ADiT2.9260.639
CDVAE0.2990.264
Chemeleon0.8390.679
Chemeleon20.1440.130
Crystalite1.1260.824
DiffCSP0.8420.635
DiffCSP++0.6920.587
MatterGen0.6860.615
Average0.9440.546
  • MACE conditioning Yes, indeed, stability improves for empirical features: This means that they were drawn from the training set, hence also novelty decreases. The raw MSUN and SUN numbers can indeed be a bit noisy since they use a hard cut-off, but especially for the learned, empirical, and unconditional models, there is a clear trend! The learned MACE features, learned with a simple Multilayer Perceptron, have high novelty and low stability, the unconditional model sits in the middle, and the empirical model has high stability and low novelty. Since we did not explore the space of how to learn MACE conditions, we think this can be greatly improved for the learned model. But this shows that MACE features are usable downstream, and also capable of producing novel structures. In particular, when combined with a rejection sampler (that rejects crystals for the learned model with a too low cosine agreement), this outperforms all the variants!

  • MACE conditioning cosine We thank the reviewer for raising this important point regarding the interpretation of the MACE cosine similarities. They use different normalizations. For the generative modelling experiments, we standardized the MACE features, i.e., so that they have mean 0 and variance 1. This also means that two randomly picked crystals from e.g., the MP20 set have cosine close to 0, and indeed the model really uses the conditioning. We updated the figure to reflect that two "random" crystals do not have that agreement.

  • Two featurizers? Agreed, we fixed the sentence that was a bit weirdly worded. The claim is that different quantities are likely to be measured better by different "lenses". Also, without a Goldilocks zone and a single featurizer, every improvement in one regime is a loss in the other, which is also shown in the TNovD bounds. However, even then, to judge what should be novel is a very different question from what is good "quality". This is also reflected in the MSUN metrics, where we measure stability using an MLIP (or DFT) and novelty using a structure matcher procedure.

We thank you for your review and hope that our response cleared up your concerns.

Dear reviewer,

thanks for your review. Since we are close to the end of the discussion period, we wanted to follow up whether we addressed your concerns! We would appreciate your feedback on the updated version. Thanks!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their extensive and helpful reviews. In this answer, we want to discuss the most common criticisms among reviewers as well as extensions that greatly strengthened our paper. We revised the manuscript according to the reviews, added some related work. The changes are marked in blue. The main changes comprise the new LeMatgen Table, which now includes a learned MACE version, UMA-CFTD, DFT experiments as well as some insights on energies, MACE features and mode collapse.

  1. Implicit Stability: Among 3 out of 4 reviewers, the way we treated stability caused some confusion. We refer to the stability captured by the (pooled) MLIP features as implicit stability. The rationale for this is as follows: MLIPs estimate energies from structures. MACE describes local environments to derive local energies. Hence, the hidden layers must contain some energy-related representation of the crystal at hand. We show this via multiple experiments in the paper, and have added more experiments in the revision (see Appendix N, O and P). We show that MACE features outperform purely structural features by a factor of >2>2 in predicting the lowest-energy polymorph for a given composition. This also relates to a previous experiment in the Appendix E that demonstrated a strong correlation between polymorph energy differences and MACE feature distances. Moreover, we added experiments that demonstrate a strong correlation of MACE feature distances to energy differences between rattled structures, using UMA as an independent energy evaluation, see Appendix N. We also show that the CFTD captures these energy differences, but only from 0.1 eV/atom onwards. Based on these experiments, we think it is reasonable to refer to MACE features as a measure of implicit stability.

  2. UMA features To test whether our results only depend on the specific MLIP (MACE) we also reran the model evaluations with an UMA featurizer. We saw that the ranking stays largely consistent, with some minor differences for the raw Chemeleon2. However, when relaxed, the Chemeleon2 is also rated badly. We added this to the main paper.

  3. Chemeleon2 One of the biggest mismatches in comparison to cSUN is Chemeleon2. We have found the major reason why distributional metrics tend to judge it much more harshly than, e.g., MSUN metrics. This is due to its tendency to over- and underproduce certain elements (see Appendix K).

  4. Comparison to other distributional metrics: We show that normal distributional metrics (for instance FID as implemented in LeMatGen) have the property that the train set ranks best, therefore unable to capture memorization.

  5. Learned MACE features: We use learned MACE features to strengthen our generative model, for which we generate 16-dimensional MACE features using a "normal" Euclidean generative model. When combined with a rejection procedure, this obtains the best MSUN among our methods. Further, there is a nice ranking: Learned tends to produce novel but not stable materials, unconditional is in the middle and empirical produces known but stable crystals. All in all, this is an exciting result that shows that learning such MLIP feature spaces can help guide generators.

  6. DFT calculations: We added many DFT experiments during the revision. We performed DFT relaxations for a subset of the generated samples and compared MACE and DFT convex hulls of those chemical systems, for an independent stability evaluation. The MACE classification as stable or metastable (energy above hull <100<100 meV/atom) is confirmed by DFT in 5 of the 6 studied systems. Also, we performed static DFT calculations on some generated metastable samples to answer the question of whether MACE relaxations improve stability - they do, in all cases.

We thank the reviewers for their time and their helpful reviews and hope that the revised version addresses the concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.