- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes CFTD, a metric for crystal generation evaluation. The metric has two views. A fine-grained one flags crystals too close to training data, and thus evaluates novelty using a contrastive GNN. A second, coarse view uses embeddings from MACE, a foundation model trained on DFT data from the Materials Project, to flag crystals too far from training data, and thus measures quality. Both views are combined into a single score using optimal transport: crystals in a “safe zone” between the two thresholds are not penalized, so the metric avoids the artificial quality-novelty trade-off caused by using a single feature space. Using toy experiments, the authors show that CFTD catches mode collapse better than cSUN and broken crystals better than TNovD, and that it detects memorization. They also rank 8 existing generators with CFTD and show that relaxation improves quality but increases memorization, and that some rankings differ strongly from cSUN (e.g., Chemeleon2). They also show that feeding MACE embeddings of training crystals into a flow-matching generator gives more stable but less novel crystals.
Critical strengths
- The proposed CFTD metric detects simulated mode collapse (distribution shifts) that cSUN misses. It also penalizes broken or unrealistic crystals (lattice strain, noise, bad substitutions) more strongly than TNovD.
- It combines two competing goals, novelty and quality, in a single distributional metric without a forced trade-off between them, thanks to the separate feature spaces and the “safe zone”.
- It is computationally cheaper than DFT-based evaluation: it reuses pretrained MACE features, with no costly DFT or energy-above-hull computations, much as FID reuses a pretrained classifier.
- It also detects memorization and overfitting, which makes it a potential checkpoint-selection signal.
Critical weaknesses
- Limited novelty. CFTD builds directly on TNovD and reuses its optimal transport framework and contrastive GNN. The main additions are a second feature space based on MACE and a zone with no penalty. It is mostly a combination of existing methods rather than a new approach.
- Limited experiments. The comparisons with TNovD and cSUN rely mostly on toy experiments, and other metrics (FID-style material metrics, precision/recall) are not considered. Also, CFTD ranks some generators very differently from cSUN (e.g., Chemeleon2), but the authors do not check what causes these differences.
- Quality is only measured indirectly. CFTD uses MACE features as a proxy for stability, but it is never compared to DFT energy above hull on generated crystals. So it is unclear how well it reflects real stability.
- Generalization and scalability are not fully shown. CFTD is only evaluated on MP20 and Perov-5, with no larger structures. The MP20-trained contrastive GNN is reused on Perov-5, but only for toy tests, so transfer across datasets is not studied systematically. Exact optimal transport scales as ; the authors mention faster alternatives but do not test them or report runtimes.
- Only one feature representation. The coarse view only uses MACE-MP-0b3 with mean pooling. It is not compared to other pretrained MLIPs or classical structural descriptors, and the authors themselves say the pooling choice is arbitrary. So it is hard to know if the gain comes from MACE itself or just from adding a second, physics-based feature space.
- Possible evaluation bias. MACE is used to evaluate, relax, and condition the generated crystals, so a MACE-based metric could favor the MACE-conditioned generator. LeMat-GenBench uses other MLIPs, which helps, but an independent check (another representation or DFT) would be more convincing.
Questions for authors
- Does CFTD correlate with DFT energy above hull on generated crystals, even on a small subset?
- How does CFTD compare to other distributional or memorization-aware metrics, such as FID-style material metrics or precision/recall?
- Did you try other pretrained MLIPs, classical descriptors, or other pooling choices for the coarse view? Does the gain come from MACE specifically, or from any physics-based feature space?
- How do CFTD and its runtime behave on larger datasets and larger unit cells (exp: MPTS-52)? How well does the contrastive GNN generalize to these datasets?
- Since MACE is used to evaluate, relax, and condition the crystals, could this introduce a bias in favor of the MACE-conditioned generator? Would the CFTD results hold with a different pretrained representation, beyond the LeMat-GenBench evaluation?
4 comments
Thank you for the detailed rebuttal and the new experiments. They address most of my concerns. Two questions remain:
-
Regarding stability: As far as I can tell, the DFT results validate MACE, not CFTD itself. Also, CFTD only reacts from about 0.1 eV/atom in the perturbation experiments. How should this sensitivity be interpreted when comparing models that generate materials with smaller energy differences, especially within the metastable window (0–0.1 eV/atom above hull)? Could this limitation be discussed explicitly in the paper?
-
Regarding the penalty rule: Algorithm 1 adds both penalties, but Appendix F, which is used in the experiments, keeps only the memorization penalty when both apply. Do the guarantees in Appendix B still hold for the Appendix F rule, and is the proof included in the revision?
Response
Dear reviewer,
thank you for your thoughtful response.
-
Indeed, the experiment in Appendix N suggests that CFTD reacts to perturbations resulting in energetic differences larger than 0.1 eV/atom. We referenced this accordingly in the paper and stated more precisely that we view CFTD related to stability information. However CFTD measures more, for instance the bad rating of Chemeleon2 is mainly related to compositional/elemental collapse. Therefore, a precise experiment only identifying the relation of thermodynamic information and CFTD is hard. We state this limitation more clearly. In particular, that for precise stability information classical measures are still needed. We also recommend to use CFTD together with relaxation (as can be seen in the overfitting experiment) so that structures close to energetic equilibrium move to that equilibrium and minor energetic differences do not matter as much anymore. Indeed, the DFT experiments mainly referred to checking the content of the MACE features as suggested by other reviewers.
-
Regarding the CFTD penalty, we now included the proof of the precise notion we use in the code, called \mathrm{CFTD^{mem}. Only the coverage part needs adaptation and you can find it in Theorem F.1.
We thank you again for your questions, and hope that these additional points resolve your concerns.
I thank the authors for their detailed responses. As most of my concerns have been addressed, I will raise my score to Weak Acceptance, provided that the final version includes the stated revisions and clarifications.
Response
We thank the reviewer for their constructive and helpful review. We answer point by point.
Limited Novelty In contrast to TNovD, we added a stability-aware featurizer, changed the training of the contrastive GNN (Pettifor augmentation), proved coverage-novelty bounds, performed extensive analysis on the MACE features, ablated this to other MLIPs, and used these MACE features for generative modeling. Note that for TNovD, the score barely changes as we perturb the lattice or add Gaussian fractional noise. Further, every quality decrease of TNovD corresponds to a memorization increase, which is not the case for CFTD.
With this, we have several new findings, namely that CFTD spots Chemeleon2, the RL fine-tuned model, as off-distribution, we found that MACE features serve as good coarse coordinates, and found that FID-style metrics do not detect memorization. We spotted an insufficiency in instance-level metrics (cSUN) and found that feeding in empirical MACE features increases stability but lowers novelty, and that naively learned ones can be used to get a more novel sampler.
Limited experiments: Good suggestion. We added a distributional metric, FID from LeMatGen, to argue that such metrics would not penalize the training set. This can be seen in Fig. 17 and Appendix L. For TNovD, you can see from their figures or relative differences that TNovD with the contrastive featurizer is not responsive to large perturbations in the same way our CFTD is (see Section 3). For Chemeleon2, we believe we have found the reason: It mode collapses with respect to composition (see Appendix K), which explains, in our opinion, why distributional metrics detect this.
Quality of CFTD/Question regarding correlation Good question. We first want to clarify that we see our metric as complementary to classical thermodynamic stability measures. We checked in Fig. 18 how the MACE features relate to the energetic differences and, similarly, how the CFTD relates. The MACE features are very responsive, the CFTD reacts from a perturbation of 0.1 eV/atom onwards. This was tested with a different MLIP, so there is no MACE overlap here. We would say it can tell the difference between 0.2 and 0.05 eV/atom, however we need to remark that it is not easy to extract this effect since CFTD measures strictly more than stability. Further, we also computed MACE and DFT convex hulls for 6 generated samples and observed that the MACE metastability assessment is accurate for 5 out of 6 generated samples. You can find these DFT results in Appendix O and P.
Generalization and Scalability The main scaling bottleneck is not large crystals but a large number of generated crystals. We therefore checked how the run time of featurization and CFTD behaves when we scale up the number of generated samples to 50k. In that case, it is still feasible in roughly 15 mins (Appendix M), which is still cheaper than relaxation or thermodynamic stability measures. We still think this is an important area of future research, and speeding it up substantially is important.
Only MACE We added CFTD-UMA, where we mirrored our MACE feature construction with the UMA MLIP. We see that they mostly have the same ranking, with the exception of Chemeleon2 in the raw case, but in the relaxed case UMA also identifies the Chemeleon2-produced samples as out of distribution.
We also compared the prediction of the lowest structure in polymorphs (using DFT energies) with a purely structural featurizer AMD, where we show that MACE features strongly outperform AMD at predicting the minimum energy structure, see Appendix O.
Evaluation bias We split up the LeMatGen into the MACE+ORB and ORB score (UMA energy hull does not seem to be reliable in the current LeMatGen version, see for instance this commit), so we see that the improved stability holds up under independent MLIP evaluations. Further, we added independent DFT assessments for a small number of structures in Appendix P.
Other distributional metrics Good point. There are indeed a few, but none of them has emerged as standard. They all share, however, a flaw: a model that would generate only training crystals would score extremely well. We exemplify this for the FID as provided in LeMatGen and showcase the results in Appendix L and Figure 17, where we indeed see that the copier scores best! This will be the case for any other distributional metric on the space of probability measures (e.g., Wasserstein, MMD) as well. Similarly, Precision and recall measures would be maximized with a purely memorizing model, so they suffer from the same problem.
We thank the reviewer for their helpful evaluation of our manuscript and hope that we addressed the concerns!