- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper main focus is to show the reward hacking by RL models on LeMat-GenBench: the oversampling of unary materials and materials for which the reference hull has fewer than 12 structures on or near it. They design a reward function which prevents said reward hacking, and show that it works with OMatG and Chemeleon2.
Methodology-wise, they apply GRPO to the discrete flow-matching composition channel of a crystal generator, using exact per-event likelihoods so that the elements of a structure are actions of the policy in their native discrete.
Critical strengths
The papers addresses an important problem in the field. There is an overabundance of crystal generation models regurgitating DFT datasets and not so many real discoveries made. The proxy metrics used for benchmarking the generators are indeed deficient, and RL indeed hacks them.
On the low-level the paper appears sound - it reasonably measures what it claims to measure. Can’t be sure without the code though.
Critical weaknesses
I What, exactly, is wrong with single-element structures?
A. “face-centeredcubic Mg” in Figure 1 is an interesting example, as it both supports and undermines the authors’ point. An fcc Mg phase has been synthesised by Ji and Jona - so it is possible that the particular model-generated structure is not novel and is just absent from the DFT database. On the other hand, it shows that thermodynamically metastable single-element phases can be valuable enough to warrant experimental synthesis.
On top of that, LeMat-Bulk does contain fcc Mg, e. g. agm002159588 & mp-1056702. Could the authors please:
- Supply the CIF file for the fcc MG structure depicted in Figure 1
- Confirm that the Figure 1 fcc MG has been relaxed with UMA prior to novelty-scoring
- Use
pymatgen.core.StructureMatcherto compare this supposedly novel phase against all the Mg entries in LeMat-Bulk
The reason I’m asking is because a novel fcc Mg can be registered in a few ways, it’s important to distinguish between them:
- A novel local energy minimum for Mg system
- BAWL fingerprint bug
- A configuration of Mg atoms which is novel but is not at the local energy minimum
B. Multi-element compounds also have a lot of metastable local energy minima
https://www.science.org/doi/10.1126/sciadv.1600225
https://doi.org/10.1063/1.3079326
C. The burden is on authors to show why discovering new single-element metastable local minima is “meaningless”, but discovering mutli-element metastable local minima is not. As the authors correctly write in the Related Work section, RL can produce many metastable generations which are duplicates or substitution variants of training structures (Negishi & Walsh, 2026), and, quoting (Negishi et.al), “While it is natural that the sample distribution becomes biased toward a specific area of material space after RL, the generation of 900 or 611 novel, stable structures for the same composition is unrealistic.”
II Sparse hull impact
On a hull with only a few references a value near zero says only that the structure is no worse than the few phases it was compared against, and every benchmark that scores stability against a reference set inherits this limit.
The number of structures on the hull depends on the chemical system (it doesn’t have to be the same for all), not only on the arbitrary database construction procedure.
On top of this, there is a purely combinatoric aspect. The 12 threshold for near-hull reference phases is the same for all arities - practically impossible to satisfy for unary systems, and easily satisfiable for quarterly (4 unary + 6 binary already give 10).
The authors show a correlation and a direction, but they don’t prove the magnitude of the effect. Had the true hull been known, how many of the structures scored as metastable would turn out to actually not be metastable? I see two possible ways to address this:
- Take some chemical systems with sparse hulls, generate a large number of structures in those chemical systems, relax with a MLIP, add them to the MLIP-based hull. MatterGen and WyFormer can do this out of the box. // Just in case, LeMat used MLIPs to compute single point energy of DFT-relaxed structures - it's likely that MLIP-relaxed structures will have slightly lower energy, this needs to be accounted for
- Take some chemical systems with dense hulls, exclude some structures on the hull, check how it affects the stability verdicts. This is cheap but random removal won’t settle the question
Table 1 contains the SUN counts, 8-72 out of 2500 generated structures. This doesn't conclusively proof anything, but intuitively one would expect that if the database hull was somehow deficient, the models would have identified a large number of structures below it as well.
III DFT
This is not critical and doesn’t have a big impact on Accept/Reject, but you still might want to fix.
DFT settings are wrong, MP used MPGGADoubleRelaxStaticMaker, not a single MPRelaxSet+MPStaticSet. The impact is likely not catastrophic: Miller et al. (2024) had the same mistake, see Table 1 and Section H.1, in Kazeev et al. (2025).
DFT spot checks only include the structures with MLIP E_hull < 0.1 eV/atom, missing errors where MLIP overestimates the energy.
IV Minor bits
They are not critical and don’t have an impact on Accept/Reject, but you still might want to fix.
“known elements” is a weird phrase. What is the alternative, ununennium-119?
Figure 3 caption has weird commas: “a,”
Questions for authors
See Critical Weaknesses
11 comments
I.A, continued. The three ways in which a single-element structure can be registered as novel
The reviewer lists three ways in which a novel fcc Mg can be registered. For the structure of Fig. 1 the answer is the possibility the reviewer raised first, that it is not novel. It is the known fcc phase and was novel only with respect to the MP-20 training set, which our internal evaluation uses and not in LeMat-GenBench. We should have stated this in the submission.
To answer the question for the structures that the benchmark does count, we examined every single-element structure that it classifies as novel, in five structure sets (1,531 structures). Each was relaxed with UMA to a tight tolerance (0.005 eV/Å on atomic and cell forces), then perturbed three times and relaxed again. A structure is a known phase if it matches a LeMat-Bulk entry of its element. It is not a local minimum if it leaves its geometry or lowers its energy by more than 5 meV/atom. The rest are candidate new minima. These criteria were fixed before the calculations.
The table refers to the training run without the single-element guard (left side of Fig. 1), which has 499 single-element structures in mSUN.
| Class | Structures |
|---|---|
| Known phase (matches a LeMat-Bulk entry) | 140 (28%) |
| Not a local energy minimum, case (3) of the reviewer | 231 (46%) |
| Candidate new local energy minimum, case (1) of the reviewer | 128 (26%) |
Case (2) does not occur. No structure that the benchmark classifies as novel matches a LeMat-Bulk entry under the settings of the benchmark. Our evaluation uses StructureMatcher and does not use BAWL. For the candidates, DFT calculations on ten of them retain the geometry of all ten, at 3 to 138 meV/atom above the ground state of the element. The perturbation test is not a phonon calculation, which is why we call them candidates. With the guard, OMatGRPO has 8 single-element structures that the benchmark classifies as novel (5 known phases, 2 not local minima, 1 candidate).
Fig. 1 now shows a Sn structure of the run without the guard. The benchmark counted it in mSUN, and after tight relaxation it matches the known body-centered tetragonal Sn entry mp-55 (). The caption names the training run and the evaluation. A new appendix table reports the classification for the five sets.
I.B and I.C. Metastable minima of compounds, and substitution variants
We agree that compounds also have many metastable local minima. Metastable configurations of compounds can carry properties of interest. One example is the superconductivity created in WB$_2$ by pressure-induced metastable planar defects (Lim et al., Nat. Commun. 13, 7901 (2022), https://doi.org/10.1038/s41467-022-35191-8). We focus on compounds for this reason.
We applied the criterion that the reviewer quotes, the number of novel and stable structures of one composition.
| Number of mSUN structures | |
|---|---|
| With the guard, largest number of structures sharing one formula | 8 (1,153 compounds over 1,071 formulas) |
| Without the guard, largest number of structures of one element | 73 (Sn) |
Of the 73 Sn structures, 13 are known phases, 41 are not local minima and 19 are candidates. Compounds are less concentrated, although the chemical system with the most mSUN structures of OMatGRPO, Hg-K, has 38.
We also classified the mSUN structures against the MP-20 training set with the released code of Negishi and Walsh.
| Model | Substitution variants of training structures |
|---|---|
| Pretrained model | 48.2% |
| OMatGRPO | 68.0% |
| Chemeleon2 | 80.3% |
Two thirds of the mSUN structures of OMatGRPO are substitution variants by this definition. The classification follows their pipeline, which relaxes each structure with MACE before the comparison. The revision will reports both tables in the appendix and does not present mSUN as a number of discoveries.
Regarding the single-element structures, I think it would be useful to view the issue at two levels.
Material discovery I and the authors appear to be in agreement that both single-element and multi-element (meta)stable structures can be valuable novel materials - but also that formal MSUN wrt a DFT database doesn't gurantee that.
There is also a chemical reason. For an element, the convex hull consists of the ground state alone, so measures only the energy difference to that one phase. For many metals a large family of stacking variants and distorted cells of the close-packed structures lies within 0.1 eV/atom of it. The metastability criterion is therefore less selective for elements than for compounds, which also have to compete with phases at other compositions.
This qualitive assertion requires evidence and loops us back to "what counts as a new material". Graphite and diamond, martensitic and austenitic steels, , , -iron share chemical composition and are genuinely different real materials. On the other hand, ordered realizaitons of the same disordered material can have different chemical compostions.
ML model proxy metrics Now, we don't have to precisely solve what counts as discovery to advance machine learning.
We observed that in the MP-20 training set, 0.8% of the structures are single-element (226 of 27,136). In the training run without the guard, 52.9% of the mSUN structures are single-element (499 of 944). With the guard, the share is 0.4% (5 of 1,158). Without the guard the policy moves far from the training distribution, toward the structures for which the reward is easiest to obtain.
I agree with the authors here. A dramatic collapse in the diversity of chemical compositions is a bad trait for a generative model. Also, the observation that the reward is easiest to obtain on single-element structures is interesting; qualitively, one might expect that uniqueness & novelty would suffer.
I.A. The fcc Mg example
Good, I'm glad that it's settled.
We thank the reviewer for this expert assessment. It has led to the most extensive changes in the manuscript, which are described below. It also made the paper scientifically more sound, because the revision now discusses what a new material means and states that an improvement in a benchmark does not always translate into new materials. Section numbers refer to the general response. All values are the re-scored values (Section 1).
I. Regarding single-element structures
We agree that single-element structures are not meaningless. The wording of the submission was unfair, and we take responsibility for it. We see this work as a step toward the inverse design of bulk materials with properties of interest for applications. Such properties arise in compounds more often than in elemental structures. The single-element guard is therefore a restriction of the task to compounds and not a statement about the value of elemental phases.
There is also a chemical reason. For an element, the convex hull consists of the ground state alone, so Ehull measures only the energy difference to that one phase. For many metals a large family of stacking variants and distorted cells of the close-packed structures lies within 0.1 eV/atom of it. The metastability criterion is therefore less selective for elements than for compounds, which also have to compete with phases at other compositions.
We observed that in the MP-20 training set, 0.8% of the structures are single-element (226 of 27,136). In the training run without the guard, 52.9% of the mSUN structures are single-element (499 of 944). With the guard, the share is 0.4% (5 of 1,158). Without the guard the policy moves far from the training distribution, toward the structures for which the reward is easiest to obtain.
Change in the manuscript. The words "meaningless" and "repackings" are removed. The guard is described as a restriction of the task to compounds. Changes are in the abstract, the caption of Fig. 1, Sec. 3 and Sec. 4.4.
I.A. The fcc Mg example
Response. The assessment of the reviewer is correct. We thank the reviewer for identifying this error, and we apologize for it. The example has been removed.
The three requests.
ltol=0.1). Under both it matches two entries, mp-1056702 and agm002159588, which are the two that the reviewer names. We also applied the novelty test of the benchmark to the file. It classifies the structure as not novel.Further checks. The structure has space group Fm3ˉm at a symmetry tolerance of 0.05 Å and a lattice constant of 4.494 Å. The benchmark set of the same model contains a corresponding fcc Mg structure, and the benchmark classifies it as not novel as well. In DFT the structure of Fig. 1 remains fcc and lies 12 meV/atom above hcp Mg at the densest k-point mesh.
CIF file of the fcc Mg structure of Fig. 1 of the submission, as relaxed with UMA (request 1 of point I.A). The cell is the generated 10-atom cell. Its space group is Fm3ˉm at a symmetry tolerance of 0.05 Å.