- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In this paper, the authors addressed an important problem in materials science called crystal material generation and proposed a new method, ANCHOR. They have two components:
- A small composition policy, trained with GRPO, proposes formulas or atom types
- A frozen CSP diffusion model generates a structure for each formula.
Also, to improve the diversity and novelty of the generated materials, they have incorporated continuous adaptive novelty (CAN), which scores a structure against both MP-20 and the search’s own history. During training, only the composition policy is updated and CSP diffusion model is pretrained and remains frozen.
Critical strengths
- The idea of the paper is simple yet interesting. They propose a small composition policy, trained with GRPO, which first samples the atomic composition or formula. Then, given that atom type, a frozen CSP model generates a structure x for that composition. The composition policy is trained using a continuous adaptive novelty and stability reward, which improves its SUN/MSUN values.
- Another strength of the paper is its experiments section. The authors have done extensive experiments to show effectiveness of their proposed model.
Critical weaknesses
-
Firstly, the paper is a bit difficult for me to go through. The authors could work on making it easier to read.
-
One of the central promises of the work is that it assumes a frozen CSP model remains effective on compositions rarely seen during training, so that discovery is limited mainly by composition choice. But there is no evidence or citation for that. I am not sure, for any unseen or less-observed composition during training, how well the CSP model will learn to generate a stable structure. If the compositional prior fails there, promising compositions may be paired with poor structures and wrongly rejected, giving the policy misleading feedback. More concrete evidence or justification for this would be good.
-
The method trains the composition policy network using stability, validity, and novelty rewards, which raises two issues.
- Firstly, computing actual stability needs DFT, and the authors use MatterSim as a proxy to do that. Since they did not evaluate their model using DFT, the results are good. But I wonder, if we evaluate this with DFT, how much improvement we will observe
- Next, even with MatterSim, relaxing the sampled structure, computing SUN reward, and training the policy network will be a computationally costly process. Could the authors (a) report the total computational cost of a discovery campaign (e.g., GPU hours, wall-clock time etc.), broken down by component; (b) clarify whether all methods are compared under an equal budget of evaluated candidates or oracle calls; and (c) show how the performance gap with baselines changes as a function of this budget?
-
One of the important contribution of the work is continuous adaptive novelty (CAN) formualtion. A simpler alternative for existing DNG models is to sample sequentially and discard candidates that match previously generated or known structures. Could the authors compare ANCHOR against such a “DNG + history-based filtering” baseline and report SUN? This would clarify how much of the gain comes from learning where to search versus simply removing duplicates.
-
Lack of citations and baseline comparisons of many recent works on DNG: FlowMM, MCFlow, SymmCD, ADiT, Crystal-Text-LLM, TGDMat, FlowLLM, CrysLLMGen. Many of them, specially the LLM ones and hybrid models like FlowLLM and CrysLLMGen, have good SUN and MSUN.
-
Also code base is not provided by the author, would love to check the code during rebuttal, if provided by the author.
-
Finally, a few minor questions which I might have missed:
- How is the number of atoms in materials to be sampled?
- How many materials you sampled for evaluation?
- Which Diffusion Model you used as frozen CSP model?
Questions for authors
Check the Weaknesses
4 comments
Thank you for the detailed responses and the additional experiments. Some of my concerns have been addressed, but the following remain unresolved:
W1 (Readability). Beyond the proposed changes, I would encourage the authors to rethink the presentation of the motivation and methodology to make the paper easier to follow.
W3 (Stability evaluation). Matbench Discovery accuracy reflects typical materials, whereas ANCHOR is optimized against MatterSim and may exploit its errors, as the drop under LeMat-GenBench (13.6% to 0.4% SUN) suggests. Structures far below the hull, i.e., more stable than all known phases, are a typical sign of potential exploitation rather than reassurance. Since the SOTA claim under MatterGen’s pipeline uses the training potential, I would encourage DFT validation of even a small random subset (e.g., 50–100 on-hull structures), which is modest relative to the paper’s total compute.
W3 (Compute). The comparison is not compute-matched. Each ANCHOR run uses 168 GPU-hours (~123k evaluated structures) before the final N = 1000 draw, whereas standard diffusion baselines are comparatively cheap to train and sample. I would suggest reporting SUN/MSUN per GPU-hour for all methods, counting total cost (training, sampling and evaluation), to show whether ANCHOR’s gains also hold in terms of efficiency.
Filtered baseline. The filtering baseline is applied only to Crystalite-DNG and KLDM-DNG. Stronger models such as MatterGen+REPA, FlowLLM and CrysLLMGen, which report higher S.U.N., should also be included under the same pipeline.
W5 (Baselines). Since published numbers are not comparable across protocols, I would encourage the authors to evaluate all baselines, including FlowLLM, CrysLLMGen and other SOTA models, under one common pipeline, reporting Stable, SUN, Metastable and MSUN. This would provide a clear and fair benchmark comparison.
Finally, as ICLR allows, I would ask the authors to reflect these changes in the revised manuscript during the discussion period.
Thank you for the follow-up. A revised manuscript is uploaded, and a second upload follows before 18 Nov.
W1 (readability). The revision adds an overview paragraph at the start of Sec. 3, states in the caption of Table 1 what each ablation cell isolates, and gives the sampling settings at the start of Sec. 4. A reworked presentation of the motivation and of Sec. 3 comes with the second upload.
W3 (stability). The two numbers are different steps: the validity gate (23.2 → 13.6% SUN) and the hull construction (21.5 → 0.4% on the 716 structures that pass both gates). LeMat-GenBench is also scored by potentials and is not a DFT reference (App. E.3.2). With ORB-v3, which the policy never saw, 632 of the 665 structures on the MatterSim hull stay on the hull and 664 stay within 0.1 eV/atom (App. D.3).
Regarding DFT validation, we would like to clarify some points:
-
The claim is methodological. The paper asks where reward optimisation should act, and answers it under one fixed evaluator. We do not claim DFT-verified stability. The abstract states the state-of-the-art result as holding under MatterGen's evaluation pipeline, and states that the on-hull rate of a model optimised against a potential depends on that potential.
-
Every reward-optimised method had the same evaluator and the same reward. This holds for the fine-tuned DNG models, the genetic algorithm, and ANCHOR (Table 2). Each has the same opportunity to exploit errors of MatterSim. Under these equal conditions the DNG fine-tunes do not raise their on-hull fraction, the genetic algorithm reaches 5.9% SUN, and ANCHOR reaches 22.1%. Our conclusion about where reward should act rests on this comparison.
-
Evaluator error is a property of the evaluator, not of the algorithm. Any reward-optimised generator inherits the errors of the potential it is trained against, and the paper says so in Sec. 4 and Sec. 5. In ANCHOR the evaluator is a plug-in (App. E.3.2), so a better potential can replace MatterSim without changing the method. This is why we studied evaluator dependence directly, including the ORB-v3 check above.
-
We plan a DFT check for the final version. We plan to check a random subset of on-hull structures with DFT, with settings matched to the Materials Project reference hull, and to report the result whatever it shows. This cannot be done reliably before 18 November. Until then, the paper states the limitation in Sec. 5 and App. D.3.
Does this resolve the concern? If a specific statement in the paper reads as a claim of DFT stability, we will also reword it in the second upload.
W3 (compute). We agree, and report the comparison in GPU-hours (App. F.8). Both methods start from a released checkpoint (KLDM-CSP for ANCHOR, Crystalite-DNG for the baseline), so we count what comes after. At small budgets the DNG is ahead: after 24 GPU-hours it found 380 on-hull and 4,369 metastable formulas, against 133 and 404. Its on-hull rate then falls (19 to 13 per GPU-hour) while ANCHOR’s rises (5 to 38). Even if its rate stopped falling, the DNG would reach about 2,200 on-hull formulas by 168 GPU-hours, against 2,885 for ANCHOR (extrapolated past 50k structures). For metastable formulas the DNG stays more efficient. So the gain is in on-hull formulas at larger budgets. As in our W4 reply, more samples do not change what a DNG proposes (App. F.4).
Filtered baseline and W5. A common pipeline is in place for every model we can evaluate: Table 3 scores MatterGen, KLDM, Crystalite, MatterGen+REPA and ANCHOR under MatterGen's pipeline. It reports Stable, SUN, Metastable and MSUN for Crystalite and KLDM (our own sampling) and for ANCHOR, and now also the values reported for MatterGen+REPA (8.2% Stable and 2.6% SUN, against 16.0% SUN for ANCHOR). For the second upload we are sampling the released MatterGen+REPA checkpoint and scoring it under our pipeline, with and without the history filter.
FlowLLM and CrysLLMGen release training code but neither trained weights nor generated samples. Evaluating them would mean retraining both from scratch, including fine-tuning a large language model, which is not possible within the discussion period. The FlowLLM repository includes language-model samples, but these are training input for its flow model, not FlowLLM output. Both models keep the atom types proposed by their language model and only refine coordinates and lattice, so their compositions come from a model of the Crystal-Text-LLM type. Tables 27 and 28 evaluate Crystal-Text-LLM compositions through our frozen CSP: 0.3% and 1.1% SUN under our evaluator, against 22.1% for ANCHOR, and 0.3% and 1.0% under MatterGen's pipeline, against 16.0%.
We have now answered each of your remaining points. The revised manuscript contains the changes.
We hope this resolves your concerns and we would be grateful if you would raise your rating
The 2nd upload adds MatterGen+REPA results and the reworked Sec. 3. We will post a note when it is in.
Dear Reviewer D4qD,
We have uploaded the second revision of the paper, which contains the following for each of your open points:
W1 (readability).
We rethought how the motivation and the method are presented. We have now rewritten both such that:
- The introduction now explains the idea in three steps, in plain language, before any method is introduced.
- In Sec. 3, the reasoning for separating search from structure generation used to be two long paragraphs. It is now three numbered reasons with a heading each, so the argument can be followed from the headings alone.
- Terms that were used without explanation are now defined where they first appear: MP-20, E_hull and "on the hull" (Sec. 2), and GRPO, the policy's state and the validity gate (Sec. 3). The overview also points to Algorithm 1 for the full training loop.
W2
We have clarified our stance on whether a frozen CSP model is effective on rare compositions.
The paper no longer states this as an assumption. The first reason in Sec. 3 now clarifies that this is possible in principle/theory but with the natural caveat that a CSP model (like any model) is usually weaker away from its training data. We therefore also agree that how often it works is an empirical question, and we state this together with the numbers, i.e. that 92% of the compositions the search proposes are in element systems absent from the training data, while the frozen CSP model still produces a stable, unique and novel structure for 19.7% of them.
Sec. 3 also keeps the point that a CSP failure can only make a good composition look bad, never a bad one look good.
Filtered baseline with a stronger model.
We sampled MatterGen+REPA from its released checkpoint and scored it under the same pipeline as Table 2 (Table 27, App. F.7). MSUN / SUN:
- MatterGen+REPA, unfiltered: 14.3% / 1.5%
- MatterGen+REPA with the history filter (1,779 draws for 1,000 accepted): 22.5% / 2.4%
- ANCHOR: 47.6% / 22.1% It is the strongest of the three filtered models, and the conclusion is the same as for the other two: a filter removes repeats, but it does not change which compositions the model proposes.
W3 (stability).
Sec. 5 now has a separate Limitations paragraph. It states that:
- the gains hold under the reward the policy was trained against
- that our stability estimates rely on MatterSim potentials
- that the LeMat-GenBench check is only partial.
For FlowLLM and CrysLLMGen, our earlier answer still holds, since neither of these works release weights or samples.
We hope the paper is now much easier to read, and we would be glad if you would look at it again and reconsider your assessment and rating.
Follow-up to Reviewer D4qD
Follow-up to W6: code
The anonymised repository is available here: https://anonymous.4open.science/r/anchor-code-78C8/. It contains the training loop, the reward and evaluation code, and the configuration of the reported run. External components (KLDM, MatterSim, MatterGen's reference hull) are listed in the README.
Follow-up to W5: Crystal-Text-LLM under the MatterGen protocol.
For completeness, SUN / MSUN:
70B:
7B:
For reference: ANCHOR scores 16.0% SUN and 41.3% MSUN.
So what we reported in the previous reply, their compositions through our CSP under our internal evaluation (0.3 / 31.6% for 70B and 1.1 / 14.3% for 7B), is the most favourable setting for 70B and about equivalent for 7B.
If these results address your concerns, we would be grateful if you would consider updating your assessment. If something is still open, please tell us what, and we will answer before the discussion ends on 18 November.