- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes CrystalJTF, a training-free way to use a frozen MLIP as feedback during sampling from pretrained crystal diffusion models. The method reweights per-atom forces from the MLIP on the clean structure estimate and uses them to update fractional coordinates while keeping lattice and species intact. Experiments on a wide range of base models and training data show higher SUN and MSUN in all settings, with CHGNet used for relaxation and scoring. An ablation suggests that atom-wise scales improve results over uniform scaling and over using POINT Q-JTF instead of Endpoint forces.
Critical strengths
- The method really is a plug-in and does not require retraining. One controller with the same hyperparameters improves SUN/MSUN on three different generators, including an LLM+diffusion.
- The evaluation is more careful than is common in this area. I would like to point out the analysis of matcher tolerances and the dependency on the number of generated samples.
Critical weaknesses
The paper is generally clear. My concerns are mostly about how well the specific design choices are supported.
- The 2x2 ablation switches two components on top of the full controller, but simpler alternatives are not tested:
- a sweep over the guidance interval.
- a sweep over .
- a constant step size instead of the cap-derived .
- plain Point Q-JTF with several guidance strengths.
Without these, it is hard to say whether Relative-BB and the cap are needed, or whether a tuned DPS-style step would do the same.
-
The isolated effects are also small. Experiments in Table 3 differ by 0.2-0.6 SUN. CIs over several independent sets of 10,000 generations are needed, and even more so for the MatterGen rows with n = 1,000.
-
Guidance uses a single MACE MLIP, and all evaluation uses CHGNet. So the "two evaluation potentials" are two versions of the same, by now fairly old, model. Three additions would address this:
- Separate the effect of in-loop guidance from simple pre-relaxation, e.g., by relaxing baseline samples post hoc with the same MACE model before the CHGNet step.
- Recompute relaxation and E_hull with a more recent model from Matbench Discovery (https://matbench-discovery.materialsproject.org).
- Try 2-3 other MLIPs as the source of guidance gradients to check that the gains are not specific to MACE.
Questions for authors
- How often does the pair term in Eq. (8) bind rather than the atom term? What fraction of proposals is rejected by the 0.5A check?
- GSS (Qin et al., 2026, arXiv:2604.27636), which uses MatterSim forces inside a MatterGen-style sampler, is missing from related work. Could you discuss it in the relevant works?
- De Lambilly et al. (2026) report that MatterSim energy guidance on MatterGen performed about the same as unguided sampling. Could you discuss their results in comparison to the proposed approach?
What could increase my score is the set of experiments listed under weaknesses:
- CIs over independent 10k runs.
- Sweeps over and the guidance interval.
- Constant-step and plain Point Q-JTF baselines.
- A newer evaluation MLIP and other guidance MLIPs.
- The post-hoc pre-relaxation control.
4 comments
Response Part 2: Additional Experiments and Validation
-
Confidence intervals across three 10,000-sample runs
CrystalJTF improves SUN and MSUN over unguided sampling in all three reported DiffCSP MP-20 runs, each generating 10,000 structures per method. We summarize the mean metrics and 95% Student-t confidence intervals of the run-level paired improvements below. Values are percentages; Δ denotes CrystalJTF − Baseline, with Δ and its confidence interval expressed in percentage points.
Matcher Method N U S MS SUN MSUN Strict Baseline 90.31 97.23 11.43 58.75 8.71 48.28 CrystalJTF 91.33 97.38 12.75 63.55 10.13 54.08 Δ +1.02 +0.15 +1.32 +4.80 +1.42 +5.80 95% CI of Δ [0.560, 1.480] [−0.012, 0.319] [1.201, 1.446] [3.539, 6.068] [1.395, 1.452] [4.701, 6.899] SUN and MSUN increase in every run, with mean paired gains of 1.423 and 5.800 percentage points, respectively. Both confidence intervals lie entirely above zero, and the positive gains across all three runs demonstrate that the observed improvement is not confined to a single run.
-
Parameter search and comparison with tuned simple guidance
CrystalJTF achieves higher SUN and MSUN than the simple-guidance baseline selected through a ten-configuration parameter search in the reported 10,000-sample test. We evaluated five geometric-cap strengths and five constant-step sizes on a separate development set, then froze the selected configuration before testing. CrystalJTF retained its original settings, and , throughout this comparison.
Both simple-guidance variants use uniform atom weighting. Uniform-fixed uses a normalized Point Q-JTF direction without Relative-BB or adaptive geometric scaling. Its reference step size, Å, was calibrated independently without using SUN labels.
Development search: 2,000 structures per configuration. All metric values are percentages.
Method Configuration N U S MS SUN MSUN Baseline — 90.15 99.10 12.95 59.20 10.10 49.45 Uniform-cap 90.20 99.10 12.70 60.65 9.95 50.95 Uniform-cap 90.85 99.05 13.95 63.10 11.35 53.90 Uniform-cap 91.00 99.00 13.40 62.35 11.00 53.25 Uniform-cap , selected 91.65 98.80 13.85 63.10 11.55 54.65 Uniform-cap 94.35 99.10 11.20 59.55 9.75 53.40 Uniform-fixed 89.90 99.05 13.20 60.60 10.20 50.45 Uniform-fixed 90.50 98.95 13.30 61.80 10.45 52.25 Uniform-fixed 91.25 99.05 13.15 62.65 10.80 53.70 Uniform-fixed 94.50 99.30 10.85 58.25 9.40 52.45 Uniform-fixed 98.60 98.80 9.05 41.85 8.70 40.10 CrystalJTF Original settings 90.80 98.95 13.10 63.40 10.55 54.10 Uniform-cap with achieved the highest development SUN and was selected for the final comparison. The sweep also shows that stronger guidance is not uniformly beneficial: increasing the fixed step from to raises novelty but reduces MSUN from 53.70% to 40.10%.
Final comparison: 10,000 structures per method. Δ denotes CrystalJTF − Uniform-cap, in percentage points.
Method N U S MS SUN MSUN Selected Uniform-cap () 91.83 97.30 11.90 62.83 9.52 53.61 CrystalJTF 91.38 97.53 12.65 63.42 10.17 54.05 Δ −0.45 +0.23 +0.75 +0.59 +0.65 +0.44 Although the selected simple baseline scored higher on the development set, CrystalJTF obtained 65 additional SUN structures and 44 additional MSUN structures in this independent test. Thus, the development-selected baseline did not match the complete controller’s final SUN/MSUN, despite receiving a ten-configuration parameter search. This comparison supports the value of the complete atom-resolved, geometry-constrained feedback design beyond tuning a global guidance strength within the tested search space.
Response Part 2: Additional Experiments and Validation
-
Cross-MLIP validation and post-hoc pre-relaxation controls
CrystalJTF improves SUN and MSUN across all four tested combinations of guidance and relaxation MLIPs, demonstrating cross-MLIP transferability within the evaluated settings. We vary the guidance potential between MACE and MatterSim and the final relaxation/evaluation potential between CHGNet and MatterSim. Improvements persist under both matched-potential and cross-potential evaluation, showing that the observed benefits are not restricted to MACE guidance or CHGNet evaluation.
All experiments use DiffCSP on MP-20 with 10,000 structures per group. Values are percentages. Each Δ row compares the preceding method with the unguided Baseline under the same evaluation MLIP, in percentage points.
Method Guidance / pre-relaxation MLIP Evaluation MLIP N U S MS SUN MSUN Baseline — CHGNet 90.22 97.30 11.35 58.88 8.76 48.41 CrystalJTF MACE CHGNet 91.38 97.53 12.65 63.42 10.17 54.05 Δ +1.16 +0.23 +1.30 +4.54 +1.41 +5.64 CrystalJTF MatterSim CHGNet 91.27 97.59 12.35 63.50 9.95 54.07 Δ +1.05 +0.29 +1.00 +4.62 +1.19 +5.66 Baseline + post-hoc pre-relaxation MACE CHGNet 90.25 97.30 11.62 60.32 9.04 49.86 Δ +0.03 +0.00 +0.27 +1.44 +0.28 +1.45 Baseline — MatterSim 90.22 97.23 7.97 43.20 5.72 33.01 CrystalJTF MACE MatterSim 91.12 97.38 8.05 44.84 6.00 35.50 Δ +0.90 +0.15 +0.08 +1.64 +0.28 +2.49 CrystalJTF MatterSim MatterSim 90.93 97.46 8.06 45.05 5.90 35.62 Δ +0.71 +0.23 +0.09 +1.85 +0.18 +2.61 Under CHGNet evaluation, MACE-guided and MatterSim-guided CrystalJTF improve SUN/MSUN over unguided sampling by 1.41/5.64 and 1.19/5.66 percentage points, respectively. Under MatterSim evaluation, both guidance potentials retain positive gains: 0.28/2.49 and 0.18/2.61 percentage points, respectively. These results support our central contribution: an atom-resolved, geometry-constrained controller that translates feedback from different MLIPs into effective sampling updates without retraining the generator.
The post-hoc control further distinguishes this contribution from simply adding an energy-minimization stage. With the same MACE potential and subsequent CHGNet evaluation, in-loop CrystalJTF exceeds post-hoc pre-relaxation by 1.13 percentage points in SUN and 4.19 percentage points in MSUN. Thus, the tested post-hoc procedure does not reproduce the gains of sampling-time feedback, supporting the value of controlling updates during generation rather than relying solely on terminal relaxation.
-
Expanded MatterGen evaluation from 1,000 to 10,000 samples
Our original manuscript reported MatterGen results using only 1,000 generated structures per method. To strengthen the evidence for the reliability of CrystalJTF, we expanded the evaluation to 10,000 structures per method on both MP-20 and Alex-MP-20, comparing Baseline and CrystalJTF in four groups comprising 40,000 structures in total. CrystalJTF improves SUN and MSUN on both datasets, with particularly strong MSUN gains on Alex-MP-20.
Values are percentages. Each Δ row compares CrystalJTF with its corresponding Baseline within the same dataset and evaluation protocol, in percentage points.
Dataset Method N U S MS SUN MSUN MP-20 Baseline 91.19 97.93 9.61 62.10 7.68 52.83 CrystalJTF 93.04 98.38 10.10 61.86 8.57 54.55 Δ +1.85 +0.45 +0.49 −0.24 +0.89 +1.72 Alex-MP-20 Baseline 61.21 99.48 22.53 87.71 10.84 50.36 CrystalJTF 80.55 99.72 17.07 81.46 11.42 62.75 Δ +19.34 +0.24 −5.46 −6.25 +0.58 +12.39 On MP-20, CrystalJTF produces 89 additional SUN structures and 172 additional MSUN structures at the fixed 10,000-sample budget. On Alex-MP-20, SUN increases from 10.84% to 11.42%, while MSUN increases substantially from 50.36% to 62.75%. These gains correspond to 58 additional SUN structures and 1,239 additional MSUN structures.
The expanded experiments therefore provide larger-scale evidence that CrystalJTF improves the yield of stable or metastable, unique, and novel crystals on MatterGen. In particular, the Alex-MP-20 results demonstrate a substantial improvement in joint discovery yield, even though not every marginal metric increases.
Thank you for the comments
I thank the authors for the detailed response. All my questions are addressed. The results look well motivated and supported by the experiments, so I increase the score to 3.
Response Part 1: Geometric Constraints and Related Work
We thank the reviewer for the careful assessment and constructive suggestions, and for recognizing the plug-in nature of CrystalJTF and our evaluation efforts. We organize our response into two parts. This first part addresses the three Questions for Authors; Part 2 will address the experimental concerns and requested additional analyses.