- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this paper, the authors introduce OMatG-Flash and OMatG-Flash-RAM, a version with post-training. Their model closely matches the recent SOTA on most metrics and, through their flow map based approach, is much faster, which makes this model Pareto optimal for the crystal generation task.
Critical strengths
Critical strengths include:
- much faster than current SOTA for similar performance
- introduce post-training for flow-map-based crystal generation
- well-presented, clear writing, intuitive figures
- overall interesting ideas and good results
Critical weaknesses
Critical weaknesses include:
- As the authors note, the base model lacks geometric accuracy compared to Crystalite.
- The authors discuss issues with match rates, which are legitimate, but then they say they’re going to use METRe, which I agree is a better metric. But table 1 then reports MR (an acronym that isn’t introduced), and Table 2 is below table 3. Overall, I just found the results section slightly confusing to read because of the combination of these things. Easy to fix, though.
- The post-training objective isn’t ablated at all (and also can’t compensate for the base model’s RMSE gap to Crystallite).
Questions for authors
It would be helpful to see some ablations on the post-training objective and some more discussion on the stability of the post-trained checkpoints. If no ablations are possible, more discussion on this would be appreciated.
3 comments
Response to Authors
I am satisfied with the authors response and commitment to improve areas of the manuscript I found fault with in my initial review. I think this is a solid contribution to the field, and the further discussion of the alternative reward for de novo generation seems very promising.
We thank the reviewer for their engagement and feedback! We believe that their critical assessment has helped us to enhance the quality and clarity of our work.
Response to Reviewer yigr
We thank the reviewer for their feedback and provide some further information and clarification below.
Geometric accuracy
We agree that sacrificing geometric accuracy on as-generated structures is a weakness of OMatG-flash. We point the reviewer to our response to zMwX where we explore this issue further. Briefly, we find that, after force field relaxation, the RMSE gap between our model and selected baselines is narrowed substantially.
Clarity
We thank the reviewer for pointing out that we neglect to properly introduce “MR” as an evaluation metric. MR is meant to stand for the flawed one-to-one match rate, where the StructureMatcher algorithm is used to compare two index-matched crystals to determine if they match or not. We will revisit our discussion of match rates, and the results section as a whole, such that the flow of our results is more coherent. We also agree that table placement should be improved and will account for this in our preparation for a camera-ready version of this work.
Post-training ablation
We agree with the reviewer that a more thorough ablation and discussion of post-training stability and performance is warranted. We did observe that, for the CSP energy-minimization reward, collapse occurs almost always. Various choices of reward hyperparameters can delay the onset of this collapse, but for CSP it is quite difficult to eliminate entirely. We do, however, encourage the reviewer to see our exploration of an alternative reward for de novo generation (DNG) in our response to whKH as this may fit the reviewer’s interest in ablation of a different reward. Using a reward from the literature on reinforcement learning for DNG improvement, we post-train OMatG-flash and find that not only does the S.U.N. and M.S.U.N. of generated samples increase, but the stability of the RAM optimization under this reward is greatly improved. In fact, for this case, we do not observe any reward collapse as we observed in the CSP case. We argue that, therefore, reward choice has a critical role in determining the stability of the proposed post-training scheme and that RAM is not inherently unstable when applied to flow map post-training.
As for comparison to Crystalite's geometric accuracy, we find that after relaxation the RMSE gap to Crystalite decreases substantially and we point the reviewer to our response to zMwX for more discussion on this. We also note that energy minimization is only a proxy for improving the CSP model. The CSP task is a structure matching task and it is not always true that the lowest energy crystal polymorph is the reference structure which must be matched. We therefore study energetics in our response to whKH to understand how post-training impacts the energetics of crystals produced by OMatG-flash. We find that combining RAM post-training on minimizing energy with amortized flow map inference for k=10 gives state-of-the-art performance in terms of generating low energy structures at a fraction of the cost of a conventional flow or diffusion model.