- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper attempts to introduce a multi-source benchmark for measuring experimental generalization in disordered crystal generation based on structures from COD, ICSD, AMCSD, and Springer Materials, but because it bears all signs of a fully AI-generated paper (ranging from hallucinated references to hallucinated numbers in other publications to unfinished experiments) it fails to do so.
Critical strengths
None.
Critical weaknesses
This paper has a series of wrong claims and hallucinated facts. In the following, I will only list the most serious problems:
-
The authors claim that the "vast majority of crystal generation models are trained and evaluated on the Crystallography Open Database (COD)." This is just wrong. DiffCSP, MatterGen, CrystalLLM, FlowMM, and CrysLLMGen, all models that were considered in this paper, are not trained on COD.
-
Table 2 claims to take COD results from the original publications that are just not present in the original publications. Also, the work is unfinished ("FlowMM deferred due to PyTorch manifold operator compability; CrysLLMGen deferred due to upstream matgl realization dependencies and inference latency.").
-
Most of the considered generative models were not designed with disordered materials in mind. The authors never explain how they adapted, if at all, these models to the disordered setting.
-
Several references are hallucinated. The very first Dis-CSP reference is claimed to be by "Anonymous". The EquiCSP reference misses several authors. The review by De Breuck et al. reference has the wrong article number.
Since the authors explicitly state that they take "responsibility for the final content," I have to be blunt: Submitting a paper with such fundamental errors and apparently fabricated results is disrespectful to the research community and to the reviewers who volunteer their time to evaluate it. The authors should refrain from submitting work of this quality in the future.
Questions for authors
None.