- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents PACKORA, a flow-based generative model for molecular crystal structure prediction (CSP). The proposed framework integrates several architectural and training design choices and supports flexible conditioning on molecular conformers, stereochemistry, and space-group information. The authors systematically investigate model architecture, training objectives, conditioning strategies, inference algorithms, and scaling. Extensive experiments demonstrate that PACKORA consistently outperforms competitive baselines across multiple structure-generation benchmarks and achieves improved performance in downstream structure ranking.
Critical strengths
Important and relevant research topic. Molecular crystal structure prediction is an important and challenging problem with broad applications in materials science and drug discovery. Developing effective generative models for this task is valuable to the research community.
-
Clear writing and presentation. The paper is well organized and clearly written. The motivation, methodology, and experimental setup are presented in a logical manner.
-
Well-designed evaluation protocol. I appreciate the effort to establish a comprehensive evaluation framework that separates structure generation from downstream relaxation and ranking. This distinction is important for understanding the performance of generative CSP models and provides a useful benchmark protocol for future research.
-
Comprehensive empirical study and strong experimental results. A major strength of this paper is its systematic investigation of important model design choices, including architecture, training objectives, conditioning, inference, and scaling. The proposed model consistently achieves strong performance against competitive baselines across multiple benchmarks. More importantly, the systematic findings provide useful empirical insights into effective modeling strategies for flow-based molecular CSP.
Critical weaknesses
-
Limited methodological novelty. Most of the individual components employed in PACKORA. Consequently, the methodological novelty is relatively limited. However, I view this work primarily as a systematic design study rather than a proposal of a fundamentally new generative modeling approach. From this perspective, the comprehensive empirical investigation constitutes meaningful contributions.
-
Suggestions for improving the evaluation. While the experimental evaluation is comprehensive overall, I have a few suggestions that could further strengthen the paper:
-
Statistical robustness of structure ranking. Although the results are encouraging, it would be helpful to report the variability across different random seeds or provide uncertainty estimates to better assess the robustness of the improvements.
-
Evaluation under matched computational budgets. The current experiments mainly compare different methods using the same number of generated candidates. Since the methods have different generation costs, it would be interesting to compare their performance under the same computational budget (e.g., GPU time).
-
Analysis of generated structure diversity. The improvements in crystal solve rate may result from better individual predictions, greater diversity among generated structures, or both.
Questions for authors
See weakness.
4 comments
W2c. Analysis of generated structure diversity: do the solve-rate gains result from better individual predictions, greater diversity among generated structures, or both?
The gain comes mainly from better individual predictions: Packora candidates are correct about twice as often, and concentrate near low-energy minima. Three analyses support this.
- Per-candidate success (Table A). Packora doubles Sol_S, the chance that a single candidate is correct.
- Gain decomposition (Table B). We split Packora-L's Sol_C@30 gain over CLARI-H by which model solves each target. About 80% of the gain comes from targets both models solve, where Packora finds correct candidates more often, and the remaining 20% from targets only Packora-L solves. Packora-M and the standard criterion show the same pattern.
- Unique structures after relaxation. We relax all candidates and cluster those within 10 kJ/mol of the lowest energy with StructureMatcher. Packora places 1.3–1.6× more candidates in this window than CLARI-H, but they form the same number of unique structures or fewer: 123 and 138 vs 141 per target on single-polymorph, and 329 and 325 vs 435 per system on multi-polymorph. Its gain therefore does not come from greater diversity.
Table A. Per-candidate success Sol_S and solve rate Sol_C@30 (%; best in bold, second best in italics)
| Model | Sol_S ↑ | Sol_C@30 ↑ |
|---|---|---|
| CLARI-H | 4.7 | 27.3 |
| Packora-M | 9.0 | 36.4 |
| Packora-L | 9.2 | 38.9 |
Table B. Contribution to Packora-L's Sol_C@30 gain over CLARI-H, by target group (points)
| Target group | Targets | Contribution |
|---|---|---|
| Both models solve | 514 | +9.1 |
| Only Packora-L solves | 138 | +2.9 |
| Only CLARI-H solves | 46 | −0.4 |
| Neither solves | 194 | 0.0 |
| Total | 892 | +11.6 |
We added this analysis to Appendix H.1.
Thank you again for suggestions that strengthened the paper; we are happy to answer any further questions.
Thanks for the response from the authors. I do not have further concerns. I will keep my positive rating.
Thank you for reviewing our response and for keeping your positive rating. We appreciate your time and helpful feedback throughout the review process.
Dear Reviewer fdkW,
Thank you for your positive review and for viewing Packora as a systematic design study. We incorporated all three of your suggestions:
In the revised PDF, changes for your review are in green.
W1. Limited methodological novelty: most of the individual components employed in Packora are not new.
We agree, and thank you for assessing Packora as a design study whose systematic investigation is a meaningful contribution. While the methods are not new, several of its findings are new for generative molecular CSP:
W2a. Statistical robustness of structure ranking: please report the variability across different random seeds or provide uncertainty estimates.
We incorporated your comment by repeating the structure-ranking evaluation with three sampling seed sets. Packora's lead holds across seeds: on both benchmarks, Packora-M and Packora-L recover more experimental forms than CLARI-H, rank them higher, and relax about twice as fast. Specifically, for each model, we generated two more sets of 1,000 candidates per CSD entry with new sampling seeds and evaluated them with the same relaxation and ranking pipeline. The table reports the mean ± SD over the three sets (best in bold, second best in italics).
Single-polymorph (26 targets)
Multi-polymorph (29 forms)
†Convergence rate is the fraction of candidates converged within 100 BFGS steps.
We revised Section 4.3, Fig. 5 and Appendix N (Tables 19–22) to report the three-seed results. We appreciate your suggestion.
W2b. Evaluation under matched computational budgets: how do the methods compare under the same computational budget (e.g., GPU time)?
At same GPU time budget, Packora is more accurate than CLARI-H. We optimized our inference procedure (will be publicly released), and at equal throughput of about 35 candidates/s, Packora-M solves 36.4% of targets versus 27.3% for CLARI-H, where the solve rates (strict) are averaged over the test set with 30 candidates per target:
Also, please note that Packora's candidates also relax faster than CLARI-H's, because they converge in fewer optimization steps. Relaxation dominates the per-candidate cost of the pipeline, so Packora is also cheaper end to end.
We revised Appendix F.2 to add these results and will report our optimized inference code with our paper.