- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Synthon-CGR, a GATv2-based model that predicts co-crystal formation and ranks supramolecular synthons (interacting functional groups of the two molecules). The authors curate a dataset of molecular pairs and derive synthon labels from resolved crystal structures. They show that the full model outperforms its ablations on a held-out test set, and report results on internal co-crystallization experiments, where it performs similarly to the strongest baseline.
I want to preface the review by stating that my knowledge of crystal engineering and molecular chemistry is very limited, and that my comments on the chemical or physical aspects of the work are, at most, educated guesses. However, I have a background in geometric deep learning; therefore, I am quite certain about my assessment of the machine learning parts of the review.
Critical strengths
- The paper is well motivated: the introduction does a good job of conveying why this setup is important and the current limitations of the datasets (most datasets only record whether a pair formed a co-crystal, and they are biased) and of the models (which only predict whether two molecules form a co-crystal).
- The authors propose a dataset creation and synthon annotation pipeline, which could be very useful from a practical standpoint for practitioners interested in co-crystal engineering.
- The method, while relatively simple from an ML perspective, seems to be effective.
- I appreciated the inclusion of experimentally validated results from the authors' laboratory.
- The paper is relatively easy to follow and understand.
Critical weaknesses
- (This is not greatly influencing my score.) The paper's aesthetics and writing could be improved:
- Aesthetics: I do like Figure 1, but I would recommend that the authors improve the other figures (e.g., Figure 2 and the figures in the appendix), for instance, by increasing the font size and including them as PDF/SVG so that they are rendered as vector graphics.
- Writing: I am not a fan of including so many numbers in the prose when discussing the experiments and results. For instance, when describing the dataset in Section 5.1, I would recommend using a table (e.g., Table 7 from the appendix) instead of listing the number of samples. The same holds for Section 6: referring to the tables and discussing the results at a higher level would help the reader understand them better.
- From an ML perspective, the experiments and models are quite simple, and I am unsure whether they are conclusive enough. The simplicity of the pipeline is a strength, but the experimental evaluation has some flaws:
- As far as I can tell, the number of seeds is not reported. The differences are generally large enough to be somewhat convincing, but since both the dataset and the models are small, I believe that a mean and std over at least 3 seeds should be reported for all experiments.
- No hyperparameter search is reported (except for the backbone comparison in Table 10). At this (small) scale, the hyperparameters could be crucial.
- The results reported in Table 3 on the experimental validation set seem relatively weak: Synthon-CGR is essentially tied with DeepCocrystal (balanced accuracy 0.636 vs. 0.632), and with only 17 positives the difference is within noise.
- The ablation study is confusing and hard to follow; several changes are made at once between the variants in Table 1. As the authors themselves note, "The comparison between V2 and V3 should not be read as isolating a single architectural factor because the contextual reasoner and the training regime change together." A V3 variant fine-tuned without the synthon loss would isolate the paper's central claim.
- For the @K synthon-ranking metrics (Table 2), it would be important to include baselines (e.g., a random predictor): with a median of six candidates per pair, a random ranking already reaches HitRate@5 = 5/6 0.83 on a six-candidate pair with a single positive.
- Some numbers do not add up: the Table 1 metrics are inconsistent with a balanced 350/350 test set (for V3, precision 0.884 and recall 0.902 imply a balanced accuracy of 0.89, not 0.873), and 245 masked test pairs plus 729 masked Stage 2 pairs give 974, not 916 (Section 3.2).
Overall, I appreciate the motivation behind the proposed method and analysis --- the overall narrative and intuitions are strong, with the model architecture and dataset construction pipeline being reasonable. However, I believe the writing can be improved and that the paper has limitations in its empirical evaluation. I recommend rejection, but I encourage the authors to iterate on the manuscript a few more times and strengthen their empirical evaluation; the work has the potential to be a good paper.
Questions for authors
- Have the authors performed any hyperparameter search? How is the validation set constructed?
- Have the authors used multiple seeds for their experiments?
- Could any other meaningful baselines be added for the experimental evaluation?
- Will the curated dataset, the synthon annotations, and the laboratory dataset be released? This might be a major contribution of the paper.