- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OmniInvent is a chemical language model which approaches molecular design as autoregressive blank filling over unmodified SMILES. Both de novo and substructure constrained generation are therefore handled by a single framework rather than via separate pipelines. It uses a LLaMA style decoder with GLM style prefix attention and 2D positional embeddings. The pretraining masks are generated on the fly by sampling connected subgraphs of the molecular graph and replacing the SMILES segments they map to (thereby eliminating the need for a hand-crafted fragmentation step which fragment based generalists are so reliant on). As it is autoregressive the exact log likelihoods are available and REINVENT4 policy gradient optimisation can be used as before. Results are shown for both direct sampling and RL guided generation, with PMO for the de novo cases and LibINVENT and LinkINVENT for the constrained cases.
Critical strengths
-
Removing the fragmentation step is a big one. The vast majority of current generalist molecular generators employ some form of fragment based representation (e.g. SAFE) which in turn require hand-crafted bond breaking and bond reassembly rules. By working directly on the raw SMILES with graph guided masking you eliminate that entire layer and yet still have an autoregressive model so exact log likelihoods are available and policy gradient optimisation can be used directly. It's a clean combination rather than two pieces just glued together.
-
The authors are extremely upfront about the limitations of their own work. The fact that the direct sampling differences are so small, that the prescreening setting is not the PMO setting, that their models are vastly smaller and trained on far less data than the baselines are all explicitly stated by the authors rather than left for the reader to infer. In the ablation they only describe the fully-masked-fraction variant as not harmful rather than presenting it as a benefit which again is a very unusual level of honesty for a paper of this nature.
Critical weaknesses
-
The methodological contribution is negligible. Prefix attention and 2D positional embeddings are simply lifted from GLM. The architecture is a standard decoder. The only real innovation is the graph guided masking. That's a perfectly reasonable engineering choice, but again nothing particularly novel in terms of methodology. You're just taking an existing infilling approach and applying it to a different sequence type. The paper doesn't really go on to explain why the molecular graph case necessitates a different design rather than just applying the existing approach.
-
The direct sampling results are saturated and are presented as a headline claim. All the models are at ceiling on validity and uniqueness, one baseline is actually higher on validity and diversity (it only differs from the nearest baselines in the third decimal place). The body of the paper states all of this and notes that the differences are negligible and that these metrics may not be good indicators of goal directed performance. The abstract and conclusion however present the direct sampling results as having high validity, high uniqueness and high diversity and go on to state that performance is either better than or at least on a par with state of the art. The framing is inconsistent with what the body of the paper actually says.
-
The ablations are done on a different model to the one used in the main results and so in no way validate the design decisions. They use the smaller ChEMBL trained variant rather than the PubChem trained models used in the main results. The masking comparison does demonstrate that aligning the masks to the graph structure is an improvement. The fully masked fraction comparison only shows that the choice isn't harmful. The ablation doesn't actually test the capability that the choice is supposed to provide (i.e. the ability to do de novo design) as that's precisely what the fully masked examples are there for.
-
Most of the claimed advantages lie within the range of the reported variability and there is absolutely no significance testing in the paper. On PMO the gap to the best baseline is less than one standard deviation (this is stated quite explicitly in the paper) and a number of the gaps in the constrained generation table are of a similar order of magnitude.
-
The source metric (distance to source) is not actually directional at all (we don't know whether a lower or higher value is better) but it is presented as being good. It's simply the average Tanimoto distance between the generated molecules and the unfragmented source molecule using the fingerprint settings from the prior work. There is absolutely no statement as to whether a lower or higher value is better. OmniInvent is lower than all of the baselines that report it and this is presented as the generated molecules being more similar to the source drug. In the substructure constrained design case, being close to the source could just as easily mean that you haven't explored sufficiently so the fact that it's presented in a positive light at all without actually stating the direction is quite selective.
Questions for authors
-
In the prescreening environment the oracle call time used for prescreening isn't counted against the budget (as you point out). Does the ordering in the table change if you do include those calls in the total budget?
-
The ablations use the smaller ChEMBL variant. Do the conclusions on the masking strategy and the fully-masked fraction still hold with respect to the PubChem models that were used for the main results?
-
The fully masked fraction ablation is only on constrained tasks. How does that affect de novo performance (i.e. the ability that the choice is trading off against)?
-
Several of the conclusions are based on differences within one standard deviation. Can you provide the paired across seed comparisons to show which of the gaps remain after correction?
4 comments
Weaknesses
3)
The choice of ablation dataset reflects what was done during the development of OmniInvent: experiment on the smaller ChEMBL dataset, and then train on the much larger PubChem database. As shown in Appendix H, the performance differences between models trained on the two datasets are small. We see no reason as to why the findings would not translate to the PubChem-trained model. We do not believe rerunning the ablations with the larger dataset be a good use of compute.
Furthermore, we have now evaluated the ablations's capabilities of conducting de novo design, as shown below:
| Validity (%) | Uniqueness (%) | Diversity | Mean molecular weight (Da) | |
|---|---|---|---|---|
| OmniInvent | 378 | |||
| OmniInvent$_\text{L}$ | 381 | |||
| 326 | ||||
| Training data | 404 |
Surprisingly enough, the ablated model is capable of de novo generation. This implies that blank infilling without fully masked training examples gives a sufficient grasp of SMILES syntax to be effective in this setting. However, the ablated model exhibits a slightly lower validity and uniqueness than all other OmniInvent models trained with ; in particular, the uniqueness was 100 % after rounding for all models covered in Section H.1. This indicates some difference in what the ablated model has learned. The average molecular weight of the de novo samples from the ChEMBL-trained models were 378 Da and 381 Da for the ChEMBL-trained OmniInvent and OmniInvent_L models, respectively; this value vas 326 Da for the ablated model. As the average molecular weight of our training dataset was 404 Da, this implies that including fully masked training samples helps align the output distribution with the training data.
We will include the table and discussion above in the manuscript.
4)
See our reply to Question 3 below.
5)
The distance metric was included here because it was reported for both SAFE-GPT and GenMol, with the latter work bolding higher values in their results table; this implies that they consider higher numbers to be better. (As a side note, neither work discusses the interpretation of the value beyond stating its definition.) This is situational, however; while staying close to the source is highly desirable in analogue generation (see e.g. Tibo et al., 2024: https://www.nature.com/articles/s41467-024-51672-4), broader exploration is beneficial for e.g. scaffold hopping.
We will address this subtlety in the manuscript, with the new addition bolded:
Source molecule distances were lower than for the baselines that reported this metric; in other words, OmniInvent produced internally diverse sets of samples that were more similar to the source drug than those of the baselines. This similarity is not necessarily beneficial, however, since it may reflect a narrower exploration of chemical space. This is desirable in analogue generation (Tibo et al., 2024), while broader exploration may be advantageous in e.g. scaffold hopping.
Questions
1)
Accounting for the prescreening oracle calls would have a negative impact on the rankings of prescreening methods in comparison to non-prescreening methods. Prescreening effectively conducts virtual screening on ZINC 250K, for which the PMO benchmark paper reports an overall PMO score of 8.635 (Gao et al., 2022: https://arxiv.org/pdf/2206.12411) under the PMO oracle budget. Doing so would effectively extend the oracle budget to 260 000 calls to allow for the generative models to generate 10 000 molecules after prescreening; at that stage, recovering from the low score of virtual screening would effectively be impossible, and the slight differences between the methods drowned out.
2)
See our reply to Weakness 3) above.
3)
See our reply to Weakness 3) above.
4)
GenMol (masked diffusion) and InVirtuoGen (flow matching) consume randomness differently from OmniInvent and from each other. Differences in molecular representation (SAFE vs. SMILES) and tokenization further exacerbates this by changing what a given random draw results in. We therefore do not believe that a shared seed yields correlated outputs across models, and seed pairing is thus not meaningful. Our main message, however, is that blank infilling on SMILES is a competitive approach; we do not believe that a slight gap on e.g. PMO is practically meaningful.
We will make the following modifications to the manuscript to better reflect OmniInvent's positioning:
-
Removing "or exceeds" from this sentence in the abstract: In goal-directed design, it matches or exceeds state-of-the-art performance on the Practical Molecular Optimization benchmark for de novo design and achieves competitive performance in substructure-constrained design despite being trained as a general-purpose model.
-
Removing "surpassing or" from this sentence in the conclusion: Despite its general-purpose pretraining, it achieves performance comparable to both generalist and specialized models, with high validity, uniqueness, and diversity in direct sampling as well as surpassing or matching state-of-the-art results on the PMO benchmark for goal-directed de novo design.
Thanks very much for the clarifications and the additional de novo ablation. I do accept the more conservative performance claims and the revised discussion on source distance. The similar overall performance between the ChEMBL and PubChem models doesn't in any way validate that the ablation results would transfer to the models that were actually used in the main results. I also agree that shared seeds are unlikely to give us any meaningful pairing across these generators, though that doesn't really get us any further in terms of the very small differences that are reported. All of that said, combined with my continuing doubts about the methodological novelty, I remain with my original rating.
We thank the reviewer for their time and for their feedback!
Weaknesses
1)
The architecture itself is indeed lifted from other work. Re-purposing successful model architectures to new domains is in our opinion a valid and widely successful strategy, however. The main novelty, from our perspective, lies in the following:
We have rewritten the contribution statement in the introduction to better reflect this:
Our contributions are threefold: we present what is, to our knowledge, the first application of autoregressive blank infilling to molecular design, enabling a single model to handle de novo and substructure-constrained design on standard SMILES without rule-based fragmentation; we introduce OmniInvent, a transformer-based chemical language model trained with a random graph-based masking strategy that aligns training examples with the structure of molecular graphs; and we demonstrate competitive unguided and RL-guided generation across de novo and substructure-constrained design.
2)
We partially agree with the notion that the direct sampling results are effectively saturated, particularly in the de novo setting. We do point out that OmniInvent has an advantage in terms of validity for at one least task per baseline in the substructure-constrained setting, however; for example, despite InVirtuoGen exhibiting a slightly higher uniqueness than OmniInvent for fragment linking, our model yields a larger number of unique molecules due to the advantage in validity. A good molecular generative model needs to be able to generate a sufficient number of unique samples for downstream tasks; if only a handful of unique samples are generated, the RL learning signal would be rather weak in OmniInvent's case. We therefore pointed out that OmniInvent accomplishes this the in the abstract, as it is in our view a minimum requirement for molecular design.
The framing regarding SotA in our abstract and conclusion refers to the goal-directed setting. We will clarify this in the manuscript. (See our reply to the last question below for precise wording changes.)