acceptodds
Under review as a conference paper at ICLR 2027

OmniInvent: Generalist Generative Chemical Language Models through Blank Infilling

Abstract

Drug design requires navigating a vast and sparsely populated chemical space while balancing structural novelty with multiple property constraints. Most generative approaches to the problem remain specialized to individual tasks, however. Here, we introduce OmniInvent, a chemical language model that operates on unmodified SMILES and supports both de novo and substructure-constrained molecular generation within a single framework. OmniInvent formulates molecular design as an autoregressive blank infilling problem over partially masked SMILES and is pretrained using an on-the-fly graph-based masking strategy that randomly selects connected molecular substructures and masks the corresponding SMILES segments without the need for task-specific fragmentation strategies. In direct sampling, OmniInvent generates highly valid, unique, and diverse molecules in de novo design, fragment linking, scaffold decoration, and motif extension. In goal-directed design, it matches or exceeds state-of-the-art performance on the Practical Molecular Optimization benchmark for de novo design and achieves competitive performance in substructure-constrained design despite being trained as a general-purpose model. These results show that blank infilling on SMILES provides an effective foundation for building generalist molecular generative models across diverse drug design tasks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
drug design, blank infilling, chemical language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zqZVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OmniInvent is a chemical language model which approaches molecular design as autoregressive blank filling over unmodified SMILES. Both de novo and substructure constrained generation are therefore handled by a single framework rather than via separate pipelines. It uses a LLaMA style decoder with GLM style prefix attention and 2D positional embeddings. The pretraining masks are generated on the fly by sampling connected subgraphs of the molecular graph and replacing the SMILES segments they map to (thereby eliminating the need for a hand-crafted fragmentation step which fragment based generalists are so reliant on). As it is autoregressive the exact log likelihoods are available and REINVENT4 policy gradient optimisation can be used as before. Results are shown for both direct sampling and RL guided generation, with PMO for the de novo cases and LibINVENT and LinkINVENT for the constrained cases.

Critical strengths
  1. Removing the fragmentation step is a big one. The vast majority of current generalist molecular generators employ some form of fragment based representation (e.g. SAFE) which in turn require hand-crafted bond breaking and bond reassembly rules. By working directly on the raw SMILES with graph guided masking you eliminate that entire layer and yet still have an autoregressive model so exact log likelihoods are available and policy gradient optimisation can be used directly. It's a clean combination rather than two pieces just glued together.

  2. The authors are extremely upfront about the limitations of their own work. The fact that the direct sampling differences are so small, that the prescreening setting is not the PMO setting, that their models are vastly smaller and trained on far less data than the baselines are all explicitly stated by the authors rather than left for the reader to infer. In the ablation they only describe the fully-masked-fraction variant as not harmful rather than presenting it as a benefit which again is a very unusual level of honesty for a paper of this nature.

Critical weaknesses
  1. The methodological contribution is negligible. Prefix attention and 2D positional embeddings are simply lifted from GLM. The architecture is a standard decoder. The only real innovation is the graph guided masking. That's a perfectly reasonable engineering choice, but again nothing particularly novel in terms of methodology. You're just taking an existing infilling approach and applying it to a different sequence type. The paper doesn't really go on to explain why the molecular graph case necessitates a different design rather than just applying the existing approach.

  2. The direct sampling results are saturated and are presented as a headline claim. All the models are at ceiling on validity and uniqueness, one baseline is actually higher on validity and diversity (it only differs from the nearest baselines in the third decimal place). The body of the paper states all of this and notes that the differences are negligible and that these metrics may not be good indicators of goal directed performance. The abstract and conclusion however present the direct sampling results as having high validity, high uniqueness and high diversity and go on to state that performance is either better than or at least on a par with state of the art. The framing is inconsistent with what the body of the paper actually says.

  3. The ablations are done on a different model to the one used in the main results and so in no way validate the design decisions. They use the smaller ChEMBL trained variant rather than the PubChem trained models used in the main results. The masking comparison does demonstrate that aligning the masks to the graph structure is an improvement. The fully masked fraction comparison only shows that the choice isn't harmful. The ablation doesn't actually test the capability that the choice is supposed to provide (i.e. the ability to do de novo design) as that's precisely what the fully masked examples are there for.

  4. Most of the claimed advantages lie within the range of the reported variability and there is absolutely no significance testing in the paper. On PMO the gap to the best baseline is less than one standard deviation (this is stated quite explicitly in the paper) and a number of the gaps in the constrained generation table are of a similar order of magnitude.

  5. The source metric (distance to source) is not actually directional at all (we don't know whether a lower or higher value is better) but it is presented as being good. It's simply the average Tanimoto distance between the generated molecules and the unfragmented source molecule using the fingerprint settings from the prior work. There is absolutely no statement as to whether a lower or higher value is better. OmniInvent is lower than all of the baselines that report it and this is presented as the generated molecules being more similar to the source drug. In the substructure constrained design case, being close to the source could just as easily mean that you haven't explored sufficiently so the fact that it's presented in a positive light at all without actually stating the direction is quite selective.

Questions for authors
  1. In the prescreening environment the oracle call time used for prescreening isn't counted against the budget (as you point out). Does the ordering in the table change if you do include those calls in the total budget?

  2. The ablations use the smaller ChEMBL variant. Do the conclusions on the masking strategy and the fully-masked fraction still hold with respect to the PubChem models that were used for the main results?

  3. The fully masked fraction ablation is only on constrained tasks. How does that affect de novo performance (i.e. the ability that the choice is trading off against)?

  4. Several of the conclusions are based on differences within one standard deviation. Can you provide the paired across seed comparisons to show which of the gaps remain after correction?

4 comments

We thank the reviewer for their time and for their feedback!

Weaknesses

1)

The architecture itself is indeed lifted from other work. Re-purposing successful model architectures to new domains is in our opinion a valid and widely successful strategy, however. The main novelty, from our perspective, lies in the following:

  1. Our random graph masking strategy is to our knowledge unique and serves and important inductive bias in aligning the model with downstream tasks (i.e. scaffold decoration or scaffold hopping), as evidenced by the ablation in Table 4. This approach is substantially different from other pretraining strategies demonstrated in other works.
  2. To our knowledge, the first application of blank infilling to molecule generation tasks, utilizing a powerful generalist formulation to allow our model to solve essentially all molecular design tasks without resorting to rule-based fragmentation, which is a limitation shared by all existing generalist models.

We have rewritten the contribution statement in the introduction to better reflect this:

Our contributions are threefold: we present what is, to our knowledge, the first application of autoregressive blank infilling to molecular design, enabling a single model to handle de novo and substructure-constrained design on standard SMILES without rule-based fragmentation; we introduce OmniInvent, a transformer-based chemical language model trained with a random graph-based masking strategy that aligns training examples with the structure of molecular graphs; and we demonstrate competitive unguided and RL-guided generation across de novo and substructure-constrained design.

2)

We partially agree with the notion that the direct sampling results are effectively saturated, particularly in the de novo setting. We do point out that OmniInvent has an advantage in terms of validity for at one least task per baseline in the substructure-constrained setting, however; for example, despite InVirtuoGen exhibiting a slightly higher uniqueness than OmniInvent for fragment linking, our model yields a larger number of unique molecules due to the advantage in validity. A good molecular generative model needs to be able to generate a sufficient number of unique samples for downstream tasks; if only a handful of unique samples are generated, the RL learning signal would be rather weak in OmniInvent's case. We therefore pointed out that OmniInvent accomplishes this the in the abstract, as it is in our view a minimum requirement for molecular design.

The framing regarding SotA in our abstract and conclusion refers to the goal-directed setting. We will clarify this in the manuscript. (See our reply to the last question below for precise wording changes.)

Weaknesses

3)

The choice of ablation dataset reflects what was done during the development of OmniInvent: experiment on the smaller ChEMBL dataset, and then train on the much larger PubChem database. As shown in Appendix H, the performance differences between models trained on the two datasets are small. We see no reason as to why the findings would not translate to the PubChem-trained model. We do not believe rerunning the ablations with the larger dataset be a good use of compute.

Furthermore, we have now evaluated the pfull=0p_{full}=0 ablations's capabilities of conducting de novo design, as shown below:

Validity (%)Uniqueness (%)DiversityMean molecular weight (Da)
OmniInvent98.1±0.198.1\pm0.1100.0±0.0100.0\pm0.00.888±0.0000.888\pm0.000378
OmniInvent$_\text{L}$98.8±0.198.8\pm0.1100.0±0.0100.0\pm0.00.887±0.0000.887\pm0.000381
pfull=0p_\text{full}=097.3±0.097.3\pm0.099.6±0.099.6\pm0.00.895±0.0000.895\pm0.000326
Training data404

Surprisingly enough, the ablated model is capable of de novo generation. This implies that blank infilling without fully masked training examples gives a sufficient grasp of SMILES syntax to be effective in this setting. However, the ablated model exhibits a slightly lower validity and uniqueness than all other OmniInvent models trained with pfull>0p_\text{full} > 0; in particular, the uniqueness was 100 % after rounding for all models covered in Section H.1. This indicates some difference in what the ablated model has learned. The average molecular weight of the de novo samples from the ChEMBL-trained models were 378 Da and 381 Da for the ChEMBL-trained OmniInvent and OmniInvent_L models, respectively; this value vas 326 Da for the ablated model. As the average molecular weight of our training dataset was 404 Da, this implies that including fully masked training samples helps align the output distribution with the training data.

We will include the table and discussion above in the manuscript.

4)

See our reply to Question 3 below.

5)

The distance metric was included here because it was reported for both SAFE-GPT and GenMol, with the latter work bolding higher values in their results table; this implies that they consider higher numbers to be better. (As a side note, neither work discusses the interpretation of the value beyond stating its definition.) This is situational, however; while staying close to the source is highly desirable in analogue generation (see e.g. Tibo et al., 2024: https://www.nature.com/articles/s41467-024-51672-4), broader exploration is beneficial for e.g. scaffold hopping.

We will address this subtlety in the manuscript, with the new addition bolded:

Source molecule distances were lower than for the baselines that reported this metric; in other words, OmniInvent produced internally diverse sets of samples that were more similar to the source drug than those of the baselines. This similarity is not necessarily beneficial, however, since it may reflect a narrower exploration of chemical space. This is desirable in analogue generation (Tibo et al., 2024), while broader exploration may be advantageous in e.g. scaffold hopping.

Questions

1)

Accounting for the prescreening oracle calls would have a negative impact on the rankings of prescreening methods in comparison to non-prescreening methods. Prescreening effectively conducts virtual screening on ZINC 250K, for which the PMO benchmark paper reports an overall PMO score of 8.635 (Gao et al., 2022: https://arxiv.org/pdf/2206.12411) under the PMO oracle budget. Doing so would effectively extend the oracle budget to 260 000 calls to allow for the generative models to generate 10 000 molecules after prescreening; at that stage, recovering from the low score of virtual screening would effectively be impossible, and the slight differences between the methods drowned out.

2)

See our reply to Weakness 3) above.

3)

See our reply to Weakness 3) above.

4)

GenMol (masked diffusion) and InVirtuoGen (flow matching) consume randomness differently from OmniInvent and from each other. Differences in molecular representation (SAFE vs. SMILES) and tokenization further exacerbates this by changing what a given random draw results in. We therefore do not believe that a shared seed yields correlated outputs across models, and seed pairing is thus not meaningful. Our main message, however, is that blank infilling on SMILES is a competitive approach; we do not believe that a slight gap on e.g. PMO is practically meaningful.

We will make the following modifications to the manuscript to better reflect OmniInvent's positioning:

  1. Removing "or exceeds" from this sentence in the abstract: In goal-directed design, it matches or exceeds state-of-the-art performance on the Practical Molecular Optimization benchmark for de novo design and achieves competitive performance in substructure-constrained design despite being trained as a general-purpose model.

  2. Removing "surpassing or" from this sentence in the conclusion: Despite its general-purpose pretraining, it achieves performance comparable to both generalist and specialized models, with high validity, uniqueness, and diversity in direct sampling as well as surpassing or matching state-of-the-art results on the PMO benchmark for goal-directed de novo design.

Thanks very much for the clarifications and the additional de novo ablation. I do accept the more conservative performance claims and the revised discussion on source distance. The similar overall performance between the ChEMBL and PubChem models doesn't in any way validate that the ablation results would transfer to the models that were actually used in the main results. I also agree that shared seeds are unlikely to give us any meaningful pairing across these generators, though that doesn't really get us any further in terms of the very small differences that are reported. All of that said, combined with my continuing doubts about the methodological novelty, I remain with my original rating.

Reviewer_8EorOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • Main contribution: The paper developed a SMILES-based generative model which can perform generation and multiple partial-generation tasks (fragment linking, scaffold decoration, motif extension) with a single model checkpoint. The model is trained with autoregressive blank-infilling, which fills masked spans with variable-length content given the context.
  • OmniInvent utilizes GLM-style techniques (prefix attention and 2D positional encoding) to reconstruct the masked chunk of SMILES. For molecule optimization, it utilized a REINVENT4-based RL strategy.
  • The paper conducts benchmarks and comparisons on generation, partial generation, and 23 PMO tasks.
Critical strengths
  • A SMILES-based model incorporating multiple tasks by graph-based masking: While previous works (SAFE-based models) developed a new representation for molecules, OmniInvent utilizes SMILES and introduces graph masking which masks SMILES strings in a graph-aware manner. The ablation study also supports the efficacy of graph-based masking which strengthens the model's contribution.
Critical weaknesses
  • Unclear advancements compared to previous SAFE-based general-purpose generative models: SAFE-GPT and GenMol have demonstrated partial molecular generation, with SAFE-GPT also supporting RL-based optimization. Thus, the paper’s main contribution lies in SMILES-graph masking and GLM-style infilling design. However, the current experiments do not clearly show this design choice’s additional benefits in generative performance or generalizability. The graph-masking ablation shows the masking strategy’s contribution but does not isolate the additional gain compared to SAFE-based methods.
Questions for authors
  • Benefits over SAFE representation: Can the authors provide a rationale or an experiment that specifies the limitation of SAFE-based methods that SMILES-graph masking can address? Or, is there a reason why the authors attribute OmniInvent's higher performance compared to previous models to its use of a SMILES-based method?
  • Preservation of substructure constraints: For partial generation evaluations, beyond RDKit validity, can the authors provide metrics for the preservation of all specified substructures and attachment constraints? For instance, when there are multiple attachment points, even RDKit-valid molecules may fail to preserve the specified substructures or may connect fragments at incorrect attachment points.
3 comments

We thank the reviewer for taking the time to review our manuscript! The reply below covers most concerns raised by the author, and we will get back with the additional result as soon as possible.

Weaknesses

Unclear advancements compared to previous SAFE-based general-purpose generative models

As the reviewer points out above, the main advantage of SMILES over SAFE is avoiding fragmentation and thus reliance on semi-arbitrary bond-breaking rules. We do not mean to claim that the relative performance gain of OmniInvent is due to using plain SMILES rather than a fragment-based representation. Our motivation for building OmniInvent was to see if it was possible to build a generalist chemical language model using unfragmented SMILES, so its performance is circumstantial.

Furthermore, sidestepping fragmentation avoids unintended inductive biases introduced by training on fragmented molecules, and thus potentially allows for greater exploration of chemical space. Additionally, the choice of fragmentation rule may render some molecules unrepresentable. The SAFE-GPT manuscript acknowledges this: they break BRICS bonds to produce fragments, but fall back to Louvain community detection for molecules that have no BRICS bonds. Finally, bond breaking may destroy stereochemical information; the simplest example of this would be C/C=C/C (where the slashes indicate the stereo-bond configuration), whose safe representation is CC=1.C=1C. The SAFE encoder that is part of the SAFE-GPT Python package emits a warning for such cases.

We will modify the introduction in the following way to reflect this, with additions bolded:

Recent developments in CLMs have shown that a single model can support multiple design tasks by operating on fragment-based representations such as SAFE (Noutahi et al., 2024; Lee et al., 2025; Kaech et al., 2026). These representations require explicit fragmentation and reassembly conventions, however, introducing potentially unintentional inductive biases.

We will additionally modify the Related Work section on generalist molecular generation models like so, with additions bolded:

These methods show that general-purpose molecular generators can be competitive across tasks. However, fragment-based representations require molecular fragmentation and thus rely on hand-crafted bond-breaking rules. This renders some molecules unrepresentable (Noutahi et al., 2024) and discards stereochemical information carried by the unfragmented SMILES sequence if stereo bonds are broken (Berenger et al., 2021).

Questions

Benefits over safe representation

See above.

Preservation of substructure constraints

We will check this for the generated samples and get back to you as soon as possible.

Thank you for the detailed responses and additional analysis. I understand the motivation for supporting multiple molecular generation tasks without rule-based fragmentation, and appreciate the explanation of the differences from SAFE-based methods.

I think graph-based masking itself is a useful design choice for this purpose, though I see it mainly as a refinement of existing generalist molecular generation methods, and I am still not convinced that it offers a sufficiently distinct methodological contribution. The additional analysis supports its practical usefulness, but it does not fully address the original concern about novelty. For these reasons, I maintain my current rating.

Thank you for getting back to us! We would again like to emphasize that our graph masking strategy is entirely novel to our knowledge, as is the application of blank infilling to molecular design.

Substructure preservation

We have recomputed our main results for substructure-constrained design, including substructure preservation in the validity metric; "new" corresponds to validity accounting for substructure preservation, "old" without accounting for substructure preservation.

ModelExperimentValidity (%)Uniquen. (%)DiversityDistance
Linker design99.5±0.299.5\pm0.283.8±1.283.8\pm1.20.532±0.0030.532\pm0.0030.524±0.0010.524\pm0.001
OmniInvent, oldScaffold dec.97.6±0.697.6\pm0.693.8±0.993.8\pm0.90.568±0.0050.568\pm0.0050.474±0.0050.474\pm0.005
Motif ext.99.0±0.399.0\pm0.397.4±0.497.4\pm0.40.664±0.0030.664\pm0.0030.656±0.0020.656\pm0.002
Linker design99.5±0.299.5\pm0.283.8±1.283.8\pm1.20.532±0.0030.532\pm0.0030.524±0.0010.524\pm0.001
OmniInvent, newScaffold dec.96.8±0.796.8\pm0.793.8±0.993.8\pm0.90.566±0.0050.566\pm0.0050.472±0.0050.472\pm0.005
Motif ext.99.0±0.399.0\pm0.397.4±0.497.4\pm0.40.664±0.0030.664\pm0.0030.656±0.0020.656\pm0.002
Linker design99.4±0.299.4\pm0.280.8±1.180.8\pm1.10.530±0.0040.530\pm0.0040.496±0.0050.496\pm0.005
OmniInvent$_\text{L}$, newScaffold dec.98.0±0.698.0\pm0.692.2±1.392.2\pm1.30.553±0.0050.553\pm0.0050.449±0.0060.449\pm0.006
Motif ext.99.3±0.399.3\pm0.397.9±0.997.9\pm0.90.656±0.0020.656\pm0.0020.637±0.0010.637\pm0.001
Linker design99.4±0.299.4\pm0.280.8±1.180.8\pm1.10.530±0.0040.530\pm0.0040.496±0.0050.496\pm0.005
OmniInvent$_\text{L}$, newScaffold dec.97.0±0.397.0\pm0.392.2±1.492.2\pm1.40.550±0.0050.550\pm0.0050.447±0.0060.447\pm0.006
Motif ext.99.3±0.399.3\pm0.397.9±0.997.9\pm0.90.656±0.0020.656\pm0.0020.637±0.0010.637\pm0.001

As can be seen, this leads to a minor degradation on the scaffold decoration experiment, while the other settings are unaffected. This does not change any conclusions in the manuscript, however. We will update the manuscript to reflect this change.

Reviewer_R5xhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents OmniInvent, a SMILES-based autoregressive blank-infilling model for both de novo and substructure-constrained molecular generation. It uses graph-based masking during pretraining and is evaluated on direct generation and RL-based molecular optimization.

Critical strengths

The unified formulation supports several molecular generation tasks within a single model, and the experiments cover both unguided and goal-directed settings.

Critical weaknesses
  1. The methodological novelty is limited. The method mainly combines existing techniques, including blank infilling, prefix attention, positional encoding, a LLaMA backbone, and REINVENT-style RL. Its main novelty lies in adapting these components to unified SMILES generation with graph-based masking.
  2. The ablations are limited. They examine the masking strategy and training-data composition, but do not isolate the contributions of blank infilling, prefix attention, or positional encoding.
  3. The PMO results are strongly affected by pretraining data. Performance varies substantially across pretraining datasets, making it difficult to attribute the strongest results solely to the proposed method.
  4. Evidence for broad generality remains limited. Substructure-constrained generation is evaluated on a relatively small set of molecules, and performance is sensitive to the masking configuration.
Questions for authors
  1. What performance gains come specifically from prefix attention and positional encoding?
  2. How much of the PMO improvement remains under matched pretraining conditions?
  3. How robust is the method to different masking configurations and substructure constraints?
3 comments

We thank the reviewer for taking the time to review our manuscript! In the interest of fostering discussion, we post a mostly complete reply to the reviewer's concerns below and will post another update once we have ran and analyzed additional experiments.

Weaknesses.

1)

We respectfully disagree and believe that our method is meaningfully distinct from other approaches for molecular generation. While the core architecture indeed consists of a modified LLaMA decoder, re-purposing successful model architectures to new domains is in our opinion a valid and widely successful strategy. In our view, the main novelty of the contribution can be summarized as:

  1. Our random graph masking strategy is to our knowledge unique and serves and important inductive bias in aligning the model with downstream tasks (i.e. scaffold decoration or scaffold hopping), as evidenced by the ablation in Table 4. This approach is substantially different from other pretraining strategies demonstrated in other works.
  2. To our knowledge, the first application of blank infilling to molecule generation tasks, utilizing a powerful generalist formulation to allow our model to solve essentially all molecular design tasks without resorting to rule-based fragmentation, which is a limitation shared by all existing generalist models.

We have rewritten the contribution statement in the introduction to better reflect this:

Our contributions are threefold: we present what is, to our knowledge, the first application of autoregressive blank infilling to molecular design, enabling a single model to handle de novo and substructure-constrained design on standard SMILES without rule-based fragmentation; we introduce OmniInvent, a transformer-based chemical language model trained with a random graph-based masking strategy that aligns training examples with the structure of molecular graphs; and we demonstrate competitive unguided and RL-guided generation across de novo and substructure-constrained design.

2)

As the blank infilling, prefix attention, and positional encoding components are effectively identical to those in GLM, and thus not directly our contributions, we initially opted not to run these ablations. We will nevertheless get back to you with these results as soon as possible.

3)

We would like to emphasize that we, in contrast to many other works that ignore this aspect of PMO, already report results for OmniInvent pretrained on ZINC 250K as per the original conditions outlined in the PMO benchmark (Table 3). Doing so yields an overall score of 15.3, compared to previous state-of-the-art for REINVENT-style RL of 14.2 for REINVENT_PMO.

4)

There is a notable lack of standardized benchmarks for substructure constrained task in the literature (as opposed to de novo generation). We will however supplement the existing examples with additional goal-directed tasks in the substructure constrained setting from primary sources.

Questions

1)

See 2) above.

2)

See 3) above.

3)

Section H.2 in the Appendix covers this. In particular, uniqueness suffers without restrictions on the model input. As mentioned above, we will run some additional substructure-constrained experiments to evaluate robustness across more substructure constraints. We believe that the SAFE-GPT test set is sufficient for the fragment linking, scaffold decoration, and motif extension direct sampling scenarios, however.

We have now conducted additional experiments in the goal-directed substructure-constrained setting. To this end, we selected a known ligand each for GSK3β and JNK3 from the Protein Data Bank (ligand 6QH from PDB entry 5K5N and ligand JK1 from PDB entry 3FI2, respectively). We selected substructures from each to serve as scaffolds/decorators and conducted fragment linking as well as scaffold decoration. As scoring functions, we used the geometric mean of Quantitative Estimate of Drug-Likeness and the corresponding target's activity as predicted by machine learning models from Therapeutics Data Commons. The precise setups are described in Section F.2.3 of the updated manuscript.

As shown in Figure 4, OmniInvent achieves performance comparable to (or slightly better than, albeit within one standard deviation) LinkINVENT and LibINVENT in these settings.

We have now put together a set of ablations on the prefix attention and positional encoding. The training setup is the same as for the other ablations in the main paper. For the causal ablation, we replace prefix attention with causal attention and compute the loss over the entire sequence. For the positional encoding ablation, we replace our 2D encoding with standard 1D RoPE based on the sequential position over the entire sequence. We additionally ablate both components at once, resulting in a standard LLaMA implementation.

ModelTaskValid. (%)Uniq. (%)DiversityDistance
Fragm. link.98.1±0.498.1\pm0.486.6±1.186.6\pm1.10.553±0.0040.553\pm0.0040.508±0.0030.508\pm0.003
CausalScaffold dec.96.7±0.896.7\pm0.889.9±1.389.9\pm1.30.546±0.0030.546\pm0.0030.463±0.0030.463\pm0.003
Motif ext.98.7±0.398.7\pm0.394.4±1.094.4\pm1.00.653±0.0010.653\pm0.0010.635±0.0030.635\pm0.003
Fragm. link.99.1±0.299.1\pm0.282.8±1.182.8\pm1.10.540±0.0020.540\pm0.0020.499±0.0030.499\pm0.003
1D RoPEScaffold dec.95.6±0.595.6\pm0.589.7±1.489.7\pm1.40.530±0.0030.530\pm0.0030.458±0.0010.458\pm0.001
Motif ext.99.3±0.199.3\pm0.193.2±0.393.2\pm0.30.644±0.0020.644\pm0.0020.631±0.0010.631\pm0.001
Fragm. link.98.2±0.498.2\pm0.484.4±1.884.4\pm1.80.551±0.0030.551\pm0.0030.510±0.0020.510\pm0.002
Causal + 1D RoPEScaffold dec.96.1±0.996.1\pm0.991.7±1.091.7\pm1.00.553±0.0030.553\pm0.0030.466±0.0070.466\pm0.007
Motif ext.98.5±0.398.5\pm0.393.8±0.893.8\pm0.80.654±0.0040.654\pm0.0040.638±0.0030.638\pm0.003
Fragm. link.98.6±0.698.6\pm0.680.6±1.580.6\pm1.50.541±0.0010.541\pm0.0010.500±0.0020.500\pm0.002
OmniInventScaffold dec.97.8±0.397.8\pm0.388.6±1.288.6\pm1.20.533±0.0040.533\pm0.0040.452±0.0040.452\pm0.004
Motif ext.98.9±0.298.9\pm0.293.1±0.793.1\pm0.70.651±0.0030.651\pm0.0030.631±0.0030.631\pm0.003

Across all three tasks, the ablated models perform comparably to OmniInvent on every metric. Neither the 2D positional encoding nor prefix attention appears necessary for substructure-constrained design with blank infilling.

These results surprised us. As noted in the manuscript, we adopted both components from Du et al., who showed that they improve performance on natural language modeling tasks in the original GLM. SMILES syntax differs significantly from natural language, however, so the benefits may not transfer. We will nevertheless include these ablations in the manuscript for transparency.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of initial batch of changes

We thank all the reviewers for their comments so far. No matter the final outcome, we feel that the points raised in the reviews will help us substantially improve the manuscript.

Two reviewers remarked that they considered the novelty of our work to be limited. We believe this is worth highlighting again, so we reproduce our reply here as well. The main novelty of our contribution can be summed up as:

  1. Our random graph masking strategy is to our knowledge unique and serves and important inductive bias in aligning the model with downstream tasks (i.e. scaffold decoration or scaffold hopping), as evidenced by the ablation in Table 4. This approach is substantially different from other pretraining strategies demonstrated in other works.
  2. To our knowledge, the first application of blank infilling to molecule generation tasks, utilizing a powerful generalist formulation to allow our model to solve essentially all molecular design tasks without resorting to rule-based fragmentation, which is a limitation shared by all existing generalist models.

In response to reviewer comments, we have made the following modifications to our manuscript with smaller changes bolded:

New contribution statement (Introduction)

Our contributions are threefold: we present what is, to our knowledge, the first application of autoregressive blank infilling to molecular design, enabling a single model to handle de novo and substructure-constrained design on standard \gls{smi} without rule-based fragmentation; we introduce OmniInvent, a transformer-based chemical language model trained with a random graph-based masking strategy that aligns training examples with the structure of molecular graphs; and we demonstrate competitive unguided and RL-guided generation across de novo and substructure-constrained design.

Elaboration on SAFE pitfalls

Introduction: Recent developments in CLMs have shown that a single model can support multiple design tasks by operating on fragment-based representations such as SAFE (Noutahi et al., 2024; Lee et al., 2025; Kaech et al., 2026). These representations require explicit fragmentation and reassembly conventions, however, introducing potentially unintentional inductive biases.

Related work: These methods show that general-purpose molecular generators can be competitive across tasks. However, fragment-based representations require molecular fragmentation and thus rely on hand-crafted bond-breaking rules. This renders some molecules unrepresentable (Noutahi \textit{et al.}, 2024) and discards stereochemical information carried by the unfragmented SMILES sequence if stereo bonds are broken (Berenger, 2021).

Softening of our performance claims in abstract and conclusion (removals bolded)

Abstract: In goal-directed design, it matches or exceeds state-of-the-art performance on the Practical Molecular Optimization benchmark for de novo design and achieves competitive performance in substructure-constrained design despite being trained as a general-purpose model.

Conclusion: Despite its general-purpose pretraining, it achieves performance comparable to both generalist and specialized models, with high validity, uniqueness, and diversity in direct sampling as well as surpassing or matching state-of-the-art results on the PMO benchmark for goal-directed de novo design.

Elaboration on interpretation of source-to-molecule distance metric (Results)

Source molecule distances were lower than for the baselines that reported this metric; in other words, OmniInvent produced internally diverse sets of samples that were more similar to the source drug than those of the baselines. This similarity is not necessarily beneficial, however, since it may reflect a narrower exploration of chemical space. This is desirable in analogue generation (Tibo et al., 2024), while broader exploration may be advantageous in e.g. scaffold hopping.

Additional ablation on de novo capabilities of the pfull=0p_\text{full}=0 model (Ablations)

Table 5: Results from de novo generation for the pfull=0p_\text{full}=0 ablation on de novo design. Results were averaged over three independent runs, with errors representing the standard deviation. Mean molecular weights (MW in the table) were computed for all valid molecules across the three runs.

AblationValid. (%)Uniq. (%)DiversityMean MW (Da)
OmniInvent98.1±0.198.1\pm0.1100.0±0.0100.0\pm0.00.888±0.0000.888\pm0.000378
OmniInvent$_\text{L}$98.8±0.198.8\pm0.1100.0±0.0100.0\pm0.00.887±0.0000.887\pm0.000381
pfull=0p_\text{full}=097.3±0.097.3\pm0.099.6±0.099.6\pm0.00.895±0.0000.895\pm0.000326
Training data---------404

We additionally find that the model trained with pfull=0p_\text{full}=0 yields de novo samples with high validity, uniqueness, and diversity as shown in Table 5. This implies that blank infilling with the random graph masking strategy on its own gives the model the capability to generate full molecules. However, the slightly lower validity and uniqueness than the OmniInvent variants trained with pfull>0p_\text{full}>0 implies a difference in what the ablated model learns; in particular, all of these OmniInvent variants exhibited a uniqueness of 100 % after rounding (Table 11). Furthermore, the average molecular weight of samples drawn from the ablated model was significantly lower than those from the ChEMBL-trained models in Table 5, and importantly further away from that of the training dataset. This implies that the inclusion of fully masked training examples helps align the output distribution with the training data.

Additional experiments

We are currently running additional experiments, which will be added as soon as possible:

  1. Prefix attention and positional encoding ablation (done; see below).
  2. Additional substructure-constrained design experiments (done; see below).
  3. Substructure preservation verification (done; see below).
AuthorsOpenReview ↗

Additional experiments

Additional susbtructure-constrained design experiments

We have now conducted additional experiments in the goal-directed substructure-constrained setting. To this end, we selected a known ligand each for GSK3β and JNK3 from the Protein Data Bank (ligand 6QH from PDB entry 5K5N and ligand JK1 from PDB entry 3FI2, respectively). We selected substructures from each to serve as scaffolds/decorators and conducted fragment linking as well as scaffold decoration. As scoring functions, we used the geometric mean of Quantitative Estimate of Drug-Likeness and the corresponding target's activity as predicted by machine learning models from Therapeutics Data Commons. The precise setups are described in Section F.2.3 of the updated manuscript.

As shown in Figure 4, OmniInvent achieves performance comparable to (or slightly better than, albeit within one standard deviation) LinkINVENT and LibINVENT.

Substructure preservation

As noted by one of the reviewers, chemical validity in the substructure-constrained setting does not necessarily guarantee preservation of the substructures that are conditioned on. We therefore recomputed our main results, with validity now requiring both substructure preservation as well as chemical validity:

ModelExperimentValidity (%)Uniquen. (%)DiversityDistance
Linker design99.5±0.299.5\pm0.283.8±1.283.8\pm1.20.532±0.0030.532\pm0.0030.524±0.0010.524\pm0.001
OmniInvent, oldScaffold dec.97.6±0.697.6\pm0.693.8±0.993.8\pm0.90.568±0.0050.568\pm0.0050.474±0.0050.474\pm0.005
Motif ext.99.0±0.399.0\pm0.397.4±0.497.4\pm0.40.664±0.0030.664\pm0.0030.656±0.0020.656\pm0.002
Linker design99.5±0.299.5\pm0.283.8±1.283.8\pm1.20.532±0.0030.532\pm0.0030.524±0.0010.524\pm0.001
OmniInvent, newScaffold dec.96.8±0.796.8\pm0.793.8±0.993.8\pm0.90.566±0.0050.566\pm0.0050.472±0.0050.472\pm0.005
Motif ext.99.0±0.399.0\pm0.397.4±0.497.4\pm0.40.664±0.0030.664\pm0.0030.656±0.0020.656\pm0.002
Linker design99.4±0.299.4\pm0.280.8±1.180.8\pm1.10.530±0.0040.530\pm0.0040.496±0.0050.496\pm0.005
OmniInvent$_\text{L}$, newScaffold dec.98.0±0.698.0\pm0.692.2±1.392.2\pm1.30.553±0.0050.553\pm0.0050.449±0.0060.449\pm0.006
Motif ext.99.3±0.399.3\pm0.397.9±0.997.9\pm0.90.656±0.0020.656\pm0.0020.637±0.0010.637\pm0.001
Linker design99.4±0.299.4\pm0.280.8±1.180.8\pm1.10.530±0.0040.530\pm0.0040.496±0.0050.496\pm0.005
OmniInvent$_\text{L}$, newScaffold dec.97.0±0.397.0\pm0.392.2±1.492.2\pm1.40.550±0.0050.550\pm0.0050.447±0.0060.447\pm0.006
Motif ext.99.3±0.399.3\pm0.397.9±0.997.9\pm0.90.656±0.0020.656\pm0.0020.637±0.0010.637\pm0.001

As shown above, this leads to a minor degradation on the scaffold decoration experiment, while the other settings are unaffected. This does not change any conclusions in the manuscript, however. We will incorporate this change into the results of the manuscript shortly.

Additional ablations

We have added ablations on the position encoding scheme as well as prefix attention to Appendix I.

ModelTaskValid. (%)Uniq. (%)DiversityDistance
Fragm. link.98.1±0.498.1\pm0.486.6±1.186.6\pm1.10.553±0.0040.553\pm0.0040.508±0.0030.508\pm0.003
CausalScaffold dec.96.7±0.896.7\pm0.889.9±1.389.9\pm1.30.546±0.0030.546\pm0.0030.463±0.0030.463\pm0.003
Motif ext.98.7±0.398.7\pm0.394.4±1.094.4\pm1.00.653±0.0010.653\pm0.0010.635±0.0030.635\pm0.003
Fragm. link.99.1±0.299.1\pm0.282.8±1.182.8\pm1.10.540±0.0020.540\pm0.0020.499±0.0030.499\pm0.003
1D RoPEScaffold dec.95.6±0.595.6\pm0.589.7±1.489.7\pm1.40.530±0.0030.530\pm0.0030.458±0.0010.458\pm0.001
Motif ext.99.3±0.199.3\pm0.193.2±0.393.2\pm0.30.644±0.0020.644\pm0.0020.631±0.0010.631\pm0.001
Fragm. link.98.2±0.498.2\pm0.484.4±1.884.4\pm1.80.551±0.0030.551\pm0.0030.510±0.0020.510\pm0.002
Causal + 1D RoPEScaffold dec.96.1±0.996.1\pm0.991.7±1.091.7\pm1.00.553±0.0030.553\pm0.0030.466±0.0070.466\pm0.007
Motif ext.98.5±0.398.5\pm0.393.8±0.893.8\pm0.80.654±0.0040.654\pm0.0040.638±0.0030.638\pm0.003
Fragm. link.98.6±0.698.6\pm0.680.6±1.580.6\pm1.50.541±0.0010.541\pm0.0010.500±0.0020.500\pm0.002
OmniInventScaffold dec.97.8±0.397.8\pm0.388.6±1.288.6\pm1.20.533±0.0040.533\pm0.0040.452±0.0040.452\pm0.004
Motif ext.98.9±0.298.9\pm0.293.1±0.793.1\pm0.70.651±0.0030.651\pm0.0030.631±0.0030.631\pm0.003

Across all three tasks, the ablated models perform comparably to OmniInvent on every metric. Neither the 2D positional encoding nor prefix attention appears necessary for substructure-constrained design with blank infilling.

These results surprised us. As noted in the manuscript, we adopted both components from Du et al., who showed that they improve performance on natural language modeling tasks in the original GLM. SMILES syntax differs significantly from natural language, however, so the benefits may not transfer. We will nevertheless include these ablations in the manuscript for transparency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.