- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a method for conducting molecular design with LLMs. Unlike many other approaches, the LLM does not directly generate molecular representations (e.g. SMILES). Instead, a hierarchical representation of molecular scaffolds is introduced; a harness allows the LLM to navigate the resulting scaffold search space to select molecular scaffolds that are subsequently locally modified to produce new molecules. The authors name this framework MolHop and investigate its performance compared to a series of baselines. The focus is not on traditional molecular optimization, where the generative model optimizes chemical matter for certain properties; instead, the work focuses on property matching under novelty constraints, i.e., proposing molecules that match the properties of a refereence without exhibiting a distinct chemical structure.
They run comparisons using 1) LLMs that directly generate SMILES, 2) off-the-shelf LLMs with the MolHop harness, as well 3) LLMs fine-tuned to make use of the harness (MolHop-2B, MolHop-4B, and MolHop-9B). In addition, they include a state-of-the-art generative model for molecular design (GenMol) as as a baseline. The harness significantly improves the reported metrics compared to direct LLM generation of SMILES, and the fine-tuned agents yield further improvements.
Critical strengths
The idea of using structured search spaces is a compelling concept in chemical design; it may serve to anchor the broader exploration of chemical space in the vicinity of known scaffolds while allowing for local exploration through chemical edits. Finding optimal molecules under multiple constraitns is incredibly important for many chemical industries, and MolHop performs well on the evaluations presented in the paper. The emphasis on property matching with different scaffolds is to my knowledge novel in the context of LLM-based molecular design and is an interesting angle; most conventional approaches to molecular generation focus on traditional molecular optimization.
Critical weaknesses
\textbf{Reproducibility.} As it stands, several things are lacking from the paper from a reproducibility perspective. Several of these factors additionally make it impossible to judge whether or not fair baseline comparisons are made. Below are some of the more egregious omissions I found during reading.
Promts for the LLM-based methods appear to be missing from both the main paper and the appendix; I imagine performance could be rather sensitive to that.
Similarly, it is not clear how the GenMol baseline experiments were performed. During my first readthrough I assumed that GenMol was used for scaffold hopping; that does not seem reasonable in light of the approach MolHop takes to drug design, however. Allowing GenMol to only modify the scaffold/core would be unfair in this setting, given that the other methods modify the entire starting molecule. It would thus be instructive to elaborate on how it was used in this work, including inference-time hyperparameters and the database used to construct the initial fragment sets if applicable.
It is not fully clear to me what objectives were optimized for, and when. Section C.2 states that "MolHop and GenMol remain largely unchanged because their objectives already include H1 scaffold transition." First off, it is never stated what GenMol is optimizing for beyond this sentence from what I can tell. Second, the cost function used for RL with MolHop involves MD as well as improvement in SA. Precisely how is H1 scaffold transition involved in model training and inference? Were the optimization objectives different between MolHop and the baselines? The aforementioned sentence seems to imply that, which makes the comparison with MolHop unfair.
\textbf{Statistical significance of results.} Results between molecular optimization runs can vary a fair bit. For the experimental results, were duplicate experiments ran? If so, (at the very minimum) adding the interrun standard deviation to the results tables would give a better idea of how dependable MolHop is in comparison to the baselines. If not, it is entirely possible that part of the gap between MolHop and baselines is due to lucky seeds.
\textbf{Agent actions and trajectory memory.} The explanations of precisely what the MolHop harness allows the agent to do are underdeveloped. I would suggest that the authors take a fresh look at all of Appendix A and fill in the details that are missing. Some of the most glaring omissions are listed below, but it is not unlikely that I have missed something.
What exactly is shown to the agent when it performs the nav_pick step and how does it make its choice? I imagine many potential scaffolds fit into a given query, so understanding what the agent uses to make its choice would improve understanding of the proposed methodology.
One category of molecular editing actions regards fragments. Deleting a fragment requires defining a fragmentation scheme for the molecule, while fragment replacement and swapping requires a list of fragments given that harness-using agents are stated to not generate free-form SMILES to modify molecules. How were they defined, and where did they come from?
How are editable sites defined for the mol_edit action, and how are the allowable actions defined?
Figure A2 illustrates the structure of the MolHop trajectory memory, but an example snapshot showing what the agent actually sees would be helpful for understanding. Similarly, it is mentioned that the trajectory contains a summary generated by the harness according to predefined rules; these rules do not appear to be stated in the paper nor appendix.
\textbf{Scaffold hopping?} The paper's title and introduction heavily implies that MolHop conducts scaffold hopping. In the context of molecular design, however, scaffold hopping usually entails replacing only the core of a molecule, keeping any decorators (or, loosely speaking, functional groups) intact. (There are exceptions to this, but none of them really conform to what MolHop does; see e.g. doi:10.1016/j.drudis.2011.10.024) That is not what MolHop does; when the scaffold changes, the molecule in the previous step is entirely discarded (but kept in the trajectory memory). I am not entirely sure what I would call that, but I would be careful in appropriating established terms without nuance.
Furthermore, there are several published generative machine learning models designed for scaffold hopping in the traditional sense. While GenMol in principle supports scaffold hopping, the accompanying paper does not demonstrate that capability in a goal-directed setting, nor is it apparent that this capability was used in the baseline runs. It would thus be appropriate to include models that demonstrate this capability as baselines.
\textbf{Task complexity.} While tasks involving multi-parameter/property optimization are challenging, the chosen sub-tasks (logP, TPSA, MW, ASA, and MolMR) are all properties that can, in principle, be reasoned about from a molecular structure by an expert chemist. They can thus be considered to be relatively easy. Drug design campaigns typically involve target binding as well; such tasks are a lot tricker, as it involves interactions between the target and the designed ligand. The absense of evaluations on this class of tasks undermines the potential impact of the work in drug design, especially given the use of a drug molecule-derived scaffold dataset. Similar requirements on molecules are present in other domains, e.g., agrochemicals, sensor design, and catalysis. While commonly used methods used for binding estimation are computationally expensive (e.g. docking or binding free energy), there are inexpensive alternatives available (e.g. the JNK3 activity oracle in the Therapeutics Data Commons package).
Questions for authors
See above. The "Reproducibility," "Statistical significance of results," and "Agent actions" sections above are the largest reason for my low score; I believe those issues need to be rectified before acceptance could be considered. The "Scaffold hopping?" and "Task complexity" sections had less impact on my final score, but are nevertheless weaknesses that should be adressed and/or acknowledged.
Finally, I was hoping that the authors could clarify how rings were counted; Figs. E2b, E3b, and E4b appear to show molecules with fewer rings than they are labeled with.