acceptodds
Withdrawn from ICLR 2027

MolHop: Scaffold-Hopping Molecular Design with Agentic Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Molecular design is often formulated as an optimization problem over existing molecular spaces. However, molecular optimization and the discovery of structurally distinct replacement molecules represent two fundamentally different search problems. LLM-based molecular design agents typically perform the former, conducting local refinement within the neighborhood of existing structures to improve target properties through structural modifications. In contrast, the latter requires exploration of a broader chemical space to discover molecules with entirely different scaffolds while preserving the functional characteristics of reference molecules.We propose MolHop, an agentic framework for cross-scaffold molecular discovery that enables models to navigate and edit within a pre-organized chemical space. MolHop organizes approximately 1.6 million molecular scaffolds from commercial molecules into a coarse-to-fine hierarchical action space. The agent selects new exploration regions through scaffold navigation actions and adjusts candidate properties through molecular editing. The framework maintains historical trajectory memory to support long-horizon search processes.On drug molecule and commercial compound tasks, connecting Grok-4.6 and DeepSeek-Flash-V4.1 to the MolHop framework substantially improves their cross-scaffold joint success rates. We develop MolHop-9B through agentic reinforcement learning, and MolHop-9B outperforms generative baselines including GPT-5.6-sol, Claude-5-Opus, and Gemini-3.1-Pro, while generating candidate molecules with larger scaffold differences from the reference molecules.These results demonstrate that effective molecular discovery beyond the traditional optimization paradigm requires structured chemical and action spaces, enabling agents to explore new structural regions rather than merely refine existing molecules.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Molecular design, LLM agents, agent harness, agentic reinforcement learning, multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1iPdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a method for conducting molecular design with LLMs. Unlike many other approaches, the LLM does not directly generate molecular representations (e.g. SMILES). Instead, a hierarchical representation of molecular scaffolds is introduced; a harness allows the LLM to navigate the resulting scaffold search space to select molecular scaffolds that are subsequently locally modified to produce new molecules. The authors name this framework MolHop and investigate its performance compared to a series of baselines. The focus is not on traditional molecular optimization, where the generative model optimizes chemical matter for certain properties; instead, the work focuses on property matching under novelty constraints, i.e., proposing molecules that match the properties of a refereence without exhibiting a distinct chemical structure.

They run comparisons using 1) LLMs that directly generate SMILES, 2) off-the-shelf LLMs with the MolHop harness, as well 3) LLMs fine-tuned to make use of the harness (MolHop-2B, MolHop-4B, and MolHop-9B). In addition, they include a state-of-the-art generative model for molecular design (GenMol) as as a baseline. The harness significantly improves the reported metrics compared to direct LLM generation of SMILES, and the fine-tuned agents yield further improvements.

Critical strengths

The idea of using structured search spaces is a compelling concept in chemical design; it may serve to anchor the broader exploration of chemical space in the vicinity of known scaffolds while allowing for local exploration through chemical edits. Finding optimal molecules under multiple constraitns is incredibly important for many chemical industries, and MolHop performs well on the evaluations presented in the paper. The emphasis on property matching with different scaffolds is to my knowledge novel in the context of LLM-based molecular design and is an interesting angle; most conventional approaches to molecular generation focus on traditional molecular optimization.

Critical weaknesses

\textbf{Reproducibility.} As it stands, several things are lacking from the paper from a reproducibility perspective. Several of these factors additionally make it impossible to judge whether or not fair baseline comparisons are made. Below are some of the more egregious omissions I found during reading.

Promts for the LLM-based methods appear to be missing from both the main paper and the appendix; I imagine performance could be rather sensitive to that.

Similarly, it is not clear how the GenMol baseline experiments were performed. During my first readthrough I assumed that GenMol was used for scaffold hopping; that does not seem reasonable in light of the approach MolHop takes to drug design, however. Allowing GenMol to only modify the scaffold/core would be unfair in this setting, given that the other methods modify the entire starting molecule. It would thus be instructive to elaborate on how it was used in this work, including inference-time hyperparameters and the database used to construct the initial fragment sets if applicable.

It is not fully clear to me what objectives were optimized for, and when. Section C.2 states that "MolHop and GenMol remain largely unchanged because their objectives already include H1 scaffold transition." First off, it is never stated what GenMol is optimizing for beyond this sentence from what I can tell. Second, the cost function used for RL with MolHop involves MD as well as improvement in SA. Precisely how is H1 scaffold transition involved in model training and inference? Were the optimization objectives different between MolHop and the baselines? The aforementioned sentence seems to imply that, which makes the comparison with MolHop unfair.

\textbf{Statistical significance of results.} Results between molecular optimization runs can vary a fair bit. For the experimental results, were duplicate experiments ran? If so, (at the very minimum) adding the interrun standard deviation to the results tables would give a better idea of how dependable MolHop is in comparison to the baselines. If not, it is entirely possible that part of the gap between MolHop and baselines is due to lucky seeds.

\textbf{Agent actions and trajectory memory.} The explanations of precisely what the MolHop harness allows the agent to do are underdeveloped. I would suggest that the authors take a fresh look at all of Appendix A and fill in the details that are missing. Some of the most glaring omissions are listed below, but it is not unlikely that I have missed something.

What exactly is shown to the agent when it performs the nav_pick step and how does it make its choice? I imagine many potential scaffolds fit into a given query, so understanding what the agent uses to make its choice would improve understanding of the proposed methodology.

One category of molecular editing actions regards fragments. Deleting a fragment requires defining a fragmentation scheme for the molecule, while fragment replacement and swapping requires a list of fragments given that harness-using agents are stated to not generate free-form SMILES to modify molecules. How were they defined, and where did they come from?

How are editable sites defined for the mol_edit action, and how are the allowable actions defined?

Figure A2 illustrates the structure of the MolHop trajectory memory, but an example snapshot showing what the agent actually sees would be helpful for understanding. Similarly, it is mentioned that the trajectory contains a summary generated by the harness according to predefined rules; these rules do not appear to be stated in the paper nor appendix.

\textbf{Scaffold hopping?} The paper's title and introduction heavily implies that MolHop conducts scaffold hopping. In the context of molecular design, however, scaffold hopping usually entails replacing only the core of a molecule, keeping any decorators (or, loosely speaking, functional groups) intact. (There are exceptions to this, but none of them really conform to what MolHop does; see e.g. doi:10.1016/j.drudis.2011.10.024) That is not what MolHop does; when the scaffold changes, the molecule in the previous step is entirely discarded (but kept in the trajectory memory). I am not entirely sure what I would call that, but I would be careful in appropriating established terms without nuance.

Furthermore, there are several published generative machine learning models designed for scaffold hopping in the traditional sense. While GenMol in principle supports scaffold hopping, the accompanying paper does not demonstrate that capability in a goal-directed setting, nor is it apparent that this capability was used in the baseline runs. It would thus be appropriate to include models that demonstrate this capability as baselines.

\textbf{Task complexity.} While tasks involving multi-parameter/property optimization are challenging, the chosen sub-tasks (logP, TPSA, MW, ASA, and MolMR) are all properties that can, in principle, be reasoned about from a molecular structure by an expert chemist. They can thus be considered to be relatively easy. Drug design campaigns typically involve target binding as well; such tasks are a lot tricker, as it involves interactions between the target and the designed ligand. The absense of evaluations on this class of tasks undermines the potential impact of the work in drug design, especially given the use of a drug molecule-derived scaffold dataset. Similar requirements on molecules are present in other domains, e.g., agrochemicals, sensor design, and catalysis. While commonly used methods used for binding estimation are computationally expensive (e.g. docking or binding free energy), there are inexpensive alternatives available (e.g. the JNK3 activity oracle in the Therapeutics Data Commons package).

Questions for authors

See above. The "Reproducibility," "Statistical significance of results," and "Agent actions" sections above are the largest reason for my low score; I believe those issues need to be rectified before acceptance could be considered. The "Scaffold hopping?" and "Task complexity" sections had less impact on my final score, but are nevertheless weaknesses that should be adressed and/or acknowledged.

Finally, I was hoping that the authors could clarify how rings were counted; Figs. E2b, E3b, and E4b appear to show molecules with fewer rings than they are labeled with.

Reviewer_h2EaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MolHop turns scaffold selection into explicit search over a hierarchy of approximately 1.67 million Enamine-derived Bemis–Murcko scaffolds. An agent can navigate and filter this space, edit a selected molecule, inspect property feedback, and use environment-maintained trajectory memory. The authors train Qwen3.5 models by SFT on tool-use trajectories followed by GRPO and test whether output molecules change H1 scaffold category, preserve five calculated physicochemical properties, and improve synthetic-accessibility score. The frozen-model harness comparisons and component ablations show that structured search is useful for this benchmark.

Critical strengths
  1. Explicit scaffold navigation is an interesting proposal over free-form SMILES editing for long-horizon search.

  2. Connecting the harness to frozen Grok and DeepSeek models improves joint@H1 on both 200-molecule test sets (Table 1), showing that the environment itself contributes beyond training a new language model.

Critical weaknesses
  1. Limited methodological novelty. The method section primarily describes an engineered search environment. The agent is then trained using standard SFT and GRPO.

  2. Missing recent baselines. The comparison omits Kimi K3 and Qwen3.8-Max, although both were publicly available well before the submission deadline. The comparison should use a matched tool interface and interaction budget.

  3. The training contribution is not fully isolated. Table 3 compares base and SFT models on tool-protocol reliability, and the GRPO curves show improvement, but the main test tables do not include SFT-only joint@H1 under the same 50-turn budget. Since Grok-generated demonstrations include both successful submissions and budget-exhausted trajectories, this comparison is needed to quantify what RL adds beyond imitation and tool access.

  4. The scaffold library is built from the same Enamine collection used for the commercial test references. Exact test InChIKeys are excluded from SFT/RL starting points, but the paper does not establish that their exact scaffolds or close analogues are absent from the searchable library. This is a legitimate retrieval setting, yet it weakens claims of generalization beyond the indexed chemical space. The restricted-library experiment is helpful but uses only 83 targets and a property-based masking procedure.

  5. The best 9B model improves over the strongest one-shot model by just 2.0 percentage points on the drug test set. Also, the continuous RL reward optimizes property distance and SA ratio but does not directly include H1 transition, making the mechanism behind scaffold transitions important to explain.

Questions for authors
  1. What is joint@H1 for SFT-only 2B/4B/9B agents evaluated with the identical harness, turn limit, and test sets?

  2. How much of the gain remains against a 50-turn iterative baseline with the same property oracle and a flat, budget-matched scaffold search interface?

  3. How often does the searchable library contain the exact H7 scaffold of a test reference or a very close analogue, and how does performance change when those entries are withheld?

Reviewer_6YmxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of cross-scaffold molecular design, aiming to address the limitations of existing LLM-based molecular design methods that primarily rely on local structural editing and struggle to explore structurally distinct molecular scaffolds. The authors propose MolHop, a framework that organizes approximately 1.66 million molecular scaffolds into a seven-level hierarchical search space and integrates scaffold navigation, molecular editing, and trajectory memory to enable cross-scaffold exploration while preserving target physicochemical properties and reducing synthetic accessibility (SA) scores. The framework employs supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO) to train agents to coordinate multi-step structural exploration and property optimization. Experiments on the ChEMBL and Enamine datasets, each containing 200 test molecules, show that MolHop-9B achieves joint success rates of 55.5% and 73.5%, respectively. Ablation studies examine the contributions of the hierarchical scaffold library, trajectory memory, and molecular editing, while experiments with a restricted scaffold library further investigate the framework's ability to optimize molecular properties beyond the library's coverage.

Critical strengths
  1. Explicit scaffold navigation is a useful design choice. The framework gives the agent actions to select a new scaffold and then edit it, while the environment keeps a record of previous candidates and property feedback (Section 3; Appendices A.2–A.3). This provides a concrete way to coordinate structural exploration and property adjustment, which is central to the task.

  2. The experiments provide useful evidence for the harness. For frozen Grok-4.6, joint@H1 rises from 41.0% to 64.0% on drug molecules and from 36.5% to 69.0% on commercial molecules when the harness is added (Table 1). The retrained MolHop-4B ablations also show lower success after removing editing, memory, or the scaffold library, with larger drops from removing memory or the library (Figure 4). These results support the value of the complete search environment, although they do not fully separate its benefits from additional search and feedback.

Critical weaknesses

W1. The one-shot comparisons do not isolate the benefit of the search design. Agent methods can use up to 50 assistant turns with tool feedback, while the LLM generation baselines directly generate candidate SMILES (Section 4.1.4). The results show the performance of complete workflows, but leave uncertainty about how much of the gain comes from the proposed search design and how much comes from repeated attempts and property feedback. GenMol is an important exception: it receives 10,000 oracle calls, so this concern should not be applied to every baseline. Table C5 reports tool-use costs for agents, but does not provide a comparison with an iterative generation baseline under a matched evaluation budget. Such a comparison would make the method's advantage more convincing. On the drug test set, the lead over the best one-shot model is only 2.0 percentage points, corresponding to 111 versus 107 successful tasks out of 200 (Table 2; Table C3). The paired outcomes or uncertainty estimates would help assess the reliability of this small gap.

W2. The additional test-set benefit of GRPO is not clearly quantified. Table 3 compares base models with SFT models on tool-use metrics. Figure C1 shows validation joint@H1 improving during GRPO, which is meaningful evidence that RL helps. However, I could not find an SFT-only joint@H1 result on the two final test sets using the same harness and inference budget. Without that comparison, it is difficult to quantify how much RL adds to the final results after SFT has taught the tool protocol. This limits my confidence in the paper's specific claim about the contribution of agentic RL, rather than in the usefulness of the harness itself.

Questions for authors
  1. Comparison under the same search budget. Could you compare the same frozen LLM using MolHop and an iterative SMILES-generation workflow, with the same property feedback and cap on property evaluations? Please clarify how property evaluations are counted separately from assistant turns and tool calls. Table C5 already reports agent tool-use costs, and the GenMol comparison includes oracle-driven optimization. This control would help determine what the structured search design adds beyond repeated generation and feedback.

  2. Additional benefit of GRPO. Figure C1 shows validation improvement during GRPO, while Table 3 evaluates tool-use capability after SFT. Could you report SFT-only and SFT+GRPO joint@H1 on both final test sets using the same harness, decoding settings, and inference budget, with checkpoints selected using validation data only? Is the first validation point in Figure C1 the SFT checkpoint before any GRPO updates? This would directly quantify the additional test-set benefit of RL while recognizing the evidence already in the training curves.

  3. Reliability of the drug-task improvement. The reported lead over the best one-shot baseline is 2.0 percentage points on the drug test set (Table 2; Table C3). Could you provide the paired success/failure counts for MolHop-9B and that baseline, together with an uncertainty estimate for the difference in success rates? In particular, how many tasks are solved only by each method? This would help assess how much confidence to place in the small aggregate lead.

  4. Contribution beyond library access. The no-edit variant reaches 43.0% and 50.0% joint success on the drug and commercial tasks (Figure 4). How does simple retrieval with property ranking and the same H1 constraint compare, with evaluation costs counted consistently? This would help distinguish the benefit of access to the scaffold library from the benefit of learning its hierarchical search policy.

  5. Scaffold constraints during RL. Appendix B.3 defines the continuous reward using property distance and normalized SA, while joint@H1 also requires a scaffold change. Can a valid submission with unchanged H1 receive the ordinary continuous reward, or does the environment reject or penalize it separately? Please clarify where this constraint enters training. This affects how I interpret the learned policy's contribution to scaffold hopping.

  6. Restricted-library result under the main budget. Appendix D uses 100 turns, compared with the main limit of 50, and selects targets without a matching bare scaffold in the restricted library. What joint success is reached on those same targets at the main turn limit? If existing trajectories allow this analysis, it would help assess how much editing beyond library coverage is possible at the main inference budget. The zero-success retrieval comparator should be interpreted in light of the target selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.