- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce ToolMol - a genetic algorithm incorporating LLM-agent that is used to generate new candidates based on parent structures. LLM-agent has access to several RDKit-based tool for different molecular modifications which prevents agent from generating invalid and low-quality SMILES strings. The authors demonstrate that ToolMol outperforms several existing approaches in three biding affinity, QED and SA-based metrics. ToolMol is also shown to surpass MF-LAL in ABFE scores across 2 targets.
Critical strengths
- The paper addresses an important limitation of LLMs as molecule generators. The idea to supply LLM with RDKit-based tools is interesting.
- The authors perform computationally expensive ABFE simulations. Although the comparison is limited this provides additional evidence that the generated molecules can achieve favorable binding-affinity estimates under a different computational evaluation method.
Critical weaknesses
o1. My main concern is that the same scores (binding affinity, QED, SA) are used both as optimization objectives and comparison criteria in Table 1. This introduces a risk of oracle overoptimization meaning that ToolMol might become particularly good at discovering molecules that e.g. Boltz-2 systematically overestimates. If the authors added ABFE evaluation for all methods in Table 1, ToolMol would not necessarily remain the best-performing method, so it requires clarification. 2. The reported ABFE improvements are substantially smaller after applying drug-likeness and synthesizability filters. In particular, the filtered improvements over MF-LAL are only 0.6 kcal/mol for c-MET and 0.2 kcal/mol for BRD4, against considerable variation in the reported scores. The evidence does not establish that these differences are statistically significant. 3. It remains unclear whether the reported gains in ToolMol's performance arise from the LLM's chemical reasoning or simply from the constrained RDKit action space and deterministic molecular editing operations. What if we remove LLM and randomly apply the tools to molecules? 4. One of the main motivations for ToolMol is the high rate of invalid molecules generated by LLMs. However, it is unclear whether the proposed multi-step tool-calling procedure is more computationally efficient than simply generating molecules directly with an LLM and discarding invalid outputs. ToolMol allows up to 10 sequential tool-calling steps per molecule, potentially requiring substantially more LLM inference than direct molecular generation. The paper does not provide a comparison of the total inference cost, number of LLM calls, or wall-clock time required to obtain a fixed number of valid molecules. 5. The evaluation covers only three protein targets which can be insufficient because it may not represent the diversity of binding-pocket geometries and target classes encountered in drug discovery.
Questions for authors
- Why are ABFE results in Table 2 restricted to MF-LAL and other baselines are excluded? Why is ACAA1 excluded from Table 2? Could the authors provide ABFE results for at least top-3/5 molecules generated by other baselines?
- Could the authors report the average number of LLM calls, generated tokens, and inference time required per valid molecule? How does ToolMol compare against direct generation with validity filtering under a matched computational budget?
- Could the authors compare ToolMol against a random algorithm that applies exactly the same molecular editing tools within the same genetic algorithm and oracle budget?
- In Table 2, the ABFE improvements of filtered ToolMol molecules over MF-LAL are relatively small compared to the reported standard deviations. Are these differences statistically significant? Could the authors report confidence intervals or statistical tests to better support the claimed improvements?
8 comments
Rebuttal by Authors (2/2)
The evaluation covers only three protein targets which can be insufficient because it may not represent the diversity of binding-pocket geometries and target classes encountered in drug discovery.
While we understand this concern, we raise that our protein targets were intently chosen with structural and functional diversity in mind. c-MET is a receptor tyrosine kinase, BRD4 is a bromodomain-containing protein, and ACAA1 is a peroxisomal thiolase. We believe that ToolMol’s consistent performance across these structurally diverse binding sites reflects a genuine methodological advancement rather than coincidence, given that these three handpicked binding tasks reflect unique chemical functions, structures, and properties. We would also like to note that many published, established work in this field evaluate on a similar number of protein targets. MOLLEO evaluates on 3 targets, LIMO evaluates 2, GENTRL focuses on a single target (DDR1) and achieves fantastic results on it. Our work focuses on slower but more accurate oracles such as Boltz-2 and ABFE, as this reflects higher potential for real-world applicability, which is always the ultimate goal of computational drug discovery. It is our belief that while this restricts feasible evaluations to fewer targets, our careful selection of targets and higher fidelity evaluations provides results that are still important and impactful.
Why are ABFE results in Table 2 restricted to MF-LAL and other baselines are excluded? Why is ACAA1 excluded from Table 2?
ABFE is extremely expensive to evaluate, and as such we did not have the compute or time to evaluate on every single baseline found in Table 1. We hope that the added MOLLEO result above is helpful and informative. ACAA1 is excluded because the primary baseline, MF-LAL, is only evaluated on c-MET and BRD4. Evaluating on ACAA1 would require reproducing results with the MF-LAL method, which involves computing further ABFE signals within their multi-fidelity active training pipeline, which was also outside the scope of our compute budget.
How does ToolMol compare against direct generation with validity filtering under a matched computational budget?
This ablation already exists in Table 3 of section 4.3 of our manuscript; we ran MOLLEO but allowed the LLM to retry invalid generations up to 10 times, ensuring that every molecule in the final generated set was LLM-produced. As shown by Table 3, this direct generation scheme is significantly outperformed by ToolMol.
Could the authors compare ToolMol against a random algorithm that applies exactly the same molecular editing tools within the same genetic algorithm and oracle budget?
We thank the reviewer for suggesting this valuable ablation, and have designed a random version of ToolMol, where we reuse the exact same seven tools and input random constrained parameters that are guaranteed to be valid. The results with the multi-objective metrics from Table 1 (filtered affinity, hypervolume) are as follows:
| ToolMol (random) | ToolMol (LLM) | |
|---|---|---|
| c-MET FA | ||
| c-MET HV | ||
| BRD4 FA | ||
| BRD4 HV | ||
| ACAA1 FA | ||
| ACAA1 HV |
The random tool-use ablation is significantly outperformed by ToolMol guided by LLM reasoning, showing that the LLM operator provides valuable decision-making within the design process. These random tools yield similar results to Graph-GA in Table 1 (average rank of 5.0), which makes sense since Graph-GA is also a random rule-based operator.
Thank you for your clarifications and additional results. I appreciate that ABFE calculations are computationally expensive, and I acknowledge the evidence supporting Boltz-2 as a useful proxy for real-world binding affinity. I am also satisfied with ToolMol (random) results and clarifications regarding the number of LLM calls. However, my main concern (point 1 in "Critical Weaknesses") remains unresolved. In Table 1, ToolMol explicitly optimizes binding affinity, QED, and SA, and is subsequently evaluated using these same objectives. While this is appropriate for assessing optimization performance, the comparison may favor methods that explicitly optimize these objectives over methods that do not have access to the same iterative optimization procedure or oracle feedback (Pocket2Mol, TAGMol, and PAFlow). I understand that you provide ABFE results as an independent molecules assessment, but you do it only for MF-LAL (which is not included in Table 1 for some reason) and for MOLLEO, leaving the performance of the other baselines under an independent evaluation unclear. Given that ABFE is computationally expensive, you could think of other possible independent metrics that can support results reported in Table 1, e.g. QSAR IC50 predictors, medicinal chemistry experts assessments, alternative synthetic accessibility scores (RAscore) etc.
Rebuttal by Authors (1/2)
We acknowledge the reviewer’s concern about limited evaluation. Due to the extremely expensive cost of ABFE, we are unable to report ABFE evaluation for all methods in Table 1. However, we were able to evaluate molecules generated by MOLLEO, the closest methodological neighbor and most competitive baseline. We ran experiments on the top 15 molecules from the filtered sets for MOLLEO on c-MET and BRD4, consistent with the existing ABFE experiments shown in Table 2. For MOLLEO on c-MET, we obtain ABFE predicted binding affinity of −6.57±4.17, with corresponding average QED and SA of 0.60±0.08 and 2.53±0.17 respectively. For MOLLEO on BRD4, we obtain ABFE predicted binding affinity of −6.40±4.50, with corresponding average QED and SA of 0.61±0.11 and 2.63±0.28 respectively. Contrasting these results with ToolMol (filtered) and MF-LAL in Table 2, we note that MOLLEO's c-MET affinity result is not only significantly worse than ToolMol, but is also worse than the MF-LAL baseline. MOLLEO's BRD4 affinity result is also weaker than ToolMol’s, and demonstrates weaker QED (but slightly stronger SA). Altogether, we believe that this additional ABFE result demonstrates that the Boltz-2 scores in Table 1 have good correlation with ABFE scores, and that the high ABFE scores achieved by ToolMol over multiple existing baselines are non-trivial and non-coincidental.
As an aside, the reviewer’s concern about overoptimizing for Boltz-2 is understandable, but we would like to reference a recent study [1] in which candidates filtered by Boltz-2 predicted affinity were experimentally verified in a wet-lab. Candidate compounds that scored highly on Boltz-2 reported a 70% success rate in inhibitory activity. We would like to note that we chose a more expensive oracle in Boltz-2 over cheaper oracles such as Vina exactly for the benefit of relying on an oracle that has demonstrated real-world success, so that we are less concerned about overoptimizing for this expensive oracle. We would also like to note that much well-regarded literature in this field trains their generative method on CrossDocked2020 and evaluates on the CrossDocked2020 test set, using Vina both as a training signal and as the evaluation metric. Our pattern of relying on Boltz-2 in optimization and in evaluation is fundamentally the same.
[1] Kawano, Shinya, et al. “Efficient Discovery of Potent AKR1B10 Inhibitors Using Co-folding Model Boltz-2.” Journal of Chemical Information and Modeling, vol. 66, no. 18, 2026, pp. 12100–14, https://doi.org/10.1021/acs.jcim.6c00874.
We acknowledge that our gains on the filtered sets are not statistically significant, but would like to clarify that it was never the intent/focus of this work to achieve state-of-the-art in ABFE. We included these evaluations as a supporting result alongside Boltz-2, and the most surprising/important result from our manuscript is that we are able to match and slightly exceed the ABFE results of MF-LAL, which is a method explicitly tuned for this oracle that includes explicit ABFE signal within its active learning pipeline. ToolMol receives zero ABFE guidance and has no knowledge of this oracle whatsoever, and yet yields such strong results simply by optimizing Boltz-2 predicted affinity with a tool-assisted LLM. This is the only result and idea that we wanted to raise in section 4.2.
We find that the average number of LLM prompts per generated molecule is around 5.4 in ToolMol, compared to a single call for MOLLEO. However, we observe that since compute is generally bottlenecked by the Boltz-2 evaluation rather than LLM inference (which is widely available and supported by massive infrastructure in the modern era), the total actual runtime of ToolMol is negligibly greater than that of MOLLEO.