- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SMDD-Bench, a benchmark of 502 small-molecule drug design tasks spanning five task types and 102 protein targets. Agents work through a multi-turn tool interface with limited access to Boltz2 and ADMET-AI. A central contribution is witness-aware task generation: a hidden molecule satisfying the evaluation criteria is used to construct each scaffold hopping, lead optimization, or fragment assembly instance. Seven LLMs are evaluated. The best overall success rate is 40.2%, with most successes coming from lead optimization. Additional analyses examine output diversity, candidate selection, and failures across reasoning trajectories.
Critical strengths
(1)Constructing tasks around a hidden feasible molecule gives the evaluation a firmer basis. This is particularly useful for lead optimization and fragment assembly, where satisfying several requirements simultaneously is central to the task. (2)The paper goes beyond reporting model rankings.
Critical weaknesses
(1)The connection between benchmark success and real-world design quality remains insufficiently established. An independent assessment of a representative subset of witnesses and successful submissions would help establish whether the benchmark rewards chemically meaningful improvements rather than predictor-specific behavior. (2)The current baselines do not adequately calibrate task difficulty or support a broad interpretation of the low success rates.
Questions for authors
(1)Can the authors provide an independent assessment of a representative subset of witness molecules and successful agent submissions? The key question is whether improvements defined by the benchmark remain credible outside Boltz2 and ADMET-AI. (2)For scaffold hopping and fragment assembly, what proportion of submissions fail at each evaluation gate? (3)How should the scaffold hopping retrieval result of 5.8% in Table 16 be reconciled with the statement that non-LLM baselines do not outperform LLMs? (4) How stable are the reported model differences across repeated runs? The overall scores of GPT-5.4, Gemini, and Claude are close, while several task types contain only 25–60 instances.
3 comments
Thank you for the detailed response. The breakdown of failures by evaluation stage and the clarification of the retrieval baseline’s access to experimental data address some of my questions. My main concern remains whether the improvements observed in successful agent submissions hold up under independent evaluation beyond Boltz2 and ADMET-AI. The repeated runs on selected tasks also leave open whether the small differences among the leading models are stable across the full benchmark. Given the evidence in the current manuscript and response, my concerns are only partially resolved.Therefore, I will maintain my original rating.
Thank you for the response!
Our ultimate goal is to understand current model capabilities in small molecule drug design tasks as opposed to precisely differentiating the performance of leading models. However, we are currently running multiple runs of each model to get a notion of spread and share our early results on GPT5.4 and Claude:
GPT5.4
| Task type | SD |
|---|---|
| 2D pharmacophore | 0.00 |
| Interaction point discovery | 0.00 |
| Scaffold hopping | 2.36 |
| Lead optimization | 1.39 |
| Fragment assembly | 2.14 |
| Overall | 0.43 |
Claude 4.6 Sonnet
| Task type | SD |
|---|---|
| 2D pharmacophore | 0.00 |
| Interaction point discovery | 0.00 |
| Scaffold hopping | 1.86 |
| Lead optimization | 1.20 |
| Fragment assembly | 1.91 |
| Overall | 1.15 |
The increase in task count from 20 in the SMDD-Bench Diversity split to 502 for the entire SMDD-Bench contributes to a significant reduction in the overall standard deviation. We hope this alleviates your concerns regarding the variance of the runs, and we hope this addresses your concerns regarding differentiating performance of the leading models. And once again, we are running the rest of the models to get a thorough understanding of the spread of the model's performance!
My main concern remains whether the improvements observed in successful agent submissions hold up under independent evaluation beyond Boltz2 and ADMET-AI.
We want to be very clear that the benchmark is designed such that the oracles themselves are the ground truth for reasons outlined at the end of section 6. We wanted to circle back to a comment you made in your original review:
The key question is whether improvements defined by the benchmark remain credible outside Boltz2 and ADMET-AI.
The independent assessment (at greater accuracies than Boltz2 and ADMET-AI) of such improvements can only be performed at the wet lab scale -- human experts cannot simply look at molecules and determine success or favorability. The computational chemistry community already has significant precedent in using computational oracles such as Boltz2 and ADMET-AI and treating those as ground truth due to the infeasibility of sending everything to the wet lab.
We feel that there may be some gap between your expectations as well as the prior precedent within the community that we have established. We hope that we were able to clear up this gap! Please let us know if you have any other questions.
Author Response 1
First off, we would like to thank the reviewer for the review! Below are our point-by-point responses to your concerns.
We had human experts validate for task validity, threshold appropriateness, constraint satisfaction, and witness evaluation against all oracles, optimization properties, hold-constant properties, and hard constraints. We will include this in the manuscript
We strongly disagree with this statement. Small molecule drug design is a very challenging field, for both humans and LLMs. The low success rates of baseline methods calibrate how much lift LLMs can achieve beyond simple mechanisms of understanding the task.
We think there may have been a misunderstanding here – the witness molecules are derived from experimental databases such as ChEMBL. These molecules have already been lab validated at high fidelity and are then re-validated under the SMDD-Bench computational oracles. The existence of the witness molecule ensures the existence of a successful agent submission. The relationship between benchmark oracles and wet lab labels has already been validated by [1] and [2].
[1] Boltz-2: Towards Accurate and Efficient Binding Affinity Prediction, https://www.biorxiv.org/content/10.1101/2025.06.14.659707v1 [2] ADMET-AI: a machine learning ADMET platform for evaluation of large-scale chemical libraries, https://pmc.ncbi.nlm.nih.gov/articles/PMC11226862/
Below are tables with the proportion of overall submissions which fail at each gate for the scaffold hopping and fragment assembly tasks:
Scaffold Hopping:
Fragment Assembly:
The scaffold hopping retrieval baseline is retrieval and sorting over the experimental potency values from the ChEMBL database itself, which gives the retrieval baseline highly privileged information. The reasoning behind including this baseline was to measure if LLM performance came from memorizing the ChEMBL database – to which we find this is not the case.
We performed 20 repeated runs of specific tasks for the selected tasks in the SMDD-Bench Diversity experiment. Below are the results of those runs:
As seen in the table, the standard deviation is quite tight.
Once again, we would like to thank the reviewer for the reviews, and we hope we were able to address all of your concerns!