- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new paradigm for evaluating drug molecules, where a large language model acts as medicinal chemistry judge and can yield both quality scores and explanatory texts. The authors were motivated by the belief that existing quantitative indicator assessments and multifactorial considerations in real drug discovery are insufficient. Its core idea is to use a large number of molecules already evaluated by medicinal chemistry experts, covering a wide range of medical chemistry fields, and dividing them into three independent environments to serve as testing benchmarks to train the optimal large model evaluation configuration in terms of molecular representation, prompt style, and configuration. The main experimental results show that using the evaluation results of the trained large language model as a FREED++ reward function can effectively improve candidate molecular quality and better reflect real-world drug design scenarios.
Critical strengths
-
Based on literature and actual drug design data, this paper indeed addresses the current gap between quantitative evaluation indicators and actual drug design needs, and uses the understanding and reasoning capabilities of large models to solve them.
-
The evaluation paradigm of LLM can serve as a flexible intermediate layer between traditional quantitative metrics and human experts. It can serve as a reward function for reinforcement learning-type models to assist quantitative metrics in optimizing molecular generation, and can also serve as guidance and interpretability for explanatory texts in actual wet experiments, forming a complete closed loop throughout the generation process.
-
From the experimental results, the evaluation paradigm of LLM can, in most cases, indeed lead to improvements in quantitative indicators of the final candidate molecules.
-
From the perspective of specific method design, the evaluation paradigm of LLM provides multiple representation methods, prompt styles, and large model configurations, making it easier to replicate subsequent experiments and initiate optimization tasks targeting specific targets.
Critical weaknesses
-
Training LLMs solely through prompt tuning as an evaluation paradigm results are closely tied to the model's own performance without a precise metric to measure the quality of molecular generation.
-
From the experimental results of the five targets, the evaluation paradigm of LLM does bring optimization to most indicators, but it does not necessarily align with expert evaluations. This seems to contradict the original design aimed at compensating for the deviation between quantitative indicators and actual drug design, rather than simply improving a single quantitative indicator in a particular aspect.
-
Current experiments do not compare results from other reward functions, human expert rules, and multi-objective optimization methods, which is insufficient to prove the effectiveness of the evaluation paradigm of LLM in final experiments.
-
The evaluation paradigm of LLM proposed in this paper includes two outcomes, numerical indicators and interpretive analysis, but the consistency judgment between explanatory text and continuous numerical indicator results is insufficient and unclear.
Questions for authors
-
MedChemJudge, which serves as the benchmark, is evaluated by pharmaceutical chemistry experts without a completely fair evaluation, and its results are discrete quantities, whereas LLM evaluation paradigms are continuous values. Will this bias affect the pre-training configuration of subsequent large models?
-
On the main experimental table for five targets, how is it determined that the results of molecules generated after adding the evaluation paradigm of LLM better align with the pharmacology experts' judgment, and does this truly meet the expected results?
-
How are the numerical results of the evaluation paradigm of LLM calculated, and how are they subsequently used to optimize the reward function of reinforcement learning generation models?
-
Have ablation experiments been conducted on reinforcement learning generative models based on the evaluation paradigm of LLM and other quantitative metrics, or on MedChemJudge's test benchmarks?
3 comments
Q1. We thank the reviewer for raising this concern. First, as we clarified earlier, MedChemJudge is not used to pretrain or fine-tune the evaluated LLMs, but only to compare and select configurations of pretrained models. Therefore, potential biases in expert annotations do not affect the models' pretrained parameters. Regarding the expert annotations, MedChemJudge was constructed using a structured evaluation protocol with predefined medicinal chemistry criteria, as detailed in Appendix A1.1. Importantly, the difference between discrete expert labels and continuous LLM scores is intentional. We evaluate agreement primarily through ranking-based metrics, which assess whether continuous LLM scores reproduce the ordering of expert-defined quality categories without requiring identical scoring scales. We would appreciate it if the reviewer could clarify what specific aspects of the benchmark evaluation are considered insufficiently fair, as this would help us better understand and address the concern.
Q2. The alignment with medicinal chemistry expert preferences is determined through an independent expert evaluation, as described in Appendix A9. For each target and reward regime, we generate 1,000 molecules and select the top five candidates based on a combination of QED, SA Score, Brenk alerts, and docking score. These molecules are then independently assessed by a medicinal chemistry expert, without relying on the LLM-generated scores. As reported in Table 1, LLM-augmented and Hybrid regimes achieve higher expert scores than the Baseline for four out of five targets, suggesting that incorporation of the LLM-based reward improves expert alignment.
Q3. As described in Section 3, the LLM judge receives a molecule represented as SMILES and assigns a continuous medicinal chemistry quality score between 0 and 1. The score is generated directly by the pretrained LLM using the selected balanced prompt, without additional training or fine-tuning. To incorporate this evaluation into molecular generation, we integrate the LLM score into the reward function of FREED++. During training, FREED++ constructs molecules by sequentially adding molecular fragments. Each generated molecule is evaluated by the LLM judge, and its numerical score contributes to the total reward alongside the docking score and, in the Hybrid regime, conventional molecular-property penalties. The resulting reward is used by the reinforcement learning algorithm to update the generative policy, increasing the probability of generating molecular structures associated with higher rewards. We appreciate the reviewer's suggestion and would be happy to include a more detailed explanation of this process in the camera-ready version to improve clarity.
Q4. We thank the reviewer for this question. Yes, our study includes ablation experiments for both LLM-based molecular evaluation and its integration into reinforcement learning. For molecular generation, we compare three reward configurations within FREED++: Baseline (docking and conventional molecular descriptors), LLM-augmented (docking and LLM score), and Hybrid (docking, conventional descriptors, and LLM score). This experimental design allows us to isolate the contribution of the LLM-based reward and investigate its interaction with traditional molecular quality metrics. The results are presented in Section 6 and Table 1. For MedChemJudge, we systematically evaluate the effects of three key design factors: LLM choice, prompting strategy, and molecular representation. Specifically, we compare three primary LLMs, three prompting strategies, and four molecular representations, as shown in Figure 2A. We further evaluate selected configurations on the SMILES Invariance and Pairwise Structural Ranking benchmarks (Appendix A3.2–A3.3).
We hope our detailed responses address your concerns. As you recognized the significance of our work to the scientific community, we would be grateful if you could reassess your evaluation and consider increasing the score.
Dear Reviewer,
Thank you for your initial positive assessment of our paper. We were pleased to see that you recognize the significance of our work. As we have not yet received any further feedback from you, we kindly ask again to update the evaluation of our work.
We sincerely thank the reviewer for the constructive feedback! Below we would like to comment on the weaknesses and questions outlined by the reviewer.