acceptodds
Under review as a conference paper at ICLR 2027

Toward Medicinal Chemistry-Aware Molecular Generation through LLM Guidance

Abstract

Molecular optimization and evaluation in AI-driven drug discovery relies primarily on established molecular metrics and heuristics such as QED, Lipinski's Rule-of-Five, structure filters, and synthetic accessibility scores. While effective at assessing specific aspects of molecular quality, these metrics cannot fully capture the holistic reasoning that medicinal chemists apply when prioritizing candidate molecules. In this work, we propose a new evaluation paradigm in which a large language model acts as a medicinal chemistry judge, assigning each molecule an overall quality score while explaining the medicinal chemistry considerations underlying its assessment. To systematically evaluate this approach, we introduce MedChemJudge, the first benchmark specifically designed for LLM-based medicinal chemistry assessment, comprising over 500 expert-annotated molecules across three complementary evaluation settings. Using this benchmark, we identify an effective judge configuration achieving a C-index of 0.84 in reproducing expert quality rankings. The analysis shows that LLM-derived scores can distinguish between structurally similar molecules even when conventional molecular metrics are nearly identical. Finally, incorporating the proposed score into RL–based molecular generative model improves the medicinal chemistry quality of generated molecules across 5 targets while maintaining strong docking performance and high diversity. Together, these results highlight the potential of LLM-based evaluation as a complementary assessment layer for AI-assisted drug design and support molecular optimization that more closely reflects medicinal chemistry decision-making.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
large language models, medicinal chemistry, molecular evaluation, drug-likeness, molecular generation, reinforcement learning, AI-assisted drug discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JpntOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new paradigm for evaluating drug molecules, where a large language model acts as medicinal chemistry judge and can yield both quality scores and explanatory texts. The authors were motivated by the belief that existing quantitative indicator assessments and multifactorial considerations in real drug discovery are insufficient. Its core idea is to use a large number of molecules already evaluated by medicinal chemistry experts, covering a wide range of medical chemistry fields, and dividing them into three independent environments to serve as testing benchmarks to train the optimal large model evaluation configuration in terms of molecular representation, prompt style, and configuration. The main experimental results show that using the evaluation results of the trained large language model as a FREED++ reward function can effectively improve candidate molecular quality and better reflect real-world drug design scenarios.

Critical strengths
  1. Based on literature and actual drug design data, this paper indeed addresses the current gap between quantitative evaluation indicators and actual drug design needs, and uses the understanding and reasoning capabilities of large models to solve them.

  2. The evaluation paradigm of LLM can serve as a flexible intermediate layer between traditional quantitative metrics and human experts. It can serve as a reward function for reinforcement learning-type models to assist quantitative metrics in optimizing molecular generation, and can also serve as guidance and interpretability for explanatory texts in actual wet experiments, forming a complete closed loop throughout the generation process.

  3. From the experimental results, the evaluation paradigm of LLM can, in most cases, indeed lead to improvements in quantitative indicators of the final candidate molecules.

  4. From the perspective of specific method design, the evaluation paradigm of LLM provides multiple representation methods, prompt styles, and large model configurations, making it easier to replicate subsequent experiments and initiate optimization tasks targeting specific targets.

Critical weaknesses
  1. Training LLMs solely through prompt tuning as an evaluation paradigm results are closely tied to the model's own performance without a precise metric to measure the quality of molecular generation.

  2. From the experimental results of the five targets, the evaluation paradigm of LLM does bring optimization to most indicators, but it does not necessarily align with expert evaluations. This seems to contradict the original design aimed at compensating for the deviation between quantitative indicators and actual drug design, rather than simply improving a single quantitative indicator in a particular aspect.

  3. Current experiments do not compare results from other reward functions, human expert rules, and multi-objective optimization methods, which is insufficient to prove the effectiveness of the evaluation paradigm of LLM in final experiments.

  4. The evaluation paradigm of LLM proposed in this paper includes two outcomes, numerical indicators and interpretive analysis, but the consistency judgment between explanatory text and continuous numerical indicator results is insufficient and unclear.

Questions for authors
  1. MedChemJudge, which serves as the benchmark, is evaluated by pharmaceutical chemistry experts without a completely fair evaluation, and its results are discrete quantities, whereas LLM evaluation paradigms are continuous values. Will this bias affect the pre-training configuration of subsequent large models?

  2. On the main experimental table for five targets, how is it determined that the results of molecules generated after adding the evaluation paradigm of LLM better align with the pharmacology experts' judgment, and does this truly meet the expected results?

  3. How are the numerical results of the evaluation paradigm of LLM calculated, and how are they subsequently used to optimize the reward function of reinforcement learning generation models?

  4. Have ablation experiments been conducted on reinforcement learning generative models based on the evaluation paradigm of LLM and other quantitative metrics, or on MedChemJudge's test benchmarks?

3 comments

We sincerely thank the reviewer for the constructive feedback! Below we would like to comment on the weaknesses and questions outlined by the reviewer.

  1. First, we would like to clarify that our approach does not involve training or fine-tuning LLMs. Instead, we systematically compare pretrained models, prompting strategies, and molecular representations to identify the configuration that best aligns with medicinal chemistry expert judgments. Importantly, our evaluation does not rely solely on the LLM's own predictions. We introduce the MedChemJudge benchmark to quantitatively assess agreement with expert annotations using established metrics, including the C-index and Spearman's correlation. Furthermore, the quality of generated molecules is evaluated independently using QED, synthetic accessibility, Brenk structural alerts, docking scores, and scaffold diversity, as well as independent medicinal chemistry expert assessments (Table 1). Thus, while we acknowledge the model dependence of LLM-based evaluation, we respectfully believe that the reviewer's concern does not fully reflect our evaluation methodology.
  2. We would like to emphasize that, for four out of five targets, incorporating the LLM score into the reward function leads to expert evaluations that are improved compared to the Baseline. Therefore, we believe that these results support rather than contradict our original motivation of bringing molecular optimization closer to medicinal chemistry expert preferences. Regarding 3FQS, we acknowledge that the Baseline regime achieves higher expert scores than both LLM-guided regimes. This suggests that the effectiveness of LLM-based guidance may be target-dependent and does not guarantee improvements in every case. We explicitly acknowledge this limitation in Section 6.2 of the manuscript.
  3. We would like to clarify that comparing different multi-objective optimization methods is beyond the scope of our study. Our primary goal was to investigate whether LLM-based medicinal chemistry evaluation can serve as an effective reward signal for molecular generation. To this end, we selected a single generative framework, FREED++, and compared three reward configurations within it: Baseline (docking and conventional molecular descriptors), LLM-augmented (docking and LLM score), and Hybrid (docking, conventional descriptors, and LLM score). This allows us to isolate the contribution of the LLM-based reward while keeping the underlying generative architecture consistent. Importantly, the Baseline already incorporates established physicochemical constraints and expert-derived structural filters. Therefore, we believe that our experimental setup is sufficient for the specific research question addressed in this work.
  4. The consistency between the LLM-generated textual explanations and numerical scores is explicitly evaluated in Appendix A10 (Expert Evaluation of LLM-Generated Comments). Furthermore, Table 4 provides representative examples of both successful evaluations and failure cases, including factual errors and discrepancies between textual explanations and numerical scores. Therefore, we respectfully believe that this concern is already addressed by the expert-based analysis.

Q1. We thank the reviewer for raising this concern. First, as we clarified earlier, MedChemJudge is not used to pretrain or fine-tune the evaluated LLMs, but only to compare and select configurations of pretrained models. Therefore, potential biases in expert annotations do not affect the models' pretrained parameters. Regarding the expert annotations, MedChemJudge was constructed using a structured evaluation protocol with predefined medicinal chemistry criteria, as detailed in Appendix A1.1. Importantly, the difference between discrete expert labels and continuous LLM scores is intentional. We evaluate agreement primarily through ranking-based metrics, which assess whether continuous LLM scores reproduce the ordering of expert-defined quality categories without requiring identical scoring scales. We would appreciate it if the reviewer could clarify what specific aspects of the benchmark evaluation are considered insufficiently fair, as this would help us better understand and address the concern.

Q2. The alignment with medicinal chemistry expert preferences is determined through an independent expert evaluation, as described in Appendix A9. For each target and reward regime, we generate 1,000 molecules and select the top five candidates based on a combination of QED, SA Score, Brenk alerts, and docking score. These molecules are then independently assessed by a medicinal chemistry expert, without relying on the LLM-generated scores. As reported in Table 1, LLM-augmented and Hybrid regimes achieve higher expert scores than the Baseline for four out of five targets, suggesting that incorporation of the LLM-based reward improves expert alignment.

Q3. As described in Section 3, the LLM judge receives a molecule represented as SMILES and assigns a continuous medicinal chemistry quality score between 0 and 1. The score is generated directly by the pretrained LLM using the selected balanced prompt, without additional training or fine-tuning. To incorporate this evaluation into molecular generation, we integrate the LLM score into the reward function of FREED++. During training, FREED++ constructs molecules by sequentially adding molecular fragments. Each generated molecule is evaluated by the LLM judge, and its numerical score contributes to the total reward alongside the docking score and, in the Hybrid regime, conventional molecular-property penalties. The resulting reward is used by the reinforcement learning algorithm to update the generative policy, increasing the probability of generating molecular structures associated with higher rewards. We appreciate the reviewer's suggestion and would be happy to include a more detailed explanation of this process in the camera-ready version to improve clarity.

Q4. We thank the reviewer for this question. Yes, our study includes ablation experiments for both LLM-based molecular evaluation and its integration into reinforcement learning. For molecular generation, we compare three reward configurations within FREED++: Baseline (docking and conventional molecular descriptors), LLM-augmented (docking and LLM score), and Hybrid (docking, conventional descriptors, and LLM score). This experimental design allows us to isolate the contribution of the LLM-based reward and investigate its interaction with traditional molecular quality metrics. The results are presented in Section 6 and Table 1. For MedChemJudge, we systematically evaluate the effects of three key design factors: LLM choice, prompting strategy, and molecular representation. Specifically, we compare three primary LLMs, three prompting strategies, and four molecular representations, as shown in Figure 2A. We further evaluate selected configurations on the SMILES Invariance and Pairwise Structural Ranking benchmarks (Appendix A3.2–A3.3).

We hope our detailed responses address your concerns. As you recognized the significance of our work to the scientific community, we would be grateful if you could reassess your evaluation and consider increasing the score.

Dear Reviewer,

Thank you for your initial positive assessment of our paper. We were pleased to see that you recognize the significance of our work. As we have not yet received any further feedback from you, we kindly ask again to update the evaluation of our work.

Reviewer_ZzWqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes using a large language model as a "medicinal chemistry judge" that takes a molecule as input and returns both a scalar holistic quality score in [0,1] and a free-text rationale, positioned as a complement to conventional descriptors such as QED, SAscore, and structural alert filters. Three contributions follow from this framing. First, the authors introduce MedChemJudge, a benchmark of 520 expert-annotated molecules drawn from ChEMBL, PubChem, and the medicinal chemistry literature, organized into three settings. Second, they run a grid search over three prompt formulations, four input representations (SMILES, IUPAC, their combination, and SMILES plus a nine-descriptor vector), and three compact models, with supplementary comparisons against a chemistry-specialized model and larger frontier models. Third, they inject the resulting score as an additional reward term in FREED++ across five protein targets under three reward methods (baseline descriptors, LLM-only, and hybrid), generating 1000 molecules per configuration and evaluating QED, SAscore, Brenk alerts, docking score, scaffold diversity, and a human expert score on the top-5 molecules.

My overall reading is that the engineering is competent and the narrative is clean, but the three central claims are each weaker than stated: the method is prompt engineering over off-the-shelf models rather than a methodological contribution, the benchmark is small and appears to rest on a single annotator with severe pretraining-contamination exposure, and Table 1 does not actually support the abstract's claim of improvement across five targets.

Critical strengths

The most consequential strength is that the paper targets a real and widely acknowledged gap with an artifact the community can reuse.

A second, lesser strength is the honesty of the negative results that do appear. The authors report that appending precomputed descriptors to the prompt degrades performance, that the chemistry-specialized ChemDFM does not beat general-purpose models, that scaling to larger models yields only modest gains, and that the judge fails outright on the thalidomide and desipramine/imipramine pairs.

Critical weaknesses
  1. The first critical weakness is that the novelty claims rest on an incomplete treatment of prior work, and the single most directly competing method is neither cited nor compared against. MolSkill (Choung et al., Nature Communications 2023) applies learning-to-rank to pairwise preference feedback collected from 35 medicinal chemists in order to produce exactly the kind of holistic "chemist intuition" score this paper proposes, and it already demonstrates compound prioritization, motif-level attribution, and biasing of de novo generation. Those are, almost line for line, contributions 1, 3, and 4 of this submission, and MolSkill's annotation base is far larger than one expert over 520 molecules; the cheminformatics community has discussed it specifically as a QED replacement for several years. Moreover, the claim that "LLM-as-a-judge" has not been evaluated systematically in Chem is not accurate. Existing works like MOLLEO (Wang et al. ICLR 2025) and Varvara Voinarovska et al. Digital Discovery 2026 have evaluated LLM-as-a-judge in synthesis plan and molecule discovery tasks.

  2. The second critical weakness is that the generation experiments do not statistically support their stated conclusion. Each reward was trained exactly once per target, and FREED++'s own authors documented substantial implementation and evaluation fragility in the FREED line of work, so seed variance in Table 1 is plausibly on the order of the reported effects. Within the single run that exists, the results are internally contradictory rather than favorable: the LLM-augmented method on 5VFI collapses to QED 0.19, SA 4.19, and Brenk 0.95; the hybrid method raises Brenk alerts from 0.07 to 0.31 on 1KZN and from 0.16 to 0.61 on 3FQS; the 3FQS expert score falls from 1.60 to 0.20 under both LLM methods; and docking degrades relative to baseline on most targets. No method dominates on more than a subset of targets and metrics, which is not what "improves the medicinal chemistry quality of generated molecules across 5 targets" describes.

Questions for authors

(1) On the 1000 generated molecules per configuration, what is the correlation between the LLM score and QED, SA, logP, and MW? Can the Table 1 improvements in "properties not explicitly optimized" be distinguished from the LLM score acting as a proxy for QED?

(2) How many annotators produced the benchmark labels, and what is the inter-annotator agreement (κ or Krippendorff's α)? Is the expert who scored the generated molecules in Section 6.2 the same person who built the benchmark, and was that scoring blinded and order-randomized?

(3) For the well-known drug pairs, how do you separate chemical reasoning from memorization? Is there a contamination-free held-out subset, for example novel analogues or FREED++-generated molecules with freshly collected expert preferences?

(4) What happens if the LLM reward term is replaced by QED or MolSkill at equal weight, holding everything else fixed?

4 comments

We thank the reviewer for the careful assessment and constructive suggestions. Below, we provide our responses to the outlined critical weaknesses and questions:

  1. We thank the reviewer for highlighting these works. MOLLEO addresses a different research question: LLMs are used as molecular generation operators rather than molecular quality evaluators. Therefore, we did not consider it directly relevant to our LLM-as-a-Judge setting. We agree that MolSkill is highly relevant and should have been cited. We will add it to the Related Work section and clarify the differences between preference learning and our approach based on pretrained LLMs without task-specific training. Voinarovska et al. (2026) focuses on evaluating retrosynthetic routes rather than molecular quality. Nevertheless, we recognize its relevance as a study of LLMs as chemistry experts and will include the corresponding citation. Finally, we would like to clarify that we did not claim that LLM-as-a-Judge has not been systematically evaluated in chemistry in general. Our statement specifically concerns “medicinal chemistry assessment of candidate molecules.” To the best of our knowledge, this particular task has not previously been systematically investigated using LLMs as judges.
  2. We thank the reviewer for this valuable suggestion to conduct multiple training runs with different random seeds. We are currently conducting these additional experiments and will provide the results before the end of the rebuttal period. As these experiments require additional time, we would appreciate the opportunity to address the reviewer's other concerns in the meantime. Regarding the current results, we agree that the claim “improves the medicinal chemistry quality of generated molecules across 5 targets” is too strong. As explicitly discussed in Section 6.2, the effects of LLM-based rewards are target-dependent and not uniformly positive across all metrics. We will therefore soften the corresponding claims in the Abstract and Contributions to better reflect the observed results and their limitations.

Q1. We thank the reviewer for this valuable suggestion. We already compare LLM scores with conventional molecular metrics in Figure 2B, demonstrating that LLM assessments often differ from traditional descriptors and can better reflect expert preferences in certain cases. Nevertheless, we appreciate the suggestion to quantify these relationships explicitly and have calculated the correlations between LLM scores and conventional molecular properties.

DescriptorCorrelation with LLM score
QED+0.272
SA−0.291
LogP+0.106
MW−0.201

The correlations are relatively weak, suggesting that the LLM score does not simply reproduce conventional molecular descriptors. In particular, the low correlation with QED suggests that the LLM score captures additional information rather than acting solely as a proxy for QED. We will include these results in the revised manuscript.

Q2. As described in Appendix A1.1, the MedChemJudge benchmark was annotated by a single medicinal chemistry expert following a structured annotation protocol. Therefore, inter-annotator agreement metrics cannot be calculated. The expert who evaluated the generated molecules in Section 6.2 was different from the expert who constructed the benchmark, providing an independent human assessment. As described in Appendix A9, the evaluation was blinded: the expert had access only to molecular structures, without information about the protein targets, generation regimes, LLM scores, or other computational metrics.

Q3. We thank the reviewer for raising this important point. We agree that, for well-known drug pairs, it is difficult to fully disentangle chemical reasoning from memorized knowledge, and we explicitly acknowledge this limitation in Section 7.2. Our choice of well-characterized molecules was motivated by the need for reliable reference preferences supported by known pharmacological and safety data. For entirely novel molecules, such experimentally established information is generally unavailable, making it difficult to objectively validate LLM judgments. To complement this evaluation, Section 6.2 includes FREED++-generated molecules assessed by an independent medicinal chemistry expert. Importantly, for these generated molecules, the expert evaluates their potential medicinal chemistry quality based on structural characteristics and anticipated liabilities. Thus, the two evaluation settings serve complementary purposes: known compounds allow us to assess agreement with established medicinal chemistry knowledge, while generated molecules provide an additional assessment of the approach's applicability to novel molecular structures.

Q4, point 2 in Weaknesses. We thank the reviewer for these valuable suggestions. To address these questions, we (1) repeated the FREED++ training and molecular generation experiments 3 times with different random seeds for each target and reward configuration and (2) conducted an additional ablation in which the reward consisted of MolSkill, docking score and standard FREED++ components (3 runs, similarly).

TargetRegimeQED ↑SA ↓Brenk ↓Docking ↓Diversity ↑
1KZNBaseline0.42 ± 0.042.81 ± 0.030.07 ± 0.02−10.36 ± 0.120.43 ± 0.06
MolSkill0.58 ± 0.044.59 ± 0.580.13 ± 0.05-7.43 ± 0.610.74 ± 0.10
LLM-augmented0.30 ± 0.042.64 ± 0.190.25 ± 0.31−10.16 ± 0.120.63 ± 0.09
Hybrid0.53 ± 0.082.94 ± 0.110.31 ± 0.26−9.71 ± 0.330.36 ± 0.06
3FQSBaseline0.59 ± 0.032.87 ± 0.120.10 ± 0.05−7.71 ± 0.140.26 ± 0.09
MolSkill0.65 ± 0.044.15 ± 0.290.25 ± 0.04-5.76 ± 0.160.69 ± 0.15
LLM-augmented0.66 ± 0.042.18 ± 0.220.22 ± 0.15−7.36 ± 0.130.18 ± 0.11
Hybrid0.66 ± 0.032.32 ± 0.170.36 ± 0.30−7.33 ± 0.070.21 ± 0.09
4J1RBaseline0.62 ± 0.093.71 ± 1.000.61 ± 0.23−7.83 ± 0.080.25 ± 0.15
MolSkill0.61 ± 0.034.09 ± 0.960.16 ± 0.03-5.66 ± 0.520.79 ± 0.07
LLM-augmented0.52 ± 0.113.07 ± 0.270.05 ± 0.03−8.13 ± 0.250.31 ± 0.11
Hybrid0.67 ± 0.142.71 ± 0.410.13 ± 0.09−7.89 ± 0.140.26 ± 0.10
5VFIBaseline0.40 ± 0.022.99 ± 0.140.06 ± 0.05−11.59 ± 0.150.58 ± 0.07
MolSkill0.56 ± 0.073.82 ± 1.090.11 ± 0.10-9.09 ± 0.590.75 ± 0.07
LLM-augmented0.19 ± 0.003.97 ± 0.130.96 ± 0.02−13.36 ± 0.090.83 ± 0.08
Hybrid0.38 ± 0.042.95 ± 0.360.14 ± 0.14−11.31 ± 0.070.60 ± 0.12
8AFBBaseline0.56 ± 0.063.06 ± 0.080.65 ± 0.27−9.22 ± 0.150.22 ± 0.03
MolSkill0.59 ± 0.044.26 ± 0.490.17 ± 0.07-5.55 ± 0.490.65 ± 0.12
LLM-augmented0.64 ± 0.083.05 ± 0.190.53 ± 0.25−9.07 ± 0.490.25 ± 0.02
Hybrid0.69 ± 0.032.97 ± 0.430.21 ± 0.04−9.08 ± 0.230.26 ± 0.10

The updated results confirm that the overall trends observed in our original experiments remain consistent, supporting the reproducibility of our findings. Importantly, the LLM-guided configurations generally outperform MolSkill across key molecular quality metrics. While MolSkill achieves high scaffold diversity and competitive QED values, it consistently produces substantially worse docking scores and higher SA scores across all five protein targets. In contrast, the LLM-augmented and Hybrid configurations maintain docking performance closer to the conventional Baseline while achieving more favorable synthetic accessibility and Brenk alert profiles in several cases. These results demonstrate that LLM-based rewards provide a more balanced trade-off between molecular quality and predicted binding affinity than MolSkill. Although the improvements remain target-dependent, the additional experiments strengthen our conclusion that LLM-based medicinal chemistry evaluation provides a valuable and competitive optimization signal for molecular generation.

We hope our clarifications and additional results will resolve your concerns and demonstrate the value of our contribution. We would be grateful if you could reconsider your initial rating in light of our responses. We will also provide the results of the additional experiments addressing point 2 before the end of the rebuttal period. We remain available to address any further questions.

Dear Reviewer,

We appreciate your patience while we conducted additional experiments. We have updated our previous response to include results from multiple independent runs, which further support the reproducibility of our findings.

Thank you for your positive initial assessment of our paper. We were happy to hear that you consider our work honest and highly relevant for the drug design scientific community. As we have not yet received further feedback, we would kindly ask whether our responses and additional results have adequately addressed your concerns. If so, we would be grateful if you could increase your initial rating in light of this new evidence.

Reviewer_9uzLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores the role of LLMs as drug chemistry evaluators in the assessment and guidance of molecular generation. The authors proposed MedChemJudge, which covers three evaluation scenarios: expert alignment scoring, SMILES invariance, and paired structure ranking. It was found that using SMILES input and balanced prompts in the Qwen3-14B configuration is an effective evaluation scheme, outperforming the molecular quality categories defined by experts. The proposed LLM scoring method was further integrated into FREED++, which is a reinforcement learning framework for molecular generation, and was evaluated on five protein targets. The results show that the evaluation based on LLMs can capture drug chemistry quality features beyond traditional molecular descriptors and is expected to improve the molecular generation effect when combined with existing optimization goals.

Critical strengths

The paper addresses a practically relevant problem in molecular generation. Conventional molecular descriptors and structural filters cannot fully represent the multifactorial considerations involved in medicinal chemistry assessment. Exploring LLM-based evaluation as a complementary optimization signal is a meaningful research direction.

The study connects molecular evaluation with downstream generation. Rather than evaluating LLM judgments in isolation, the authors incorporate the resulting scores into FREED++ and compare conventional, LLM-augmented, and hybrid reward formulations. The experiments provide an initial investigation of whether LLM-based assessments can be used in molecular optimization.

Critical weaknesses

The expert's scoring benchmark relies solely on the annotation of a single medicinal chemistry expert, lacking consensus among multiple annotators or independent verification, and thus lacks credibility.

The annotation process incorporates known toxicity information and information that can be searched online, making it difficult to distinguish between true medicinal chemistry reasoning based on structure and the identification and search of known compounds. This is a potential hacking issue. Since this benchmark mainly consists of molecules from public databases, potential pre-training contamination further weakens the evidence for its generalization to new molecules.

The method has limited innovation and insufficient baseline comparison.

The results of various experiments do not consistently support the hypothesis and do not demonstrate the claimed improvements. Table 1 shows that the target dependency differences are significant, and in multiple cases, the molecules generated by the LLM-guided strategy have worse effects.

The experimental data lacks statistical robustness. Each FREED++ configuration is trained only once for each target, and the reported standard deviations are based on the calculation results of the generated molecules rather than independent training runs.

Furthermore, human evaluation only covers five selected molecules under each target and reward mechanism, totaling only 75 generated molecules evaluated by experts. This sample is too limited to support broad conclusions regarding the improvement of medicinal chemistry quality.

Questions for authors

How much of the judge's performance is attributable to memorized chemical knowledge? Given that many benchmark molecules are established drugs or known toxic compounds, could the authors evaluate the judge on previously unseen or recently synthesized molecules to better distinguish generalizable structural reasoning from knowledge recall?

Could the authors clarify the benchmark and evaluation statistics? How many unique molecular structures are included in MedChemJudge?

3 comments

We would like to thank the reviewer for taking the time to revisit and revise their initial review. Below, we address each of the reviewer's comments and questions in detail.

  1. We thank the reviewer for this important comment. We acknowledge that annotations from multiple independent experts would strengthen the reliability of the benchmark. However, the expert annotations were obtained using a structured protocol with predefined medicinal chemistry and safety criteria (Appendix A1.1), reducing the subjectivity of individual judgments. Furthermore, MedChemJudge includes complementary evaluation settings, such as SMILES Invariance and Pairwise Structural Ranking, while the practical utility of LLM-based scoring is additionally assessed through molecular generation experiments using established computational metrics.
  2. We thank the reviewer for raising this important concern. We agree that the use of well-known compounds makes it difficult to fully distinguish medicinal chemistry reasoning from memorized knowledge, and we explicitly acknowledge this limitation in Section 7.2. However, these molecules were intentionally selected because their known pharmacological and safety profiles provide reliable reference information for evaluating LLM judgments. For entirely novel molecules, such experimentally validated information is generally unavailable, making objective evaluation more challenging. To complement this benchmark, Section 6.2 presents experiments on newly generated molecules evaluated by a medicinal chemistry expert based on their structural features and potential liabilities. Thus, these two evaluation settings serve complementary purposes: known compounds enable comparison with established medicinal chemistry knowledge, while generated molecules provide additional evidence of the approach's applicability to novel molecular structures.
  3. We respectfully disagree that the novelty and baseline comparisons are insufficient. The key contribution of our work is the introduction of a new paradigm for holistic medicinal chemistry evaluation, where LLMs serve as judges capable of integrating multiple chemical considerations into a single interpretable score. To the best of our knowledge, we introduce the first benchmark specifically designed for this task, MedChemJudge, comprising three complementary evaluation settings. Our study provides extensive comparisons across multiple LLM families, model sizes, molecular representations, and prompting strategies, as well as nine established molecular descriptors. Moreover, we go beyond benchmarking by integrating LLM-based scoring into reinforcement learning for molecular generation and systematically comparing conventional descriptor-based, LLM-based, and hybrid reward functions across five protein targets. These contributions establish both the methodological novelty and practical relevance of our approach, demonstrating how LLM-based evaluation can complement conventional molecular optimization methods.
  4. We would like to emphasize that our results are not intended to demonstrate uniform improvements across all targets and molecular properties, but rather the potential of LLM-based scoring as a complementary optimization signal. As shown in Table 1, the Hybrid approach improves expert-assessed molecular quality for three of the five targets and achieves comparable performance for another, while generally maintaining competitive docking scores and scaffold diversity. Importantly, improvements are also observed in molecular quality metrics that were not explicitly optimized, suggesting that the LLM captures useful information beyond conventional descriptors. We acknowledge that the benefits are target-dependent, as explicitly discussed in Section 6.2, but this does not undermine our central hypothesis.
  5. We thank the reviewer for this valuable comment and agree that evaluating multiple independent training runs would strengthen the statistical robustness of our results. To address this concern, we are conducting additional FREED++ experiments with multiple random seeds to assess the reproducibility and variability of the observed improvements. We will provide the results before the end of the rebuttal period. As these experiments require additional time, we would appreciate the opportunity to address the reviewer's other concerns in the meantime.
  1. We acknowledge that evaluating a larger number of generated molecules would provide stronger statistical support for the expert assessment. We would like to clarify that evaluating all 1,000 generated molecules for each target and reward configuration (15,000 molecules in total) would require an enormous amount of expert time and effort, making such an evaluation practically infeasible. We therefore provide computational assessment of all generated molecules using established molecular quality metrics and assess the expert quality of the most promising candidates from each generation setting, reflecting the practical drug discovery workflow where medicinal chemists prioritize a limited number of compounds for further investigation.
  2. MedChemJudge contains 520 molecular entries across three evaluation settings: 300 molecules in Expert-Aligned Scoring, 60 pairs (120 SMILES representations) in SMILES Invariance, and 50 pairs (100 molecules) in Pairwise Structural Ranking. Importantly, the SMILES Invariance subset contains two different representations of the same molecular structure in each pair. Therefore, the benchmark comprises up to 460 unique molecular structures.

We hope that our clarifications address your concerns and further highlight the significance of our contributions. We would sincerely appreciate reconsideration of the initial rating in light of our responses. The results of the additional experiments addressing point 5 will be provided before the rebuttal deadline. We remain open to provide any further clarification or address additional questions.

The experimental data lacks statistical robustness. Each FREED++ configuration is trained only once for each target, and the reported standard deviations are based on the calculation results of the generated molecules rather than independent training runs.

We appreciate reviewer's patience while we conducted additional experiments. To address this issue, we repeated the FREED++ training and molecular generation experiments 3 times with different random seeds for each target and reward configuration. The updated results are presented in the table above as mean ± standard deviation across 3 independent runs, rather than variability across molecules from a single run.

TargetRegimeQED ↑SA ↓Brenk ↓Docking ↓Diversity ↑
1KZNBaseline0.42 ± 0.042.81 ± 0.030.07 ± 0.02−10.36 ± 0.120.43 ± 0.06
LLM-augmented0.30 ± 0.042.64 ± 0.190.25 ± 0.31−10.16 ± 0.120.63 ± 0.09
Hybrid0.53 ± 0.082.94 ± 0.110.31 ± 0.26−9.71 ± 0.330.36 ± 0.06
3FQSBaseline0.59 ± 0.032.87 ± 0.120.10 ± 0.05−7.71 ± 0.140.26 ± 0.09
LLM-augmented0.66 ± 0.042.18 ± 0.220.22 ± 0.15−7.36 ± 0.130.18 ± 0.11
Hybrid0.66 ± 0.032.32 ± 0.170.36 ± 0.30−7.33 ± 0.070.21 ± 0.09
4J1RBaseline0.62 ± 0.093.71 ± 1.000.61 ± 0.23−7.83 ± 0.080.25 ± 0.15
LLM-augmented0.52 ± 0.113.07 ± 0.270.05 ± 0.03−8.13 ± 0.250.31 ± 0.11
Hybrid0.67 ± 0.142.71 ± 0.410.13 ± 0.09−7.89 ± 0.140.26 ± 0.10
5VFIBaseline0.40 ± 0.022.99 ± 0.140.06 ± 0.05−11.59 ± 0.150.58 ± 0.07
LLM-augmented0.19 ± 0.003.97 ± 0.130.96 ± 0.02−13.36 ± 0.090.83 ± 0.08
Hybrid0.38 ± 0.042.95 ± 0.360.14 ± 0.14−11.31 ± 0.070.60 ± 0.12
8AFBBaseline0.56 ± 0.063.06 ± 0.080.65 ± 0.27−9.22 ± 0.150.22 ± 0.03
LLM-augmented0.64 ± 0.083.05 ± 0.190.53 ± 0.25−9.07 ± 0.490.25 ± 0.02
Hybrid0.69 ± 0.032.97 ± 0.430.21 ± 0.04−9.08 ± 0.230.26 ± 0.10

The new results are consistent with our original findings and provide stronger evidence for their reproducibility. In particular, the LLM-augmented or Hybrid configurations achieve better mean values than the Baseline in most of target-metric comparisons. However, these improvements are not uniform across all targets and metrics. The Baseline achieves better docking scores for 1KZN, 3FQS, and 8AFB; however, the differences compared to the best LLM-guided configurations are relatively small, and the LLM-guided molecules still exhibit favorable docking scores. Similarly, the Baseline achieves lower Brenk alert values for 1KZN, 3FQS, and 5VFI. These findings reinforce our previous conclusions that the benefits of LLM-based rewards are target-dependent and that LLM scoring is most appropriately considered a complementary optimization signal rather than a universally superior alternative to conventional descriptors.

We believe these additional experiments directly address the reviewer's concern. We will incorporate the updated results into the revised manuscript.

As we have not yet received further feedback, we would appreciate it if you could let us know whether our responses and additional experimental results have addressed your concerns. If so, we would be grateful if you could increase your initial rating in light of these clarifications.

Comments on the paper

1 comment

Dear Reviewers and Area Chairs,

We would like to thank you for your time and careful consideration of our submission.

Although we have not received further feedback from the reviewers regarding our rebuttal responses and additional experimental results, we have made every effort to thoroughly address all the concerns raised. We conducted additional experiments, strengthened our baseline comparisons, clarified the limitations of our approach, and updated the manuscript accordingly. All changes and additions are highlighted in blue in the revised version for ease of reference.

We would also like to emphasize that the reviewers recognized the high significance of our research, along with several other important strengths, including the transparency of our findings, the experimental design, and the practical relevance of the proposed approach. At the same time, the reviewers indicated relatively low confidence in their assessments, which we believe is worth considering alongside the additional clarifications provided during the rebuttal.

We believe that the additional experiments and revisions have addressed the major concerns raised during the review process. We would therefore be very grateful if these improvements could be taken into account in the final evaluation and if our submission could be given favorable consideration for acceptance at ICLR 2027.

Thank you again for your time and consideration. We sincerely hope to have the opportunity to present our work at ICLR 2027.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.