- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces PharmLens, which uses multiple agents to obtain literature-based evidence and applies it to the prediction of molecular properties. PharmLens collects ligand knowledge using physicochemical agents, and pathway knowledge related to targets using protein-relevant and interaction agents; if lacking, it additionally finds reference ligands that bind to the relevant targets from the document, verifies their specific binding, and then compares them with the query molecule. The evidence obtained in the above process (LCE, PCE, IE) is combined to predict the molecular characteristics. Experiments were conducted using TDC and UniTox dataset, and additionally, a property-cliff distinction experiment was performed, showing overall superior performance compared to several existing llms.
Critical strengths
For predicting molecular properties, It is a good approach to use multiples agents to collect various pieces of evidence and determine the property by considering a robust methodlogy, especially using pathway information.
In both evaluation datasets(TDC & UniTox), this methodology contributed to the highest performance and score improvements among multiple LLMs. In particular, it demonstrated higher discriminative ability than other models when evaluating the cliff subset which is hard to distinguish. Moreover, after conducting Rationale evaluations with an LLM judge, human evaluators and fine-tuning ModernBERT, it demonstrated acceptable quality of reasoning.
Critical weaknesses
- Several choices in the docking pipeline are not sufficiently justified. First, the criteria for selecting a PDB structure when multiple structures are available for a protein are unclear. Second, only the top-ranked docking pose is used for PLIP analysis, without examining the effects of alternative poses. Finally, restricting docking to a single binding site may overlook relevant interactions at alternative or allosteric sites. These choices raise concerns about the reliability of the resulting interaction evidence.
- The experimental evaluation has two limitations. First, comparisons with more recent frontier LLMs, such as GPT-5.x and newer Claude models, are missing, making it difficult to assess the framework's effectiveness against stronger contemporary baselines. Second, the UniTox evaluation aggregates eight toxicity endpoints into four physiological-system groups and reports group-level averages rather than individual endpoint results. This aggregation may obscure substantial performance variations across endpoints and potentially overstate the consistency of the reported improvements.
- The ablation results show that PCE and IE provide relatively small improvements, with all Macro-F1 gains below 0.1. This raises questions about the effectiveness of these components given the added complexity. In addition, the paper does not include an ablation study for LCE, making it difficult to evaluate its contribution.
- Although the paper evaluates rationale quality using LLM judges and human experts, it lacks detailed case studies showing how intermediate evidence leads to final predictions. In particular, the absence of failure case analysis makes it difficult to assess the reliability of the proposed mechanistic reasoning process.
-
The manuscript contains several inconsistencies and typos:
- Model naming: The framework is inconsistently referred to as PharmLens, PharmaLens, and TheKnow.
- Table reference: On Line 304, "(backbone ablation table)" should be replaced with "Table 3".
- Model categorization: TxGemma is categorized under "Domain LLMs" in Tables 1 and 2, but under "Multi-Agents" in Table 4.
- Typographical error: Table 5 contains "Qwnen" instead of "Qwen".
These issues should be corrected to improve the manuscript's clarity and consistency.
Questions for authors
Q1. Experimental Evaluation
Could the authors provide additional results using more recent frontier LLMs, such as GPT-5.x and newer Claude Sonnet models? Additionally, could the authors report the Macro-F1 scores for each of the eight UniTox endpoints separately, rather than only presenting the aggregated results across four physiological-system groups?
Q2. Ablation Study
Could the authors provide an ablation study for LCE to clarify its contribution to the overall performance? Additionally, given the relatively modest absolute improvements from PCE and IE, how do the authors justify the additional complexity of these components?
Q3. Docking Settings
When multiple PDB structures are available for a protein, how is the final structure selected? Are factors such as species, resolution, and protein conformations considered? Additionally, why is only the top-ranked docking pose used for PLIP analysis, and how does the framework handle proteins with multiple binding sites, including allosteric sites?
Q4. Qualitative Analysis
Could the authors provide representative case studies showing the complete reasoning process, including intermediate evidence, protein–ligand relationship predictions, and final property predictions? In particular, failure cases would help assess the reliability of the proposed mechanistic reasoning.
6 comments
Response to Reviewer 8rLz (2/4)
We thank the reviewer for highlighting the importance of comparisons with more recent frontier LLMs. Accordingly, we evaluated GPT-5.1 and Claude-Sonnet-5 and report their results on the four TDC endpoints and the UniTox cliff set in the Table.2 below.
Table.2 Performance comparison with recent frontier LLMs.
| Model | BBBP | DILI | HIA | Bioav | Avg. | Circ. | Metab. | Sens. | Repro. | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Llama3.1 | 0.421 | 0.427 | 0.534 | 0.537 | 0.480 | 0.373 | 0.447 | 0.404 | 0.364 | 0.397 |
| GPT-4o | 0.613 | 0.553 | 0.637 | 0.626 | 0.607 | 0.458 | 0.503 | 0.513 | 0.410 | 0.471 |
| GPT-5.1 | 0.594 | 0.596 | 0.801 | 0.663 | 0.663 | 0.424 | 0.478 | 0.574 | 0.469 | 0.486 |
| DeepSeek-V4-Pro | 0.668 | 0.659 | 0.718 | 0.659 | 0.676 | 0.428 | 0.470 | 0.479 | 0.488 | 0.466 |
| Qwen3.7-max | 0.701 | 0.615 | 0.805 | 0.616 | 0.684 | 0.426 | 0.477 | 0.516 | 0.515 | 0.483 |
| Claude-Sonnet-5 | 0.750 | 0.650 | 0.872 | 0.659 | 0.719 | 0.459 | 0.537 | 0.575 | 0.524 | 0.524 |
| PharmLens (ours) | 0.726 | 0.669 | 0.875 | 0.694 | 0.741 | 0.516 | 0.554 | 0.583 | 0.531 | 0.546 |
Even against these stronger baselines, PharmLens achieves the highest average Macro-F1 across both evaluations. On TDC, PharmLens yields relative improvements of 11.8% over GPT-5.1 and 3.1% over Claude-Sonnet-5. Similarly, on the UniTox cliff set, it achieves relative improvements of 12.3% and 4.0%, respectively, while outperforming both additional baselines across all four toxicity groups.
These results further demonstrate that the performance advantage of PharmLens is not merely attributable to the choice of comparatively weaker baselines. In particular, this advantage persists on property-cliff molecules, which exhibit high structural similarity despite having opposing property labels. This further supports the utility of acquiring and integrating molecule-specific evidence guided by property-specific mechanistic knowledge, even when compared against stronger frontier LLMs.
We understand the reviewer's concern regarding whether the performance gains justify the additional model complexity. However, the practical significance of each component's contribution should be assessed not only in terms of absolute performance gains, but also in the context of task difficulty and performance gaps among competing models.
On the cliff evaluation, the performance drops resulting from removing and are 5.2 and 2.7 times larger, respectively, than the performance range among the top three baselines (0.017). Moreover, their relative performance improvements increase substantially from 6.1% and 4.4% on the full UniTox dataset to 19.2% and 9.2% on the property-cliff set (Relative improvements were calculated with respect to the corresponding baseline performance). Notably, removing reduces performance below that of the strongest baseline, highlighting its critical role in establishing PharmLens's performance advantage. further complements 's functional-relation predictions with structural interaction evidence, providing additional performance gains.
The additional ablation revealed distinct contributions of the evidence layers. (All ablation results are provided in the Appendix C.1.) On the full UniTox evaluation, removing causes the largest performance drop, highlighting its primary contribution to property prediction through intrinsic molecular features. In contrast, removing leads to the largest drop on the cliff set, as shown in the Appendix C.1. These findings suggest that provides a foundation for capturing intrinsic properties, whereas and contribute to distinguishing molecules with different property outcomes despite structural similarity by assessing their functional relationships and interactions with biological targets. We thank the reviewer for suggesting this additional analysis, which revealed these complementary roles.
Response to Reviewer 8rLz (3/4)
In this study, we prioritized binding sites that are supported by literature and experimental structures as being associated with the target functional relation and offer sufficient reference ligands to construct representative interaction profiles. Accordingly, we did not restrict our evaluation to orthosteric sites or categorically exclude allosteric sites; allosteric sites with established evidence of mediating the same functional relation were also included.
The primary objective of this selection strategy is to construct generalized interaction profiles by integrating interaction patterns from diverse reference ligands sharing the same functional relation. Conversely, indiscriminately including alternative sites with only a limited number of reference ligands may cause the interpretation to rely disproportionately on contact patterns specific to a small number of references.
Nevertheless, we acknowledge the reviewer's concern that this approach may not capture all interactions occurring at other functionally relevant binding sites. To investigate this, we performed additional docking analyses on alternative sites supported by evidence of their association with the same functional relation, constructing separate reference interaction profiles for each site (Appendix A.4.1).
Table.3 Alternative binding-site evaluation results (Macro-F1).
| BBBP | HIA | Bioav | Cardio | Dermato | Hemato | Infertility | Ototo | Pulmonary | Renal | Mean | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| # Replaced proteins | 1 | 2 | 3 | 2 | 3 | 1 | 3 | 1 | 3 | 1 | – |
| Baseline macro-F1 | 0.7302 | 0.8754 | 0.6939 | 0.5054 | 0.5401 | 0.5058 | 0.6700 | 0.6986 | 0.5087 | 0.5104 | 0.6238 |
| Δ macro-F1 | +0.0075 | −0.0098 | +0.0165 | −0.0371 | −0.0044 | +0.0233 | −0.0732 | +0.0402 | +0.0251 | −0.0220 | −0.0034 |
The results (Table.3) showed no consistent performance improvements across endpoints, while the overall mean Macro-F1 slightly decreased. This does not imply that alternative sites lack biological significance; rather, it indicates that incorporating alternative-site profiles based on limited reference ligands does not necessarily improve predictive performance or complement the existing representative interaction profile or improve predictive performance.
In this study, we selected the Top-1 pose, ranked highest by the docking scoring function, as the representative binding hypothesis for each molecule. This choice was not based on an assumption that the Top-1 pose necessarily reflects the experimentally correct binding conformation, but rather on the need to consistently compare the interaction patterns of query and reference ligands using a single predicted binding configuration.
Nevertheless, we recognize that evaluating the sensitivity of predictive performance to pose selection is essential for assessing the robustness of this workflow. Accordingly, we conducted additional sensitivity analyses using the Top-1, Top-2, and Top-3 poses, applying the same selection criteria to both query and reference ligands. As shown below, the average Macro-F1 differences across pose ranks were minimal on both TDC and the UniTox set, with overall predictive performance remaining consistent despite some variations across individual endpoints.
Table.4 Macro-F1 comparison across docking pose ranks on TDC (Top-1, Top-2, and Top-3)
| Task | Top-1 | Top-2 | Top-3 |
|---|---|---|---|
| BBBP | 0.726 | 0.730 | 0.716 |
| DILI | 0.669 | 0.659 | 0.671 |
| HIA | 0.875 | 0.840 | 0.875 |
| Bioavailability | 0.694 | 0.718 | 0.718 |
| Average | 0.742 | 0.737 | 0.745 |
| vs. Top-1 | – | −0.005 | +0.003 |
Table.5 Macro-F1 comparison across docking pose ranks on UniTox (Top-1, Top-2, and Top-3)
| Task | Top-1 | Top-2 | Top-3 |
|---|---|---|---|
| Cardiotoxicity | 0.505 | 0.476 | 0.477 |
| Dermatological | 0.540 | 0.512 | 0.539 |
| Hematological | 0.506 | 0.523 | 0.505 |
| Infertility | 0.670 | 0.652 | 0.638 |
| Liver | 0.606 | 0.609 | 0.620 |
| Ototoxicity | 0.699 | 0.699 | 0.739 |
| Pulmonary | 0.509 | 0.522 | 0.524 |
| Renal | 0.510 | 0.496 | 0.507 |
| Average | 0.568 | 0.561 | 0.569 |
| vs. Top-1 | – | −0.007 | +0.001 |
We thank the reviewer for prompting this sensitivity analysis, which enabled us to explicitly assess the impact of pose selection. The results suggest that average predictive performance remains largely consistent across the evaluated pose ranks. We have incorporated these additional analyses into the revised manuscript.
Response to Reviewer 8rLz (1/4)
We sincerely thank the reviewer for carefully examining our work and raising insightful and thought-provoking questions regarding our methodology. Revisiting our workflow in light of the reviewer's comments allowed us to identify several interesting and meaningful interpretations that we had not fully recognized at the time of submission, ultimately helping us further strengthen our work.
We have organized our responses to the reviewer's comments into the following sections to facilitate the review process.
Q1: Experimental Evaluation and Baseline Comparisons
[Answer 1-1] Reporting Macro-F1 scores for all eight UniTox endpoints.
[Answer 1-2] Additional evaluations using GPT-5.1 and Claude-Sonnet-5.
Q2: Contributions of Evidence Components and Ablation Analysis
[Answer 2-1] Interpretation of the performance contributions and distinct roles of EPCE and EIE.
[Answer 2-2] Analysis of the ELCE ablation results.
Q3: Docking Protocol and Binding-Site Selection
[Answer 3-1] Evaluation of alternative and allosteric binding sites.
[Answer 3-2] Top-1/2/3 pose sensitivity analysis.
[Answer 3-3] PDB structure selection criteria and ensemble docking evaluation.
Q4: Qualitative Analysis and Typographical Corrections
[Answer 4-1] Failure case studies.
Detailed responses to each comment are provided below.
[Answer 1-1] Reporting Macro-F1 scores for all eight UniTox endpoints.
The original evaluation aggregated the endpoints into four physiological-system groups due to the imbalance in the number of cliff pairs across endpoints. In particular, Ototoxicity contains only six cliff pairs, making its Macro-F1 highly sensitive to changes in a small number of predictions. Therefore, rather than assigning equal weight to performance estimates from endpoints with limited samples, we grouped biologically related endpoints into four systems and reported weighted averages.
Nevertheless, we agree with the reviewer that group-level aggregation may obscure performance variations across individual endpoints. To address this concern, we have added a consolidated table reporting the individual Macro-F1 scores for all eight endpoints, including those of the existing baselines.
Table.1 Performance across eight endpoints. (For clarity, only the top two models in each category are shown; general-purpose LLMs are excluded. Full performance results for all models are provided in the Table.9 of Appendix B.1)
At the individual endpoint level, PharmLens achieves the highest performance on five of the eight endpoints. Furthermore, PharmLens obtains the highest performance among all evaluated models in both the unweighted average across the eight endpoints and the weighted average.
Therefore, although performance variations exist across individual endpoints, the overall performance advantage of PharmLens remains consistent across both endpoint-level and physiological-system-level evaluations.