- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a novel competition-aware deep learning based framework for bioactivity prediction. Different from existing drug discover methods, which only consider the interactions between proteins and ligands, the proposed DrugComp also includes the effects caused by cellular environments and drug action. Additionally, a new dataset named CompDB is curated by the authors, which contains a large number of test ligands and radioligands for experimental validation.
Critical strengths
- The new dataset curated by the authors could be very beneficial to the machine learning community. Unlike conventional deep learning datasets that rely primarily on straightforward data annotation, this dataset incorporates substantial biological prior knowledge into its annotation, collection, and curation processes, enabling a more meaningful evaluation of whether the proposed model can effectively generalize to real-world biological downstream tasks.
Critical weaknesses
- The authors employed LLMs for certain information filtering and extraction tasks. My main concern is the potential risk of LLM hallucinations during this process. Given the sensitivity and complexity of biological data, even a small proportion of hallucinated or inaccurate information could significantly compromise the reliability and integrity of the entire dataset.
- The proposed method constructs a graph using radioligands and test ligands to identify potential test ligands that could competitively displace radioligands. While this approach is biologically reasonable, from a deep learning perspective, it may introduce a bias toward the radioligands already represented in the dataset, potentially limiting the model's generalization ability. In real-world drug discovery scenarios, however, such models are expected to generalize beyond known ligand interactions and identify previously unseen candidates. Therefore, I am concerned that the proposed graph construction strategy may lead the model to rely heavily on existing radioligand-associated patterns rather than learning more generalizable protein–ligand interaction mechanisms.
Questions for authors
- Regarding the first weakness, I would like to see the authors to provide more details regarding the LLM uses during dataset curation. Also the authors need to verify the accuracy of the information extracted by the LLM.
- Regarding generalization, I would like the authors to provide additional zero-shot evaluation cases to demonstrate that the proposed method has not overfitted to the protein–ligand pairs in the training set and can effectively generalize to previously unseen interactions.
4 comments
We sincerely thank the reviewer for the careful review and constructive feedback. We have provided point-by-point responses, including additional analyses and clarifications, and would greatly appreciate your feedback on whether these adequately address your concerns. If so, we hope you might consider revisiting your overall assessment.
Thanks for the authors' response. I think the rebuttal has fully addressed my questions. Therefore I'm willing to raise my score.
Thank you for your positive feedback and for raising your rating. We sincerely appreciate your time in reviewing our paper and responses.
Response
Response to Weakness 1 and Question 1: LLM extraction
We carefully considered the potential risk of LLM hallucinations during dataset curation and therefore designed a highly constrained prompt for radioligand extraction:
In particular, we explicitly instruct the LLM to abstain from extraction when the assay description does not specify a radioligand. To further address the reviewer's concern, we conducted an additional manual verification, in which two annotators independently reviewed all 861 distinct descriptions from 10,675 assay records. After standardizing radioligand names, the annotators achieved 100% inter-annotator agreement (Cohen's κ=1.0). Comparing the LLM outputs against these manually verified records, we found that 99.27% of the extractions were correct. The remaining 0.73% exclusively involved cases where no radioligand was explicitly specified in the assay description, but the LLM inferred a radioligand.
Upon further inspection, we found that these inferred compounds are commonly used radioligands in relevant assays and are therefore plausible candidates. Although they cannot be regarded as experimentally verified radioligands for the corresponding assays, we believe such candidates can still be useful in practical applications of our framework. Specifically, large-scale bioactivity prediction often involves protein–ligand pairs without prior wet-lab measurements or assay descriptions. In such scenarios, plausible radioligands can be introduced to construct the competitive binding context required by DrugComp, enabling its application to previously untested protein–ligand pairs.
Response to Weakness 2 and Question 2: Generalisation
We provide additional zero-shot evaluations to assess generalization to unseen targets and ligands.
Practical assay setting. In competitive binding assays, radioligands are typically established tracers reused across experiments, whereas test ligands are the new candidates being screened. Introducing a new radioligand requires additional experimental characterization and validation [1]. This is reflected in CompDB-GPCR, where each target is associated with an average of 71 distinct test ligands but only 2 radioligands. Thus, generalization to new candidates using established radioligands represents a primary practical application of DrugComp.
Zero-shot generalization. To further assess generalization, we evaluated DrugComp under a target-disjoint split, restricting test samples to those whose test ligands were also absent from training. Thus, both targets and test ligands are unseen during training.
DrugComp consistently improves RMSE across all three backbones, suggesting that its benefits extend to previously unseen target–ligand combinations rather than relying solely on memorized interactions.
[1] Hulme, Edward C., and Mike A. Trevethick. "Ligand binding assays at equilibrium: validation and interpretation." British journal of pharmacology 161.6 (2010): 1219-1237.