- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper focuses on biological assay prediction. It is a challenging problem given the sparsity of measured activities per assay. Particularly for new assays with only few compounds labeled. Therefore authors present ASAP. They first estimate across-assay similarities by evaluating the prediction accuracy for one assay based on trained model of another assay. Then they leverage the similarity matrix to establish a collaborative assay space using contrastive learning, drawing similar assays together, and pushing dissimilar assay away from each other, where the assays are encoded through the protein sequence and assay description in text. When it comes to bioactivity training, ASAP combines an molecular encoder and the assay encoder pretrained in the collaborative assay space and applies an MLP to predict a score that is optimized by the ranking across compounds on the specific bioactivity. The authors benchmark their approach in zero-shot and few-shot scenarios, and demonstrate superior performance in most settings based on correlation metrics like Pearson’s coefficient and Concordance Index.
Critical strengths
- The ASAP aims at true generalization towards zero-shot and few-shot scenarios for new assays in low data regime with few measured compound activities. The authors formulate their solution appropriately so that, any new assay could be encoded in the collaborative assay space, either by the text description or protein sequence, giving a boost start to the new assay model training.
- The authors formulate their experiments rigorously. Particularly for zero-shot and few-shot settings, they purposefully prevent the data leakage issue by removing any overlapping compounds and similar target assays between training and unseen testing datasets.
Critical weaknesses
- It is unclear how biologically sound the assay similarity matrix is by the cross-assay predictivity. It can be largely affected by the compound data availability and diversity. Some assays may have very few measured compounds, some may severe class-imbalance among data with measurements, some may only cover very limited scope of chemical space. It’s likely not revealing the real biology in assay similarity given it’s very dependent on the existing limited data. Using that assay similarity matrix as a reference for contrast learning to establish the collaborative assay space could be risky, and therefore further affect the generalizability for the full framework. The authors showed a couple examples for biological relevance for computationally similar assays, and demonstrated the protein-level clustering on the latent space, but I would expect a more systematic biological validation given this is the biggest assumption of this paper. If most assays are protein target related, I find it surprising that the authors did not try to compare with reference datasets of protein-protein interactions.
- While ranking based objective makes sense for training, using only ranking or correlation based metrics for evaluation is not ideal. The relative binding likelihood or bioactivity strengths could be helpful, but in many cases scientists would still take the absolute binding affinity/probability or bioactivity magnitude to guide their decisions. Missing benchmarks on the absolute assay value predictions makes it less actionable.
Questions for authors
- How to put ASAP into practical use? Especially with a relative score outcome per outcome, how would biologists and chemists use it for compound mechanism understanding or screening?
- The authors promotes generalization of their method through zero-shot and few-shot testing. There’s a strong focus on assay-level knowledge transfer which essentially enables the generalization. It has its value, but to what degree does it help the assay prediction be usable? Would it make more sense to have a better predictor module than MLP? Would it be possible to leverage additional measured data from other similar assays to improve the prediction of one particular assay?