- Rating
- 1
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The protein-ligand binding affinity may not be accurately predicted due to OOD test issue, therefore, this work proposes PAIRank to predict a score demonstrating the reliability of the original prediction, which is a plug-in for any protein–ligand binding affinity prediction model. This method adopts training embeddings and computes cluster-conditioned Mahalanobis distances, showing how an evaluation sample is relative to the training distribution, and finally learns a model providing an error-risk score.
Critical strengths
- The problem is meaningful, it is necessary to know how reliable the prediction is in OOD test for drug discovery task.
- The method presentation is easy to understand.
Critical weaknesses
-
The paper describes the OOD test issue with many words, however, it is not convincing that the design of RAIRank properly addresses this problem. PAIRank learns a mapping from Mahalanobis-distance features to prediction-error risk using pairwise supervision from training samples. However, it is unclear whether the learned relationship between distributional distance feature and prediction error remains valid for OOD test samples, especially the learning is only within the training set. This concern is further motivated by the relatively weak risk–error correlations reported in the experiments (e.g., 0.198 for TrustAffinity and 0.086 for BACPI on ChEMBL). First, the relationship between distributional distance and prediction error itself is not straightforward. Second, it is not convincing that this learned mapping on the training distribution is still valid for OOD test, the paper also recognized that there are many factors for OOD prediction failures, can these factors all be solved by embedding distribution deviation? Therefore, the method is questionable.
-
The novelty is somewhat limited. The proposed method mainly combines existing techniques, including K-means clustering, Mahalanobis-distance features, and pairwise ranking loss. The main difference from existing error-modelling approaches, particularly eMOSAIC, appears to be the replacement of error regression with pairwise ranking, which has been commonly used in current drug discovery research.
-
The writing does not sufficiently support the motivation and the design. Although it is clearly written that the problem is meaningful, the introduction is more like listing the related work on the OOD issue, without a strong motivation and clear support for the design.
Questions for authors
Stronger justification and further experiments for the question in Weakness1 are needed, for example, comparisons with direct distance-based risk estimation and evaluations of how the distance–error relationship changes under distribution shifts.