acceptodds
Withdrawn from ICLR 2027

PAIRank: A MODEL-AGNOSTIC RELIABILITY PLUGIN FOR PROTEIN–LIGAND BINDING AFFINITY PREDICTION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

tional drug screening, but their performance can deteriorate on novel chemical scaffolds and other out-of-distribution (OOD) samples. Most predictors do not provide sample-level reliability information for identifying predictions with po- tentially large errors. Here, we introduce PAIRank (Protein–ligand Affinity In- dividual Reliability Ranking), a model-agnostic reliability plug-in for protein– ligand binding affinity prediction. PAIRank extracts joint protein–ligand embed- dings from a frozen base affinity predictor, partitions the training embeddings into local regions and computes cluster-conditioned Mahalanobis distances from each evaluated sample to every region. Together, these distances describe how the sam- ple is positioned relative to the training distribution. Pairwise supervision derived from base-model prediction errors then enables PAIRank to learn the relative error risk of different samples and assign a score for reliability ranking. These scores allow lower-risk predictions to be retained preferentially. We evaluated PAIRank under ligand-scaffold extrapolation on ChEMBL and on a temporally external BindingDB set using multiple affinity predictors and uncertainty-quantification methods. Experimental results showed that PAIRank reduced error in the re- tained subsets and exhibited competitive and relatively stable selective-retention performance across base models and data distributions. PAIRank requires neither modification nor retraining of the base predictor and it can serve as a lightweight post-processing plug-in for assessing the reliability of individual predictions from different affinity-prediction frameworks under distribution shift.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein--ligand binding affinity, individual reliability ranking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YJWFOpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The protein-ligand binding affinity may not be accurately predicted due to OOD test issue, therefore, this work proposes PAIRank to predict a score demonstrating the reliability of the original prediction, which is a plug-in for any protein–ligand binding affinity prediction model. This method adopts training embeddings and computes cluster-conditioned Mahalanobis distances, showing how an evaluation sample is relative to the training distribution, and finally learns a model providing an error-risk score.

Critical strengths
  1. The problem is meaningful, it is necessary to know how reliable the prediction is in OOD test for drug discovery task.
  2. The method presentation is easy to understand.
Critical weaknesses
  1. The paper describes the OOD test issue with many words, however, it is not convincing that the design of RAIRank properly addresses this problem. PAIRank learns a mapping from Mahalanobis-distance features to prediction-error risk using pairwise supervision from training samples. However, it is unclear whether the learned relationship between distributional distance feature and prediction error remains valid for OOD test samples, especially the learning is only within the training set. This concern is further motivated by the relatively weak risk–error correlations reported in the experiments (e.g., 0.198 for TrustAffinity and 0.086 for BACPI on ChEMBL). First, the relationship between distributional distance and prediction error itself is not straightforward. Second, it is not convincing that this learned mapping on the training distribution is still valid for OOD test, the paper also recognized that there are many factors for OOD prediction failures, can these factors all be solved by embedding distribution deviation? Therefore, the method is questionable.

  2. The novelty is somewhat limited. The proposed method mainly combines existing techniques, including K-means clustering, Mahalanobis-distance features, and pairwise ranking loss. The main difference from existing error-modelling approaches, particularly eMOSAIC, appears to be the replacement of error regression with pairwise ranking, which has been commonly used in current drug discovery research.

  3. The writing does not sufficiently support the motivation and the design. Although it is clearly written that the problem is meaningful, the introduction is more like listing the related work on the OOD issue, without a strong motivation and clear support for the design.

Questions for authors

Stronger justification and further experiments for the question in Weakness1 are needed, for example, comparisons with direct distance-based risk estimation and evaluations of how the distance–error relationship changes under distribution shifts.

Reviewer_MVdiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies methods for quantification of error prediction risk for a basemodel used for prediction of ligand-target binding affinity. The work is based on the a few steps:

  1. Embeddings of ligands and targets are extracted from intermediate layers of basemodels.

  2. K-means are used for creating clusters of ligands and targets.

  3. Distances between new data examples to cluster centroid are used as features to train a risk prediction models.

  4. Instead of predicting risk as regression the authors propose to use learning to rank to mitigate the difficulty in risk regression.

The authors for show that the top 15% test instances selected by their methods have MAE reduction compared to the overall MAE of the model on 100% of the test data. The authors also compared their methods to relevant baselines.

Critical strengths

The paper studies an interesting approach for quantifying prediction uncertainty under the context of OOD, the strength of the paper includes a new methods based on the eMOSAIC framework with a minor modification, turning the regression problem into a learning to rank problems which could be considered as a minor modification to existing work but help improving the results compared to the baselines.

The paper was well written and it tackles an important research problem in drug discovery.

Critical weaknesses

For drug discovery application, demonstration of better risk estimation on top 15% examples may not enough as those low risks example might just concern very easy to predict examples and likely known in-distribution hits or non-binders. I have a strong concern reagarding this point.

The proposed approach is based on the eMOSAIC framework with a minor modification, turning the regression problem into a learning to rank problems. While for this type of problems it is a reasonable change but the novelty is limited by the existence of the eMOSAIC framework. Therefore I have another concern on the novelty of the work in terms of overall new concepts proposal and new solutions to the problem significant enough to be qualified as a solid publication at ICLR.

I have another concern on the OOD claims. The split is based on scaffold splits, this is not enough to demonstrate OOD under scaffold split as two molecules with different scaffolds can be similar on an embedding space. Additional experiments on more sostificated splits to demonstrate the effectiveness under OOD.

Questions for authors
  1. Could you please do the following experiments:

+. Evaluate PAIRank on more challenging OOD settings, particularly target-cold and protein-family-cold splits.

  • Consider split based on the embeddings spaces, the splits by clusters could be relevant.

  • compare to simple distance-based baselines, such as minimum Mahalanobis distance, k-nearest-neighbor distance

Reviewer_8i2DOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PAIRank, a model-agnostic reliability ranking framework for protein-ligand binding affinity prediction. The method extracts joint protein-ligand embeddings from a frozen affinity predictor, constructs cluster-conditioned Mahalanobis distance features, and trains a pairwise ranking head using prediction errors to estimate the relative risk of individual predictions. The authors evaluate PAIRank on ChEMBL scaffold splits and an external BindingDB dataset using four affinity prediction models. The results show that PAIRank can reduce prediction errors in retained subsets and generally outperform regression-based error modeling, particularly at low retention rates.

Critical strengths

The paper addresses an important problem in protein-ligand binding affinity prediction, where identifying unreliable predictions is valuable for virtual screening and candidate prioritization. The proposed method is relatively simple and can be integrated with existing affinity predictors without retraining the base models. The experiments cover four affinity predictors with different architectures and include both scaffold-based and temporal external evaluations. The comparison between pairwise ranking and regression-based error modeling is also useful.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework is closely related to the existing eMOSAIC method, which also uses cluster-conditioned Mahalanobis distance features derived from joint protein-ligand embeddings for individual uncertainty estimation. The main methodological difference appears to be replacing error regression with pairwise ranking. However, pairwise ranking itself is an established technique. Therefore, the methodological contribution is relatively incremental.

  2. Potential bias in the error supervision used to train PAIRank. Section 2 states that PAIRank uses the absolute prediction errors of training samples to construct pairwise supervision. However, it is unclear whether these errors are obtained from the same samples used to train the base affinity predictor or from held-out predictions. If the former, the errors may not represent the generalization errors encountered on unseen samples. This could introduce a mismatch between the training supervision and the actual objective of identifying unreliable OOD predictions.

  3. Fairness of the virtual screening evaluation. In Figure 2, the authors report improvements in AUROC, BEDROC and enrichment factors after retaining only 15% of the samples. However, it is unclear whether these metrics are calculated on the retained subset and compared directly with those calculated on the complete test set. Such a comparison may not be fair because removing difficult samples can improve ranking metrics without necessarily improving virtual screening performance on the original candidate library.

  4. Limited generalization across models and datasets. Although the proposed method is described as model-agnostic, its reliability ranking performance varies considerably across base models and data distributions. For example, on the external BindingDB dataset, BACPI shows almost no improvement in MAE, an increase in RMSE, and a negative Spearman correlation between the predicted risk scores and actual errors. Moreover, PAIRank performs worse than RIO and MC dropout in terms of AUSE-MAE on BindingDB. These results suggest that the effectiveness of PAIRank depends strongly on the quality and transferability of the base-model embeddings.

  5. Insufficient evidence for the advantage of pairwise ranking. Although the comparison with regression-based error modeling supports the use of pairwise ranking on ChEMBL, the improvements are relatively small in terms of error-risk correlation and less consistent on BindingDB. Furthermore, Table S6 uses the compression ratio between the standard deviations of predicted risk scores and true errors to explain the limitations of regression-based modeling. However, this ratio depends on the arbitrary scale of the predicted scores, while ranking performance is invariant to positive rescaling. Therefore, a low compression ratio does not necessarily indicate poor ranking performance. More direct evidence is needed to explain why the pairwise ranking objective is preferable to regression-based error modeling.

Questions for authors
  1. Are the prediction errors used to train PAIRank obtained from in-sample predictions of the base model or from held-out or out-of-fold predictions? If they are in-sample errors, how do the authors address the potential mismatch between training errors and generalization errors?

  2. How are AUROC, BEDROC and enrichment factors calculated after retaining only 15% of the candidates? Are these metrics evaluated on the retained subset or the original candidate library? How does PAIRank compare with affinity-only ranking under the same screening budget?

  3. Why does PAIRank show little benefit for BACPI on BindingDB, and why does it underperform some uncertainty baselines on this external dataset? What characteristics of the base-model embeddings determine whether PAIRank is effective?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.