acceptodds
Under review as a conference paper at ICLR 2027

Error-Controlled Discovery of Protein Interactions via Conformal Calibration

Abstract

Protein language models (PLMs) are powerful predictors of protein interactions, but strong benchmark performance does not ensure reliable discoveries. We introduce PPI-Caliper, a model-agnostic framework that adapts conformal calibration to select interactions at user-specified false discovery rate (FDR) levels using held-out labels without retraining the predictor. Across five species, two PLM-based predictors, and two human training-data versions for each predictor, PPI-Caliper keeps empirical FDR at or below the target level under in-distribution calibration. Under distribution shift, transferred thresholds tend to become conservative when predictive performance improves, but can fail to control FDR when performance deteriorates. PPI-Caliper connects pre-trained interaction scores to explicit error targets and demonstrates why reliable biological discovery requires evaluating calibration alongside predictive accuracy.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
False discovery rate control, protein–protein interactions, conformal calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YtjfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors present PPI-Caliper, a post-hoc threshold calibration framework for protein–protein interaction (PPI) predictors that uses labeled reference data to select interactions at a desired False Discovery Rate (FDR). The method fundamentally uses ground-truth labels from a held-out calibration set to estimate which prediction score threshold yields an acceptable error rate. The authors evaluate the method on PPI benchmarks from five species, demonstrating that empirical FDR is maintained near the target when calibration and test data come from the same species. They further investigate cross-species generalization by calibrating thresholds on one species and applying them to another. They find that FDR control generally transfers between species with similar predictive performance, but can fail when model performance deteriorates in the target species.

Critical strengths

PPI-Caliper provides a straightforward, model-agnostic approach to post-hoc FDR calibration that does not require retraining the underlying PPI predictor. This makes it potentially useful for selecting candidate interactions for experimental follow-up when representative labeled calibration data are available. The authors evaluate the framework across five species and multiple PPI predictors, examining both in-distribution FDR calibration and cross-species transfer. In particular, the experiments highlight the limitations of transferring calibrated thresholds under distribution shift, which is a useful practical observation.

Critical weaknesses
  1. Limited methodological novelty and lack of meaningful baselines. The proposed framework is fundamentally a relatively simple threshold calibration procedure using labeled reference data. In particular, the choice of δ=0.5\delta = 0.5 in the Learn-then-Test procedure effectively reduces threshold acceptance to checking whether the mean estimated bootstrap FDP is below the target FDR. The authors also acknowledge that the method does not provide finite-sample FDR guarantees. Without comparisons to simpler baselines, such as directly selecting thresholds based on empirical FDP on a calibration set, it is unclear what additional benefit the proposed framework provides.

  2. Limited evaluation of FDR control in realistic discovery settings. The experiments rely on constructed PPI benchmarks with a fixed 1:10 positive-to-negative ratio, where negative interactions are randomly sampled rather than experimentally verified. Calibration and test sets are obtained through random pair-level splits of these benchmarks. Since FDR is sensitive to the underlying prevalence of true interactions, the observed FDR control may not generalize to realistic PPI discovery settings with unknown interaction prevalence. Although the authors investigate cross-species transfer and demonstrate that FDR control can fail under distribution shift, they do not establish a quantitative or actionable criterion for determining when calibration will transfer reliably.

Questions for authors
  1. Given that the choice of δ=0.5\delta=0.5 effectively reduces threshold selection to checking whether the mean bootstrap FDP is below the target FDR, what benefit does the proposed framework provide over directly selecting a threshold using empirical FDP on the labeled calibration set? A comparison against this simple baseline in terms of FDR control and statistical power would help clarify the methodological contribution.

  2. The evaluation relies on PPI benchmarks with a fixed 1:10 positive-to-negative ratio, where negative examples are randomly sampled rather than experimentally verified. How representative are these benchmarks of practical PPI discovery settings, where the prevalence of true interactions may differ substantially? In particular, does the proposed framework maintain FDR control under different class ratios or imperfect negative labels? Evidence addressing these concerns would substantially strengthen the practical claims.

  3. The authors observe that differences in predictive performance across species are associated with the reliability of transferred thresholds. However, evaluating predictive performance (e.g., AUPRC) on a target species requires labeled target data. How would this observation be used in practice when such labels are unavailable? Is there an objective, quantitative criterion for determining when FDR control is likely to transfer successfully?

Reviewer_MF3gOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Most protein-protein interaction (PPI) prediction studies would simply choose a deployment decision threshold by selecting a point from the precision-recall curve of the validation set (e.g., choose the threshold that maximizes recall for a precision of at least X %). This paper presents a complicated statistical approach to do the same thing. The authors propose a method to determine a decision threshold that limits the false discovery rate (the proportion of predicted positives that will be FPs, or 1-Pr) at a given value. Prediction scores on the calibration set are ranked and binned. The decision threshold is then swept from most permissive to least. At each threshold, bootstrapping is used to estimate the mean and SD of the expected FDR for that threshold. Hypothesis testing is then used to determine whether the expected true/test FDR will fall below the required threshold. The most permissive threshold that meets the requirement is kept.

Results show that the proposed method works well when calibrating the decision threshold and then testing said threshold on the same species, but often fails when the calibration and test species differ.

Critical strengths

The paper is clearly written and easy to follow.

Critical weaknesses

The proposed method is quite complex, does not seem to work out-of-distribution, and is not demonstrated to outperform a much simpler/standard approach for selecting the decision threshold (see summary above).

Many of the conclusions presented in this paper are already well established in the PPI prediction literature. For example:

  • 307-308: Higher power (recall) with higher FDR (lower precision) is an entirely expected result.
  • 309-316: The fact that higher power (recall) can be achieved for the same FDR (precision) on different species is a well-established fact. PPI performance in E.coli is often lower than other species. A more difficult problem will result in a lower recall for the same level of precision (or vice versa).
  • 317-323: The fact that accuracy is not a useful metric for a highly imbalanced ML problem is well understood. This result is entirely expected.

333-342: The finding that a decision threshold (for a given FDR) tuned on one species fails to generalize to other species is not unexpected. One would hope that such a complex approach to determining the decision threshold would be more robust to such domain shifts than would be a simpler method (e.g., determine the decision threshold that maximizes recall at some minimum precision from the PR curve). These results indicate that the method will have limited utility in the field.

As stated above, no baseline approach for solving this problem is provided, so any incremental benefit of the proposed method cannot be demonstrated

249-251: The authors claim that, by using models trained on both STRING v11 and STRING v12, this permits assessment of calibration and transfer behavior "across supervised training datasets". Clearly, v11 and v12 of the same dataset are not independent and thus this experiment cannot establish behavior "across datasets".

Questions for authors

Please respond to the Weaknesses stated above. Additional questions:

  • Rather than just computing the FDR for a given threshold, the authors use bootstrapping to get a mean and SD of the expected FDR for a given calibration set of protein pairs. It is unclear why we must use a hypothesis test to see if a given threshold leads to an FDR < q (equation 4). If the FDR is a random variable with both mean and SD (equation 3), then OK... But, empirically, how important is it to consider the variability of the FDR estimate (for a reasonably sized calibration set)? Can we simply treat the FDR estimated over the entire calibration set as a point estimate without considering the uncertainty of this estimate? How much would we lose by doing this?

  • Equation 2 seems to indicate that scores are converted to ranked bins (bin 1...1000). This is fit on the calibration set and applied to the test set. How is this preferable to using the raw score? Are the predictions in the test set T also sorted and assigned a rank-bin, or does the same mapping from score-->bin apply (without refitting)? It appears not, and that a fixed mapping is used even when we expect a domain shift to cause the preciction scores to shift up/down and/or spread. In this case, wouldn't we expect a fixed mapping to fail?

  • The Related Work section mentions two FDR estimation methods already designed for PPI prediction, and several more designed for link prediction (of which PPI prediction is an example application). All of these are dismissed without direct experimental comparison. The authors state that the proposed PPI-Caliper does not require "a network observation model" (why is this difference important/useful?) and that the present study examines performance under distribution shift (which proves to be weak). Why was a direct comparison with these SOTA alternatives completed?

Minor

  • "discovery power" is defined as "the fraction of true interactions in T that are selected". How does this differ from "recall" (a term used much more frequently in ML and PPI prediction literature)
  • 260-261: Why would the bootstrap samples be smaller than the calibration set (10% of pre-split dataset size = 20% of calibration set size?) Aren't bootstrap samples typically the same size as the entire dataset, but chosen with replacement?
  • If we expect the scores to be different/shifted when moving to another species, why would we use the same score-to-quantile mapping for both species? Why not run all predictions and then determine the quantiles at test time?
  • Why are only two PPI prediction methods included in the comparison? Several methods are publicly available and often appear as baselines in contemporary papers.
  • Lines 54-56: The suggestion that PPI negative samples would be generated by shuffling or reversing sequences is completely out of sync with SOTA approaches to creating negative PPI data (Bernett et al 2024, (Chatterjee et al., 2025, others). Instead, negatives are formed by swapping proteins within positive pairs for random proteins, while controlling for data leakage. Why mention sequence shuffling/reversing here?
  • Lines 37-39: The statement that "AUPRC... do not indicate how many interactions selected at a given threshold are false" is strange. The PR-curve (that AUPRC summarizes) does precisely that: for each possible decision threshold, the precision is reported, which reports what proportion of predictions are TRUE at a given threshold, and FDR = 1-Pr. What is meant by this claim (only that summarizing the curve with a since AUC value loses information?)
Reviewer_gD4POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a technique for selecting a threshold of an existing PPI predictor (PLM-Interact, MINT) to avoid False Discovery Rate.

They test on a five-species benchmark, using models trained on human data.

Critical strengths

Well motivated and impactful research question.

The approach seems being able to control the FDR.

The code is available for reproducing experiments.

Critical weaknesses

There is no comparison with a naive cutoff.

The approach is only tested on two baselines.

The approach is quite related with ranking. But the models are trained for classification. There is recent work that introduces PPI Candidate Ranking (Russo et al., ICML 2026). The approach could be more suited on such method.

The novelty is limited: The method is a close adaptation of Boger et al. (2025) and the Learn-then-Test framework. The contribution is mostly the application and empirical study.

Questions for authors

How does PPI-Caliper compare to naive threshold selection techniques? For instance, could the authors test wrt using the precision-recall curve to pick a threshold?

Could the authors explain why only two baselines are tested? Could the authors test on PPI-candidate ranking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.