- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors present PPI-Caliper, a post-hoc threshold calibration framework for protein–protein interaction (PPI) predictors that uses labeled reference data to select interactions at a desired False Discovery Rate (FDR). The method fundamentally uses ground-truth labels from a held-out calibration set to estimate which prediction score threshold yields an acceptable error rate. The authors evaluate the method on PPI benchmarks from five species, demonstrating that empirical FDR is maintained near the target when calibration and test data come from the same species. They further investigate cross-species generalization by calibrating thresholds on one species and applying them to another. They find that FDR control generally transfers between species with similar predictive performance, but can fail when model performance deteriorates in the target species.
Critical strengths
PPI-Caliper provides a straightforward, model-agnostic approach to post-hoc FDR calibration that does not require retraining the underlying PPI predictor. This makes it potentially useful for selecting candidate interactions for experimental follow-up when representative labeled calibration data are available. The authors evaluate the framework across five species and multiple PPI predictors, examining both in-distribution FDR calibration and cross-species transfer. In particular, the experiments highlight the limitations of transferring calibrated thresholds under distribution shift, which is a useful practical observation.
Critical weaknesses
-
Limited methodological novelty and lack of meaningful baselines. The proposed framework is fundamentally a relatively simple threshold calibration procedure using labeled reference data. In particular, the choice of in the Learn-then-Test procedure effectively reduces threshold acceptance to checking whether the mean estimated bootstrap FDP is below the target FDR. The authors also acknowledge that the method does not provide finite-sample FDR guarantees. Without comparisons to simpler baselines, such as directly selecting thresholds based on empirical FDP on a calibration set, it is unclear what additional benefit the proposed framework provides.
-
Limited evaluation of FDR control in realistic discovery settings. The experiments rely on constructed PPI benchmarks with a fixed 1:10 positive-to-negative ratio, where negative interactions are randomly sampled rather than experimentally verified. Calibration and test sets are obtained through random pair-level splits of these benchmarks. Since FDR is sensitive to the underlying prevalence of true interactions, the observed FDR control may not generalize to realistic PPI discovery settings with unknown interaction prevalence. Although the authors investigate cross-species transfer and demonstrate that FDR control can fail under distribution shift, they do not establish a quantitative or actionable criterion for determining when calibration will transfer reliably.
Questions for authors
-
Given that the choice of effectively reduces threshold selection to checking whether the mean bootstrap FDP is below the target FDR, what benefit does the proposed framework provide over directly selecting a threshold using empirical FDP on the labeled calibration set? A comparison against this simple baseline in terms of FDR control and statistical power would help clarify the methodological contribution.
-
The evaluation relies on PPI benchmarks with a fixed 1:10 positive-to-negative ratio, where negative examples are randomly sampled rather than experimentally verified. How representative are these benchmarks of practical PPI discovery settings, where the prevalence of true interactions may differ substantially? In particular, does the proposed framework maintain FDR control under different class ratios or imperfect negative labels? Evidence addressing these concerns would substantially strengthen the practical claims.
-
The authors observe that differences in predictive performance across species are associated with the reliability of transferred thresholds. However, evaluating predictive performance (e.g., AUPRC) on a target species requires labeled target data. How would this observation be used in practice when such labels are unavailable? Is there an objective, quantitative criterion for determining when FDR control is likely to transfer successfully?