acceptodds
Under review as a conference paper at ICLR 2027

Constructed Negatives Shape the Evaluation of Sequence-Based Protein–Protein Interaction Models

Abstract

Many machine-learning tasks, including representation learning, rely on negative examples that are constructed rather than directly observed. This issue is especially important for protein–protein interaction (PPI) prediction, where experimentally established non-interactions are scarce. PPI models are therefore commonly trained and evaluated against sampled, unobserved protein pairs treated as negatives. We study how this construction affects model comparison. Keeping positives, protein-disjoint splits, class balance, and model selection fixed, we vary training and evaluation negatives independently across six constructions and four sequence-based models, three of which use protein language model (pLM) representations. Changing only the test negatives shifts AUPRC by up to 0.26, and the top-ranked architecture changes with the construction. Negatives matched to true partners in pLM representation space reduce all pLM-based models to near-chance performance, and shuffling partners within each class shows that the original pairing contributes little to AUROC. Constructed training negatives rarely outperform uniform sampling: on external data, they underperform it in 108 of 120 comparisons for the pLM-based models, with gains never exceeding 0.05 AUROC and losses up to 0.23. We release splits, negative-generation procedures, and evaluation code that allow training and evaluation negatives to be varied separately. Our results show that with constructed negatives, model comparisons are not just noisy but systematically sensitive to how negatives are defined, a methodological choice that is often left implicit.

Primary area:
datasets and benchmarks
Keywords:
protein-protein interaction prediction, negative sampling, representation learning, benchmark design, model evaluation, distribution shift, sequence-based models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_8vimOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

The paper asks how the way negative examples are constructed affects the evaluation of sequence-based PPI models. Experimentally confirmed non-interactions are rare, so PPI datasets usually treat unobserved protein pairs as negatives. Starting from the leakage-free human Gold benchmark of Bernett et al. (2024), the authors keep the positives, the protein-disjoint split, class balance and model selection fixed, and change only the negatives. They use six constructions: uniform random, degree-matched, sequence-matched, ESM-2-matched, the bias-reducing ILP of Bernett et al. (2026), and topology-driven TPPNI. Training and test negatives are crossed in a 6 × 6 design for four models (a frozen ESM-2 MLP, TUnA, D-SCRIPT-ESM2 and RAPPPID) with three seeds each. The models are also evaluated on external data without retraining: Y2H assay outcomes, curated IntAct negatives, and BioGRID/STRING positives paired with constructed negatives. They find that changing only the test negatives shifts AUPRC by up to 0.26, and that the best-ranked model changes with the construction. They also observe that ESM-matched negatives push all ESM-based models to near chance, and that shuffling partners within each class barely changes AUROC, so the models make little use of the specific protein pair. Finally, constructed training negatives rarely beat uniform sampling on external data. The authors conclude that negative construction should be reported and varied as an evaluation axis, not treated as a preprocessing detail.

Critical strengths
  1. The design is clean and well controlled. Positives, splits, class balance, checkpoint selection and thresholds are held fixed, so differences in performance can be attributed to the negatives. I find the crossing of training and test negatives the right way to separate the two effects.

  2. The partner-shuffle control is the most useful contribution. It directly tests whether a model needs the original A–B pairing, and the answer (it barely does) is important for anyone who reads high benchmark scores as evidence of learned interaction biology.

  3. The authors are careful about what their labels mean. They consistently separate assay non-detections, curated negatives and constructed unobserved pairs, they avoid calling constructed negatives "non-interactions", and they distinguish cross-source transfer from true out-of-distribution generalisation. Most PPI papers are not this careful.

  4. The evaluation goes beyond the constructed benchmark. Testing on Y2H outcomes and curated IntAct negatives, even though both are small, is the right direction and rarely done.

  5. The practical recommendations at the end are sensible and easy to adopt: report the candidate set, test at least one alternative construction (including one matched in the model's own representation space), include controls that don't need the original pair, and evaluate on assay data where possible.

  6. The data support a stronger conclusion than the paper draws. In the released figure data, a simple logistic regression on pooled single-protein ESM embeddings performs about as well as the full PPI models. Performance also drops from about 0.70 AUPRC on the constructed Gold test to roughly chance against experimental negatives. I would encourage the authors to put both findings in the main text.

Critical weaknesses
  1. The external results hide that performance on experimental negatives is close to chance. External transfer is reported only as differences from Uniform training. In the released figure data, absolute AUROC averaged over training constructions is 0.51-0.56 on Y2H and 0.51-0.52 on IntAct; the best single condition is 0.62, and several are below 0.5. The headline "constructed training negatives underperform Uniform in 108 of 120 comparisons" therefore mostly compares near-chance numbers, which makes it misleading on its own. It also obscures what I consider one of the paper's most important results: about 0.70 AUPRC on the constructed Gold test versus close to chance on experimentally tested negatives. Request: report absolute external performance in the main text and state this gap explicitly.

  2. Label noise in the constructed negatives is not measured, so the hardness of some constructions cannot be interpreted. The authors suggest that matched negatives, especially ESM-matched ones, may contain many real but unrecorded interactions, because close homologs of a partner often bind the same protein. If so, the near-chance results on these sets may reflect mislabelling rather than difficulty, and this is never checked. In the code, training negatives are filtered only against Gold training positives, and the collision count in the sampler summary is hard-coded to zero rather than computed. Request: for each construction, report the fraction of negatives that appear as positives in BioGRID, STRING, IntAct or HuRI.

  3. The quantitative summaries overstate the strength of the evidence. With three seeds, seed-to-seed spread in external AUROC reaches 0.12, so many of the 108/120 comparisons are within noise. They are also not independent, since they share models and test sets. The degree-matched test set reduces to a random draw on a protein-disjoint split (all test proteins have zero training degree), and its scores are nearly identical to Uniform (e.g. 0.633 vs 0.631 for Frozen-ESM2), yet it is counted in summaries such as "best in 20 of 36 settings". The external sets are also described too loosely. The Y2H set is mainly a transcription-factor isoform screen, not a random sample of human pairs, and results are not separated for proteins that overlap with Gold training and proteins that do not. Request: replace win/loss counts with effect sizes and confidence intervals, exclude the degree-matched test set from summary counts, describe the Y2H set accurately (and check that the cited source is the study the data came from), and report external results separately for overlapping and new proteins.

  4. The contribution is not clearly positioned against prior work, and the main new result is underdeveloped. Park and Marcotte (2011) already showed, with positives fixed, that changing only the test negatives shifts AUC substantially. Reim et al. (2025) showed on the same benchmark that ESM-2 embeddings, not architecture, explain the performance plateau. The paper does not engage with Park and Marcotte's central argument: that uniform random test negatives are the unbiased choice, and that a protein's general tendency to interact is a legitimate signal. That argument bears directly on how the near-zero pair dependence should be read. The shuffle analysis, which I consider the clearest new contribution, gets one paragraph. On the sequence- and ESM-matched sets it mainly reflects how the test set was built. Evidence that supports it is left out of the main text: a simple logistic regression on pooled single-protein ESM embeddings performs about as well as the full PPI models. The conclusion that models do not use the specific pair is also stated more broadly than tested, since only ESM-2-based sequence models were evaluated. Request: state what the paper adds beyond these works, discuss Park and Marcotte's argument, expand the shuffle analysis (including the pooled-ESM comparison), and limit the pair-dependence conclusion to the model class tested.

  5. The iteration history behind the "fixed in advance" claim is not disclosed. The paper states that the models "were fixed before final-test inspection, so no architecture was added post hoc". The repository shows that the reported experiment is the third iteration, that half of the 432 result cells are reused from an earlier version whose test results had already been examined, and that a fifth model (SENSE-PPI) was dropped between versions. The stated reason is logistical and plausible, and I do not think the results are necessarily biased, but readers cannot judge this from the paper. Request: describe the earlier iterations, the reused cells, and the removal of SENSE-PPI.

  6. The released materials do not match the release claims. The abstract promises released splits, negative-generation procedures and evaluation code. The anonymised repository contains code and per-figure tables, but no negative sets, splits or result registries, and most artifacts cited in its own documentation are missing. Its own audit also notes that the results were produced from code with uncommitted changes that cannot be reconstructed. Since a reusable evaluation resource is one of the stated contributions, the paper does not currently deliver it. Request: release the negative sets, splits and result registries.

Questions for authors
  1. What fraction of each negative construction consists of pairs reported as interacting in BioGRID, STRING, IntAct or HuRI? In particular, is this fraction substantially higher for the ESM-matched and sequence-matched sets? If so, how much of the near-chance performance on those sets can be attributed to mislabelled negatives rather than to genuine difficulty?

  2. How do you interpret the near-zero pair dependence in light of Park and Marcotte (2011), who argue that a protein's general tendency to interact is a legitimate predictive signal and that uniform random test negatives are the unbiased choice? Do you view the per-protein signal the models rely on as a shortcut, as partly legitimate, or as dependent on the intended use of the model?

  3. Did results from the earlier iterations (E001v3 and E002) influence any design decision in the final experiment, such as the choice of constructions, metrics, checkpoint selection, or the removal of SENSE-PPI? Will the negative sets, splits and result registries be publicly released?

12 comments
AuthorsOpenReview ↗

Major correction: Partner shuffling

Thank you for the detailed review. It highlighted the interpretation of pair dependence, weak performance hidden by relative comparisons, and gaps in the original code release. Our subsequent checks uncovered the shuffling implementation error. We corrected the errors we could verify and now state the remaining limits directly.

Major correction: Partner shuffling. We withdraw the submitted conclusion that the models barely use the original protein pair. The shuffled rows carried a new partner sequence but kept identifiers from the old partner. As a consequence, the lookup could score the wrong sequence. We fixed this and added tests across all four model input paths. In our 1024-residue experiment with Uniform negatives in both training and testing, AUROC drops by 0.134 (Frozen-ESM2) and 0.182 (TUnA) upon shuffling. On ESM-matched test negatives the effects are smaller, 0.020 and 0.026 respectively. We report the 512-residue experiment separately because its corrected analysis is not an exact replay of the historical execution. Figure 3 shows individual seeds and their means; Appendix Table S8 gives the corresponding selected paired test-row intervals (Sec. 5, partner-shuffling results).

AuthorsOpenReview ↗

W1: Absolute performance on external data

We agree, and we now treat this gap as a result rather than a reporting detail. Figure 4 reports absolute AUROC per model, training construction and seed for the original 512-residue experiment; the full 512/1024 comparison is Appendix Figure S16. In the 512-residue experiment, after averaging over six training constructions and three seeds within each model, model means range from 0.507 to 0.561 on Y2H-iso and 0.505 to 0.525 on IntAct. In the 1024-residue experiment, Frozen-ESM2 trained on Uniform reaches AUROC 0.691 on Gold Uniform but 0.527 on Vidal/Y2H. We state the weak absolute measured-source performance in the abstract and as a main finding, and we now present the 108/120 comparison only as a statement about training constructions, explicitly conditioned on these weak absolute levels. In the 1024-residue experiment several conditions fall below chance (down to 0.429 on Y2H-iso), but these results alone do not identify the cause, since eligible proteins, training data and negative candidates also change. The practical consequence is that rankings on constructed negatives need not transfer to the selected experimentally measured outcomes, which is why we keep assay non-detections, curated IntAct negatives and constructed BioGRID/STRING negatives separate throughout.

AuthorsOpenReview ↗

W2/Q1: Known-positive overlaps and close homologs

We checked all 49,164 training negatives (1024-residue experiment) under each of the six samplers for exact-sequence-pair matches to known positives in BioGRID, STRING and IntAct. The largest source-specific overlap is 151/49,164 negative rows (0.3071%). Appendix Table S3 reports the counts as negative rows.

ESM-matched has more known-positive collisions than Uniform in STRING and IntAct, whereas Bernett-ILP has the most in BioGRID and STRING (Table S3; a mapping-conditioned check on the 185,737 HuRI-screenable negative rows, 33.14% of 560,382 evaluated negatives, likewise found zero unexcluded HuRI positives). In our fixed score-independent sample of 2,000 ESM-matched negatives per experiment, excluding close replacement homologs (≥30% identity and ≥80% dual coverage; 166/1,988 evaluable pairs at 512 residues and 164/1,986 at 1024 residues) together with their matched positives shifted AUPRC by less than 0.006 across all four Uniform-trained models and three seeds. Neither check explains the near-chance scores: annotation collisions affect at most 0.31% of negative rows, and removing close replacement homologs shifts AUPRC by less than 0.006. What they cannot measure is the rate of true but unrecorded interactions among close embedding neighbours, which is why we treat ESM matching as a representation-coupled stress test rather than as verified non-interaction evidence (Secs. 6–7; Appendix C).

Reviewer_YR2DOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines different ways to construct negative (non-interacting) protein pair examples for training and testing sequence-based protein-protein interaction (PPI) prediction systems. Since we tend to have only positive training and testing data (i.e., those protein pairs experimentally demonstrated to interact), PPI prediction is a positive-unlabeled learning (PU Learning) problem, and negative data must be constructed. Six negative construction strategies are compared, and are applied in all possible combinations during the training and testing of four different PPI predictors across several datasets. Additionally, cross-dataset evaluation reinforces the findings, where two assay-based datasets, external to the original investigation, are used to confirm findings.

The study finds that none of the construction methods are particularly stronger than random pair selection, and that even matching the same strategy during training and testing is not particularly important. They further find that negative construction strategies that match positive and negative proteins in ESM-2 embedding space make the PPI prediction problem nearly impossible.

Critical strengths
  • Overall, this benchmarking of negative construction strategies is thorough, methodologically sound, and asks an interesting question. Where other studies would report performance on a single dataset, this study examines multiple datasets with varying properties to explore issues such as dataset difficulty, expected opportunities for short-cut learning, and pair dependence.
  • The authors include both complex negative construction strategies and simple baselines. They explicitly test for "shortcut learning" (e.g., where unintended systematic differences between positive and negative pairs emerge due to the construction strategy, leading to optimistically biased evaluations)
  • The authors include external datasets that include a small number of experimentally validated negatives.
  • The paper is well written and easy to follow.
Critical weaknesses
  • Interesting conclusions?: While the research question is interesting, the findings are somewhat uninteresting (no negative construction strategy is especially preferable), and there is a lack of actionable conclusions.
  • No clear path forward: The results do not clearly point to a new way of constructing negatives (other than avoiding matching negatives in ESM-2 space).
  • Expected impact: given the lack of actionable conclusions, it is unclear how the field of sequence-based PPI prediction will change as a result of this study. However, the summary of findings on lines 444-459 could guide future negative construction strategies, while the methodological advice given on lines 461-466 could guide future PPI predictor research.
  • Findings representative?: By only examining Gold positive pairs where both proteins are < 512 AAs, only 16.3% of the Gold dataset remains. This was done since the four sequence-based methods compared in this study cannot natively handle longer proteins. However, this apparently stringent threshold eliminates the vast majority of pairs. One must question whether the findings in this study generalize to all PPIs, when (apparently) most PPIs do not resemble the (short) protein pairs used in this study. The authors acknowledge this weakness, but have not explored the expected impact of this limitation (e.g., when relaxing the sequence length limit, do the findings change?)

Minor weaknesses

  • The Bernett-ILP and TPPNI-balanced negative construction methods could be described in greater detail for the reader (but references are certainly provided)
  • Several acronyms are used without definition (MCC, AUPRC, etc). These are commonly used, but they should be defined space-permitting.
Questions for authors

I encourage the authors to respond to the Weaknesses listed above. Additional questions follow:

  • It is surprising that the constructed training negatives fail to improve upon the "Uniform" strategy (where one protein in a positive pair is replaced with a random protein, so long as that does not form a known positive pair). I would like to see a yet-simpler baseline: negatives chosen as random pairs (where both proteins are chosen randomly, again so long as they do not represent a known positive pair). This is a reasonable strategy since most pairs should be negative but will likely lead to data leakage/short-cut learning since low-degree proteins will dominate the negatives. For this reason, we should expect all six negative (more complex) construction strategies to outperform such a baseline, no? This would serve as a control of sorts...
  • (Curiosity question only...) One of the two external evaluation datasets is Y2H-iso, where negatives are "non-detections". Presumably, the creators of this dataset only examined protein pairs that they had some suspicion of being PPIs (and not just randomly performing Y2H experiments on random pairs, when the vast majority of random pairs are expected to be negative. This makes me wonder if these test 'negatives' are representative of how a PPI prediction model will be used in actual practice... If these are "looks like a PPI" negatives, then this is a biased (difficult?) sample of all negatives.
  • Figure 2 makes it look like 'uniform' negative construction during training is highly competitive with the more complex strategies, regardless of which strategy is used during testing (perhaps except for using TPPNI in both train & test, but that strategy underperforms relative to "uniform" for other training strategies in most cases). Does this mean that "uniform" negative construction should be preferred since it is simplest?
10 comments
AuthorsOpenReview ↗

W1: Actionable conclusions

Thank you for requesting clearer practical conclusions and a simpler Fully Random control. We ran this control for all four models and revised the discussion. We also corrected an identifier-lookup error in the submitted shuffling analysis (Sec. 5, partner-shuffling results). Our point-by-point answers below rely on the updated results.

W1: Actionable conclusions. Our framing understated the result. We do not find that negative construction is inconsequential; we observe different effects of negative construction during training and evaluation, which cautions against assuming that a single sampler suits both. In our tested settings, constrained negatives provide useful evaluation stress tests, but they do not consistently improve external transfer when used for training. Three recommendations follow.

  • (1) Train on anchor-fixed Uniform and do not assume heuristic hard negatives transfer. Across three pLM-based models and 512-residue external evaluations, constructed training negatives underperform anchor-fixed Uniform in 108 of 120 comparisons (Fig. S5).
  • (2) Report performance on simple controls. Following Reviewer 8vim, who highlighted these two analyses, within-class partner shuffling and a low-capacity pooled-embedding readout have been moved to the main text (Sec. 5, Figs. 3 and 5). Shuffling probes dependence on partner identity: with the corrected within-class procedure, 1024-residue Uniform AUROC drops by 0.134 (Frozen-ESM2) and 0.182 (TUnA), suggesting that these models do exploit partner properties rather than only anchor-level properties. The pooled readout probes what architectural capacity adds: a logistic classifier on [u + v, |u − v|] of mean-pooled ESM-2 embeddings reaches 0.643 AUROC in the same setting (against 0.570 for the strictly additive u + v alone). This result indicates that a substantial part of the pair-dependent signal is already linearly accessible from mean-pooled embeddings, without residue-level interaction modelling. Reporting both addresses “does the model use the pair” and “does the architecture earn its complexity”.
  • (3) Report robustness to the matching representation. Matching negatives in a model’s own embedding space is the sharpest diagnostic in our panel: it drops all three pLM-based models to near-chance AUPRC, including models reaching 0.72 on uniform negatives. Near-chance performance here is consistent with sensitivity to global embedding compatibility, but does not isolate the effects of shortcuts, dataset geometry, and negative-label uncertainty — and this motivates future evaluations of biologically relevant candidate partners that are close in embedding space, such as paralogues with diverged interaction profiles, splice isoforms differing by an interaction domain, or missense variants that abolish binding. Because these pairs are constructed rather than verified non-interactions, this is a representation-coupled stress test rather than direct evidence of binding recognition.

Consistent with this, a frozen-embedding MLP is the best model in 16 of 30 settings (excluding the degenerate Degree-matched test), ahead of residue-level interaction architectures — a frozen-embedding baseline should therefore be reported before any claim that architectural complexity contributes.

AuthorsOpenReview ↗

W2: A path forward

We agree that our positioning with respect to prior work was insufficient, which obscured the path our results support. We have revised the manuscript to improve our positioning.

Park & Marcotte (2011) argue that train and test negatives must be decoupled, and that test negatives should be uniformly random such that evaluation matches the deployment distribution. Bernett and co-authors' works (2024, 2026) argue the opposite for evaluation: uniform negatives allow models to exploit degree and dataset geometry, and they report that a decade of apparent progress rests on this artefact. The disagreement bundles three questions that our design separates: (1) whether training and test construction should be decoupled, (2) which construction to train on, and (3) what an evaluation must control for. Our results come out differently on each.

  • (1) Our factorial design clearly shows that the diagonal is not always optimal. For most test constructions, the best-performing training negatives are not those built from the same construction. This result supports Park & Marcotte (2011) view.

  • (2) Constraining training negatives to avoid shortcut and leakage, as advocated by Bernett and co-authors, leads to collapsed diversity and degraded transfer, including when testing on constrained splits. Unbiased training remains the better baseline, now shown for pLM-based models.

  • (3) Our results corroborate Bernett and co-authors’ findings that a strong Uniform score is not evidence of molecular recognition, even when partner shuffling costs 0.134–0.182 AUROC: a logistic classifier on [u + v, |u − v|] of pooled ESM-2 embeddings already reaches 0.643, and performance falls to near chance on Y2H-iso and IntAct. Constrained negatives and our two controls are necessary as stress tests.

The path we propose is therefore: train on Uniform until verified experimental negatives are available, and report Uniform scores together with partner shuffling and the frozen-embedding readout. Longer term, progress is more likely to come from discrimination tasks where negatives are biologically grounded — paralogues with diverged partners, splice isoforms differing by an interaction domain, binding-abolishing variants — than from further heuristic constructions.

AuthorsOpenReview ↗

W3: Expected impact

We thank the reviewer for identifying where the practical contribution already sits. We agree that lines 444–459 and 461–466 carry it, and the revised version promotes it to the abstract, the end of the introduction, and the opening of the discussion. The concrete change we expect is a reporting standard. A sequence-based PPI predictor claiming improvement should show that it clears a linear readout on pooled embeddings (0.643 AUROC in our Uniform setting) and that its score depends on partner identity. Both controls are cheap, and we provide the code and the frozen protocol to run them (Sec. 6; Reproducibility Statement).

Reviewer_VX1xOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyses how the choice of negative examples affects the results of PPI models. The authors keep the positive pairs and the train/test split fixed, and try six different ways of building negatives for both training and testing, with four models. They find that scores and model rankings highly depend on how negatives are built.

Critical strengths

Clean design and important question.

Code available for reproducibility.

Thorough evaluation of the different methods including several baselines.

Critical weaknesses

No error bars. The authors could provide standard deviations to assess statistical significance, even though 3 seeds are not many.

Negative construction doubt. Each positive pair is (A, B), two proteins known to interact. To make a negative, the authors keep A and swap B for another protein C, giving (A, C), which is assumed not to interact. In the “ESM-matched” construction, C is chosen to be the protein whose ESM-2 representation (the protein language model’s numerical summary of a sequence) is closest to B’s. The ESM-based models see exactly those ESM-2 representations as input. So for every positive (A, B), there is a negative (A, C) where C looks almost identical to B from the model’s point of view. The inputs of the two examples are nearly the same, but the labels are opposite. A model that only sees these inputs has almost nothing to separate them with, so performance falls to near chance. In other words, the sampler was designed against the very representation the model uses, so it’s not surprising the models fail. Instead of matching negatives only by embedding similarity, I suggest the authors match them by how the model uses the embedding, for example by selecting candidates that are similar in input but produce different internal activations, to separate genuinely hard negatives from probable false negatives; Russo et al. (ICML 2026) use a related idea of analysing a model’s internal representations of known interactions in the context of PPI candidate ranking.

Only 16.3% of the training data is kept because of the 512-residue limit. The low scores of some models (RAPPPID, D-SCRIPT-ESM2) may come from this rather than from the architecture.

Questions for authors

Can you add confidence intervals or seed standard deviations to the main comparisons?

If you remove close homologs of the true partner when building ESM-matched negatives, do the results change?

Do the findings hold with the full training set rather than the 512-residue subset?

5 comments
AuthorsOpenReview ↗

W1/Q1: Uncertainty and error bars

Thank you for the questions about uncertainty, ESM-matched negatives, and the 512-residue limit. We ran the 1024-residue experiment and revised the paper. We also found an identifier-lookup error in the submitted partner-shuffling analysis. We withdraw the claim that our models barely use the original protein pair (Sec. 5, partner-shuffling results).

W1/Q1: Uncertainty and error bars. In the revised version, we show the results for seeds 11, 22, and 33 separately. For reference, the AUPRC mean ± SD in the 512-residue experiment with Uniform negatives used for both training and testing is 0.723 ± 0.003 (TUnA), 0.697 ± 0.005 (Frozen-ESM2), 0.587 ± 0.038 (D-SCRIPT), and 0.518 ± 0.004 (RAPPPID). These SDs are indicative only, as three seeds give a rough estimate of training variance. We independently recomputed all 144 paired bootstrap intervals comparing the 1024-residue and 512-residue experiments, using 2,000 stratified resamples of the same test pairs and averaging over the three fixed training seeds. Appendix Table S7 gives representative Uniform-trained/Uniform-test comparisons: TUnA changes by +0.0058 AUPRC, with a 95% test-row interval of [+0.0021, +0.0092]. The complete set of 144 intervals was checked against the authors' private audit evidence; selected intervals are reported in Appendix Table S7. Figure 4 shows descriptive mean ± sample SD for the original 512-residue measured-source results; the full 512/1024 comparison is in Appendix Figure S16. Appendix Tables S8 and S10 give selected paired effect-size intervals for shuffling and external transfer. Although these intervals do not capture variance due to training, they quantify the sensitivity of the results to the choice of the test pairs. For the Fully Random split, we resample pairs grouped by protein to account for shared proteins; since each pair involves two proteins, some dependence remains and hence, we report these intervals as descriptive rather than as formal significance tests (Secs. 4–5; Appendix).

AuthorsOpenReview ↗

W2/Q2: Does ESM matching make the task artificially hard?

W2: Negative construction. We agree, and have reframed this construction accordingly. The ESM-matched negative construction is now presented as an adversarial setting for the models that use ESM representations rather than a set of verified non-interactions. Moreover, nearest-neighbor matching concentrates negatives on a few “hub” proteins: in the 1024-residue experiment, the most frequent replacement appears in 172 test negatives (vs. 22 with Uniform), and 996 test-negative pairs are duplicates (vs. 129), which reduces the effective diversity of the negative set. We also report overlaps with known positives and the effect of removing close homologs. We now discuss Russo et al. (ICML 2026), which ranks PPI candidates for experimental testing using model-derived and complementary evidence, with evaluation across successive STRING releases. Candidate ranking and validating biological non-interactions are different tasks. Activation-guided negative construction is a possible extension, but we have not evaluated it and do not claim that it separates hard negatives from false negatives (Sec. 6).

Q2: Excluding close homologs. We selected 2,000 ESM-matched negatives per experiment using a fixed rule independent of model scores. We aligned each replacement with the original positive partner. At ≥30% identity and ≥80% coverage of both proteins, 166/1,988 (respectively 164/1,986 pairs) evaluable pairs met the threshold in the 512-residue (resp. 1024-residue) experiment. Removing these pairs and their matched positives left 1,822 negative-positive dyads (3,644 evaluated rows) in each experiment. AUPRC changed by less than 0.006 in absolute value across four Uniform-trained models and three seeds. Thus, discrimination changes only slightly in this sampled sensitivity analysis. This is not a rebuilt full test set or a retraining experiment, and it cannot estimate the true rate of mislabeled negatives (Sec. 7; Appendix).

AuthorsOpenReview ↗

W3/Q3: Longer proteins and more training data

We substantially expanded the scope of our analysis by training 72 new model instances with a 1024-residue limit. Retained Gold training positives increased by 3.70 fold, from 13,304/81,596 (16.30%) to 49,164/81,596 (60.25%). Performance remains sensitive to how negatives are chosen. The mean AUPRC change was +0.0065 (D-SCRIPT), +0.0002 (Frozen-ESM2), −0.0009 (TUnA), and −0.0026 (RAPPPID) across the 36 training/test settings on the common set of test pairs. The largest change was −0.1773 for D-SCRIPT trained and tested on TPPNI, consistent with the higher seed-to-seed variance we observe for D-SCRIPT (W1). Appendix Figure S15 reports the matched-test comparison. All 144 paired test-row bootstrap intervals have been independently recomputed; selected comparisons appear in Appendix Table S7. The 1024-residue experiment does not cover the longest proteins, and it varies the amount of training data rather than isolating sequence length as a factor (Sec. 5; Appendix Fig. S15 and Table S7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.