- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+2)
Read review
Paper summary
The paper asks how the way negative examples are constructed affects the evaluation of sequence-based PPI models. Experimentally confirmed non-interactions are rare, so PPI datasets usually treat unobserved protein pairs as negatives. Starting from the leakage-free human Gold benchmark of Bernett et al. (2024), the authors keep the positives, the protein-disjoint split, class balance and model selection fixed, and change only the negatives. They use six constructions: uniform random, degree-matched, sequence-matched, ESM-2-matched, the bias-reducing ILP of Bernett et al. (2026), and topology-driven TPPNI. Training and test negatives are crossed in a 6 × 6 design for four models (a frozen ESM-2 MLP, TUnA, D-SCRIPT-ESM2 and RAPPPID) with three seeds each. The models are also evaluated on external data without retraining: Y2H assay outcomes, curated IntAct negatives, and BioGRID/STRING positives paired with constructed negatives. They find that changing only the test negatives shifts AUPRC by up to 0.26, and that the best-ranked model changes with the construction. They also observe that ESM-matched negatives push all ESM-based models to near chance, and that shuffling partners within each class barely changes AUROC, so the models make little use of the specific protein pair. Finally, constructed training negatives rarely beat uniform sampling on external data. The authors conclude that negative construction should be reported and varied as an evaluation axis, not treated as a preprocessing detail.
Critical strengths
-
The design is clean and well controlled. Positives, splits, class balance, checkpoint selection and thresholds are held fixed, so differences in performance can be attributed to the negatives. I find the crossing of training and test negatives the right way to separate the two effects.
-
The partner-shuffle control is the most useful contribution. It directly tests whether a model needs the original A–B pairing, and the answer (it barely does) is important for anyone who reads high benchmark scores as evidence of learned interaction biology.
-
The authors are careful about what their labels mean. They consistently separate assay non-detections, curated negatives and constructed unobserved pairs, they avoid calling constructed negatives "non-interactions", and they distinguish cross-source transfer from true out-of-distribution generalisation. Most PPI papers are not this careful.
-
The evaluation goes beyond the constructed benchmark. Testing on Y2H outcomes and curated IntAct negatives, even though both are small, is the right direction and rarely done.
-
The practical recommendations at the end are sensible and easy to adopt: report the candidate set, test at least one alternative construction (including one matched in the model's own representation space), include controls that don't need the original pair, and evaluate on assay data where possible.
-
The data support a stronger conclusion than the paper draws. In the released figure data, a simple logistic regression on pooled single-protein ESM embeddings performs about as well as the full PPI models. Performance also drops from about 0.70 AUPRC on the constructed Gold test to roughly chance against experimental negatives. I would encourage the authors to put both findings in the main text.
Critical weaknesses
-
The external results hide that performance on experimental negatives is close to chance. External transfer is reported only as differences from Uniform training. In the released figure data, absolute AUROC averaged over training constructions is 0.51-0.56 on Y2H and 0.51-0.52 on IntAct; the best single condition is 0.62, and several are below 0.5. The headline "constructed training negatives underperform Uniform in 108 of 120 comparisons" therefore mostly compares near-chance numbers, which makes it misleading on its own. It also obscures what I consider one of the paper's most important results: about 0.70 AUPRC on the constructed Gold test versus close to chance on experimentally tested negatives. Request: report absolute external performance in the main text and state this gap explicitly.
-
Label noise in the constructed negatives is not measured, so the hardness of some constructions cannot be interpreted. The authors suggest that matched negatives, especially ESM-matched ones, may contain many real but unrecorded interactions, because close homologs of a partner often bind the same protein. If so, the near-chance results on these sets may reflect mislabelling rather than difficulty, and this is never checked. In the code, training negatives are filtered only against Gold training positives, and the collision count in the sampler summary is hard-coded to zero rather than computed. Request: for each construction, report the fraction of negatives that appear as positives in BioGRID, STRING, IntAct or HuRI.
-
The quantitative summaries overstate the strength of the evidence. With three seeds, seed-to-seed spread in external AUROC reaches 0.12, so many of the 108/120 comparisons are within noise. They are also not independent, since they share models and test sets. The degree-matched test set reduces to a random draw on a protein-disjoint split (all test proteins have zero training degree), and its scores are nearly identical to Uniform (e.g. 0.633 vs 0.631 for Frozen-ESM2), yet it is counted in summaries such as "best in 20 of 36 settings". The external sets are also described too loosely. The Y2H set is mainly a transcription-factor isoform screen, not a random sample of human pairs, and results are not separated for proteins that overlap with Gold training and proteins that do not. Request: replace win/loss counts with effect sizes and confidence intervals, exclude the degree-matched test set from summary counts, describe the Y2H set accurately (and check that the cited source is the study the data came from), and report external results separately for overlapping and new proteins.
-
The contribution is not clearly positioned against prior work, and the main new result is underdeveloped. Park and Marcotte (2011) already showed, with positives fixed, that changing only the test negatives shifts AUC substantially. Reim et al. (2025) showed on the same benchmark that ESM-2 embeddings, not architecture, explain the performance plateau. The paper does not engage with Park and Marcotte's central argument: that uniform random test negatives are the unbiased choice, and that a protein's general tendency to interact is a legitimate signal. That argument bears directly on how the near-zero pair dependence should be read. The shuffle analysis, which I consider the clearest new contribution, gets one paragraph. On the sequence- and ESM-matched sets it mainly reflects how the test set was built. Evidence that supports it is left out of the main text: a simple logistic regression on pooled single-protein ESM embeddings performs about as well as the full PPI models. The conclusion that models do not use the specific pair is also stated more broadly than tested, since only ESM-2-based sequence models were evaluated. Request: state what the paper adds beyond these works, discuss Park and Marcotte's argument, expand the shuffle analysis (including the pooled-ESM comparison), and limit the pair-dependence conclusion to the model class tested.
-
The iteration history behind the "fixed in advance" claim is not disclosed. The paper states that the models "were fixed before final-test inspection, so no architecture was added post hoc". The repository shows that the reported experiment is the third iteration, that half of the 432 result cells are reused from an earlier version whose test results had already been examined, and that a fifth model (SENSE-PPI) was dropped between versions. The stated reason is logistical and plausible, and I do not think the results are necessarily biased, but readers cannot judge this from the paper. Request: describe the earlier iterations, the reused cells, and the removal of SENSE-PPI.
-
The released materials do not match the release claims. The abstract promises released splits, negative-generation procedures and evaluation code. The anonymised repository contains code and per-figure tables, but no negative sets, splits or result registries, and most artifacts cited in its own documentation are missing. Its own audit also notes that the results were produced from code with uncommitted changes that cannot be reconstructed. Since a reusable evaluation resource is one of the stated contributions, the paper does not currently deliver it. Request: release the negative sets, splits and result registries.
Questions for authors
-
What fraction of each negative construction consists of pairs reported as interacting in BioGRID, STRING, IntAct or HuRI? In particular, is this fraction substantially higher for the ESM-matched and sequence-matched sets? If so, how much of the near-chance performance on those sets can be attributed to mislabelled negatives rather than to genuine difficulty?
-
How do you interpret the near-zero pair dependence in light of Park and Marcotte (2011), who argue that a protein's general tendency to interact is a legitimate predictive signal and that uniform random test negatives are the unbiased choice? Do you view the per-protein signal the models rely on as a shortcut, as partly legitimate, or as dependent on the intended use of the model?
-
Did results from the earlier iterations (E001v3 and E002) influence any design decision in the final experiment, such as the choice of constructions, metrics, checkpoint selection, or the removal of SENSE-PPI? Will the negative sets, splits and result registries be publicly released?
12 comments
W1: Absolute performance on external data
We agree, and we now treat this gap as a result rather than a reporting detail. Figure 4 reports absolute AUROC per model, training construction and seed for the original 512-residue experiment; the full 512/1024 comparison is Appendix Figure S16. In the 512-residue experiment, after averaging over six training constructions and three seeds within each model, model means range from 0.507 to 0.561 on Y2H-iso and 0.505 to 0.525 on IntAct. In the 1024-residue experiment, Frozen-ESM2 trained on Uniform reaches AUROC 0.691 on Gold Uniform but 0.527 on Vidal/Y2H. We state the weak absolute measured-source performance in the abstract and as a main finding, and we now present the 108/120 comparison only as a statement about training constructions, explicitly conditioned on these weak absolute levels. In the 1024-residue experiment several conditions fall below chance (down to 0.429 on Y2H-iso), but these results alone do not identify the cause, since eligible proteins, training data and negative candidates also change. The practical consequence is that rankings on constructed negatives need not transfer to the selected experimentally measured outcomes, which is why we keep assay non-detections, curated IntAct negatives and constructed BioGRID/STRING negatives separate throughout.
W2/Q1: Known-positive overlaps and close homologs
We checked all 49,164 training negatives (1024-residue experiment) under each of the six samplers for exact-sequence-pair matches to known positives in BioGRID, STRING and IntAct. The largest source-specific overlap is 151/49,164 negative rows (0.3071%). Appendix Table S3 reports the counts as negative rows.
ESM-matched has more known-positive collisions than Uniform in STRING and IntAct, whereas Bernett-ILP has the most in BioGRID and STRING (Table S3; a mapping-conditioned check on the 185,737 HuRI-screenable negative rows, 33.14% of 560,382 evaluated negatives, likewise found zero unexcluded HuRI positives). In our fixed score-independent sample of 2,000 ESM-matched negatives per experiment, excluding close replacement homologs (≥30% identity and ≥80% dual coverage; 166/1,988 evaluable pairs at 512 residues and 164/1,986 at 1024 residues) together with their matched positives shifted AUPRC by less than 0.006 across all four Uniform-trained models and three seeds. Neither check explains the near-chance scores: annotation collisions affect at most 0.31% of negative rows, and removing close replacement homologs shifts AUPRC by less than 0.006. What they cannot measure is the rate of true but unrecorded interactions among close embedding neighbours, which is why we treat ESM matching as a representation-coupled stress test rather than as verified non-interaction evidence (Secs. 6–7; Appendix C).
Major correction: Partner shuffling
Thank you for the detailed review. It highlighted the interpretation of pair dependence, weak performance hidden by relative comparisons, and gaps in the original code release. Our subsequent checks uncovered the shuffling implementation error. We corrected the errors we could verify and now state the remaining limits directly.
Major correction: Partner shuffling. We withdraw the submitted conclusion that the models barely use the original protein pair. The shuffled rows carried a new partner sequence but kept identifiers from the old partner. As a consequence, the lookup could score the wrong sequence. We fixed this and added tests across all four model input paths. In our 1024-residue experiment with Uniform negatives in both training and testing, AUROC drops by 0.134 (Frozen-ESM2) and 0.182 (TUnA) upon shuffling. On ESM-matched test negatives the effects are smaller, 0.020 and 0.026 respectively. We report the 512-residue experiment separately because its corrected analysis is not an exact replay of the historical execution. Figure 3 shows individual seeds and their means; Appendix Table S8 gives the corresponding selected paired test-row intervals (Sec. 5, partner-shuffling results).