- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a new form of protein-protein interaction prediction that fuses sequence-based information (for each query protein) with network information (the pattern of known PPIs connecting the two query proteins). Specifically, it tackles the case where neither of the query proteins appears in the training PPI network (i.e., the "neither seen/NS" or "cold start" problem that many others have investigated). In that NS case, network-based features are unavailable for the test pair. Instead, they use sequence similarity (MMseqs2 matching score) to identify "anchor" (~homologous) proteins within the training PPI network, create virtual links from the test proteins to those anchor proteins and also to some internal nodes along short PPI paths connecting the anchor proteins. A "relation aware graph encoder" then encodes the proteins/nodes and the subgraph connecting the query, anchor, and internal nodes.
Experiments demonstrate increases in accuracy (AUPR) and throughput/efficiency on 5 non-human test PPI datasets (D-SCRIPT) when human data are used for training. Several methods are included as SOTA baselines, including three network-based methods that are modified to account for the NS case. Results are convincing.
Critical strengths
The authors spend considerable time establishing the case for adding surrogate network topology information in the NS use case, which motivates their method.
The paper is clearly written and easy to follow. The method is explained well, but some details feel unspoken (see below).
The comparison with the SOTA is convincing and the selection of baseline methods appears to cover the SOTA.
Critical weaknesses
The authors equate cross-species PPI prediction (where the training and test data arise from different species) with the neither-seen (NS) use case. I would argue that this is not representative of interesting NS cases... Since so many protein complexes and key protein pathways are preserved between species, we expect good matches for many non-human (test) protein pairs within the human (training) PPI network (e.g., interlogs). So, we can expect that many of the NS non-human test pairs are likely represented by direct equivalent pairs in the human training data. To me, the more interesting case is where there is no direct homolog available for the test sequences (e.g., engineering a novel peptide in virgin sequence space). Cross-species PPI prediction is a well-established use case and does not feel equivalent to the NS use case...
Several hyperparameters are required in the proposed method. For example, Section 5.2.2 includes "L_path" and "B", while Section 5.2.3 mentions adding virtual edges between the query and internal nodes when their matching score exceeds "a fixed threshold" (an unnamed hyperparameter?). How are these chosen/optimized? Is a sensitivity analysis completed for these hyperparameters?
In Section 4.2, the authors show that train-test sequence similarity does not explain PPI prediction accuracy, whereas others (Bernett 2024 and others) have shown the opposite: that sequence similarity between training and testing proteins explains much of the performance of PPI predictors. How do the authors explain the difference in findings here?
Similarly, Bernett and others have shown that node degree explains much of the remainder of PPI prediction accuracy (where a node with high degree in the training data would be expected to also participate in test PPIs, when compared with a training protein of low degree). Using the authors' approach, wouldn't a hub protein in a non-human test set, when found to be sequence-similar to a human hub protein in the training set, thus be predicted to interact with many test proteins? Isn't this a form of short-cut learning and data leakage?
Line 340 says "with a relation-aware graph encoder". What is its structure? Is it trained from scratch on the human training data? Is it entirely frozen during non-human tests?
Minor: the tone of the early sections of the paper makes it feel like the authors are discovering/demonstrating the challenge of the NS use case for the first time (e.g., Introduction and Section 4.1). The challenge associated with the NS use case was described at least as early as 2012 by Park & Marcotte (in which they defined the C1/C2/C3 test classes, where C3 pairs share neither protein with the training set, so no known interactions of either test protein can be used.) More recently, in the "protein disjoint" Gold PPI dataset (Bernett 2024) test proteins are excluded from the training dataset to avoid such data leakage.
Questions for authors
I encourage the authors to respond to the Weaknesses listed above.
8 comments
Response: Cross-species NS evaluation and direct human PPI templates
We thank the reviewer for raising this question. Our cross-species evaluation satisfies the NS split: neither protein in a test pair appears in the training set. At inference, PPI information is restricted to the human training network; no known interactions, non-interaction labels, or network neighbors of the test-species proteins are provided to the model. NS does not, however, imply an absence of sequence similarity. We therefore asked two questions: does ERCL retain an advantage on weakly matched pairs, and does it retain an advantage on pairs without detected direct human PPI templates?
Sequence-matching strata. We compared test proteins against all 15,816 human training proteins using CD-HIT-2D, requiring ≥40% identity and ≥80% coverage of both sequences. Pairs with no qualifying hit for either endpoint account for 94.8%, 76.3%, 56.0%, and 73.1% of the E. coli, yeast, fly, and worm tests, respectively. These test sets thus contain substantial subsets without qualifying matches, rather than only highly matched pairs. CD-HIT is used only for this stratification; “no qualifying hit” does not mean that ERCL has no anchor.
Within each stratum, ERCL and our internal sequence-only control are evaluated on identical test pairs. Their AUPRs below are three-seed means; differences are computed before rounding.
| Species | Qualifying hit | Pairs (positives) | ERCL | Sequence-only | ΔAUPR |
|---|---|---|---|---|---|
| E. coli | Neither endpoint | 20,858 (1,739) | 0.745 | 0.736 | +0.009 |
| E. coli | ≥1 endpoint | 1,142 (261) | 0.894 | 0.851 | +0.042 |
| Mouse | Neither endpoint | 1,139 (8) | — | — | — |
| Mouse | ≥1 endpoint | 53,861 (4,992) | 0.931 | 0.887 | +0.044 |
| Yeast | Neither endpoint | 41,969 (2,285) | 0.620 | 0.594 | +0.026 |
| Yeast | ≥1 endpoint | 13,031 (2,715) | 0.833 | 0.806 | +0.028 |
| Fly | Neither endpoint | 30,777 (946) | 0.870 | 0.767 | +0.103 |
| Fly | ≥1 endpoint | 24,223 (4,054) | 0.956 | 0.916 | +0.041 |
| Worm | Neither endpoint | 40,178 (1,321) | 0.768 | 0.680 | +0.088 |
| Worm | ≥1 endpoint | 14,822 (3,679) | 0.959 | 0.933 | +0.026 |
Note: — denotes fewer than 100 positives; AUPR is not reported. “No qualifying hit” means no match detected under the stated identity and coverage criteria, not an absence of sequence similarity or homology.
ERCL outperforms sequence-only in all four evaluable no-hit groups. In fly and worm, the gains are +0.103 and +0.088, compared with +0.041 and +0.026 in their respective groups with at least one qualifying hit.
Direct human PPI templates. To address the reviewer’s concern about interolog-based transfer, we used two complementary checks for direct human PPI templates: (1) a training PPI edge between the two sets of selected Top-5 anchors; and (2) a training PPI edge connecting a human protein matched to one query with a human protein matched to the other, considering all qualifying matches rather than only the selected anchors. Across species, 90.2%–94.8% of test pairs have no direct PPI edge detected by either check. We evaluated all three models on these same pairs:
| Species | Pairs (positives) | Share of species’ positives | ERCL | Sequence-only | PLM-Interact |
|---|---|---|---|---|---|
| E. coli | 20,862 (1,422) | 71.1% | 0.649 | 0.660 | 0.630 |
| Mouse | 49,924 (1,027) | 20.5% | 0.633 | 0.567 | 0.572 |
| Yeast | 50,849 (2,609) | 52.2% | 0.474 | 0.467 | 0.427 |
| Fly | 49,628 (1,012) | 20.2% | 0.708 | 0.613 | 0.649 |
| Worm | 49,946 (1,374) | 27.5% | 0.634 | 0.592 | 0.580 |
Note: The second check requires ≥40% sequence identity and ≥80% coverage of both query and target. PLM-Interact uses its existing frozen per-sample predictions.
On these pairs, ERCL outperforms PLM-Interact in all five species and sequence-only in four. ERCL’s predictive advantage is therefore not confined to pairs with a detected direct human PPI template.
As supporting evidence, we counted three-hop PPI paths between the two anchor sets in the context graphs of these same pairs: using path count alone to distinguish positive from negative pairs yields ROC-AUCs of 0.596, 0.862, 0.692, 0.886, and 0.810 for E. coli, mouse, yeast, fly, and worm, respectively. These results show that multi-hop PPI connectivity between the two anchor sets is informative for interaction prediction, even when no direct human PPI template is detected.
Together, these analyses show that ERCL’s predictive gains extend to weakly matched pairs and to pairs without detected direct human PPI templates.
Our cross-species benchmark evaluates one NS setting; de novo designed peptides without homologous counterparts were not directly evaluated in this study.
Response: Sequence proximity and the findings of Bernett et al
We thank the reviewer for highlighting this point. We agree that sequence similarity between training and test proteins affects PPI prediction performance. Section 4.2 does not reject this relationship; it explicitly notes that sequence proximity broadly aligns with AUPR across the five test sets. Our narrower observation is that the measured test-set-level sequence proximity does not fully account for the differences in PLM-Interact’s reported performance across these species.
Bernett et al. changed the training/test partitions to examine how existing models perform under stricter generalization conditions. They partitioned each of the human and yeast proteomes into two non-overlapping subsets while minimizing overall sequence similarity between the subsets. When trained on PPIs within one subset and tested on PPIs within the other, the tested models performed at or near chance level. Their findings highlight the importance of protein overlap, sequence similarity, and node-degree information that models can exploit under conventional evaluation settings. We agree with these findings. Bernett et al., 2024
Section 4.2 asks a different, concrete question: with the human training set and five non-human NS test sets fixed, are the species whose proteins more closely resemble the training proteins also those with higher prediction performance? We compare each test set’s sequence proximity, summarized by the median protein-level score, with PLM-Interact’s reported AUPR. For example, worm and yeast have similar sequence-proximity values, but their AUPRs are 0.888 and 0.706, respectively. Mouse has substantially higher sequence proximity than fly, yet its AUPR is not higher: 0.904 versus 0.913. Thus, across these five test sets, the relative sequence-proximity values alone do not adequately explain why prediction performance differs between species.
These observations are compatible with Bernett et al.’s findings: sequence similarity can strongly affect performance without a single test-set-level proximity measure fully explaining cross-species performance differences. This motivated us to investigate whether PPI relational context provides complementary predictive information, with its contribution examined through the subsequent topology diagnostics and ablation experiments. Our aim is to extend the understanding of NS prediction difficulties, not to discount the importance of sequence similarity.
Response to the minor comment: NS definition and prior work
We thank the reviewer for pointing out prior work on the NS setting. At the level of train/test partitioning, NS means that neither protein in a test pair appears in any training pair. We agree that this criterion is not new: it corresponds to the C3 class of Park and Marcotte (2012) and is also reflected in the protein-disjoint evaluation of Bernett et al. (2024). We will tighten wording that might suggest we were the first to identify this split or its associated challenge, and acknowledge these studies more explicitly.
While we adopt this established split-level definition of NS, the setting studied here also specifies what PPI information is available at inference: neither query protein has observed PPI edges in the reference network accessible to the model. A protein-disjoint training split alone does not guarantee this, because a separate reference network may still contain test proteins and their known neighbors. For example, the original GNN-PPI reports an NS test subset but predicts the types of known interactions; in its original graph-based evaluation, test-pair edges can enter the graph structure while their interaction-type labels remain withheld. LLaPA removes test edges during training and inference, but uses a reference PPI network assembled from multiple sources. A test protein absent from the training pairs may therefore still have other known PPI neighbors in that network. Thus, methods using an NS split can differ in the network information available at inference.
In our cross-species evaluation, the only observed PPI edges available to the predictor come from the human training network. Target-species proteins are absent from that reference network. At inference, the model is given neither observed PPI edges incident to either query protein nor any positive or negative pair labels involving them; test-pair labels are used to compute evaluation metrics, not as model inputs. Their sequences are used to retrieve anchors in the human network, while their precomputed sequence representations serve as node features. The query–anchor links are virtual similarity edges rather than observed PPIs. ERCL builds relational context from PPI relationships within the human training network. Our contribution is therefore not a new NS split, but a way to construct relational context under this explicit information constraint.