- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OMP, a protein interaction language model based on ESM-2, which introduces Siamese cross-attention modules at every transformer layer. The method processes two protein sequences separately and supports order-invariant interaction modeling. The model is further pretrained on interacting protein pairs from STRING and evaluated on PPI prediction, TCR-epitope binding, and interface prediction. The results show some improvements over ESM-2 and other baselines, mainly on the PPI prediction task.
Critical strengths
-
The problem is meaningful. Modeling protein interactions directly during PLM pretraining is an interesting direction, and handling long protein sequences and input order is useful.
-
The proposed architecture is reasonable. The Siamese design provides order invariance by construction, and separating intra- and inter-protein attention is a natural way to model protein pairs.
-
The paper evaluates several downstream tasks and provides relatively detailed analysis. I also appreciate that the authors discuss some limitations of their results and release the pretrained model.
Critical weaknesses
-
The technical novelty seems quite limited, and the main contribution is not well validated. The architecture mainly adds standard cross-attention and gating modules to ESM-2, which is not a particularly new idea. More importantly, the paper does not provide proper ablations to show that the proposed layer-wise cross-attention is actually responsible for the improvements. Compared with ESM-2, OMP has about 325M additional parameters and is further pretrained on STRING interaction pairs. So it is unclear whether the gains come from the proposed architecture, additional pretraining, or simply larger model capacity. I think this is a fundamental missing experiment for the paper.
-
The experimental improvements are relatively small and not consistent across tasks. On the gold-standard PPI benchmark, the accuracy only improves from 64.9% to 66.5%, and AUPRC from 0.701 to 0.712. For TCR-epitope prediction, the mean AUROC improvement is only 0.008, and is significant in only two of the five splits. On interface prediction, OMP is even worse than the original ESM-2. Considering the additional 325M parameters and 4,032 GPU-hours of pretraining, I don't think the current results are strong enough to justify the proposed method.
-
I have concerns about the potential pretraining data overlap with the PPI benchmark. The model is pretrained using interacting protein pairs from STRING, while some positive pairs in the gold-standard PPI benchmark may also appear in STRING. Although the paper discusses this issue, the actual overlap between the pairs used for pretraining and the test pairs is not measured. The reported 9.3% is based on the broader curated STRING links involving test/validation protein families, rather than the final pretraining subset. This is different from ESM-2, which only has single-sequence pretraining. Given the relatively small performance improvement, I think this issue needs to be carefully checked before claiming that OMP breaks the existing performance plateau.
-
The claimed architectural advantages are not sufficiently connected to the experimental gains. For example, the longer context window is a reasonable advantage over concatenation-based models, but the main ESM-2 baseline already supports the same per-chain length. There is no length-wise analysis showing that OMP benefits more from longer proteins. Also, the order invariance seems largely achievable through a symmetric prediction head. In fact, Appendix E shows that MINT's order sensitivity mainly comes from its prediction head rather than the encoder. I am therefore not convinced that these advantages require adding cross-attention to every layer. The interpretability results are also not very convincing as an independent contribution, since similar structural knowledge already exists in ESM-2.
Questions for authors
-
Can the authors provide an ablation that separates the contribution of layer-wise cross-attention from additional STRING pretraining? For example, continued MLM pretraining of ESM-2 on the same data, or a simpler cross-attention adaptation. I think this comparison is necessary to justify the proposed architecture.
-
Can the authors check the actual overlap between the 18.4M pretraining pairs and the downstream PPI test pairs? It would be useful to report the performance separately on test pairs without pretraining exposure, including closely related interaction pairs.
-
Can the authors provide more analysis of where OMP actually improves over ESM-2? For example, performance grouped by protein sequence length, and the effect of disabling the pretrained cross-attention modules. This would help understand whether the proposed architectural changes really contribute to the improvements.
2 comments
Thank you for the rebuttal and additional experiments. The new overlap analysis addresses my concern regarding data leakage. However, the ablations still don't demonstrate the contribution of the proposed cross-attention architecture, and the reported improvement is comparable to the variation across head-training seeds. Therefore, I will keep my original score.
Answer to reviewer
W1 and Q1. Ablation
We ran both suggested ablations twice (Tables 9 and 10). ESM-2 with continued MLM reaches 0.7200 and 0.7046 AUROC, and the difference between the GLU and the scalar gate is 0.0111 in the first run and 0.0020 in the second one. Hence, the ablations do not resolve the source of the gain, and we revised our claims (lines 337–345).
W2. Size of the improvements
We agree that the improvements are modest, and ESM-2 remains ahead on interface prediction. Appendix K reports the variation between repeated trainings of the head and its consequence for the comparison with ESM-2.
W3 and Q2. Overlap with the test pairs
The released checkpoint saw 128 of the 26,024 positive test pairs (0.49%) and no negative pair (Table 6). Counting closely related pairs (i.e., 50% identity families), it saw 207 positive pairs (0.80%). On the pairs without exposure, the difference with ESM-2 is unchanged (AUROC CI [0.0120, 0.0199], Table 7).
W4. Architectural advantages
We agree. OMP does not benefit more from longer proteins relative to ESM-2 (Table 12), and a symmetric head suffices for order invariance. We corrected both claims (lines 58–60 and 347–350). Regarding interpretability, cross-attention heads specialize by amino acid with a low correlation with BLOSUM62 (Spearman 0.22, against 0.76 for self-attention), so they encode a signal that is not inherited from ESM-2 (Table 14). In contrast, cross-attention does not localize interface residues in a systematic manner (mean AUROC over heads of 0.483), and we report it in Appendix N.
Q3. Where OMP improves
The lead over ESM-2 is largest between 511 and 1,022 residues (Table 12). Disabling the cross-attention modules changes the gold-standard AUROC by 0.0005 (CI [-0.0017, 0.0028], Table 10). Thus, the contribution of the pathway to this benchmark is not resolved, and we state this limitation (line 499).