acceptodds
Under review as a conference paper at ICLR 2027

OMP: Order-Invariant Protein Interaction Language Modeling with Layer-wise Siamese Cross-Attention

Abstract

Protein language models (PLMs) adapted to protein–protein interactions (PPIs) typically concatenate the two protein input sequences, which forces aggressive cropping, makes predictions depend on input order, and requires every residue to split a single attention budget between its own chain and its partner. We introduce OMP, a dual encoder that extends ESM-2 (650M) with Siamese cross-attention modules running in parallel to self-attention in every layer, each followed by a gated linear unit that is initialized nearly closed. OMP processes up to 1,022 residues per chain, twice the per-chain context of concatenation-based models built on the same backbone, and is order-invariant by construction. We pre-train OMP with masked language modeling on interacting sequence pairs from STRING and evaluate its frozen embeddings on three downstream tasks. Pre-training takes 4,032 GPU-hours over a fifth of the curated corpus, so adding pair-level structure to a pre-trained PLM does not require re-pretraining at the scale of the backbone. On the gold-standard PPI benchmark, OMP reaches 66.5% accuracy and 0.712 AUPRC, surpassing the 65% accuracy plateau of sequence-based models on this leakage-controlled benchmark and significantly outperforming ESM-2 with an identical prediction head. Among the models evaluated here, only OMP exceeds 0.60 mean AUROC on TCR–epitope binding prediction, and it outperforms the interaction language model MINT on interface prediction, where the single-sequence ESM-2 baseline remains strongest. Without structural supervision, OMP's attention maps recover protein domains: 86% of the strongest intra-protein energetic contacts in a target protein fall within the top 15% of layer-30 self-attention. Anonymized code and pre-trained weights are available at https://anonymous.4open.science/r/OMP and https://huggingface.co/OMP-ICLR/OMP.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein language models, protein–protein interaction, cross-attention, order invariance, ESM-2, TCR–epitope binding, interface prediction, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vXZhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OMP, a protein interaction language model based on ESM-2, which introduces Siamese cross-attention modules at every transformer layer. The method processes two protein sequences separately and supports order-invariant interaction modeling. The model is further pretrained on interacting protein pairs from STRING and evaluated on PPI prediction, TCR-epitope binding, and interface prediction. The results show some improvements over ESM-2 and other baselines, mainly on the PPI prediction task.

Critical strengths
  1. The problem is meaningful. Modeling protein interactions directly during PLM pretraining is an interesting direction, and handling long protein sequences and input order is useful.

  2. The proposed architecture is reasonable. The Siamese design provides order invariance by construction, and separating intra- and inter-protein attention is a natural way to model protein pairs.

  3. The paper evaluates several downstream tasks and provides relatively detailed analysis. I also appreciate that the authors discuss some limitations of their results and release the pretrained model.

Critical weaknesses
  1. The technical novelty seems quite limited, and the main contribution is not well validated. The architecture mainly adds standard cross-attention and gating modules to ESM-2, which is not a particularly new idea. More importantly, the paper does not provide proper ablations to show that the proposed layer-wise cross-attention is actually responsible for the improvements. Compared with ESM-2, OMP has about 325M additional parameters and is further pretrained on STRING interaction pairs. So it is unclear whether the gains come from the proposed architecture, additional pretraining, or simply larger model capacity. I think this is a fundamental missing experiment for the paper.

  2. The experimental improvements are relatively small and not consistent across tasks. On the gold-standard PPI benchmark, the accuracy only improves from 64.9% to 66.5%, and AUPRC from 0.701 to 0.712. For TCR-epitope prediction, the mean AUROC improvement is only 0.008, and is significant in only two of the five splits. On interface prediction, OMP is even worse than the original ESM-2. Considering the additional 325M parameters and 4,032 GPU-hours of pretraining, I don't think the current results are strong enough to justify the proposed method.

  3. I have concerns about the potential pretraining data overlap with the PPI benchmark. The model is pretrained using interacting protein pairs from STRING, while some positive pairs in the gold-standard PPI benchmark may also appear in STRING. Although the paper discusses this issue, the actual overlap between the pairs used for pretraining and the test pairs is not measured. The reported 9.3% is based on the broader curated STRING links involving test/validation protein families, rather than the final pretraining subset. This is different from ESM-2, which only has single-sequence pretraining. Given the relatively small performance improvement, I think this issue needs to be carefully checked before claiming that OMP breaks the existing performance plateau.

  4. The claimed architectural advantages are not sufficiently connected to the experimental gains. For example, the longer context window is a reasonable advantage over concatenation-based models, but the main ESM-2 baseline already supports the same per-chain length. There is no length-wise analysis showing that OMP benefits more from longer proteins. Also, the order invariance seems largely achievable through a symmetric prediction head. In fact, Appendix E shows that MINT's order sensitivity mainly comes from its prediction head rather than the encoder. I am therefore not convinced that these advantages require adding cross-attention to every layer. The interpretability results are also not very convincing as an independent contribution, since similar structural knowledge already exists in ESM-2.

Questions for authors
  1. Can the authors provide an ablation that separates the contribution of layer-wise cross-attention from additional STRING pretraining? For example, continued MLM pretraining of ESM-2 on the same data, or a simpler cross-attention adaptation. I think this comparison is necessary to justify the proposed architecture.

  2. Can the authors check the actual overlap between the 18.4M pretraining pairs and the downstream PPI test pairs? It would be useful to report the performance separately on test pairs without pretraining exposure, including closely related interaction pairs.

  3. Can the authors provide more analysis of where OMP actually improves over ESM-2? For example, performance grouped by protein sequence length, and the effect of disabling the pretrained cross-attention modules. This would help understand whether the proposed architectural changes really contribute to the improvements.

2 comments
AuthorsOpenReview ↗

Answer to reviewer

W1 and Q1. Ablation

We ran both suggested ablations twice (Tables 9 and 10). ESM-2 with continued MLM reaches 0.7200 and 0.7046 AUROC, and the difference between the GLU and the scalar gate is 0.0111 in the first run and 0.0020 in the second one. Hence, the ablations do not resolve the source of the gain, and we revised our claims (lines 337–345).

W2. Size of the improvements

We agree that the improvements are modest, and ESM-2 remains ahead on interface prediction. Appendix K reports the variation between repeated trainings of the head and its consequence for the comparison with ESM-2.

W3 and Q2. Overlap with the test pairs

The released checkpoint saw 128 of the 26,024 positive test pairs (0.49%) and no negative pair (Table 6). Counting closely related pairs (i.e., 50% identity families), it saw 207 positive pairs (0.80%). On the pairs without exposure, the difference with ESM-2 is unchanged (AUROC CI [0.0120, 0.0199], Table 7).

W4. Architectural advantages

We agree. OMP does not benefit more from longer proteins relative to ESM-2 (Table 12), and a symmetric head suffices for order invariance. We corrected both claims (lines 58–60 and 347–350). Regarding interpretability, cross-attention heads specialize by amino acid with a low correlation with BLOSUM62 (Spearman 0.22, against 0.76 for self-attention), so they encode a signal that is not inherited from ESM-2 (Table 14). In contrast, cross-attention does not localize interface residues in a systematic manner (mean AUROC over heads of 0.483), and we report it in Appendix N.

Q3. Where OMP improves

The lead over ESM-2 is largest between 511 and 1,022 residues (Table 12). Disabling the cross-attention modules changes the gold-standard AUROC by 0.0005 (CI [-0.0017, 0.0028], Table 10). Thus, the contribution of the pathway to this benchmark is not resolved, and we state this limitation (line 499).

Thank you for the rebuttal and additional experiments. The new overlap analysis addresses my concern regarding data leakage. However, the ablations still don't demonstrate the contribution of the proposed cross-attention architecture, and the reported improvement is comparable to the variation across head-training seeds. Therefore, I will keep my original score.

Reviewer_x55xOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OMP, a protein interaction language model built upon ESM-2 650M. OMP processes two protein sequences through separate encoding streams while enabling cross-protein information exchange at each layer through siamese cross-attention and GLU. The model is further pre-trained using MLM on interacting protein pairs collected from STRING. The proposed model is evaluated on three downstream tasks, including binary PPI prediction, TCR–epitope binding prediction, and residue-level interface prediction. OMP achieves 0.665 accuracy on the gold-standard PPI benchmark. However, the improvements on TCR–epitope prediction are modest, and OMP performs worse than ESM-2 on residue-level interface prediction.

Critical strengths
  1. The paper addresses practical limitations of concatenation-based protein interaction models, particularly sequence truncation and input-order dependence. The proposed architecture provides a straightforward solution with order invariance.
  2. The paper provides detailed implementation and training configurations.
Critical weaknesses
  1. Limited novelty over existing protein interaction language models - The proposed architecture shares substantial similarities with MINT, both methods extend ESM-2 with additional attentions and adopt MLM pre-training on samples from STRING. The primary architectural differences appear to be the siamese attn, GLU and symmetric head. These modifications are reasonable, but their individual contributions remain unclear. (lack of ablation studies for each component) In particular, Appendix E shows that MINT's encoder representations are approximately invariant under input swapping, while its prediction head introduces the observed order dependence. Therefore, a direct comparison with MINT+OMP head would help verify the contribution of the proposed architecture.

  2. Insufficient experimental evidence to justify the additional model capacity and pre-training cost - OMP introduces about 325M additional parameters. According to Appendix A, the matrix multiplication cost per layer increases by approximately 50% at an equal total input length. The comparison with the frozen ESM-2 backbone does not isolate the contribution of the proposed modules, since model capacity, architecture, and pre-training are changed simultaneously, as well as comparing with MINT.

  3. The effectiveness of cross-protein information exchange remains unclear - The motivation of cross-protein information exchange makes sense. However, the analysis of GLU openness raises further questions about the role of the newly introduced cross-attention pathway. Appendix E reports an average gate openness of approximately 0.005, with 19 of the 33 layers having an average openness below 0.001. Cross-attention activity is concentrated in a small number of layers and feature dimensions. Also, according to the analysis of the encoder representations of MINT in Appendix E.

Questions for authors

Besides the core weaknesses, I have several questions for authors:

  1. contribution of each component Have the authors evaluated continued MLM pre-training of the original ESM-2 on the same STRING data, as well as MINT with OMP head? How much improvement can be attributed specifically to the proposed cross-attention modules and GLU gating?

  2. sequence length performance As the author mentioned a benifit of the design is extending the input from 512 in MINT to 1024. Also, the performance weakness of PINDER is related to input length. Can the authors provide a full-length comparison on PINDER, together with performance stratified by protein length on the gold-standard benchmark?

  3. evaluation selection The evaluation of general binary PPI prediction is restricted to the gold-standard benchmark. Additional evaluations on established PPI datasets would strengthen the evidence for the general applicability of the proposed approach.

2 comments
AuthorsOpenReview ↗

Answer to reviewer

W1 and Q1. Contribution of each component

We trained the three configurations twice (Tables 9 and 10). The GLU exceeds the scalar gate by 0.0111 AUROC in the first run and by 0.0020 in the second one (CI [-0.0014, 0.0057]). Hence, we cannot attribute the gain to a component.

W2. Capacity and cost

We agree. Since our controls do not resolve the components, we cannot separate capacity from the architecture, and we lack a capacity-matched control. We state that the contribution of the pathway is not resolved (line 499).

W3. Cross-protein information exchange

Removing the pathway increases the masked-token cross-entropy by 0.0020, and a random partner increases it by 0.0068 (Table 8). Hence, the pathway is used despite the low gate openness. This result does not depend on a trained head. MINT shows a similar dependence on the partner (0.0095 against 0.0085 for OMP at a matched context).

Q2. Sequence length and full-length PINDER

OMP leads ESM-2 in the three length bands, but the lead does not grow with length (Table 12). We corrected this claim (lines 347–350). On full-length PINDER, ESM-2 remains ahead of OMP (median AUROC 0.7116 against 0.7011, p = 0.0035), and we report it (lines 404–407).

Q3. Additional PPI datasets

We agree. We could not add them within the discussion period.

Thank you for the detailed rebuttal and the additional experiments. The revisions substantially improve the empirical completeness of the paper.

However, my main concern regarding methodological novelty and performance remains, especially additional results on length-stratified experiments and cross-attn pathway.

Overall, I appreciate the authors' substantial efforts and the more transparent analysis. However, the additional evidence does not sufficiently resolve my concerns. I therefore maintain my original score.

Reviewer_DLWJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OMP extends ESM-2 with shared, layer-wise cross-attention modules and gated residual updates to model interacting protein pairs. It uses separate sequence streams and a symmetric prediction head to ensure order-invariant predictions. The model is further pretrained on STRING interaction pairs and evaluated through frozen embeddings on PPI prediction, TCR–epitope binding, and interface prediction.

Critical strengths

The motivation is sound. Giving each protein sufficient context, learning partner-dependent representations, and ensuring that interaction predictions remain unchanged when the inputs are swapped are sensible objectives. Separating intra-chain and inter-chain attention is a reasonable architectural approach to these objectives.

Critical weaknesses
  1. The experiments do not separate architectural improvements from additional pretraining and model capacity. OMP has approximately 975M parameters, compared with 650M for ESM-2, and receives additional pretraining on interacting sequence pairs. Using an identical downstream head controls the head architecture but does not resolve these differences. The PPI accuracy improvement from 64.9% to 66.5% therefore cannot be attributed specifically to layer-wise cross-attention or gating. A backbone continued-pretrained on the same pairs, together with controlled architectural ablations, is needed to establish the source of the gain.

  2. The MINT order-sensitivity example primarily demonstrates a prediction-head issue. Appendix E reports that swapping the inputs changes MINT’s encoder representations by at most approximately 8×10−68 \times 10^{-6}. The substantial prediction changes, including 16/30 label flips, arise from its head concatenating the two protein representations in input order. This example establishes a problem with the evaluated pipeline, but does not demonstrate that redesigning the encoder is necessary to resolve it. A direct control would retain MINT’s encoder and train a symmetric prediction head. Without this comparison, the example provides limited evidence for the additional value of OMP’s encoder design.

Questions for authors
  1. Does OMP retain its advantage over a baseline continued-pretrained on the same interaction pairs under comparable training conditions?
  2. How does MINT perform with a symmetric prediction head?
1 comment
AuthorsOpenReview ↗

Answer to reviewer

W1 and Q1. Architecture versus additional pre-training

We trained the requested baseline twice (Table 9). It reaches 0.7200 and 0.7046 AUROC in the two runs, against 0.7106 for ESM-2 and 0.7262 for OMP. Hence, this control does not separate the architecture from the additional pre-training, and we state it (lines 337–345 and 499).

W2 and Q2. MINT with a symmetric head

We agree that the order sensitivity of MINT originates in its head, and we now state it (lines 58–60). MINT with our symmetric head reaches 0.7067 AUROC, against 0.7106 for ESM-2 with the same head (Table 9). Thus, a symmetric head removes the order dependence, and it does not exceed ESM-2 under the same head.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to reviewers

We thank the reviewers. We ran the requested controls and uploaded a revised PDF with all changes marked in blue. Intervals are 99% paired bootstrap CIs over the test pairs.

Our main new results are as follows.

• We measured the pre-training exposure of the gold-standard test set. The released checkpoint saw 0.49% of the positive test pairs and no negative pairs, and removing these pairs leaves the difference with ESM-2 unchanged (Appendix I).

• We show that the cross-attention pathway depends on the partner. A random partner is 3.4 times more harmful to the masked-token cross-entropy than removing the pathway (Appendix J).

• We trained the requested controls under the identical prediction head (i.e., ESM-2 with continued MLM on the same pairs and no cross-attention, cross-attention with a scalar gate, and cross-attention with the gated linear unit). We trained each control head twice, and the differences between the controls do not hold across the two runs (Appendix K).

Accordingly, we revised our claims. The controls do not resolve the contribution of the additional pre-training, the pathway or the gating, and we no longer claim that layer-wise cross-attention is the source of the gain.

The changes in the revised PDF are as follows.

• We report the measured pre-training exposure in the abstract (lines 27–28).

• We scoped the order-dependence example to the prediction head (Section 1, lines 58–60).

• We removed the causal claim on layer-wise cross-attention (Section 1, lines 70–71).

• We added a contribution on the component analysis (Section 1, lines 92–98).

• We replaced the exposure bound with the measured value (Section 4, lines 299–308).

• We added the component analysis and the analysis by chain length (Section 4.1, lines 337–350).

• We added the full-length interface comparison (Section 4.3, lines 404–407).

• We rewrote the conclusion and the limitations around the revised claims (Section 5, lines 482–516).

• We added the new experiments as Appendices I to N (line 1109 onward).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.