OMP: Order-Invariant Protein Interaction Language Modeling with Layer-wise Siamese Cross-Attention

Protein language models (PLMs) adapted to protein–protein interactions (PPIs) typically concatenate the two protein input sequences, which forces aggressive cropping, makes predictions depend on input order, and requires every residue to split a single attention budget between its own chain and its partner. We introduce OMP, a dual encoder that extends ESM-2 (650M) with Siamese cross-attention modules running in parallel to self-attention in every layer, each followed by a gated linear unit that is initialized nearly closed. OMP processes up to 1,022 residues per chain, twice the per-chain context of concatenation-based models built on the same backbone, and is order-invariant by construction. We pre-train OMP with masked language modeling on interacting sequence pairs from STRING and evaluate its frozen embeddings on three downstream tasks. Pre-training takes 4,032 GPU-hours over a fifth of the curated corpus, so adding pair-level structure to a pre-trained PLM does not require re-pretraining at the scale of the backbone. On the gold-standard PPI benchmark, OMP reaches 66.5% accuracy and 0.712 AUPRC, surpassing the 65% accuracy plateau of sequence-based models on this leakage-controlled benchmark and significantly outperforming ESM-2 with an identical prediction head. Among the models evaluated here, only OMP exceeds 0.60 mean AUROC on TCR–epitope binding prediction, and it outperforms the interaction language model MINT on interface prediction, where the single-sequence ESM-2 baseline remains strongest. Without structural supervision, OMP's attention maps recover protein domains: 86% of the strongest intra-protein energetic contacts in a target protein fall within the top 15% of layer-30 self-attention. Anonymized code and pre-trained weights are available at https://anonymous.4open.science/r/OMP and https://huggingface.co/OMP-ICLR/OMP.