acceptodds
Under review as a conference paper at ICLR 2027

State-space, attention or hybrid? PRIMA: a family of foundation models for protein-protein interactions.

Abstract

Protein–protein interactions (PPIs) govern essential biological processes, yet the vast majority of protein language models (PLMs) are transformers trained on single sequences. Our goal is to encode the semantics of the interaction: we ask which architectures are best suited to model PPIs, whether the sequence mixer matters, and at what scale it begins to. We present PRIMA, a family of PPI foundation models that share one training recipe and differ only in their sequence mixer: bidirectional state-space layers (PRIMA-M), full FlashAttention (PRIMA-A), and a hybrid that interleaves one attention layer every state-space layers (PRIMA-H). Each is trained at 8M and 150M parameters in two phases, pre-training on single sequences and post-training on concatenated PPI pairs. We evaluate against the ESM2 family and MINT, the state-of-the-art PPI specialist at 813M parameters, and we rebuild SKEMPI into SKEMPI-R to expose homology leakage and models that resort to recall. At 8M the benchmarks cannot separate the mixers. At 150M they separate, and PRIMA-H matches or outperforms models several times larger. On SKEMPI-R it is first on every metric, reaching Pearson against for ESM2 650M ( larger) and for MINT ( larger), with a partition-to-partition spread tighter, and it leads on the rows where models cannot exploit recall. On Gold Standard PPI, the classification benchmark built to remove protein-identity shortcuts, it matches MINT and PRIMA-M. Scaling PRIMA-H to 650M does not improve its ability to encode interaction specific properties, it only buys a better language model: the extra capacity goes into representing each chain rather than the relationship between them, a different approach is necessary. PRIMA mixers are equivalent after pre-training and diverge after post-training, placing attention's deficit in the interaction objective rather than in protein modelling. We find that PRIMA-H needs no positional encoding at all: permutation-equivariant attention layer can use the position encoded by state-space layers, previously demonstrated only for causal models. At 16,384 tokens PRIMA-H 150M encodes a complex in 74ms and 0.8GB against 10.5s and 78GB for MINT, and runs 65,536-token inputs on which MINT runs out of memory on an 80 GB GPU; below 2,000 tokens the state-space stacks are launch-bound and hold no speed advantage. PRIMA shows that a mixed state-space/attention stack produces specialist-level PPI representations at a fraction of the size and training budget.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein Language Models, Protein-protein interactions, State-space models, mamba, attention, hybrid architectures

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kyzyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PRIMA, a family of pLMs comparing bi-directional SSMs, transformers, and hybrid models under the same shared training recipe. They pertained the models on single proteins and then on concatenated interacting pairs. The paper also introduced SKEMPI-R and then evaluated the representation quality and inference efficiency.

Critical strengths
  • The comparison on different architecture designs is useful. It's informative evaluation to compare three model families at several parameter scales using the same training recipe.
  • It's nice to see small models achieved decent representation quality. For example, on SKEMPI-R, PRIMA-H (150M) outperformed ESM2 (650M) and HINT. On gold standard PPI, it performed on par with MINT with substantially fewer parameters.
  • The evaluation criteria is good. They reported within-complex ranking, improvement over a mean predictor, and mutation-coverage subsets provides useful context beyond aggregate correlation. They also acknowledged weak within-complex performance, short-input efficiency limitations, and implementation effects.
Critical weaknesses
  • LC-PLM is not positioned properly. The description of LC-PLM as a 1B-parameter model is not accurate. It's a main model is 740M with several other scales studied as well. Besides, it's not precise to say LC-PLM is never trained on concatenated complexes. The variant LC-PLM-G already constructs multi-protein sequences from graph walks and conducts MLM training. It's fair to say that graph contexts differ from curated physical complex inputs, but, this difference should not hinder the overlap in multi-protein contextual learning. PRIMA also explicitly adopts LC-PLM's shared forward/reverse projections. The statement of LC-PLM on lack of binding affinity evaluation is valid. However, it doesn't explain why the authors exclude it as a representation baseline.
  • Besides LC-PLM, some other works are missing and need proper positioning. For example, PPLM (2026), PLM-interact (2025), PPIformer (2024) and CATH-ddG (2025). These work should be discussed and their relationship to the proposed benchmark should be properly addressed as well.
  • The current study fails to prove that structural differences stem from interaction-specific learning. The authors should compare downstream performance before and after interaction training, and include shuffled-partner controls in Appendix A to isolate true interaction info from sequence correlations. Furthermore, ESM2's zero difference vector is a direct result of its independent encoding setup, which does not prove that its performance relies on memorization. A concatenated-input ESM2 baseline is missing.
  • The author explicitly acknowledges that uncovered mutations can still occur on chains homologous to training chains. Consequently, performance on this subset cannot by itself establish freedom from homology-based shortcuts. Evaluation on an existing strict split can augment this claims. Moreover, the error bars in Table 5 show probe-training randomness but not test-sample uncertainty. Non-overlapping error bars themselves don't establish reliable generalization difference.
  • The scaling conclusion is a bit oveclaimed. Although PRIMA-H (650M) performed worse using final-layer embeddings, the best intermediate-layer results are comparable, and gold standard AUPRC increased. The larger model also has fewer tokens/parameter. These findings support limited improvements under the used training recipe and readout protocol, but not the general conclusion that additional capacity improves only single-chain learning.
Questions for authors
  • Can the authors correct and expand the LC-PLM discussion, and include proper comparison if possible?
  • How does PRIMA differ from PPLM and PLM-interact in its learned representations and training objective? Which comparisons can be made under shared splits and comparable supervision?
  • Do the gains persist with concatenated-input ESM2, before/after post-training comparisons, and shuffled or incorrect partners?
  • Do the main rankings hold under a homology-disjoint SKEMPI evaluation and paired uncertainty estimates that account for shared proteins or complexes?
1 comment

Thank you for the detailed response and substantial revisions. I appreciate your efforts in addressing my concerns.

The revised manuscript now provides a more accurate positioning of LC-PLM and related work, additional baselines and controls for interaction-specific learning, and more rigorous analyses of homology leakage and statistical uncertainty. I also appreciate the more carefully qualified scaling conclusions.

While the homology-disjoint results remain inconclusive and generalization to unseen protein families is still an open question, you now explicitly acknowledge this limitation.

Overall, I find that my main concerns have been substantially addressed, and the revisions have meaningfully strengthened the paper. I'll raise my rating to 3.

Reviewer_FRfeOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a family of PPI foundation models, namely PRIMA. This paper compares three sequence-mixing architectures including attention, SSM, and hybrid. The experiments cover PPI classification, binding affinity change prediction, and inference efficiency across sequence lengths. The authors also introduce SKEMPI-R, a revised evaluation protocol designed to address limitations of existing SKEMPI benchmarks, including homologous protein overlap and mutation memorization.

Critical strengths
  1. systematic empirical comparison of sequence-mixing architectures

The paper provides a reasonably controlled comparison of state-space, attention, and hybrid architectures under similar parameter budgets and training conditions. The experiments across model scales offer useful empirical insights.

  1. comprehensive efficiency evaluation

The authors evaluate inference latency and memory consumption across a wide range of sequence lengths, using efficient attention implementations for the baselines. The results demonstrate the practical advantages of state-space and hybrid architectures for long protein sequences while also identifying their computational disadvantages for shorter inputs.

Critical weaknesses

Although this paper presents a useful empirical study for PPI modeling, my main concerns are the limited methodological novelty and the incomplete validation.

  1. limited methodological novelty.

The main methodological components of PRIMA are largely established in prior work. The two-stage training paradigm, involving single-sequence protein language modeling followed by PPI-specific MLM, closely follows the framework explored in MINT. The present work primarily extends these approaches through a systematic comparison of existing sequence mixers. Although this empirical investigation is useful, the proposed architecture introduces limited methodological innovation specific to PPI modeling. In particular, the hybrid design does not incorporate an explicit mechanism or learning objective tailored to inter-protein interactions.

  1. insufficient validation between architecture and training strategy.

The paper aims to identify which sequence-mixing architecture is better suited for PPI representation learning. However, architecture selection and pre-training strategy are closely related design choices. First, sec 4.1 primarily evaluates the pre-trained models using cross-entropy and perplexity. These metrics characterize the models' language modeling capacity, particularly their ability to reconstruct masked amino acids, but provide limited insight into the quality of learned representations for biological tasks. Second, sec 4.2 reports PPI performance after post-training, without providing corresponding results from the models before post-training. This makes it difficult to determine the contributions of the two training stages and whether the observed architectural differences originate from single-sequence pre-training or PPI-specific post-training.

Questions for authors
  1. Could the authors provide a more comprehensive evaluation of the representations learned at each training stage? (both linear probe or FFT OK)
  2. Can the partner-removal analysis be extended to all three architectures at 150M parameters?
  3. Does the reported scaling behavior persist under more comparable training budgets or alternative representation readouts? Although the models are trained with approximately the same total number of tokens, larger models may require more training tokens to reach convergence. Could the authors evaluate the scaling behavior with appropriately increased total training tokens and report the corresponding learning curves? This would help determine whether the observed degradation in interaction-related performance at 650M is attributable to scaling issue or insufficient training.
Reviewer_TM8UOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PRIMA, a family of protein language models comparing bidirectional state-space, attention, and hybrid architectures under a shared two-stage training recipe. It also proposes SKEMPI-R, a revised evaluation protocol for mutation-induced binding affinity changes. The strongest result is that PRIMA-H 150M achieves the best SKEMPI-R scores among the evaluated models while matching the substantially larger MINT model on Gold Standard PPI. The paper additionally investigates representation quality across model scales and measures inference efficiency across sequence lengths.

Critical strengths
  1. The architecture comparison is reasonably controlled and produces a concrete result. The three variants use the same training corpora, token budgets, and objectives at approximately matched parameter counts. These results provide useful evidence for choosing among the three architectures under this training recipe.
  2. The efficiency evaluation reports useful limitations alongside the gains. The authors use memory-efficient attention for ESM2 and explicitly report that PRIMA-M 150M is 4.6× slower at 512 tokens, overtaking ESM2 only around 2,048 tokens. They also distinguish implementation effects from architectural effects in the MINT comparison. Together with PRIMA-H 150M matching MINT’s Gold Standard PPI performance, this provides a practical case for the smaller model without implying a speed advantage at every input length.
Critical weaknesses
  1. The claim that the architectures diverge specifically because of interaction training is not directly established. The evidence compares single-sequence MLM performance before post-training with downstream PPI performance afterward. Similar MLM losses do not establish that the initial checkpoints had equivalent PPI representations. Evaluating the same downstream tasks before and after pair post-training would directly test this attribution. The current results establish a difference between the final models, but do not isolate when that difference emerges.
  2. The strongest improvement remains concentrated in one benchmark, and generalization to unfamiliar interactions is incompletely tested. On Gold Standard PPI, PRIMA-H’s 0.687 AUPRC is effectively tied with PRIMA-M’s 0.689 and MINT’s 0.688. On SKEMPI-R, even the best within-complex Spearman is only 0.152. Moreover, as Appendix C acknowledges, the uncovered subset excludes repeated substitutions on homologous chains but still permits other homology overlap. Its results therefore support robustness to that particular shortcut, without establishing generalization to unseen protein families.
Questions for authors

Can you evaluate the single-sequence pretrained checkpoints on the same downstream tasks to show how pair post-training changes each architecture’s performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.