- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PRIMA, a family of pLMs comparing bi-directional SSMs, transformers, and hybrid models under the same shared training recipe. They pertained the models on single proteins and then on concatenated interacting pairs. The paper also introduced SKEMPI-R and then evaluated the representation quality and inference efficiency.
Critical strengths
- The comparison on different architecture designs is useful. It's informative evaluation to compare three model families at several parameter scales using the same training recipe.
- It's nice to see small models achieved decent representation quality. For example, on SKEMPI-R, PRIMA-H (150M) outperformed ESM2 (650M) and HINT. On gold standard PPI, it performed on par with MINT with substantially fewer parameters.
- The evaluation criteria is good. They reported within-complex ranking, improvement over a mean predictor, and mutation-coverage subsets provides useful context beyond aggregate correlation. They also acknowledged weak within-complex performance, short-input efficiency limitations, and implementation effects.
Critical weaknesses
- LC-PLM is not positioned properly. The description of LC-PLM as a 1B-parameter model is not accurate. It's a main model is 740M with several other scales studied as well. Besides, it's not precise to say LC-PLM is never trained on concatenated complexes. The variant LC-PLM-G already constructs multi-protein sequences from graph walks and conducts MLM training. It's fair to say that graph contexts differ from curated physical complex inputs, but, this difference should not hinder the overlap in multi-protein contextual learning. PRIMA also explicitly adopts LC-PLM's shared forward/reverse projections. The statement of LC-PLM on lack of binding affinity evaluation is valid. However, it doesn't explain why the authors exclude it as a representation baseline.
- Besides LC-PLM, some other works are missing and need proper positioning. For example, PPLM (2026), PLM-interact (2025), PPIformer (2024) and CATH-ddG (2025). These work should be discussed and their relationship to the proposed benchmark should be properly addressed as well.
- The current study fails to prove that structural differences stem from interaction-specific learning. The authors should compare downstream performance before and after interaction training, and include shuffled-partner controls in Appendix A to isolate true interaction info from sequence correlations. Furthermore, ESM2's zero difference vector is a direct result of its independent encoding setup, which does not prove that its performance relies on memorization. A concatenated-input ESM2 baseline is missing.
- The author explicitly acknowledges that uncovered mutations can still occur on chains homologous to training chains. Consequently, performance on this subset cannot by itself establish freedom from homology-based shortcuts. Evaluation on an existing strict split can augment this claims. Moreover, the error bars in Table 5 show probe-training randomness but not test-sample uncertainty. Non-overlapping error bars themselves don't establish reliable generalization difference.
- The scaling conclusion is a bit oveclaimed. Although PRIMA-H (650M) performed worse using final-layer embeddings, the best intermediate-layer results are comparable, and gold standard AUPRC increased. The larger model also has fewer tokens/parameter. These findings support limited improvements under the used training recipe and readout protocol, but not the general conclusion that additional capacity improves only single-chain learning.
Questions for authors
- Can the authors correct and expand the LC-PLM discussion, and include proper comparison if possible?
- How does PRIMA differ from PPLM and PLM-interact in its learned representations and training objective? Which comparisons can be made under shared splits and comparable supervision?
- Do the gains persist with concatenated-input ESM2, before/after post-training comparisons, and shuffled or incorrect partners?
- Do the main rankings hold under a homology-disjoint SKEMPI evaluation and paired uncertainty estimates that account for shared proteins or complexes?
Thank you for the detailed response and substantial revisions. I appreciate your efforts in addressing my concerns.
The revised manuscript now provides a more accurate positioning of LC-PLM and related work, additional baselines and controls for interaction-specific learning, and more rigorous analyses of homology leakage and statistical uncertainty. I also appreciate the more carefully qualified scaling conclusions.
While the homology-disjoint results remain inconclusive and generalization to unseen protein families is still an open question, you now explicitly acknowledge this limitation.
Overall, I find that my main concerns have been substantially addressed, and the revisions have meaningfully strengthened the paper. I'll raise my rating to 3.