- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper uses contrastive fune-tuning on protein language models to create stronger embeddings for biological related proteins across sequences. They use a unique data mixture (Pfam, protein families, AlphaFoldDB for structure, and STRING for interactions) to capture different notions of similarity. Their main result is that retrieval improves significantly, but linear probing deteriorates.
Critical strengths
The empirical results are strong across the tasks presented. On retrieval, MAP goes from .42 -> .7 CATH retrieval similarly improves about 20% and clustering seems to improve from basically irrelevant on a 35M parameter model to 0.5 on ARI. This is not incredibly surprising -- the tasks are all directly aligned with the dataset mixture (e.g., structural alignment with AlphaFoldDB), but it's still nice to see.
Critical weaknesses
I'm evaluating this paper on two axes, methodology and whether the actual method is sufficiently strong to be of interest to the community.
Methodologically, this paper does not really introduce anything new. Contrastive fine-tuning is used both in proteins and other domains (see [1]), and the results themselves are really testing things that the model explicitly trains to optimize. For this reason, acceptance of this paper turns on the strength of the results.
While the results themselves seem strong, like I mentioned, they seem to be mostly just stronger results in a regime that the model was trained on. This is unlikely to be of broad interest to the community, it isn't particularly surprising to me and I'd imagine most folks who need this capability are already doing this sort of training (e.g., structural contrastive learning has already been done). This is compounded by the decontamination issues (which the authors do raise, which I appreciate), but it makes it harder to tell if this technique is actually useful.
The only remaining question is whether we get stronger general purpose embeddings from contrastive fine tuning. I think this would be quite an important result; the dominant reason to use protein language models is as a strong initial embedding with fine-tuning on some downstream specific task for which we have less data. This doesn't seem to be the case, the linear probe results in particular suggestion that this may make other task performance even worse. Without this evidence, I don't see a situation where I would ever want to actually use these fine-tuned embeddings, except for cases where my task is literally what the model was trained on -- in which case, I should just use tools designed for those tasks.
[1] Heinzinger, Michael, et al. "Contrastive learning on protein embeddings enlightens midnight zone." NAR genomics and bioinformatics 4.2 (2022): lqac043.
As an aside, the text of the paper strongly reads as written by language model. This was disclosed in the AI statement and did not affect my recommendation. Nonetheless, I'd urge the authors to reconsider this for future submissions.
Questions for authors
- Are there tasks on which contrastive fine-tuning actually improves the general purpose strength of the embedding?
3 comments
Response to authors: maintaining score
Thanks for your response. My core issue from my original rebuttal was not focused on generalization to unseen folds. I was raising the issue that the tasks on which the model is evaluated -- retrieval, clustering, and nearest neighbor transfer -- are all things that both intuitively and empirically are being trained for by contrastive methods. So the work is essentially noting that by fine-tuning on a task, we can improve model performance on that task. This isn't interesting in of itself, and if the general embedding strength doesn't improve, I do not see this being of interest to the community. For these reasons, I will maintain my score.
Thanks for clarifying. We do see a broad improvement overall and showed this in the benchmarks. Across the backbones, most tasks improved given the contrastive pretraining (Table 3). That includes predicting mutation effects, peptide–HLA binding and subcellular localization, which are not trained for explicitly. We trained on protein families, structural similarity and interactions, and the same embeddings proved useful across diverse downstream tasks - i.e universal/foundational improvements. We agree that the base models generally do better with a trained linear probe, but it still matters that the gains transfer.
Thank you for the direct review. We agree that contrastive training of protein embeddings is not new, and that the work should be judged by its practical results.
Novelty. ProtTucker keeps ProtT5 frozen and trains a small head on curated CATH labels. ProtSent updates the whole backbone, combines three relation types (Pfam families, AlphaFold DB structural clusters, STRING interactions) and never sees a CATH or SCOPe label. On ProtTucker's own CATH benchmark, ProtSent on ESM-C reaches 70.7% at the H level. Published ProtTucker reaches 70% on ESM-1b and 76% on ProtT5.
Are the gains confined to the training regime? We tested this (global comment, point 1). SCOPe families track the Pfam families we train on, so we removed each query's own family from the gallery, and every hit must then come from a different family. ProtSent still leads its frozen base by +0.134, +0.195 and +0.496 R@1 at the three model sizes, with intervals excluding zero. The gain also holds one level up, with the query's whole superfamily removed. We do not claim transfer to unseen folds.
We also added the simplest alternative, a label-free whitening of the base embeddings (global comment, point 2). It recovers a large part of the ESM-2 gain and matches ProtSent at 150M. ProtSent stays ahead at 35M on SCOPe-40 and everywhere on ESM-C, where it leads by +0.137 R@1 and by 34.7 points on CATH. We narrowed the claims to match.
Why not specialist tools? Paired against filters-off phmmer on SCOPe-40, phmmer leads at top-1 at 35M, the two tie at 150M, and ProtSent-ESMC leads (+0.043 [+0.021, +0.064] R@1). All three ProtSent models lead in ranking depth (MAP +0.045, +0.104 and +0.169). phmmer finds no significant hit (E ≤ 10⁻³) for 36% of queries, while the embedding ranks every query. Foldseek needs a structure for each query. ProtSent complements these tools.
Q1. Does it improve general-purpose embeddings? Not under a trained linear probe. Section 5.4 reports this as the main negative result, and structure distillation (ISM-C) shows a similar cost. The gains are in retrieval, clustering and nearest-neighbour transfer. For a trained linear readout, the base model is the better starting point, revised section 6 will include this.
Writing. Thank you, we will remove the repetitive conclusions and separated observations. The AI-use statement describes our use accurately.
We would be grateful if you would reconsider your scores in light of the new controls.