acceptodds
Under review as a conference paper at ICLR 2027

ProtSent: Protein Sentence Transformers

Abstract

Protein language models produce per-residue representations rich in evolutionary and structural information. The sequence embeddings obtained by mean-pooling those representations, however, are never trained to place related proteins near one another in the embedding space, so the geometry on which retrieval, clustering and nearest-neighbor transfer depend is left to chance. We present ProtSent, a contrastive post-training framework that supervises a protein language model with four types of biological relations at once: Pfam family co-membership, AlphaFold DB Foldseek structural clusters, STRING interactions, and deep mutational scanning fitness ordering. It uses no structure encoder at training or inference, and every training corpus is filtered against the benchmark test sets at 40% identity and 80% coverage. Post-training makes structural family membership recoverable from a frozen embedding without labels. Clustering 2,207 SCOPe-40 domains at their 917 true families raises the adjusted Rand index of the 35M model from 0.054 to 0.507. Family retrieval gains +0.185, +0.190 and +0.424 Recall@1 over three frozen backbones spanning two independently pretrained model families (95% intervals exclude zero). The result reproduces on the CATH v4.3 midnight-zone benchmark, where Homologous-superfamily accuracy rises from 40.7% to 56.7% at 35M and from 43.3% to 62.7% at 150M; these gains of +16 and +19 points exceed the +12 that a CATH-supervised specialist gains over its own ProtT5 base, even though that base is twenty times larger than our backbones. Against sequence-only alignment the frozen embedding leads at ranking depth, MAP 0.705 against 0.607 for a maximally sensitive profile search, and returns a ranking for every query, whereas that search returns no hit at all for 31% of them. We also measure what the reorganization costs, and find that under a trained linear probe the same embeddings fall below the untuned backbone, and that structure distillation pays the same price against its own matched base. Contrastive post-training improves the neighborhoods of the embedding space at the cost of linear-probe accuracy, and that trade is the choice a practitioner faces. We release models at three sizes, the data, and the training recipe.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
protein language models, PLMs, Sentence transformers, contrastive learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_D5ZWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses contrastive fune-tuning on protein language models to create stronger embeddings for biological related proteins across sequences. They use a unique data mixture (Pfam, protein families, AlphaFoldDB for structure, and STRING for interactions) to capture different notions of similarity. Their main result is that retrieval improves significantly, but linear probing deteriorates.

Critical strengths

The empirical results are strong across the tasks presented. On retrieval, MAP goes from .42 -> .7 CATH retrieval similarly improves about 20% and clustering seems to improve from basically irrelevant on a 35M parameter model to 0.5 on ARI. This is not incredibly surprising -- the tasks are all directly aligned with the dataset mixture (e.g., structural alignment with AlphaFoldDB), but it's still nice to see.

Critical weaknesses

I'm evaluating this paper on two axes, methodology and whether the actual method is sufficiently strong to be of interest to the community.

Methodologically, this paper does not really introduce anything new. Contrastive fine-tuning is used both in proteins and other domains (see [1]), and the results themselves are really testing things that the model explicitly trains to optimize. For this reason, acceptance of this paper turns on the strength of the results.

While the results themselves seem strong, like I mentioned, they seem to be mostly just stronger results in a regime that the model was trained on. This is unlikely to be of broad interest to the community, it isn't particularly surprising to me and I'd imagine most folks who need this capability are already doing this sort of training (e.g., structural contrastive learning has already been done). This is compounded by the decontamination issues (which the authors do raise, which I appreciate), but it makes it harder to tell if this technique is actually useful.

The only remaining question is whether we get stronger general purpose embeddings from contrastive fine tuning. I think this would be quite an important result; the dominant reason to use protein language models is as a strong initial embedding with fine-tuning on some downstream specific task for which we have less data. This doesn't seem to be the case, the linear probe results in particular suggestion that this may make other task performance even worse. Without this evidence, I don't see a situation where I would ever want to actually use these fine-tuned embeddings, except for cases where my task is literally what the model was trained on -- in which case, I should just use tools designed for those tasks.

[1] Heinzinger, Michael, et al. "Contrastive learning on protein embeddings enlightens midnight zone." NAR genomics and bioinformatics 4.2 (2022): lqac043.

As an aside, the text of the paper strongly reads as written by language model. This was disclosed in the AI statement and did not affect my recommendation. Nonetheless, I'd urge the authors to reconsider this for future submissions.

Questions for authors
  1. Are there tasks on which contrastive fine-tuning actually improves the general purpose strength of the embedding?
3 comments

Thank you for the direct review. We agree that contrastive training of protein embeddings is not new, and that the work should be judged by its practical results.

Novelty. ProtTucker keeps ProtT5 frozen and trains a small head on curated CATH labels. ProtSent updates the whole backbone, combines three relation types (Pfam families, AlphaFold DB structural clusters, STRING interactions) and never sees a CATH or SCOPe label. On ProtTucker's own CATH benchmark, ProtSent on ESM-C reaches 70.7% at the H level. Published ProtTucker reaches 70% on ESM-1b and 76% on ProtT5.

Are the gains confined to the training regime? We tested this (global comment, point 1). SCOPe families track the Pfam families we train on, so we removed each query's own family from the gallery, and every hit must then come from a different family. ProtSent still leads its frozen base by +0.134, +0.195 and +0.496 R@1 at the three model sizes, with intervals excluding zero. The gain also holds one level up, with the query's whole superfamily removed. We do not claim transfer to unseen folds.

We also added the simplest alternative, a label-free whitening of the base embeddings (global comment, point 2). It recovers a large part of the ESM-2 gain and matches ProtSent at 150M. ProtSent stays ahead at 35M on SCOPe-40 and everywhere on ESM-C, where it leads by +0.137 R@1 and by 34.7 points on CATH. We narrowed the claims to match.

Why not specialist tools? Paired against filters-off phmmer on SCOPe-40, phmmer leads at top-1 at 35M, the two tie at 150M, and ProtSent-ESMC leads (+0.043 [+0.021, +0.064] R@1). All three ProtSent models lead in ranking depth (MAP +0.045, +0.104 and +0.169). phmmer finds no significant hit (E ≤ 10⁻³) for 36% of queries, while the embedding ranks every query. Foldseek needs a structure for each query. ProtSent complements these tools.

Q1. Does it improve general-purpose embeddings? Not under a trained linear probe. Section 5.4 reports this as the main negative result, and structure distillation (ISM-C) shows a similar cost. The gains are in retrieval, clustering and nearest-neighbour transfer. For a trained linear readout, the base model is the better starting point, revised section 6 will include this.

Writing. Thank you, we will remove the repetitive conclusions and separated observations. The AI-use statement describes our use accurately.

We would be grateful if you would reconsider your scores in light of the new controls.

Reviewer_D5ZWOpenReview ↗

Response to authors: maintaining score

Thanks for your response. My core issue from my original rebuttal was not focused on generalization to unseen folds. I was raising the issue that the tasks on which the model is evaluated -- retrieval, clustering, and nearest neighbor transfer -- are all things that both intuitively and empirically are being trained for by contrastive methods. So the work is essentially noting that by fine-tuning on a task, we can improve model performance on that task. This isn't interesting in of itself, and if the general embedding strength doesn't improve, I do not see this being of interest to the community. For these reasons, I will maintain my score.

Thanks for clarifying. We do see a broad improvement overall and showed this in the benchmarks. Across the backbones, most tasks improved given the contrastive pretraining (Table 3). That includes predicting mutation effects, peptide–HLA binding and subcellular localization, which are not trained for explicitly. We trained on protein families, structural similarity and interactions, and the same embeddings proved useful across diverse downstream tasks - i.e universal/foundational improvements. We agree that the base models generally do better with a trained linear probe, but it still matters that the gains transfer.

Reviewer_BCpgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ProtSent, a contrastive post-training recipe for protein language models in the spirit of Sentence-BERT. ESM-2 (35M, 150M) and ESM-C (300M) are fine-tuned end-to-end with Multiple Negatives Ranking Loss over three relation sources in one objective: Pfam family co-membership, AlphaFold DB Foldseek structural clusters, and STRING interactions, with a CoSENT loss on deep mutational scanning fitness ordering as an optional fourth. No structure is needed at inference.

Validation: (1) embedding-geometry and retrieval metrics on SCOPe-40 (e.g., ARI of the 35M model rises from 0.054 to 0.507; Recall@1 gains of +0.185 / +0.190 / +0.424 across the three backbones, with paired bootstrap intervals); (2) nearest-neighbour superfamily transfer on the CATH midnight-zone set (40.7% to 56.7% at 35M, 43.3% to 62.7% at 150M); (3) a comparison with filters-off phmmer, where alignment keeps the best single hit but the embedding leads at ranking depth and returns a ranking for every query; (4) a 23-task linear-probe suite, on which ProtSent falls below its own untuned backbone on most tasks, which the authors report openly; and (5) leave-one-source-out ablations. The paper also discloses several process problems: the 512-residue cap was not applied, hard negatives in V1 were effectively positives, the configuration was selected on the evaluation suite, and each model has one seed.

Critical strengths
  1. Unusually transparent, careful reporting. Paired bootstrap intervals throughout, strong alignment baselines (filters-off phmmer, profile HMMs), a pre-stated tie band, and open reporting of the negative linear-probe result and of the process problems listed above. This matters to my assessment because it makes the paper's numbers trustworthy as reported, and it lets me separate rigor of reporting (high) from rigor of experimental design (weaker, see below).

  2. Large, consistent retrieval gains from a sequence-only embedding. The gains on SCOPe-40 and the CATH midnight zone are large relative to each base and hold across two model families. The coverage result (a ranking for queries where alignment returns no hit) is practically relevant for remote-homology search. If the gains survive the controls requested below, this would be a useful, cheap tool.

Critical weaknesses
  1. The two headline benchmarks were not decontaminated against the training data, and the training label closely matches the SCOPe test label. Section 3.3 states that only the TAPE remote-homology set and the Bernett PPI set were used as filtering targets. The abstract's statement that the corpora were "filtered against the benchmark test sets they most resemble" therefore overstates what was done: SCOPe-40 and CATH test-219 were not filtered. By the paper's own analysis, the median maximum identity of a SCOPe-40 domain to the training corpus is 0.908, and only 164 of 1,693 eligible queries fall below 0.4. Because training positives are Pfam family co-membership, and SCOPe families track Pfam families closely, SCOPe-40 family retrieval is close to an in-distribution test of the training label. For CATH, "no alignment-detectable relative" is defined against the lookup set, not against the training corpus. The memorization argument offered does not resolve this: memorizing family labels does not predict a positive correlation between identity and gain, and the correlation is not corrected for headroom. This is consequential because SCOPe-40 and CATH carry the paper's central claim.

  2. The cheapest strong baselines are missing, so the size of the gain attributable to the method is unclear. (a) Whitened frozen backbone: the base embedding is highly anisotropic (mean random-pair cosine 0.848; participation ratio 7.9 of 480 dimensions), which by itself depresses base ARI and compresses its distance range. The paper's own Table 19 shows whitening raises base 3-NN remote homology from 0.584 to 0.680, against ProtSent's 0.667 raw and 0.701 whitened, so an unsupervised linear map recovers most of the post-training gain on that task (+0.083 shrinks to about +0.02 when both are whitened). The CATH control uses only L2 normalization, which neither centers the embedding nor removes the dominant shared direction. This is the BERT-whitening point from NLP and speaks directly to the Sentence-BERT analogy. (b) Direct competitors: Foldseek (the SCOPe-40 test set is essentially Foldseek's own benchmark, and AFDB structures exist for these sequences, so it is at least an upper reference), DHR (trained on SCOPe domains, the most direct competitor), and ProtTucker run on the same backbone; ProtTucker currently appears only through published ProtT5 numbers. The abstract's comparison with "the 12 points a CATH-supervised model gains" compares gains from different starting points (ProtT5 at 64 vs ESM-2 at 40.7); in absolute terms ProtTucker reaches 76 and profile HMMER 77, against ProtSent-150M's 62.7.

Other concerns (less critical). (i) The linear-probe loss is reported but not explained; the only explanation offered is one sentence, and Table 19's caption itself says it "does not test the cause". (ii) The multi-relation contribution is never tested directly: no single-source model (Pfam-only or AFDB-only) is trained on the final recipe, the leave-one-out ablations are single runs on the V1 configuration (with the defective hard negatives), and the PPI payoff is small (Bernett AUC 0.572 to 0.583 at 35M). (iii) On the 23-task 3-NN suite the median signed deltas are +0.004 (35M) and +0.005 (150M), within the paper's own 0.005 tie band, so the positive result is retrieval-specific; "net positive at every scale" holds clearly only for ESM-C. (iv) The largest headline number (+0.424 R@1) comes from ProtSent-ESMC-300M, which also trains on the non-decontaminated DMS corpus, starts from a very weak base (R@1 0.371), and has no CATH row in Table 4. (v) Presentation: no figures in the main text, the V1/V2/V2.5 naming is hard to track, and the density of caveats obscures the core claim.

Questions for authors

Answers to Q1 to Q3 could change my rating.

Q1 (experiment / clarification). Please report the sequence identity of CATH test-219 queries to the training corpus, and SCOPe-40 and CATH results stratified by maximum identity to training (or results on an identity-filtered subset, e.g., below 30% or 40%). Does the gain over the base model hold in the low-identity stratum?

Q2 (experiment). Please add a centered and whitened frozen ESM-2 baseline (whitening fitted on held-out data) on SCOPe-40 (ARI, R@1, MAP) and CATH, and report ProtSent whitened the same way. How much of the gain remains?

Q3 (experiment). Can you provide one controlled comparison with ProtTucker (its recipe on ESM-2, or ProtSent on ProtT5), and Foldseek and DHR on SCOPe-40 (Foldseek at least as an upper reference)?

Q4 (experiment). What does STRING buy on the retrieval metrics, and what does it cost? A single-source model (Pfam-only, AFDB-only) trained on the final V2 recipe would test the multi-relation claim directly.

Q5 (experiment). For ESM-C: results for the V2 recipe without DMS, and its CATH numbers.

Q6 (clarification). The no-hit coverage figure (691 queries) and the Recall@10 to Recall@30 flattening use default-filter phmmer, while Table 2 uses filters-off phmmer. What is coverage under filters-off phmmer? A hybrid baseline (alignment first, embedding ranking when alignment returns nothing) is the natural deployment comparison.

Q7 (clarification). The CATH confidence-interval caption gives n=150, but the test set has 219 queries; which is it? A paired test of each base against its ProtSent model would be stronger than the unpaired ±8 interval.

Q8 (clarification). The corpus counts do not reconcile: Pfam has 32.94M sequences in Appendix B.1, but decontamination starts from 28.53M rows; STRING has 36.5M pairs in Appendix B.4, but decontamination starts from 76.07M rows, and the 4,178,737 removed rows are also called "pairs". Also, the 150M V2 model ran 3,890 steps at batch 1,024 (about 4M pairs), a small fraction of the corpora; please state how many examples per source the model actually saw.

Q9 (suggestion; would strengthen the paper but does not affect my rating). For the linear-probe loss: probes on intermediate layers for the tasks that lose; concatenating ProtSent with the base embedding; partial-layer, LoRA, or MLM-regularized fine-tuning; and cross-validated probe regularization (ridge α=1 and default logistic regression may disadvantage contrastively trained features).

1 comment

Thank you for the detailed review. We ran the two controls you put first (Q1, Q2), and they changed what we claim.

W1 / Q1. Training overlap. You are right, and the abstract is corrected. The global comment gives the new analysis. Identity strata are not informative here, since only 12 SCOPe-40 queries and 1 CATH query are below 40% identity to our training sources. We instead removed each query's own family (or superfamily) from the gallery. The gain over the frozen base holds at both levels and all three model sizes.

W2(a) / Q2. Whitening. The main table is in the global comment. Whitening is fit without labels on each benchmark's training side: the lookup set for CATH, and 50,000 domains from our training corpus for SCOPe-40. Domains with 40% or more identity to any test query are excluded from the fit.

At 150M, ProtSent and the whitened base tie on SCOPe-40 R@1 and MAP and on CATH. ProtSent leads only on R@10 (+0.040 [+0.027, +0.054]). At 35M it keeps its SCOPe-40 gain and ties on CATH. On ESM-C it leads everywhere.

ESM-2 35MESM-2 150M
R@1, both models whitened+0.086 [+0.064, +0.109]+0.020 [+0.001, +0.040]
MAP, both models whitened+0.071 [+0.057, +0.085]+0.034 [+0.022, +0.046]
Clustering ARI, ProtSent / whitened base0.507 / 0.4800.592 / 0.551

We narrowed the claim to match (global comment).

W2(b) / Q3. Competitors. We removed the "+12 vs. +16" claim. ProtTucker trains a head on frozen ProtT5 with CATH labels and reaches 76 at the H level (70 on ESM-1b). ProtSent on ESM-C reaches 70.7 without CATH labels. We did not run a matched ProtTucker, DHR or Foldseek. Foldseek needs a structure for each query.

Q4. STRING. Removing STRING is the only ablation that turns the PPI gain negative (+5.3% → −0.5%), and it slightly helps remote homology (+40.5% → +43.3%). After the Bernett filter, PPI 3-NN AUC rises from 0.572 to 0.583 (35M) and 0.594 (150M). Single-source runs on the final recipe are not available; we state this as a limitation.

Q5. ESM-C, DMS and CATH. New CATH rows (C / A / T / H, %):

C / A / T / H
ESM-C 300M66.7 / 36.1 / 21.9 / 18.7
ESM-C 300M, whitened61.6 / 41.1 / 31.4 / 36.0
ProtSent-ESMC-300M84.5 / 73.1 / 62.4 / 70.7

ProtSent-ESMC trains with DMS, so we checked overlap. 268 SCOPe-40 and 45 CATH queries share at least 40% identity with a DMS wild type. On the remaining queries, the gain over the whitened base is unchanged: +0.138 [+0.114, +0.162] R@1 on SCOPe-40 and +36.8 [+27.4, +46.2] points on CATH. We did not train ESM-C without DMS.

Q6. phmmer coverage and hybrid. We corrected the coverage claim. The 31% came from default-filter phmmer. Filters-off phmmer returns some hit for every query at E ≤ 10, but many are chance hits. At E ≤ 10⁻³, 36% of all queries and 23% of the 1,693 eligible queries have no significant hit. A hybrid (phmmer hits first, then the embedding) adds only +0.012 MAP. Paired against filters-off phmmer:

Model minus phmmerR@1MAP
ProtSent 35M−0.068 [−0.092, −0.044]+0.045 [+0.027, +0.062]
ProtSent 150M−0.010 [−0.033, +0.012]+0.104 [+0.087, +0.121]
ProtSent-ESMC-300M+0.043 [+0.021, +0.064]+0.169 [+0.152, +0.185]

phmmer leads at top-1 at 35M, the two tie at 150M, and ProtSent-ESMC leads. All three models lead on MAP. Under stricter scoring, where a positive that phmmer never returned counts as a miss, phmmer's MAP is 0.600 (printed as 0.607).

Q7. n=150 vs. 219. Both are right. A query is dropped at a level when no other domain carries its label, so n is 219 at C and A, 210 at T and 150 at H. The ±8 is the interval of one model's H accuracy. Paired exact McNemar tests at H against the raw base: ProtSent alone is right on 29 queries and the base alone on 5 at 35M, and 35 vs. 6 at 150M (both p < 0.001). Against the whitened base the two tie (17 vs. 14, and 10 vs. 10), consistent with Q2.

Q8. Corpus counts. The 35M model saw 34.8M pairs in one pass: 0.78M from Pfam (2.2%), 18.99M from AlphaFold DB (54.6%) and 15.0M from STRING (43.1%). The 150M model drew from a 23.9M-pair pool; one resume restarted its data order, so exact per-source counts are not available. The revision replaces the Appendix B figures with one reconciled table per source.

Other concerns, Q9.

  • The space does not collapse (participation ratio 7.9 → 52.5). To locate the linear-probe loss, we will add intermediate-layer probes and a base+ProtSent concatenation.
  • One training run exists per model, and we state this as a limitation.
  • The initial ablations are labelled as run with the faulty hard negatives.
  • We dropped the broad 3-NN claim. The suite medians (+0.004 / +0.005) do not exceed the tie band.
  • LoRA and MLM-regularized variants are future work.
Reviewer_rM2QOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

In this paper the authors propose ProtSent, a new method to address a limitation in protein language models. While PLM-learned residue representations are indeed shown to support downstream biological tasks, mean-pooled sequence embeddings are not necessarily optimized for nearest-neighbor retrieval or clustering. In response, the authors apply contrastive post-training using Pfam family membership, AlphaFold DB structural clusters, STRING interactions (and optionally deep mutational scanning fitness info).

The authors report substantial improvements on clustering, family retrieval, and remote homolog retrieval. Reported gains are also shown over the frozen backbone. Results also seem competitive against sequence-alignment and structure-based methods on selected benchmarks. The presented work has a potentially useful empirical contribution in demonstrating that contrastive post-training can improve embedding-based retrieval at relatively small model scales.

The paper's main contribution is that contrastive post-training can improve protein retrieval. To strengthen the paper, the authors need to clarify how the proposed method is different from prior work, test the contribution of each supervision source under controlled conditions, examine possible overlap between training and test structures, and explain why retrieval improves while linear-probe performance declines.

Critical strengths

(1) Mean-pooled embeddings are convenient for large-scale search, clustering, and annotation transfer, but general-purpose pretraining does not directly optimize these uses. So, ProtSent is timely and targets a specific limitation.

(2) Reported gains in SCOPe clustering, family retrieval, and CATH midnight-zone homolog retrieval are substantial for smaller models. Results suggest post-training can improve usefulness of existing PLMs without requiring a much larger backbone.

(3) The paper considers family relationships, structural similarity, interaction information, remote homology, and protein-property tasks. This provides a broader view than an evaluation restricted to a single retrieval benchmark. Additionally, comparisons with HMMER and other embedding- and structure-based methods contextualize ProtSent among protein search approaches.

(4) Neighborhood-based performance is shown to improve while linear-probe results declines relative to the frozen backbone. This is potentially a valuable scientific contribution about how contrastive post-training changes the utility of protein representations (although the authors miss an opportunity to explain this sufficiently).

Critical weaknesses

(1) I have concerns about novelty. Contrastive optimization of protein embeddings for structural relationships predates ProtSent. For instance, ProtTucker uses hierarchical CATH labels to improve protein embedding similarity and distant-homology recognition. At a minimum, the authors need to better distinguish the paper's contribution. In particular, is the contribution the combination of relation types, end-to-end backbone updating, the scale of the improvements, or generalization beyond the supervised relationships?

(2) Pfam membership, structural clustering, and STRING interactions encode different biological relationships. Family membership may reflect homology. Structural clusters encode structural resemblance. Interaction data capture protein relationships that are not necessarily structurally or evolutionarily similar. The authors need to clarify what a shared embedding space is supposed to preserve, as well as how the space handles conflicting or non-equivalent supervision signals.

(3) Better retrieval alongside weaker linear-probe performance raises the question of whether ProtSent improves general-purpose representations or primarily adapts them to the geometry that is favored by the training objectives. The MaxSim results further suggest that the choice of pooling and readout really affect the outcome. Without separating these, the paper cannot confidently attribute the gains to a general improvement in the underlying representations as stated.

(4) It is good that the authors acknowledge that fold-level overlap between training corpora and SCOPe-40 or CATH test sets has not been established through appropriate filtering. However, the concern now is that if related structural examples occur in the training and evaluation data, then the reported results overestimate generalization to truly unseen structural families or folds.

(5) Some ablations in the paper use an initial configuration that is differnt from the final configuration. As a result, differences between ablation results may not identify the effects of individual supervision sources or training decisions. The paper needs controlled comparisons using otherwise matched settings.

(6) The final configuration is selected using the same benchmark suite on which it is evaluated. Each model has only one reported training run. The 23-task aggregate is descriptive rather than a basis for statistical inference. These limitations make it difficult to assess factors such as sensitivity to seeds, selection decisions, or benchmark composition.

(7) In the DMS results, the authors acknowledge that fitness supervision can conflate high fitness with WT likeness, and that the corpus is not fully filtered for contamination. These issues complicate any claim that the optional fitness objective improves biological representation. Instead, it could potentially introduce a particular preference into the embedding space.

(8) The paper is difficult to read. Results are repeatedly followed with a short, emphatic conclusion that give the paper an unnatural rhythm. Many sentences have an overly dramatic feel to them, ending with active verbs. This can obscure the distinction between what an experiment demonstrates vs. what the authors infer from it.

Questions for authors

What is the paper's principal contribution relative to ProtTucker and other prior work on contrastive protein embeddings? Is the novelty primarily the use of multiple biological relation types, the end-to-end updating of the backbone, the scale of the retrieval gains, or generalization beyond the supervision sources?

How does ProtSent compare with the most relevant recent sequence–structure contrastive methods under matched backbones, data access, training budgets, and evaluation protocols?

What biological notion of similarity is ProtSent intended to represent? Given that family membership, structural clustering, and protein interactions encode different relationships, what justifies optimizing them within one embedding space. How, if at all, is the resulting geometry evaluated for coherence?

Why does contrastive post-training improve retrieval but reduce linear-probe performance? Do the authors interpret this as a specialization trade-off, a change in embedding geometry, or a limitation of mean pooling? And also, what evidence for any?

I wonder how much of the reported retrieval improvement comes from contrastive training itself rather than from the choice of pooling or readout? In particular, can the authors compare mean pooling and MaxSim or other residue-level aggregation under matched training conditions?

What does each supervision source contribute independently? What benefit from combining them? Are there controlled, matched-config ablations for each source, relevant pairs of sources, and full combination? Do different sources help different benchmarks or produce conflicting effects?

Does the training procedure produce one coherent geometry or a compromise among source-specific objectives? Since the model receives different kinds of supervision across training steps, can the authors show how performance on one relation type changes when another is introduced? It would be good to undersatnd this.

What is the evidence that full-backbone fine-tuning is necessary? How might it compare with training a projection head or another lightweight adapter under matched compute and data budgets?

Can the authors establish the degree of overlap between the training corpora and SCOPe-40 and CATH test sets and so evaluate on a split designed to exclude close structural relationships?

Can the authors report multiple training runs, uncertainty estimates at least for key metrics, and a clearer separation between configuration selection and final evaluation?

2 comments

Thank you for the detailed review. We agree that retrieval gains should not be presented as a general improvement in protein representations, and the revision is framed that way.

Q1. Contribution relative to ProtTucker. We do not claim novelty for contrastive training. ProtTucker keeps ProtT5 frozen and trains a two-layer head on curated CATH labels. ProtSent updates the whole backbone, combines three relation types (Pfam families, AlphaFold DB structural clusters, STRING interactions) and never sees a CATH or SCOPe label. Pfam families are annotated, so we now say "no structural labels" instead of "label-free". On ProtTucker's CATH benchmark, ProtSent on ESM-C reaches 70.7% at the H level. Published ProtTucker reaches 70% on ESM-1b and 76% on ProtT5.

Q2. Sequence–structure baselines. We did not run matched versions of these methods. They differ in backbone, training data and inputs, and Foldseek needs a structure for each query. The revision reports absolute numbers and states these differences.

Q3. What similarity is learned? A positive pair shares a family, a structural cluster or an interaction. These are different relations, and the intended geometry is nested: distance grows with evolutionary distance, and interaction partners sit closer than random pairs. Table 1 shows that distance decreases monotonically with shared SCOPe levels, which the base model's does not. In the same model, Bernett PPI 3-NN AUC rises from 0.572 to 0.594 at 150M.

Q4. Retrieval vs. linear probes. We see this as specialization for pooled-embedding similarity, and Section 5.4 reports the linear-probe loss as the main negative result. The space does not collapse: the participation ratio rises from 7.9 to 52.5. We will add intermediate-layer probes to test where the property information goes.

Q5. Training vs. readout. With the same MaxSim readout on both models, ProtSent leads its base by +0.139 [+0.124, +0.154] MAP at 35M and +0.144 [+0.130, +0.158] at 150M (Table 23), so the gain is not an artifact of mean pooling. Whitening is the stronger control, and the global comment reports it. It matches ProtSent at 150M and not at 35M or on ESM-C.

Q6. Source contributions. Table 5 ran on the initial configuration, with the faulty hard negatives. It is labelled that way, and we use it only for the direction of each source's effect. Matched single-source runs on the final recipe are not available, and we state this as a limitation.

Q7. One geometry or a compromise? The initial-configuration ablations show small trade-offs. Removing any relation source lowers the mean gain (from +6.7% to between +3.2% and +5.9%). The one cross-source cost: removing STRING hurts PPI but slightly helps remote homology (+40.5% → +43.3%).

Q8. Is full-backbone tuning necessary? We have not trained a head on the frozen base, so we do not claim it is. The whitening result shows that at ESM-2 150M a linear map recovers most of the gain. At 35M and on ESM-C it does not.

Q9. Training overlap. See the global comment. Identity strata are not informative here, since only 12 SCOPe-40 queries and 1 CATH query are below 40% identity to our training sources. We instead removed each query's own family from the gallery, so that every hit comes from another family. ProtSent still leads its base, by +0.134, +0.195 and +0.496 superfamily R@1 at the three sizes, with intervals excluding zero. The gain also holds at the fold level, with the query's own superfamily removed. We do not claim transfer to unseen folds.

Q10. Selection and uncertainty. SCOPe-40 was not among the ablation tasks used to choose the configuration (Table 24), and CATH was first run after both ESM-2 models were trained. Our bootstrap intervals measure variation across queries, not across training seeds. One run exists per model, and we state this as a limitation. On CATH we added paired McNemar tests (see our reply to BCpg, Q7).

DMS and writing. The primary ESM-2 models never train on DMS. Only ProtSent-ESMC does, and its gains are unchanged on queries with no DMS overlap (reply to BCpg, Q5). We shortened the results text and separated measurements from interpretation.

Reviewer_rM2QOpenReview ↗

Keeping overall score

Thank you for the detailed rebuttal. The gallery-removal control is convincing. It addresses my original concern about overlap better than the identity-stratified analysis. Excluding each query's own family and superfamily a stronger test than the control I had previously suggested. The MaxSim comparison also makes clear that the improvement does not depend on mean pooling. I also appreciate the corrections to the ProtTucker description and to the “+12 against +16” claim. To acknowledge these efforts by the authors, I have raised my soundness score accordingly.

However, I am keeping my overall rating at 2 for three main reasons: (1) whitened base now matches ProtSent at ESM-2 150M on SCOPe-40 R@1 and MAP, and the paired tests on CATH show a tie. This evidence supports a narrower claim than the submission originally made, with the remaining advantage on 35M and ESM-C. (2) matched comparisons against ProtTucker, DHR, and Foldseek are still lacking. In absolute terms, the H-level result also remains below the published ProtTucker and profile HMMER results. (3) matched single-source runs using the final recipe are also still missing leaving multi-relation claim untested.

I do want to recognize the quality of the original writing and rebuttal. The paper is very well written and clear. The retrieval-versus-linear-probe trade-off is also an interesting result on its own regardless of the outcome of this submission.

Comments on the paper

1 comment
AuthorsOpenReview ↗

global comment for rebuttal

Thank you for the careful reviews. We ran the main controls you asked for. All intervals below are paired 95% bootstrap intervals over queries.

1. Overlap with training data. Identity filtering cannot settle this. Against our training sources, only 12 of 2,207 SCOPe-40 queries and 1 of 219 CATH queries fall below 40% identity, because AlphaFold DB covers almost all of UniProt. The frozen base saw the same proteins in pretraining, so both models share this exposure.

The sharper concern is that SCOPe families track the Pfam families we train on. We tested this by removing the nearer level from the gallery. At the superfamily level the query's own family is removed, so every hit comes from another family. At the fold level its own superfamily is removed.

ProtSent minus frozen baseESM-2 35MESM-2 150MESM-C 300M
Superfamily R@1+0.134 [+0.111, +0.157]+0.195 [+0.172, +0.219]+0.496 [+0.472, +0.522]
Fold R@1+0.063 [+0.034, +0.093]+0.142 [+0.113, +0.172]+0.284 [+0.253, +0.315]
Fold MAP, against whitened base+0.040 [+0.030, +0.050]+0.041 [+0.030, +0.052]+0.132 [+0.122, +0.143]

The gain holds when same-family and same-superfamily hits are excluded. Pfam supplies 2.2% of the 35M model's training pairs. We corrected the abstract: only the remote-homology and PPI test sets were decontamination targets.

2. Whitening. We added a centered and whitened frozen base, fit without labels.

ProtSent minus whitened baseSCOPe-40 R@1SCOPe-40 MAPCATH H (points)
ESM-2 35M+0.070 [+0.048, +0.093]+0.055 [+0.041, +0.069]+2.0 [−5.3, +9.3]
ESM-2 150M+0.018 [−0.002, +0.038]+0.011 [−0.001, +0.024]−0.7 [−6.7, +4.7]
ESM-C 300M+0.137 [+0.115, +0.159]+0.189 [+0.173, +0.206]+34.7 [+26.7, +42.7]

Whitening explains a large part of the ESM-2 gain, and we narrow the claim accordingly. Post-training adds retrieval structure beyond whitening at 35M on SCOPe-40 and everywhere on ESM-C. At 150M its gain is matched by whitening on R@1 and MAP, and on CATH both ESM-2 models tie the whitened base. When both models are whitened, ProtSent leads on SCOPe-40 at every size. The revision adds whitened rows to Tables 1, 2 and 4.

3. Relation to ProtTucker. ProtTucker keeps ProtT5 frozen and trains a two-layer head on curated CATH labels. Our related work wrongly said it fine-tunes ProtT5, and we corrected this. ProtSent updates the whole backbone, combines several relation types and never sees a CATH or SCOPe label. On ProtTucker's CATH benchmark, ProtSent on ESM-C reaches 70.7% at the H level. Published ProtTucker reaches 70% on ESM-1b and 76% on ProtT5.

4. Scope and limits. Our findings are about retrieval, clustering and nearest-neighbour transfer from frozen embeddings. Under a trained linear probe ProtSent trails its backbone, as structure distillation (ISM-C) also does. One training run exists per model, and the single-source ablations ran on the initial configuration only. The revision will state both as limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.