- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
Functional search/retrieval with proteins. Authors show their method, using generative models that incorporate protein sequence and text outperform others, including foldseek, other protein language and structure models. They present a benchmark of GO annotations retrieval from swissprot of sequences with structure: FURETON, a benchmark for retrieving functionally related proteins, using GO annotations from 89,116 Swiss-Prot proteins. They compare sequence and structure search, protein language models, and text-based retrieval methods. Generated functional descriptions perform best, with Evolla reaching 7.65% MAP, compared with 5.71% for ProtST and 4.65% for the best contrastively adapted protein language model and 11.55 for expert annotated swissprot text as ground truth.
Critical strengths
- Good broad evaluation of models, and task is reasonable (many GO annots, swissprot, etc). Use of text also good example.
- Practical relevant problem and results. sequence and structure split is a good idea (although temporal split as in CAFA may be more appropriate here, see below).
Critical weaknesses
-
Linked codebase clearly gives author identity and institution; breaks anonymity. :(
-
Text model memorization: The main finding is that text-generating models outperform other protein representations. (protein function annotation datasets already exist, e.g. cafa and works by others). However, these models were trained on Swiss-Prot-derived functional descriptions, and the benchmark also uses Swiss-Prot proteins and their GO annotations. The paper does not check whether the models had already seen the test proteins and learned their functions during training. They might therefore be recalling known annotations rather than inferring functions from sequence or structure. The sequence/structure train-test split within FURETON does not prevent this. Without checking the models' original training data, it is unclear how much of the reported advantage would hold for proteins whose functions were not previously known to the models. A temporal split (as in cafa) could solve this or from scratch training of the models. This is a serious limitation of the paper's main conclusion. It is not discussed or evaluated.
Questions for authors
-
Were the FURETON proteins (or families i.e close homologs), together with their descriptions or annotations or names, included in the original training data of Evolla, Prot2Text-V2, or InstructBioMol? Can you report the overlap for each model and results if using from scratch models instead of pretrained?
-
Can you repeat the main comparison on sequences whose functions were not included in the models' training data (i.e stratified by functions)? Does the advantage over naive / text based (e.g. InterProScan) remain?
-
What was the species distribution and overlap?
6 comments
Part 2. Interpreting the retrieval results and the request for training from scratch
Although the reviewer specifically asks about Evolla, Prot2Text-V2, and InstructBioMol, our analysis suggests that potential protein-level overlap is widespread across the evaluated baselines. Such exposure does not by itself invalidate their use as baselines: pretraining on individual proteins or protein–text pairs is distinct from evaluating protein-to-protein functional-similarity retrieval. Prior exposure to individual proteins does not necessarily imply accurate recovery of their pairwise functional relationships. A similar distinction arises in protein–protein interaction prediction, where previously seen proteins can form new evaluation pairs. (See Appendix A.2 for the protein-to-protein retrieval task and Section 6 for the limitation concerning prior exposure.)
Our main comparison asks how well methods that incorporate functional text retrieve functionally related proteins under joint controls on sequence and structural similarity. ProtST learns from paired proteins and functional descriptions and uses only protein embeddings for retrieval. Evolla generates functional descriptions and ranks candidates by the similarity of their text embeddings. On the main FURETON test set, ProtST-ESM2 and Evolla achieved 5.42% and 7.39% MAP, respectively, compared with 4.72% for the strongest contrastively adapted protein encoder. These results suggest that functional text can support retrieval in both settings, although performance varies across models and prior exposure to functional annotations may have contributed to the observed differences. (See Section 4.1 for the retrieval methods, Section 5.1 and Table 2 for overall MAP, and Section 6 for possible pretraining exposure.)
Regarding training from scratch, reproducing these models' large-scale pretraining is beyond our available computational resources, and we therefore do not report from-scratch results. To address part of the reviewer's concern, we additionally conducted a temporal evaluation, described in the response below, to examine performance on more recent data. (For the temporal evaluation, see Section 5.1 and Appendix F, Tables 18–19.)
Can you repeat the main comparison on sequences whose functions were not included in the models' training data (i.e stratified by functions)? Does the advantage over naive / text based (e.g. InterProScan) remain?
Part 1. Temporal evaluation design and query eligibility
We thank the reviewer for this important question. We added a temporal evaluation to examine performance on newly added Swiss-Prot proteins. (See Section 5.1 and Appendix F.1 for the temporal evaluation and the definition of newly added entries.)
Using Swiss-Prot 2025_03 as a common temporal reference, we identified 537 proteins present in 2026_02 but absent from 2025_03 that satisfy the FURETON inclusion criteria. These proteins occur in 308 clusters (see Appendix F.1 and Table 18):
| Cluster size | Clusters | Total proteins | New proteins |
|---|---|---|---|
| 1 | 124 | 124 | 124 |
| 2–5 | 60 | 184 | 77 |
| 6–10 | 22 | 178 | 39 |
| 11–50 | 42 | 1,012 | 67 |
| 51–99 | 13 | 917 | 34 |
| 100–500 | 38 | 7,849 | 87 |
| >500 | 9 | 11,654 | 109 |
| Total | 308 | 21,918 | 537 |
We selected all 261 clusters containing fewer than 100 proteins and used their complete membership as the query set: 2,415 proteins, including 341 new and 2,074 older entries. The remaining 89,373 proteins formed a shared candidate database, including 196 new entries from the larger clusters. No cluster spans the query and candidate sets. (See Appendix F.1, p. 31, for the cluster selection and the composition of the query set and gallery.)
All methods used this same split and the original FURETON relevance definition and evaluation procedure. We report MAP for all temporal queries and separately for new-protein queries. Under the ALL criterion, positives are the union of those defined by Molecular Function (MF), Biological Process (BP), and Cellular Component (CC), rather than an average of three separate MAP scores. Queries without positive candidates are excluded from the macro-average, leaving 1,654 evaluable queries overall and 206 evaluable new-protein queries. (See Appendix F.1, p. 31, for query eligibility and the ALL criterion, and Table 19 for the MAP results.)
Were the FURETON proteins (or families i.e close homologs), together with their descriptions or annotations or names, included in the original training data of Evolla, Prot2Text-V2, or InstructBioMol? Can you report the overlap for each model and results if using from scratch models instead of pretrained?
Part 1. Accession overlap across Swiss-Prot releases
We thank the reviewer for raising the issue of potential overlap between FURETON and the models' pretraining data. We examined the reported data sources and compared the corresponding Swiss-Prot reference releases with Swiss-Prot 2026_02, the source release of FURETON. (See Section 3.1 for the FURETON source release and Section 6 for the discussion of possible pretraining exposure.)
Evolla explicitly reports Swiss-Prot 2023_03 [1]. For Prot2Text-V2, release information differs across the arXiv paper, conference paper, and linked dataset card; we use the dataset card's 2022_04 release as a conditional reference [2], [3], [4]. For InstructBioMol, 2023_02 is inferred from the upstream UniProtQA dataset size reported by BioMedGPT [5], [6]. (Section 6 discusses possible pretraining exposure. The model-specific release information comes from references [1]–[6] above.)
ESM-2 and ESM-C report UniRef releases 2021_04 and 2023_02, respectively [7], [8]; we use the contemporaneous Swiss-Prot releases as proxies for this database-release comparison. ProtST does not specify the Swiss-Prot release used for ProtDescribe. We use 2022_05 as a publication-based upper-bound reference, since it was the latest release available before the initial manuscript's submission on January 28, 2023 [9]. ProteinDT reports a SwissProtCLAP data date of September 8, 2022, from which we infer Swiss-Prot 2022_03 [10]. For ProTrek, 2023_03 is inferred from the reported source-database size [11]. Inferred releases are conditional references. (See Section 4.1 and Table 4 for the evaluated encoders. The release assignments and proxies are documented in this response.)
The table compares primary accessions in these releases with the 575,503 entries in Swiss-Prot 2026_02 [12]. Overlap is the number of shared primary accessions divided by 575,503. Historical release counts are available in the UniProt archive. (See Appendix A.3, Table 6, for the 575,503 reviewed entries in the reference release.)