acceptodds
Under review as a conference paper at ICLR 2027

Lost in Sequence, Found in Text: Retrieving Functionally Related Proteins Under Joint Sequence and Structure Similarity Controls

Abstract

Convergent evolution can give rise to proteins with similar functions despite dissimilar sequences and structures, motivating functional retrieval beyond conventional similarity search. Yet the ability of existing methods to recover such functional relationships remains insufficiently evaluated under joint controls on sequence and structural similarity. We introduce Fureton, a FUnctional RETrieval benchmark for this setting built from 89,116 reviewed UniProtKB/Swiss-Prot proteins with AlphaFold-predicted structures and selected Gene Ontology (GO) annotations. Designed to assess functional retrieval beyond conventional similarity, the benchmark controls both sequence and structural similarity between query–target pairs and defines relevance through informative shared GO ancestors. We evaluate five families of methods in zero-shot settings: conventional sequence-, structure-, and signature-based search, protein sequence language models, protein structure-aware language models, protein–text contrastive models, and protein-to-text generative models. We additionally adapt protein language models by training projection heads with a contrastive objective while keeping the pretrained backbones frozen. This contrastive adaptation improves retrieval across Molecular Function, Biological Process, Cellular Component, and the overall task, suggesting that it can make pretrained protein representations more effective for functional retrieval. Nevertheless, the best adapted model achieves 4.65% MAP in overall task, compared with 7.65% for retrieval using protein-to-text generative models. These results highlight functional text as a promising representation for retrieving functionally related proteins under joint sequence and structural similarity controls. Code, data, and evaluation protocols are available at https://anonymous.4open.science/r/FURETON-EA8D.

Primary area:
datasets and benchmarks
Keywords:
Protein function retrieval, Text-mediated protein search, Joint sequence–structure similarity control, Protein language models, Contrastive adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hMUTOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

Functional search/retrieval with proteins. Authors show their method, using generative models that incorporate protein sequence and text outperform others, including foldseek, other protein language and structure models. They present a benchmark of GO annotations retrieval from swissprot of sequences with structure: FURETON, a benchmark for retrieving functionally related proteins, using GO annotations from 89,116 Swiss-Prot proteins. They compare sequence and structure search, protein language models, and text-based retrieval methods. Generated functional descriptions perform best, with Evolla reaching 7.65% MAP, compared with 5.71% for ProtST and 4.65% for the best contrastively adapted protein language model and 11.55 for expert annotated swissprot text as ground truth.

Critical strengths
  1. Good broad evaluation of models, and task is reasonable (many GO annots, swissprot, etc). Use of text also good example.
  2. Practical relevant problem and results. sequence and structure split is a good idea (although temporal split as in CAFA may be more appropriate here, see below).
Critical weaknesses
  1. Linked codebase clearly gives author identity and institution; breaks anonymity. :(

  2. Text model memorization: The main finding is that text-generating models outperform other protein representations. (protein function annotation datasets already exist, e.g. cafa and works by others). However, these models were trained on Swiss-Prot-derived functional descriptions, and the benchmark also uses Swiss-Prot proteins and their GO annotations. The paper does not check whether the models had already seen the test proteins and learned their functions during training. They might therefore be recalling known annotations rather than inferring functions from sequence or structure. The sequence/structure train-test split within FURETON does not prevent this. Without checking the models' original training data, it is unclear how much of the reported advantage would hold for proteins whose functions were not previously known to the models. A temporal split (as in cafa) could solve this or from scratch training of the models. This is a serious limitation of the paper's main conclusion. It is not discussed or evaluated.

Questions for authors
  1. Were the FURETON proteins (or families i.e close homologs), together with their descriptions or annotations or names, included in the original training data of Evolla, Prot2Text-V2, or InstructBioMol? Can you report the overlap for each model and results if using from scratch models instead of pretrained?

  2. Can you repeat the main comparison on sequences whose functions were not included in the models' training data (i.e stratified by functions)? Does the advantage over naive / text based (e.g. InterProScan) remain?

  3. What was the species distribution and overlap?

6 comments

Were the FURETON proteins (or families i.e close homologs), together with their descriptions or annotations or names, included in the original training data of Evolla, Prot2Text-V2, or InstructBioMol? Can you report the overlap for each model and results if using from scratch models instead of pretrained?

Part 1. Accession overlap across Swiss-Prot releases

We thank the reviewer for raising the issue of potential overlap between FURETON and the models' pretraining data. We examined the reported data sources and compared the corresponding Swiss-Prot reference releases with Swiss-Prot 2026_02, the source release of FURETON. (See Section 3.1 for the FURETON source release and Section 6 for the discussion of possible pretraining exposure.)

Evolla explicitly reports Swiss-Prot 2023_03 [1]. For Prot2Text-V2, release information differs across the arXiv paper, conference paper, and linked dataset card; we use the dataset card's 2022_04 release as a conditional reference [2], [3], [4]. For InstructBioMol, 2023_02 is inferred from the upstream UniProtQA dataset size reported by BioMedGPT [5], [6]. (Section 6 discusses possible pretraining exposure. The model-specific release information comes from references [1]–[6] above.)

ESM-2 and ESM-C report UniRef releases 2021_04 and 2023_02, respectively [7], [8]; we use the contemporaneous Swiss-Prot releases as proxies for this database-release comparison. ProtST does not specify the Swiss-Prot release used for ProtDescribe. We use 2022_05 as a publication-based upper-bound reference, since it was the latest release available before the initial manuscript's submission on January 28, 2023 [9]. ProteinDT reports a SwissProtCLAP data date of September 8, 2022, from which we infer Swiss-Prot 2022_03 [10]. For ProTrek, 2023_03 is inferred from the reported source-database size [11]. Inferred releases are conditional references. (See Section 4.1 and Table 4 for the evaluated encoders. The release assignments and proxies are documented in this response.)

The table compares primary accessions in these releases with the 575,503 entries in Swiss-Prot 2026_02 [12]. Overlap is the number of shared primary accessions divided by 575,503. Historical release counts are available in the UniProt archive. (See Appendix A.3, Table 6, for the 575,503 reviewed entries in the reference release.)

ModelSwiss-Prot reference releaseFull-release entriesAccession overlap
Evolla2023_03; reported569,79398.99%
Prot2Text-V22022_04; conditional568,36398.73%
InstructBioMol2023_02; inferred569,51698.94%
ESM-22021_04; UniRef proxy565,92898.30%
ESM-C2023_02; UniRef proxy569,51698.94%
ProtST2022_05 or earlier (publication-based upper bound)568,74498.83%
ProteinDT2022_03; inferred568,00298.67%
ProTrek 35M / 650M2023_03; inferred569,79398.99%

Part 2. Interpreting the retrieval results and the request for training from scratch

Although the reviewer specifically asks about Evolla, Prot2Text-V2, and InstructBioMol, our analysis suggests that potential protein-level overlap is widespread across the evaluated baselines. Such exposure does not by itself invalidate their use as baselines: pretraining on individual proteins or protein–text pairs is distinct from evaluating protein-to-protein functional-similarity retrieval. Prior exposure to individual proteins does not necessarily imply accurate recovery of their pairwise functional relationships. A similar distinction arises in protein–protein interaction prediction, where previously seen proteins can form new evaluation pairs. (See Appendix A.2 for the protein-to-protein retrieval task and Section 6 for the limitation concerning prior exposure.)

Our main comparison asks how well methods that incorporate functional text retrieve functionally related proteins under joint controls on sequence and structural similarity. ProtST learns from paired proteins and functional descriptions and uses only protein embeddings for retrieval. Evolla generates functional descriptions and ranks candidates by the similarity of their text embeddings. On the main FURETON test set, ProtST-ESM2 and Evolla achieved 5.42% and 7.39% MAP, respectively, compared with 4.72% for the strongest contrastively adapted protein encoder. These results suggest that functional text can support retrieval in both settings, although performance varies across models and prior exposure to functional annotations may have contributed to the observed differences. (See Section 4.1 for the retrieval methods, Section 5.1 and Table 2 for overall MAP, and Section 6 for possible pretraining exposure.)

Regarding training from scratch, reproducing these models' large-scale pretraining is beyond our available computational resources, and we therefore do not report from-scratch results. To address part of the reviewer's concern, we additionally conducted a temporal evaluation, described in the response below, to examine performance on more recent data. (For the temporal evaluation, see Section 5.1 and Appendix F, Tables 18–19.)

Can you repeat the main comparison on sequences whose functions were not included in the models' training data (i.e stratified by functions)? Does the advantage over naive / text based (e.g. InterProScan) remain?

Part 1. Temporal evaluation design and query eligibility

We thank the reviewer for this important question. We added a temporal evaluation to examine performance on newly added Swiss-Prot proteins. (See Section 5.1 and Appendix F.1 for the temporal evaluation and the definition of newly added entries.)

Using Swiss-Prot 2025_03 as a common temporal reference, we identified 537 proteins present in 2026_02 but absent from 2025_03 that satisfy the FURETON inclusion criteria. These proteins occur in 308 clusters (see Appendix F.1 and Table 18):

Cluster sizeClustersTotal proteinsNew proteins
1124124124
2–56018477
6–102217839
11–50421,01267
51–991391734
100–500387,84987
>500911,654109
Total30821,918537

We selected all 261 clusters containing fewer than 100 proteins and used their complete membership as the query set: 2,415 proteins, including 341 new and 2,074 older entries. The remaining 89,373 proteins formed a shared candidate database, including 196 new entries from the larger clusters. No cluster spans the query and candidate sets. (See Appendix F.1, p. 31, for the cluster selection and the composition of the query set and gallery.)

All methods used this same split and the original FURETON relevance definition and evaluation procedure. We report MAP for all temporal queries and separately for new-protein queries. Under the ALL criterion, positives are the union of those defined by Molecular Function (MF), Biological Process (BP), and Cellular Component (CC), rather than an average of three separate MAP scores. Queries without positive candidates are excluded from the macro-average, leaving 1,654 evaluable queries overall and 206 evaluable new-protein queries. (See Appendix F.1, p. 31, for query eligibility and the ALL criterion, and Table 19 for the MAP results.)

Reviewer_DpfjOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

FURETON benchmarks functional protein retrieval while controlling sequence and structure similarity across splits. It groups 89116 Swiss-Prot proteins using MMseqs2 and Foldseek matches and defines relevance through shared informative GO ancestors. Methods retrieve from a common gallery of 88616 proteins. The evaluation covers conventional search, protein embeddings, protein-text models, and generated descriptions. Evolla reaches better overall MAP. Projection-head adaptation also improves the evaluated encoders.

Critical strengths
  1. The shared retrieval protocol and joint sequence and structure controls are useful. The paper describes the filtering, split construction, and effective query counts clearly and provides a public implementation.

  2. The comparison covers a broad range of methods. The text-encoder analysis and query-level adaptation study help explain the results, including why no model wins on every metric.

Critical weaknesses
  1. Relevant work is missing from the discussion. ProteinCLIP and ProtCLIP align protein embeddings with functional text. Aligning Proteins and Language studies structure-text retrieval, while ProCyon supports protein retrieval and functional description generation. These relate directly to the comparison of embeddings and generated descriptions. Please explain which methods can use the same protocol. Rethinking Text-based Protein Understanding also reports leakage in protein-text benchmarks, making pretraining overlap relevant here. Removing GO identifiers at inference does not rule out prior exposure to Swiss-Prot annotations, although such overlap has not been established here.

  2. The similarity controls still allow remote homologs and partial-domain matches across splits. Matches above 50% sequence identity or TM-score 0.5 are excluded only when coverage reaches 80% for both proteins. The GO threshold and annotation coverage also affect relevance. Only 364 of 500 test queries contribute to overall metrics, and 74 to CC. These choices are documented, but it is unclear whether the method ranking would hold under different thresholds or splits.

Questions for authors
  1. What is known about overlap between the test proteins or annotations and the models' pretraining data? Is any existing non-overlap analysis available?
  2. Are there existing analyses of remaining homology or sensitivity to the sequence, structure, coverage, and GO thresholds?
  3. Can you report bootstrap intervals for the main MAP differences, especially for CC's 74 effective queries?
  4. Which of the related methods above can be evaluated under the same protocol, and what prevents comparison with the others?
5 comments

What is known about overlap between the test proteins or annotations and the models' pretraining data? Is any existing non-overlap analysis available?

Part 1. Accession overlap across Swiss-Prot releases

We thank the reviewer for raising the issue of potential overlap between FURETON and the models' pretraining data. We examined the reported data sources and compared the corresponding Swiss-Prot reference releases with Swiss-Prot 2026_02, the source release of FURETON. (See Section 3.1 for the FURETON source release and Section 6 for the discussion of possible pretraining exposure.)

Evolla explicitly reports Swiss-Prot 2023_03 [1]. For Prot2Text-V2, release information differs across the arXiv paper, conference paper, and linked dataset card; we use the dataset card's 2022_04 release as a conditional reference [2], [3], [4]. For InstructBioMol, 2023_02 is inferred from the upstream UniProtQA dataset size reported by BioMedGPT [5], [6].(Section 6 discusses possible pretraining exposure. The model-specific release information comes from references [1]–[6] above.)

ESM-2 and ESM-C report UniRef releases 2021_04 and 2023_02, respectively [7], [8]; we use the contemporaneous Swiss-Prot releases as proxies for this database-release comparison. ProtST does not specify the Swiss-Prot release used for ProtDescribe. We use 2022_05 as a publication-based upper-bound reference, since it was the latest release available before the initial manuscript's submission on January 28, 2023 [9]. ProteinDT reports a SwissProtCLAP data date of September 8, 2022, from which we infer Swiss-Prot 2022_03 [10]. For ProTrek, 2023_03 is inferred from the reported source-database size [11]. Inferred releases are conditional references. (See Section 4.1 and Table 4 for the evaluated encoders. The release assignments and proxies are documented in this response.)

The table compares primary accessions in these releases with the 575,503 entries in Swiss-Prot 2026_02 [12]. Overlap is the number of shared primary accessions divided by 575,503. Historical release counts are available in the UniProt archive. (See Appendix A.3, Table 6, for the 575,503 reviewed entries in the reference release.)

ModelSwiss-Prot reference releaseFull-release entriesAccession overlap
Evolla2023_03; reported569,79398.99%
Prot2Text-V22022_04; conditional568,36398.73%
InstructBioMol2023_02; inferred569,51698.94%
ESM-22021_04; UniRef proxy565,92898.30%
ESM-C2023_02; UniRef proxy569,51698.94%
ProtST2022_05 or earlier (publication-based upper bound)568,74498.83%
ProteinDT2022_03; inferred568,00298.67%
ProTrek 35M / 650M2023_03; inferred569,79398.99%

Part 2. Retrieval on newly added Swiss-Prot entries

We added a temporal evaluation to examine performance on proteins newly added to Swiss-Prot. Specifically, we evaluated how well each method retrieved functionally similar proteins for queries whose primary accessions were absent from a common earlier Swiss-Prot release. (See Section 5.1 and Appendix F.1 for the temporal evaluation and the definition of newly added entries.)

Using Swiss-Prot 2025_03 as a common temporal reference, we identified 537 proteins present in 2026_02 but absent from 2025_03 that satisfy the FURETON inclusion criteria. These proteins occur in 308 clusters (see Appendix F.1 and Table 18):

Cluster sizeClustersTotal proteinsNew proteins
1124124124
2–56018477
6–102217839
11–50421,01267
51–991391734
100–500387,84987
>500911,654109
Total30821,918537

We selected all 261 clusters containing fewer than 100 proteins and used their complete membership as the query set: 2,415 proteins, including 341 new and 2,074 older entries. The remaining 89,373 proteins formed a shared candidate database, including 196 new entries from the larger clusters. No cluster spans the query and candidate sets. (See Appendix F.1, p. 31, for the cluster selection and the composition of the query set and gallery.)

All methods used this same split and the original FURETON relevance definition and evaluation procedure. We report MAP for all temporal queries and separately for new-protein queries. Under the ALL criterion, positives are the union of those defined by Molecular Function (MF), Biological Process (BP), and Cellular Component (CC), rather than an average of three separate MAP scores. Queries without positive candidates are excluded from the macro-average, leaving 1,654 evaluable queries overall and 206 evaluable new-protein queries. (See Appendix F.1, p. 31, for query eligibility and the ALL criterion, and Table 19 for the MAP results.)

MethodAll temporal queries MAP (%)New-protein queries MAP (%)
Random0.150.10
MMseqs22.011.35
Foldseek2.201.58
ESM-2 8M2.171.15
ESM-2 35M2.011.58
ESM-2 150M1.521.41
ESM-2 650M1.291.70
ESM-C 300M0.851.01
ESM-C 600M0.880.94
SaProt 35M1.351.13
SaProt 650M1.411.40
ProTrek 35M2.151.99
ProTrek 650M3.983.06
ProtST-ESM1b5.063.50
ProtST-ESM25.393.93
ProtST-ProtBert4.342.52
ProteinDT-InfoNCE1.911.34
ProteinDT-EBM-NCE-11.361.10
ProteinDT-EBM-NCE-21.340.99
Evolla6.963.87
Prot2Text-V25.052.58
InstructBioMol5.232.59
InterProScan5.843.59
Swiss-Prot curated annotation text (reference)10.399.07

Most methods show lower MAP on new-protein queries than on the full temporal query set, including MMseqs2, Foldseek, and InterProScan. This suggests that generalization to newly added proteins poses a challenge across methods, rather than a limitation specific to text-mediated approaches. Nevertheless, all three protein-to-text generative models, all three ProtST variants, and both ProTrek variants retain their advantage over the listed sequence- and structure-based baselines on new queries, supporting our central finding that text-mediated representations improve protein functional-similarity retrieval. (See Appendix F.2 and Table 19 for the comparison across the two temporal query sets.)

Are there existing analyses of remaining homology or sensitivity to the sequence, structure, coverage, and GO thresholds?

We thank the reviewer for raising this point. Our choices draw on prior work in protein clustering, structure comparison, and GO-based assessment of functional similarity. (See Sections 3.2–3.3 for the sequence, structure, and GO criteria used in FURETON.)

Sequence and structure filters. Uniclust50 uses a 50% sequence identity cutoff with at least 80% sequence overlap to group proteins with similar domain architectures. FURETON uses a comparable sequence criterion: a match requires more than 50% sequence identity over an alignment covering at least 80% of each protein. This coverage requirement prevents a short shared domain from linking two otherwise dissimilar proteins. Mirdita et al., 2017 (For the FURETON sequence-match criterion, see Section 3.2 and Appendix A.4, Eq. (7).)

For structural matches, FURETON requires a Foldseek alignment-normalized TM-score above 0.5 and the same 80% coverage of each protein. The choice of 0.5 is motivated by the empirical observation that the probability of two proteins sharing a fold rises sharply around a conventional TM-score of 0.5. Guo and He likewise combined a TM-score cutoff of 0.5 with 80% coverage of both proteins in Foldseek searches using TM-align mode. In FURETON, we assign each connected component to a single split, so every protein pair detected by either the sequence or the structure filter stays in the same split. Xu and Zhang, 2010 · Guo and He, 2025 (For the FURETON structural threshold and component-disjoint split, see Section 3.2 and Appendix A.4, Eqs. (8)–(13).)

GO-based functional relevance. We measure functional similarity using the information content (IC) of the most informative GO term shared by two proteins within a GO aspect. Fan et al. used the same Resnik-based measure and, in a separate evaluation, defined cross-species protein pairs as functionally similar when they shared a GO term associated with at most 100 proteins in each species. FURETON applies this specificity principle to the benchmark corpus. On our base-10 scale, IC ≥ 3 requires a shared term to occur in at most 0.1% of the 91,788 benchmark proteins after annotation propagation. For example, sharing the term WASH complex (28 proteins) establishes relevance under the Cellular Component criterion, whereas sharing cytoskeleton (870 proteins) alone does not (Appendix A.5, Table 8). Fan et al., 2019 (For the propagated term frequencies, base-10 IC, and relevance threshold, see Appendix A.5, Eqs. (17)–(20).)

Reviewer_VB2VOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new benchmark FURETON for protein function retrieval. It focuses on proteins with very different amino-acid sequences or 3D structures that carry similar biological functions. The authors construct a dataset based on SwissProt, segment data with joint distribution and structural similarity constraints, and label protein function relevance with GO annotations. The paper systematically evaluates various existing methods and discovers that generative models producing functional text description usually led to better retrieval performance. Furthermore, the paper demonstrates that supervised contrastive adaptation with frozen pretrained backbones can improve the retrieval performance.

Critical strengths
  1. The paper addresses a biological meaningful research problem and studies protein function relevance beyond conventional sequential and structural similarities. The proposed FURETON benchmark provides an interesting perspective to evaluate protein models which was largely underexplored in the past.

  2. The systematic construction methodology of the FURETON benchmark is of potentially great value to the research community. The detailed data filtering protocol, similarity-based clustering and evaluation policy are all provided in detail, which are of great help for reproduction and further studies.

  3. The experiments and analysis cover a large and comprehensive scale of the problem. The paper compares many categories of protein models and reports different metrics. Furthermore, it studies the effect of supervised adaptation and various test queries on the performance, providing a deeper understanding on advantages and limitations of different protein representation approaches.

Critical weaknesses
  1. The definition of function relevance is partially limited. The authors rely solely on GO annotations and the clustered graph structure, which can be incomplete and lack details. Two proteins without a common GO label are not necessarily irrelevant, which may produce false negatives in the clustering. The problem is even tricker on Biological Process and Cellular Component annotations, which might affect the effectiveness of the definition of relevance on real biological functions.

  2. The reliability of sequence and structure similarity definition may require further examination. Two proteins with similar local motifs/domains/regions/structures can be of low similarity on the global level. In this case, the paper may need a bit more analysis on the positive pairs that are used for clustering and graph construction.

Questions for authors
  1. As mentioned in the “weaknesses”, could the authors provide further validation on the reliability of relevance defined by GO annotations? Maybe a small-scale human validation would be of great help, or a test with different thresholds could be designed.

  2. Since the function relevance in the benchmark is defined based on GO annotations, would the authors consider adding a baseline that retrieve functions based on GO Function Prediction label similarities? This could be helpful to further validate the advantage of text-generation approaches in the function retrieval task.

  3. The test set of the benchmark contains only 500 queries, especially that the Cellular Component task has only 74. For top-tier models like Evolla, Prot2Text-V2 and InstructBioMol where the performance on metrics are sometimes not that distinguishable, could the authors provide further statistical significance tests or confidence intervals?

3 comments

As mentioned in the “weaknesses”, could the authors provide further validation on the reliability of relevance defined by GO annotations? Maybe a small-scale human validation would be of great help, or a test with different thresholds could be designed.

We conducted a GPT-assisted assessment of 100 individual-GO triplets (30 MF, 40 BP, 30 CC). Query GO terms were selected by first sampling a test query protein and then sampling one of its direct annotations in the selected GO aspect; query GO terms were kept unique. A sampled query term was rejected if it had already been accepted or lacked a distinct positive or non-positive candidate. Candidate GO terms came from direct annotations in the original train-plus-validation gallery, within the same GO aspect. Each triplet contains a query GO term, a randomly sampled positive GO term and a randomly sampled non-positive GO term. Positive GO pairs share a frozen is_a ancestor (including self) with IC at least 3; non-positive pairs do not meet that criterion. The judge received only GO aspect, English term names and official definitions, with GO identifiers and positive/non-positive assignments withheld. Each triplet was evaluated twice in separate contexts with candidate order reversed. The eligible query-term pools contained 66 MF, 337 BP, 33 CC terms. Results are conditional on this eligibility filtering and the protein-first sampling procedure. (See Appendix A.6, pp. 20–21, and Table 9 for triplet sampling, judging, and counts by GO aspect.)

The positive GO term was preferred consistently in 100/100 triplets (100.0%). Because every observed triplet has the same binary outcome, the 95% within-aspect percentile bootstrap interval degenerates to 100.0%–100.0%. (See Appendix A.6, Table 9, for the observed agreement counts.)

The requested model was gpt-6-astra. Judgments were generated through the Codex CLI harness. We retain the full prompts, per-request inputs, model settings, raw responses and all disagreements for inspection. (See the reproducibility note in Appendix A.6, p. 21, for the requested model and the Codex CLI.)

The existing appendix also provides illustrative examples of the GO-based relevance criterion. This assessment measures agreement between the GO-based relation rule and GPT's interpretation of GO term text. (See Table 8 in Appendix A.5 for GO term examples and Table 10 in Appendix A.6 for example triplets.)

Since the function relevance in the benchmark is defined based on GO annotations, would the authors consider adding a baseline that retrieve functions based on GO Function Prediction label similarities? This could be helpful to further validate the advantage of text-generation approaches in the function retrieval task.

We added a baseline using the authors' released DeepGO-SE checkpoints to predict GO functions from sequence. For each protein and each of MF, BP and CC, we first average the raw sigmoid probabilities from the six checkpoints selected by the authors for that namespace, using sequential float32 summation followed by division by six. We then select the single highest-scoring term from each namespace, without ancestor-score propagation, and concatenate its official English name and definition from the checkpoints' frozen GO ontology in that fixed aspect order. We encode this text with the same fixed all-MiniLM-L6-v2 encoder used for the three generative baselines, then rank candidates by cosine similarity. This experiment tests a compact top-1 GO-label text representation. (See Appendix G, p. 32, for the construction and encoding of the GO text.)

All methods use the same frozen 501 test queries and 91,287 candidates, with 364/189/272/73 eligible queries for ALL/MF/BP/CC. One ranking per query is evaluated against all four criteria. The following table reports MAP in percent. (See Appendix G, pp. 32–33, and Table 20.)

MethodALLMFBPCC
DeepGO-SE top-1 GO text + MiniLM2.334.330.250.14
ESM2-3B1.421.361.292.92
Evolla7.3910.604.098.57
Prot2Text-V27.2510.892.976.04
InstructBioMol6.499.103.007.68
InterProScan + MiniLM4.547.073.649.78

We also include the ESM2-3B sequence representation as a control for the protein language model underlying the GO predictor. Differences below are baseline minus comparator, in percentage points, with paired-query 95% BCa intervals based on 50,000 resamples (seed 20261117). Whole-component paired BCa intervals are provided as a sensitivity analysis. All intervals are pointwise and conditional on the fixed protocol. (See Appendix G, pp. 32–33, for the ESM2-3B control and resampling procedure, and Table 21 for the paired-query intervals.)

Baseline minus comparatorALLMFBPCC
Evolla-5.06 [-6.38, -4.07]-6.27 [-8.80, -4.44]-3.83 [-4.89, -3.07]-8.44 [-12.76, -5.52]
Prot2Text-V2-4.92 [-6.06, -4.05]-6.56 [-8.37, -5.15]-2.71 [-3.51, -2.14]-5.90 [-10.29, -3.38]
InstructBioMol-4.16 [-5.17, -3.42]-4.77 [-6.23, -3.71]-2.75 [-3.47, -2.19]-7.54 [-13.11, -4.48]
ESM2-3B+0.91 [0.13, 1.85]+2.97 [1.43, 4.74]-1.04 [-1.36, -0.79]-2.78 [-4.26, -1.85]

For this top-1 GO-text representation, all three generative baselines achieved higher MAP in ALL, MF, BP and CC. All baseline-minus-generative-comparator pointwise 95% BCa intervals lie below zero under both paired-query and whole-component resampling. Relative to ESM2-3B, the top-1 baseline has higher MAP in ALL and MF, with paired-query intervals above zero; however, the corresponding whole-component intervals include zero (ALL: [-1.46, 8.16] percentage points; MF: [-1.69, 14.59] percentage points). These apparent ALL/MF advantages are therefore not robust to the component-resampling sensitivity analysis. (See Appendix G, Tables 20–21 and the accompanying paired-comparison results on p. 33.)

The test set of the benchmark contains only 500 queries, especially that the Cellular Component task has only 74. For top-tier models like Evolla, Prot2Text-V2 and InstructBioMol where the performance on metrics are sometimes not that distinguishable, could the authors provide further statistical significance tests or confidence intervals?

We quantified uncertainty in the main MAP differences using a paired query bootstrap as the primary analysis. For each GO task, we resample its eligible test queries with replacement, drawing the same number of queries as in the original evaluation. Each sampled query retains both methods' AP scores, and the same query draws are used for all seven comparisons within a task. We calculate each difference as 100 times the mean paired AP difference, so the units are percentage points. We use 50,000 resamples and report two-sided 95% bias-corrected and accelerated (BCa) intervals, with a delete-one-query jackknife and random seed 20261117. (See Appendix B.3, Eq. (38), for the underlying AP and MAP definitions.)

The ALL/MF/BP/CC analyses contain 364/189/272/73 eligible queries, respectively. Eligibility requires at least one relevant gallery candidate; eligible queries with zero AP remain included. ALL uses the union relevance relation and is evaluated directly, rather than by averaging the aspect scores. Models, generated descriptions, relevance labels, rankings, and the 91,287-protein gallery are held fixed. The primary analysis treats queries as independent sampling units while preserving the pairing between methods. (For the query counts, test gallery, and metric definitions, see Table 1, Appendix A.7, and Appendix B.3.)

The table reports the observed MAP difference followed by its paired-query 95% BCa interval, in percentage points. Positive differences favor the first method. Evolla denotes Evolla-10B-DPO, and CA denotes contrastive adaptation. The three generative methods and InterProScan use the same fixed all-MiniLM-L6-v2 text encoder. (See Table 15 for the model configurations and Appendix C for the shared text encoder.)

Comparison (first − second)ALLMFBPCC
Evolla − Prot2Text-V2+0.14 [−0.78, 1.17]−0.29 [−2.24, 1.99]+1.12 [0.54, 1.92]+2.54 [1.00, 5.35]
Evolla − InstructBioMol+0.90 [0.13, 1.90]+1.50 [−0.20, 3.76]+1.08 [0.41, 2.05]+0.90 [−2.22, 4.05]
Prot2Text-V2 − InstructBioMol+0.76 [0.14, 1.43]+1.79 [0.71, 2.94]−0.04 [−0.64, 0.66]−1.64 [−6.54, 0.73]
Evolla − ProtST-ESM2+1.97 [1.29, 2.88]+3.27 [1.93, 5.20]+0.86 [0.39, 1.42]+1.10 [−1.35, 3.18]
Evolla − ESM2-650M-CA+2.67 [1.54, 4.01]+5.87 [3.69, 8.31]+0.83 [0.12, 1.60]−0.57 [−3.39, 1.86]
InterProScan − Evolla−2.86 [−4.08, −1.82]−3.53 [−5.99, −1.23]−0.44 [−1.03, 0.31]+1.20 [−1.00, 4.53]
InterProScan − ESM2-650M-CA−0.19 [−1.01, 0.89]+2.34 [0.52, 4.50]+0.39 [−0.44, 1.52]+0.64 [−1.90, 5.12]
Effective queries36418927273

In ALL, Evolla and Prot2Text-V2 are not clearly separated: their difference is +0.14 [−0.78, 1.17]. Evolla and Prot2Text-V2 have positive pointwise differences relative to InstructBioMol, at +0.90 [0.13, 1.90] and +0.76 [0.14, 1.43], respectively. Evolla's differences relative to ProtST-ESM2 and ESM2-650M-CA are +1.97 [1.29, 2.88] and +2.67 [1.54, 4.01]. (See Table 2 for the individual methods’ ALL MAP scores.)

For CC's 73 eligible queries, InterProScan's differences relative to Evolla and ESM2-650M-CA are +1.20 [−1.00, 4.53] and +0.64 [−1.90, 5.12], respectively. Both intervals include zero, so its highest CC MAP remains a point-estimate ranking rather than a statistically established advantage over these two methods. (See Appendix D and Table 15 for the CC query count and the individual methods’ CC MAP scores.)

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their feedback. During a recheck, we corrected errors in benchmark construction that changed the corpus from 89,116 to 91,788 proteins. We updated the dataset statistics and reran the main experiments. The revised test set contains 501 queries and a gallery of 91,287 proteins. Of these queries, 364 have a relevant gallery protein under the combined GO criterion (ALL) and contribute to overall MAP (§§3.2–3.3, Table 1).

The revision clarifies how functional text enters retrieval. Protein–text contrastive models use functional descriptions during training and rank proteins by their protein embeddings. Description-based methods use functional text at retrieval time. The descriptions can be generated by a protein model or assembled from signature annotations such as those provided by InterProScan. In the main comparison, ProtST-ESM2 and Evolla-10B-DPO achieved 5.42% and 7.39% overall MAP, respectively. The strongest contrastively adapted protein encoder, ESM-2 650M, achieved 4.72%. Expert-curated Swiss-Prot text reached 11.04%, indicating room for improvement (§5.1, Table 2).

We added four analyses to the manuscript.

  1. On 206 evaluable queries newly added to Swiss-Prot, ProtST-ESM2, Evolla, and InterProScan-derived text achieved 3.93%, 3.87%, and 3.59% MAP. The highest MAP among conventional search methods and unadapted sequence- or structure-based encoders was 1.70% (§5.1, Appendix F, Tables 18–19).
  2. A language-model judge preferred the GO-positive candidate in both presentation orders for all 100 sampled GO term triplets. This tests agreement with judgments based on term names and definitions (Appendix A.6, Tables 9–10).
  3. Of the 70 species in the test set, 69 also occur in training, accounting for 500 of the 501 test proteins. The split holds out sequence/structure similarity components and permits shared species (Appendix A.8, Tables 11–12).
  4. Descriptions assembled from the top DeepGO-SE prediction in each GO aspect achieved 2.33% overall MAP. With the same frozen text encoder, InterProScan-derived descriptions reached 4.54% and the three generators reached 6.49–7.39% (Appendix G, Tables 20–21).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.