acceptodds
Under review as a conference paper at ICLR 2027

Beyond Static Annotation Matching: A Temporal Benchmark for Free-Form Protein Function Prediction

Abstract

Advances in large language models have enabled protein models to generate detailed natural-language descriptions of protein function. However, evaluating such free-form predictions remains challenging, as existing benchmarks are primarily built around fixed labels or static annotations. We introduce ProLong-Bench, a temporal benchmark for free-form protein function prediction that evaluates models as protein annotations develop over time. Across six representative Protein LLM systems, we observe substantial differences in performance across temporal settings. Models that perform well on existing functional annotations do not necessarily capture knowledge added later, while performance on newly added proteins shows a different pattern. These results highlight temporal evaluation as an important complement to conventional static benchmarking. We intend ProLong-Bench to provide a standardized and extensible evaluation framework for free-form protein function prediction, facilitating more comprehensive model comparison and supporting future progress in the field.

Primary area:
datasets and benchmarks
Keywords:
Protein function prediction, temporal benchmark, free-form functional annotation, large language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R5MUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ProLong-Bench, a temporal benchmark for evaluating free-form protein function prediction. The benchmark is constructed from three Swiss-Prot snapshots and considers three evaluation settings: recovery of existing functional knowledge (A.0), coverage of newly added annotations for existing proteins (A.1), and function prediction for newly added proteins (Track B). The authors evaluate six protein language model systems using a claim-level evaluation framework called FactJudge. The results show that models performing well on existing annotations often struggle to recover newly added functional information. The paper also finds that tool-augmented systems perform relatively better on temporally novel knowledge, while sequence-based models show stronger agreement with historical annotations. Additional analyses examine the relationship between model performance and historically available functional information.

Critical strengths
  1. Most existing benchmarks evaluate predictions against static functional annotations. Separating established knowledge from newly added annotations provides a useful perspective on model performance. In particular, evaluating both types of annotations for the same protein sequence helps isolate differences in functional knowledge coverage.

  2. The comparison across six systems shows that strong performance on existing annotations does not necessarily translate to better coverage of newly added knowledge. The finding that tool-augmented systems perform relatively better on temporally novel information is interesting and may help guide future model development.

Critical weaknesses
  1. Unclear definition of temporal novelty.

The paper treats functional annotations added in later Swiss-Prot releases as emerging knowledge. However, newly curated annotations do not necessarily represent newly discovered biological functions, as the underlying information may have already been available in earlier literature. This makes it difficult to interpret A.1 performance as the ability to predict new functional knowledge. Moreover, the comparison between A.1 and Track B may be affected by differences in functional specificity and annotation complexity. Although Appendix E examines annotation density and sequence-family representation, these analyses do not fully address such differences. Therefore, the lower performance on A.1 does not necessarily demonstrate that annotation novelty is more difficult than protein novelty.

  1. Limited benchmark size and evaluation validation.

The benchmark is relatively small, particularly W2 Track B, which contains only 90 proteins. This raises concerns about the reliability of model rankings and conclusions across different functional categories. In addition, FactJudge is validated using only 50 gold-support cases and 99 prediction-relevance cases. The gold-support validation covers A.0 and A.1 but does not include Track B. Reference decomposition also satisfies both completeness and atomicity criteria in only 43 of 50 cases, while the component-level scores are not independently validated. More extensive validation would be helpful to establish the reliability of the reported results.

  1. Unclear control of information access across models.

The evaluated models have different training data and inference-time resources, which may affect the fairness of the comparison. Although Table 11 reports the available temporal information, it is unclear whether these dates correspond to the exact resources used during evaluation. In particular, the paper does not clearly specify whether the models had access to additional databases, online retrieval, or cached information, or how the knowledge cutoffs of the underlying LLMs were determined. Since tool-augmented models perform better on A.1 and Track B, it is difficult to determine how much of this advantage comes from the model architecture rather than differences in information access.

  1. Overinterpretation of the historical-reference analysis.

The historical-reference analysis shows a correlation between model performance and the presence of similar functional information in historical annotations. However, this does not directly demonstrate memorization or explain the underlying mechanisms of different models. For example, PFUA's confidence interval for the performance difference between historically unseen and represented claims includes zero. In addition, as noted in Appendix F, these two groups do not necessarily contain the same proteins. Therefore, I do not think the current evidence is sufficient to conclude that some models mainly rely on repeating historical knowledge while tool-augmented models operate through a fundamentally different mechanism.

Questions for authors
  1. Appendix A defines temporal status by Swiss-Prot annotation availability. Is citation-date information available for a small sample of A.1 claims to assess how often the supporting literature predates the baseline snapshot? If not, could the paper consistently qualify its claims as later-curated annotation coverage?

  2. Beyond the aggregation checks in Appendix E, could a small matched analysis assess the A.1–Track B recall gap at comparable levels of functional specificity or contextual detail? Could the Miss comparison also be checked after matching or subsampling the number of claims per protein?

  3. Given the existing cluster-based analyses, could you report paired uncertainty for close model comparisons, particularly PFUA versus BioReason-Pro on W2 A.1 and Track B, and clarify which ordering claims are supported?

  4. Do the dates in Table 11 describe the exact resources used in these evaluation runs? Please clarify whether any additional databases, online retrieval, or caches were accessible and how their temporal boundaries were determined.

  5. Table 9 provides encouraging initial validation. Could a small model- and regime-stratified expert audit assess differential judge bias and the component-level conclusions? A second judge on the same subset would also help assess evaluator sensitivity.

6 comments
AuthorsOpenReview ↗

Response to W1 and Q1

W1 and Q1

We thank the reviewer for pointing out the ambiguity in our use of “emerging knowledge.” We will explicitly scope this term to sequence–function associations newly recorded in Swiss-Prot after the baseline, with novelty defined relative to historical sequence–function annotations.

To examine the timing of the supporting evidence, we checked 64 A.1 claims from 30 proteins:

Literature checkResult
Linked papers published by the baseline119/119
Claims with explicit pre-baseline abstract support, identified through machine-assisted checks21/64

These results show that later curation can incorporate previously published biological findings, supporting the interpretation of A.1 as later-curated annotation coverage. This does not diminish the value of the evaluation. First, the evaluated systems were not provided with the supporting papers and did not retrieve literature during inference; their predictions relied on the target sequence and the specified model and tool resources. Second, an existing protein–function association in the literature does not by itself establish a model’s ability to recover that association from the amino acid sequence and express it as a coherent functional description. This sequence-to-description capability is precisely what our benchmark evaluates.

AuthorsOpenReview ↗

Respinse to W1 and Q2

W1 and Q2

We thank the reviewer for asking whether reference content and claim counts explain the A.1–Track B gap. Lower A.1 recall persists for the four directly aligned sequence-to-text systems across three matching settings, and higher A.1 Miss persists for all six systems after equalizing claim counts.

Content matching. To test whether more specific or complex reference content explains the recall gap, we human-labeled all reference claims for 30 A.1 and 30 Track B proteins:

Reference groupHuman-labeled proteinsClaims
A.13064
Track B3083
Total60147

Experts rated each claim from 0–2 for specificity (broad functions to specified targets or reactions) and semantic complexity (single relations to additional conditions or mechanistic steps). Ratings were averaged within each protein and used for one-to-one matching, maximizing pair counts and minimizing content differences without using model performance. A maximum difference of 0.5 per attribute yielded 22 pairs; stricter (0.25) and exact matching yielded 17 and 13 pairs, respectively. Primary matching reduced the content imbalance:

Mean reference-content ratingBefore matching: A.1 / Track BAfter primary matching: A.1 / Track B
Specificity1.689 / 1.3131.667 / 1.495
Semantic complexity0.711 / 0.4400.583 / 0.532

For each matched subset, we averaged each system's existing per-protein recall separately for A.1 and Track B. Exact matching requires identical ratings on both attributes. All difference columns report Track B recall minus A.1 recall:

SystemPrimary: A.1 recallPrimary: Track B recallPrimary difference (22 pairs)Stricter difference (17 pairs)Exact-match difference (13 pairs)
BioMatrix0.10700.2451+0.1381+0.2027+0.1843
Prot2Text-V20.09190.2223+0.1305+0.1532+0.1298
BioT5+0.08620.1051+0.0189+0.0245+0.0240
EvoLLa0.08240.1460+0.0636+0.0507+0.0881
PFUA0.23960.2968+0.0572+0.1027−0.0369
BioReason-Pro0.17330.3848+0.2116+0.2728+0.2099

The A.1 deficit persists across matching settings for the directly aligned sequence-to-text systems; BioReason-Pro also retains this direction. PFUA shows a more matching-dependent A.1–Track B difference. This pattern is consistent with PFUA’s ability to acquire and integrate functional evidence through tools and LLM reasoning, reducing its dependence on sequence–function associations already represented in task-specific training. Consequently, the distinction between annotation novelty and protein novelty may be less decisive for PFUA than for directly aligned systems.

Claim-count control for Miss. More claims provide more opportunities for usable support. We therefore randomly sampled one or two claims per eligible protein in both regimes and averaged Miss over repeated draws. A.1 Miss remains higher for all six systems under both settings:

System1 claim: A.1 Miss1 claim: Track B Miss2 claims: A.1 Miss2 claims: Track B Miss
BioMatrix0.89170.75970.80800.5714
Prot2Text-V20.93260.82000.92060.6675
BioT5+0.94910.87820.90290.7452
EvoLLa0.95630.85530.92520.6937
PFUA0.81920.61920.77460.4058
BioReason-Pro0.82920.63200.73570.4118
AuthorsOpenReview ↗

Response to W2 and Q3

W2 and Q3

We thank the reviewer for raising concerns about benchmark size and evaluation reliability. We have added paired uncertainty estimates for the model comparisons and expanded human validation to cover all six systems and all three regimes, including checks of support, specificity, and conditions. The human evaluation includes 90 support cases, balanced across the 18 system–regime combinations, 107 cases for each component, and reference-decomposition checks on 60 paired cases. Replacing the audited support scores with human ratings preserves the W2 rankings and main comparison directions; the validation and sensitivity results are presented under W2 and Q5 below.

The paired results support describing PFUA and BioReason-Pro as comparable on W2 A.1 and Track B. Both systems have higher recall than BioMatrix on these targets.

ComparisonA.1 difference [95% CI]Track B difference [95% CI]
PFUA − BioReason-Pro−0.0004 [−0.0356, 0.0388]+0.0101 [−0.0446, 0.0663]
PFUA − BioMatrix+0.0620 [0.0330, 0.0929]+0.1110 [0.0489, 0.1770]
BioReason-Pro − BioMatrix+0.0632 [0.0336, 0.0922]+0.1009 [0.0481, 0.1563]

We will use this supported ordering in the text. The retained-versus-added finding is also consistent across systems: every A.1-A.0 recall interval is below zero.

SystemA.1 - A.0 recall [95% CI]
BioMatrix-0.6810 [-0.7329, -0.6250]
Prot2Text-V2-0.2973 [-0.3668, -0.2337]
BioT5+-0.0242 [-0.0438, -0.0049]
EvoLLa-0.0432 [-0.0723, -0.0181]
PFUA-0.4080 [-0.4696, -0.3483]
BioReason-Pro-0.1341 [-0.1787, -0.0934]
Reviewer_TqM2OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ProLong-Bench, which evaluates free-form protein function predictions against evolving Swiss-Prot annotations. It separates retained statements, A.0, from later-added statements, A.1, while holding the protein sequence and model answer fixed. Newly added proteins form a separate track. Six systems are evaluated with claim-level support scores and historical-information diagnostics.

Critical strengths

The paired A.0/A.1 comparison is a useful contribution. It asks something that an aggregate annotation score cannot answer: which parts of the same answer cover retained information, and which cover later additions? The large gap for BioMatrix makes the distinction consequential. The expert support-score validation and aggregation checks are also useful, and the authors correctly avoid treating every reference-external prediction as false.

Critical weaknesses

My main uncertainty is how A.0 and A.1 are assigned. Appendix A gives evidence-eligibility rules, and Appendix B explains decomposition and scoring, but I could not find the procedure that compares the old and new annotations and decides which content is genuinely added. The expert audit checks support and decomposition, not the temporal assignments themselves. Since this split defines the benchmark, I would want the construction rule and an audit covering paraphrases and partially updated statements.
The “fully clean” description of W2 also needs a more precise basis. Appendix C lists reported temporal information, including retrieval-database versions, but does not clearly establish the complete information boundary used in these runs. A statement newly entered into Swiss-Prot need not be new to every source available to a tool-augmented system. I am not inferring contamination; I need to know which cutoff the comparison actually enforces before interpreting the ranking change between system types.

Questions for authors
  • How are retained and added content separated, and has the accuracy of that separation been checked against both annotation versions?
  • Were the retrieval versions in Table 11 the versions actually used for evaluation? Please specify the checkpoints and any other inference-time information sources relevant to the W2 cutoff.
  • Could the authors provide an anonymized repository containing the scripts for the A.0/A.1 temporal split and the LLM-based evaluation pipeline to substantiate reproducibility? This transparency is crucial for a benchmark contribution.
Reviewer_eKsdOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ProLong-Bench, a temporal benchmark for free-form protein function prediction based on historical Swiss-Prot annotations. It separately evaluates how well models recover established functions, capture newly added functional knowledge, and predict functions for newly introduced proteins. Using claim-level semantic evaluation, the authors compare different protein language model systems and conduct additional robustness and historical-information analyses. The results show that strong performance on existing annotations does not necessarily translate to good coverage of newly added knowledge, and that model rankings can change across these evaluation settings.

Critical strengths
  • The temporal comparison is well-designed. A.0 and A.1 evaluate retained and newly added knowledge using the same protein sequences and model outputs, making the comparison more controlled.

  • There are meaningful differences in model performance. The results show that models strong on existing annotations do not necessarily perform well on newly added knowledge. The change in model rankings highlights a limitation of static evaluation.

  • The claim-level evaluation is useful. Evaluating individual functional claims provides more insight than whole-text similarity. The main findings also remain consistent across different aggregation methods.

Critical weaknesses
  • The conventional and temporal evaluations use different protein sets and metrics, and the relationship between these datasets is not fully explained. This makes it difficult to attribute changes in model rankings specifically to temporal evaluation.

  • The historical copying and homology transfer baselines are evaluated using TF–IDF rather than FactJudge. Comparing them under the same semantic metric would better establish how much the models improve over historical information alone

  • The construction of retained and newly added claims is not explained clearly enough. It is unclear how paraphrases, refinements of existing annotations, or corrections are handled. These decisions directly affect the A.0/A.1 split.

Questions for authors

N/A

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and reviewers

We thank the reviewers for their constructive feedback. Substantive revisions are highlighted in blue.

Our central contribution is to distinguish newly added sequence–function associations from retained knowledge and protein novelty, and to systematically characterize model performance across these settings. The concerns raised primarily involve the precision of the construction protocol and the validation of specific analyses. Additional audits and controlled comparisons strengthen the supporting evidence while preserving the central finding: strong performance on retained annotations does not ensure coverage of later-added functional content.

I. Response to the Reviews

Reviewer eKsd. The two experimental settings form a progression from complete descriptions to retained/added content. Answer-length statistics, common-reference metric comparisons, and FactJudge baselines now make this relationship explicit. Direct temporal review confirms 60/60 A.0 and 58/60 A.1 assignments; excluding flagged cases preserves the tested comparison directions.

Reviewer TqM2. Strict pairs deliberately isolate standalone additions, while whole-description evaluation retains broader updates. Manual review shows that 84.2% of excluded records only reorganize or rephrase existing functional content. We clarify this scope and document actual backbone, task-supervision, database, and cache timelines.

Reviewer R5MU. The added controls refine the supporting interpretations while preserving the paired A.0/A.1 finding. Content matching retains the A.1–Track B gap for directly aligned systems, while PFUA shows different sensitivity. Expanded expert validation and a second judge strengthen evaluator validation; replacing audited scores with human labels changes recall by at most 0.0033 and preserves model ordering.

II. Manuscript Revisions

  • Construction and resources: clarified definitions and protocols, added construction audits, and documented resource provenance (Section 3.1; Appendices A and C).
  • Evaluation controls: updated Tables 2–3 and added historical baselines, content matching, claim-count controls, and uncertainty estimates (Sections 4.1–4.3; Appendices B.2, E, and G).
  • Evaluator validation: expanded expert coverage, second-judge comparisons, and human-score sensitivity (Appendix B.5).
  • Interpretation: revised historical-reference explanations using within-protein controls (Section 4.3; Appendix F).

Anonymous code: ProLong-Bench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.