- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ProLong-Bench, a temporal benchmark for evaluating free-form protein function prediction. The benchmark is constructed from three Swiss-Prot snapshots and considers three evaluation settings: recovery of existing functional knowledge (A.0), coverage of newly added annotations for existing proteins (A.1), and function prediction for newly added proteins (Track B). The authors evaluate six protein language model systems using a claim-level evaluation framework called FactJudge. The results show that models performing well on existing annotations often struggle to recover newly added functional information. The paper also finds that tool-augmented systems perform relatively better on temporally novel knowledge, while sequence-based models show stronger agreement with historical annotations. Additional analyses examine the relationship between model performance and historically available functional information.
Critical strengths
-
Most existing benchmarks evaluate predictions against static functional annotations. Separating established knowledge from newly added annotations provides a useful perspective on model performance. In particular, evaluating both types of annotations for the same protein sequence helps isolate differences in functional knowledge coverage.
-
The comparison across six systems shows that strong performance on existing annotations does not necessarily translate to better coverage of newly added knowledge. The finding that tool-augmented systems perform relatively better on temporally novel information is interesting and may help guide future model development.
Critical weaknesses
- Unclear definition of temporal novelty.
The paper treats functional annotations added in later Swiss-Prot releases as emerging knowledge. However, newly curated annotations do not necessarily represent newly discovered biological functions, as the underlying information may have already been available in earlier literature. This makes it difficult to interpret A.1 performance as the ability to predict new functional knowledge. Moreover, the comparison between A.1 and Track B may be affected by differences in functional specificity and annotation complexity. Although Appendix E examines annotation density and sequence-family representation, these analyses do not fully address such differences. Therefore, the lower performance on A.1 does not necessarily demonstrate that annotation novelty is more difficult than protein novelty.
- Limited benchmark size and evaluation validation.
The benchmark is relatively small, particularly W2 Track B, which contains only 90 proteins. This raises concerns about the reliability of model rankings and conclusions across different functional categories. In addition, FactJudge is validated using only 50 gold-support cases and 99 prediction-relevance cases. The gold-support validation covers A.0 and A.1 but does not include Track B. Reference decomposition also satisfies both completeness and atomicity criteria in only 43 of 50 cases, while the component-level scores are not independently validated. More extensive validation would be helpful to establish the reliability of the reported results.
- Unclear control of information access across models.
The evaluated models have different training data and inference-time resources, which may affect the fairness of the comparison. Although Table 11 reports the available temporal information, it is unclear whether these dates correspond to the exact resources used during evaluation. In particular, the paper does not clearly specify whether the models had access to additional databases, online retrieval, or cached information, or how the knowledge cutoffs of the underlying LLMs were determined. Since tool-augmented models perform better on A.1 and Track B, it is difficult to determine how much of this advantage comes from the model architecture rather than differences in information access.
- Overinterpretation of the historical-reference analysis.
The historical-reference analysis shows a correlation between model performance and the presence of similar functional information in historical annotations. However, this does not directly demonstrate memorization or explain the underlying mechanisms of different models. For example, PFUA's confidence interval for the performance difference between historically unseen and represented claims includes zero. In addition, as noted in Appendix F, these two groups do not necessarily contain the same proteins. Therefore, I do not think the current evidence is sufficient to conclude that some models mainly rely on repeating historical knowledge while tool-augmented models operate through a fundamentally different mechanism.
Questions for authors
-
Appendix A defines temporal status by Swiss-Prot annotation availability. Is citation-date information available for a small sample of A.1 claims to assess how often the supporting literature predates the baseline snapshot? If not, could the paper consistently qualify its claims as later-curated annotation coverage?
-
Beyond the aggregation checks in Appendix E, could a small matched analysis assess the A.1–Track B recall gap at comparable levels of functional specificity or contextual detail? Could the Miss comparison also be checked after matching or subsampling the number of claims per protein?
-
Given the existing cluster-based analyses, could you report paired uncertainty for close model comparisons, particularly PFUA versus BioReason-Pro on W2 A.1 and Track B, and clarify which ordering claims are supported?
-
Do the dates in Table 11 describe the exact resources used in these evaluation runs? Please clarify whether any additional databases, online retrieval, or caches were accessible and how their temporal boundaries were determined.
-
Table 9 provides encouraging initial validation. Could a small model- and regime-stratified expert audit assess differential judge bias and the component-level conclusions? A second judge on the same subset would also help assess evaluator sensitivity.
6 comments
Respinse to W1 and Q2
W1 and Q2
We thank the reviewer for asking whether reference content and claim counts explain the A.1–Track B gap. Lower A.1 recall persists for the four directly aligned sequence-to-text systems across three matching settings, and higher A.1 Miss persists for all six systems after equalizing claim counts.
Content matching. To test whether more specific or complex reference content explains the recall gap, we human-labeled all reference claims for 30 A.1 and 30 Track B proteins:
| Reference group | Human-labeled proteins | Claims |
|---|---|---|
| A.1 | 30 | 64 |
| Track B | 30 | 83 |
| Total | 60 | 147 |
Experts rated each claim from 0–2 for specificity (broad functions to specified targets or reactions) and semantic complexity (single relations to additional conditions or mechanistic steps). Ratings were averaged within each protein and used for one-to-one matching, maximizing pair counts and minimizing content differences without using model performance. A maximum difference of 0.5 per attribute yielded 22 pairs; stricter (0.25) and exact matching yielded 17 and 13 pairs, respectively. Primary matching reduced the content imbalance:
| Mean reference-content rating | Before matching: A.1 / Track B | After primary matching: A.1 / Track B |
|---|---|---|
| Specificity | 1.689 / 1.313 | 1.667 / 1.495 |
| Semantic complexity | 0.711 / 0.440 | 0.583 / 0.532 |
For each matched subset, we averaged each system's existing per-protein recall separately for A.1 and Track B. Exact matching requires identical ratings on both attributes. All difference columns report Track B recall minus A.1 recall:
| System | Primary: A.1 recall | Primary: Track B recall | Primary difference (22 pairs) | Stricter difference (17 pairs) | Exact-match difference (13 pairs) |
|---|---|---|---|---|---|
| BioMatrix | 0.1070 | 0.2451 | +0.1381 | +0.2027 | +0.1843 |
| Prot2Text-V2 | 0.0919 | 0.2223 | +0.1305 | +0.1532 | +0.1298 |
| BioT5+ | 0.0862 | 0.1051 | +0.0189 | +0.0245 | +0.0240 |
| EvoLLa | 0.0824 | 0.1460 | +0.0636 | +0.0507 | +0.0881 |
| PFUA | 0.2396 | 0.2968 | +0.0572 | +0.1027 | −0.0369 |
| BioReason-Pro | 0.1733 | 0.3848 | +0.2116 | +0.2728 | +0.2099 |
The A.1 deficit persists across matching settings for the directly aligned sequence-to-text systems; BioReason-Pro also retains this direction. PFUA shows a more matching-dependent A.1–Track B difference. This pattern is consistent with PFUA’s ability to acquire and integrate functional evidence through tools and LLM reasoning, reducing its dependence on sequence–function associations already represented in task-specific training. Consequently, the distinction between annotation novelty and protein novelty may be less decisive for PFUA than for directly aligned systems.
Claim-count control for Miss. More claims provide more opportunities for usable support. We therefore randomly sampled one or two claims per eligible protein in both regimes and averaged Miss over repeated draws. A.1 Miss remains higher for all six systems under both settings:
| System | 1 claim: A.1 Miss | 1 claim: Track B Miss | 2 claims: A.1 Miss | 2 claims: Track B Miss |
|---|---|---|---|---|
| BioMatrix | 0.8917 | 0.7597 | 0.8080 | 0.5714 |
| Prot2Text-V2 | 0.9326 | 0.8200 | 0.9206 | 0.6675 |
| BioT5+ | 0.9491 | 0.8782 | 0.9029 | 0.7452 |
| EvoLLa | 0.9563 | 0.8553 | 0.9252 | 0.6937 |
| PFUA | 0.8192 | 0.6192 | 0.7746 | 0.4058 |
| BioReason-Pro | 0.8292 | 0.6320 | 0.7357 | 0.4118 |
Response to W2 and Q3
W2 and Q3
We thank the reviewer for raising concerns about benchmark size and evaluation reliability. We have added paired uncertainty estimates for the model comparisons and expanded human validation to cover all six systems and all three regimes, including checks of support, specificity, and conditions. The human evaluation includes 90 support cases, balanced across the 18 system–regime combinations, 107 cases for each component, and reference-decomposition checks on 60 paired cases. Replacing the audited support scores with human ratings preserves the W2 rankings and main comparison directions; the validation and sensitivity results are presented under W2 and Q5 below.
The paired results support describing PFUA and BioReason-Pro as comparable on W2 A.1 and Track B. Both systems have higher recall than BioMatrix on these targets.
| Comparison | A.1 difference [95% CI] | Track B difference [95% CI] |
|---|---|---|
| PFUA − BioReason-Pro | −0.0004 [−0.0356, 0.0388] | +0.0101 [−0.0446, 0.0663] |
| PFUA − BioMatrix | +0.0620 [0.0330, 0.0929] | +0.1110 [0.0489, 0.1770] |
| BioReason-Pro − BioMatrix | +0.0632 [0.0336, 0.0922] | +0.1009 [0.0481, 0.1563] |
We will use this supported ordering in the text. The retained-versus-added finding is also consistent across systems: every A.1-A.0 recall interval is below zero.
| System | A.1 - A.0 recall [95% CI] |
|---|---|
| BioMatrix | -0.6810 [-0.7329, -0.6250] |
| Prot2Text-V2 | -0.2973 [-0.3668, -0.2337] |
| BioT5+ | -0.0242 [-0.0438, -0.0049] |
| EvoLLa | -0.0432 [-0.0723, -0.0181] |
| PFUA | -0.4080 [-0.4696, -0.3483] |
| BioReason-Pro | -0.1341 [-0.1787, -0.0934] |
Response to W1 and Q1
W1 and Q1
We thank the reviewer for pointing out the ambiguity in our use of “emerging knowledge.” We will explicitly scope this term to sequence–function associations newly recorded in Swiss-Prot after the baseline, with novelty defined relative to historical sequence–function annotations.
To examine the timing of the supporting evidence, we checked 64 A.1 claims from 30 proteins:
These results show that later curation can incorporate previously published biological findings, supporting the interpretation of A.1 as later-curated annotation coverage. This does not diminish the value of the evaluation. First, the evaluated systems were not provided with the supporting papers and did not retrieve literature during inference; their predictions relied on the target sequence and the specified model and tool resources. Second, an existing protein–function association in the literature does not by itself establish a model’s ability to recover that association from the amino acid sequence and express it as a coherent functional description. This sequence-to-description capability is precisely what our benchmark evaluates.