acceptodds
Under review as a conference paper at ICLR 2027

Composing Protein Evidence: A Benchmark for Fine-Grained Protein-Text Understanding

Abstract

Protein-text understanding requires more than assigning functional labels: a biological claim is most informative when it is associated with the different levels of evidence. Current protein understanding evaluations often fragment this relation, measuring single global label, local feature, or protein-text matching without testing whether these elements remain bound to the same protein. We introduce PannotGround, a benchmark that turns curated protein annotations into a diagnostic test of evidence composition. PannotGround projects each protein record into synchronized levels and composes them into protein-to-text and text-to-protein tasks with biologically structured hard negatives. Across protein language models, protein-text alignment models, protein-LLMs, and text LLMs, we find that single-level performance does not reliably transfer to compositional binding. Alignment models are strongest, yet remain sensitive to local-evidence contrasts, and many failures are biologically plausible near misses. PannotGround establishes evidence composition as a key diagnostic axis for fine-grained protein-text models.

Primary area:
datasets and benchmarks
Keywords:
Protein understanding, Compositional task, Reasoning, Benchmark, LLM

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_DiNrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PannotGround is a benchmark for fine-grained protein-text understanding built from UniProtKB/Swiss-Prot annotations. It contains 282,021 proteins and approximately 1.2 million task-view records covering global function recognition, local feature identification, and residue-level localization. Its compositional tasks match proteins and descriptions through 8-way protein-to-text selection and 64-candidate text-to-protein retrieval; these tasks combine global and local descriptors without explicit residue coordinates. The authors evaluate multiple model families on in-distribution and family-held-out splits, analyze controlled task variants and error patterns, and introduce Pannot-FG with residue-level supervision. The results show substantial performance differences across model families and improvements over the ProtT3 base model after fine-grained training.

Critical strengths
  1. A useful benchmark connecting global and local protein annotations. The three task levels and compositional matching tasks extend evaluation beyond global function labels. Hard negatives that preserve one component while changing another make partial matches observable, providing a practical resource for studying fine-grained protein-text alignment.

  2. Broad evaluation with informative diagnostics and a trainable baseline. Comparisons across protein language models, text models, protein-text alignment models, and protein LLMs reveal meaningful differences in behavior. Pannot-FG also improves P2T accuracy over ProtT3 base from 44.60% to 57.95% on ID and from 37.00% to 50.75% on OOD (Table 27), providing initial evidence that task-relevant training can improve performance.

Critical weaknesses
  1. Composition-specific difficulty is not fully separated from individual recognition errors. Attr1 and compositional tasks differ in answer spaces and scoring settings, so their performance gap does not isolate the ability to combine individually recognized attributes. In the ProGen example in Table 23, the incorrect and correct options have identical local/evidence scores and differ in the global score; the authors explicitly identify a marginal global-recognition error. These limitations weaken the interpretation of benchmark scores as a distinct measure of evidence composition.

  2. The effects of split overlap and potentially ambiguous negatives remain insufficiently quantified. Appendix A.2 acknowledges a small number of exact cross-split sequence collisions, but their effect on the reported results is unclear. Although equivalent annotation signatures and duplicate candidates are excluded, different signatures do not necessarily imply biological incompatibility when annotations are incomplete. These issues could affect both generalization estimates and answer validity. They warrant targeted audits, rather than a blanket dismissal of the benchmark.

Questions for authors
  1. Distinguishing recognition from composition. How do the authors distinguish individual attribute-recognition failures from composition failures? Is there evidence that models still fail on a compositional task when they correctly recognize all constituent attributes for the same examples?

  2. Exact sequence overlap. How many exact cross-split duplicate sequences are identified in Appendix A.2, and how many evaluation questions are affected? Do the main results and ID/OOD conclusions remain unchanged after excluding these cases?

  3. Hard-negative validity. How are hard negatives validated when annotations are incomplete or differ in granularity, and how are potentially valid alternative answers handled? Is the evaluation intended to recover the exact source record or assess biological compatibility?

1 comment

Thank you for acknowledging the utility and breadth of our benchmark. We appreciate your careful assessment and constructive suggestions. We further respond to the questions as follows.

W1 & Q1 Distinguishing recognition from composition

To distinguish individual attribute recognition failure and compositional failure, we paired 130 Attr2 Test-ID and 165 Test-OOD-family questions, yielding 260 and 330 individual questions, respectively. Each gold caption supplies the two correct attributes; negative options are extracted from that compositional question's seven distractor captions, excluding known supported alternatives. The same model answers all three questions. The randomly selected examples pass annotation-validity checks and require both attributes to identify the gold option.

The table separates wrong compositional predictions according to whether both individual-attribute answers are correct or at least one is incorrect. Cells report all individual answers correct / at least one individual answer incorrect, with counts and percentages calculated among compositional errors in each model and split.

ModelTest-IDTest-OOD
ProTrek-35M22 (30.56%) / 50 (69.44%)23 (25.56%) / 67 (74.44%)
ProTrek-650M15 (23.44%) / 49 (76.56%)28 (30.77%) / 63 (69.23%)
ProtST-ESM26 (6.06%) / 93 (93.94%)10 (7.25%) / 128 (92.75%)
ProtST-ProtBERT4 (3.88%) / 99 (96.12%)13 (9.56%) / 123 (90.44%)
ProtT3 Stage110 (10.53%) / 85 (89.47%)7 (5.51%) / 120 (94.49%)
Pannot-FG15 (21.13%) / 56 (78.87%)13 (16.67%) / 65 (83.33%)

All six alignment models make compositional errors despite answering both individual-attribute questions correctly, while most compositional errors also involve an individual-attribute error. The Test-ID example P0CP59, evaluated with ProTrek-650M, illustrates successful individual recognition followed by an incorrect compositional answer:

QuestionChoicesGold answerModel answer
Global attribute3RNA binding (GO:0003723)RNA binding (correct)
Local attribute6coiled coilcoiled coil (correct)
Compositional matching8RNA binding; coiled coilRNA binding; Helicase ATP-binding (incorrect)

This shows that correct individual-attribute recognition does not guarantee correct compositional matching.

W2 & Q2 Split overlap

We identified 95 groups of identical sequences spanning different dataset splits; 13 groups contribute query proteins to the original Full, Attr2 and Attr4 P2T test sets. The table reports affected questions and the largest absolute accuracy change after excluding them across 19 saved model/readout configurations. Each test set contains 2,000 questions. Cross-split matches include overlap between evaluation splits as well as with training.

VariantTest-ID: affected queriesTest-OOD-family: affected queriesTest-ID: maximum absolute accuracy change (pp)Test-OOD-family: maximum absolute accuracy change (pp)
Full2/2,000 (0.10%)6/2,000 (0.30%)0.0620.114
Attr22/2,000 (0.10%)5/2,000 (0.25%)0.0540.129
Attr42/2,000 (0.10%)2/2,000 (0.10%)0.0460.054

Excluding these questions changed accuracy by at most 0.129 percentage points and preserved the highest-scoring model in every variant and test split. Whether ID or OOD accuracy was higher also remained unchanged, except for one initially tied result that became a 0.015-point OOD advantage. These exclusions therefore have little effect on the reported comparisons.

Restricting overlap to sequences also present in the canonical training partition gives zero affected Test-ID queries and 2/1/0 Test-OOD-family queries for Full/Attr2/Attr4, respectively. The revised P2T evaluations exclude both query and caption-source proteins belonging to cross-split duplicate groups.

W2 & Q3 Hard-negative validity

Full P2T asks which option most completely matches the protein's recorded annotations. Construction excludes duplicate candidates and equivalent annotation signatures, but different signatures do not establish biological incompatibility. We audited the seven alternative captions in every original test question against the query protein's retained annotations, checking whether an alternative was fully supported and covered every attribute in the gold caption. Under literal annotation-field matching, we found no equally complete supported alternatives in Test-ID (0/2,000) or Test-OOD-family (0/2,000). Supported captions covering fewer attributes are less complete answers to this best-match question. This evaluation measures matching to recorded annotations. The audit does not establish semantic uniqueness or rule out biological compatibility of other options. Missing annotations, synonyms, and differences in annotation granularity remain limitations, which we will state explicitly.

Reviewer_wrgtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1(−2)
Soundness
2
Read review
Paper summary

This paper introduces PannotGround, a benchmark for fine-grained protein–text understanding. It organizes protein annotations into three levels: global protein function (P1), local feature types (P2), and residue/span evidence localization (P3). The benchmark further constructs compositional protein-to-text (P2T) and text-to-protein (T2P) matching tasks by combining global functional annotations with local feature descriptors. Through biologically structured hard negatives that share global functions, local features, or partially overlapping attributes, PannotGround evaluates whether global and local annotations are correctly matched to the same protein. The authors evaluate four representative model families and find that strong single-level performance does not necessarily translate into reliable compositional matching. Protein–text alignment models perform best overall, but still struggle to accurately match complete global–local annotation compositions.

Critical strengths
  1. This paper evaluates compositional matching between global protein functions and local annotations by constructing P2T and T2P tasks with biologically structured hard negatives. These tasks assess whether global and local annotations are correctly matched to the same protein Compared with individual attribute recognition or general protein–text matching evaluations, this design further distinguishes matching errors across different global–local attribute relationships, providing more fine-grained diagnostic information.

  2. The paper constructs PannotGround from manually reviewed UniProtKB/Swiss-Prot entries, containing 282,021 proteins, over 1.2 million task-view records, and 12,000 P2T/T2P compositional evaluation instances. The dataset covers global functional annotations, local feature types, and residue/span localization. It also uses C30/C90 homology-based clustering to construct ID and family-level OOD splits, which provides a rich data basis for multi-level protein understanding ability evaluation of different models.

  3. This paper compares the performance of Protein language model(PLMs), protein-text alignment model, Protein-LLM and text LLM on P2T task. It analyzes the matching performance under different attribute composition by variants such as glob-only /Local-only, Attr2/Attr4. These experiments compare overall model performance and reveal differences across model families in matching global and local annotations.

Critical weaknesses
  1. In §5.2, the authors use the performance gap between Attr1 and Attr2/Attr4 in Table 5 to argue that single-level prediction proficiency does not reliably transfer to compositional matching. However, Appendix E.4 explicitly states that Attr1 is an aggregate reference from separate P1/P2/P3 tasks, whereas Attr2/Attr4 are 8-way compositional P2T tasks. These settings differ in task format, answer space, and evaluation metrics. Therefore, the observed performance gap cannot be attributed solely to limitations in cross-attribute compositional ability. Although the benchmark requires at least one global and one local attribute when both are available, the paper does not report the distribution of global/local attribute ratios in Attr4. Considering the obvious difference between the performance of the model in Table 4 on Global-only and Local-only tasks, the ratio of global and local attributes may affect task difficulty and accuracy.

  2. Both §3 and Appendix G explicitly mention that the local descriptions of P2T and T2P contain only feature types and textual instructions, and no specific residue positions are used. Although the P3 single-layer task can assess the localization of local features, it cannot directly verify whether the model correctly associates the correct local features with the correct sequence positions during the combination matching process.

Questions for authors
  1. Can you give the number of samples and the model accuracy in Attr4 with different global/local attribute ratios (e.g., 1G+3L, 2G+2L, 3G+1L)? This would help determine whether the composition of different attributes affects the difficulty of the combined task.

  2. To determine whether the main bottleneck is cross-attribute compositional binding, rather than individual attribute recognition or score calibration, could the authors report the proportion of cases where a model correctly recognizes all individual attributes ( P1✓∧P2✓∧P3✓P_1 \checkmark \land P_2 \checkmark \land P_3 \checkmark) but fails to match their combination ( P1+P2+P3P_1 + P_2 + P_3), using the same test proteins and attribute subsets?

  3. PannotGround already contains the P3 annotations that have been verified by coordinates, Can you further design a combination evaluation task for location awareness? For example, given two proteins with the same global annotation but with clear differences in local features, the model is asked to identify the local features that distinguish the two and indicate the residue intervals of these features in the corresponding protein.

4 comments
AuthorsOpenReview ↗

Response – Part 1

Thank you for your thoughtful review and for recognizing the value of evaluating global–local compositional matching. We appreciate your suggestions for examining attribute ratios, distinguishing recognition from composition, and clarifying localization in compositional tasks. We address these points below.

W1 & Q1 Attribute composition

Global/local composition is definitely worthwhile delving into. We thank the reviewer for the question and suggestion. Attr4 has 1G+3L and 2G+2L composition with percentages shown as follows. We produce no 3G1L questions in these sets.

Split fileTotal1G+3L, n (%)2G+2L, n (%)
Val-ID1,000635 (63.50%)365 (36.50%)
Val-OOD1,000612 (61.20%)388 (38.80%)
Test-ID2,0001,290 (64.50%)710 (35.50%)
Test-OOD2,0001,224 (61.20%)776 (38.80%)

Accuracy (%) on the revised tests is reported below for all 15 available model/readout configurations. Each cell gives Test-ID / Test-OOD. After support and duplicate screening, the 1G+3L groups contain 1,026 / 935 questions and the 2G+2L groups contain 534 / 578; these denominators differ from the original counts above.

Model/readout1G+3L2G+2L
ProTrek-35M61.31 / 60.9671.54 / 74.91
ProTrek-650M61.11 / 62.2573.22 / 73.88
ProtST-ESM235.87 / 38.5047.00 / 50.35
ProtST-ProtBERT33.82 / 35.9438.01 / 48.27
ProtT3 Stage133.43 / 35.2943.26 / 44.29
Pannot-FG62.67 / 61.6069.85 / 73.01
Ankh (Align)12.57 / 9.5215.73 / 18.86
ESM2 (Align)19.01 / 17.0126.03 / 29.24
ProGen (Align)12.96 / 10.2715.36 / 17.13
ProstT5 (Align)15.79 / 13.8019.85 / 24.91
Llama-3.1-8B-Instruct15.69 / 16.6814.23 / 15.92
Qwen2.5-7B10.43 / 7.496.37 / 9.34
Qwen3-4B-Base2.14 / 2.571.87 / 2.25
ProLLaMA13.06 / 13.5811.61 / 10.03
BioReason-Pro SFT8.09 / 7.4914.23 / 15.05

All six protein–text alignment models and four PLM alignment readouts score higher on 2G+2L in both splits, whereas Llama, Qwen3 and ProLLaMA show the opposite pattern. Aggregate Attr4 accuracy therefore depends on both the attribute mixture and the evaluated model. These groups contain different proteins and annotations, so the comparison does not isolate a causal effect of the ratio. No 3G+1L examples occur because the sampler prioritizes proteins with multiple local evidence annotations. Every sampled gold option retains at least two local descriptors.

On the other hand, we rebuilt Attr1 as eight-choice single-attribute P2T scored by accuracy, replacing the aggregate of separate P1/P2/P3 tasks. For each protein, we select one recorded global annotation or local feature description as the gold answer. Seven alternatives are sampled from other proteins in the same split and broad global/local category, excluding attributes already supported by the query protein's annotations and duplicate or equivalent options.

Each initial test split contained 1,000 global and 1,000 local questions. Screening for complete model inputs, cross-split sequence duplicates, and supported or duplicate alternatives leaves 1,831 Test-ID / 1,828 Test-OOD-family questions, shared by all evaluated models. The table averages model accuracies equally within each family, and the final column gives equal weight to the two test splits.

FamilyAvailable modelsTest-ID (%)Test-OOD (%)Mean (%)
Random—12.5012.5012.50
PLM413.2212.5712.89
PLM-TextAlign432.8232.5432.68
Text LLM38.457.848.14
Alignment649.6151.1350.37
Protein-LLM211.4711.8411.66

This Attr1 setup could be used as the uniform baseline.

AuthorsOpenReview ↗

Response – Part 2

W1 & Q2 Examine compositional failure and individual recognition

To examine the same test proteins and attribute subsets, we paired 130 Attr2 Test-ID and 165 Test-OOD-family questions with individual global- and local-attribute questions, yielding 260 and 330 individual questions. Correct attribute answers come from each gold caption; distractors come from its seven alternative captions, excluding known supported alternatives.

The table separates wrong compositional predictions into whether all individual attributes are correct or at least one is incorrect. Each cell reports all individual answers correct / at least one individual answer incorrect, with counts and percentages calculated among compositional errors.

ModelTest-IDTest-OOD
ProTrek-35M22 (30.56%) / 50 (69.44%)23 (25.56%) / 67 (74.44%)
ProTrek-650M15 (23.44%) / 49 (76.56%)28 (30.77%) / 63 (69.23%)
ProtST-ESM26 (6.06%) / 93 (93.94%)10 (7.25%) / 128 (92.75%)
ProtST-ProtBERT4 (3.88%) / 99 (96.12%)13 (9.56%) / 123 (90.44%)
ProtT3 Stage110 (10.53%) / 85 (89.47%)7 (5.51%) / 120 (94.49%)
Pannot-FG15 (21.13%) / 56 (78.87%)13 (16.67%) / 65 (83.33%)

All six alignment models make compositional errors after correctly recognizing both individual attributes, while most compositional errors also involve an individual-attribute error. Thus, successful individual recognition does not guarantee correct compositional matching.

W2 & Q3 Localization in compositional task

We totally agree on the value of including P3 (where) in the binding experiments. Following your suggestion, we added the recorded P3 intervals to the existing single-protein P2T format and constructed three setups. We set 3 types of ablations for position: A preserves the original annotation alternatives, while B/C require distinguishing positions for the same feature description. The setup details are presented as follows:

ColumnChoicesConstructionInterpretation
A08Original attr2 task without locationMatched control
A8Existing semantic candidates with source position addedPositional annotation matched
B8Replace one A negative option with gold annotations but at a wrong locationPosition contrast present in one negative option
C8Gold descriptions in all options and only the position variesPosition selection

Incorrect intervals in setup B/C are sampled by shifting the gold interval pattern within the sequence, preserving lengths and gaps and avoiding recorded intervals of the same feature type. In C, all eight candidate windows are mutually non-overlapping. Positional compositional results on the attr2 sets are presented as follows

Model/readoutA0ABC
ProTrek-35M51.97 / 62.2859.84 / 66.6750.39 / 55.2648.03 / 38.60
ProTrek-650M55.91 / 49.1260.63 / 59.6549.61 / 57.0237.80 / 55.26
ProtST-ESM227.56 / 23.6832.28 / 18.4218.90 / 13.1613.39 / 11.40
ProtST-ProtBERT24.41 / 17.5431.50 / 18.4221.26 / 14.0412.60 / 7.89
ProtT3 Stage124.41 / 19.3023.62 / 19.3013.39 / 12.2812.60 / 18.42
Pannot-FG51.18 / 42.9846.46 / 40.3525.20 / 30.708.66 / 11.40
Ankh (align)7.09 / 19.307.87 / 20.181.57 / 12.289.45 / 13.16
ESM2 (align)10.24 / 23.6812.60 / 22.815.51 / 11.409.45 / 14.04
ProGen (align)7.09 / 18.429.45 / 18.422.36 / 11.409.45 / 13.16
ProstT5 (align)9.45 / 20.1811.81 / 22.812.36 / 10.537.87 / 12.28
Llama-3.1-8B-Instruct39.37 / 38.6025.20 / 21.0527.56 / 25.4414.96 / 12.28
Qwen2.5-7B13.39 / 21.0512.60 / 17.5414.17 / 25.442.36 / 8.77
Qwen3-4B-Base1.57 / 1.755.51 / 4.395.51 / 7.025.51 / 7.02
ProLLaMA7.87 / 13.166.30 / 20.187.09 / 19.3014.96 / 29.82
BioReason-Pro SFT7.09 / 7.893.15 / 3.512.36 / 3.5111.02 / 13.16

Adding location in the attr2 introduces a different influence on accuracy when comparing performances in Setup A to those in the original setup A0. Most models’ accuracy is increased, possibly because the locations serve as additional binding evidence for the model. Setup B introduces the wrong location with gold annotation as a negative option. From Setup A to B, the incorrect location in the negative options leads to a performance drop for most of the models. Compared to Setup A, Setup C tests interval selection directly beyond the presence or lack of locations. We found that the accuracy of Setup C is lower than other setups, revealing that the pure localization task in compositional options is hard.

These setups evaluate compositional prediction with positions. In our framework, we study localization as a capability for models in the single-level task, and we add the compositional location-awareness capabilities as extra ablations.

Reviewer_wrgtOpenReview ↗

Thanks for the rebuttal

Thanks for the rebuttal. I appreciate the authors' additional experiments, where the localization-aware evlauation reveals an interesting finding: ProTrek remains robust to incorrect positional distractors in Setup B and achieves performance comparable to A0 in the position-only Setup C. This suggests that ProTrek can effectively recognize position-specific protein features, a valuable finding that deserves greater emphasis in the paper.

I really appreciate the big idea of this paper. The introduction is well-motivated, and the benchmark design is promising. However, the experimental section lacks clarity and suffers from poor organization and formatting, making the results difficult to follow.

  1. From line 152 to line 158, there are repeated description of global and local entries.
  2. I suggest authors to organize quesions as follows in Evaluation section:
    • Q1: XXX
    • Q2: XXX
  3. For example, the sentence in Line 370, “Thus, the main-table comparison shows that solving the full caption task requires more than isolated evidence recognition,” reads like a generic, possibly AI-generated conclusion. It provides little insight into the experimental findings.
  4. I strongly encourage the authors to make the experimental findings more focused and accessible. Each paragraph should begin with a clear takeaway, preferably highlighted in bold, followed by supporting evidence and analysis. Readers should be able to grasp the key findings without having to work through lengthy descriptions to understand the main point.

Given the current lack of clarity in the presentation and formatting, I am unable to raise my score. However, I would be willing to reconsider my rating if the authors substantially improve the organization, clarity, and overall presentation of the paper.

AuthorsOpenReview ↗

Response on revision

Thank you for your helpful feedback. We have revised the manuscript to address your concerns about the organization, clarity, and presentation of the experiments. The revised parts are marked in blue.

  1. Removed the repeated definitions in Section 3.

    We removed the duplicated descriptions of global and local annotation entries previously appearing in Lines 152–158.

  2. Explicitly listed the research questions in Section 4.

    Following your suggested format, the Evaluation section now presents three clearly labeled questions:

    • RQ1 (Individual capabilities): How well do models recognize global functions, identify local feature types, and localize residue/span evidence?
    • RQ2 (Compositional prediction): How well do models match compositions of global functions and local features to proteins, and how does performance vary with the composition?
    • RQ3 (Failure diagnosis): Which biological attributes are preserved or mismatched in incorrect predictions, and which local features are difficult to distinguish?

    The subsequent paragraph explains how the evaluation addresses these questions, and Sections 5.1–5.3 follow the same order.

  3. Replaced the generic conclusion with specific analysis.

    We removed the sentence you identified on line 370 and revised Section 5.2 to connect the interpretation directly to the reported results. For example, Table 4 shows that all six alignment models follow the ordering Global-only > Full > Local-only, while all four PLM-TextAlign models show the reverse ordering. The revised discussion showcase these contrasting patterns and their implications for matching global and local annotations.

  4. Reorganized Section 5 around the findings and their supporting evidence.

    We revised the progression of the experimental discussion:

    • Section 5.1 establishes how model strengths differ across biological levels, using concrete comparisons to explain why separate evaluations are informative.
    • Section 5.2 examines individual versus compositional performance, then considers global/local composition, attribute count, and the contribution of structure.
    • Section 5.3 first examines which annotations incorrect P2T predictions preserve, then identifies difficult local-feature categories, and finally presents the complementary T2P retrieval analysis.

    We added bold takeaway sentences at the beginning of the main result paragraphs, followed by supporting results and analysis. Examples include “Models performing similarly at one level can differ at another” and “Strong models’ compositional errors usually retain gold annotations.” These changes make the central findings visible while providing a clear progression.

We also agree with your observation about ProTrek’s positional discrimination. We emphasize this finding in Appendix E.5, Table 32, and its accompanying discussion.

We substantially revised the Results section. We would appreciate it if you could revisit the revised manuscript regarding previous concerns.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response – Part 1

Thank you for the detailed review and recognition of PannotGround’s evaluation target, structured hard negatives, and diagnostic analyses. We provide the corresponding revisions below.

w1 Localization in compositional task

We thank the reviewer for the suggestion. To clarify, we did bind global function (P1 - What) with local residue- and region-level features (P2 - Why) in P2T and T2P tasks (as stated in Section 3, line 190); what was not included in P2T/T2P was the residue indices of the local evidence (P3 - Where). The reviewer’s suggestion to include P3 makes total sense. Following your suggestion, we added a positional compositional variant on the attr2 P2T sets for ablation studies, considering that attr2 provides a controlled two-attribute setting. We designed 3 new setups to introduce the position content based on our original attr2 sets.

ColumnChoicesConstructionInterpretation
A08Original attr2 task without locationMatched control
A8Existing semantic candidates with source position addedPositional annotation matched
B8Replace one A negative option with gold annotations but at a wrong locationPosition contrast present in one negative option
C8Gold descriptions in all options and only the position variesPosition selection

Incorrect intervals in setup B/C are sampled by shifting the gold interval pattern within the sequence, preserving lengths and gaps and avoiding recorded intervals of the same feature type. In C, all eight candidate windows are mutually non-overlapping. Positional compositional results on the attr2 sets are presented as follows

Model/readoutA0ABC
ProTrek-35M51.97 / 62.2859.84 / 66.6750.39 / 55.2648.03 / 38.60
ProTrek-650M55.91 / 49.1260.63 / 59.6549.61 / 57.0237.80 / 55.26
ProtST-ESM227.56 / 23.6832.28 / 18.4218.90 / 13.1613.39 / 11.40
ProtST-ProtBERT24.41 / 17.5431.50 / 18.4221.26 / 14.0412.60 / 7.89
ProtT3 Stage124.41 / 19.3023.62 / 19.3013.39 / 12.2812.60 / 18.42
Pannot-FG51.18 / 42.9846.46 / 40.3525.20 / 30.708.66 / 11.40
Ankh (align)7.09 / 19.307.87 / 20.181.57 / 12.289.45 / 13.16
ESM2 (align)10.24 / 23.6812.60 / 22.815.51 / 11.409.45 / 14.04
ProGen (align)7.09 / 18.429.45 / 18.422.36 / 11.409.45 / 13.16
ProstT5 (align)9.45 / 20.1811.81 / 22.812.36 / 10.537.87 / 12.28
Llama-3.1-8B-Instruct39.37 / 38.6025.20 / 21.0527.56 / 25.4414.96 / 12.28
Qwen2.5-7B13.39 / 21.0512.60 / 17.5414.17 / 25.442.36 / 8.77
Qwen3-4B-Base1.57 / 1.755.51 / 4.395.51 / 7.025.51 / 7.02
ProLLaMA7.87 / 13.166.30 / 20.187.09 / 19.3014.96 / 29.82
BioReason-Pro SFT7.09 / 7.893.15 / 3.512.36 / 3.5111.02 / 13.16

Adding location in the attr2 introduces a different influence on accuracy when comparing performances in Setup A to those in the original setup A0. Most models’ accuracy is increased, possibly because the locations serve as additional binding evidence for the model. Setup B introduces the wrong location with gold annotation as a negative option. From Setup A to B, the incorrect location in the negative options leads to a performance drop for most of the models. Compared to Setup A, Setup C tests interval selection directly beyond the presence or lack of locations. We found that the accuracy of Setup C is lower than other setups, revealing that the pure localization task in compositional options is hard. Both ProTrek models performed better than the other models.

These setups evaluate compositional prediction with positions. In our framework, we study localization as a capability for models in the single-level task, and we add the compositional location-awareness capabilities as extra ablations.

W2 Pannot-FG clarification

Appendix §F offers details of the model. Pannot-FG is initialized on the ProtT3 stage1 checkpoints. It is continually trained on a training set of the PannotGround split and tested on the Val/Test-ID/OOD as other models. It ensures our models are not trained with test records. The training samples are task-view record based instruction set (before the MCQ construction; stats shown in Table 2) including GO, EC, catalytic activities, cofactors, regions, and sites, within the train_condensed protein split. We would further clarify the details of training in the appendix §F.

AuthorsOpenReview ↗

Response – Part 2

W3 Statistical treatments

The significant transfer gap denotes the performance drop from Attr1 to Attr2/4 in Table 5, from individual attributes to multiple-attribute binding, rather than the transferability from ID to OOD.

We agree on the need for more statistical evidence. For the reported structure comparison, we recomputed paired differences on the identical questions available to both models:

Historical test populationNESM2 accuracy %ESM2+ESMIF accuracy %Paired Δ, pp [95% CI]
Test-ID1,18726.2828.39+2.11 [+0.49,+3.73]
Test-OOD23727.4329.96+2.53 [−0.83,+5.79]
Equal-weight test-split mean1,187 / 23726.8629.17+2.32 [+0.47,+4.15]

The last row gives equal weight to the two test splits and reproduces the paper's 26.86 versus 29.17 comparison.

W4 Options' length

The length of the P2T MCQ options is not distinguishable by construction between gold options and negative options. In fact, negative options are sampled from other proteins that share some attributes with the gold option but differ in other attributes. Only 2.65%/3.01% of negative options in the original ID/OOD tests are strict subsets of the gold's retained annotations. Length below is measured in characters(Q1 and Q3 are the 25th and 75th percentiles).

Original Full splitGold length, median [Q1, Q3]Negative length, median [Q1, Q3]Gold strictly longer than all 7 negative options, n/N (%)
Test-ID480 [381, 593.25]470 [370, 585]166/2,000 (8.30%)
Test-OOD-family482 [389.75, 595]470 [367, 586]145/2,000 (7.25%)

For the stats presented, the dataset does not leak the gold option by length since the length of gold answers and negative answers are comparable.

To directly examine the effect of option length, we replaced negative options of several samples so that all eight captions have the same word count, keeping the protein, gold answer and negative-option relation counts fixed. This was possible for 233 ID / 301 OOD questions(12.59%/16.83%) of the revised Full tests. Both conditions below use these same questions.

ModelTest-ID: revised baseline → equal-word-count accuracy (%)Test-OOD: revised baseline → equal-word-count accuracy (%)
ProTrek-35M78.97 → 81.1278.07 → 75.08
ProTrek-650M81.97 → 83.2673.75 → 76.74
ProtST-ESM254.94 → 55.7960.47 → 57.14
ProtST-ProtBERT52.36 → 52.3654.82 → 51.83
ProtT3 Stage151.93 → 51.0757.48 → 56.48
Pannot-FG80.26 → 75.1171.10 → 72.76

Matching all eight options by word count largely preserves performance: mean accuracy changes from 66.34% to 65.73%, a decrease of 0.62 percentage points. This suggests that word-count differences are not the primary driver of the observed performance on the matched subset. Because matching requires replacing negative options, this is evidence of robustness rather than an isolated causal estimate of length effects.

Native PLMs remain low-performing after word matching: mean accuracy changes from 9.33% to 10.45%, below 12.5% chance. This means equal weight to the four models and two test splits.

Native PLMTest-ID: baseline → word-matched (%)Test-OOD: baseline → word-matched (%)
Ankh7.73 → 8.588.31 → 10.30
ESM215.02 → 11.599.63 → 14.29
ProGen13.73 → 14.596.98 → 11.30
ProstT58.58 → 7.304.65 → 5.65

These PLMs score the supplied attributes rather than the caption text. Their changes reflect the replacement options.

AuthorsOpenReview ↗

Response – Part 3

W5 Weak baseline

The benchmark aims to provide a relatively fair comparison across models on complex protein understanding tasks. We do not fine-tune all models, considering the computing costs and different model architectures. The reason we provide fine-tuned PLM headers is that it is the direct way to showcase the PLM embedding’s prediction capability. We choose to adopt the most common and native harness in baselines. To provide optional stronger baselines, we added small-scale external-model fine-tuning, a training-only TF-IDF baseline, and few-shot and constrained-scoring comparisons.

We fine-tuned ProTrek-35M on 4,096 Full train_condensed pairs, using three seeds, three epochs and final checkpoints without test selection. As baselines, Sequence 3-mer TF-IDF transfers annotations from either those same 4,096 pairs or 81,854 training references. All methods below use the same revised questions; mean ± SD describes the three fine-tuning seeds.

Full test splitNTF-IDF, 4k (%)TF-IDF, 81k (%)Pretrained ProTrek-35M (%)Fine-tuned ProTrek-35M (%), mean ± SD
Test-ID1,85040.7072.9761.4164.74 ± 0.92
Test-OOD1,78836.4759.8454.1957.12 ± 0.86

For text models, we selected 128 full compositional questions per test split. The conditions use zero-shot generation, two fixed Val-ID demonstrations. Cells give accuracy (%) / valid-answer rate (%).

CheckpointTest splitZero-shot generationTwo-shot generation
Llama-3.1-8B-InstructTest-ID31.25 / 100.0013.28 / 100.00
Llama-3.1-8B-InstructTest-OOD30.47 / 100.0014.06 / 100.00
Qwen2.5-7B (base)Test-ID7.81 / 41.4110.16 / 80.47
Qwen2.5-7B (base)Test-OOD5.47 / 39.069.38 / 80.47
Qwen3-4B (base)Test-ID2.34 / 17.9722.66 / 100.00
Qwen3-4B (base)Test-OOD4.69 / 21.8824.22 / 100.00

These tables provide optional baselines for the benchmark.

W6 Attr1 reconstruct

We rebuilt Attr1 as eight-choice single-attribute P2T scored by accuracy, replacing the aggregate of separate P1/P2/P3 tasks. For each protein, we select one recorded global annotation or local feature description as the gold answer. Seven alternatives are sampled from other proteins in the same split and broad global/local category, excluding attributes already supported by the query protein's annotations and duplicate or equivalent options. The eight options are shuffled. Sampling is not restricted to the same annotation namespace or local feature type.

Each initial test split contained 1,000 global and 1,000 local questions. Screening for complete model inputs, cross-split sequence duplicates, and supported or duplicate alternatives leaves 1,831 Test-ID / 1,828 Test-OOD-family questions, shared by all evaluated models. The table averages model accuracies equally within each family, and the final column gives equal weight to the two test splits.

FamilyAvailable modelsTest-ID (%)Test-OOD (%)Mean (%)
Random—12.5012.5012.50
PLM413.2212.5712.89
PLM-TextAlign432.8232.5432.68
Text LLM38.457.848.14
Alignment649.6151.1350.37
Protein-LLM211.4711.8411.66

This Attr1 setup could be used as the uniform baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.