- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PannotGround is a benchmark for fine-grained protein-text understanding built from UniProtKB/Swiss-Prot annotations. It contains 282,021 proteins and approximately 1.2 million task-view records covering global function recognition, local feature identification, and residue-level localization. Its compositional tasks match proteins and descriptions through 8-way protein-to-text selection and 64-candidate text-to-protein retrieval; these tasks combine global and local descriptors without explicit residue coordinates. The authors evaluate multiple model families on in-distribution and family-held-out splits, analyze controlled task variants and error patterns, and introduce Pannot-FG with residue-level supervision. The results show substantial performance differences across model families and improvements over the ProtT3 base model after fine-grained training.
Critical strengths
-
A useful benchmark connecting global and local protein annotations. The three task levels and compositional matching tasks extend evaluation beyond global function labels. Hard negatives that preserve one component while changing another make partial matches observable, providing a practical resource for studying fine-grained protein-text alignment.
-
Broad evaluation with informative diagnostics and a trainable baseline. Comparisons across protein language models, text models, protein-text alignment models, and protein LLMs reveal meaningful differences in behavior. Pannot-FG also improves P2T accuracy over ProtT3 base from 44.60% to 57.95% on ID and from 37.00% to 50.75% on OOD (Table 27), providing initial evidence that task-relevant training can improve performance.
Critical weaknesses
-
Composition-specific difficulty is not fully separated from individual recognition errors. Attr1 and compositional tasks differ in answer spaces and scoring settings, so their performance gap does not isolate the ability to combine individually recognized attributes. In the ProGen example in Table 23, the incorrect and correct options have identical local/evidence scores and differ in the global score; the authors explicitly identify a marginal global-recognition error. These limitations weaken the interpretation of benchmark scores as a distinct measure of evidence composition.
-
The effects of split overlap and potentially ambiguous negatives remain insufficiently quantified. Appendix A.2 acknowledges a small number of exact cross-split sequence collisions, but their effect on the reported results is unclear. Although equivalent annotation signatures and duplicate candidates are excluded, different signatures do not necessarily imply biological incompatibility when annotations are incomplete. These issues could affect both generalization estimates and answer validity. They warrant targeted audits, rather than a blanket dismissal of the benchmark.
Questions for authors
-
Distinguishing recognition from composition. How do the authors distinguish individual attribute-recognition failures from composition failures? Is there evidence that models still fail on a compositional task when they correctly recognize all constituent attributes for the same examples?
-
Exact sequence overlap. How many exact cross-split duplicate sequences are identified in Appendix A.2, and how many evaluation questions are affected? Do the main results and ID/OOD conclusions remain unchanged after excluding these cases?
-
Hard-negative validity. How are hard negatives validated when annotations are incomplete or differ in granularity, and how are potentially valid alternative answers handled? Is the evaluation intended to recover the exact source record or assess biological compatibility?
Response
Thank you for acknowledging the utility and breadth of our benchmark. We appreciate your careful assessment and constructive suggestions. We further respond to the questions as follows.
W1 & Q1 Distinguishing recognition from composition
To distinguish individual attribute recognition failure and compositional failure, we paired 130 Attr2 Test-ID and 165 Test-OOD-family questions, yielding 260 and 330 individual questions, respectively. Each gold caption supplies the two correct attributes; negative options are extracted from that compositional question's seven distractor captions, excluding known supported alternatives. The same model answers all three questions. The randomly selected examples pass annotation-validity checks and require both attributes to identify the gold option.
The table separates wrong compositional predictions according to whether both individual-attribute answers are correct or at least one is incorrect. Cells report all individual answers correct / at least one individual answer incorrect, with counts and percentages calculated among compositional errors in each model and split.
All six alignment models make compositional errors despite answering both individual-attribute questions correctly, while most compositional errors also involve an individual-attribute error. The Test-ID example P0CP59, evaluated with ProTrek-650M, illustrates successful individual recognition followed by an incorrect compositional answer:
This shows that correct individual-attribute recognition does not guarantee correct compositional matching.
W2 & Q2 Split overlap
We identified 95 groups of identical sequences spanning different dataset splits; 13 groups contribute query proteins to the original Full, Attr2 and Attr4 P2T test sets. The table reports affected questions and the largest absolute accuracy change after excluding them across 19 saved model/readout configurations. Each test set contains 2,000 questions. Cross-split matches include overlap between evaluation splits as well as with training.
Excluding these questions changed accuracy by at most 0.129 percentage points and preserved the highest-scoring model in every variant and test split. Whether ID or OOD accuracy was higher also remained unchanged, except for one initially tied result that became a 0.015-point OOD advantage. These exclusions therefore have little effect on the reported comparisons.
Restricting overlap to sequences also present in the canonical training partition gives zero affected Test-ID queries and 2/1/0 Test-OOD-family queries for Full/Attr2/Attr4, respectively. The revised P2T evaluations exclude both query and caption-source proteins belonging to cross-split duplicate groups.
W2 & Q3 Hard-negative validity
Full P2T asks which option most completely matches the protein's recorded annotations. Construction excludes duplicate candidates and equivalent annotation signatures, but different signatures do not establish biological incompatibility. We audited the seven alternative captions in every original test question against the query protein's retained annotations, checking whether an alternative was fully supported and covered every attribute in the gold caption. Under literal annotation-field matching, we found no equally complete supported alternatives in Test-ID (0/2,000) or Test-OOD-family (0/2,000). Supported captions covering fewer attributes are less complete answers to this best-match question. This evaluation measures matching to recorded annotations. The audit does not establish semantic uniqueness or rule out biological compatibility of other options. Missing annotations, synonyms, and differences in annotation granularity remain limitations, which we will state explicitly.