- Rating
- 2
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces POOR, a multi-axis OOD benchmark covering 10 protein understanding tasks, where test samples are annotated with non-exclusive sequence/evolutionary, structural, and functional OOD flags. It further proposes ProDPR, a reference-relative metric for measuring predictive utility retained under distribution shift. Experiments across protein language models and structure-aware models show that OOD robustness is shift-dependent, and that fused multi-axis shifts are generally more challenging than individual shifts.
Critical strengths
- Comprehensive multi-axis OOD evaluation. POOR evaluates multiple biologically meaningful and overlapping forms of distribution shift across a broad range of protein tasks, providing a more realistic view of generalization than single-axis benchmarks.
- Useful empirical analysis. The experiments reveal meaningful differences between predictive performance and OOD robustness, as well as shift-dependent effects of model scale, structural information, and fine-tuning strategy.
Critical weaknesses
-
Benchmark availability is unclear. I could not find a clear repository or release statement for the processed splits, OOD annotations, or evaluation implementation. For a benchmark paper, public access to the exact data and evaluation scripts is important.
-
The null predictor in ProDPR is underspecified. The paper describes it only as a task-specific non-informative predictor, such as a label-prior baseline, without clearly defining it for every task and metric.
-
ProDPR sensitivity is insufficiently studied. The effects of the null model, , , weighting strategy, and reference-pool choice are not systematically evaluated.
-
Some InD reference pools are very small. For example, the InD subset constitutes only 1.7% of the pH test set and around 3% for several EC/GO tasks, potentially making the denominator of ProDPR unstable.
-
The interpretation of fused OOD shifts may be too strong. Sequence, structure, and functional novelty are biologically and statistically correlated, and lower performance on their intersection does not necessarily imply a genuine compounding interaction rather than selection of intrinsically harder samples.
-
The empirical model coverage and uncertainty are somewhat limited. The benchmark is broad, but experiments mainly involve ESM-2 variants, SaProt, and GVP-GNN, and GVP-GNN is reported with only one seed while the PLM results use five runs.
-
Several task-specific confounders complicate the OOD interpretation. In particular, the pH task mixes biological OOD with an AlphaFold-to-experimental structure provenance shift, while PPI performance is strongly affected by negative-pair composition.
Questions for authors
-
Will the complete POOR benchmark be publicly released, including the exact train/validation/test splits, OOD annotations, structure identifiers, and ProDPR evaluation code?
-
How is the task-specific null predictor $f_{\mathrm{null}}$ defined for each metric, and what values are used for and ? How sensitive are the main ProDPR rankings to these choices and to the InD versus Auto reference pool?
-
Do the conclusions about fused multi-axis OOD shifts remain when using background-matched Auto references or another analysis that controls for sample composition? This would help distinguish genuine compounding effects from harder intersection subsets.
-
How much do the benchmark-level conclusions change after controlling for the PPI negative-pair composition and the pH structure-provenance shift, or after excluding these two tasks from the aggregate results?
-
What is the rationale for moving chemically dissimilar kcat/LBA validation or test samples back into the training set, rather than retaining them as explicit chemical-OOD subsets or excluding them from evaluation?
-
Why are five independent runs reported for the PLM models but only one run for GVP-GNN? Could comparable uncertainty estimates be provided across all evaluated models?
3 comments
Thank you for the rebuttal, which partially addresses my concern. However, I still have a few remaining concerns:
ProDPR sensitivity: The additional analysis is helpful, but some results actually reinforce my concerns about the stability of ProDPR. For example, changing the minimum stratum size to 50 moves ESM-2-650M from fifth to first, while size-weighted aggregation leads to substantial differences in ProDPR scores. I understand that some sensitivity to hyperparameters is expected, but such large changes make me wonder how reliable the overall model comparisons are. The uncertainty associated with very small InD reference pools also remains a concern.
Chemical OOD: I understand the motivation for controlling chemical OOD to focus on protein OOD. However, moving validation/test samples into training can change the protein distribution and invalidate the original OOD annotations. How can the authors guarantee the original split constraints (e.g., protein-level disjointness for kcat and temporal splitting for LBA) are still preserved after reassignment?
Benchmark availability: I appreciate the commitment to releasing the benchmark, but I would really like to see an anonymous repository during the review process. Since this is a benchmark paper, being able to examine the actual data splits, OOD annotations, and evaluation code would help me better assess its reproducibility. If the complete benchmark is too large to share at this stage, providing one representative task with its processed splits, OOD annotations, and evaluation scripts would be helpful as well.
Thank you for the follow-up questions.
1. ProDPR sensitivity and reference uncertainty
Minimum stratum size. and 20 yield identical strata and scores. Raising it to 50 removes four of eighteen strata (22.2%), but only 159 of 14,980 samples (1.06%). ESM-2-650M moves from fifth to second, mainly because poorly retained rare strata are excluded; GVP remains first. We use 20 to balance scenario coverage and minimum support; this threshold is a compromise dictated by current data sizes, and larger datasets may support higher thresholds.
Aggregation weights. Equal weighting reflects our goal of evaluating robustness across OOD conditions without allowing frequent conditions to dominate rare ones. Sample-size weighting increases the influence of common strata and reduces that of rare OOD scenarios; it answers a different evaluation question and changes rankings in our experiments. We therefore use equal weighting as a default suited to the benchmark's goal, without claiming that rankings are invariant to weighting. Deployment-aware weighting is also supported by related work: Fernández-Díaz et al. [1] incorporate prior knowledge of the target deployment distribution into AU-GOOD. Exploring such weighting schemes and curves that vary the emphasis on different test conditions is a useful future direction for POOR.
ProDPR provides a summary reference for comparing model generalization, rather than a standalone judgment of model capability. We interpret it alongside predictive performance. More detailed diagnosis can use the original task metrics on individual OOD subsets, as illustrated by the subset-level comparisons in our fourth experimental analysis (Section 4.4). We respectfully ask that these limitations of ProDPR be weighed separately from the contributions of the dataset and its multi-axis OOD design, which also support evaluation using the original task metrics without ProDPR.
2. Chemical reassignment and split integrity
Chemical-support reassignment is part of split construction and precedes OOD annotation. The annotation scripts read the resulting split files and compute similarity-based OOD flags against the resulting train/validation reference; training-based functional and holdout annotations likewise use the resulting training split. Thus, the intended pipeline does not retain annotations computed against the provisional splits.
For kcat, the initial split is defined over UniProt groups, and chemical reassignment moves entire UniProt groups, not individual protein–substrate records. This avoids splitting a reassigned protein's records between training and evaluation.
For LBA, the temporal constraint is within each ligand-similarity cluster, rather than a single global date cutoff. Clusters require pairwise Tanimoto similarity ≥0.6. Every non-singleton cluster initially has a training member, so its held-out members already satisfy the chemical-support threshold. Singleton clusters without chemical support are moved to training without reversing the order of retained members within a cluster. Subsequent ratio adjustments move the earliest test entries to validation and the earliest validation entries to training. OOD annotation follows these adjustments.
3. Anonymous benchmark access
The anonymous repository now includes complete kcat train/validation/test CSVs (15,479 / 2,150 / 5,002 records) and test OOD annotations under POOR/datasets/enzyme_kinetics_prediction/splits/, together with processing, annotation, and evaluation scripts and the ProDPR implementation. The README and task documentation describe usage. The complete benchmark data, including all remaining task splits and the large structure files, will be publicly released together after the review process concludes, under the long-term maintenance plan described in our response to Comment 1.
Reference
[1] Fernández-Díaz et al. (2025). A New Framework for Evaluating Model Out-of-Distribution Generalisation for the Biochemical Domain. ICLR. Paper.
1. Availability (C1, Q1)
Links were withheld for anonymity. We will release the splits, OOD annotations, structure identifiers, reference pools, processing pipeline, and evaluation code, with long-term maintenance, task expansion, and issue correction.
2. ProDPR definition and sensitivity (C2–C4, Q2)
Appendix A.1 now specifies the nulls: training-label prevalence vectors for EC/GO; majority classes for Fold/SSP; constant training-prevalence scores for AUROC (0.5 when both classes exist); and zero utility for Spearman by convention. Training priors remain fixed across strata and references.
We use ϵ=10−8, nmin=20, equal stratum weights, and InD references, excluding undefined metrics and near-zero reference gaps. Sensitivity checks show:
Small reference pools remain uncertain. ProDPR supports comparisons of model generalization, but we acknowledge that substantial changes in some hyperparameters can affect its validity for particular datasets. We therefore interpret ProDPR alongside task-specific predictive performance rather than judging model capability from ProDPR alone.
3. Fused shifts (C5, Q3)
With Auto references, fused retention remains below every single axis for all five models (Table 1; mean gap 0.3170). This persists beyond the InD reference choice, but does not prove an interaction or eliminate subset-composition differences. We have removed the super-additivity claim.
Table 1. Auto retention; gap = hardest single-axis retention minus fused retention.
4. Model coverage and seeds (C6, Q6)
GVP training from scratch is costly; resources and time limit seed and model coverage. Three-seed results on eight configurations preserve the GVP–SaProt mean performance ordering despite seed variability (Table 2). We will complete five seeds across all configurations and update the final paper.
Table 2. Predictive performance; three-seed GVP: mean ± SD. Bold indicates the larger new GVP/SaProt mean, not significance.
5. pH/PPI confounders (C7, Q4)
For pH, predicted structures compensate for missing experimental coverage in the source data. However, 93% of test entries use experimental structures, and OOD subsets are defined only within this test set. Thus, the subsets predominantly share the same structure type, substantially limiting structure-type differences between OOD conditions, although the training-to-test provenance shift remains. Appendix E discloses PPI negative-composition effects; released data will label negative types for controlled evaluation.
Excluding both tasks preserves the overall model ranking (Table 3). This supports the aggregate comparison, without resolving within-task confounding.
Table 3. Macro ProDPR from rounded Figure 3b scores, using the original single-seed GVP.
6. Chemical OOD (Q5)
POOR targets protein OOD across tasks; chemical OOD concerns only tasks with small-molecule inputs. For kcat/LBA, chemically unsupported validation/test examples are identified using a 0.6 maximum-Tanimoto threshold and reassigned to training (Section B.3). This retains measurements for fitting and reduces chemical-OOD interference, focusing evaluation on protein OOD under chemical support.