acceptodds
Under review as a conference paper at ICLR 2027

A Multi-Axis Out-of-Distribution Benchmark for Protein Understanding Tasks

Abstract

Protein representation learning has advanced rapidly, driven by large-scale language models and structure encoders. However, most existing benchmarks evaluate models under independent and identically distributed (IID) settings or rely on a single out-of-distribution (OOD) axis, such as sequence identity. In real-world applications, test proteins frequently diverge from the training distribution along multiple biological dimensions—sequence, structure and function—yet current benchmarks do not systematically measure model robustness under these diverse shifts. To address this gap, we introduce POOR (Protein Out-Of-distribution Robustness), a multi-axis OOD benchmark for protein understanding that spans 10 protein tasks, each annotated with 24–28 OOD scenario flags covering sequence/evolutionary OOD, structural OOD, and functional OOD. We further propose ProDPR (Protein Distributional Performance Retention), a unified OOD evaluation metric that enables comparison of model generalization difficulty across both single-axis OOD scenarios and their high-dimensional combinations. Using this framework, we benchmark protein language models at multiple scales and structure-aware models, and further examine the effects of different fine-tuning regimes. We find that stronger in-distribution performance does not necessarily imply greater OOD robustness: model scale and structural inductive bias confer distinct, shift-dependent advantages, while fused multi-axis shifts expose substantially larger failures than conventional single-axis evaluation.

Primary area:
datasets and benchmarks
Keywords:
out of distribution, generalization ability, protein function prediction, protein representation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qk13OpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces POOR, a multi-axis OOD benchmark covering 10 protein understanding tasks, where test samples are annotated with non-exclusive sequence/evolutionary, structural, and functional OOD flags. It further proposes ProDPR, a reference-relative metric for measuring predictive utility retained under distribution shift. Experiments across protein language models and structure-aware models show that OOD robustness is shift-dependent, and that fused multi-axis shifts are generally more challenging than individual shifts.

Critical strengths
  1. Comprehensive multi-axis OOD evaluation. POOR evaluates multiple biologically meaningful and overlapping forms of distribution shift across a broad range of protein tasks, providing a more realistic view of generalization than single-axis benchmarks.
  2. Useful empirical analysis. The experiments reveal meaningful differences between predictive performance and OOD robustness, as well as shift-dependent effects of model scale, structural information, and fine-tuning strategy.
Critical weaknesses
  1. Benchmark availability is unclear. I could not find a clear repository or release statement for the processed splits, OOD annotations, or evaluation implementation. For a benchmark paper, public access to the exact data and evaluation scripts is important.

  2. The null predictor in ProDPR is underspecified. The paper describes it only as a task-specific non-informative predictor, such as a label-prior baseline, without clearly defining it for every task and metric.

  3. ProDPR sensitivity is insufficiently studied. The effects of the null model, ϵ\epsilon, nmin⁡n_{\min}, weighting strategy, and reference-pool choice are not systematically evaluated.

  4. Some InD reference pools are very small. For example, the InD subset constitutes only 1.7% of the pH test set and around 3% for several EC/GO tasks, potentially making the denominator of ProDPR unstable.

  5. The interpretation of fused OOD shifts may be too strong. Sequence, structure, and functional novelty are biologically and statistically correlated, and lower performance on their intersection does not necessarily imply a genuine compounding interaction rather than selection of intrinsically harder samples.

  6. The empirical model coverage and uncertainty are somewhat limited. The benchmark is broad, but experiments mainly involve ESM-2 variants, SaProt, and GVP-GNN, and GVP-GNN is reported with only one seed while the PLM results use five runs.

  7. Several task-specific confounders complicate the OOD interpretation. In particular, the pH task mixes biological OOD with an AlphaFold-to-experimental structure provenance shift, while PPI performance is strongly affected by negative-pair composition.

Questions for authors
  1. Will the complete POOR benchmark be publicly released, including the exact train/validation/test splits, OOD annotations, structure identifiers, and ProDPR evaluation code?

  2. How is the task-specific null predictor $f_{\mathrm{null}}$ defined for each metric, and what values are used for ϵ\epsilon and nmin⁡n_{\min}? How sensitive are the main ProDPR rankings to these choices and to the InD versus Auto reference pool?

  3. Do the conclusions about fused multi-axis OOD shifts remain when using background-matched Auto references or another analysis that controls for sample composition? This would help distinguish genuine compounding effects from harder intersection subsets.

  4. How much do the benchmark-level conclusions change after controlling for the PPI negative-pair composition and the pH structure-provenance shift, or after excluding these two tasks from the aggregate results?

  5. What is the rationale for moving chemically dissimilar kcat/LBA validation or test samples back into the training set, rather than retaining them as explicit chemical-OOD subsets or excluding them from evaluation?

  6. Why are five independent runs reported for the PLM models but only one run for GVP-GNN? Could comparable uncertainty estimates be provided across all evaluated models?

3 comments

1. Availability (C1, Q1)

Links were withheld for anonymity. We will release the splits, OOD annotations, structure identifiers, reference pools, processing pipeline, and evaluation code, with long-term maintenance, task expansion, and issue correction.

2. ProDPR definition and sensitivity (C2–C4, Q2)

Appendix A.1 now specifies the nulls: training-label prevalence vectors for EC/GO; majority classes for Fold/SSP; constant training-prevalence scores for AUROC (0.5 when both classes exist); and zero utility for Spearman by convention. Training priors remain fixed across strata and references.

We use ϵ=10−8\epsilon=10^{-8}, nmin⁡=20n_{\min}=20, equal stratum weights, and InD references, excluding undefined metrics and near-zero reference gaps. Sensitivity checks show:

  • Raising ϵ\epsilon to 10−410^{-4} changes scores by at most 0.00028 across 47 runs.
  • Size-weighted aggregation changes scores by up to 0.250.
  • On four fully replayed configurations, rankings are unchanged for nmin⁡=10n_{\min}=10 versus 20; at 50, ESM-2-650M moves from fifth to first.
  • The current InD–Auto comparison reports unchanged rankings on nine of twelve configurations.

Small reference pools remain uncertain. ProDPR supports comparisons of model generalization, but we acknowledge that substantial changes in some hyperparameters can affect its validity for particular datasets. We therefore interpret ProDPR alongside task-specific predictive performance rather than judging model capability from ProDPR alone.

3. Fused shifts (C5, Q3)

With Auto references, fused retention remains below every single axis for all five models (Table 1; mean gap 0.3170). This persists beyond the InD reference choice, but does not prove an interaction or eliminate subset-composition differences. We have removed the super-additivity claim.

Table 1. Auto retention; gap = hardest single-axis retention minus fused retention.

ModelAA: SeqIDBB: TM-scoreCC: NewEC-L4A∩B∩CA\cap B\cap CGap
ESM2-35M0.80210.69530.79520.24340.4519
ESM2-150M0.77880.64290.74690.28230.3606
ESM2-650M0.85430.76550.82850.37890.3866
SaProt-650M0.81890.73340.90320.53050.2029
GVP-GNN0.77190.69671.06100.51380.1829

4. Model coverage and seeds (C6, Q6)

GVP training from scratch is costly; resources and time limit seed and model coverage. Three-seed results on eight configurations preserve the GVP–SaProt mean performance ordering despite seed variability (Table 2). We will complete five seeds across all configurations and update the final paper.

Table 2. Predictive performance; three-seed GVP: mean ± SD. Bold indicates the larger new GVP/SaProt mean, not significance.

TaskMetricOriginal GVP-GNNThree-seed GVP-GNNSaProt-650M
ECFmax0.5940.5915±0.00740.5915 \pm 0.00740.8010.801
GO-BPFmax0.3060.3656±0.02860.3656 \pm 0.02860.5970.597
GO-CCFmax0.4570.4775±0.01170.4775 \pm 0.01170.6070.607
GO-MFFmax0.6010.5951±0.00800.5951 \pm 0.00800.6880.688
SSPResidue accuracy0.9690.9667±0.00110.9667 \pm 0.00110.888
kcatk_{\mathrm{cat}}Spearman0.3870.3722±0.02000.3722 \pm 0.02000.282
pHSpearman0.2840.2992±0.00990.2992 \pm 0.00990.4330.433
LBASpearman0.6590.6604±0.00180.6604 \pm 0.00180.594

5. pH/PPI confounders (C7, Q4)

For pH, predicted structures compensate for missing experimental coverage in the source data. However, 93% of test entries use experimental structures, and OOD subsets are defined only within this test set. Thus, the subsets predominantly share the same structure type, substantially limiting structure-type differences between OOD conditions, although the training-to-test provenance shift remains. Appendix E discloses PPI negative-composition effects; released data will label negative types for controlled evaluation.

Excluding both tasks preserves the overall model ranking (Table 3). This supports the aggregate comparison, without resolving within-task confounding.

Table 3. Macro ProDPR from rounded Figure 3b scores, using the original single-seed GVP.

ModelAll 12 configurationsRemaining 10Rank (before →\to after)
ESM-2-35M0.59980.55745→55\to5
ESM-2-150M0.61720.57504→44\to4
ESM-2-650M0.64640.61083→33\to3
SaProt-650M0.67670.63362→22\to2
GVP-GNN0.73580.68681→11\to1

6. Chemical OOD (Q5)

POOR targets protein OOD across tasks; chemical OOD concerns only tasks with small-molecule inputs. For kcatk_{\mathrm{cat}}/LBA, chemically unsupported validation/test examples are identified using a 0.6 maximum-Tanimoto threshold and reassigned to training (Section B.3). This retains measurements for fitting and reduces chemical-OOD interference, focusing evaluation on protein OOD under chemical support.

Thank you for the rebuttal, which partially addresses my concern. However, I still have a few remaining concerns:

ProDPR sensitivity: The additional analysis is helpful, but some results actually reinforce my concerns about the stability of ProDPR. For example, changing the minimum stratum size to 50 moves ESM-2-650M from fifth to first, while size-weighted aggregation leads to substantial differences in ProDPR scores. I understand that some sensitivity to hyperparameters is expected, but such large changes make me wonder how reliable the overall model comparisons are. The uncertainty associated with very small InD reference pools also remains a concern.

Chemical OOD: I understand the motivation for controlling chemical OOD to focus on protein OOD. However, moving validation/test samples into training can change the protein distribution and invalidate the original OOD annotations. How can the authors guarantee the original split constraints (e.g., protein-level disjointness for kcat and temporal splitting for LBA) are still preserved after reassignment?

Benchmark availability: I appreciate the commitment to releasing the benchmark, but I would really like to see an anonymous repository during the review process. Since this is a benchmark paper, being able to examine the actual data splits, OOD annotations, and evaluation code would help me better assess its reproducibility. If the complete benchmark is too large to share at this stage, providing one representative task with its processed splits, OOD annotations, and evaluation scripts would be helpful as well.

Thank you for the follow-up questions.

1. ProDPR sensitivity and reference uncertainty

Minimum stratum size. nmin⁡=10n_{\min}=10 and 20 yield identical strata and scores. Raising it to 50 removes four of eighteen strata (22.2%), but only 159 of 14,980 samples (1.06%). ESM-2-650M moves from fifth to second, mainly because poorly retained rare strata are excluded; GVP remains first. We use 20 to balance scenario coverage and minimum support; this threshold is a compromise dictated by current data sizes, and larger datasets may support higher thresholds.

Aggregation weights. Equal weighting reflects our goal of evaluating robustness across OOD conditions without allowing frequent conditions to dominate rare ones. Sample-size weighting increases the influence of common strata and reduces that of rare OOD scenarios; it answers a different evaluation question and changes rankings in our experiments. We therefore use equal weighting as a default suited to the benchmark's goal, without claiming that rankings are invariant to weighting. Deployment-aware weighting is also supported by related work: Fernández-Díaz et al. [1] incorporate prior knowledge of the target deployment distribution into AU-GOOD. Exploring such weighting schemes and curves that vary the emphasis on different test conditions is a useful future direction for POOR.

ProDPR provides a summary reference for comparing model generalization, rather than a standalone judgment of model capability. We interpret it alongside predictive performance. More detailed diagnosis can use the original task metrics on individual OOD subsets, as illustrated by the subset-level comparisons in our fourth experimental analysis (Section 4.4). We respectfully ask that these limitations of ProDPR be weighed separately from the contributions of the dataset and its multi-axis OOD design, which also support evaluation using the original task metrics without ProDPR.

2. Chemical reassignment and split integrity

Chemical-support reassignment is part of split construction and precedes OOD annotation. The annotation scripts read the resulting split files and compute similarity-based OOD flags against the resulting train/validation reference; training-based functional and holdout annotations likewise use the resulting training split. Thus, the intended pipeline does not retain annotations computed against the provisional splits.

For kcat, the initial split is defined over UniProt groups, and chemical reassignment moves entire UniProt groups, not individual protein–substrate records. This avoids splitting a reassigned protein's records between training and evaluation.

For LBA, the temporal constraint is within each ligand-similarity cluster, rather than a single global date cutoff. Clusters require pairwise Tanimoto similarity ≥0.6. Every non-singleton cluster initially has a training member, so its held-out members already satisfy the chemical-support threshold. Singleton clusters without chemical support are moved to training without reversing the order of retained members within a cluster. Subsequent ratio adjustments move the earliest test entries to validation and the earliest validation entries to training. OOD annotation follows these adjustments.

3. Anonymous benchmark access

The anonymous repository now includes complete kcat train/validation/test CSVs (15,479 / 2,150 / 5,002 records) and test OOD annotations under POOR/datasets/enzyme_kinetics_prediction/splits/, together with processing, annotation, and evaluation scripts and the ProDPR implementation. The README and task documentation describe usage. The complete benchmark data, including all remaining task splits and the large structure files, will be publicly released together after the review process concludes, under the long-term maintenance plan described in our response to Comment 1.

Reference

[1] Fernández-Díaz et al. (2025). A New Framework for Evaluating Model Out-of-Distribution Generalisation for the Biochemical Domain. ICLR. Paper.

Reviewer_VderOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces POOR (Protein Out‑Of‑distribution Robustness), a multi‑axis out‑of‑distribution(OOD) benchmark for protein‑understanding tasks. Real‑world proteins deviate from training data across sequence‑evolutionary, structural, and functional axes simultaneously, yet most existing protein benchmarks rely on independent‑and‑identically‑distributed(IID) splits or only single‑axis OOD shifts (e.g. sequence identity) and cannot evaluate combined, high‑dimensional distribution shifts. POOR covers 10 protein tasks (12 total task configurations) spanning function annotation, structure classification, interaction, binding and biophysical property prediction.

Critical strengths
  1. Multi‑axis composable OOD benchmark design reflecting real‑world protein distribution shifts. Existing protein OOD evaluations mostly focus on one single dimension (sequence homology alone). POOR systematically disentangles sequence‑evolutionary, structural and functional novelty with non‑exclusive scenario flags, supporting evaluation for both individual shifts and their joint combinations.

  2. ProDPR provides a cross‑task comparable OOD robustness metric. Protein prediction tasks use highly divergent raw evaluation metrics (F‑max, accuracy, Spearman correlation, AUROC), making direct OOD‑robustness comparison across tasks infeasible with original scores.

Critical weaknesses
  1. The empirical evaluation only includes ESM‑2 variants, SaProt‑650M and GVP‑GNN. It does not incorporate more recent multimodal protein models such as ESM‑3, ESM-C or other contemporary structure‑sequence joint models.
Questions for authors
  1. You observe that full fine‑tuning leads to severe overfitting on OOD strata for ESM‑2 at intermediate scales. Have you explored whether SFT techniques can close the OOD‑performance
2 comments

Comment 1

Reviewer:

The empirical evaluation only includes ESM‑2 variants, SaProt‑650M and GVP‑GNN. It does not incorporate more recent multimodal protein models such as ESM‑3, ESM-C or other contemporary structure‑sequence joint models.

Response:

We selected ESM-2, SaProt, and GVP-GNN as established representatives of sequence-based protein language models, structure-aware protein language models, and graph neural networks, respectively. Our focus is to validate the value of the benchmark and its multi-axis evaluation design.

We have now evaluated ESM-C-600M and ESM-3-1.4B on pH and PPIS using three seeds (2024, 2025, and 2026). Table 1 compares ESM-C with ESM-2-650M and ESM-3 with SaProt-650M. On pH, ESM-C has lower Spearman correlation than ESM-2 (0.399 vs. 0.438) despite similar mean ProDPR (0.545 vs. 0.540), whereas ESM-3 improves on SaProt in both metrics. On PPIS, ESM-C has lower AUROC than ESM-2 (0.759 vs. 0.772), while ESM-3 matches SaProt at the reported precision (0.818). The preliminary PPIS ProDPR means are higher for both new models. Thus, these newer models do not show a consistent advantage across both predictive performance and retention on the evaluated tasks; these results do not establish a broader model ranking.

Table 1. Additional evaluations. New models: mean ± sample standard deviation over three seeds. ESM-2 and SaProt: original Figure 3 point estimates.

TaskModelPredictive performanceProDPR
pH (Spearman)ESM-2-650M0.4380.540
pH (Spearman)ESM-C-600M0.399±0.0470.399 ± 0.0470.545±0.1650.545 ± 0.165
pH (Spearman)SaProt-650M0.4330.498
pH (Spearman)ESM-3-1.4B0.449±0.0110.449 ± 0.0110.543±0.0210.543 ± 0.021
PPIS (AUROC)ESM-2-650M0.7720.805
PPIS (AUROC)ESM-C-600M0.759±0.0030.759 ± 0.0030.821±0.0190.821 ± 0.019
PPIS (AUROC)SaProt-650M0.8180.931
PPIS (AUROC)ESM-3-1.4B0.818±0.0030.818 ± 0.0030.960±0.0300.960 ± 0.030

Question 1

Reviewer:

You observe that full fine‑tuning leads to severe overfitting on OOD strata for ESM‑2 at intermediate scales. Have you explored whether SFT techniques can close the OOD‑performance

Response:

Thank you for highlighting this important direction. Our primary contribution is the benchmark and multi-axis evaluation framework, so the present study includes only a limited analysis of training strategies. We have not yet systematically evaluated whether alternative fine-tuning techniques close the OOD gap on POOR.

Related work provides strong motivation: Schmirler et al. [1] study supervised and parameter-efficient fine-tuning across diverse protein tasks, while Xu and Bonvin [2] investigate layer-wise selective fine-tuning for antibodies. These studies motivate further experiments, but do not establish whether the same benefits hold across POOR's OOD conditions. We plan to use POOR for more extensive analyses of fine-tuning strategies and pursue this question in follow-up work. We appreciate the reviewer's suggestion, which identifies an important application of the benchmark.

References

[1] Schmirler, R., Heinzinger, M., & Rost, B. (2024). Fine-tuning protein language models boosts predictions across diverse tasks. Nature Communications, 15, 7407. DOI: 10.1038/s41467-024-51844-2.

[2] Xu, X., & Bonvin, A. M. J. J. (2026). AbTune: layer-wise selective fine-tuning of protein language models for antibodies. Briefings in Bioinformatics, 27(4), bbag374. DOI: 10.1093/bib/bbag374.

Thanks for the rebuttal, and the provided extra experimental results.

Reviewer_33knOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper has three main contributions: 1) a new perspective in how OOD should be considered in protein benchmarks not along a single axis or properties, but across multiple; 2) a benchmark showcasing such an analysis with test proteins annotated with multiple OOD labels/properties/flags; 3) a metric to integrate the multivariate observations into a single value to streamline method comparisons.

Critical strengths
  1. Paper is clearly written and easy to follow.
  2. Choice of the major OOD directions: function, sequence, and structure; follows traditional conventions of the field and is intuitive. Also informative for major application domains.
  3. Choice of the OOD subdirections, it is also quite good and well founded in previous literature, with all main directions having been previously explored (separately) in previous studies.
  4. Analysis of axis overlap is interesting.
  5. The empirical observations regarding failure modes are aligned with previous studies and our established domain understanding of the mechanisms underlying the tasks. For example sequence function annotation failing under structural OOD settings or sequence length or holdout being the hardest settings for binding prediction methods (or enzyme activity).
  6. The quality control checks described in the Appendices are well thought-out and thorough
Critical weaknesses
  1. Although the overlap analysis was a good choice. It is limited in that it only considers test set label overlap. It does not consider to what extent two different axis may be measuring the exact same underlying property. For example, NewFunction and LongTailFunction could lead to similar failure modes that are highly correlated, making the two metrics redundant. However, the current analysis is not able to capture this potential problem. This is important because if two directions are highly correlated they might tend to skew results for methods that are better in those kinds of extrapolation problems, seeming like they are overall better generalization methods, when in reality it would be an artifact of the benchmark/multi-axis design.
Questions for authors
  1. You have an interesting amount of information regarding the relationship between similarity (sequence or TM) and model performance [Table 6]. I think from the paper that only the highest and lowest thresholds are considered, is that correct or am I misunderstading something? If not, how are you integrating the information from the different similarity levels?
  2. What proportion of the data needs to be hold-out for the test set for the different datasets? (I apologize if this described in the paper, I could not find it).
  3. If my understanding of how the method is implemented is correct, all the axis are evaluated through a single training-test iteration. Is this correct? If so, I'll move this comment to the strengths section, as I think that it is a major contribution. As other OOD analysis methods like SPECTRA [1] or Hestia-GOOD [2], require multiple training-test iterations to obtain their metric.

  1. Ektefaie, Y., Shen, A., Bykova, D., Marin, M. G., Zitnik, M., & Farhat, M. (2024). Evaluating generalizability of artificial intelligence models for molecular datasets. Nature Machine Intelligence, 6(12), 1512-1524.
  2. Fernandez-Diaz, R., Thanh Lam, H., Lopez, V., & Shields, D. (2025, May). A new framework for evaluating model out-of-distribution generalisation for the biochemical domain. In International Conference on Learning Representations (Vol. 2025, pp. 74207-74232).
1 comment

C1. Correlated OOD scenarios

We agree that annotation overlap does not establish independence of model failure modes, and some scenarios are inherently nested. At the same EC level, NewFunction (training frequency zero) is a subset of LongTailFunction when the latter includes zero-frequency labels. Likewise, SeqID$<30%$ is a subset of SeqID$<70%$.

ProDPR does not simply average scores over these overlapping subsets. It partitions the selected descriptor space into mutually exclusive strata, evaluates retention within each valid stratum, and aggregates those values; each protein contributes to at most one retained stratum (Appendix A.1). This avoids directly counting the same sample in several overlapping OOD subsets, but does not fully remove the influence of correlated descriptors on partitioning and weighting.

Users can select the axes and severity levels relevant to their application rather than include all annotations. For example, our joint model comparison selects sequence, structure, and NewEC axes. We will clarify that these axes need not be independent and that ProDPR comparisons should use a common, explicitly specified partition.

Q1. Similarity thresholds and aggregation

Table 6 reports coverage for the complete OOD scenario annotations, including sequence-identity thresholds from 30% to 90%, rather than only the endpoints. These threshold flags describe nested test subsets; they need not all be included in an individual evaluation.

The experimental settings differ by analysis. Section 4.1 reports performance on representative scenarios, including length, disorder, and functional shifts. Sections 4.2 and 4.3 focus on the sequence, structural, and NewEC dimensions; the explicit single-axis versus intersection comparison uses SeqID$<0.7$, TM-score$<0.7$, and NewEC-L4, with task-specific exceptions such as NewEC-L1 for Fold. Section 4.4 uses the full set of available OOD axes for its ProDPR analysis.

Overall predictive performance is generally reported on the Default test subset, while scenario-specific results use the corresponding subsets. ProDPR instead evaluates the selected axes and severity levels by partitioning their joint descriptor space into disjoint strata and aggregating stratum-level retention. Thus, the same metric handles different OOD combinations without restricting evaluation to the highest and lowest similarity thresholds.

Q2. Test-set proportions

The task-specific split protocols are described in Appendix B, and the final train/validation/test counts are reported in the dataset-scale table in Appendix C. We apologize that these details were not easy to locate. The final test proportions (test samples divided by total samples) are: Fold: 20.65%, EC: 22.81%, GO-MF: 21.68%, GO-CC: 26.45%, GO-BP: 23.25%, PPI: 19.45%, kcatk_{\mathrm{cat}}: 22.10%, pH: 25.44%, LBA: 20.29%, LBS: 17.89%, PPIS: 22.77%, SSP: 20.27%.

These proportions can differ from the base split ratios because of task-specific filtering, holdout rules, and sample reassignment. The complete data-processing pipeline, including split construction and OOD annotation scripts, will be available in the public dataset repository when the benchmark is released.

Q3. One training run for multiple OOD evaluations

Yes, your understanding is correct. For each model, task, and random seed, a single training run on the fixed split suffices to evaluate all applicable OOD axes, severity levels, and their combinations using the same test predictions. Changing the evaluated OOD conditions does not require retraining or constructing a new train/test split. Enabling this capability was one of our original motivations: to support flexible, multi-axis OOD evaluation without repeatedly training models for different evaluation scenarios. Thank you for highlighting this advantage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.