acceptodds
Under review as a conference paper at ICLR 2027

TabBench-Bio: A Living Benchmark for Machine Learning on High-Dimensional Biomedical Tables

35 views

Abstract

Biomedical tables often combine thousands of measured variables with only tens or hundreds of labelled samples, a regime that is poorly represented in general-purpose tabular benchmarks. We introduce TabBench-Bio, a living and interactive benchmark of 43 biomedical datasets spanning multiple domains. Under a shared cross-validation protocol, we compare classical estimators, neural networks, and tabular foundation models across 28 feature-by-sample operating points. At the reference cell of 10,000 features and 100 training samples, RealTabPFN 2.5 has the highest point estimate, closely followed by TabPFN 3 and Logistic Regression. A paired bootstrap over the target pool separates RealTabPFN 2.5 from TabPFN 3 by 87 Elo (95% interval [43, 129]). Tabular foundation models generally occupy the leading ranks, while the strongest configuration depends on the operating point and biomedical modality. The AutoML framework AutoGluon, using its one-hour “extreme” preset, is configured as a separate resource-intensive reference and reported here at the reference and full cell. Fold-level predictions, run status, and deterministic aggregations make every reported result reproducible and reusable. The benchmark is open to contributions of new biomedical datasets. The interactive leaderboard is available at https://anonymous_iclr.codeberg.page/TabBench-Bio-Anonymous/.

Primary area:
datasets and benchmarks
Keywords:
Tabular Foundation Models, High-Dimensional Low-Sample-Size, Benchmark, TabPFN, AutoGluon

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hHXNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a new benchmark for the tabular domain focusing on biomedical datasets. The benchmark is comprised of already published datasets, a strict and very detailed evaluation setup and a first analysis of existing state-of-the-art models. The current benchmark page (anonymised) contains very recent models that have been even released after the paper submission.

Critical strengths
  1. The paper is well written and very detailed when it comes to the experimental setup. This is very much appreciated. I also like that issues such as memory problems and other reasons for model training failures have been handled and analyzed in depth.
  2. Since there are plenty of applications dealing with biomedical tabular data - the proposed benchmark is indeed relevant.
  3. The paper is using benchmarking techniques and metrics used in recent prominent benchmarks like TabArena.
Critical weaknesses
  1. What are the common properties of the datasets in the benchmark? The targeted applications are pretty diverse (from molecular-property tasks to gene expressions etc.). A benchmark that compiles multiple datasets and aggregates results across these datasets suggests that there is a common prior also relevant for other applications. As a practitioner, If I have a new biomedical tabular dataset, how likely is it that the model tanking holds or should I rather look for the ranking within a certain modality group. Their common attribute of being "biomedical" might be not enough to draw relevant conclusions. I would love to see these questions tackled in more depth, which also relate the questions how close is this benchmark (in data distribution) to TabArena or RamanBench for example.
  2. Of course, not doing any hyperparameter optimization is unfortunate, but I know that this requires a completely different magnitude of compute. It is a weakness, but not a critical one in my opinion.
  3. I do not get, why analyzing feature width restrictions is important. In my opinion, this leads to more issues than answers to research questions. Feature representations are cut-off in experiments (making feature order a relevant factor), etc. - these limitations were mentioned by the authors themselves, but I am not sure it was the right decision for the paper. However, varying the training size nn is reasonable and studying its effects led to some insights whether a certain model is able to deal with a data-scarce application (as illustrated e.g. in Fig. 3).
Questions for authors

Minor comments and questions:

  1. Indeed not a critical question, but maybe an interesting discussion aspect in our community: given the fact that we see so many "living" benchmarks appearing in our field, how can we guarantee their maintainability in the future (you do not have to answer this in the rebuttal, since this question is beyond this single paper).
  2. L200: "budgets" might be not the right wording here.
  3. It would be reasonable to list the differences to the experimental setup used in TabArena (beyond L83ff).
  4. I haven't seen a plot showing task-type specific performance, e.g. what is the best model for gene expression for DNA methylation?
2 comments

Thank you for the careful review. We have clarified the shared evaluation setting and feature-width rationale, added modality-specific results, and explained the differences from TabArena. We address your concerns and questions below:


Regarding CW1 (Common properties and transfer to new datasets):

The datasets share an evaluation setting with many numerical features and limited labelled training data; we do not assume a common biological prior, additionally, their representations and endpoints differ.. TabArena covers broader IID tabular tasks, which are not HDLSS, while RamanBench uses physically ordered spectra whereas TabBench-Bio has 32 of the 43 tables with more features than rows.

For a new dataset, the modality-specific rankings on the interactive leaderboard can help form a shortlist, considering representation, endpoint, sample size and computing resources. Final selection still requires validation with the intended biological grouping and deployment setting. We have not tested transfer to external sites.


Regarding CW2 (Hyperparameter optimization):

We agree that the absence of hyperparameter optimization is a limitation. Since submission, we have added tuned variants of Logistic Regression/Ridge, the MLP, Random Forest, Extra Trees and XGBoost to the live leaderboard, each selected from a small predefined grid on a training-only holdout. These variants are not part of the comparison reported in the paper.


Regarding CW3 (Feature-width rationale and effective dimensions):

At the reference operating point, 28 of the 42 eligible datasets retain the full 10,000 features in all five folds; the remaining 14 retain between 1,142 and 8,192 features. Riboflavin is not eligible at this point because it has fewer than 100 training rows. We now report this count in the main text and show the retained widths by modality in Appendix F (Figure A5b).

We have also added the rationale for varying feature width to the operating-point description: For gene-expression, methylation and genomic-prediction data, smaller subsets approximate assays that measure fewer genes, CpG sites or variants; for metagenomic profiles, fingerprints and embeddings, they test sensitivity to retaining only part of the representation. Because increasing the width changes both dimensionality and the available information, the sweep does not isolate dimensionality alone, and random subsets do not reproduce a particular assay or biologically selected panel.


Regarding Q1 (Maintenance of the living benchmark):

TabBench-Bio is maintained by the authors and open to community contributions. Datasets and models are proposed through GitHub issues or PRs. A dataset must be publicly accessible under an open licence, have a defined classification or regression target, and record its source, licence and citation; it then is evaluated on checks described in Section 3.1.

A model only needs an adapter and declared metadata. The maintainers review each contribution and evaluate it under the unchanged operating-point configurations before it enters a versioned release on GitHub and PyPI. TabBench-Bio thus combines community contributions with curation by dedicated maintainers. We are committed to maintaining TabBench-Bio and evaluating new models and datasets.


Q2

True, we’ve replaced "budgets" with "settings".


Regarding Q3 (Experimental-protocol differences from TabArena):

Compared with TabArena, our protocol differs in several respects:

  1. Scope: TabBench-Bio focuses on biomedical HDLSS tasks and varies both feature width and training-sample count.
  2. Configurations: the submitted comparison evaluates fixed configurations without per-dataset hyperparameter search or benchmark-level bagging/stacking; TabArena evaluates defaults, tuned configurations and ensembles.
  3. Validation: we use one five-fold outer partition shared across operating points, preserving biological groups where supplied; TabArena uses repeated three-fold outer cross-validation.
  4. Metrics: we use macro-F1 for classification, whereas TabArena uses AUROC for binary tasks and log loss for multiclass tasks; both use RMSE for regression.

We also explicitly score attempted memory/time-limit failures at constant-predictor performance. These differences mean that contrasts between the benchmarks cannot be attributed to domain alone.


Regarding Q4 (Modality-specific performance):

We have added a modality-by-model heatmap in Appendix E (Figure A4) at the strict reference cell (10,000-feature cap, 100 training samples). It reports ranks from the same Bradley-Terry Elo procedure within each modality, with dataset counts and classification/regression composition shown. RealTabPFN 2.5 leads gene expression, metagenomics and molecular properties; TabFM leads embeddings, TabPFN-Wide leads methylation, Random Forest leads genomic prediction. Those results can also be inspected on the website.

Reviewer_hHXNOpenReview ↗

Thanks for the rebuttal

Thank you very much for the detailed rebuttal, this helped clarify some of my questions. As mentioned in the original review, I do not see CW2 as really critical, Q1 was just a thought mostly related to often unseen work that is required to keep such a benchmark alive. I also appreciate the explicit list of differences that allows comparison to TabArena.

Reviewer_Qtc4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents TabBench-Bio, a benchmark for ML approaches for learning biomedical tabular data. To this end, the proposed benchmark comprises 43 high-dimensional, low-sample size datasets across different modalities.
The benchmark also includes 19 model configurations (reported in the paper) from different model families, ranging from traditional logistic/ridge regression to the most recent tabular in-context models. The claimed contribution is the systematic evaluation of these different model variants, benchmarking their performance across scenarios of different numbers of samples and features. The general conclusion is that recent tabular foundation models generally perform well, though their performance can vary across different data modalities and tasks. Classical linear regressors are also competitive (at a fraction of the computational cost compared to large ICL models), which is not surprising for these regimes. TabBench-Bio is a living benchmark, and since the submission of this paper has extended in number of models, and provides additional leader board and resources for reproducibility. Overall, I think this is a useful resource, but IMO the contributions are very limited.

Critical strengths
  • An important and practically relevant benchmark, as many biomedical applications rely on HDLSS tabular data
  • The experimental setup is rigorous, the findings are useful and reasonably discussed.
  • The live leaderboard can be helpful for quick reference in practice.
Critical weaknesses
  • W* I understand that this is a live benchmark, which allows easy extension to more models and datasets. However, for the purpose of this submission, I find this rather lacking fairness. The link the authors provide, although anonymized, leads to the live benchmark. Seems that, since the submission deadline, additional models and configurations have been added. As such, in the reminder, I will focus solely on the submitted version (ie. what is stated in the manuscript).

  • W1. The biggest weakness IMO is the novelty of this work. Beyond the selection of the particular datasets/tasks (many of which have already been available), I do not really see a sufficiently original scientific contributions relative to existing benchmarks.

  • W2. There are many details missing on how (and why) the tasks were chosen/designed for different datasets. Some of the tasks in this benchmark have been defined by the original authors, but I believe there are several that have been developed by the authors of TabBench-Bio. This needs to be discussed, as some of the chosen tasks seem too easy (most models perform almost perfectly; eg the TCGA tasks) and therefore may be a rather saturated benchmark. The authors state that they explicitly handle such cases for regression, but there is no mention of classification.

  • W3. In terms of scope, I don't see how the decision to evaluate fixed configurations across the different HDLSS tasks, rather than tuning the models, is a benefit in this case. While it reads as such (L082-L084), I'd argue that tuning is more important in such a regime. (Note that I am referring to what is reported in the manuscript)

Questions for authors
  • See weaknesses

  • What's the protocol of deciding what to add/consider and adding models and dataset (or a task)? Is TabBench-Bio suppose to be a community effort or it will have dedicated maintainers/curators? Both of these can have their pros and cons, so I'm curious how the authors see their resource evolving.

  • What's the class/target distribution of the chosen tasks?

  • How does the implemented Bradley-Terry ranking account for task coverage? As some models are evaluated only on subset of tasks (L230-L231), this may affect the overall ranking (as it's the case when excluding the failure runs)

  • The authors acknowledge that 3 evaluated datasets originate from the same GEMLeR/expO collection as data used to pretrain TabDPT. Why wasn't TabDPT excluded from these datasets, as it's the case for the other models that can't be applied?

  • Could the authors clarify whether the evaluated methods use their original implementations or have been re-implemented/adapted for the benchmark?

  • What is considered to be "fit" time (Fig A1), when certain models are fitted from scratch, some are fine-tuned, and some used only for inference?

  • To what extent does TabBench-Bio adhere to the FAIR (Findable, Accessible, Interoperable, Reusable) principles? In particular across benchmark releases, given that the repository does not seem to provide fully pinned software dependencies and environments?

3 comments

Thank you for the careful review. We have clarified task provenance and difficulty, added target distributions, and separated the submitted comparison from subsequent leaderboard additions. We address your concerns and questions below:


Regarding W* (Submitted comparison and the living benchmark):

We agree that the submission should be assessed against a fixed roster. We now provide an anonymous paper snapshot at https://anonymous_iclr.codeberg.page/TabBench-Bio-Anonymous/paper-snapshot/ containing only the 43 datasets and 19 predictive configurations evaluated in the paper, with the constant baseline and separate AutoGluon reference.

Section 8, "Outlook and Living Benchmark", separates the submitted comparison from subsequent additions and illustrates changes in the leading group (Figure 4). These later additions are not part of the main comparison reported in the paper.


Regarding W1 (Scientific contribution):

We agree that TabBench-Bio introduces no new learning algorithm and that much of the underlying data is public. Nevertheless, to the best of our knowledge, no published living benchmark is dedicated to comparing tabular learners across heterogeneous biomedical HDLSS modalities. The main contribution is the feature-by-sample grid on shared folds, which shows how model rankings depend on the operating point. Across the 28 operating points, RealTabPFN 2.5 leads at 23, whereas Logistic Regression ranks among the top three at 10 operating points, all with at most 100 training samples, and at none with more. An evaluation at a single operating point would not reveal this dependence.


Regarding W2 (Task provenance and difficulty):

Most endpoints were taken from their original sources. When no prediction task was predefined, we selected a target from the available sample annotations, aiming for a familiar, domain-relevant prediction task. For TCGA, we used tumour-versus-normal classification from expression; for MGnify, prediction of recorded sample metadata. For the protein and variant tasks, we retained the source labels (such as Variant Effect) and constructed features using pretrained language-model embeddings. Section 3.1 describes the review of each task’s target, prediction unit, licence, group identifiers and reproducible table construction.

We added Appendix I with per-dataset performance for all 30 classification and 13 regression tasks, at 100 training samples and with all available training samples. At 100 samples, median macro-F1 reaches 0.95 on two of the 30 classification tasks; with all samples, it reaches 0.95 on six. A few tasks therefore approach saturation, particularly with more training data, but the benchmark as a whole spans a broad range of task difficulties.


Regarding W3 (Fixed configurations and hyperparameter tuning):

We agree that tuning can be important in HDLSS settings, and have clarified this limitation in the manuscript. Tuning every model across the full feature-by-sample grid would require substantially more computation. We therefore initially evaluated how specific models behave as these settings change, without claiming the best attainable performance of each model family or that fixed configurations are preferable. This allowed us to evaluate a broader set of methods and operating points.

Since submission, the living benchmark has added further tabular foundation models and tuned variants of Logistic Regression/Ridge, MLP, Random Forest, Extra Trees and XGBoost. Each tuned variant uses a small predefined grid, balancing search space and computation, selects its configuration on a training-only holdout and is refitted on the full training fold. These additions are outside the evaluation reported in the manuscript.

Regarding Q2 (Admission rules and maintenance):

TabBench-Bio is maintained by the authors and open to community contributions. Datasets and models are proposed through GitHub issues or pull requests, following a public contributor guide. A dataset must be publicly accessible under an open licence, have a defined classification or regression target, and record its source, licence and citation; it then is evaluated on checks described in Section 3.1.

A model only needs an adapter and declared metadata. The maintainers review each contribution and evaluate it under the unchanged operating-point configurations before it enters a versioned release on GitHub and PyPI. TabBench-Bio thus combines community contributions with curation by dedicated maintainers. We are committed to maintaining TabBench-Bio and evaluating new models and datasets.


Regarding Q3 (Target distributions):

We have added Appendix J.1 with separate target-distribution tables for all 43 datasets. Table 5 reports each retained classification label, its count and proportion, and the number of classes; Table 6 reports regression medians, interquartile intervals and ranges on the evaluated target scale. These summaries describe the evaluated cohorts after filtering and were computed from the five disjoint held-out folds, with cohort sizes matching the registry. We also clarify that classification sampling is stratified rather than balanced to equal class sizes.


Regarding Q4 (Bradley–Terry ranking, task coverage and failures):

The Bradley-Terry model is fitted only on pairwise comparisons between models that were both evaluated on the same dataset and fold, so a model that is not applicable to a task, such as the classification-only TabPFN-Wide on regression, simply has no comparisons there and is not penalized. Failed runs are not excluded: memory or time-limit failures are scored with the constant predictor's result on the same fold and enter the comparisons like any other result.

Removing the two classification-only TabPFN-Wide configurations gives all remaining models identical coverage of the same 42 datasets. Recomputing Elo produces no substantive change in the leaderboard; only two previously rounded ties separate.


Regarding Q5 (TabDPT pretraining overlap):

We retained these results for two reasons. First, the three evaluated tasks (OVA Uterus, AP Colon Kidney and AP Ovary Lung) are not directly in TabDPT's documented pretraining list, which instead contains two other tasks from the same GEMLeR/expO collection (OVA_Lung and OVA_Endometrium). Our other exclusions are based on model applicability, such as task type, so this relatedness did not fall under them; we instead flag it in the limitations, since tasks from one collection may share expression profiles.

Second, the effect on the ranking is small: excluding TabDPT's results on the three tasks, while retaining all 42 datasets, leaves the leaderboard order unchanged and TabDPT in seventh place (Elo 1,032 to 1,026).


Regarding Q6 (Original implementations and benchmark adaptations):

We used standard scikit-learn or AutoGluon implementations wherever possible. For methods unavailable through either library, we wrote minimal adapters that call the original model implementations; we did not reimplement any learning algorithm. The adapters handle compatibility and lift wrapper-level feature or sample guards where needed. Models receive the shared benchmark inputs, while model-native preprocessing and applicability restrictions remain part of each method.


Regarding Q7 (Meaning of fit time):

We have clarified in Appendix B that fit time measures the adapter's fit-call wall time, covering training for parameter-updating models and training-context preparation for in-context learning models.


Regarding Q8 (FAIR principles and reproducibility):

We agree that the submitted version did not fully pin its environment: several dependencies, including the AutoGluon fork, were specified without exact revisions. They now do. For FAIR: datasets are findable through their source accessions (OpenML, GEO, TCGA, MGnify, TDC, ChEMBL) and a versioned registry; code and releases are accessible on GitHub and PyPI; results are stored in open formats (JSON, CSV, SQLite) with stable keys; and fold-level predictions, split manifests and evaluation code allow rankings to be recomputed without retraining. The three locally generated embedding datasets are distributed as checksum-verified files in a versioned release.

I would like to thank the authors for their clarifications and improvements. As I stated in my original review, I think such a benchmark can be useful, but I still maintain that, in terms of novelty, it is rather narrow. Nevertheless, with a proper buy-in from the community and a proper curation, this has the potential to be a nice resource.

Reviewer_RLvWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TabBench-Bio, a living benchmark for evaluating machine learning methods on high-dimensional, low sample size biomedical datasets. The benchmark contains 43 datasets spanning different biomedical modalities and evaluates 19 predictive model configurations, alongside a constant baseline and a separate AutoGluon reference. The authors propose a 4 x 7 feature-by-sample grid, shared five-fold cross-validation, and Bradley Terry Elo rankings with bootstrap confidence intervals.

The experimental results suggest that tabular foundation models generally perform well in this setting, with RealTab PFN 2.5 achieving the highest Elo ranking at the reference operating point. However, Logistic Regression remains competitive, and model rankings vary across operating points and biomedical modalities. The authors also provide an interactive leaderboard, implementation code, and reproducible evaluation procedures.

Critical strengths
  1. The paper addresses an important benchmarking gap. Existing general-purpose tabular benchmarks do not specifically focus on biomedical datasets with many features and relatively few labelled samples. The collection of 43 datasets across different biomedical modalities provides a useful basis for understanding model behaviour in this setting.

  2. The experimental framework is reasonably comprehensive and carefully implemented. The authors evaluate a broad range of conventional models and tabular foundation models under shared cross-validation splits. The inclusion of failure handling, bootstrap confidence intervals, metric sensitivity analysis, and reproducible evaluation artefacts strengthens the benchmark's usefulness and transparency.

Critical weaknesses
  1. The dataset selection procedure raises concerns about the representativeness and potential selection bias of the benchmark. In Section 3.1, the authors describe using Random Forest with label-permutation tests and minimum predictive-performance thresholds to screen candidate datasets. Although this helps exclude datasets without detectable predictive signals, it also needs dataset inclusion on the performance of a particular model family that participates in the benchmark. Consequently, the resulting collection may favour tasks that are learnable by Random Forest and exclude difficult datasets where other model families could offer advantages. This is particularly relevant because the benchmark aims to establish comparative model performance across biomedical HDLSS settings. Further justification of this selection strategy and its influence on the benchmark composition is necessary.

  2. The experimental results do not fully isolate the effects of feature dimensionality and training sample size, despite this being a central motivation of the benchmark. For example, Figure 3 shows substantial changes in model performance and relative rankings as the training sample budget increases. However, the set of eligible datasets also changes across budgets, meaning that these differences may partly reflect changes in dataset composition rather than sample size alone. Additionally, feature budgets are implemented through random feature subsampling, and the nominal feature cap does not necessarily reflect the actual dimensionality of each dataset. For instance, several molecular-property datasets contain only 2048 features, even at the 10000-feature reference operating point. Although the authors acknowledge these limitations and provide a shared-target comparison for two feature budgets, a more systematic analysis using matched datasets across sample budgets, together with clearer reporting of effective feature dimensionality, would strengthen the conclusions regarding model robustness across operating points.

Questions for authors

How many candidate datasets were excluded based on the Random Forest permutation and predictive-performance screening? Could the authors clarify how this selection procedure might affect the relative performance of different model families?

In Figure 3, can the authors provide or clarify the model-ranking trends when restricting comparisons to the same set of datasets across all sample budgets? This would help establish whether the reported changes are attributable to sample size rather than changes in dataset composition.

At the reference operating point of 10,000 features and 100 training samples, how many datasets actually contain 10,000 retained features? Reporting effective feature dimensions would help interpret the feature-budget comparisons.

2 comments

Thank you for the careful review. We have clarified dataset selection, added comparisons using the same datasets across training-sample settings, and reported the effective feature dimensions. We address your concerns and questions below:


Regarding Q1 and CW1 (Dataset selection):

In Section 3.1 we have clarified that the automated random-forest screen produced only a preliminary shortlist, and that the final collection also contains manually curated additions selected for relevance and quality.

Out of all datasets available through the different apis, 214 matched initial criteria, from which 10 fell below the random forest minimum score. The thresholds are deliberately low (eg.: an adjusted balanced accuracy of at least 0.10, or R² of at least 0.05), so the screen removes candidates with little detectable signal rather than requiring strong random-forest performance. We agree that the screen may still favour tasks that forests can learn, and the limitations now state this. The manual additions broaden the collection, and the new Appendix I shows that it spans a wide range of difficulty.


Regarding Q2 and CW2 (Effect Isolation):

We agree that the original plot mixes training-sample changes with changes in dataset composition. We’ve added a second panel directly below it in Figure 3, repeating the regular Elo calculation on the 14 datasets that are available at all six fixed training-sample settings. The same six configurations are shown. The linear baseline still declines in relative rank, from fourth at 20 training samples to 16th at 1,000. Individual model curves change slightly, showing that composition influences the aggregate results. We therefore interpret the result as a sensitivity analysis of the complete budgeted learning procedure, rather than an isolated causal effect of sample count.


Regarding Q3 and CW2:

At the reference operating point, 28 of the 42 eligible datasets retain the full 10,000 features in all five folds; the remaining 14 retain between 1,142 and 8,192 features. Riboflavin is not eligible at this point because it has fewer than 100 training rows. We now report this count in the main text and show the retained widths by modality in Appendix F (Figure A5b).

Many thanks for your answers to the questions, along with the additional content in the updated version. This generally addressed my main concerns regarding dataset selection bias and the issue of dataset composition across sample budgets. The review has been updated accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.