- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces a new benchmark for the tabular domain focusing on biomedical datasets. The benchmark is comprised of already published datasets, a strict and very detailed evaluation setup and a first analysis of existing state-of-the-art models. The current benchmark page (anonymised) contains very recent models that have been even released after the paper submission.
Critical strengths
- The paper is well written and very detailed when it comes to the experimental setup. This is very much appreciated. I also like that issues such as memory problems and other reasons for model training failures have been handled and analyzed in depth.
- Since there are plenty of applications dealing with biomedical tabular data - the proposed benchmark is indeed relevant.
- The paper is using benchmarking techniques and metrics used in recent prominent benchmarks like TabArena.
Critical weaknesses
- What are the common properties of the datasets in the benchmark? The targeted applications are pretty diverse (from molecular-property tasks to gene expressions etc.). A benchmark that compiles multiple datasets and aggregates results across these datasets suggests that there is a common prior also relevant for other applications. As a practitioner, If I have a new biomedical tabular dataset, how likely is it that the model tanking holds or should I rather look for the ranking within a certain modality group. Their common attribute of being "biomedical" might be not enough to draw relevant conclusions. I would love to see these questions tackled in more depth, which also relate the questions how close is this benchmark (in data distribution) to TabArena or RamanBench for example.
- Of course, not doing any hyperparameter optimization is unfortunate, but I know that this requires a completely different magnitude of compute. It is a weakness, but not a critical one in my opinion.
- I do not get, why analyzing feature width restrictions is important. In my opinion, this leads to more issues than answers to research questions. Feature representations are cut-off in experiments (making feature order a relevant factor), etc. - these limitations were mentioned by the authors themselves, but I am not sure it was the right decision for the paper. However, varying the training size is reasonable and studying its effects led to some insights whether a certain model is able to deal with a data-scarce application (as illustrated e.g. in Fig. 3).
Questions for authors
Minor comments and questions:
- Indeed not a critical question, but maybe an interesting discussion aspect in our community: given the fact that we see so many "living" benchmarks appearing in our field, how can we guarantee their maintainability in the future (you do not have to answer this in the rebuttal, since this question is beyond this single paper).
- L200: "budgets" might be not the right wording here.
- It would be reasonable to list the differences to the experimental setup used in TabArena (beyond L83ff).
- I haven't seen a plot showing task-type specific performance, e.g. what is the best model for gene expression for DNA methylation?
2 comments
Thanks for the rebuttal
Thank you very much for the detailed rebuttal, this helped clarify some of my questions. As mentioned in the original review, I do not see CW2 as really critical, Q1 was just a thought mostly related to often unseen work that is required to keep such a benchmark alive. I also appreciate the explicit list of differences that allows comparison to TabArena.
Thank you for the careful review. We have clarified the shared evaluation setting and feature-width rationale, added modality-specific results, and explained the differences from TabArena. We address your concerns and questions below:
Regarding CW1 (Common properties and transfer to new datasets):
The datasets share an evaluation setting with many numerical features and limited labelled training data; we do not assume a common biological prior, additionally, their representations and endpoints differ.. TabArena covers broader IID tabular tasks, which are not HDLSS, while RamanBench uses physically ordered spectra whereas TabBench-Bio has 32 of the 43 tables with more features than rows.
For a new dataset, the modality-specific rankings on the interactive leaderboard can help form a shortlist, considering representation, endpoint, sample size and computing resources. Final selection still requires validation with the intended biological grouping and deployment setting. We have not tested transfer to external sites.
Regarding CW2 (Hyperparameter optimization):
We agree that the absence of hyperparameter optimization is a limitation. Since submission, we have added tuned variants of Logistic Regression/Ridge, the MLP, Random Forest, Extra Trees and XGBoost to the live leaderboard, each selected from a small predefined grid on a training-only holdout. These variants are not part of the comparison reported in the paper.
Regarding CW3 (Feature-width rationale and effective dimensions):
At the reference operating point, 28 of the 42 eligible datasets retain the full 10,000 features in all five folds; the remaining 14 retain between 1,142 and 8,192 features. Riboflavin is not eligible at this point because it has fewer than 100 training rows. We now report this count in the main text and show the retained widths by modality in Appendix F (Figure A5b).
We have also added the rationale for varying feature width to the operating-point description: For gene-expression, methylation and genomic-prediction data, smaller subsets approximate assays that measure fewer genes, CpG sites or variants; for metagenomic profiles, fingerprints and embeddings, they test sensitivity to retaining only part of the representation. Because increasing the width changes both dimensionality and the available information, the sweep does not isolate dimensionality alone, and random subsets do not reproduce a particular assay or biologically selected panel.
Regarding Q1 (Maintenance of the living benchmark):
TabBench-Bio is maintained by the authors and open to community contributions. Datasets and models are proposed through GitHub issues or PRs. A dataset must be publicly accessible under an open licence, have a defined classification or regression target, and record its source, licence and citation; it then is evaluated on checks described in Section 3.1.
A model only needs an adapter and declared metadata. The maintainers review each contribution and evaluate it under the unchanged operating-point configurations before it enters a versioned release on GitHub and PyPI. TabBench-Bio thus combines community contributions with curation by dedicated maintainers. We are committed to maintaining TabBench-Bio and evaluating new models and datasets.
Q2
True, we’ve replaced "budgets" with "settings".
Regarding Q3 (Experimental-protocol differences from TabArena):
Compared with TabArena, our protocol differs in several respects:
We also explicitly score attempted memory/time-limit failures at constant-predictor performance. These differences mean that contrasts between the benchmarks cannot be attributed to domain alone.
Regarding Q4 (Modality-specific performance):
We have added a modality-by-model heatmap in Appendix E (Figure A4) at the strict reference cell (10,000-feature cap, 100 training samples). It reports ranks from the same Bradley-Terry Elo procedure within each modality, with dataset counts and classification/regression composition shown. RealTabPFN 2.5 leads gene expression, metagenomics and molecular properties; TabFM leads embeddings, TabPFN-Wide leads methylation, Random Forest leads genomic prediction. Those results can also be inspected on the website.