acceptodds
Under review as a conference paper at ICLR 2027

Beyond Foundation Models: PFN Priors Improve Tabular Hyperparameter Tuning

16 views

Abstract

Synthetic priors have so far served one purpose: pretraining tabular foundation models. We ask whether the same priors can improve conventional tabular models, which are trained from scratch on each dataset. We propose PriorTune, which meta-tunes hyperparameter configurations on synthetic tasks alone. Offline, for a given model, we fit a pool of configurations on datasets sampled from a published PFN prior and greedily select the subset with the highest winrate over the random baseline. Online, a new dataset fits only the selected configurations, so the gain comes at no extra cost. Empirically, we observe that scaling both the number of configurations and synthetic datasets has an equal scaling effect on the performance of the final subset of configurations, which we also explain theoretically. On BeyondArena, the models tuned with our hyperparameter configurations approach the performance of the recent TabPFN-3, while being much more efficient at inference. Overall, our results establish existing PFN priors as a scalable source of meta-training data for hyperparameter selection for non-foundational tabular models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
tabular, hyperparameter optimization, meta-learning, synthetic data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rQKWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose to use synthetic datasets from priors used to train tabular foundation models to select default well-performing hyperparameter configurations for four candidate models. On well established benchmarks, the authors argue that the default hyperparameter configurations found outperform the ones obtained from random search.

Critical strengths
  • The experiments of the work are run on well established public tabular data benchmarks.
Critical weaknesses
  • The introduction section is difficult to understand. For example, reading Lines 76-84 and Lines 85-90, it is difficult to comprehend what the contributions of the work are. Same for the bulleted list of contributions.

  • There is an undefined reference to an algorithm in Line 212.

  • "Under both deployment rules, the model (or ensemble) is selected on the validation split and scored on the test split. Our objective mimics this, which selects configurations that are stable across validation and test." How should the following paragraph be understood, do the authors select hyperparameter configurations that are well-performing on the validation split and just score them on the test split. Or do they select based on both splits. Because selecting on the validation set, is the proper way in my perspective, otherwise you are leaking the test split. If it is the first case, how does that guarantee that the configuration is stable?

  • The work in it's entirety is difficult to understand. In my perspective the work needs a serious rewriting. First to highlight what the contributions of the work are in the introduction, secondly, to show how the authors conducted the experiments in detail. It is difficult to comprehend both points from the main paper.

  • The authors consider four models in their results, how would the authors apply what they offered to other models, or different hyperparameter search spaces? Would practitioners need to rebuild what the authors offered, how would that time spent compare to fine-tuning the models via say a hyperparameter optimization method?

  • How does the aforementioned approach fare, when for a method XX on a task AA, while instead of fitting on the best hyperparameter configurations found from the synthetic datasets, we run a gray-box HPO method [1][2][3][4]? In the time it takes to fit one full configuration, is it possible to evaluate more configurations by pruning on lower budgets to find something better for a few full hyperparameter configurations? If the total budget is 50 epochs (1 full training), one could in theory run 10 configurations for 3 epochs, and one for 20 and potentially achieve something better?

[1] Rakotoarison, H., Adriaensen, S., Mallik, N., Garibov, S., Bergman, E. & Hutter, F.. (2024). In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization. Proceedings of the 41st International Conference on Machine Learning

[2] Lee, D. B., Zhang, A., Kim, B., Park, J., Adriaensen, S., Lee, J., ... & Lee, H. B. (2026). Cost-sensitive freeze-thaw bayesian optimization for efficient hyperparameter tuning. Advances in Neural Information Processing Systems, 38, 169531-169564.

[3] Kadra, A., Janowski, M., Wistuba, M., & Grabocka, J. (2023). Scaling laws for hyperparameter optimization. Advances in Neural Information Processing Systems, 36, 47527-47553.

[4] Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2018). Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of machine learning research, 18(185), 1-52.

Questions for authors
  • The work in my opinion should be rewritten and the contributions/experimental protocol be made clear. I have raised a few questions for the authors in the critical weaknesses section, to help improve the paper for future submissions.
4 comments
AuthorsOpenReview ↗

Response [pt1]

We thank the reviewer for their input. However, we respectfully disagree that our paper is hard to understand. Reviewers ghNY and V1FT rated the presentation 3 ("Clear and easy to follow") and 2 ("Generally understandable") respectively, and raised no objections to the writing. We will do our best to address your comments and to improve the writing in the revised manuscript. For convenience, we first list the key terms and summarize the paper, and then answer each point.

Key terms

  • Configuration. One setting of a model's hyperparameters (learning rate, tree depth and so on).
  • Prior. A generator of synthetic datasets. Tabular foundation models such as TabPFN and TabICL are trained on datasets drawn from a prior.
  • TabRepo. A collection of 200 real datasets. It is the standard source of real data for selecting configurations that transfer to new datasets.
  • TabArena. The standard benchmark for tabular models: 51 real datasets with random train-test splits. Every model is tuned in the same way, by training 200 random configurations per dataset.
  • BeyondArena. A newer and larger benchmark: 142 real datasets, including time-ordered and group-wise splits and datasets with over a million rows. Every model is tuned with 25 random configurations per dataset.
  • Tuned (T). Train the configurations and keep the one with the best validation score. Its test score is reported.
  • Tuned and ensembled (T+E). Train the same configurations and combine them into an ensemble, with weights chosen on the validation set (greedy Caruana ensemble). The test score of the ensemble is reported. This usually gives the best results, and both benchmarks treat it as the main measure of a model's strength.
  • Elo. The score both benchmarks use to rank methods, computed from wins and losses against all other methods on every dataset.

The paper in plain terms

  • Idea 1: reusing hyperparameters across datasets. TabRepo showed that one can pick a small set of hyperparameter configurations that works well on about 200 real datasets and reuse it on new datasets. The idea has a longer history in the hyperparameter optimization literature, see also [1, 2, 3, 4]. A related line transfers knowledge into the optimizer itself, for example [5, 6, 7]. All of these are discussed in our related work.
  • Idea 2: synthetic data for tabular models. Tabular foundation models (TabPFN, TabICL) are trained only on synthetic datasets drawn from a generator called a prior, and they are among the strongest tabular models today.
  • Our question. Can we combine the two ideas and select hyperparameter configurations on synthetic datasets instead of real ones?
  • Our answer. Yes. With few datasets, real data is better. But real datasets are limited, and the result stops improving. Synthetic datasets are unlimited, keep improving, and overtake real data (Figure 3). The selected configurations also do better than random search over up to 4,000 configurations (Figure 4), and the gain grows steadily with the amount of synthetic data and the number of candidate configurations.

Q1. The introduction and contributions are difficult to understand

In one sentence, we meta-select good hyperparameter configurations on synthetic data. For reference, TabRepo has explored a similar idea on real data. We compare to TabRepo on TabArena (Figure 3, p. 7) and on BeyondArena (Table 13, p. 31). We will revise the introduction to make this point clearer.

Q2. Undefined algorithm reference in Line 212

Thank you for catching this. The reference points to Algorithm 1 (lines 230–247). We have corrected the reference to Algorithm 1 (p. 5).

Q3. Is selection done on validation or on test? Is the test split leaked?

We do not tune on datasets that we benchmark on. Our algorithm exclusively uses synthetic data to meta-select good hyperparameter configurations. Each synthetic dataset has its own train, validation and test parts, and we use them exactly as a practitioner would:

S = empty set, R = a random set of K configurations
while |S| < K:
    for each candidate configuration c:
        for each synthetic dataset:
            pick the best configuration in S + {c} on the VALIDATION part
            score that pick on the TEST part
            record a win if it beats the pick from R, scored the same way
    add to S the candidate with the most wins

So within a synthetic dataset, the choice among configurations uses validation only. The test part is used to judge whether that choice was a good one.

Why this favors stable configurations. A configuration that looks good on validation by chance gets picked and then scores poorly on test. It loses, so the procedure does not select it.

AuthorsOpenReview ↗

Response [pt2]

Q4. The work as a whole is difficult to understand

The summary above and the general comment restate the contributions and the experimental protocol. If any specific term or step remains unclear, we would be glad to explain it.

Q5. How does the method apply to other models or search spaces?

The offline step is done once per model and search space, and its result is reused by everyone. We expect the authors of new models to run it when they release the model, as the authors of a foundation model pretrain it once. A practitioner then only trains the released configurations, at the same cost as a random search of the same size.

For a new model, the offline step costs 300 to 400 GPU-hours in our experiments, which is less than pretraining one tabular foundation model. We release the code and the cached results, so that others can apply the method to their own models or extend ours.

Q6. How does the method compare with gray-box HPO?

This is a fair question. We have not run freeze-thaw or Hyperband, and we do not claim that they are ineffective. We see two reasons why a fixed list of configurations is attractive for tabular models, and we add one new experiment.

  • Ensembling. On tabular benchmarks, the best results come from ensembling the trained configurations, not from keeping the single best one. Methods that prune early leave few fully trained models, and adaptive search concentrates on similar configurations, so both give weaker ensembles. Diversity-aware methods such as DivBO address this, but they have not been adopted by the standard benchmarks. PriorTune can select its configurations for how well they ensemble.
  • Parallelism. A fixed list can be trained all at once, whereas adaptive methods wait for the results of earlier trials.
  • Performance. We compared our 25 configurations with TPE for LightGBM-TabPrep on the 51 TabArena datasets. TPE did not catch up at any budget up to 200 trials (win rate against random search at 200 trials: 63.7% for ours, 58.7% for TPE). Our list leads clearly at small budgets. At 200 trials it is still ahead, but the difference is not significant (+5.0 points, 95% CI [-5.2, 15.0]). Figure 14 on p. 27 of the updated PDF shows both curves.

The two approaches can also be combined: our list can serve as the starting set of candidates for a gray-box method. We consider a direct comparison with such methods a valuable next step and will say so in the paper.

References

[1] Feurer, Springenberg, Hutter. Initializing Bayesian Hyperparameter Optimization via Meta-Learning. AAAI, 2015.

[2] Wistuba, Schilling, Schmidt-Thieme. Sequential Model-Free Hyperparameter Tuning. ICDM, 2015.

[3] Winkelmolen, Ivkin, Bozkurt, Karnin. Practical and Sample Efficient Zero-Shot HPO. arXiv:2007.13382, 2020.

[4] Pfisterer, van Rijn, Probst, Müller, Bischl. Learning Multiple Defaults for Machine Learning Algorithms. GECCO, 2021.

[5] Perrone, Jenatton, Seeger, Archambeau. Scalable Hyperparameter Transfer Learning. NeurIPS, 2018.

[6] Wang, Dahl, Swersky, Lee, Nado, Gilmer, Snoek, Ghahramani. Pre-trained Gaussian Processes for Bayesian Optimization. JMLR, 2024.

[7] Müller, Feurer, Hollmann, Hutter. PFNs4BO: In-Context Learning for Bayesian Optimization. ICML, 2023.

Dear Reviewer rQKW,

Thank you again for your review. We would like to kindly ask whether our response has addressed your concerns. In short:

  • Clarity. Our response gives a plain summary of the paper, definitions of the key terms, and the contributions in three sentences. We will carry these into the introduction.
  • Validation and test splits. Selection uses the validation split only, and the test split is used only for scoring. Our response includes pseudo-code of the protocol.
  • Other models and search spaces. The procedure is the same for any model, and it is run once, offline. The selected configurations are then reused on every new dataset at no extra cost.
  • Comparison with HPO methods. We added a comparison with TPE (Figure 14, p. 27 of the revised paper) and a discussion of Hyperband and freeze-thaw methods in the General Comment.

If the response resolves your concerns, we would be grateful if you could consider updating your score. If any part of the paper is still unclear, please point us to it, and we will explain it and revise the text. Thank you for your time!

I would like to thank the authors for the reply. Below are my responses:

  • Work difficult to understand

    I thank the authors for the extra simplified explanations, however, I would like to kindly point out that I have a confidence level of 4 in my review. It is not the case that I am unfamiliar with the related work, with the current work, with the notation of hyperparameter optimization etc. The other reviewers have their own opinion of the work, maybe they might have also used LLMs. I did not do that and I believe I pointed out exactly what I think was not clear to understand the work in detail, not in general. As such, the AC during the final evaluation is additionally welcome to revise the initial paper, the lines I pointed to and the rest of the paper and decide whether it is clear for the AC or if the AC shares my opinion.

  • Q5

    The question I made was deliberate. Because that is a shortcoming of the work, the aim of this work, is to have a good set of defaults that perform well on an unseen dataset, as such, if there is a new method, a new search space, these results cannot be reused. The same experiment would have to be repeated for that method and the practitioner that does it would have to release the results for others to use, while in the meantime, the practitioner can just run hyperparameter optimization. Moreover, there are multiple cases where hyperparameter optimization has task dependent hyperparameters, for example, maybe the type of categorical encoding.

  • Q6

    An additional issue I observe, is that these priors have been created so that they allow tabular foundation models to generalize to unseen problems. However, the priors have obviously been skewed so that they perform well on TabArena, given that TabArena and now Beyond, are the benchmarks that are used to advertise the performance of these models. As such, using data from these priors even though synthetic, is optimizing for performance on the aforementioned benchmarks. Lastly, 25 configurations are a considerable amount, given that they would have to be evaluated. What would be the performance of the aforementioned configurations to HyperBand? to IFBO? to DPL?

Combining the work being difficult to understand, for a work which in my opinion is not complex, with the additional aforementioned concerns, I am unable to recommend acceptance for the submission at the current iteration.

Reviewer_ghNYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether synthetic datasets drawn from the priors used to pretrain tabular foundation models can improve hyperparameter selection for conventional tabular models. The proposed method, PriorTune, fits a pool of 4,000 hyperparameter configurations on 4,000 datasets sampled from the TabICLv2 prior. It then greedily selects a subset by its win rate against random subsets of the same size, scored under the deployment rule: the validation-best model or a greedy ensemble. On a new dataset, only the selected configurations are fitted. The study covers LightGBM, LightGBM with TabPrep, an MLP, and TabPack. On TabArena, synthetic tasks overtake real-data meta-training on TabRepo once the compute budget is large enough, and the selected subsets beat random subsets of every size up to the full pool. On BeyondArena, the selected subsets reduce median relative error over random subsets by 0.1–0.4 points, and LightGBM-TabPrep with its selected subset matches RealMLP. The paper also studies how results change with the number of synthetic tasks, the pool size, and the choice of prior. It gives theoretical arguments for the observed scaling, and releases the selected subsets and cached results.

Critical strengths
  1. A controlled test of a new source of meta-training data.

The paper tests whether synthetic tasks sampled from PFN priors can replace real datasets when selecting hyperparameter configurations. The comparison with TabRepo holds compute fixed (B = C·N), removes the 18 TabRepo datasets that overlap with or share a source with TabArena, and uses the same selection objective for both sources. Real data is better at small budgets, but synthetic data overtakes it as the budget grows. At the largest budget, synthetic selection ends 0.02–0.08 above TabRepo’s best for every model and subset size, while TabRepo gains at most 0.008 over its last two doublings. On TabArena, a meta-tuned subset of 200 configurations also beats random subsets of every size, up to fitting all 4,000, and beats the TabRepo-selected subset. This shows that PFN priors can be a scalable source of meta-training data for conventional models, a use beyond pretraining PFNs.

  1. A deployment-aware objective and a useful released resource.

PriorTune scores each candidate by its win rate against random subsets under the rule that will actually be used: the validation-best model or a greedy ensemble, chosen on validation and scored on test splits of the synthetic tasks. Because it uses win rates, the objective works across tasks with different metrics and scales, and it penalizes configurations whose validation and test results disagree. The gain over random search adds no training or inference cost on a new dataset, since only the K selected configurations are fitted. The authors also release the selected subsets and score caches covering 16 million fits per model family, which others can use to reproduce the results, test other selection methods, or adopt the configurations directly.

Critical weaknesses
  1. The gains on real data are small, not tested for significance, and overstated in the abstract.

On BeyondArena, the selected subsets improve median relative error over random subsets by only about 0.1–0.4 percentage points. For example, the MLP under the tuned rule improves by 0.16 points, while LightGBM-TabPrep under the tuned-and-ensembled rule reaches 1.75, compared with 1.57 for its random subset. The paired win rates are 0.55 to 0.68, but the paper reports no confidence intervals or statistical tests for these comparisons. With 142 datasets, the standard error of a win rate near 0.5 is approximately 0.04, so a win rate of 0.55 is not clearly distinguishable from chance. Several dataset-size groups also have win rates below 0.5, including LightGBM-TabPrep FT on large datasets under T+E (0.36) and TabPack on medium datasets (0.45).

Several claims also go beyond the reported results:

  • The abstract claims that the tuned models "approach the performance" of TabPFN-3, but TabPFN-3 achieves a median improvement of 3.62, compared with 1.75 for LightGBM-TabPrep, and wins on 59% of datasets.
  • The reported large-dataset advantage over TabPFN-3 covers only 39 datasets. The paper should say whether any TabPFN-3 results in this comparison are imputed, and how the comparison changes without them.
  • The claim of a "new state of the art among non-foundation models" rests on a 52% win rate against RealMLP, which is better described as a near tie.
  • The abstract claims an "equal scaling effect" for the number of configurations and synthetic tasks, but Section 5.4 reports that doubling the configuration pool is worth approximately 1.2–1.7 doublings of the number of tasks. The curve for the number of tasks also begins to flatten above approximately 1,000 tasks.

These results suggest that PriorTune is promising, but the size and reliability of its improvements on real datasets remain uncertain.

  1. The main benchmark compares only against random search.

On BeyondArena, each selected subset is compared with a random subset from the same search space. This shows that meta-tuning can outperform random search, but it does not establish that synthetic meta-training is better than real-data meta-training on new datasets, which is the paper's main practical claim. The TabRepo-selected subset appears only in the TabArena ablations and is not evaluated on BeyondArena. Comparisons with Bayesian optimization at a matched evaluation budget and existing portfolios, such as AutoGluon's, are also missing. In addition, the uncertainty reported for the ablations comes from selection repetitions on one fixed 51-dataset benchmark, rather than variation across datasets. This limits the conclusions that can be drawn about generalization to new datasets.

Questions for authors
  1. Are the BeyondArena gains distinguishable from noise? Could you report paired confidence intervals or statistical tests over datasets (e.g., a bootstrap or sign test) for the win rates in Table 8 and the differences in Table 3? Also, are any TabPFN-3 results in the large-dataset comparison partially imputed? If so, how does the comparison change when those results are excluded?

  2. How does a real-data-selected portfolio perform on BeyondArena? Could you evaluate a TabRepo-selected subset on BeyondArena using the same K, search space, selection objective, and deployment rule as PriorTune, while excluding datasets that overlap with TabRepo? This would directly test whether synthetic meta-training transfers better than real-data meta-training to unseen datasets.

  3. How independent is the main evaluation from the design choices? Were any design choices, such as the prior settings, number of repetitions R, selection objective, or subset size K, decided on the basis of TabArena results? How many BeyondArena datasets overlap with TabArena or TabRepo, either directly or through shared data sources?

5 comments
AuthorsOpenReview ↗

Response [pt1]

We thank the reviewer for the valuable comments and suggestions. Addressing them has made the paper better.

Weaknesses

W1. Gains on real data are small and not tested for significance

For this response, we finished the runs on the full BeyondArena (all outer train-test splits). Below we report the win rate and the Elo gain of PriorTune configurations over random ones, with confidence intervals.

vs. randomWin rate [95% CI]Elo gain [95% CI]p
LightGBM T56% [48, 64]+27 [8, 46]0.18
LightGBM T+E59% [51, 67]+25 [11, 39]0.04
LightGBM-TabPrep T65% [57, 73]+45 [25, 64]0.0004
LightGBM-TabPrep T+E71% [63, 78]+44 [31, 59]5e-7
MLP T48% [39, 56]+5 [-17, 28]0.68
MLP T+E56% [48, 65]+21 [5, 38]0.15
TabPack T+E62% [54, 70]+23 [7, 41]0.005

How to read the table:

  • Elo gain. The Elo of the model with PriorTune configurations minus the Elo with random configurations, both computed among 58 methods.
  • Intervals. 95% bootstrap intervals over datasets.
  • P-values. Two-sided sign test over datasets on the win rate.

PriorTune is ahead of random configurations by Elo in all seven cases, and the interval excludes zero in six of them. The effect is strongest for LightGBM-TabPrep and weakest for the MLP under T.

W1a. "Approach the performance of TabPFN-3" is overstated

After expanding our results from the reduced BeyondArena to the full benchmark, we indeed surpass TabPFN-3 by Elo with LightGBM-TabPrep T+E with PriorTune (FTE) configurations. The new results are in the General Comment and in Table 10 on p. 26 of the revised paper. We also include them here for your convenience:

RankMethodTypeElo [95% CI]Time per row
7LightGBM-TabPrep, PriorTune (ours)parametric1313 [1287, 1344]4.4 ms (CPU)
9RealMLPparametric1297 [1272, 1326]11 ms (GPU)
10TabPFN-3foundation1279 [1238, 1323]243–3,308 ms (GPU)
11LightGBM-TabPrep, randomparametric1268 [1247, 1296]2.5 ms (CPU)

Important points on how to read this result:

  • Our model is ahead on average but statistically tied with TabPFN-3 and RealMLP, while being faster at inference (2.6 times against RealMLP, over 50 times against TabPFN-3) and running on a CPU. The Elo differences are +33 (95% CI [-14, 81]) against TabPFN-3 and +16 (95% CI [-18, 49]) against RealMLP. We will word the claim accordingly.
  • What PriorTune adds. Without PriorTune, LightGBM-TabPrep ranks 11th, below both RealMLP and TabPFN-3. PriorTune alone moves it above both, and this gain is significant (+44 Elo, 95% CI [31, 59]).

W1b. Are TabPFN-3 results on large datasets imputed?

We use the official TabPFN-3 results published on BeyondArena, and we checked them manually. None of the TabPFN-3 results are imputed, including for large datasets.

W1c. "New state of the art" rests on a near tie with RealMLP

Yes, we will adjust the wording. The advantage is that our model is faster at inference and needs no GPU, while staying slightly ahead, albeit not significantly.

W1d. "Equal scaling effect" contradicts Section 5.4

Both more synthetic tasks and more candidate configurations help on the tested grid, with diminishing returns, but their empirical rates and marginal values differ, which is also supported by our theory. The revised wording will be: “Scaling both the number of synthetic tasks and candidate configurations improves the selected portfolio, with different empirical rates.”

AuthorsOpenReview ↗

Response [pt2]

W2. No real-data-selected portfolio on BeyondArena

We have added an experiment with TabRepo-selected configuration sets on BeyondArena (Table 13 on p. 31 of the revised paper). For each model, we selected configurations on the TabRepo datasets using matched search spaces and portfolio sizes (25 configurations, 64 for TabPack), and evaluated them under the same T+E protocol.

ModelScored datasetsElo: Random / TabRepo / SyntheticWin rate, synthetic vs. TabRepo [95% CI]pElo, synthetic minus TabRepo [95% CI]
LightGBM-TabPrep141 / 1421283 / 1286 / 130657% [49, 65]0.127+21 [-4, 46]
LightGBM141 / 1421209 / 1237 / 123048% [40, 56]0.673-8 [-30, 14]
MLP139 / 1421201 / 1220 / 123060% [53, 68]0.017+10 [-11, 33]
TabPack142 / 1421215 / 1223 / 124056% [47, 63]0.208+17 [-5, 40]

By Elo, we have Random < TabRepo < Synthetic for three of the four models. With this sample size, most of the differences are not significant.

Caveats:

  • Coverage. The runs use the first train-test split of each BeyondArena dataset. LightGBM and LightGBM-TabPrep exclude one dataset on which no configuration finished, and three MLP datasets are still running. These are unlikely to change the pattern. We will report the full BeyondArena evaluation in the camera-ready version.
  • Statistics. Elo is computed per row among the paper's 58 methods plus that model's TabRepo-selected set, so Elo values are not comparable across rows. Win rates, intervals and p-values are computed as in W1.
  • Overlap. Before selecting on TabRepo, we removed the TabRepo datasets that share a source with BeyondArena, so the comparison does not favor TabRepo through leakage.

W2a. No Bayesian optimization or AutoGluon comparison

We added a comparison with TPE in the General Comment. AutoGluon builds a multi-model portfolio. Its latest version includes only one configuration for LightGBM and LightGBM-TabPrep and none for the MLP or TabPack, so a direct comparison is not possible. Our selected sets of configurations can, however, be added to AutoGluon.

W2b. Ablation uncertainty comes from repetitions, not datasets

Indeed, our error bars in the TabArena ablations only show variation across repeated configuration set selection. Uncertainty over datasets is covered in two places:

  • BeyondArena. All intervals and tests in W1 are computed over datasets.
  • TabArena. The new comparison with TPE (General Comment) reports a bootstrap interval over the 51 datasets.

Questions

Q3a. Were design choices decided on the basis of TabArena results?

None of the design choices was selected to optimize performance on TabArena or BeyondArena. All decisions were made on synthetic datasets or fixed in advance, for the reasons below.

  • Prior settings. We chose TabICLv2 in advance, as the newest public prior, and used the official settings from its GitHub repository. Our prior ablation later showed that the older TabPFNv1 prior is competitive and sometimes ahead on TabArena, but we kept TabICLv2 for all main experiments.
  • Selection objective. Our algorithm greedily selects configurations, mimicking the exact model selection (single best or ensembling) used in testing on real data. We ablated our algorithm vs. TabRepo’s algorithm using synthetic datasets.
  • Subset size K. We used 25, 64, and 200, because BeyondArena uses 25, TabPack uses 64, and TabArena uses 200.

Q3b. How many BeyondArena datasets overlap with TabArena or TabRepo?

We traced every dataset back to its original source. 49 of the 142 BeyondArena datasets share a source with a TabArena dataset, and 25 share a source with a TabRepo dataset. A shared source means the data come from the same origin. The rows, preprocessing and splits can still differ.

How we handled the overlap when selecting on TabRepo:

  • For TabArena: we removed 18 of the 200 TabRepo datasets, leaving 182.
  • For BeyondArena: we removed a further 29, leaving 153. Of these, 12 share a source with a BeyondArena dataset, and 17 have a likely overlapping source, so we removed them to be safe.

We matched datasets by source, not row by row, so we cannot rule out that a few identical rows remain. Selection on synthetic data does not have this problem, because it uses no real datasets.

Reviewer_ghNYOpenReview ↗

Reply to authors’ response

Thank you for the detailed response and the revised paper. The new results mostly address my main concerns. On full BeyondArena, the Elo gain over random configurations now comes with dataset-level intervals. I also appreciate the check that no TabPFN-3 results are imputed, and the revised abstract and conclusion now state the comparisons with RealMLP and TabPFN-3 as statistically unresolved, which matches the evidence. I will keep my rating.

Reviewer_V1FTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PriorTune, which is a method of selecting hyperparameter configurations for classical machine learning models by evaluating them against synthetically sampled datasets from a PFN prior, specifically TabICLv2. Offline, this expensive evaluation of configurations is done once, while online only the K selected configurations are fitted on a new dataset. The authors show that this selection mechanism is effective at finding good hyperparameter configurations on the BeyondArena benchmark, compare their synthetic meta-training data with TabRepo’s real datasets and investigate scaling of their approach.

Critical strengths
  • Using PFN priors as an unlimited, contamination-free source of meta-training data is both a simple and practical idea that can help classical methods benefit from the advancements of tabular foundation model development. Furthermore, it provides a way to scale offline tuning classical methods while incurring no additional deployment cost.
  • The authors perform a comprehensive comparison on the BeyondArena benchmark and investigate relevant questions such as scaling laws of their approach and comparing it to TabRepo’s real-world meta-tuning datasets.
Critical weaknesses
  • Full TabArena and absolute performance results are missing. Given this is one of the two main benchmarks used in this paper absolute performances should be reported here and for the baselines. The reporting of improvement against a random subset configuration is interesting but makes it challenging to judge the actual performance of the model.
  • Limitations should be discussed more openly. For example, for small to medium sized datasets Table 3 shows that foundation models like TabPFN-3 are clearly better and PriorTune mostly catches up on large datasets. Additionally, compared to RealMLP the improvements are very minor and no confidence intervals or significance tests are provided that show a significant advantage. A clear discussion on where the model performs well and where it underperforms would help clarify this.
Questions for authors
  • Could you please report a full TabArena leaderboard for PriorTune and the baselines, as you do for BeyondArena? Also, if possible, could you add a further split here in small, medium and large datasets?
  • Could you please add TabRepo- and AutoGluon-selected portfolios to the TabArena and BeyondArena comparisons?
  • Would it be possible to report a similar comparison as the BeyondArena paper did in its Figure 1 and report the performance of the model across the different sub-benchmarks? It would be very interesting here to see how PriorTune performs in each category.
  • I found the comparison between the TabPFNv1 and the TabICLv2 prior very interesting. Do you have ideas why TabPFNv1 prior is competitive even though the performance difference between the models is large?
6 comments
AuthorsOpenReview ↗

Response [pt1]

We thank the reviewer for the thoughtful comments and valuable insights. They have helped us improve the paper.

Weaknesses

W1. Full TabArena and absolute performance results are missing

We ran LightGBM-TabPrep with PriorTune (FTE) configurations on the full TabArena: all 51 datasets and all outer train-test splits, with 200 configurations (TabArena's budget). The remaining models and configuration sets (LightGBM, MLP, TabPack with PriorTune) are running, and we will add them to this table as they finish. Here is what we have now, next to the published TabArena results of the baselines. The full leaderboard of 86 methods is in Tables 11–12 on pp. 29–30 of the revised paper.

ModelElo, tuned (T)Elo, tuned + ensembled (T+E)
TabFM (default)1760
AutoGluon 1.5 (extreme, 4h)1663
TabPFN-3 (default)1644
TabICLv2 (default)1576
LightGBM-TabPrep, PriorTune (ours)14741548
LightGBM-TabPrep, random14631498
RealMLP14081485
TabM13761429
LightGBM13661409
CatBoost13851398
XGBoost13331356

Under T+E, PriorTune adds +50.5 Elo over random configurations (95% CI [31.3, 67.5]) and wins on 62.5% of train-test splits (p = 0.0015). It is the highest-ranked single parametric model in the table. Under T, the gain is smaller and not significant (+11.3 Elo, 95% CI [-7.4, 29.1]).

The table below compares our model under T+E directly with its random-configuration version, with RealMLP (the strongest parametric baseline), and with TabPFN-3.

Our model againstWin rate, 95% CIpElo gain, 95% CI
LightGBM-TabPrep, random62.5% [55.2, 69.6]0.0015+50.5 [31.3, 67.5]
RealMLP58.4% [48.1, 68.1]0.119+63.2 [15.9, 122.6]
TabPFN-334.1% [24.8, 44.1]0.0104-95.4 [-169.7, -22.4]

We also report the improvement by dataset size for LightGBM-TabPrep on full TabArena (T+E, against random configurations):

Size (total rows)DatasetsWin rate, 95% CIElo gain, 95% CIp
Small (<2500)1766.3% [58.8, 73.5]+53.3 [30.4, 77.3]0.0026
Medium (2500–49999)2762.3% [50.4, 73.3]+55.5 [26.4, 86.7]0.050
Large (>=50000)754.0% [34.9, 74.6]+36.5 [-19.9, 73.9]0.91

Win rates are averaged over datasets. Intervals are 95% bootstrap intervals over datasets, and p-values are from two-sided Wilcoxon tests over datasets.

W2a. Limitations against foundation models should be discussed more openly

Foundation and parametric models are hard to compare in general. Foundation models are strongest on small datasets, slower at inference (due to quadratic attention), and require GPU. Parametric models are usually better at scale, faster at inference (e.g. CatBoost is used in high frequency trading), and require no GPU. The breakdown in Q3 shows where each is ahead, and we will state this openly in the limitations section.

W2b. Improvements over RealMLP are minor and untested

We will adjust the wording. The advantage is that our model is faster at inference and needs no GPU, while being slightly better (albeit statistically tied).

AuthorsOpenReview ↗

Response [pt2]

Questions

Q2. TabRepo- and AutoGluon-selected portfolios on TabArena and BeyondArena

We ran this comparison on BeyondArena (Table 13 on p. 31 of the revised paper). For each model, we selected configurations on the TabRepo datasets using matched search spaces and portfolio sizes (25 configurations, 64 for TabPack), and evaluated them under the same T+E protocol.

ModelScored datasetsElo: Random / TabRepo / SyntheticWin rate, synthetic vs. TabRepo [95% CI]pElo, synthetic minus TabRepo [95% CI]
LightGBM-TabPrep141 / 1421283 / 1286 / 130657% [49, 65]0.127+21 [-4, 46]
LightGBM141 / 1421209 / 1237 / 123048% [40, 56]0.673-8 [-30, 14]
MLP139 / 1421201 / 1220 / 123060% [53, 68]0.017+10 [-11, 33]
TabPack142 / 1421215 / 1223 / 124056% [47, 63]0.208+17 [-5, 40]

By Elo, we have Random < TabRepo < Synthetic for three of the four models.

Caveats:

  • Coverage. The runs use the first train-test split of each BeyondArena dataset. LightGBM and LightGBM-TabPrep exclude one dataset on which no configuration finished, and three MLP datasets are still running. These are unlikely to change the pattern. We will report the full BeyondArena evaluation in the camera-ready version.
  • Statistics. Elo is computed per row among the paper's 58 methods plus that model's TabRepo-selected set. Intervals are 95% bootstrap intervals over datasets, and p-values are from two-sided sign tests over datasets.
  • Overlap. Before selecting on TabRepo, we removed the TabRepo datasets that share a source with BeyondArena, so the comparison does not favor TabRepo through leakage.
  • AutoGluon has only 1 configuration for LightGBM and none for the MLP or TabPack.

Q3. Performance across the BeyondArena sub-benchmarks

We have computed this figure and include it in the revised paper. It is Figure 15 on p. 28 of the updated manuscript and reports all BeyondArena sub-benchmarks, with the same groups as in the BeyondArena paper, on all 142 datasets and all outer train-test splits. The table below gives the Elo difference between LightGBM-TabPrep with PriorTune configurations (T+E) and three references, with 95% confidence intervals. Positive values mean our model is ahead.

Group (datasets)vs. random configurationsvs. RealMLP (T+E)vs. TabPFN-3
All (142)+44.4 [31.4, 58.7]+15.7 [-18.3, 48.8]+33.4 [-13.5, 80.6]
IID (103)+52.1 [35.4, 68.0]+24.7 [-19.5, 66.4]-53.3 [-106.8, -2.7]
Grouped (18)+40.8 [13.5, 69.9]-8.4 [-90.3, 64.9]+234.0 [95.9, 401.1]
Temporal (21)+21.2 [-21.6, 68.3]-1.0 [-90.5, 77.8]+211.7 [97.3, 354.7]
Tiny (52)+35.8 [19.8, 52.0]-39.1 [-77.7, -2.3]-105.0 [-160.6, -47.4]
Small (38)+72.4 [41.7, 108.5]+60.2 [-2.3, 129.2]-58.5 [-142.9, 6.5]
Medium (30)+49.4 [19.9, 85.5]+112.9 [23.1, 217.9]+162.0 [56.5, 264.7]
Large (22)+32.1 [-26.7, 90.3]-21.5 [-165.0, 109.2]+387.3 [231.4, 587.5]
Low-dim (91)+41.3 [26.0, 56.2]+25.4 [-14.4, 65.3]-22.4 [-74.5, 27.9]
High-dim (51)+50.7 [23.8, 77.6]-1.5 [-66.7, 58.3]+125.3 [31.5, 222.3]
Text (16)+54.5 [0.2, 114.4]-55.1 [-235.2, 85.9]+63.4 [-134.5, 249.5]
High card. (30)+28.5 [-13.1, 70.6]+72.3 [-31.8, 183.3]+276.8 [164.4, 413.0]

PriorTune improves over random configurations in every group, and the gain is significant in 9 of the 12. Against TabPFN-3, the picture depends on the type of data: TabPFN-3 is ahead on tiny and i.i.d. datasets, and our model is ahead on medium and large datasets, on grouped and temporal splits, and on high-dimensional and high-cardinality data. Against RealMLP, most differences are not significant.

Q4. Why is the TabPFNv1 prior competitive?

We attribute this to the Bayesian inference inside foundation models, which our method does not have. A foundation model approximates the posterior predictive distribution under its prior p(m) over data-generating mechanisms m:

p(y∗∣x∗,D)=∫p(y∗∣x∗,m)p(m∣D)dmp(y^\ast \mid x^\ast, D) = \int p(y^\ast \mid x^\ast, m) p(m \mid D) dm

p(m∣D)∝p(D∣m)p(m).p(m \mid D) \propto p(D \mid m) p(m).

The term p(m∣D)p(m \mid D) puts more weight on the mechanisms in the prior that are close to the dataset at hand. But if p(m)=0 for the true mechanism, no amount of data can fix this. The best prior for a foundation model is therefore the most diverse one, and each new prior is more diverse than the previous one. Our method cannot adapt in the same way, because it does not use the dataset D when it selects configurations. For us, extra diversity in the prior is not automatically helpful. Developing a prior specifically for this task (for example, using TabRepo as a measure of prior quality instead of as a source of selection data) may bring an improvement as large as new priors bring to foundation models.

AuthorsOpenReview ↗

Follow-up Response

Follow-up on Q2: TabRepo-selected configurations on full BeyondArena

In our response to Q2, the comparison between configurations selected on synthetic data and on TabRepo used only the first train-test split of each BeyondArena dataset. The runs for TabPack have now finished on the full BeyondArena (all datasets and all outer train-test splits). On the full benchmark, the configurations selected on synthetic data are significantly ahead of those selected on TabRepo by Elo.

All three sets have 64 configurations and are evaluated under the same T+E protocol.

TabPack configurationsElo
Synthetic (PriorTune)1246.8
TabRepo1225.5
Random1222.9
ComparisonWin rate [95% CI]Elo difference [95% CI]
Synthetic vs. Random61.7% [53.9, 69.5]+23.9 [8.0, 41.3]
Synthetic vs. TabRepo56.7% [48.9, 64.5]+21.3 [5.0, 37.3]
TabRepo vs. Random53.9% [45.4, 62.4]+2.6 [-11.6, 15.5]
  • Synthetic vs. TabRepo. PriorTune is ahead by +21.3 Elo, and the interval excludes zero. On the first split alone this difference was +17 [-5, 40]. The win rate is 56.7%, and its interval still includes 50%.
  • Synthetic vs. Random. PriorTune is ahead by both win rate and Elo, and both intervals exclude the no-difference value.
  • TabRepo vs. Random. The TabRepo-selected set is not distinguishable from random configurations.

For TabPack, selecting on synthetic data gives a significant gain over random configurations, while selecting on TabRepo does not. Intervals are 95% bootstrap intervals over datasets, as in our first response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

We thank the reviewers. Headline new result on the full BeyondArena (142 real datasets, 58 methods): LightGBM-TabPrep with PriorTune is the highest-ranked non-foundation model, ahead of RealMLP and TabPFN-3 by Elo, at 4.4 ms per row on a CPU. With random hyperparameters, the same model ranks below both, and the difference is significant.

The problem

Models trained from scratch cannot learn from other datasets.

  • Foundation models (TabPFN, TabICL) are pretrained once on millions of synthetic datasets and applied to new data without training. They are most accurate on small and medium datasets, need a GPU, and predict slowly.
  • Parametric models (gradient-boosted trees, MLPs) are trained from scratch on each dataset. They are cheap, fast, and the production default for large datasets.

Progress for parametric models has come from manual feature engineering, per-dataset tuning, and hand-found architectures and training techniques (RealMLP, TabM). Meanwhile, synthetic data generators ("priors") improve every year but serve only foundation models.

Our approach

We propose a new mechanism for improving parametric models. We show that the same synthetic data can improve them. What such a model can carry across datasets is its hyperparameters. PriorTune learns a short list of good configurations once, offline:

  1. Generate 4,000 synthetic datasets from a published prior (TabICLv2).
  2. Train 4,000 random configurations on each.
  3. Select the 25 that together most often beat 25 random ones on these datasets.

On a new dataset, one trains these 25 instead of 25 random ones. The cost is that of random search, the list is known in advance, and no real data is used to choose it.

Difference from other HPO methods

TPE, Hyperband and freeze-thaw BO search adaptively on each new dataset and use no other datasets (freeze-thaw only in its surrogate). PriorTune chooses once, offline.

  • Ensembling. The best tabular results come from ensembles of tuned models. HPO trials make a poor ensemble: TPE's are too similar, and Hyperband and freeze-thaw fully train few models. Fixes such as DivBO exist, but no standard benchmark uses them. PriorTune can select configurations for how well they ensemble.
  • Parallelism. A fixed list trains all at once. TPE and freeze-thaw wait for earlier trials, and Hyperband parallelizes only within a round.
  • Trials needed. Adaptive methods need many trials to beat random search and can overfit validation on small data. This is why TabArena and BeyondArena tune by random search, our baseline.
  • Performance. For LightGBM-TabPrep on TabArena (51 datasets), 12–15 of our configurations match TPE with 200 trials, and TPE stops improving after about 60 (win rate vs. 25 random configurations at 200 trials: 63.7% vs. 58.7%, difference +5.0, 95% CI [-5.2, 15.0]). See Figure 14, p. 27 of the updated paper.

Results

Full BeyondArena, all outer splits (the submission used a reduced version). In the revised paper: Figure 13, p. 25 (Elo and inference cost), Table 10, p. 26 (full leaderboard), Figure 15, p. 28 (dataset groups). F = foundation, P = parametric.

RankMethodElo [95% CI]ms per row
1Kumo-Tabular (F)1455 [1399, 1507]236, GPU
2TabPFN-3.5 (F)1427 [1381, 1484]
3–6Kumo-Tabular M/S, TabPFN-3.5-Fast, Causilo (F)1361–141681+, GPU
7LightGBM-TabPrep + PriorTune (FTE) (P)1313 [1287, 1344]4.4, CPU
9RealMLP (P)1297 [1272, 1326]11, GPU
10TabPFN-3 (F)1279 [1238, 1323]243–3,308, GPU
11LightGBM-TabPrep, random (P)1268 [1247, 1296]2.5, CPU
16TabM (P)1228 [1197, 1258]
20CatBoost (P)1219 [1199, 1245]
22TabICLv2 (F)1215 [1172, 1255]102–1,412, GPU
  • Highest-ranked parametric model. Only six methods rank above. PriorTune alone moves it from 11th place past RealMLP and TabPFN-3 (+44 Elo, 95% CI [31, 59]).
  • RealMLP: ours is slightly ahead (win rate 52.0%, p = 0.58), with 2.6x faster inference and no GPU.
  • TabPFN-3: different strengths. It wins on IID datasets, most of which are small. Ours wins on large datasets.
  • Newest foundation models are more accurate, but ours costs 4.4 ms on a CPU against 81–236 ms for Kumo-Tabular and ties it on large datasets (win rate 50.9%).
  • Not specific to one model. PriorTune also improves LightGBM, TabPack and the MLP over random configurations (see reply to ghNY).

Changes to the paper

  • New evidence (pp. 25–31): full BeyondArena Elo and inference (Figure 13, Table 10), TPE curves (Figure 14), dataset groups (Figure 15), full TabArena (Tables 11–12), TabRepo-selected sets on BeyondArena (Table 13).
  • Planned: add the new statistical and HPO comparisons, tighten the claims, make the introduction and method plainer, run TabRepo-selected sets on full BeyondArena, add all models with PriorTune on full TabArena.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.