- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors propose to use synthetic datasets from priors used to train tabular foundation models to select default well-performing hyperparameter configurations for four candidate models. On well established benchmarks, the authors argue that the default hyperparameter configurations found outperform the ones obtained from random search.
Critical strengths
- The experiments of the work are run on well established public tabular data benchmarks.
Critical weaknesses
-
The introduction section is difficult to understand. For example, reading Lines 76-84 and Lines 85-90, it is difficult to comprehend what the contributions of the work are. Same for the bulleted list of contributions.
-
There is an undefined reference to an algorithm in Line 212.
-
"Under both deployment rules, the model (or ensemble) is selected on the validation split and scored on the test split. Our objective mimics this, which selects configurations that are stable across validation and test." How should the following paragraph be understood, do the authors select hyperparameter configurations that are well-performing on the validation split and just score them on the test split. Or do they select based on both splits. Because selecting on the validation set, is the proper way in my perspective, otherwise you are leaking the test split. If it is the first case, how does that guarantee that the configuration is stable?
-
The work in it's entirety is difficult to understand. In my perspective the work needs a serious rewriting. First to highlight what the contributions of the work are in the introduction, secondly, to show how the authors conducted the experiments in detail. It is difficult to comprehend both points from the main paper.
-
The authors consider four models in their results, how would the authors apply what they offered to other models, or different hyperparameter search spaces? Would practitioners need to rebuild what the authors offered, how would that time spent compare to fine-tuning the models via say a hyperparameter optimization method?
-
How does the aforementioned approach fare, when for a method on a task , while instead of fitting on the best hyperparameter configurations found from the synthetic datasets, we run a gray-box HPO method [1][2][3][4]? In the time it takes to fit one full configuration, is it possible to evaluate more configurations by pruning on lower budgets to find something better for a few full hyperparameter configurations? If the total budget is 50 epochs (1 full training), one could in theory run 10 configurations for 3 epochs, and one for 20 and potentially achieve something better?
[1] Rakotoarison, H., Adriaensen, S., Mallik, N., Garibov, S., Bergman, E. & Hutter, F.. (2024). In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization. Proceedings of the 41st International Conference on Machine Learning
[2] Lee, D. B., Zhang, A., Kim, B., Park, J., Adriaensen, S., Lee, J., ... & Lee, H. B. (2026). Cost-sensitive freeze-thaw bayesian optimization for efficient hyperparameter tuning. Advances in Neural Information Processing Systems, 38, 169531-169564.
[3] Kadra, A., Janowski, M., Wistuba, M., & Grabocka, J. (2023). Scaling laws for hyperparameter optimization. Advances in Neural Information Processing Systems, 36, 47527-47553.
[4] Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2018). Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of machine learning research, 18(185), 1-52.
Questions for authors
- The work in my opinion should be rewritten and the contributions/experimental protocol be made clear. I have raised a few questions for the authors in the critical weaknesses section, to help improve the paper for future submissions.
4 comments
Response [pt2]
Q4. The work as a whole is difficult to understand
The summary above and the general comment restate the contributions and the experimental protocol. If any specific term or step remains unclear, we would be glad to explain it.
Q5. How does the method apply to other models or search spaces?
The offline step is done once per model and search space, and its result is reused by everyone. We expect the authors of new models to run it when they release the model, as the authors of a foundation model pretrain it once. A practitioner then only trains the released configurations, at the same cost as a random search of the same size.
For a new model, the offline step costs 300 to 400 GPU-hours in our experiments, which is less than pretraining one tabular foundation model. We release the code and the cached results, so that others can apply the method to their own models or extend ours.
Q6. How does the method compare with gray-box HPO?
This is a fair question. We have not run freeze-thaw or Hyperband, and we do not claim that they are ineffective. We see two reasons why a fixed list of configurations is attractive for tabular models, and we add one new experiment.
- Ensembling. On tabular benchmarks, the best results come from ensembling the trained configurations, not from keeping the single best one. Methods that prune early leave few fully trained models, and adaptive search concentrates on similar configurations, so both give weaker ensembles. Diversity-aware methods such as DivBO address this, but they have not been adopted by the standard benchmarks. PriorTune can select its configurations for how well they ensemble.
- Parallelism. A fixed list can be trained all at once, whereas adaptive methods wait for the results of earlier trials.
- Performance. We compared our 25 configurations with TPE for LightGBM-TabPrep on the 51 TabArena datasets. TPE did not catch up at any budget up to 200 trials (win rate against random search at 200 trials: 63.7% for ours, 58.7% for TPE). Our list leads clearly at small budgets. At 200 trials it is still ahead, but the difference is not significant (+5.0 points, 95% CI [-5.2, 15.0]). Figure 14 on p. 27 of the updated PDF shows both curves.
The two approaches can also be combined: our list can serve as the starting set of candidates for a gray-box method. We consider a direct comparison with such methods a valuable next step and will say so in the paper.
References
[1] Feurer, Springenberg, Hutter. Initializing Bayesian Hyperparameter Optimization via Meta-Learning. AAAI, 2015.
[2] Wistuba, Schilling, Schmidt-Thieme. Sequential Model-Free Hyperparameter Tuning. ICDM, 2015.
[3] Winkelmolen, Ivkin, Bozkurt, Karnin. Practical and Sample Efficient Zero-Shot HPO. arXiv:2007.13382, 2020.
[4] Pfisterer, van Rijn, Probst, Müller, Bischl. Learning Multiple Defaults for Machine Learning Algorithms. GECCO, 2021.
[5] Perrone, Jenatton, Seeger, Archambeau. Scalable Hyperparameter Transfer Learning. NeurIPS, 2018.
[6] Wang, Dahl, Swersky, Lee, Nado, Gilmer, Snoek, Ghahramani. Pre-trained Gaussian Processes for Bayesian Optimization. JMLR, 2024.
[7] Müller, Feurer, Hollmann, Hutter. PFNs4BO: In-Context Learning for Bayesian Optimization. ICML, 2023.
Dear Reviewer rQKW,
Thank you again for your review. We would like to kindly ask whether our response has addressed your concerns. In short:
- Clarity. Our response gives a plain summary of the paper, definitions of the key terms, and the contributions in three sentences. We will carry these into the introduction.
- Validation and test splits. Selection uses the validation split only, and the test split is used only for scoring. Our response includes pseudo-code of the protocol.
- Other models and search spaces. The procedure is the same for any model, and it is run once, offline. The selected configurations are then reused on every new dataset at no extra cost.
- Comparison with HPO methods. We added a comparison with TPE (Figure 14, p. 27 of the revised paper) and a discussion of Hyperband and freeze-thaw methods in the General Comment.
If the response resolves your concerns, we would be grateful if you could consider updating your score. If any part of the paper is still unclear, please point us to it, and we will explain it and revise the text. Thank you for your time!
I would like to thank the authors for the reply. Below are my responses:
-
Work difficult to understand
I thank the authors for the extra simplified explanations, however, I would like to kindly point out that I have a confidence level of 4 in my review. It is not the case that I am unfamiliar with the related work, with the current work, with the notation of hyperparameter optimization etc. The other reviewers have their own opinion of the work, maybe they might have also used LLMs. I did not do that and I believe I pointed out exactly what I think was not clear to understand the work in detail, not in general. As such, the AC during the final evaluation is additionally welcome to revise the initial paper, the lines I pointed to and the rest of the paper and decide whether it is clear for the AC or if the AC shares my opinion.
-
Q5
The question I made was deliberate. Because that is a shortcoming of the work, the aim of this work, is to have a good set of defaults that perform well on an unseen dataset, as such, if there is a new method, a new search space, these results cannot be reused. The same experiment would have to be repeated for that method and the practitioner that does it would have to release the results for others to use, while in the meantime, the practitioner can just run hyperparameter optimization. Moreover, there are multiple cases where hyperparameter optimization has task dependent hyperparameters, for example, maybe the type of categorical encoding.
-
Q6
An additional issue I observe, is that these priors have been created so that they allow tabular foundation models to generalize to unseen problems. However, the priors have obviously been skewed so that they perform well on TabArena, given that TabArena and now Beyond, are the benchmarks that are used to advertise the performance of these models. As such, using data from these priors even though synthetic, is optimizing for performance on the aforementioned benchmarks. Lastly, 25 configurations are a considerable amount, given that they would have to be evaluated. What would be the performance of the aforementioned configurations to HyperBand? to IFBO? to DPL?
Combining the work being difficult to understand, for a work which in my opinion is not complex, with the additional aforementioned concerns, I am unable to recommend acceptance for the submission at the current iteration.
Response [pt1]
We thank the reviewer for their input. However, we respectfully disagree that our paper is hard to understand. Reviewers ghNY and V1FT rated the presentation 3 ("Clear and easy to follow") and 2 ("Generally understandable") respectively, and raised no objections to the writing. We will do our best to address your comments and to improve the writing in the revised manuscript. For convenience, we first list the key terms and summarize the paper, and then answer each point.
Key terms
The paper in plain terms
Q1. The introduction and contributions are difficult to understand
In one sentence, we meta-select good hyperparameter configurations on synthetic data. For reference, TabRepo has explored a similar idea on real data. We compare to TabRepo on TabArena (Figure 3, p. 7) and on BeyondArena (Table 13, p. 31). We will revise the introduction to make this point clearer.
Q2. Undefined algorithm reference in Line 212
Thank you for catching this. The reference points to Algorithm 1 (lines 230–247). We have corrected the reference to Algorithm 1 (p. 5).
Q3. Is selection done on validation or on test? Is the test split leaked?
We do not tune on datasets that we benchmark on. Our algorithm exclusively uses synthetic data to meta-select good hyperparameter configurations. Each synthetic dataset has its own train, validation and test parts, and we use them exactly as a practitioner would:
So within a synthetic dataset, the choice among configurations uses validation only. The test part is used to judge whether that choice was a good one.
Why this favors stable configurations. A configuration that looks good on validation by chance gets picked and then scores poorly on test. It loses, so the procedure does not select it.