acceptodds
Under review as a conference paper at ICLR 2027

Simple Meta-Learning for HPO and CASH Using Performance Representations

Abstract

Bayesian optimization for Hyperparameter Optimization (HPO) and Combined Algorithm Selection and Hyperparameter Optimization (CASH) often relies on explicit hyperparameter encodings as surrogate inputs. In high-dimensional search spaces, however, such representations can make it difficult for standard surrogates to learn useful performance models. We instead study an alternative meta-learning representation that describes each configuration directly as its historical performance across prior tasks. This representation can be paired directly with standard surrogate models. Our scope is the iterative selection component of automated machine learning, deciding which configuration to evaluate next from a fixed candidate set. We further apply this representation to maximal marginal relevance, a diversity-based re-ranking criterion from recommender systems, as a posterior-free acquisition rule. For CASH on TabArena, a benchmark for tabular data, the performance-based representation paired with a linear Bayesian ridge regression model outperforms dedicated meta-learning baselines, achieving the same tuned performance in roughly half the number of trials. For single-model HPO, it remains a competitive second. Across both benchmarks this method closely matches or improves on regret at a fraction of the complexity and cost, requiring no pre-training or tuning and scaling independently of search space dimensionality.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Bayesian Optimization, Hyperparameter Tuning, Algorithm Selection, Recommender Systems, Collaborative Filtering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VGwoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a simple approach to HPO that uses performance on past tasks as meta-features. The authors describe their method and evaluate it empirically.

Critical strengths

The paper is well written and investigates an interesting problem. The results demonstrate some promise.

Critical weaknesses

However, some of the relevant background work is not discussed, in particular landmarkers for meta-learning (e.g. [1]). The presented idea is very similar, and it would be interesting to what extent it performs differently.

Parts of the paper are unclear. In particular, I was confused about whether a surrogate model is used at all. The description of the method discusses Bayesian ridge regression as the choice for surrogate model, but then in section 3.4 the authors state that the surrogate model is dropped altogether -- is this only for the baseline?

It was also unclear to me how exactly the hyperparameter search space is encoded, in particular with respect to the choice of learner. Section 5 states that the GP surrogates did not show improved performance with the proposed representation, whereas RF surrogates did -- given that RFs are better suited to modelling the discontinuous and categorical spaces considered here, I would have expected the opposite. In my opinion, this warrants some more explanation/investigation.

Another detail that was unclear to me was the cost of the data collection for the historical performances. This essentially amounts to additional evaluations of the target function (training and evaluating models on datasets), so there seems to be a trade-off between obtaining this information and simply running the HPO process for longer. This should at least be discussed.

[1] Pfahringer, Bernhard, Hilan Bensusan, and Christophe Giraud-Carrier. "Meta-Learning by Landmarking Various Learning Algorithms." In Proceedings of the 17th International Conference on Machine Learning (ICML 2000), 743–750. San Francisco: Morgan Kaufmann, 2000.

Questions for authors

See above.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer VGwo

Thank you for the feedback and complimenting the writing quality. Your questions led to interesting insights that have been added to the paper. We address them individually:

some of the relevant background work is not discussed, landmarkers for meta-learning (e.g. [1]). The presented idea is very similar, and it would be interesting to what extent it performs differently.

This is an interesting comparison. Landmarkers and our performance representations are both historical scores used as input features, but landmarkers describe tasks, usually by the scores of a few fast and simple algorithms, whereas we describe configurations by all observed scores.

We compared three landmark-based approaches to our method. The first two use landmarks in the usual way, the third, Linear-Perf-T, takes the symmetry literally and "transposes" our method. Normal Linear-Perf fits one model on the target task, sees each configuration as a sample and uses the historical performance of that configuration as its features. Linear-Perf-T fits one model per configuration, sees each task as a sample and uses historical performance on that task as its features. The revised paper discusses landmarking in the related work (Section 2) and reports this comparison in Appendix G.

Mean regret x100, TabArena-CASHRegret
Linear-Perf13.65
(1) Landmark kNN19.43 ▼
(2) ABLR with landmark input21.12 ▼
ABLR without landmark input19.87 ▼
(3) Linear-Perf-T16.56 ▼

▼: significantly worse than Linear-Perf (Wilcoxon signed-rank, p < 0.05). All methods first evaluate the same 4 landmark configurations (defaults of the 4 cheapest model families), counted in the budget, so Linear-Perf scores 13.65 here instead of 12.19. ABLR with vs. without landmark input: n.s.

I was confused about whether a surrogate model is used at all.

The main proposed method, Linear-Perf, does use a surrogate model. From related findings we additionally construct a surrogate-free baseline (MeanMMR). We hope this clarifies the situation. Section 3 now states this explicitly: Linear-Perf is a standard surrogate-based BO loop, and MeanMMR uses no surrogate.

Section 5 states that the GP surrogates did not show improved performance with the proposed representation, whereas RF surrogates did -- given that RFs are better suited to modelling the discontinuous and categorical spaces considered here, I would have expected the opposite.

That is a good observation, and we share your surprise at how well the GP works in the difficult space. Upon further inspection, we find that this is due to the specific prior that was used, which deals well with the high-dimensional encoding.

The main results are based on botorch's default GP prior: an RBF kernel with automatic relevance determination and the dimension-scaled lengthscale prior of Hvarfner et al. [1], which was designed specifically for high-dimensional inputs. The appendix mentioned a Matern-5/2 kernel, which was outdated and has been corrected. When using a simpler isotropic RBF kernel, the GP-Hyper variant performs worse, and the GP-Perf variant becomes significantly better than GP-Hyper, aligned with your expectation. Neither variant outperforms Linear-Perf.

Mean regret x100, TabArena-CASHGP-HyperGP-PerfPerf - Hyper
RBF-ARD, dimension-scaled prior (paper)20.2523.73+3.48 (n.s.)
RBF isotropic, no prior25.2518.82-6.43 (p = 0.006)

For reference: Linear-Perf 12.19. The corrected kernel description and this comparison are in Appendix B.6, and Section 5 now mentions the role of the prior.

Another detail that was unclear to me was the cost of the data collection for the historical performances. This essentially amounts to additional evaluations of the target function (training and evaluating models on datasets), so there seems to be a trade-off between obtaining this information and simply running the HPO process for longer. This should at least be discussed.

In the case of a novel task and a novel search space, we generally do not expect it to be beneficial to perform evaluations on tasks other than the target task. As is common in meta-learning for HPO, we assume instead that a large effort is taken once to collect a meta-dataset with performances of various configurations on various tasks, to be reused many times. The cost of historical data collection is then not factored into the reported runtime cost of applying the method. The limitations paragraph of the discussion (Section 6) now states this explicitly.

As mentioned in our reply to reviewer ZXy6, we have now found that mean-based imputation of an incomplete historical performance matrix works well (Appendix E). It is therefore also not a necessity to run evaluations to fill in any missing values before HPO can be performed on the target task.

[1] Hvarfner et al. Vanilla Bayesian Optimization Performs Great in High Dimensions. ICML 2024.

Reviewer_BuciOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a configuration's historical performance vector across prior tasks can replace its hyperparameter encoding as the input to standard surrogates for HPO/CASH. It combines this representation with Bayesian ridge regression and a model-free MMR acquisition. And it reports that the result simple method is competitive with or better than dedicated meta-learning baselines on TabArena.

The central claim (the performance representation is preferable to hyperparameter-based meta-learning ) is not yet established, because the comparison does not isolate the representation from other factors and omits the closest matrix-based baselines.

Critical strengths

The approach is straightforward and does not require specialized meta-training. The surrogate cost also does not grow with the dimensionality of the search space.

The authors use task-level 5-fold cross-validation, keep validation and test folds separate, avoid leaking normalized values during search, and ensure that the TabArena lookups are reproducible.

Critical weaknesses

The representation ablation is not fully controlled. The Hyper variants are non-meta-learning surrogates, so "Perf vs. Hyper" mainly shows that using prior-task data helps.

The headline claim lacks statistical support and external validity. Significance is reported within-family only, the "2x fewer trials" claim has no CI, and all stochastic methods use a single seed.

Questions for authors

Could you rerun the Hyper ablations with identical intercept and min_samples_leaf settings?

Could you add at least one direct baseline among OBOE, PMF, and SMAC3 under the same candidate set and evaluation budget?

Could you report per-task variance or confidence intervals for Table 2, uncertainty for the trial-reduction claim, and results over multiple seeds?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer Buci (part 1 of 2)

Thank you for the feedback and appreciating how the idea is simple but well-motivated and carefully evaluated. We will answer your questions:

Could you rerun the Perf/Hyper ablations with identical intercept and min_samples_leaf settings?

Thank you for pointing out this difference in hyperparameters. In the case of intercept for the Linear model, the results were already from matched variants, and the text has been corrected. The min_samples_leaf settings for the Random Forest variants were different across variants. Matching the values either way, RF-Perf stays significantly better than RF-Hyper on CASH. Table 2 now uses min_samples_leaf=3 for both. This improves RF-Perf on CASH, ranking it closer behind Linear-Perf, and makes it slightly worse on HPO (13.14 vs. 11.29), where neither RF variant is significantly better than the other, as before. Other conclusions of the paper are not impacted.

Mean regret x100 on CASH, Perf vs. Hyper: RF with min_samples_leaf=3 for both, 15.35 vs. 28.59 (p < 0.0001); with min_samples_leaf=1 for both, 18.53 vs. 29.94 (p = 0.005). Adding an intercept to both linear variants hurts both, Linear-Hyper far more (14.91 vs. 39.54), so the matched no-intercept setting in the paper is the favourable one for Linear-Hyper. The corrected settings are described in Appendix B.7 (random forest) and Appendix C.1 (intercept), and the figures have been regenerated with the matched random forest.

Could you report per-task variance or confidence intervals for Table 2, uncertainty for the trial-reduction claim, and results over multiple seeds?

Yes. Confidence intervals for Table 2 have now been added based on task-level bootstrapping (Appendix A.1, Table 4), and Table 2 now marks significant differences to Linear-Perf for every method. Repeating all experiments with multiple seeds was not possible in the given time, but we expect that the uncertainty established across 51 total tasks (which includes five refits at the meta-level based on the cross-validation scheme) gives sufficient statistical support. Because tasks differ strongly in difficulty, the per-method intervals overlap considerably, differences between methods are therefore assessed with paired Wilcoxon signed-rank tests over tasks, as marked in the table.

The confidence intervals for the speedup claim depend on the budget at which it is measured. At the budget where ABLR reaches its final best score, the speedup is 2.04x on average with 95% CI of [0.75,4.90]. When averaging over all regret targets between the starting regret and ABLR's final best, the interval is narrower, averaging 1.76x with interval [1.46,2.18]. Because the first interval includes 1 and the comparison now includes stronger baselines such as SMAC3, we have removed the trial-reduction claim from the revised paper.

Our answer on the additional baselines follows in a second comment.

AuthorsOpenReview ↗

Reply to Reviewer Buci (part 2 of 2)

Could you add at least one direct baseline among OBOE, PMF, and SMAC3 under the same candidate set and evaluation budget?

Yes. They were initially excluded due to the difficulty of making controlled comparisons, caused by their additional techniques such as meta-feature based warm starting or ensembling. However, as you remind us of the importance, we re-implemented OBOE [1], PMF [2] and SMAC3 [3] and compared them under controlled ablations. See results below on the original and additional CASH benchmarks.

Mean regret x100TabArena-CASHTabArena-HPOTabBench-BioPMF-51OBOEPMF full
No warm start
Linear-Perf12.199.8413.3112.496.8916.59
SMAC3 (RF, hyperparameters)11.3814.55 ▼26.35 ▼14.4714.65 ▼19.28 ▼
PMF (without its warm start)15.93 ▼8.58 ▲14.0113.669.17 ▼29.89 ▼
With warm start
Linear-Perf-Warm11.2610.228.6912.056.5714.82
OBOE19.88 ▼12.35 ▼18.97 ▼24.34 ▼16.14 ▼41.66 ▼
PMF-----16.79 ▼

▼/▲: significantly worse/better than the Linear-Perf row of the same group (Wilcoxon signed-rank, p < 0.05). Warm starts: Linear-Perf-Warm first evaluates the two configurations with the best mean score over the training datasets; OBOE follows its paper's online stage (experiment design with budget doubling), without ensembling; PMF uses its nearest-neighbour warm start on dataset meta-features. PMF full: all 553 datasets x 38,151 pipelines, 5-fold. In the revised paper, the new baselines are added to Table 2 and to the cross-benchmark Table 3 (Section 5). Implementation details are in Appendix B.8, full results per benchmark in Appendix D.

SMAC3 is the new best result on the main TabArena-CASH benchmark. However, over the first 100 evaluations on CASH, Linear-Perf is ahead of SMAC3 (15.23 vs. 16.71, p = 0.02); SMAC3 catches up later and reaches the lowest final regret (5.2 vs. 8.0 after 200 evaluations). It is also significantly worse than Linear-Perf on four of the other benchmarks.

OBOE is significantly worse than Linear-Perf-Warm on every benchmark. In an ablation (Appendix C.3), replacing its online stage with greedy selection of the predicted best configuration makes it much stronger, which supports our paper's standpoint that simpler is often better. This change also brings OBOE closer to our method. The remaining differences, a regularized full performance vector instead of a low-rank projection and a Bayesian posterior for Expected Improvement, still leave a large gap on OBOE's own benchmark (14.38 vs. 6.57).

PMF works well specifically on the low-dimensional HPO benchmark, but is not significantly better on any other benchmark for either warm start variant. Its dependence on the warm start is analysed in Appendix F.

To summarize, our proposed Linear-Perf method is generally competitive and achieves the lowest regret on all but one CASH setting. However, our headline claims need to be nuanced given these more competitive baselines that close gaps, specifically regarding the strengths of SMAC3 on TabArena-CASH and lower significance of improvements. As our paper already reported, low-dimensional HPO can be solved well with existing methods and benefits less from the performance representation. We have revised the abstract, results, discussion and conclusion accordingly.

[1] Yang et al. OBOE: Collaborative Filtering for AutoML Model Selection. KDD 2019.

[2] Fusi et al. Probabilistic Matrix Factorization for Automated Machine Learning. NeurIPS 2018.

[3] Lindauer et al. SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization. JMLR 2022.

Reviewer_ZXy6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes to use hyperparameter's historical performance to construct useful representation for solving HPO and CASH. Instead of embedding each hyperparameter configuration individually, they suggest to make use of abundant historical information as to how each configuration has performed across diverse tasks. The proposed method seems to work reasonably well on CASH and HPO.

Critical strengths

None

Critical weaknesses
  • Paper is hard to read. Even though the proposed method is extremely simple, the writing is very unclear in many places that it took a while to fully understand the method.
  • Most importantly, this paper resorts to very strong assumption that there exists historical performances across all the tasks, for all the hyperparameter configurations. This problem setting is unrealistic, and ignore the true difficulty of Meta-learning HPO -- the performance matrix P\mathcal{P} is very sparse. This is precisely why meta-learning HPO is difficult problem and why probabilistically interpolating missing values becomes critical. And I assume that this unrealistic assumption is why this paper did experiment only on TabArena benchmark.
  • Moreover, this paper does not compare more advanced meta-learning HPO techniques such as FSBO [1] and CFBO [2]. There should be much more meta-learning HPO literature, so the authors should rigorously search the previous literature.

References

[1] Martin Wistuba, Josif Grabocka, Few-Shot Bayesian Optimization with Deep Kernel Surrogates, ICLR 2021

[2] Lee et al., Cost-Sensitive Multi-Fidelity Bayesian Optimization with Transfer of Learning Curve Extrapolation, NeurIPS 2025

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Reply to Reviewer ZXy6

Thank you for your perspective, which has led to valuable findings based on new experiments. We will address your statements one-by-one:

Most importantly, this paper resorts to very strong assumption that there exists historical performances across all the tasks, for all the hyperparameter configurations.

Indeed, this assumption is made in the submitted paper. We find below that it could be relaxed.

This problem setting is unrealistic, [...] the performance matrix P\mathcal{P} is very sparse.

Not necessarily. Dense or nearly dense matrices do exist, such as the TabArena [1] benchmark data or TabBench-Bio [2]. Other examples are the datasets collected for related matrix-based methods such as OBOE [3] and PMF [4], which operate under similar assumptions.

[...] probabilistically interpolating missing values becomes critical.

We find in new experiments that this is not always the case. Adding non-probabilistic imputation as a pre-processing step, filling a missing score with the configuration's average standardized score on the other training datasets where it was observed, our method keeps a significantly lower regret than ABLR up to the highest sparsity tested (95% missing values, randomly dropped from the CASH benchmark). This relaxes the constraint of a complete matrix to a matrix with only about two scores per configuration on average. We have incorporated these findings and adjusted the stated assumption in the limitations (Section 6), with full results in Appendix E.

Mean regret x100, TabArena-CASHComplete20% kept5% kept
Linear-Perf, no imputation12.1918.2916.43
Linear-Perf, with imputation12.1915.0514.66
ABLR15.47 ▼18.12 ▼17.07 ▼

▼: significantly worse than Linear-Perf with imputation (Wilcoxon signed-rank, p < 0.05). Training entries dropped at random; the target dataset is unchanged.

I assume that this unrealistic assumption is why this paper did experiment only on TabArena benchmark.

To demonstrate applicability beyond TabArena, we repeated experiments with TabBench-Bio, as well as the benchmarks used for OBOE and PMF. Like TabArena-CASH, all three are CASH benchmarks: configurations span multiple model families (and for PMF also preprocessing steps) with their hyperparameters. The main conclusions in the paper transfer, with Linear-Perf significantly beating Linear-Hyper on every benchmark, and beating or matching ABLR on all CASH benchmarks. As in the paper, the gains on the single-model TabArena-HPO benchmark (XGBoost, 201 configurations) are small, with ABLR slightly ahead there.

Mean regret x100TabArena-CASHTabArena-HPOTabBench-BioPMF-51OBOE
Datasets x configs, budget51 x 3622, 20051 x 201, 20046 x 63, 3051 x 3622, 200418 x 219, 75
Linear-Perf12.199.8413.3112.496.89
Linear-Hyper14.59 ▼12.79 ▼30.03 ▼21.09 ▼14.13 ▼
ABLR15.47 ▼9.02 (p=0.05)14.2017.80 ▼9.23 ▼

▼: significantly worse than Linear-Perf (Wilcoxon signed-rank on per-dataset regret, p < 0.05). PMF-51: 51 datasets x 3622 pipelines of the PMF matrix, with its natural 20% missing entries. The revised paper summarizes these benchmarks in Table 3 (Section 5), with full results, including all other methods and the full PMF matrix, in Appendix D.

this paper does not compare more advanced meta-learning HPO techniques such as FSBO [1] and CFBO [2].

Thank you for suggesting these additional baselines. FSBO is a meta-learned MLP feature extractor with a GP head and thus sits between ABLR and MTGP, which are already included. We implemented FSBO and found it to perform similarly to ABLR when neither uses a warm start, and significantly worse than our Linear-Perf.

On CASH (mean regret x100): without warm start, FSBO 16.73 vs. ABLR 15.47 (n.s.) and Linear-Perf 12.19 (p = 0.02). With FSBO's own evolutionary warm-start design, given to both methods so they start from the same 5 configurations: FSBO 13.00 vs. Linear-Perf 10.10 (p = 0.18). FSBO is now part of Table 2, with implementation details in Appendix B.8 and the warm-start comparison in Appendix F.

CFBO is specifically a multi-fidelity HPO method that needs learning curves, which is a different problem setting. We now discuss FSBO in the related work (Section 2).

[1] Erickson et al. TabArena: A Living Benchmark for Machine Learning on Tabular Data. NeurIPS 2025.

[2] Kreuer et al. TabBench-Bio: A Living Benchmark for Machine Learning on High-Dimensional Biomedical Tables. arXiv:2609.07441, 2026.

[3] Yang et al. OBOE: Collaborative Filtering for AutoML Model Selection. KDD 2019.

[4] Fusi et al. Probabilistic Matrix Factorization for Automated Machine Learning. NeurIPS 2018.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal revision

We thank all reviewers for their constructive feedback, and for noting that our idea is simple and well-motivated and the results show promise. We have uploaded a revised version of the paper, with all changes marked in blue. The method and the main experimental protocol are unchanged. The revision adds baselines and benchmarks requested by the reviewers, corrects one mismatched setting and two implementation descriptions. The claims are adjusted accordingly.

New baselines: SMAC3, OBOE, PMF and FSBO, re-implemented on the same candidate set and evaluation budget (Table 2, Appendix B.8), with ablations of their warm starts and online stages (Appendices C.3 and F).

New benchmarks: Three further CASH benchmarks: TabBench-Bio, the PMF meta-dataset (51-dataset subset and all 553 datasets) and OBOE's error matrix (Table 3, Appendix D).

Revised claims: Linear-Perf achieves the best average rank across the four CASH benchmarks against the dedicated meta-learning baselines and SMAC3. SMAC3 only catches up late in the search on TabArena-CASH. On single-model HPO, Linear-Perf significantly outperforms all baselines without meta-learning but trails PMF and ABLR. We no longer use the speedup compared to ABLR as the headline claim. The abstract, introduction, results and conclusion are updated accordingly.

Incomplete matrices: With simple mean imputation, Linear-Perf stays significantly ahead of ABLR with up to 95% of the training matrix missing, and works on the naturally incomplete PMF matrix (Section 6, Appendix E).

Corrections: Both random forest variants now use the same min_samples_leaf, and Table 2 and the figures are updated (Appendix B.7). The description of the linear models' intercept (Appendix C.1) and of the GP kernel (Appendix B.6) now match the code, and the results are unchanged. All per-task times in Table 2 were re-measured on one machine.

Further additions: 95% confidence intervals and significance marks for Table 2 (Appendix A.1), a comparison with landmarking (Section 2, Appendix G), the effect of the GP lengthscale prior (Appendix B.6), and clarifications on the use of a surrogate (Section 3) and on the cost of collecting the performance matrix (Section 6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.