- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a simple approach to HPO that uses performance on past tasks as meta-features. The authors describe their method and evaluate it empirically.
Critical strengths
The paper is well written and investigates an interesting problem. The results demonstrate some promise.
Critical weaknesses
However, some of the relevant background work is not discussed, in particular landmarkers for meta-learning (e.g. [1]). The presented idea is very similar, and it would be interesting to what extent it performs differently.
Parts of the paper are unclear. In particular, I was confused about whether a surrogate model is used at all. The description of the method discusses Bayesian ridge regression as the choice for surrogate model, but then in section 3.4 the authors state that the surrogate model is dropped altogether -- is this only for the baseline?
It was also unclear to me how exactly the hyperparameter search space is encoded, in particular with respect to the choice of learner. Section 5 states that the GP surrogates did not show improved performance with the proposed representation, whereas RF surrogates did -- given that RFs are better suited to modelling the discontinuous and categorical spaces considered here, I would have expected the opposite. In my opinion, this warrants some more explanation/investigation.
Another detail that was unclear to me was the cost of the data collection for the historical performances. This essentially amounts to additional evaluations of the target function (training and evaluating models on datasets), so there seems to be a trade-off between obtaining this information and simply running the HPO process for longer. This should at least be discussed.
[1] Pfahringer, Bernhard, Hilan Bensusan, and Christophe Giraud-Carrier. "Meta-Learning by Landmarking Various Learning Algorithms." In Proceedings of the 17th International Conference on Machine Learning (ICML 2000), 743–750. San Francisco: Morgan Kaufmann, 2000.
Questions for authors
See above.
Reply to Reviewer VGwo
Thank you for the feedback and complimenting the writing quality. Your questions led to interesting insights that have been added to the paper. We address them individually:
This is an interesting comparison. Landmarkers and our performance representations are both historical scores used as input features, but landmarkers describe tasks, usually by the scores of a few fast and simple algorithms, whereas we describe configurations by all observed scores.
We compared three landmark-based approaches to our method. The first two use landmarks in the usual way, the third, Linear-Perf-T, takes the symmetry literally and "transposes" our method. Normal Linear-Perf fits one model on the target task, sees each configuration as a sample and uses the historical performance of that configuration as its features. Linear-Perf-T fits one model per configuration, sees each task as a sample and uses historical performance on that task as its features. The revised paper discusses landmarking in the related work (Section 2) and reports this comparison in Appendix G.
▼: significantly worse than Linear-Perf (Wilcoxon signed-rank, p < 0.05). All methods first evaluate the same 4 landmark configurations (defaults of the 4 cheapest model families), counted in the budget, so Linear-Perf scores 13.65 here instead of 12.19. ABLR with vs. without landmark input: n.s.
The main proposed method, Linear-Perf, does use a surrogate model. From related findings we additionally construct a surrogate-free baseline (MeanMMR). We hope this clarifies the situation. Section 3 now states this explicitly: Linear-Perf is a standard surrogate-based BO loop, and MeanMMR uses no surrogate.
That is a good observation, and we share your surprise at how well the GP works in the difficult space. Upon further inspection, we find that this is due to the specific prior that was used, which deals well with the high-dimensional encoding.
The main results are based on
botorch's default GP prior: an RBF kernel with automatic relevance determination and the dimension-scaled lengthscale prior of Hvarfner et al. [1], which was designed specifically for high-dimensional inputs. The appendix mentioned a Matern-5/2 kernel, which was outdated and has been corrected. When using a simpler isotropic RBF kernel, the GP-Hyper variant performs worse, and the GP-Perf variant becomes significantly better than GP-Hyper, aligned with your expectation. Neither variant outperforms Linear-Perf.For reference: Linear-Perf 12.19. The corrected kernel description and this comparison are in Appendix B.6, and Section 5 now mentions the role of the prior.
In the case of a novel task and a novel search space, we generally do not expect it to be beneficial to perform evaluations on tasks other than the target task. As is common in meta-learning for HPO, we assume instead that a large effort is taken once to collect a meta-dataset with performances of various configurations on various tasks, to be reused many times. The cost of historical data collection is then not factored into the reported runtime cost of applying the method. The limitations paragraph of the discussion (Section 6) now states this explicitly.
As mentioned in our reply to reviewer ZXy6, we have now found that mean-based imputation of an incomplete historical performance matrix works well (Appendix E). It is therefore also not a necessity to run evaluations to fill in any missing values before HPO can be performed on the target task.
[1] Hvarfner et al. Vanilla Bayesian Optimization Performs Great in High Dimensions. ICML 2024.