- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents an approach toward reducing the variance in statistical estimates of acquisition functions in Bayesian optimization (BO). In particular, a control variate / orthogonalized estimator for Monte-Carlo (MC) estimates of acquisition functions (sampling, e.g., GP hyperparameters like lengthscales from an approximate hyperposterior) is adapted to this setting, and instantiated for BO based on Gaussian processes and Tree-based (TPE) methods. The method is evaluated on synthetic benchmark functions, as well as on two hyperparameter optimization tasks on training CNNs and fine-tuning VITs.
Critical strengths
- Improving the sample efficiency of BO, while widely studied, is a relevant problem, and focusing on acquisition functions that are themselves estimated via sampling is a relevant subclass of BO problems. The case of integrating out (approximate) hyperposteriors over model parameters is particularly relevant.
Critical weaknesses
-
I’m quite unsure whether the goal of reducing randomness in the acquisition function is actually a good idea. In fact, Thompson sampling (which explicitly introduces randomness to encourage exploration) is a highly effective acquisition strategy, and would arguably make sense in the considered setting. So to convince me that it is, I’d need to see clear downstream utility on relevant BO tasks on relevant applications. The most important experiments on downstream utility of the method however are rather inconclusive; Figure 4 shows little difference, and in Figure 5 it is unclear whether any of the results are significant given the large error bars.
-
The ensembling / model-selection approach via exponential weights is introduced in passing (21), without proper evaluation in the main paper. If it's meant as an important element of the algorithmic contribution, I would expect proper experiments in the main body of the paper, with ablations separating its impact from orthogonalization. It's also unclear how this interplays with the use of approximate Bayesian inference over surrogate hyperparameters, which in itself can be seen as performing a form of model selection.
-
Noncritical, but the citation formatting is confusing (often nested or entirely missing parentheses).
Questions for authors
- It is unclear why the ensembling approach is introduced on page 7, when none of the experiments in the main paper seem to evaluate it. In particular, can the authors confirm that in the hyperparameter optimization experiments, ensembling (and in particular updating ensemble weights for purposes of model selection) is not used? In case it is used, is it used equally for all methods?