acceptodds
Withdrawn from ICLR 2027

OrthoBO: Orthogonal Bayesian Hyperparameter Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Bayesian optimization (BO) is widely used for hyperparameter optimization when model evaluations are expensive. However, noisy acquisition estimates can lead to unstable decisions. We identify acquisition estimation noise as a previously overlooked failure mode: even when the surrogate model and acquisition target are correctly specified, finite-sample Monte Carlo error can perturb acquisition values. This can flip candidate rankings and lead to suboptimal BO decisions. As a remedy, we adapt classical score-function control variates to acquisition estimation. Subtracting an optimally weighted control variate yields an acquisition residual *orthogonal* to posterior score directions, with reduced variance and an unchanged marginal acquisition target. We further introduce OrthoBO: a BO framework that combines this orthogonalized acquisition estimation with ensemble surrogates and an outer log transformation. We prove that our estimator preserves the target, reduces variance, and stabilizes pairwise rankings. We further verify the theoretical properties of OrthoBO through numerical experiments where our framework reduces acquisition estimation variance, stabilizes candidate rankings, and achieves strong performance. We also demonstrate the downstream utility of OrthoBO in hyperparameter optimization for neural network training and fine-tuning.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
bayesian optimization, orthogonalization, hyperparameter optimization

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_zFz8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an approach toward reducing the variance in statistical estimates of acquisition functions in Bayesian optimization (BO). In particular, a control variate / orthogonalized estimator for Monte-Carlo (MC) estimates of acquisition functions (sampling, e.g., GP hyperparameters like lengthscales from an approximate hyperposterior) is adapted to this setting, and instantiated for BO based on Gaussian processes and Tree-based (TPE) methods. The method is evaluated on synthetic benchmark functions, as well as on two hyperparameter optimization tasks on training CNNs and fine-tuning VITs.

Critical strengths
  • Improving the sample efficiency of BO, while widely studied, is a relevant problem, and focusing on acquisition functions that are themselves estimated via sampling is a relevant subclass of BO problems. The case of integrating out (approximate) hyperposteriors over model parameters is particularly relevant.
Critical weaknesses
  • I’m quite unsure whether the goal of reducing randomness in the acquisition function is actually a good idea. In fact, Thompson sampling (which explicitly introduces randomness to encourage exploration) is a highly effective acquisition strategy, and would arguably make sense in the considered setting. So to convince me that it is, I’d need to see clear downstream utility on relevant BO tasks on relevant applications. The most important experiments on downstream utility of the method however are rather inconclusive; Figure 4 shows little difference, and in Figure 5 it is unclear whether any of the results are significant given the large error bars.

  • The ensembling / model-selection approach via exponential weights is introduced in passing (21), without proper evaluation in the main paper. If it's meant as an important element of the algorithmic contribution, I would expect proper experiments in the main body of the paper, with ablations separating its impact from orthogonalization. It's also unclear how this interplays with the use of approximate Bayesian inference over surrogate hyperparameters, which in itself can be seen as performing a form of model selection.

  • Noncritical, but the citation formatting is confusing (often nested or entirely missing parentheses).

Questions for authors
  • It is unclear why the ensembling approach is introduced on page 7, when none of the experiments in the main paper seem to evaluate it. In particular, can the authors confirm that in the hyperparameter optimization experiments, ensembling (and in particular updating ensemble weights for purposes of model selection) is not used? In case it is used, is it used equally for all methods?
Reviewer_KYJiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of high variance in estimating acquisition function in BO (specifically focused on EI). Based on the well known variance reduction technique, the authors apply it to BO setting to reduce the variance of estimation and stabilize pair-wise ranking between candidates. They further combine this observation with well-known techniques such as surrogate ensemble and outer log stabilization. Experimental results show that the method works as intended, but missing ablations make it difficult to assess the true benefit of reducing estimator variance.

Critical strengths
  • Paper is written easily.
Critical weaknesses
  • Missing ablation. The paper's main claim is precisely the importance of reducing the variance of acquisition estimator. But it has not been shown in the main paper. Figure 3 and 4 are misleading because they are not controlled experiments. More specifically, there are so many confounding factors that make it difficult to isolate the effect of variance reduction from other effects -- OrthoBO includes surrogate ensemble and out log stabilization, but it's mysterious whether the baselines include the same components or not. The critical missing ablation study is as follows: OrthoBO vs. OrthoBO without variance reduction technique, in the form of best-so-far regret vs. BO iteration curve.
  • Novelty. As already stated in the main paper (in Line 262-263), Theorem 4.1 is well known optimal control variate technique. Contribution of this paper is thus applying this to BO scenario, which in my perspective is somewhat incremental. This novelty issue is also related with the missing ablation issue mentioned above. Also, surrogate model ensemble and outer log stabilization are existing techniques and cannot be argued as a novelty of this paper.
Questions for authors
  • Could you perform the ablation study I mentioned above?
Reviewer_q959OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes OrthoBO, a Bayesian optimization (BO) algorithm that improves Monte Carlo (MC) acquisition estimation to preserve the acquisition target, reduce estimation variance, and stabilize pairwise rankings between candidates. This aims to help acquisition maximization select the most promising candidate more reliably, improving optimization performance. The main idea is to reduce MC estimation error through variance reduction. To do so, the paper proposes an orthogonalized EI, computed by subtracting a control-variate correction from the original EI. The paper presents formulations for GP and TPE-style surrogates. Furthermore, OrthoBO incorporates an ensemble of surrogate models to address surrogate misspecification and an outer log transformation for numerical stability. The manuscript provides theoretical analyses of target preservation, variance reduction, and pairwise ranking-error bounds. The experiments evaluate (1) variance reduction, (2) ranking stability, (3) efficiency with respect to the MC sampling budget, and (4) downstream performance on hyperparameter optimization tasks.

Critical strengths
  • The paper addresses a clearly defined problem, i.e., acquisition estimation noise, that has received relatively limited attention in the BO community.

  • The writing clearly explains the proposed method and its motivation.

  • Several experiments provide evidence supporting the method’s ability to reduce estimation variance and stabilize candidate rankings.

Critical weaknesses
  1. It is unclear whether the main comparison against qLogEI isolates the benefit of the proposed estimator, because the authors do not clearly describe how the baseline is implemented. In OrthoBO’s GP formulation, EI is first computed analytically for a given GP hyperparameter θ\theta using Eq. (13), then marginalized over samples from q(θ)q(\theta), with the proposed correction applied. In contrast, standard BoTorch qLogEI estimates EI by sampling objective values from the predictive posterior. It does not, by itself, introduce the second step of marginalization over GP hyperparameters; this depends on the supplied surrogate model. This raises two potentially critical differences: (1) qLogEI may use MC estimation where OrthoBO uses analytic EI, and (2) OrthoBO explicitly integrates over GP hyperparameter uncertainty, whereas the qLogEI baseline may use fixed hyperparameters and therefore omit the second step. If so, the comparison does not isolate the effect of orthogonalization, making it difficult to attribute the reported gains to the proposed estimator. These implementation details require substantial clarification.

  2. The empirical scope is limited, focusing mainly on EI-based acquisition functions, with an additional experiment for PI. The paper misses an opportunity to extend to information-theoretic acquisitions such as PES, MES, and JES, which rely on sampling-based estimation. Another relevant direction is noisy EI, where MC estimation plays an important role. Such extensions would significantly strengthen the evidence for the method’s applicability.

  3. The RQ3 conclusion that “orthogonalization is especially effective in earlier iterations” is not clearly supported by Figure 3. During approximately the first 50 iterations, TuRBO and qLogEI appear to show more consistent progress than OrthoBO in several plots.

  4. The downstream performance gains appear modest or uncertain. On the 5D CNN task, most methods substantially overlap. On ViT fine-tuning, OrthoBO achieves a higher mean score, but the uncertainty intervals also overlap substantially with competing methods. These results provide limited evidence of a clear, consistent advantage.

  5. The synthetic experiments in RQ3 omit relevant baselines such as PES and JES, despite including them in the downstream HPO experiments. Including these methods on the controlled synthetic benchmarks would strengthen the evaluation of OrthoBO’s competitiveness against other acquisition functions, beyond the main qLogEI comparison.

  6. The baseline implementations require clarification. For example, standard TuRBO uses Thompson sampling (TS), which does not depend on the MC budget. However, Figure 3 shows different TuRBO performance across values of S, without explaining the sources of these differences. This ambiguity also reduces confidence in the interpretation of the downstream comparisons in RQ4. If the authors replace TS with MC-based EI, they should explicitly report this modification. TuRBO’s acquisition and candidate-generation mechanisms (RAASP) can substantially affect its performance [1]. Consequently, a modified implementation may not represent the standard baseline, limiting the conclusions about OrthoBO’s superiority.

[1] Rashidi, Bahador, Kerrick Johnstonbaugh, and Chao Gao. "Cylindrical Thompson sampling for high-dimensional Bayesian optimization." International conference on artificial intelligence and statistics. PMLR, 2024.

Questions for authors
  1. Please clarify the baseline implementations, especially qLogEI. Does it use the same GP hyperparameter posterior and marginalization procedure as OrthoBO? As discussed in the weaknesses, these differences could affect the comparison and invalidate the performance gains across the research questions. Implementation details for other baselines, particularly TuRBO, should also be reported.

  2. It's not clear why the proposed method emphasizes on HPO application. Is there a component that will be more suitable for HPO, rather than general black-box expensive optimization?

  3. Given the paper’s emphasis on HPO, how does OrthoBO compare with established HPO methods such as TPE [1], SMAC [2], and BOHB [3]?

[1] Bergstra, James, et al. "Algorithms for hyper-parameter optimization." Advances in neural information processing systems 24 (2011).

[2] Lindauer, Marius, et al. "SMAC3: A versatile Bayesian optimization package for hyperparameter optimization." Journal of Machine Learning Research 23.54 (2022): 1-9.

[3] Falkner, Stefan, Aaron Klein, and Frank Hutter. "BOHB: Robust and efficient hyperparameter optimization at scale." International conference on machine learning. PMLR, 2018.

Reviewer_iQgTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This study investigates the noisy estimation of acquisition function values in hyper-parameter estimation. The authors proposed an orthogonalized acquisition estimation method to tackle this issue.

Critical strengths
  1. The paper identifies acquisition estimation noise as a source of instability in BO and develops a theoretically grounded variance-reduction approach using control variates.

  2. The paper evaluates variance reduction, ranking stability, and optimization performance across synthetic benchmarks and real-world hyperparameter optimization tasks, providing empirical evidence of the method's potential benefits.

Critical weaknesses
  1. A serious concern is the inconsistency between the mathematical formulation and the experimental results. In Appendix G.1, the TPE control variate is constructed by centering each sample using the empirical mean of the same MC samples. Consequently, the control variates sum to exactly zero, making the orthogonalized acquisition estimator mathematically identical to the original MC estimator, regardless of the estimated control-variate coefficient. However, Table 10 reports substantial variance reductions for this estimator, which directly contradicts the stated formulation. The authors must clarify this discrepancy and verify whether the implementation and reported results are consistent with the proposed method.

  2. The core contribution is essentially a classical Monte Carlo variance-reduction technique applied to acquisition function estimation, rather than a fundamentally new BO methodology. Although the authors employ QMC-based baselines, they do not sufficiently compare against other established variance-reduction strategies or simply increasing the MC sample size under an equal computational budget.

  3. The proposed framework is motivated by acquisition estimation noise arising from Monte Carlo sampling over surrogate hyperparameters. However, in standard GP-based BO, hyperparameters are typically fitted through maximum likelihood or MAP estimation, and common single-point acquisition functions such as Expected Improvement can then be computed analytically without Monte Carlo estimation. While MC marginalization is relevant to fully Bayesian BO and more complex acquisition functions, the paper does not adequately establish how prevalent or consequential this problem is in standard BO practice, potentially overstating the generality and practical importance of its contribution.

Questions for authors
  1. How does OrthoBO compare with established variance-reduction techniques or simply increasing the MC sample size under the same computational budget? What justifies the additional complexity of the proposed method?

  2. In standard GP-based BO, hyperparameters are typically estimated using maximum likelihood or MAP, and EI can be computed analytically. How broadly does the proposed acquisition estimation problem arise in practical BO applications, and when is MC-based acquisition estimation necessary?

  3. In Appendix G.1, the TPE control variates are centered using the same MC samples, making their average exactly zero. Does this not make the orthogonalized estimator identical to the original MC estimator? How can the authors reconcile this mathematical identity with the substantial variance reductions reported in Table 10?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.