OrthoBO: Orthogonal Bayesian Hyperparameter Optimization

Bayesian optimization (BO) is widely used for hyperparameter optimization when model evaluations are expensive. However, noisy acquisition estimates can lead to unstable decisions. We identify acquisition estimation noise as a previously overlooked failure mode: even when the surrogate model and acquisition target are correctly specified, finite-sample Monte Carlo error can perturb acquisition values. This can flip candidate rankings and lead to suboptimal BO decisions. As a remedy, we adapt classical score-function control variates to acquisition estimation. Subtracting an optimally weighted control variate yields an acquisition residual *orthogonal* to posterior score directions, with reduced variance and an unchanged marginal acquisition target. We further introduce OrthoBO: a BO framework that combines this orthogonalized acquisition estimation with ensemble surrogates and an outer log transformation. We prove that our estimator preserves the target, reduces variance, and stabilizes pairwise rankings. We further verify the theoretical properties of OrthoBO through numerical experiments where our framework reduces acquisition estimation variance, stabilizes candidate rankings, and achieves strong performance. We also demonstrate the downstream utility of OrthoBO in hyperparameter optimization for neural network training and fine-tuning.