BOING: OPTIMIZING BAYESIAN OPTIMIZATION WITH INFORMATION GEOMETRY

We study Bayesian optimization (BO) through the lens of information geometry. We use insights from existing work that the surrogate, conventionally a GP, maps the input space to a lower-dimensional statistical manifold. The Acquisition Functions (AFs) in BO are defined on this manifold. Instead of conventional gradient-based optimization methods like L-BFGS or Adam, we propose $\mathrm{BOING}$: an iteration-free, trust-region-based method for AF optimization on a low-dimensional manifold that is parallelizable on a GPU. For a certain class of AFs in sequential BO, we show that the optimization of AF can be performed on a 1-D search space. We use the Fisher Information Matrix to construct the trust regions for $\mathrm{BOING}$. Our method achieves competitive or superior overall BO performance. Our method provides a $30$ - $20\times$ computational speedup for low-dimensional BO and reduces the total BO time by $20$ - $40$%. For high-dimensional BO, we achieve a $50$ - $100\times$ speedup and reduce the total BO time by $300$ - $500$%. We show speedups across multiple benchmarks and acquisition functions.