SageLLM-BO: High-Dimensional Bayesian Optimization via Large Language Model-Guided Search Geometry

Bayesian Optimization (BO) is a powerful framework for optimizing expensive black-box functions, yet its performance deteriorates in high-dimensional settings. Instead of operating over the full domain, prior works aim to define a suitable search geometry, a structured region within which BO can operate more efficiently, typically in the form of a low-dimensional subspace and/or a local trust region. However, these methods often rely on problem-specific heuristics and predefined adaptation rules, limiting their robustness. Recent BO methods have shown that Large Language Models (LLMs) can assist the BO process, but they do not explicitly address high-dimensional search and can incur substantial inference costs in such settings. In this work, we propose SageLLM-BO, a novel high-dimensional BO algorithm that leverages the reasoning capabilities of LLMs in a cost-efficient manner to guide the search geometry selection, replacing fixed adaptation rules with adaptive LLM decisions informed by optimization indicators. We further introduce a methodology for constructing a set of example search geometries with their associated quality estimates, and various optimization indicators to guide the LLMs in selecting an effective search geometry. Extensive experiments on synthetic and real-world benchmarks demonstrate that SageLLM-BO consistently matches or outperforms state-of-the-art non-LLM and LLM-based high-dimensional BO methods while substantially reducing LLM cost.