Agentic Bayesian Optimization through Surrogate-Augmented Autoresearch

Bayesian optimization (BO) is the standard tool for sample-efficient gradient-free optimization through uncertainty-aware search driven by generic statistical priors. Richer domain priors can improve BO, but encoding them through, e.g., tailored kernels is difficult. LLMs can help bring these priors into the optimization, drawing them directly from natural language, code, and documentation. However, existing LLM-based BO methods either insert the LLM into a fixed role, surrogate, acquisition proxy, or configuration interface, or hand it broad control, sacrificing the systematic exploration that makes BO reliable. We introduce agentic Bayesian optimization: a paradigm in which an LLM agent controls the BO loop while a Bayesian backend holds the probabilistic surrogate. The agent selects evaluations and can revise the strategy during the run by tightening bounds, proposing targeted evaluations, or reframing the problem as evidence or instructions change. We instantiate this idea in Sara, a surrogate-augmented autoresearch agent, and lenz, a modular BO backend that the agent can inspect and modify through a structured interface. Across synthetic and real-world benchmarks, Sara preserves the reliability of standard BO without prior knowledge, outperforms LLM-based baselines, and uses natural-language priors to improve beyond standard BO. Furthermore, in dynamic settings, Sara can reconfigure the full optimization problem on the fly as requirements change; a capability not available in standard BO.