SemanticOpt: Towards LLM-Based Semantic Black-Box Optimization

Optimizing expensive black-box systems often requires both domain knowledge and effective sequential decision making. Bayesian optimization (BO) provides strong numerical refinement and uncertainty-aware search, but typically cannot directly use broader semantic information such as expert knowledge, scientific documentation, or prior experiments. Large language models (LLMs) can exploit such information to propose strong initial configurations, but their optimization performance often plateaus as numerical observations accumulate. We introduce SemanticOpt, an LLM-based optimizer that learns model-based optimization behavior from BO trajectories. We construct more than 100,000 optimization trajectories combining context-informed LLM initialization with multiple BO methods across diverse synthetic and real-world-like function spaces, and fine-tune an LLM to jointly propose candidate configurations and predict objective value, uncertainty, and failure probability. Across 42 scientific and engineering black-box optimization problems, SemanticOpt achieves a strong semantic initialization relative to BO while substantially improving iterative refinement over longer optimization trajectories. Its predictions also become more accurate and better calibrated as observations accumulate, and the model is more robust than the base LLM to misleading semantic information. These results suggest that supervision from numerical optimizers can teach LLMs to better incorporate accumulating numerical evidence while preserving their ability to exploit natural-language priors.