In-context learning for latent space Bayesian optimization

Latent-space Bayesian optimization (LSBO) enables sample-efficient optimization over structured objects such as molecules by searching in the continuous space of a generative model. However, its surrogate is typically fitted anew for each objective from limited observations, without transferring experience across tasks. We investigate whether pretrained in-context predictors can provide such transfer in LSBO. Using prior-data fitted networks (PFNs), we introduce PILBO (_Pretrained In-context Latent-space Bayesian Optimization_), a framework combining pretrained in-context prediction with latent-space molecular generation. Its main implementation predicts from molecular representations independent of the generator and achieves the strongest aggregate performance on GuacaMol optimization objectives. We further introduce a continued-pretraining (CPT) scheme based on handcrafted synthetic molecular objectives and BO-oriented episode construction. The pretrained PFN already captures most of the benefit, with CPT yielding only modest further gains. Our analyses suggest that this limited transfer reflects a mismatch between the offline synthetic episodes used for adaptation and the adaptive prediction problems encountered during BO. Overall, pretrained in-context surrogates provide a strong alternative to repeatedly fitted LSBO surrogates.