-
CaliBO uses one scalar coefficient to control the LLM's influence across the candidate pool, which assumes a stable relationship between LLM-GP disagreement. In practice, the LLM may be useful in some regions and misleading in others. In addition, the distribution of observed residuals is determined by the BO sampling policy and may not necessarily be adequate enough to represent the candidate regions where advice will subsequently be applied. For example, it is possible to design a method that locally adjusts LLM prediction in the form of a prior mean using observed discrepancies, which is adaptive across the whole search space, giving the advantage of adaptive shrinkage instead of a fixed coefficient.
-
The appendix addresses the compliance of output format, but numerical accuracy and consistency were unaddressed. The paper did not adequately evaluate whether LLM predictions are stable across repeated queries or reasonable prompt perturbations.
-
A thorough sensitivity analysis of design choices is not sufficiently provided, and adding multiple hyperparameters also weakens the applicability of the proposed method. CaliBO introduces several constants, selected on a separate design task set. The paper does not demonstrate robustness to these choices. As such, it is hard to tell if the reported results generalize to new problems.
-
The evaluation scope is considerably limited. Firstly, doing experiments with only 5 seeds can lead to highly unstable standard deviation. Secondly, the empirical results rely on only two proprietary OpenAI LLMs. Therefore, it limits the evidence that CaliBO can work well with other LLM families, particularly open-weight models.
-
Section 2 is not sufficiently comprehensive. Requeima et al. (2024) was a pioneering work in using LLMs as the surrogate model, but it was not mentioned. Many other works listed below about using LLMs in BO are also not mentioned, both in using LLMs as a numerical predictor or other aspects.
Requeima, J., Bronskill, J., Choi, D., Turner, R. E., & Duvenaud, D. (2024). Llm processes: Numerical predictive distributions conditioned on natural language. Advances in Neural Information Processing Systems, 37, 109609-109671.
Yuan, X., Chen, Z., Zhang, J., Xiong, H., Ye, N., Li, Y., & Gu, Q. (2026). Unleashing LLMs in Bayesian Optimization: Preference-Guided Framework for Scientific Discovery. The Fourteenth International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=LktUOZayG9
Aglietti, V., Ktena, I., Schrouff, J., Sgouritsa, E., Ruiz, F., Malek, A., … Chiappa, S. (2025). FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch. Forty-Second International Conference on Machine Learning. Retrieved from https://openreview.net/forum?id=XjbJR9374o
Ngo, G., Trong, D. P., Nguyen, D., Gupta, S., & Venkatesh, S. (2026). Adaptive Acquisition Selection for Bayesian Optimization with Large Language Models. The Fourteenth International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=EPKmSgXvRe
Zeng, Y., Maus, N., Jones, H. T., Tao, J., Wan, F., Torres, M. D. T., … Gardner, J. R. (2026). Scaling Multi-Task Bayesian Optimization with Large Language Models. The Fourteenth International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=ekmUkRYnkN