- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SageLLM-BO, a high-dimensional Bayesian optimization method in which an LLM decides the "search geometry" while a standard GP-based BO loop does the actual search. The geometry has two parts: the first is a low-dimensional subspace, defined by how many variables to optimize, which ones, and how to fill in the rest, and the second is a trust region inside that subspace, defined by its size, the shape across dimensions, and how many successes or failures trigger a resize. Existing methods such as TuRBO and BAxUS set these with fixed rules. Here an LLM sets them instead, using a compact text summary of the optimization so far and a small set of candidate subspaces scored by the GP. To keep costs low, the prompts don't grow with the dimension or the number of evaluations, and the LLM is called only when a trust region or subspace update is triggered. Experiments cover three 100-dimensional synthetic functions and three real-world problems with 102 to 180 dimensions, each run for 1000 evaluations with 10 seeds. Baselines include eleven non-LLM high-dimensional BO methods and two LLM-based BO methods. SageLLM-BO does best on the synthetic functions and is competitive on the real-world problems. It uses about 280 to 370 tokens per iteration, which is much cheaper than LLAMBO and LMABO. Ablations replace each LLM component with simpler alternatives, and sensitivity studies vary the LLM, the size of the candidate set and how often the trust region weights are updated.
Critical strengths
- The main strength is the cost-conscious design. Using the LLM to set a handful of geometry parameters, instead of asking it to read the full dataset or propose 100-dimensional points, makes sense for high-dimensional BO. The numbers back this up. SageLLM-BO costs about 0.6 to 0.8 USD for 1000 evaluations, while LLAMBO costs 10 to 23 USD for just 50 (Table 1). Keeping the prompt size independent of the dimension and the evaluation count, and calling the LLM only when an update is triggered, are simple ideas that other LLM-for-BO work could reuse.
- The second strength is the breadth and openness of the evaluation. The paper compares against eleven non-LLM high-dimensional BO methods, ablates both LLM components separately, and tests three different LLMs. It also releases code and includes full example prompts and responses in Appendix H. That last part was especially useful, because it let me see what the LLM is actually asked to do and what it returns.
Critical weaknesses
- The paper doesn't show the LLM adds much beyond the rules written into its prompts. The pitch is that the LLM replaces fixed heuristics, but the prompts contain a lot of heuristics themselves. The subspace prompt (Figure 8) tells the LLM to grow the dimension over time, to jump to a much larger dimension after stagnation, and to prefer K = 100 near the end. The model runs at temperature 0, so in effect it is a deterministic rule written in natural language.
- The example responses in Appendix H look a lot like the defaults the method claims to replace. The LLM picks a failure threshold of 26, which equals the subspace dimension and so matches TuRBO's default. It keeps the success threshold at its initial value of 3. It shrinks the trust region by 0.6, compared with TuRBO's 0.5. It picks a subspace dimension of 26, which is one of the example sizes it was offered.
- The paper never reports what the LLM decides across runs. It doesn't show how often the chosen subspace dimension jumps to 100, which would effectively switch the method to full-dimensional trust-region BO. It also doesn't show how far the shrink ratios and thresholds stray from TuRBO and BAxUS defaults, or how often the weights get changed. The ablations don't settle the question either. They swap each LLM component for a weaker alternative, but none tests a non-LLM policy that follows the same written rules. Without that, I can't tell whether the gains come from LLM reasoning or from good choices built into the candidate set and the prompt.
- The benchmarks and baselines may favor this kind of method. All three synthetic functions have their optimum at or near the center of the domain. Ackley and Rastrigin have it at the origin, and Rosenbrock has it at the all-ones point inside a box centered on the origin. The sparse ±1 linear embedding used as the default fill-in maps p = 0 to x = 0, so every such subspace contains the origin. Recent work has shown that this kind of structure can make high-dimensional BO methods look much better than they are on general problems. Recent work has also shown that plain BO with suitable lengthscale priors is competitive with or better than many specialized high-dimensional methods. Two examples are Hvarfner et al. (ICML 2024), "Vanilla Bayesian Optimization Performs Great in High Dimensions", and Xu et al. (ICLR 2025), "Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization". The "Standard BO" baseline here performs poorly, and the paper doesn't describe how it was configured. So I can't tell whether it reflects these newer findings.
- On the real-world problems, the paper's own text says SageLLM-BO is "competitive". On LassoDNA it reaches a final value similar to LassoBO and CMA-BO, and on HalfCheetah it overtakes BAxUS and BOIDS only after about 300 evaluations. Yet the abstract says it "consistently matches or outperforms" all baselines, and the Figure 2 caption says it "outperforms all baselines substantially on most problems". With 10 seeds and no significance tests, the real-world evidence doesn't support claims that strong. Beating LLAMBO and LMABO also says little on its own. Neither was designed for high-dimensional problems, and LLAMBO was only run for 50 evaluations.
- Since several baselines finish close together on the real-world problems, a table of final values with standard errors would help readers compare methods. The Figure 2 caption says "LLMABO" where it should say LLAMBO, and it describes LLAMBO's 50-evaluation run even though LLAMBO's results are only reported in Appendix F.1. Finally, the method needs about 8 to 15 seconds per iteration, compared with about 2.5 seconds for CMA-BO and about 6 to 7 seconds for TuRBO (Table 2). That's fine for expensive problems, but it's worth stating plainly in the main text.
Questions for authors
- Can you report a log of the LLM's decisions across all runs? For example, how are the chosen subspace dimension, fill-in strategy, shrink ratio, thresholds and weight changes distributed? How often does it pick K = 100? How far do its choices deviate from the TuRBO and BAxUS defaults?
- How does SageLLM-BO compare with a simple non-LLM policy that follows the prompt's own rules? Such a policy could grow the dimension on a fixed schedule, pick the example subspace with the best acquisition value, and use a fixed shrink ratio. Could you also try a prompt ablation that removes the dimension statistics or the history, to test whether the LLM actually uses them?
- Can you add BO with a dimension-scaled lengthscale prior, following Hvarfner et al. (2024) or Xu et al. (2025), as a baseline? How exactly is the current "Standard BO" baseline configured?
- How does the method do on shifted or rotated synthetic functions, so that the optimum isn't at or near the center of the domain?
- Can you add significance tests or confidence intervals for the final values on the real-world problems?
- How do you handle LLM outputs that fail to parse or fall outside the valid range? How much does performance vary across repeated runs with the same seed, given that API calls aren't fully deterministic even at temperature 0?
9 comments
Response to Reviewer qPRX
- The example responses in Appendix H look a lot like the defaults the method claims to replace. The LLM picks a failure threshold of 26, which equals the subspace dimension and so matches TuRBO's default. It keeps the success threshold at its initial value of 3. It shrinks the trust region by 0.6, compared with TuRBO's 0.5. It picks a subspace dimension of 26, which is one of the example sizes it was offered.
We would like to emphasize that the LLM’s decisions differ substantially from the defaults in TuRBO and BAxUS. We added the full logs of the LLM’s decisions to Appendix I to support this comparison.
For the failure threshold (Figure 25), TuRBO and BAxUS use a fixed rule based on the subspace dimensionality, while SageLLM-BO’s threshold tends to increase before the first subspace expansion and decrease within each subsequent subspace period. For example, for seed 1 on Ackley-100D, before the first subspace expansion, after the first expansion, and after the second expansion. On the same problem, TuRBO sets , and BAxUS sets to 1, 4, and 14 in the respective subspace periods. SageLLM-BO uses success thresholds of 2 or 3, while TuRBO always uses 3. The shrinkage ratio (Figure 23) ranges from 0.3 to 0.7, with a median of 0.6 across the six benchmarks. Depending on the benchmark, 50-70% of the ratios are higher than the default of 0.5 used by TuRBO and BAxUS. One possible explanation for the small difference between the median and the default is that the prompt explanation “0.5 halves the trust region” gave the LLM a reference point for adapting the shrinkage ratio during optimization. Other differences in decisions can be found in the response to Question 1.
Regarding the comment “It picks a subspace dimension of 26, which is one of the example sizes it was offered,” our logs in Appendix I.1 show that the LLM always selects one of the dimensionalities in the set of example subspaces . We argue that, with access to only a summary of the observed data, it is better for the LLM to rely on the explicit quality estimates provided from the example subspaces, rather than its own predicted numerical estimates, when choosing a dimensionality.
- The paper never reports what the LLM decides across runs. It doesn't show how often the chosen subspace dimension jumps to 100, which would effectively switch the method to full-dimensional trust-region BO. It also doesn't show how far the shrink ratios and thresholds stray from TuRBO and BAxUS defaults, or how often the weights get changed. The ablations don't settle the question either. They swap each LLM component for a weaker alternative, but none tests a non-LLM policy that follows the same written rules. Without that, I can't tell whether the gains come from LLM reasoning or from good choices built into the candidate set and the prompt.
We added the LLM decision logs for all runs on the 6 benchmarks to Appendix I, including the subspace dimensionality in Figure 19, the shrinkage ratio in Figure 23, the failure threshold in Figure 25, and the weight changes in Figure 27. The results show that the LLM’s choices vary across problems and seeds and differ substantially from the defaults used by TuRBO and BAxUS.
Regarding how often the LLM chooses , Figure 19 shows that SageLLM-BO never sets in some runs (e.g., seeds 7 and 8 on LassoDNA-180D and seed 3 on HalfCheetah-102D) and sets very late in some other runs (after iteration 700 in some runs on HalfCheetah-102D and Pusher-161D). BAxUS on the other hand always sets before iteration 300. For the shrinkage ratios and failure thresholds, Figures 23 and 25 show how SageLLM-BO’s decisions differ from the defaults used by TuRBO and BAxUS. The specific ratios and thresholds can be found in our response to Weakness 2. For the weight changes, Figure 27 shows that each update changes the weights of 7-10% of the subspace dimensions. BAxUS and TuRBO do not update trust region weights. Other differences in decisions can be found in the response to Question 1.
Regarding the original ablations studies in Section 4.3, they are an important analysis showing the contribution of the subspace embedding and trust region LLMs separately. Nevertheless, following the reviewer’s suggestion, we have evaluated whether the gains of SageLLM-BO can be reproduced by a non-LLM policy that follows similar written rules. As mentioned in our response to Weakness 1, we added the “Prompt Rule” variant in Appendix I.3 following the reviewer’s Question 2. Figure 28 compares SageLLM-BO with the “Prompt Rule” variant. The results show that SageLLM-BO outperforms “Prompt Rule” on all 6 benchmarks, indicating that the non-LLM policy derived from the written rules does not reproduce the performance gains of the LLM’s reasoning. Please refer to our response to Weakness 1 for the full details.
Response to Reviewer qPRX (cont. 2)
- The benchmarks and baselines may favor this kind of method. All three synthetic functions have their optimum at or near the center of the domain. Ackley and Rastrigin have it at the origin, and Rosenbrock has it at the all-ones point inside a box centered on the origin. The sparse ±1 linear embedding used as the default fill-in maps p = 0 to x = 0, so every such subspace contains the origin. Recent work has shown that this kind of structure can make high-dimensional BO methods look much better than they are on general problems. Recent work has also shown that plain BO with suitable lengthscale priors is competitive with or better than many specialized high-dimensional methods. Two examples are Hvarfner et al. (ICML 2024), "Vanilla Bayesian Optimization Performs Great in High Dimensions", and Xu et al. (ICLR 2025), "Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization". The "Standard BO" baseline here performs poorly, and the paper doesn't describe how it was configured. So I can't tell whether it reflects these newer findings.
We agree that optima at or near the center of the domain can favor BAxUS’s linear embeddings and, consequently, our proposed method. To assess the effect of this benchmark structure, we evaluated shifted versions of the synthetic functions in Appendix M. In particular, we generated shifted versions of Ackley-100D, Rastrigin-100D, and Rosenbrock-100D following Xu et al. (2025b). Given the limited time available during the rebuttal, we compared SageLLM-BO with 3 main baselines (BAxUS, TuRBO, and BOIDS). Figure 31 shows that SageLLM-BO performs similarly to TuRBO and BAxUS on 2 out of 3 problems and worse on 1 out of 3. It performs similarly to BOIDS on 1 out of 3 problems and worse on 2 out of 3. We will explicitly acknowledge this limitation of SageLLM-BO, which is inherited from BAxUS’s linear embedding, in the Limitations section. However, it is worth noting that, on the 3 real-world benchmarks, where the optima are unknown, SageLLM-BO outperforms the baselines (in Section 4 of the original manuscript). Figure 2 shows that SageLLM-BO outperforms all 13 baselines on HalfCheetah-102D and Pusher-161D. On LassoDNA-180D, SageLLM-BO converges faster over the first 200 evaluations and achieves comparable final values. Moreover, to broaden our comparisons, we followed Reviewer Ljxf’s suggestion and evaluated the original synthetic functions (Ackley, Rastrigin, and Rosenbrock) at 20D and 1000D against TuRBO and BAxUS, as reported in Appendix L. Figure 30 shows that SageLLM-BO generally outperforms TuRBO and BAxUS, especially at 1000D. Although these additional synthetic benchmarks do not have shifted optima, SageLLM-BO’s better performance than BAxUS, which benefits from the same benchmark structure, highlights SageLLM-BO’s effectiveness across dimensionalities.
We have added a new paragraph in Appendix D.2 of the revised manuscript to clarify the “Standard BO” baseline, which uses a GP with a Matern-5/2 kernel and Thompson Sampling acquisition function (similar to TuRBO’s surrogate model and acquisition function). We also added the plain BO method with suitable lengthscale priors mentioned by the reviewer. In particular, we added D-Scaled-BO (Hvarfner et al., 2024) to the main experiments in Section 4. Following Reviewer Ljxf’s suggestion, we also added LinearBO (Doumont et al., 2026), which shares a similar idea. Figure 2 shows that SageLLM-BO outperforms both new baselines on 5 out of 6 benchmarks. On LassoDNA-180D, both new baselines achieve final objective values similar to those of SageLLM-BO, while SageLLM-BO converges faster over the first 200 evaluations.
Response to Reviewer qPRX
We thank the reviewer for the insightful review. We address the questions and comments as follows:
Weakness
We would like to clarify that our prompts only provide the LLM with general guidance based on theoretical analyses and insights from prior work, rather than imposing deterministic heuristics as in prior work. We also clarify that the LLM contribute to the performance gains of SageLLM-BO via its reasoning, not just the written prompt rules.
First, in the subspace prompt (Figure 8), the guideline “grow the dimension over time” refers to the well-known strategy of expanding the subspace dimensionality, which is used in BAxUS, Bounce, and BOIDS. Theoretical analyses identify this expansion as a requirement for subspace embedding methods to converge to the global optimum. The guideline “to jump to a much higher dimension after stagnation” is based on the empirical settings of these methods. It helps avoid spending too much of the evaluation budget on subspaces that may not contain the global optimum. Finally, the guidance “to prefer K=100” near the end refers to the general recommendation to prefer K=D, where D is the problem dimensionality. This guidance is also based on theoretical analyses of subspace embedding, which encourages the method to reach the full dimensionality as iterations grow for global convergence.
Second, our guidance does not make the LLMs deterministic, whereas BAxUS uses fixed heuristics. For example, when the subspace is updated, the LLM still needs to determine the new subspace dimensionality K and decide when to set K=D. Figures 19 and 20 show the subspace dimensionality choices of SageLLM-BO and BAxUS, respectively. The figures show that SageLLM-BO selects different dimensionalities across seeds, while BAxUS follows a deterministic expansion sequence that multiplies the next dimensionality by 4. For example, on a 100D problem, BAxUS always chooses dP=K=2,8,32,96,100, while SageLLM-BO can choose dP=K=2,51,88,100 in some seeds. Moreover, the iteration at which SageLLM-BO sets K=D varies considerably across seeds and problems, spanning around 500 iterations, with most runs reaching full dimensionality between iterations 250 and 750. For BAxUS, the spread is around 100 iterations, and most runs reach full dimensionality before iteration 300. These results show that the LLM makes decisions adaptively, not just following a fixed deterministic rule as in prior work.
Third, to confirm that the LLM contribute to the performance gains of SageLLM-BO via its reasoning, following the reviewer’s Question 2, we evaluated a non-LLM policy called “Prompt Rule,” which is based on the written rules in the prompts (Appendix I.3). In this variant, at each subspace expansion, we generate the set of example subspaces CSS and select the example that maximizes the acquisition value indicator. The trust region shrinkage and expansion ratios are 0.5 and 2.0, respectively. The success threshold is set to τs=3, and the failure threshold follows the rule used in BAxUS. Figure 28 compares SageLLM-BO with this variant. The result shows that SageLLM-BO outperforms the “Prompt Rule” variant on all 6 benchmarks, indicating that the non-LLM policy derived from the written rules cannot achieve the performance gains of SageLLM-BO.