acceptodds
Under review as a conference paper at ICLR 2027

SageLLM-BO: High-Dimensional Bayesian Optimization via Large Language Model-Guided Search Geometry

Abstract

Bayesian Optimization (BO) is a powerful framework for optimizing expensive black-box functions, yet its performance deteriorates in high-dimensional settings. Instead of operating over the full domain, prior works aim to define a suitable search geometry, a structured region within which BO can operate more efficiently, typically in the form of a low-dimensional subspace and/or a local trust region. However, these methods often rely on problem-specific heuristics and predefined adaptation rules, limiting their robustness. Recent BO methods have shown that Large Language Models (LLMs) can assist the BO process, but they do not explicitly address high-dimensional search and can incur substantial inference costs in such settings. In this work, we propose SageLLM-BO, a novel high-dimensional BO algorithm that leverages the reasoning capabilities of LLMs in a cost-efficient manner to guide the search geometry selection, replacing fixed adaptation rules with adaptive LLM decisions informed by optimization indicators. We further introduce a methodology for constructing a set of example search geometries with their associated quality estimates, and various optimization indicators to guide the LLMs in selecting an effective search geometry. Extensive experiments on synthetic and real-world benchmarks demonstrate that SageLLM-BO consistently matches or outperforms state-of-the-art non-LLM and LLM-based high-dimensional BO methods while substantially reducing LLM cost.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian optimization, high-dimensional optimization, large language model-guided optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qPRXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SageLLM-BO, a high-dimensional Bayesian optimization method in which an LLM decides the "search geometry" while a standard GP-based BO loop does the actual search. The geometry has two parts: the first is a low-dimensional subspace, defined by how many variables to optimize, which ones, and how to fill in the rest, and the second is a trust region inside that subspace, defined by its size, the shape across dimensions, and how many successes or failures trigger a resize. Existing methods such as TuRBO and BAxUS set these with fixed rules. Here an LLM sets them instead, using a compact text summary of the optimization so far and a small set of candidate subspaces scored by the GP. To keep costs low, the prompts don't grow with the dimension or the number of evaluations, and the LLM is called only when a trust region or subspace update is triggered. Experiments cover three 100-dimensional synthetic functions and three real-world problems with 102 to 180 dimensions, each run for 1000 evaluations with 10 seeds. Baselines include eleven non-LLM high-dimensional BO methods and two LLM-based BO methods. SageLLM-BO does best on the synthetic functions and is competitive on the real-world problems. It uses about 280 to 370 tokens per iteration, which is much cheaper than LLAMBO and LMABO. Ablations replace each LLM component with simpler alternatives, and sensitivity studies vary the LLM, the size of the candidate set and how often the trust region weights are updated.

Critical strengths
  • The main strength is the cost-conscious design. Using the LLM to set a handful of geometry parameters, instead of asking it to read the full dataset or propose 100-dimensional points, makes sense for high-dimensional BO. The numbers back this up. SageLLM-BO costs about 0.6 to 0.8 USD for 1000 evaluations, while LLAMBO costs 10 to 23 USD for just 50 (Table 1). Keeping the prompt size independent of the dimension and the evaluation count, and calling the LLM only when an update is triggered, are simple ideas that other LLM-for-BO work could reuse.
  • The second strength is the breadth and openness of the evaluation. The paper compares against eleven non-LLM high-dimensional BO methods, ablates both LLM components separately, and tests three different LLMs. It also releases code and includes full example prompts and responses in Appendix H. That last part was especially useful, because it let me see what the LLM is actually asked to do and what it returns.
Critical weaknesses
  • The paper doesn't show the LLM adds much beyond the rules written into its prompts. The pitch is that the LLM replaces fixed heuristics, but the prompts contain a lot of heuristics themselves. The subspace prompt (Figure 8) tells the LLM to grow the dimension over time, to jump to a much larger dimension after stagnation, and to prefer K = 100 near the end. The model runs at temperature 0, so in effect it is a deterministic rule written in natural language.
  • The example responses in Appendix H look a lot like the defaults the method claims to replace. The LLM picks a failure threshold of 26, which equals the subspace dimension and so matches TuRBO's default. It keeps the success threshold at its initial value of 3. It shrinks the trust region by 0.6, compared with TuRBO's 0.5. It picks a subspace dimension of 26, which is one of the example sizes it was offered.
  • The paper never reports what the LLM decides across runs. It doesn't show how often the chosen subspace dimension jumps to 100, which would effectively switch the method to full-dimensional trust-region BO. It also doesn't show how far the shrink ratios and thresholds stray from TuRBO and BAxUS defaults, or how often the weights get changed. The ablations don't settle the question either. They swap each LLM component for a weaker alternative, but none tests a non-LLM policy that follows the same written rules. Without that, I can't tell whether the gains come from LLM reasoning or from good choices built into the candidate set and the prompt.
  • The benchmarks and baselines may favor this kind of method. All three synthetic functions have their optimum at or near the center of the domain. Ackley and Rastrigin have it at the origin, and Rosenbrock has it at the all-ones point inside a box centered on the origin. The sparse ±1 linear embedding used as the default fill-in maps p = 0 to x = 0, so every such subspace contains the origin. Recent work has shown that this kind of structure can make high-dimensional BO methods look much better than they are on general problems. Recent work has also shown that plain BO with suitable lengthscale priors is competitive with or better than many specialized high-dimensional methods. Two examples are Hvarfner et al. (ICML 2024), "Vanilla Bayesian Optimization Performs Great in High Dimensions", and Xu et al. (ICLR 2025), "Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization". The "Standard BO" baseline here performs poorly, and the paper doesn't describe how it was configured. So I can't tell whether it reflects these newer findings.
  • On the real-world problems, the paper's own text says SageLLM-BO is "competitive". On LassoDNA it reaches a final value similar to LassoBO and CMA-BO, and on HalfCheetah it overtakes BAxUS and BOIDS only after about 300 evaluations. Yet the abstract says it "consistently matches or outperforms" all baselines, and the Figure 2 caption says it "outperforms all baselines substantially on most problems". With 10 seeds and no significance tests, the real-world evidence doesn't support claims that strong. Beating LLAMBO and LMABO also says little on its own. Neither was designed for high-dimensional problems, and LLAMBO was only run for 50 evaluations.
  • Since several baselines finish close together on the real-world problems, a table of final values with standard errors would help readers compare methods. The Figure 2 caption says "LLMABO" where it should say LLAMBO, and it describes LLAMBO's 50-evaluation run even though LLAMBO's results are only reported in Appendix F.1. Finally, the method needs about 8 to 15 seconds per iteration, compared with about 2.5 seconds for CMA-BO and about 6 to 7 seconds for TuRBO (Table 2). That's fine for expensive problems, but it's worth stating plainly in the main text.
Questions for authors
  • Can you report a log of the LLM's decisions across all runs? For example, how are the chosen subspace dimension, fill-in strategy, shrink ratio, thresholds and weight changes distributed? How often does it pick K = 100? How far do its choices deviate from the TuRBO and BAxUS defaults?
  • How does SageLLM-BO compare with a simple non-LLM policy that follows the prompt's own rules? Such a policy could grow the dimension on a fixed schedule, pick the example subspace with the best acquisition value, and use a fixed shrink ratio. Could you also try a prompt ablation that removes the dimension statistics or the history, to test whether the LLM actually uses them?
  • Can you add BO with a dimension-scaled lengthscale prior, following Hvarfner et al. (2024) or Xu et al. (2025), as a baseline? How exactly is the current "Standard BO" baseline configured?
  • How does the method do on shifted or rotated synthetic functions, so that the optimum isn't at or near the center of the domain?
  • Can you add significance tests or confidence intervals for the final values on the real-world problems?
  • How do you handle LLM outputs that fail to parse or fall outside the valid range? How much does performance vary across repeated runs with the same seed, given that API calls aren't fully deterministic even at temperature 0?
9 comments
AuthorsOpenReview ↗

Response to Reviewer qPRX

We thank the reviewer for the insightful review. We address the questions and comments as follows:

Weakness

  1. The paper doesn't show the LLM adds much beyond the rules written into its prompts. The pitch is that the LLM replaces fixed heuristics, but the prompts contain a lot of heuristics themselves. The subspace prompt (Figure 8) tells the LLM to grow the dimension over time, to jump to a much larger dimension after stagnation, and to prefer K = 100 near the end. The model runs at temperature 0, so in effect it is a deterministic rule written in natural language.

We would like to clarify that our prompts only provide the LLM with general guidance based on theoretical analyses and insights from prior work, rather than imposing deterministic heuristics as in prior work. We also clarify that the LLM contribute to the performance gains of SageLLM-BO via its reasoning, not just the written prompt rules.

First, in the subspace prompt (Figure 8), the guideline “grow the dimension over time” refers to the well-known strategy of expanding the subspace dimensionality, which is used in BAxUS, Bounce, and BOIDS. Theoretical analyses identify this expansion as a requirement for subspace embedding methods to converge to the global optimum. The guideline “to jump to a much higher dimension after stagnation” is based on the empirical settings of these methods. It helps avoid spending too much of the evaluation budget on subspaces that may not contain the global optimum. Finally, the guidance “to prefer K=100” near the end refers to the general recommendation to prefer K=DK=D, where DD is the problem dimensionality. This guidance is also based on theoretical analyses of subspace embedding, which encourages the method to reach the full dimensionality as iterations grow for global convergence.

Second, our guidance does not make the LLMs deterministic, whereas BAxUS uses fixed heuristics. For example, when the subspace is updated, the LLM still needs to determine the new subspace dimensionality KK and decide when to set K=DK=D. Figures 19 and 20 show the subspace dimensionality choices of SageLLM-BO and BAxUS, respectively. The figures show that SageLLM-BO selects different dimensionalities across seeds, while BAxUS follows a deterministic expansion sequence that multiplies the next dimensionality by 4. For example, on a 100D problem, BAxUS always chooses dP=K=2,8,32,96,100d_\mathcal{P}=K=\\{2,8,32,96,100\\}, while SageLLM-BO can choose dP=K=2,51,88,100d_\mathcal{P}=K=\\{2,51,88,100\\} in some seeds. Moreover, the iteration at which SageLLM-BO sets K=DK=D varies considerably across seeds and problems, spanning around 500 iterations, with most runs reaching full dimensionality between iterations 250 and 750. For BAxUS, the spread is around 100 iterations, and most runs reach full dimensionality before iteration 300. These results show that the LLM makes decisions adaptively, not just following a fixed deterministic rule as in prior work.

Third, to confirm that the LLM contribute to the performance gains of SageLLM-BO via its reasoning, following the reviewer’s Question 2, we evaluated a non-LLM policy called “Prompt Rule,” which is based on the written rules in the prompts (Appendix I.3). In this variant, at each subspace expansion, we generate the set of example subspaces CSS\mathbb{C}_{SS} and select the example that maximizes the acquisition value indicator. The trust region shrinkage and expansion ratios are 0.5 and 2.0, respectively. The success threshold is set to τs=3\tau_s=3, and the failure threshold follows the rule used in BAxUS. Figure 28 compares SageLLM-BO with this variant. The result shows that SageLLM-BO outperforms the “Prompt Rule” variant on all 6 benchmarks, indicating that the non-LLM policy derived from the written rules cannot achieve the performance gains of SageLLM-BO.

AuthorsOpenReview ↗

Response to Reviewer qPRX

  1. The example responses in Appendix H look a lot like the defaults the method claims to replace. The LLM picks a failure threshold of 26, which equals the subspace dimension and so matches TuRBO's default. It keeps the success threshold at its initial value of 3. It shrinks the trust region by 0.6, compared with TuRBO's 0.5. It picks a subspace dimension of 26, which is one of the example sizes it was offered.

We would like to emphasize that the LLM’s decisions differ substantially from the defaults in TuRBO and BAxUS. We added the full logs of the LLM’s decisions to Appendix I to support this comparison.

For the failure threshold (Figure 25), TuRBO and BAxUS use a fixed rule based on the subspace dimensionality, while SageLLM-BO’s threshold tends to increase before the first subspace expansion and decrease within each subsequent subspace period. For example, for seed 1 on Ackley-100D, τf=1,2\tau_f=\\{1,2\\} before the first subspace expansion, τf=26,24,22,20,18,16,14,12\tau_f=\\{26,24,22,20,18,16,14,12\\} after the first expansion, and τf=63,54,48,43,38,34,30,27\tau_f=\\{63,54,48,43,38,34,30,27\\} after the second expansion. On the same problem, TuRBO sets τf=100\tau_f=100, and BAxUS sets τf\tau_f to 1, 4, and 14 in the respective subspace periods. SageLLM-BO uses success thresholds of 2 or 3, while TuRBO always uses 3. The shrinkage ratio (Figure 23) ranges from 0.3 to 0.7, with a median of 0.6 across the six benchmarks. Depending on the benchmark, 50-70% of the ratios are higher than the default of 0.5 used by TuRBO and BAxUS. One possible explanation for the small difference between the median and the default is that the prompt explanation “0.5 halves the trust region” gave the LLM a reference point for adapting the shrinkage ratio during optimization. Other differences in decisions can be found in the response to Question 1.

Regarding the comment “It picks a subspace dimension of 26, which is one of the example sizes it was offered,” our logs in Appendix I.1 show that the LLM always selects one of the dimensionalities in the set of example subspaces CSS\mathbb C_{SS}. We argue that, with access to only a summary of the observed data, it is better for the LLM to rely on the explicit quality estimates provided from the example subspaces, rather than its own predicted numerical estimates, when choosing a dimensionality.

  1. The paper never reports what the LLM decides across runs. It doesn't show how often the chosen subspace dimension jumps to 100, which would effectively switch the method to full-dimensional trust-region BO. It also doesn't show how far the shrink ratios and thresholds stray from TuRBO and BAxUS defaults, or how often the weights get changed. The ablations don't settle the question either. They swap each LLM component for a weaker alternative, but none tests a non-LLM policy that follows the same written rules. Without that, I can't tell whether the gains come from LLM reasoning or from good choices built into the candidate set and the prompt.

We added the LLM decision logs for all runs on the 6 benchmarks to Appendix I, including the subspace dimensionality in Figure 19, the shrinkage ratio in Figure 23, the failure threshold in Figure 25, and the weight changes in Figure 27. The results show that the LLM’s choices vary across problems and seeds and differ substantially from the defaults used by TuRBO and BAxUS.

Regarding how often the LLM chooses K=DK=D, Figure 19 shows that SageLLM-BO never sets K=DK=D in some runs (e.g., seeds 7 and 8 on LassoDNA-180D and seed 3 on HalfCheetah-102D) and sets K=DK=D very late in some other runs (after iteration 700 in some runs on HalfCheetah-102D and Pusher-161D). BAxUS on the other hand always sets K=DK=D before iteration 300. For the shrinkage ratios and failure thresholds, Figures 23 and 25 show how SageLLM-BO’s decisions differ from the defaults used by TuRBO and BAxUS. The specific ratios and thresholds can be found in our response to Weakness 2. For the weight changes, Figure 27 shows that each update changes the weights of 7-10% of the subspace dimensions. BAxUS and TuRBO do not update trust region weights. Other differences in decisions can be found in the response to Question 1.

Regarding the original ablations studies in Section 4.3, they are an important analysis showing the contribution of the subspace embedding and trust region LLMs separately. Nevertheless, following the reviewer’s suggestion, we have evaluated whether the gains of SageLLM-BO can be reproduced by a non-LLM policy that follows similar written rules. As mentioned in our response to Weakness 1, we added the “Prompt Rule” variant in Appendix I.3 following the reviewer’s Question 2. Figure 28 compares SageLLM-BO with the “Prompt Rule” variant. The results show that SageLLM-BO outperforms “Prompt Rule” on all 6 benchmarks, indicating that the non-LLM policy derived from the written rules does not reproduce the performance gains of the LLM’s reasoning. Please refer to our response to Weakness 1 for the full details.

AuthorsOpenReview ↗

Response to Reviewer qPRX (cont. 2)

  1. The benchmarks and baselines may favor this kind of method. All three synthetic functions have their optimum at or near the center of the domain. Ackley and Rastrigin have it at the origin, and Rosenbrock has it at the all-ones point inside a box centered on the origin. The sparse ±1 linear embedding used as the default fill-in maps p = 0 to x = 0, so every such subspace contains the origin. Recent work has shown that this kind of structure can make high-dimensional BO methods look much better than they are on general problems. Recent work has also shown that plain BO with suitable lengthscale priors is competitive with or better than many specialized high-dimensional methods. Two examples are Hvarfner et al. (ICML 2024), "Vanilla Bayesian Optimization Performs Great in High Dimensions", and Xu et al. (ICLR 2025), "Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization". The "Standard BO" baseline here performs poorly, and the paper doesn't describe how it was configured. So I can't tell whether it reflects these newer findings.

We agree that optima at or near the center of the domain can favor BAxUS’s linear embeddings and, consequently, our proposed method. To assess the effect of this benchmark structure, we evaluated shifted versions of the synthetic functions in Appendix M. In particular, we generated shifted versions of Ackley-100D, Rastrigin-100D, and Rosenbrock-100D following Xu et al. (2025b). Given the limited time available during the rebuttal, we compared SageLLM-BO with 3 main baselines (BAxUS, TuRBO, and BOIDS). Figure 31 shows that SageLLM-BO performs similarly to TuRBO and BAxUS on 2 out of 3 problems and worse on 1 out of 3. It performs similarly to BOIDS on 1 out of 3 problems and worse on 2 out of 3. We will explicitly acknowledge this limitation of SageLLM-BO, which is inherited from BAxUS’s linear embedding, in the Limitations section. However, it is worth noting that, on the 3 real-world benchmarks, where the optima are unknown, SageLLM-BO outperforms the baselines (in Section 4 of the original manuscript). Figure 2 shows that SageLLM-BO outperforms all 13 baselines on HalfCheetah-102D and Pusher-161D. On LassoDNA-180D, SageLLM-BO converges faster over the first 200 evaluations and achieves comparable final values. Moreover, to broaden our comparisons, we followed Reviewer Ljxf’s suggestion and evaluated the original synthetic functions (Ackley, Rastrigin, and Rosenbrock) at 20D and 1000D against TuRBO and BAxUS, as reported in Appendix L. Figure 30 shows that SageLLM-BO generally outperforms TuRBO and BAxUS, especially at 1000D. Although these additional synthetic benchmarks do not have shifted optima, SageLLM-BO’s better performance than BAxUS, which benefits from the same benchmark structure, highlights SageLLM-BO’s effectiveness across dimensionalities.

We have added a new paragraph in Appendix D.2 of the revised manuscript to clarify the “Standard BO” baseline, which uses a GP with a Matern-5/2 kernel and Thompson Sampling acquisition function (similar to TuRBO’s surrogate model and acquisition function). We also added the plain BO method with suitable lengthscale priors mentioned by the reviewer. In particular, we added D-Scaled-BO (Hvarfner et al., 2024) to the main experiments in Section 4. Following Reviewer Ljxf’s suggestion, we also added LinearBO (Doumont et al., 2026), which shares a similar idea. Figure 2 shows that SageLLM-BO outperforms both new baselines on 5 out of 6 benchmarks. On LassoDNA-180D, both new baselines achieve final objective values similar to those of SageLLM-BO, while SageLLM-BO converges faster over the first 200 evaluations.

Reviewer_vZPBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Bayesian optimization method that uses LLMs for high dimensional settings. The proposed method is token cost efficient. In the proposed method, a set of subspaces are generated. Using them, LLMs output a subspace using several optimization indicators, such as acquisition function values. LLMs are also used for guiding trust region. The proposed method is compared with existing subspace-based and LLM-based Bayesian optimization methods, and the experimental results show that the proposed method achieves better optimization performance with lower LLM inference costs.

Critical strengths

The token cost efficient algorithm is useful for LLM-based Bayesian optimization methods.

The proposed method achieves good Bayesian optimization performance in the experiments.

Critical weaknesses

The technical contributions are limited. The components of the proposed method are used in the existing methods.

It is not well justified why the optimization indicators used for finding subspaces and trust regions are effective for LLMs. Although the authors describe that prior works often rely on problem-specific heuristics, the proposed method seems also to rely on heuristics for the optimization indicators.

There are many hyperparameters in the proposed method.

Questions for authors

How the hyperparameters in the proposed method can be determined?

5 comments
AuthorsOpenReview ↗

Response to Reviewer vZPB

We thank the reviewer for the insightful review. We address the questions and comments as follows:

Weakness

The technical contributions are limited. The components of the proposed method are used in the existing methods.

We strongly believe that our technical contribution is substantial. The two key components of our proposed method, namely subspace embedding and trust regions, are two major research directions in high-dimensional BO. Several studies published in top-tier research venues have explored these directions to improve high-dimensional BO performance (Eriksson et al., NeurIPS 2019, Wang et al., NeurIPS 2020, Papenmeier et al., NeurIPS 2022, Papenmeier et al., NeurIPS 2023, and Ngo et al., AAAI 2025). In this work, we are the first to develop an LLM-guided decision-making framework that can determine how to construct the embedded subspace and the trust region during optimization. Our framework overcomes a limitation of existing methods that rely on manually designed heuristics for these search geometries. Our experimental results show that the proposed framework outperforms or matches 13 state-of-the-art high-dimensional BO baselines, demonstrating the effectiveness of SageLLM-BO. Developing an effective LLM-based framework that guides both decisions during optimization is not a trivial technical contribution.

Another key contribution of our work is enabling LLM-based BO to handle high-dimensional optimization problems while maintaining low LLM inference costs. Existing LLM-based BO methods struggle to operate in high-dimensional settings due to their prohibitively high LLM inference costs. In this work, we propose a novel methodology that summarizes the optimization state using informative indicators and provides the LLM with relevant information to guide the optimization process effectively while keeping the cost manageable. Table 1 in Appendix F.1 of the original manuscript shows the costs of SageLLM-BO and two state-of-the-art LLM-based BO methods (LLAMBO and LMABO). The results show that, with GPT-4.1, SageLLM-BO costs USD 0.60-0.80 per experiment (1000 iterations each). LLAMBO costs USD 10-23 per experiment (only 50 evaluations each), or 16-30 times as much as SageLLM-BO. LMABO costs about USD 2.50 per experiment (1000 iterations each), or 3-4 times as much as SageLLM-BO.

It is not well justified why the optimization indicators used for finding subspaces and trust regions are effective for LLMs. Although the authors describe that prior works often rely on problem-specific heuristics, the proposed method seems also to rely on heuristics for the optimization indicators.

We added an ablation study in Appendix K to show that the optimization indicators are effective for the LLM. In particular, we construct two variants that each remove a different set of optimization indicators from the prompts and compare their performance with that of SageLLM-BO. The “No Dimension Statistics” variant removes all dimension statistics (STR\mathbb S_{TR} and SSS\mathbb S_{SS}) from the subspace embedding and trust region prompts. The “No History Information” variant removes the corresponding history information (HTR\mathbb H_{TR} and HSS\mathbb H_{SS}) from both prompts. Figure 29 (Appendix K of the revised manuscript) compares full method (SageLLM-BO) with the 2 variants, showing that the full method performs better than both variants on 3 out of 6 problems and performs comparably on the remaining problems. This suggests that both types of information contribute meaningfully to the performance of SageLLM-BO.

Regarding the heuristics for the optimization indicators, we argue that our heuristics are not problem-specific as in prior works. For example, BAxUS and Bounce share the same linear subspace embedding strategy and hyperrectangular trust region mechanism, yet their heuristics for the trust region shrinkage ratio and failure threshold vary across benchmark problems. Moreover, we argue that the heuristics for the indicators are necessary because they ensure that the LLM receives information relevant to each decision while keeping the prompts compact. Including too much information, such as raw observations or the full search history, would make the prompts prohibitively long and increase inference costs.

AuthorsOpenReview ↗

Response to Reviewer vZPB (cont)

There are many hyperparameters in the proposed method.

Apart from the choice of LLM, SageLLM-BO has only two main hyperparameters: mm, which controls the number of example subspaces (Section 3.1.1), and nwn_w, which specifies the number of iterations between trust region weight updates (Section 3.2.2). In the original manuscript, we already reported sensitivity analyses for these two hyperparameters and the choice of LLM (Section 4.3 and Appendix F). The results show that SageLLM-BO is generally robust across the settings tested. Other design choices include the Matern-5/2 GP kernel and the Thompson Sampling acquisition function, both of which are standard choices in BO. We chose this kernel and acquisition function to ensure a fair comparison with the main baselines, TuRBO and BAxUS.

Questions

How the hyperparameters in the proposed method can be determined?

We recommend using the default values nw=10n_w=10 and m=3m=3. The sensitivity analyses in Appendix F of the original manuscript show that SageLLM-BO’s performance is generally robust to changes in these hyperparameters across the settings tested.

Conclusion

We believe we have addressed all the reviewer’s concerns. We kindly ask the reviewer to consider raising the overall score.

Dear Reviewer vZPB,

As the discussion phase comes to an end, please acknowledge the author's response and consider leaving them a comment on whether it has resolved some or all of your concerns.

Thanks!

Reviewer_LjxfOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SageLLM-BO for high-dimensional Bayesian optimization problems, using LLMs to adapt the search regions. The main idea is to use the LLM to select a low-dimensional subspace and to adapt the trust region based on summaries of the optimization history. Different elements of the trust region method are considered, including the number of dimensions, the particular dimensions to be included in the subspace, how to fill unimportant variables, and updating the trust region's size. SageLLM-BO is evaluated on three synthetic and three real-world problems from 100-180 dimensions. Ablation studies and sensitivity analysis are provided for the components of the algorithm.

Critical strengths
  1. The paper makes reasonable design choices in using LLMs to make high-level decisions about the search space, which is helpful in reducing the token consumption when LLMs are used for high-dimensional BO. In addition, subspace selection and trust region adaptation are conceptually clear together, and the incorporation of LLMs seems intuitive.

  2. The empirical results are promising on the benchmarks, demonstrating the effectiveness of SageLLM-BO, while LLM inference cost has been provided in comparison with other LLM-based BO methods, showing a clear reduction on average.

  3. The paper provides some ablation studies and sensitivity analysis regarding many of its design choices. The robustness to these choices strengthens the empirical superiority of the approach.

Critical weaknesses
  1. The paper does not attempt to ablate the use of LLMs in its method. For example, Ngo et al. (2026) observes the behaviors of the LLM in LMABO throughout multiple optimization processes to extract common heuristics that can potentially replace the LLM, then make the comparison between using these heuristics with using the LLM. Such a study not only reveals the mechanisms behind the LLM's decision making but also explores the possibility of a carefully designed policy without LLMs.

  2. The paper misses many other directions in high-dimensional BO. It only covers subspace embedding and trust region approaches, while other methods exist: additive structure (Han et al., 2021; Rolland et al., 2018), proximity of the optimum to the search space center (Oh et al., 2018), adjusting complexity assumption (Hvarfner et al., 2024), and linear models (Doumont et al., 2026). Therefore, section 2.2 (Related Works) is not comprehensive, and the baseline suite for comparison is also lacking.

Conditioned on the addition of these baselines, I will revise my rating accordingly.

Han, E., Arora, I., and Scarlett, J. (2021). High-dimensional Bayesian optimization via treestructured additive models. In Proceedings of the AAAI Conference on Artificial Intelligence.

Hvarfner, C., Hellsten, E. O., and Nardi, L. (2024). Vanilla Bayesian optimization performs great in high dimensions. In International Conference on Machine Learning.

Doumont, C., Fan, D., Maus, N., Gardner, J. R., Moss, H., & Pleiss, G. (2026). We Still Don\textquoterightt Understand High-Dimensional Bayesian Optimization. The 29th International Conference on Artificial Intelligence and Statistics.

Oh, C., Gavves, E., and Welling, M. (2018). BOCK: Bayesian optimization with cylindrical kernels. In International Conference on Machine Learning.

Rolland, P., Scarlett, J., Bogunovic, I., and Cevher, V. (2018). High-dimensional Bayesian optimization via additive models with overlapping groups. In International Conference on Artificial Intelligence and Statistics.

  1. The experiment setup is still limited.

Firstly, the benchmark problems only have dimensionality between 100 and 180. Without additional dimensionality scales, we cannot determine the scalability limit of the method. Note that many other high-dimensional works have demonstrated performances on thousands of dimensions

Secondly, the ablation study is still lacking. There are no results about subtle changes in the prompt, different choices of acquisition functions, or different choices of kernels.

Finally, no statistical tests was provided to support the claim in the paper.

Questions for authors
  1. Have the authors conducted any analysis on the decisions made by LLMs to identify common patterns that can potentially lead to a learned policy that replicates what the LLM does?

  2. How does the performance of SageLLM-BO change when the problem is 20D or 1000D?

  3. Is there a reason that SageLLM-BO does not consider multiple trust regions at the same time like TuRBO? If it does, can the LLM assist the respawn of new regions in a structured way like in Ngo et al. (2026)?

Ngo, G., Trong, D. P., Nguyen, D., & Gupta, S. (2026). High-dimensional Level Set Estimation with Trust Regions and Double Acquisition Functions. The 29th International Conference on Artificial Intelligence and Statistics.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Ljxf

We thank the reviewer for the insightful review. We address the questions and comments as follows:

Weaknesses

  1. The paper does not attempt to ablate the use of LLMs in its method. For example, Ngo et al. (2026) observes the behaviors of the LLM in LMABO throughout multiple optimization processes to extract common heuristics that can potentially replace the LLM, then make the comparison between using these heuristics with using the LLM. Such a study not only reveals the mechanisms behind the LLM's decision making but also explores the possibility of a carefully designed policy without LLMs.

We would like to clarify that we included an ablation study of the LLM components in Section 4.3 of the original manuscript. In particular, we isolate the effect of each LLM component (Subspace Embedding and Trust Region) by replacing one component at a time with a heuristic. Figure 3 compares SageLLM-BO with the ablation variants. SageLLM-BO outperforms the “No Subspace” variant on 6 out of 6 problems, the “AcqF Subspace” variant on 3, the “Random Subspace” variant on 2, the “No Trust Region” variant on 4, and the “TuRBO Trust Region” variant on 6. It performs similarly to each variant on the remaining problems. These results highlight the contribution of both components.

Nevertheless, we also followed the reviewer’s suggestion to observe the LLM’s behavior across multiple optimization runs, extract common heuristics, and test whether the resulting policy can replace the LLM entirely. All decisions made by the LLM across the optimization runs are summarized in Appendices I.1 and I.2 of the revised manuscript. We then used recurring patterns to design a non-LLM policy, which we call the “Extracted Rule” variant and describe in Appendix I.3. In particular, for the subspace update, the policy multiplies the subspace dimensionality by 4 and uses the linear fill-in strategy. For the trust region update, the shrinkage and expansion ratios are 0.6 and 2.0, respectively. The success threshold is set to τs=3\tau_s=3. The failure threshold increases linearly before the first subspace expansion and decreases linearly afterward. Every nw=10n_w=10 iterations, the policy leaves the trust region weights unchanged with 1010\\% probability. Otherwise, it randomly selects approximately 77\\% of the active subspace dimensions and independently samples each replacement weight from a uniform distribution over [0.4390,1.0840][0.4390,1.0840]. Figure 28 compares SageLLM-BO with the “Extracted Rule” variant on the same 6 problems, showing that SageLLM-BO outperforms the “Extracted Rule” variant on 4 out of 6 problems and performs similarly on 1 out of 6. These results indicate that while a policy based on recurring LLM decisions can be competitive on few problems, SageLLM-BO performs better overall.

AuthorsOpenReview ↗

Response to Reviewer Ljxf (cont.)

  1. The paper misses many other directions in high-dimensional BO. It only covers subspace embedding and trust region approaches, while other methods exist: additive structure (Han et al., 2021; Rolland et al., 2018), proximity of the optimum to the search space center (Oh et al., 2018), adjusting complexity assumption (Hvarfner et al., 2024), and linear models (Doumont et al., 2026). Therefore, section 2.2 (Related Works) is not comprehensive, and the baseline suite for comparison is also lacking.

Conditioned on the addition of these baselines, I will revise my rating accordingly.

Han, E., Arora, I., and Scarlett, J. (2021). High-dimensional Bayesian optimization via treestructured additive models. In Proceedings of the AAAI Conference on Artificial Intelligence.

Hvarfner, C., Hellsten, E. O., and Nardi, L. (2024). Vanilla Bayesian optimization performs great in high dimensions. In International Conference on Machine Learning.

Doumont, C., Fan, D., Maus, N., Gardner, J. R., Moss, H., & Pleiss, G. (2026). We Still Dont Understand High-Dimensional Bayesian Optimization. The 29th International Conference on Artificial Intelligence and Statistics.

Oh, C., Gavves, E., and Welling, M. (2018). BOCK: Bayesian optimization with cylindrical kernels. In International Conference on Machine Learning.

Rolland, P., Scarlett, J., Bogunovic, I., and Cevher, V. (2018). High-dimensional Bayesian optimization via additive models with overlapping groups. In International Conference on Artificial Intelligence and Statistics.

We clarify that we did not focus on other high-dimensional BO directions because our work focuses on the high-dimensional BO direction that defines search geometry using trust regions and subspace embeddings. It is worth noting that our original evaluation compared SageLLM-BO against up to 11 state-of-the-art methods, and Reviewer qPRX noted the breadth of this comparison. Nevertheless, we agree that including methods from other directions would strengthen the evaluation. In the revised manuscript, we extended Section 2.2 to discuss the other directions in high-dimensional BO that the reviewer highlighted, including effective surrogate models (Hvarfner et al., 2024 and Doumont et al., 2026), additive decomposition (Rolland et al., 2018 and Han et al., 2021), and the proximity of the optimum to the search-space center (Oh et al., 2018). We also reclassified RDUCB under additive decomposition.

In addition, we have further strengthened our baseline comparison by adding the two most recent baseline methods suggested by the reviewer, namely D-Scaled-BO (Hvarfner et al., 2024) and LinearBO (Doumont et al., 2026), bringing the total number of state-of-the-art baseline methods in our experimental evaluation to 13. Please note that, due to the limited time available during the rebuttal phase, we prioritized these two most recent baselines. For the other baselines mentioned by the reviewer, prior studies have already reported that RDUCB (one of our baselines) empirically outperforms the methods of Rolland et al. (2018) and Han et al. (2021), while TuRBO (also one of our baselines) empirically outperforms BOCK (Oh et al., 2018). As shown in Figure 2 of the revised manuscript, SageLLM-BO significantly outperforms D-Scaled-BO and LinearBO on 5 out of 6 problems and achieves comparable final performance on the remaining problem, on which SageLLM-BO converges faster during the first 200 evaluations while other methods require more than 700 iterations. These results further highlight the robustness and effectiveness of SageLLM-BO.

AuthorsOpenReview ↗

Response to Reviewer Ljxf (cont. 3)

  1. The experiment setup is still limited. Firstly, the benchmark problems only have dimensionality between 100 and 180. Without additional dimensionality scales, we cannot determine the scalability limit of the method. Note that many other high-dimensional works have demonstrated performances on thousands of dimensions. Secondly, the ablation study is still lacking. There are no results about subtle changes in the prompt, different choices of acquisition functions, or different choices of kernels. Finally, no statistical tests was provided to support the claim in the paper.

First, the reason we used these benchmark problems (especially the real-world ones) is because they are widely used in prior work on high-dimensional BO (CMA-BO, BAxUS, RDUCB, D-Scaled-BO, BOIDS, and LassoBO). Nevertheless, we now assess scalability beyond the original 100-180D benchmarks. Following the reviewer’s Question 2, we extended our original synthetic benchmark problems (Ackley, Rosenbrock, and Rastrigin) to 20 and 1000 dimensions in Appendix L. Given the limited time available during the rebuttal, we focused on 3 main baselines, including BAxUS, which represents the state-of-the-art subspace embedding method, TuRBO, which represents the state-of-the-art trust region method, and BOIDS, which shows the best performance among the baselines in our evaluations in Figure 2. Figure 30 (in Appendix L of the revised manuscript) compares SageLLM-BO with the 3 baselines. Compared with TuRBO, SageLLM-BO performs better on 6 out of 6 problems. Compared with BAxUS, SageLLM-BO performs better on 4 problems, similarly on 1, and worse on 1. Compared with BOIDS, it performs better on 4 problems and similarly on 2. These results indicate the scalability of SageLLM-BO to 20 and 1000 dimensions.

Second, to assess how subtle changes in prompt wording affect optimization, we added an ablation study on sensitivity to changes in LLM prompt in Appendix K. In particular, we formulated the “Subtle Wording” variant, in which we rephrased the instruction sentence in each prompt while preserving its overall meaning. For example, in the subspace embedding prompt, we changed “Use this information to reason and propose a new subspace that improves the optimization performance” to “Analyze the information provided to propose a new subspace that could improve optimization performance”. Table 5 lists the specific changes to all prompts. Figure 29 compares SageLLM-BO with this variant. The result shows that the “Subtle Wording” variant performs similarly to SageLLM-BO across all benchmark problems, indicating robustness to the wording changes tested. For the acquisition function and kernel choices, we use the same acquisition function (Thompson Sampling) as TuRBO and BAxUS and the same GP configuration (Matern 5/2 kernel) as TuRBO and BAxUS to ensure a fair comparison with these main baselines.

Third, we have now included statistical tests to further support the claim in the paper. In particular, we added Appendix J to report the results of the Wilcoxon rank-sum tests (α=0.05\alpha=0.05). Tables 3 and 4 show that, across the 96 comparisons with all baselines, SageLLM-BO performs significantly better in 82 comparisons, shows no statistically significant difference in 14, and performs significantly worse in none. These results support the claim that SageLLM-BO performs comparably or outperforms all baselines.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response & Changes to The Revised Manuscript

We thank the reviewers for the insightful reviews. We have updated our manuscript to clearly reflect the reviewers’ concerns. We list the following changes to the revised manuscript. We also provide a pdf highlighting the changes: https://anonymous.4open.science/r/sagellm-bo/comparison.pdf

  • We expanded Section 2.2 to cover additional high-dimensional BO methods (Reviewer Ljxf).
  • We updated Section 4 and Figure 2 to add D-Scaled-BO and LinearBO as baselines (Reviewers Ljxf and qPRX).
  • We expanded Appendix D.2 to clarify the Standard BO configuration and document the two new baselines (Reviewer qPRX).
  • We updated Appendix G to report runtimes for D-Scaled-BO and LinearBO.
  • We added Appendix I to report the logs of LLM’s decisions across runs and compare SageLLM-BO with 2 non-LLM policies, the “Prompt Rule” variant, which follows written rules in our prompts, and the “Extracted Rule” variant, which follows recurring patterns from LLM logs (Reviewers Ljxf and qPRX).
  • We added Appendix J to report final objective values, standard errors and statistical tests (Reviewers Ljxf and qPRX).
  • We added Appendix K to test sensitivity to small wording changes (the “Subtle Wording” variant), the removal of dimension statistics (the “No Dimension Statistics” variant), and the removal of history information (the “No History Information” variant) (Reviewers Ljxf, vZPB and qPRX)
  • We added Appendix L to evaluate SageLLM-BO at 20 and 1000 dimensions (Reviewer Ljxf).
  • We added Appendix M to evaluate SageLLM-BO on shifted synthetic benchmarks (Reviewer qPRX).
  • We added Appendix N to measure variability with a fixed optimization seed when the LLM’s temperature is zero (Reviewer qPRX).

We believe these changes and our responses have addressed all reviewers’ concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.