acceptodds
Under review as a conference paper at ICLR 2027

SemanticBBOBench: A Benchmark for Semantic Black-Box Optimization with Large Language Models

Abstract

Black-box optimization is central to many scientific and engineering disciplines. While numerical methods such as Bayesian optimization (BO) can successfully optimize black-box problems, these methods struggle to incorporate the semantic context that often accompanies real-world experimental problems, including domain knowledge, prior experiments, and expert intuition. Using this information could substantially reduce experimental costs and improve outcomes. Large language models (LLMs) can reason over both semantic and numerical data, offering a pathway for bringing this context into the optimization loop. We present SemanticBBOBench, a black-box optimization benchmark designed for testing LLM-based methods with semantic context. It provides a uniform framework containing 42 optimization problems spanning diverse scientific and engineering domains, each paired with a curated context at multiple controlled levels of detail and types of information. Our experiments show that semantic context provides its greatest benefit early in the optimization process, enabling substantially stronger initial configurations when few numerical observations are available. We also show that misleading semantic information can persistently bias LLM-based search. These results highlight both the potential and limitations of LLM-based black-box optimization and establish SemanticBBOBench as a framework for developing methods that use such information effectively.

Primary area:
datasets and benchmarks
Keywords:
Black-box optimization, large language models, Bayesian optimization, semantic optimization, optimization benchmarks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hPVLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SemanticBBOBench, a benchmark comprising 42 black-box optimization tasks across scientific, engineering, and economic domains, each paired with structured semantic context at different levels of detail. The experiments evaluate conventional numerical optimizers, directly prompted LLM optimizers, and a hybrid approach that uses LLM-generated initialization followed by HEBO. The authors also investigate the effects of different context levels and misleading semantic information. The results suggest that semantic information primarily benefits early-stage optimization, while incorrect contextual guidance can persistently degrade optimization performance.

Critical strengths
  1. Meaningful evaluation setting and comprehensive task interface. The paper addresses a relevant problem by incorporating natural-language domain knowledge into black-box optimization benchmarks. The proposed framework supports mixed-variable search spaces, infeasible evaluations, and auxiliary outputs through a unified interface. This provides a useful experimental setting for studying how semantic information influences optimization decisions beyond conventional numerical benchmark functions.
  2. Substantial validation of the surrogate evaluators. The authors evaluate surrogate accuracy not only globally but also in optimization-relevant regions. Furthermore, they rerun five optimization methods on the original simulators across 21 tasks to examine whether the surrogate-based evaluations preserve optimization performance and method rankings. These experiments provide meaningful evidence supporting the use of surrogate evaluators to reduce benchmarking costs.
Critical weaknesses
  1. Insufficient evaluation of representative existing semantic optimization methods. Although the paper discusses existing hybrid approaches such as LLAMBO, LLINBO, LGBO, and GOLLuM, none of these dedicated semantic optimization methods are included in the experiments. Fixing the LLM optimizer is reasonable for isolating the effects of different context conditions, but such ablations cannot replace evaluations of representative methods from the target research area. Consequently, the current experiments do not establish whether the observed advantages in initialization and difficulties in adapting to misleading priors are limitations of direct LLM prompting or broader challenges faced by existing semantic optimization approaches. The number and diversity of benchmark tasks alone cannot compensate for this gap. Evaluating representative hybrid methods on a compatible subset of tasks would help demonstrate the benchmark's value for comparing and developing optimization algorithms.
  2. The context comparisons do not fully isolate the effects of semantic information. The no-context condition not only removes semantic descriptions but also anonymizes variable names and normalizes numerical ranges. Meanwhile, the full-context condition introduces auxiliary observations in addition to textual descriptions. Therefore, performance differences across some context conditions may reflect changes in input representations or access to additional numerical feedback, rather than the effects of natural-language knowledge alone. Although these comparisons remain informative, the paper's central goal of isolating semantic information would benefit from more controlled comparisons in which numerical representations and auxiliary observations are held fixed while only semantic descriptions are varied.
  3. Limited availability of the complete benchmark artifacts. According to Appendix E, although the anonymized release includes task definitions, metadata, and construction pipelines for all 42 problems, complete surrogate artifacts are provided for only 10 of them. For a benchmark that uses fitted surrogates as its objective evaluators, providing reconstruction code and metadata is not equivalent to distributing the exact frozen evaluators used in the reported experiments. Similarly, reproducing figures from recorded results does not establish that new optimization methods can be evaluated on the complete benchmark. The authors should provide all frozen surrogate artifacts or clarify how the remaining artifacts can be obtained and what resources are required for reconstruction. Otherwise, the accessibility and reproducibility of the full benchmark remain limited.
Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Update Overview

Thank you for your careful reading and constructive comments. We have run several new experiments and revised the paper based on your useful feedback. All new or changed text is marked in blue within the updated paper.

Key new experiments:

  1. LLM-guided BO methods. We have added suggested LLM-based optimization methods, including LLAMBO, LLINBO-Transient, LLINBO-Justify, LGBO, LABO and GOLLuM on all 42 benchmarks. Overall results are available in Figure 9 with context type results in Figure 10.
  2. Further context levels. A names-only context level and a control for the auxiliary observations (Appendix E.3).
  3. Larger LLM suite. Claude Haiku 5.5 as a direct optimizer (Figure 5), and all context ablations repeated with Gemma 4 31B (Appendices E.3 and E.4).
  4. Longer budgets. 200 evaluations on 10 benchmarks (Appendix E.5).

Key claims:

  • The LLM's contribution. Tests over various LLM-augmented BO methods show no method significantly beats BO warm-started from the LLM's proposals, showing that LLM-based methods are good at warm-starting but remain poor iterative optimizers.
  • Misleading context. Methods that add a GP in the loop reduce, but do not remove, the damage from misleading context with LLM-based methods.
AuthorsOpenReview ↗

Response to Specific Questions

We also hope to address your specific comments below.

Q1. Other LLM-based optimization methods

We have added LLAMBO, LLINBO-Transient, LLINBO-Justify, LGBO, LABO and GOLLuM on all 42 benchmarks. Overall results are available in Figure 9, with paired tests in Appendix E.2 and a summary here:

Gemma 4 31B, full context10 evals50 evals
LGBO0.7050.860
HEBO GP LLM Warmup0.7450.857
Direct LLM0.7450.838
LLINBO-Transient0.7450.830
LLINBO-Justify0.7400.825
LABO0.6890.806
HEBO GP0.4960.804
LLAMBO0.7180.790
GOLLuM0.4810.713

The results agree with the paper's warm-start finding: LLM-based optimization methods are strong at warm-starting but struggle with iterative performance. Combining Gaussian processes with LLM results like in LGBO can improve performance, but fails to significantly improve beyond a simple hybrid approach in our tests.

Misleading priors also affect the hybrid LLM and GP-based methods, which we show in Figure 10 and summarize below:

True contextAdversarial
LGBO0.8600.738
LLINBO-Transient0.8300.698
Direct LLM0.8380.585

The GP reduces the impact, but both hybrid methods still end below plain HEBO GP.

Q2. Isolating the effects of naming and auxiliary observations

We added two controls based on your feedback. A names-only level gives the real parameter names, ranges and levels but no descriptions. Tests with both GPT-5.6 Luna and Gemma show that this is not significantly better than no context, while adding the descriptions brings most of the context gain. Second, removing the auxiliary observations from full context changes the score by less than 0.01 for both models, so the impact is mostly coming from the problem descriptions (Appendix E.3).

Q3. Release of the surrogate artifacts

Every task’s objective surrogate and failure classifier are now in the anonymous repository, so all 42 tasks can be run. The predictors of the auxiliary outputs exceed the anonymous host’s storage limit and will be released upon acceptance. The repository also includes the six LLM-guided BO methods, with a script that runs any method on all 42 tasks.

We hope these additions address your concerns, and we would be happy to answer any further questions.

Thank you to the authors for their detailed responses. I will raise my scores accordingly.

Thank you for raising your score. Your suggestions led to the evaluation of existing LLM-guided BO methods and additional context controls, which made the benchmark more complete.

Reviewer_orHTOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SemanticBBOBench contains 42 black-box optimization tasks with natural-language context. It varies how much information the optimizer receives and tests rewritten, misleading, and unrelated context. With 50 evaluations and five seeds per task, the authors compare two LLMs, numerical optimizers, and an LLM–HEBO hybrid. Context helps most early in the search, and the performance loss from misleading context remains after 50 evaluations. Starting with an LLM and then switching to HEBO gives the best average score among the tested methods. The benchmark uses learned surrogates to keep evaluations cheap, with additional checks against the original evaluators on 21 tasks.

Critical strengths
  1. A useful setup for studying semantic information. The benchmark pairs 42 tasks with several levels of context and includes tests with rewritten, misleading, and unrelated information. This lets researchers test how much context helps and how bad suggestions affect the search across a range of applications.

  2. The authors check whether surrogates preserve optimizer rankings. On 21 tasks, they rerun five optimizers with the original evaluators. The two evaluators agree on the best method on 18 tasks at both 10 and 50 evaluations, with mean rank correlations of 0.92 and 0.86. This supports using the cheaper surrogates to compare optimizers (Appendix D).

Critical weaknesses
  1. Only two LLMs are tested. The experiments use GPT-5.6 Luna and Gemma 4 31B, and most context ablations use Luna alone. For a benchmark aimed at LLM optimizers, this is a small set. There is little evidence that the findings hold across other model families, sizes, and capabilities.

  2. Few LLM-based optimization strategies are tested. Both models use the same prompting procedure to propose configurations. The hybrid switches to HEBO after ten evaluations. Other methods use semantic information differently: LLAMBO uses an LLM as a surrogate, LLINBO checks LLM proposals with a GP, and LGBO uses LLM preferences to guide BO. The paper cites these methods but does not test them.

  3. Only 50 evaluations are tested. All optimization experiments stop after 50 evaluations. The results do not show whether the gains from useful context fade with more evaluations, or whether the optimizer eventually recovers from misleading context. Longer runs, for example 100 or 200 evaluations, would help answer these questions.

  4. The contribution beyond existing work is unclear. 1) Context ablations are standard practice in LLM-based optimization, so the idea itself is not new. For example, LLAMBO compares no, partial, and full context. 2) SemanticOpt studies optimization across domains, and its Figure 8 tests domain knowledge, historical examples, and auxiliary feedback on selected tasks. The paper does not discuss SemanticOpt. It should explain what its benchmark adds beyond these studies and narrow the claim that earlier benchmarks do not control semantic information.

Questions for authors

Please address the three concerns raised in the Weaknesses section.

4 comments
AuthorsOpenReview ↗

Update Overview

Thank you for your careful reading and constructive comments. We have run several new experiments and revised the paper based on your useful feedback. All new or changed text is marked in blue within the updated paper.

Key new experiments:

  1. LLM-guided BO methods. We have added suggested LLM-based optimization methods, including LLAMBO, LLINBO-Transient, LLINBO-Justify, LGBO, LABO and GOLLuM on all 42 benchmarks. Overall results are available in Figure 9 with context type results in Figure 10.
  2. Further context levels. A names-only context level and a control for the auxiliary observations (Appendix E.3).
  3. Larger LLM suite. Claude Haiku 5.5 as a direct optimizer (Figure 5), and all context ablations repeated with Gemma 4 31B (Appendices E.3 and E.4).
  4. Longer budgets. 200 evaluations on 10 benchmarks (Appendix E.5).

Key claims:

  • The LLM's contribution. Tests over various LLM-augmented BO methods show no method significantly beats BO warm-started from the LLM's proposals, showing that LLM-based methods are good at warm-starting but remain poor iterative optimizers.
  • Misleading context. Methods that add a GP in the loop reduce, but do not remove, the damage from misleading context with LLM-based methods.
AuthorsOpenReview ↗

Response to Specific Questions

We also hope to address your specific comments below.

Q1. Additional LLMs and context ablations

We added Claude Haiku 5.5 as a direct optimizer (Figure 5) and repeated all context ablations with Gemma 4 31B, so the benchmark now covers three model families. The findings hold across them: all three LLMs are far ahead of HEBO GP after 10 evaluations but the gap narrows by 50 evaluations. We also tested Gemma over the context levels and rewrites, which follow the same ordering as with GPT-5.6 Luna in Figures 6 and 7 (Appendices E.3 and E.4).

Q2. Additional LLM-based optimization methods

We have added LLAMBO, LLINBO-Transient, LLINBO-Justify, LGBO, LABO and GOLLuM on all 42 benchmarks. Overall results are available in Figure 9, with paired tests in Appendix E.2 and a summary here:

Gemma 4 31B, full context10 evals50 evals
LGBO0.7050.860
HEBO GP LLM Warmup0.7450.857
Direct LLM0.7450.838
LLINBO-Transient0.7450.830
LLINBO-Justify0.7400.825
LABO0.6890.806
HEBO GP0.4960.804
LLAMBO0.7180.790
GOLLuM0.4810.713

The results agree with the paper's warm-start finding: LLM-based optimization methods are strong at warm-starting but struggle with iterative performance. Combining Gaussian processes with LLM results like in LGBO can improve performance, but fails to significantly improve beyond a simple hybrid approach in our tests.

Q3. Longer budgets and recovery from misleading context

Based on your advice, we test an evaluation budget of 200 on a subset of 10 benchmark problems in Appendix E.5, with the results summarized below:

Method50200
Direct LLM0.8730.877
HEBO GP0.7820.926
HEBO GP LLM Warmup0.8850.918
LGBO0.8900.926

The advantage of context does fade: the direct LLM stops improving after about 50 evaluations and HEBO GP overtakes it around evaluation 130. The direct LLM's advantage is therefore specific to small budgets.

Regarding recovery from misleading context, we test LGBO and LLINBO-Transient on our various context types in Figure 10. With these methods, keeping a GP in the loop roughly halves the loss from adversarial context, but both hybrids still end below plain HEBO GP. Misleading context is therefore a challenge for hybrid methods too.

Q4. Relation to existing work

We have clarified our contribution in the paper and below. LLAMBO ablates context on hyperparameter tuning to validate its own components: every level names the model and its hyperparameters, and the levels differ only in how many dataset statistics are added, all computed from the data and therefore correct. SemanticBBOBench instead separates kinds of information (what the variables mean, mechanisms and heuristics, historical configurations, and auxiliary measurements) across 42 tasks, and tests whether the context still helps when reworded, wrong, or about another problem.

The SemanticOpt arXiv preprint (2510.25404) assembles multiple tasks to evaluate its fine-tuned model, with context that differs by task. It does not hold the context types fixed across tasks. Additionally, the concurrent ICLR 2027 submission of SemanticOpt is evaluated on SemanticBBOBench instead, using the tasks, context levels and rewrites defined here. The two papers therefore make separate contributions: SemanticOpt is a fine-tuned optimizer, and SemanticBBOBench is a benchmark for evaluating optimizers like the one proposed in SemanticOpt and any other optimization method. With SemanticBBOBench, we provide a benchmark covering the following tests:

  • 42 tasks in six domains, with mixed search spaces, infeasible regions and validated surrogates.
  • The same nested context levels for every task.
  • Three rewrites per task (counterfactual, adversarial, mismatched) that make memorization and robustness testable.

We hope these additions address your concerns, and we would be happy to answer any further questions.

Reviewer_orHTOpenReview ↗

Final Justification

Thank you for the additional experiments and detailed responses. These address my concerns in Q1–Q3 and make the experimental evaluation substantially more complete. I will therefore raise my rating from 1 to 2.

For Q4, however, I retain my concern about the benchmark’s added contribution. The publicly available SemanticOpt arXiv preprint already presents related ideas, claims, and experiments on using different types of semantic information for black-box optimization. In my view, releasing those benchmark scenarios would be a natural extension of the SemanticOpt work and would make that work more complete and solid. For SemanticBBOBench to stand as a separate benchmark paper, I would expect a substantively distinct contribution beyond the ideas and experimental settings already presented in that preprint. The current clarification does not fully resolve this concern.

Thank you for raising your score and for your careful reading and analysis of our paper. On the overlap with the SemanticOpt preprint: none of its nine problems are among our 42 tasks, and it tests each context type on a single task, without rewritten, misleading or unrelated context. Holding the context levels and rewrites fixed across all tasks is what lets the benchmark compare LLM-based methods and measure the effect of each type of context across diverse problems. We believe this makes SemanticBBOBench a distinct contribution, as it shows where LLM-based black-box optimizers help and where they fall short, while providing a common test suite for improving them.

Reviewer_B9oAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proffers a 42-problem black-box optimization benchmark, pairing real-world tasks with controlled natural-language context at multiple levels. The key findings are that LLMs leveraging semantic context achieve strong early performance but plateau, that misleading context persistently harms search, and that an LLM to Bayesian optimization (HEBO) handoff outperforms either method alone.

Critical strengths

First, the paper addresses a real gap. The benchmark isolates semantic information's contribution to optimization, which is a question absent from existing benchmarks. The nested context design and multi-domain coverage are well-motivated and more realistic than single-problem studies. This enables structured evaluation of LLM-based optimization rather than only aggregate comparisons.

Second, the paper has diagnostic depth. Specifically, the paper includes context ablations, decomposition of domain knowledge versus historical examples, adversarial/counterfactual rewrites, per-domain results, and LLM–BO combination. The main finding is that semantic benefits are primarily warm-start, not improved iteration. Thus, this reveals a concrete limitation and points toward hybrid systems, more credible than overstated LLM claims.

Critical weaknesses

It seems that information asymmetry may limit the paper's central claims. Because there is no knowledge-informed BO baseline, the experimentation comparison shows LLMs can exploit privileged information, not necessarily that they're superior at using equivalent information. The paper's main point (i.e., semantic context beats numerical methods) may reflect information richness, not algorithmic advantage.

Questions for authors
  1. How much semantic advantage survives against BO methods enriched with the same auxiliary metrics and information (structured as priors, constraints, or warm starts)?
  2. What validation process was used for semantic annotations?
4 comments
AuthorsOpenReview ↗

Update Overview

Thank you for your careful reading and constructive comments. We have run several new experiments and revised the paper based on your useful feedback. All new or changed text is marked in blue within the updated paper.

Key new experiments:

  1. LLM-guided BO methods. We have added suggested LLM-based optimization methods, including LLAMBO, LLINBO-Transient, LLINBO-Justify, LGBO, LABO and GOLLuM on all 42 benchmarks. Overall results are available in Figure 9 with context type results in Figure 10.
  2. Further context levels. A names-only context level and a control for the auxiliary observations (Appendix E.3).
  3. Larger LLM suite. Claude Haiku 5.5 as a direct optimizer (Figure 5), and all context ablations repeated with Gemma 4 31B (Appendices E.3 and E.4).
  4. Longer budgets. 200 evaluations on 10 benchmarks (Appendix E.5).

Key claims:

  • The LLM's contribution. Tests over various LLM-augmented BO methods show no method significantly beats BO warm-started from the LLM's proposals, showing that LLM-based methods are good at warm-starting but remain poor iterative optimizers.
  • Misleading context. Methods that add a GP in the loop reduce, but do not remove, the damage from misleading context with LLM-based methods.
AuthorsOpenReview ↗

Response to Specific Questions

We also hope to address your specific questions below.

Q1. Comparison with LLM-augmented BO methods

We have added LLAMBO, LLINBO-Transient, LLINBO-Justify, LGBO, LABO and GOLLuM on all 42 benchmarks. These methods provide the same semantic context as the direct LLM method, providing a fair comparison and addressing the information asymmetry. Overall results are available in Figure 9, with paired tests in Appendix E.2 and a summary here:

Gemma 4 31B, full context10 evals50 evals
LGBO0.7050.860
HEBO GP LLM Warmup0.7450.857
Direct LLM0.7450.838
LLINBO-Transient0.7450.830
LLINBO-Justify0.7400.825
LABO0.6890.806
HEBO GP0.4960.804
LLAMBO0.7180.790
GOLLuM0.4810.713

The results agree with the paper's warm-start finding: LLM-based optimization methods are strong at warm-starting but struggle with iterative performance. Combining Gaussian processes with LLM results like in LGBO can improve performance, but fails to significantly improve beyond a simple hybrid approach in our tests.

Q2. Validation of the semantic annotations

We use Claude Cowork to draft the implementations and context from the documentation and publications behind the simulator, dataset or model used for the benchmark problem. We reviewed every context profile by hand against these sources, removing errors, unsupported claims and anything that would reveal the true optimum, with the goal of providing information that is useful but does not reveal the answer directly. Two experiments support this (Figure 7): a reworded (counterfactual) context performs nearly as well as the original, so the gain does not come from recognizing the text, and an adversarial rewrite of only the optimization-relevant claims lowers the score sharply, so the content itself is informative.

We hope these additions address your concerns, and we would be happy to answer any further questions.

Thank you for running the additional experimentation, thereby strengthening your paper, as well as answering my questions.

Thank you for your feedback and positive assessment. Your question on LLM-based baselines helped lead to the comparison with other LLM-guided BO methods, which strengthened the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.