- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SemanticBBOBench, a benchmark comprising 42 black-box optimization tasks across scientific, engineering, and economic domains, each paired with structured semantic context at different levels of detail. The experiments evaluate conventional numerical optimizers, directly prompted LLM optimizers, and a hybrid approach that uses LLM-generated initialization followed by HEBO. The authors also investigate the effects of different context levels and misleading semantic information. The results suggest that semantic information primarily benefits early-stage optimization, while incorrect contextual guidance can persistently degrade optimization performance.
Critical strengths
- Meaningful evaluation setting and comprehensive task interface. The paper addresses a relevant problem by incorporating natural-language domain knowledge into black-box optimization benchmarks. The proposed framework supports mixed-variable search spaces, infeasible evaluations, and auxiliary outputs through a unified interface. This provides a useful experimental setting for studying how semantic information influences optimization decisions beyond conventional numerical benchmark functions.
- Substantial validation of the surrogate evaluators. The authors evaluate surrogate accuracy not only globally but also in optimization-relevant regions. Furthermore, they rerun five optimization methods on the original simulators across 21 tasks to examine whether the surrogate-based evaluations preserve optimization performance and method rankings. These experiments provide meaningful evidence supporting the use of surrogate evaluators to reduce benchmarking costs.
Critical weaknesses
- Insufficient evaluation of representative existing semantic optimization methods. Although the paper discusses existing hybrid approaches such as LLAMBO, LLINBO, LGBO, and GOLLuM, none of these dedicated semantic optimization methods are included in the experiments. Fixing the LLM optimizer is reasonable for isolating the effects of different context conditions, but such ablations cannot replace evaluations of representative methods from the target research area. Consequently, the current experiments do not establish whether the observed advantages in initialization and difficulties in adapting to misleading priors are limitations of direct LLM prompting or broader challenges faced by existing semantic optimization approaches. The number and diversity of benchmark tasks alone cannot compensate for this gap. Evaluating representative hybrid methods on a compatible subset of tasks would help demonstrate the benchmark's value for comparing and developing optimization algorithms.
- The context comparisons do not fully isolate the effects of semantic information. The no-context condition not only removes semantic descriptions but also anonymizes variable names and normalizes numerical ranges. Meanwhile, the full-context condition introduces auxiliary observations in addition to textual descriptions. Therefore, performance differences across some context conditions may reflect changes in input representations or access to additional numerical feedback, rather than the effects of natural-language knowledge alone. Although these comparisons remain informative, the paper's central goal of isolating semantic information would benefit from more controlled comparisons in which numerical representations and auxiliary observations are held fixed while only semantic descriptions are varied.
- Limited availability of the complete benchmark artifacts. According to Appendix E, although the anonymized release includes task definitions, metadata, and construction pipelines for all 42 problems, complete surrogate artifacts are provided for only 10 of them. For a benchmark that uses fitted surrogates as its objective evaluators, providing reconstruction code and metadata is not equivalent to distributing the exact frozen evaluators used in the reported experiments. Similarly, reproducing figures from recorded results does not establish that new optimization methods can be evaluated on the complete benchmark. The authors should provide all frozen surrogate artifacts or clarify how the remaining artifacts can be obtained and what resources are required for reconstruction. Otherwise, the accessibility and reproducibility of the full benchmark remain limited.
Questions for authors
See Weaknesses
4 comments
Response to Specific Questions
We also hope to address your specific comments below.
Q1. Other LLM-based optimization methods
We have added LLAMBO, LLINBO-Transient, LLINBO-Justify, LGBO, LABO and GOLLuM on all 42 benchmarks. Overall results are available in Figure 9, with paired tests in Appendix E.2 and a summary here:
| Gemma 4 31B, full context | 10 evals | 50 evals |
|---|---|---|
| LGBO | 0.705 | 0.860 |
| HEBO GP LLM Warmup | 0.745 | 0.857 |
| Direct LLM | 0.745 | 0.838 |
| LLINBO-Transient | 0.745 | 0.830 |
| LLINBO-Justify | 0.740 | 0.825 |
| LABO | 0.689 | 0.806 |
| HEBO GP | 0.496 | 0.804 |
| LLAMBO | 0.718 | 0.790 |
| GOLLuM | 0.481 | 0.713 |
The results agree with the paper's warm-start finding: LLM-based optimization methods are strong at warm-starting but struggle with iterative performance. Combining Gaussian processes with LLM results like in LGBO can improve performance, but fails to significantly improve beyond a simple hybrid approach in our tests.
Misleading priors also affect the hybrid LLM and GP-based methods, which we show in Figure 10 and summarize below:
| True context | Adversarial | |
|---|---|---|
| LGBO | 0.860 | 0.738 |
| LLINBO-Transient | 0.830 | 0.698 |
| Direct LLM | 0.838 | 0.585 |
The GP reduces the impact, but both hybrid methods still end below plain HEBO GP.
Q2. Isolating the effects of naming and auxiliary observations
We added two controls based on your feedback. A names-only level gives the real parameter names, ranges and levels but no descriptions. Tests with both GPT-5.6 Luna and Gemma show that this is not significantly better than no context, while adding the descriptions brings most of the context gain. Second, removing the auxiliary observations from full context changes the score by less than 0.01 for both models, so the impact is mostly coming from the problem descriptions (Appendix E.3).
Q3. Release of the surrogate artifacts
Every task’s objective surrogate and failure classifier are now in the anonymous repository, so all 42 tasks can be run. The predictors of the auxiliary outputs exceed the anonymous host’s storage limit and will be released upon acceptance. The repository also includes the six LLM-guided BO methods, with a script that runs any method on all 42 tasks.
We hope these additions address your concerns, and we would be happy to answer any further questions.
Thank you to the authors for their detailed responses. I will raise my scores accordingly.
Thank you for raising your score. Your suggestions led to the evaluation of existing LLM-guided BO methods and additional context controls, which made the benchmark more complete.
Update Overview
Thank you for your careful reading and constructive comments. We have run several new experiments and revised the paper based on your useful feedback. All new or changed text is marked in blue within the updated paper.
Key new experiments:
Key claims: