- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces AgenticBBO-Bench, a cross-domain benchmark for evaluating LLM-based agents on black-box optimization (BBO) problems. The benchmark integrates five established task families, including synthetic optimization (BBOB), hyperparameter optimization (Bayesmark), database tuning (DBTune), chip placement (BBOPlace-Bench), and molecular design (GuacaMol), under a unified finite-budget evaluation protocol.
The authors compare agentic optimization against direct LLM-based candidate generation and conventional numerical optimizers. They further investigate the effects of numerical optimization tools, task semantics and prior knowledge, and different roles of LLMs in the optimization loop. Finally, the paper introduces a five-task frontier challenge to compare seven LLMs under a shared agent harness.
The results suggest that agentic optimization consistently improves over direct LLM candidate generation across the five domains, while additional numerical optimization tools do not necessarily improve agent performance. Task semantics can be beneficial, whereas richer domain priors have less consistent effects. Numerical optimizers can also sometimes effectively continue optimization after an agent-generated warm-up.
Critical strengths
-
The paper introduces AgenticBBO-Bench, integrating 77 tasks across five heterogeneous BBO domains under a unified evaluation protocol. The benchmark accommodates different search spaces, candidate representations, and feasibility constraints while maintaining consistent evaluation budgets and scoring rules. This provides useful infrastructure for standardized evaluation of general-purpose optimization agents.
-
Beyond introducing a benchmark, the paper investigates the effects of numerical optimization tools, task semantics and prior knowledge, and different roles of LLMs in the optimization loop. These controlled experiments provide useful insights into agentic optimization behavior, particularly that additional GP-based tools do not consistently improve performance and that numerical optimizers can sometimes effectively continue from agent-generated search trajectories.
Critical weaknesses
-
The benchmark is built upon five established BBO benchmarks (BBOB, Bayesmark, DBTune, BBOPlace-Bench, and GuacaMol). Much of the contribution lies in adapting these tasks to a shared agent interface and standardizing the evaluation protocol. While this integration is useful, it is less clear what new benchmarking challenges are addressed beyond making existing BBO tasks accessible to LLM agents. Similar ideas of unified interaction and evaluation have also been explored in the broader agent benchmarking literature. The paper would benefit from a clearer discussion of what makes this benchmark substantially different from existing BBO and agent benchmarks, and why these differences matter.
-
Although the full benchmark contains 77 tasks across five domains, the main diagnostic studies are conducted on much smaller subsets. Section 5.3 evaluates optimization tools on eight tasks, Section 5.4 studies task information on six real-world tasks (plus three controlled synthetic objectives), and Section 5.5 examines LLM roles on only four tasks from BBOB and HPO. This makes it difficult to tell how broadly the reported findings apply across different BBO problems. Furthermore, some of the takeaways, such as the benefits of task semantics and the potential harm of incorrect priors, are relatively intuitive. Given that the benchmark itself largely integrates existing tasks, I would expect the empirical study to provide stronger insights into when different agentic approaches work well or poorly. A more systematic analysis across problem characteristics, such as dimensionality and search-space structure, would make the findings more informative.
Questions for authors
-
Since the benchmark largely builds on existing BBO task suites and adopts a unified agent interaction protocol, could the authors clarify which aspects of its design go beyond adapting these tasks to an agent framework? A few concrete examples would help better establish the benchmark's contribution.
-
The analyses in Sections 5.3–5.5 use relatively small subsets of the full benchmark (8, 6, and 4 tasks, respectively). How were these tasks selected, and are they representative of the broader suite in terms of dimensionality, search-space structure, and optimization difficulty? It would be helpful to understand how much the findings depend on this selection.
-
Table 2(b) shows that adding GP tools helps on some tasks but hurts on others. Is there any observable relationship between these differences and the characteristics of the optimization problems? A more fine-grained analysis would make the experimental findings more useful for guiding future Agentic BBO research.
5 comments
W2 & Q2. Selection of diagnostic tasks
Thanks for your comments! We use the full 77-task benchmark for cross-domain evaluation and select smaller subsets for controlled studies in Sections 5.3–5.5 based on their research objectives.
Optimization tools (8 tasks). We select two tasks from each of four domains: BBOB F02/F15 (unimodal/multimodal continuous), Breast/SVM and Diabetes/RF (log-scaled continuous and mixed integer/float hyperparameter spaces), PostgreSQL-5 and Sysbench-5 (mixed integer/float and enum database knobs), and Adaptec1 and Bigblue1 (64D macro coordinates with constraint repair). These tasks cover continuous, mixed, and structured search spaces, with dimensions ranging from 3 to 64.
Task information (6 tasks). We use the HPO, DBTune, and BBOPlace tasks above because they have meaningful task semantics and domain knowledge, unlike synthetic BBOB functions. Three additional synthetic objectives are used to study correct and incorrect priors.
LLM roles (4 tasks). We select BBOB and HPO tasks to support offline optimizer development, which requires related but distinct development and test tasks. As detailed in Appendices B.5 and C.1, we develop on four BBOB functions and four HPO tasks, and test on F02/F15 and Breast/SVM/Diabetes/RF, respectively. DBTune and BBOPlace are excluded because their instances can involve substantially different environments or circuit structures, making comparable development–test splits more difficult. We use the same four test tasks across all online, handoff, and generated-optimizer settings in Table 3(a) to ensure a fair comparison across roles.
We also check how well these subsets represent the full numerical benchmark using the Table 2(a) results. Across all nine methods, average method ranks on the 8-task subset closely match those on all 67 numerical tasks (Spearman ; similarly, for the 6-task subset vs. all 43 real-world numerical tasks, and for the 4-task subset vs. all 49 BBOB and HPO tasks). Task difficulty and relative win rates are also well aligned: standalone GP achieves a mean score of 0.502 on the 8-task subset versus 0.481 on all 67 tasks, and Agentic outperforms GP on 6/8 (75.0%) subset tasks compared with 50/67 (74.6%) overall. Given the substantial computational and API costs of multiple configurations and repeated runs, these subsets allow us to balance task coverage with the depth of controlled analysis. We have added these selection criteria and representativeness checks to Appendix C.2 of the revised manuscript. Thank you!
W2 & Q3. When do GP tools help?
Thanks for your valuable suggestion! Following your comments, we further analyze the eight tasks in Table 2(b), considering problem characteristics, standalone GP performance, and agent trajectories.
The table reports mean score differences over four matched seeds, where positive values indicate improvements over the base agent (T0). T1 provides GP suggestions, while T4 provides the full GP tool interface.
| Task | Dim. | GP − T0 | T1 − T0 | T4 − T0 |
|---|---|---|---|---|
| BBOB F02 | 10 | −0.0782 | +0.0136 | −0.0558 |
| BBOB F15 | 10 | +0.0591 | +0.1061 | +0.0380 |
| Breast/SVM | 3 | −0.3027 | −0.0333 | −0.0293 |
| Diabetes/RF | 6 | −0.1493 | −0.0445 | −0.0478 |
| PostgreSQL-5 | 5 | −0.1080 | +0.1771 | −0.0113 |
| Sysbench-5 | 5 | +0.0482 | −0.1448 | −0.0049 |
| Adaptec1 | 64 | −0.2969 | −0.0436 | −0.2158 |
| Bigblue1 | 64 | −0.2348 | −0.1534 | −0.0840 |
The results show that GP tool benefits vary across tasks and cannot be explained by dimensionality or standalone GP performance alone. T1 improves both BBOB tasks on average but performs worse on HPO and placement tasks. Interestingly, PostgreSQL-5 and Sysbench-5 are both five-dimensional database tuning tasks, yet T1 achieves +0.1771 and −0.1448, respectively. Standalone GP shows the opposite pattern, performing worse than T0 on PostgreSQL but better on Sysbench. Notably, T1 improves all four seeds on PostgreSQL.
We further inspect the trajectories to understand these differences. On PostgreSQL, the agent does not simply adopt GP suggestions. Instead, it compares GP candidates with its own using a model, adjusts its search direction, and modifies or rejects suggestions based on feedback. This shows that GP can provide useful information even when its candidates are not directly adopted.
On Bigblue1, T0 constructs task-specific features from distances between macros, fits empirical models, and generates new placements accordingly. In contrast, one T1 run follows GP suggestions in 46 of 50 rounds but still performs worse than T0. This suggests that agents with coding capabilities can develop more effective task-specific strategies than generic GP suggestions.
Providing more tools also does not guarantee better performance. T1 improves three of eight tasks on average, while T4 improves only one. On PostgreSQL, T1 achieves +0.1771 compared with −0.0113 for T4. Our trajectory analysis confirms that T4 agents do use GP-related outputs, including prediction queries, rather than simply ignoring the tools.
These findings suggest that standalone optimizer performance does not necessarily predict its value as an agent tool. Agents can use tools as information sources rather than directly adopting their suggestions. Since T0 can already implement numerical models through Python, the benefit of a pre-built tool depends on what it adds to the agent's existing capabilities. This motivates task-adaptive tool selection rather than simply providing more numerical tools.
We have incorporated this task-level discussion into Section 5.3 (and the broader future directions in Section 6) and added the detailed statistics and trajectory examples to Appendix C.8 of the revised manuscript. Thank you!
We hope our responses have addressed your concerns. We have incorporated these clarifications and additional analyses into Sections 5.3, 5.7, and 6 and Appendices C.2 and C.8 of the revised manuscript. Thank you again for your valuable feedback!
Thanks for your detailed comments and valuable suggestions! We appreciate your recognition of the value of a unified Agentic BBO benchmark. Following your suggestions, we further clarify the contributions of our benchmark, explain the selection and representativeness of diagnostic tasks, and incorporate additional task-level and trajectory analyses of GP tools as well as future directions into Sections 5.3, 5.7, and 6 and Appendices C.2 and C.8 of the revised manuscript. Please find our responses below.
W1 & Q1. Contributions of the benchmark
Thanks for your valuable comments! Existing BBO benchmarks (e.g., COCO, Bayesmark, GuacaMol) only expose numerical
ask/tellinterfaces without task semantics, tool interfaces, or reasoning-cost tracking, whereas general coding-agent benchmarks (e.g., SWE-bench, MLAgentBench) evaluate open-ended task completion without shared initializations, strict black-box query budgets, or sample-efficiency metrics. Our goal is to bridge this gap with a unified and extensible benchmark for Agentic BBO, supporting both fair comparisons of existing methods and future research on agent design:Our benchmark thus goes beyond collecting optimization tasks by providing a controlled framework for studying and comparing different Agentic BBO designs. Thank you!
References
[1] Agentic Bayesian Optimization through Surrogate-Augmented Autoresearch. arXiv, 2026.
[2] PlugBO: A Modular Framework for Agentic Bayesian Optimization. Technical Blog, 2026.
[3] Purely Agent-Driven Black-Box Optimization for Biological Design. arXiv, 2026.
[4] Position: Leverage Foundational Models for Black-Box Optimization. ICML 2024.