acceptodds
Under review as a conference paper at ICLR 2027

A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

15 views

Abstract

Black-box optimization (BBO) arises in many scientific and engineering problems where objective evaluations are expensive and limited. Recent large language model (LLM) agents offer a new way to approach BBO by combining task semantics, computation, optimization tools, and feedback-driven decision making. However, existing Agentic BBO studies use different task domains and system configurations, making their results difficult to compare and the effects of individual design choices hard to isolate. We introduce AgenticBBO-Bench, a cross-domain benchmark for Agentic BBO spanning synthetic optimization, hyperparameter optimization, database tuning, chip design, and molecular design under a unified finite-budget evaluation protocol. Agentic BBO achieves higher family-averaged scores than direct candidate generation by the same LLM in all five domains and outperforms the best evaluated numerical optimizers in four. We further study three factors shaping agent performance: optimization tools, task information and prior knowledge, and the role of the LLM during search. Our results show that additional numerical tools do not consistently improve performance, task semantics are broadly useful while more specific priors are less reliable, and numerical optimizers can sometimes continue effectively from search trajectories established by the agent. Finally, we introduce a five-task frontier challenge within AgenticBBO-Bench and evaluate seven LLMs under the same Codex agent harness. GPT-6 Astra and DeepSeek-V4.1-Flash lie on the Pareto frontier of performance and cost among the evaluated models. AgenticBBO-Bench provides a common standardized leaderboard for comparing future general-purpose models and agent systems.

Primary area:
optimization
Keywords:
LLM Agents, Black-Box Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PVrxOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces AgenticBBO-Bench, a cross-domain benchmark for evaluating LLM-based agents on black-box optimization (BBO) problems. The benchmark integrates five established task families, including synthetic optimization (BBOB), hyperparameter optimization (Bayesmark), database tuning (DBTune), chip placement (BBOPlace-Bench), and molecular design (GuacaMol), under a unified finite-budget evaluation protocol.

The authors compare agentic optimization against direct LLM-based candidate generation and conventional numerical optimizers. They further investigate the effects of numerical optimization tools, task semantics and prior knowledge, and different roles of LLMs in the optimization loop. Finally, the paper introduces a five-task frontier challenge to compare seven LLMs under a shared agent harness.

The results suggest that agentic optimization consistently improves over direct LLM candidate generation across the five domains, while additional numerical optimization tools do not necessarily improve agent performance. Task semantics can be beneficial, whereas richer domain priors have less consistent effects. Numerical optimizers can also sometimes effectively continue optimization after an agent-generated warm-up.

Critical strengths
  1. The paper introduces AgenticBBO-Bench, integrating 77 tasks across five heterogeneous BBO domains under a unified evaluation protocol. The benchmark accommodates different search spaces, candidate representations, and feasibility constraints while maintaining consistent evaluation budgets and scoring rules. This provides useful infrastructure for standardized evaluation of general-purpose optimization agents.

  2. Beyond introducing a benchmark, the paper investigates the effects of numerical optimization tools, task semantics and prior knowledge, and different roles of LLMs in the optimization loop. These controlled experiments provide useful insights into agentic optimization behavior, particularly that additional GP-based tools do not consistently improve performance and that numerical optimizers can sometimes effectively continue from agent-generated search trajectories.

Critical weaknesses
  1. The benchmark is built upon five established BBO benchmarks (BBOB, Bayesmark, DBTune, BBOPlace-Bench, and GuacaMol). Much of the contribution lies in adapting these tasks to a shared agent interface and standardizing the evaluation protocol. While this integration is useful, it is less clear what new benchmarking challenges are addressed beyond making existing BBO tasks accessible to LLM agents. Similar ideas of unified interaction and evaluation have also been explored in the broader agent benchmarking literature. The paper would benefit from a clearer discussion of what makes this benchmark substantially different from existing BBO and agent benchmarks, and why these differences matter.

  2. Although the full benchmark contains 77 tasks across five domains, the main diagnostic studies are conducted on much smaller subsets. Section 5.3 evaluates optimization tools on eight tasks, Section 5.4 studies task information on six real-world tasks (plus three controlled synthetic objectives), and Section 5.5 examines LLM roles on only four tasks from BBOB and HPO. This makes it difficult to tell how broadly the reported findings apply across different BBO problems. Furthermore, some of the takeaways, such as the benefits of task semantics and the potential harm of incorrect priors, are relatively intuitive. Given that the benchmark itself largely integrates existing tasks, I would expect the empirical study to provide stronger insights into when different agentic approaches work well or poorly. A more systematic analysis across problem characteristics, such as dimensionality and search-space structure, would make the findings more informative.

Questions for authors
  1. Since the benchmark largely builds on existing BBO task suites and adopts a unified agent interaction protocol, could the authors clarify which aspects of its design go beyond adapting these tasks to an agent framework? A few concrete examples would help better establish the benchmark's contribution.

  2. The analyses in Sections 5.3–5.5 use relatively small subsets of the full benchmark (8, 6, and 4 tasks, respectively). How were these tasks selected, and are they representative of the broader suite in terms of dimensionality, search-space structure, and optimization difficulty? It would be helpful to understand how much the findings depend on this selection.

  3. Table 2(b) shows that adding GP tools helps on some tasks but hurts on others. Is there any observable relationship between these differences and the characteristics of the optimization problems? A more fine-grained analysis would make the experimental findings more useful for guiding future Agentic BBO research.

5 comments

Thanks for your detailed comments and valuable suggestions! We appreciate your recognition of the value of a unified Agentic BBO benchmark. Following your suggestions, we further clarify the contributions of our benchmark, explain the selection and representativeness of diagnostic tasks, and incorporate additional task-level and trajectory analyses of GP tools as well as future directions into Sections 5.3, 5.7, and 6 and Appendices C.2 and C.8 of the revised manuscript. Please find our responses below.

W1 & Q1. Contributions of the benchmark

Thanks for your valuable comments! Existing BBO benchmarks (e.g., COCO, Bayesmark, GuacaMol) only expose numerical ask/tell interfaces without task semantics, tool interfaces, or reasoning-cost tracking, whereas general coding-agent benchmarks (e.g., SWE-bench, MLAgentBench) evaluate open-ended task completion without shared initializations, strict black-box query budgets, or sample-efficiency metrics. Our goal is to bridge this gap with a unified and extensible benchmark for Agentic BBO, supporting both fair comparisons of existing methods and future research on agent design:

  1. A unified and extensible benchmark. We integrate 77 tasks from five established BBO benchmarks under a common evaluation protocol with matched initializations and strict evaluation budgets, while tracking internal agent execution and LLM token costs. Beyond unifying the tasks, our framework decouples backbone LLMs, agent harnesses, task information, numerical tools, and search control strategies so researchers can modify individual components while keeping the tasks and evaluation protocol fixed.
  2. Systematic analysis of Agentic BBO systems. Recent works have explored several important directions for improving BBO, including the use of numerical tools (e.g., Sara [1] and PlugBO [2]) and task knowledge and priors (e.g., PABLO [3] and Song et al. [4]). However, these studies use different tasks and agent configurations, making their findings difficult to compare directly. Our benchmark brings these research questions into a unified framework, where we systematically study optimization tools, task information and priors, and the role of the LLM during search (including mid-search handoff from a shared agent prefix) under consistent conditions.
  3. A standardized frontier challenge. We introduce a five-task challenge and evaluate seven frontier LLMs under the same Codex agent harness, comparing both optimization performance and inference cost (with repeated three-seed results reported in Appendix C.4). This provides a reusable leaderboard for future general-purpose models and agent systems.

Our benchmark thus goes beyond collecting optimization tasks by providing a controlled framework for studying and comparing different Agentic BBO designs. Thank you!

References

[1] Agentic Bayesian Optimization through Surrogate-Augmented Autoresearch. arXiv, 2026.

[2] PlugBO: A Modular Framework for Agentic Bayesian Optimization. Technical Blog, 2026.

[3] Purely Agent-Driven Black-Box Optimization for Biological Design. arXiv, 2026.

[4] Position: Leverage Foundational Models for Black-Box Optimization. ICML 2024.

W2 & Q2. Selection of diagnostic tasks

Thanks for your comments! We use the full 77-task benchmark for cross-domain evaluation and select smaller subsets for controlled studies in Sections 5.3–5.5 based on their research objectives.

Optimization tools (8 tasks). We select two tasks from each of four domains: BBOB F02/F15 (unimodal/multimodal continuous), Breast/SVM and Diabetes/RF (log-scaled continuous and mixed integer/float hyperparameter spaces), PostgreSQL-5 and Sysbench-5 (mixed integer/float and enum database knobs), and Adaptec1 and Bigblue1 (64D macro coordinates with constraint repair). These tasks cover continuous, mixed, and structured search spaces, with dimensions ranging from 3 to 64.

Task information (6 tasks). We use the HPO, DBTune, and BBOPlace tasks above because they have meaningful task semantics and domain knowledge, unlike synthetic BBOB functions. Three additional synthetic objectives are used to study correct and incorrect priors.

LLM roles (4 tasks). We select BBOB and HPO tasks to support offline optimizer development, which requires related but distinct development and test tasks. As detailed in Appendices B.5 and C.1, we develop on four BBOB functions and four HPO tasks, and test on F02/F15 and Breast/SVM/Diabetes/RF, respectively. DBTune and BBOPlace are excluded because their instances can involve substantially different environments or circuit structures, making comparable development–test splits more difficult. We use the same four test tasks across all online, handoff, and generated-optimizer settings in Table 3(a) to ensure a fair comparison across roles.

We also check how well these subsets represent the full numerical benchmark using the Table 2(a) results. Across all nine methods, average method ranks on the 8-task subset closely match those on all 67 numerical tasks (Spearman ρ=0.967\rho = 0.967; similarly, ρ=0.900\rho = 0.900 for the 6-task subset vs. all 43 real-world numerical tasks, and ρ=0.996\rho = 0.996 for the 4-task subset vs. all 49 BBOB and HPO tasks). Task difficulty and relative win rates are also well aligned: standalone GP achieves a mean score of 0.502 on the 8-task subset versus 0.481 on all 67 tasks, and Agentic outperforms GP on 6/8 (75.0%) subset tasks compared with 50/67 (74.6%) overall. Given the substantial computational and API costs of multiple configurations and repeated runs, these subsets allow us to balance task coverage with the depth of controlled analysis. We have added these selection criteria and representativeness checks to Appendix C.2 of the revised manuscript. Thank you!

W2 & Q3. When do GP tools help?

Thanks for your valuable suggestion! Following your comments, we further analyze the eight tasks in Table 2(b), considering problem characteristics, standalone GP performance, and agent trajectories.

The table reports mean score differences over four matched seeds, where positive values indicate improvements over the base agent (T0). T1 provides GP suggestions, while T4 provides the full GP tool interface.

TaskDim.GP − T0T1 − T0T4 − T0
BBOB F0210−0.0782+0.0136−0.0558
BBOB F1510+0.0591+0.1061+0.0380
Breast/SVM3−0.3027−0.0333−0.0293
Diabetes/RF6−0.1493−0.0445−0.0478
PostgreSQL-55−0.1080+0.1771−0.0113
Sysbench-55+0.0482−0.1448−0.0049
Adaptec164−0.2969−0.0436−0.2158
Bigblue164−0.2348−0.1534−0.0840

The results show that GP tool benefits vary across tasks and cannot be explained by dimensionality or standalone GP performance alone. T1 improves both BBOB tasks on average but performs worse on HPO and placement tasks. Interestingly, PostgreSQL-5 and Sysbench-5 are both five-dimensional database tuning tasks, yet T1 achieves +0.1771 and −0.1448, respectively. Standalone GP shows the opposite pattern, performing worse than T0 on PostgreSQL but better on Sysbench. Notably, T1 improves all four seeds on PostgreSQL.

We further inspect the trajectories to understand these differences. On PostgreSQL, the agent does not simply adopt GP suggestions. Instead, it compares GP candidates with its own using a model, adjusts its search direction, and modifies or rejects suggestions based on feedback. This shows that GP can provide useful information even when its candidates are not directly adopted.

On Bigblue1, T0 constructs task-specific features from distances between macros, fits empirical models, and generates new placements accordingly. In contrast, one T1 run follows GP suggestions in 46 of 50 rounds but still performs worse than T0. This suggests that agents with coding capabilities can develop more effective task-specific strategies than generic GP suggestions.

Providing more tools also does not guarantee better performance. T1 improves three of eight tasks on average, while T4 improves only one. On PostgreSQL, T1 achieves +0.1771 compared with −0.0113 for T4. Our trajectory analysis confirms that T4 agents do use GP-related outputs, including prediction queries, rather than simply ignoring the tools.

These findings suggest that standalone optimizer performance does not necessarily predict its value as an agent tool. Agents can use tools as information sources rather than directly adopting their suggestions. Since T0 can already implement numerical models through Python, the benefit of a pre-built tool depends on what it adds to the agent's existing capabilities. This motivates task-adaptive tool selection rather than simply providing more numerical tools.

We have incorporated this task-level discussion into Section 5.3 (and the broader future directions in Section 6) and added the detailed statistics and trajectory examples to Appendix C.8 of the revised manuscript. Thank you!

We hope our responses have addressed your concerns. We have incorporated these clarifications and additional analyses into Sections 5.3, 5.7, and 6 and Appendices C.2 and C.8 of the revised manuscript. Thank you again for your valuable feedback!

Reviewer_Ro1UOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AgenticBBO-Bench, a cross-domain benchmark for evaluating LLM agents on finite-budget black-box optimization (BBO). It covers five task families: synthetic numerical optimization from BBOB, hyperparameter optimization (HPO) from Bayesmark, database tuning via DBTune surrogates, macro placement from BBOPlace-Bench, and molecular design from GuacaMol. All domains are evaluated under a unified protocol featuring shared candidate initializations, strict budget accounting that counts only host-evaluated candidates, an objective evaluator isolated from the agent container, and a normalized anytime score (70% trajectory average, 30% final performance) anchored against a matched Gaussian Process Bayesian Optimization (GP-BO) reference model.

Using DeepSeek V4.1 Flash within the Codex harness, the authors find that an agent with a Bash/Python workspace beats direct in-context candidate generation across all five task families (with a family-averaged relative improvement of 30.3%). It also outperforms the best evaluated numerical baselines on BBOB, HPO, BBOPlace, and GuacaMol, while TuRBO performs best on DBTune.

Controlled ablations report three primary findings: providing explicit GP tools to the agent does not consistently improve performance on average, task semantics are broadly useful whereas more specific prior knowledge is less reliable and depends heavily on the problem, and handing off the search to TuRBO after an initial agent warm-up matches full online agent control while LLM-developed offline optimizers exhibit inconsistent cross-task transfer.

Finally, a five-task frontier challenge compares seven LLMs under the same agent harness, highlighting that performance is only weakly aligned with official token cost.

Critical strengths
  1. A unified, well-specified protocol for a fragmented research domain. Agentic BBO results are currently reported across incompatible domains, harnesses, and budgets. Shared initializations, an evaluator isolated from the agent container, and budget accounting that counts only accepted host evaluations make cross-domain comparison possible. The extensive appendix detail makes the benchmark reusable and represents a major contribution.

  2. Controlled ablations with practical design implications. Three results are particularly useful for designing agentic systems: the Agent → TuRBO handoff matches full online agent control (0.570 vs. 0.569), suggesting that the LLM's value is concentrated early in the search; exposing GP tools does not help performance on average across tasks; and in the controlled-prior study, correct support plus geometry helps substantially, while wrong support is roughly neutral rather than harmful. Together, these move the discussion past simple "agents beat baselines" claims toward understanding when and where the LLM is useful.

Critical weaknesses
  1. Thin statistical support for the headline claims. The broad benchmark uses 2 runs per task, the ablations 4 seeds, and the frontier leaderboard a single run per (task, model) pair. No standard deviations, confidence intervals, or significance tests are reported. As a result, the leaderboard margins (e.g., 59.1 vs. 58.4) and the Pareto-frontier claim cannot be clearly separated from agent stochasticity. Several ablation takeaways also rest on mean differences that are small relative to per-task swings. In Table 2b, for example, the means differ by 0.015 while individual tasks differ by more than 0.15. Since the paper’s primary contribution is a benchmark and leaderboard, the low number of runs and absence of variance estimates directly weaken its central claims.

  2. Metric design and missing controls make attribution unclear.

  • Metric: The score maps the matched GP-BO endpoint to 0.6, applies a hand-set exponential tail above it, and weights the trajectory average at 70%. This can favor fast early gains from LLM priors over exploratory BO, and it makes rankings depend on how strong the GP reference is on each task. No raw-regret or final-only results and no sensitivity analysis are given.
  • Direct vs. Agentic: The comparison changes agency, code execution, and context handling at once. For example, Direct must hold up to 250 × 196-dimensional DBTune histories in context.
  • Information ablation: I0 vs. I1 changes both semantics and representation. I0 maps parameters to [0,1] and hides their original transforms (e.g., log scaling).
  • Memorization: Possible recall on well-known benchmarks (Bayesmark sklearn tasks, GuacaMol) is not discussed. Together, these make it unclear how much of the reported advantage comes from agentic reasoning versus metric choice, tooling, representation, or recall.
Questions for authors
  1. Please report per-task standard deviations across seeds for Tables 2a, 2b, and 3a, and state whether the main rankings and takeaways survive paired statistical tests. Can the frontier challenge be rerun with at least 3 runs per model?
  2. How do the rankings change under final-only scoring, raw simple regret, or a different GP anchor point or tail coefficient?
  3. How does Direct perform when it also has a code interpreter (without a persistent agent loop)? This would help separate code execution from agency.
  4. Graph-GA scores 0.054 on GuacaMol, which is unusually low for a method that is typically strong there. Please give its population and mutation settings and a sanity check against published results. Relatedly, GP-BO optimizes acquisition over a random candidate pool even in 64–196 dimensions. Would gradient-based acquisition optimization change the DBTune and BBOPlace comparisons?
  5. For the I0 vs. I1 information ablation, can you add a condition with anonymous names but the original parameter geometry, to isolate the effect of semantics?
8 comments

Thanks for your detailed feedback! We agree that the original submission did not provide enough information about statistical uncertainty and some implementation details. Following your suggestions, we have added statistical tests, repeated evaluations, metric sensitivity analyses, and two new controlled studies. We summarize the main results below, and we have incorporated an overview in Section 5.7 together with the full per-task tables, implementation clarifications, and representative search trajectories in Appendices C.2–C.7 and C.9 of the revised manuscript.


Q1.1 Statistical significance of the benchmark results

Thanks for pointing this out! We have added mean and standard deviation across seeds for all entries in Tables 2(a), 2(b), and 3(a) in Appendix C.3. We also perform paired tests to check whether the differences are consistent across tasks. Specifically, we first average the score differences over matched seeds for each task, and then apply a two-sided Wilcoxon signed-rank test with Holm correction. These tests use the original runs reported in the paper.

Results in Table U1 (also reported in Appendix C.3) show that Agentic significantly outperforms all eight numerical baselines and all three molecular baselines after Holm correction. For example, Agentic performs better than Direct, GP, and TuRBO on 53/67, 50/67, and 49/67 numerical tasks, with Holm-adjusted pp-values of 1.91×10−61.91\times10^{-6}, 7.55×10−67.55\times10^{-6}, and 1.55×10−51.55\times10^{-5}, respectively. All other numerical comparisons are also significant (p≤1.55×10−5p\leq1.55\times10^{-5}), as are the molecular comparisons (p≤0.0098p\leq0.0098). The results remain significant after grouping related HPO and DBTune tasks into 44 groups (p≤8.2×10−4p\leq8.2\times10^{-4}). These results provide strong statistical support for our main benchmark findings.

For the diagnostic studies, the results are also consistent with our original observations. In the tool study, T0 outperforms GP Suggest on 5/8 tasks (p=0.6406p=0.6406) and GP Tools on 7/8 tasks (p=0.1172p=0.1172), suggesting that adding numerical tools does not necessarily improve performance. In the LLM-role study, Agentic and Agent-to-TuRBO achieve nearly identical mean scores (Δ=−0.0015\Delta=-0.0015, p=1.0000p=1.0000), consistent with our finding that numerical optimizers can sometimes continue the search effectively after an agent-generated prefix. All reported pp-values are Holm-adjusted. While the differences in these smaller studies are not statistically significant, the observations remain consistent with the conclusions of our paper.

Table U1: Task-level paired comparisons on the original Tables 2(a), 2(b), and 3(a). All differences are Agentic minus the named comparator (Agentic is T0 in the diagnostic panels). Positive Δ\Delta favors Agentic. Wins/losses/ties count tasks after averaging over matched seeds. Mean Δ\Delta indicates the average performance difference between the two methods; the Wilcoxon test concerns the signed-rank distribution of paired differences, not the arithmetic mean directly. Statistical support is assessed using Holm-adjusted p<0.05p<0.05.

PanelComparatorTasksMean Δ\DeltaWins / losses / tiesHolm pp
2(a), NumericalDirect67+0.111453 / 14 / 01.91e-06
2(a), NumericalGP67+0.106350 / 17 / 07.55e-06
2(a), NumericalTuRBO67+0.093949 / 18 / 01.55e-05
2(a), NumericalCMA-ES67+0.179356 / 11 / 05.17e-10
2(a), NumericalTPE67+0.185360 / 7 / 01.52e-11
2(a), NumericalGitBO67+0.254258 / 9 / 02.09e-11
2(a), NumericalSOBOL67+0.354765 / 2 / 05.96e-18
2(a), NumericalRANDOM67+0.369963 / 4 / 01.96e-17
2(a), GuacaMolDirect10+0.27529 / 1 / 00.0098
2(a), GuacaMolGPBO10+0.713310 / 0 / 00.0059
2(a), GuacaMolGraph-GA10+0.786810 / 0 / 00.0059
2(b), tools+ GP Suggest8+0.01535 / 3 / 00.6406
2(b), tools+ GP Tools8+0.05147 / 1 / 00.1172
2(b), toolsGP8+0.13286 / 2 / 00.1172
3(a), rolesGP policy4+0.05223 / 1 / 01.0000
3(a), rolesAgent to GP4+0.04452 / 2 / 01.0000
3(a), rolesAgent to TuRBO4-0.00152 / 2 / 01.0000
3(a), rolesAgent to Local4+0.03603 / 1 / 01.0000
3(a), rolesGenerated4+0.06783 / 1 / 01.0000
3(a), rolesGP4+0.11783 / 1 / 01.0000
3(a), rolesStandalone TuRBO4+0.09344 / 0 / 00.8750

Q1.2 Repeated runs for the Frontier Challenge

Following your suggestion, we have extended the five-task Frontier Challenge to three seeds for all seven models, and we have added the repeated composite scores and token/cost breakdowns to Appendix C.4. The main results are provided in Tables U2 and U3. DeepSeek-V4.1-Flash was updated after our original submission, so we reran it using the October 17 version under the same evaluation protocol.

GPT-6 Astra and DeepSeek V4.1 Flash obtain the highest average scores, 0.5931±0.03510.5931 \pm 0.0351 and 0.5699±0.03210.5699 \pm 0.0321, respectively. However, their estimated inference costs over the three seeds and five tasks differ substantially: USD 880.89 versus USD 13.17. We do not consider this small score difference sufficient to establish a clear ranking between these two models. The more relevant observation is that much higher inference cost does not necessarily lead to better optimization performance.

Table U2: Frontier repeated runs — composite score (mean ± SD across 3 seeds). Average SD is computed across seed-level five-task means.

ModelSeedsdigits_mlp_sgdsysbench_allmedian1f15_d10bigblue1Average
GPT-6 Astra30.6361 ± 0.05110.3949 ± 0.09370.6410 ± 0.00050.6744 ± 0.06530.6189 ± 0.10890.5931 ± 0.0351
DeepSeek V4.1 Flash30.5956 ± 0.03040.3580 ± 0.05020.8205 ± 0.00370.4030 ± 0.11300.6724 ± 0.02830.5699 ± 0.0321
GPT-6 Sol30.6246 ± 0.05610.1926 ± 0.00380.8182 ± 0.00590.4442 ± 0.09370.5518 ± 0.15350.5263 ± 0.0211
Gemini 3.1 Pro30.6036 ± 0.01750.3180 ± 0.06440.7264 ± 0.13440.4726 ± 0.19120.1802 ± 0.07870.4602 ± 0.0657
Kimi K330.5846 ± 0.06390.0887 ± 0.05420.8180 ± 0.00380.3498 ± 0.03890.4495 ± 0.19250.4581 ± 0.0537
DeepSeek V4 Pro30.4321 ± 0.10560.1942 ± 0.16380.8130 ± 0.00860.3240 ± 0.15380.4330 ± 0.18620.4393 ± 0.0746
GLM-5.330.5252 ± 0.07640.0909 ± 0.05620.7559 ± 0.11090.3464 ± 0.10720.3066 ± 0.10210.4050 ± 0.0506

Table U3: Frontier token usage and conservative official list-price cost across 3 seeds and 5 tasks. M denotes million tokens; Input M includes cached tokens.

ModelSeedsInput MCached MOutput MTotal MCost (USD)
GPT-6 Astra3370.063342.4763.872373.934880.89
Flash 4.13968.173948.06112.189980.36213.17
GPT-6 Sol3238.826192.5915.482244.307417.84
Gemini 3.1 Pro3527.649285.4195.610533.259608.86
Kimi K33157.642131.2712.273159.915152.58
Pro3170.291152.7684.925175.21725.38
GLM-5.33194.143136.6013.188197.331130.10
Reviewer_CZtVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AgenticBBO-Bench, a suite of 77 black-box optimization tasks spanning synthetic functions, hyperparameter optimization, database tuning, placement, and molecular design. It compares numerical optimizers, direct LLM proposals, and agents that can maintain state and implement optimization routines through tools. The evaluation matches initial observations and objective-evaluation budgets and combines anytime and final solution quality. Further experiments examine task information, numerical-tool access, agent-to-optimizer handoffs, reusable optimizer code, and a small frontier-model comparison. Agentic execution improves over direct prompting in all five reported domain aggregates, while comparisons with numerical optimizers depend on the domain.

Critical strengths
  1. The common evaluation protocol is a useful contribution. The five domains expose different representations, constraints, and feedback structures, and the numerical baselines make clear that agentic optimization is not uniformly superior.
  2. The diagnostic experiments make the paper more useful than a leaderboard alone. They explore how tool access and task information affect performance, when agents should hand off to numerical optimizers, and whether agents can produce reusable optimizer code. Matching the initial observations and number of objective evaluations makes these comparisons informative about sample efficiency, though it does not establish savings in total compute or runtime.
Critical weaknesses

Some of the reported differences are too small to interpret confidently given the limited number of runs. Appendix B.2 reports two runs per task for the broad comparison, four for the diagnostic experiments, and only one per task and model for the frontier comparison. The frontier results cover just five tasks, with aggregate scores as close as 0.591 and 0.584. Likewise, the handoff score of 0.570 versus 0.569 for the full agent suggests similar performance, but does not establish equivalence. Paired confidence intervals and additional runs for these close comparisons would help determine whether the differences are reliable. This concern mainly affects close rankings, rather than the larger gains over direct prompting.

Questions for authors

Thank you for presenting the paper. I have several questions listed below: Can you report paired uncertainty for the main method differences and repeat the closest frontier/handoff comparisons on a small matched subset? If additional runs are unavailable, which ranking claims remain supported by the current data?

2 comments

Thanks for your suggestions! Following your comments, we have conducted paired analyses and additional experiments to better assess the uncertainty of our results, summarized these checks in Section 5.7, and added the full tables and discussions to Appendices C.3 and C.4 of the revised manuscript. Please find our responses below.

Q1.1 Paired Analysis of Agentic vs. Direct

We calculated the paired score differences between Agentic and Direct across five benchmark families (now reported in Appendix C.3). As shown in Table R1, Agentic achieves higher average scores in all five domains, with positive improvements in both matched seeds. The gains are especially large on BBOB (+0.2005), DBTune (+0.1373), and GuacaMol (+0.2752), while HPO shows a smaller improvement (+0.0351). These results are consistent with our main findings.

Table R1. Paired comparison of Agentic and Direct across five benchmark families. For each seed, the score difference is averaged equally across tasks within a family; means and sample standard deviations (SD) of the paired differences are then computed across the two matched seeds. Higher scores are better.

BenchmarkSeedsAgenticDirectPaired Δ (± SD)
BBOB20.66060.4601+0.2005 ± 0.0167
HPO20.55370.5186+0.0351 ± 0.0484
DBTune20.52510.3878+0.1373 ± 0.0883
BBOPlace20.54400.4648+0.0792 ± 0.0403
GuacaMol20.84050.5652+0.2752 ± 0.0492

Note: Δ = Agentic − Direct. Means in the Agentic and Direct columns are also averaged across the same tasks and seeds.

Q1.2 Additional Runs for Agent-to-TuRBO Handoff

Following your suggestion, we conducted additional handoff experiments with 8 matched seeds on 4 tasks (added to Appendix C.4). Since the September version of DeepSeek-V4.1-Flash is no longer available, we reran both methods using the October 17 version. Both methods share the same initial observations and evaluation budget, and TuRBO inherits the corresponding agent's search history.

As shown in Table R2, handoff achieves similar performance to continued Agent control (0.5451 vs. 0.5404 on average). More importantly, handoff reduces LLM token consumption by 52.8% (Table R3). These results suggest that numerical optimizers can sometimes effectively continue the search from agent-generated trajectories, reducing LLM cost without a clear loss in optimization performance.

Table R2. Eight-seed matched comparison of continued Agent and Agent-to-TuRBO handoff using the October 17, 2026 version of DeepSeek-V4.1-Flash. Both conditions were rerun with the same model version, matched observations and evaluation budgets. TuRBO inherits the corresponding Agent's search history at handoff.

TaskSeedsContinued AgentAgent → TuRBOPaired Δ (± SD)
BBOB F0280.60230.6056+0.0033 ± 0.0251
BBOB F1580.49960.5103+0.0107 ± 0.0970
Breast/SVM80.40650.4098+0.0033 ± 0.0252
Diabetes/RF80.65320.6549+0.0017 ± 0.0113

Note: Δ = (Agent → TuRBO) − Continued Agent. SD is the sample SD of eight within-seed paired differences.

Table R3. Observed LLM token usage of continued Agent and Agent-to-TuRBO handoff over 32 matched trajectory pairs (eight seeds for each of four tasks). Values are millions of tokens per run. The handoff count includes the recorded shared agent prefix; TuRBO makes no subsequent LLM calls.

TaskContinuedHandoffReduction
BBOB F0220.28110.25649.4%
BBOB F1518.7408.73553.4%
Breast/SVM1.6790.49470.6%
Diabetes/RF1.2550.31774.7%
Average (32 pairs)10.4894.95152.8%

Q1.3 Additional Runs for Frontier Challenge

We also repeated the Frontier experiments with 3 seeds per model and task (added to Appendix C.4). Results are shown in Table R4. GPT-6 Astra and DeepSeek V4.1 Flash remain the top two models, with average scores of 0.5931 and 0.5699, respectively. However, their difference is small relative to the observed variance, so we do not claim a clear winner between them.

We also observe substantial task-level differences. For example, Flash performs better on Median1, while Astra performs better on BBOB F15. We believe these differences are more informative than small changes in the overall leaderboard rankings.

Table R4. Frontier Challenge results: mean score ± sample SD across seeds. The Average first assigns equal weight to the five tasks within each seed, then averages across 3 seeds.

ModelSeedsDigits/MLPSysbench-allMedian1BBOB F15Bigblue1Average
GPT-6 Astra30.6361 ± 0.05110.3949 ± 0.09370.6410 ± 0.00050.6744 ± 0.06530.6189 ± 0.10890.5931 ± 0.0351
DeepSeek V4.1 Flash30.5956 ± 0.03040.3580 ± 0.05020.8205 ± 0.00370.4030 ± 0.11300.6724 ± 0.02830.5699 ± 0.0321
GPT-6 Sol30.6246 ± 0.05610.1926 ± 0.00380.8182 ± 0.00590.4442 ± 0.09370.5518 ± 0.15350.5263 ± 0.0211
DeepSeek V4 Pro30.4321 ± 0.10560.1942 ± 0.16380.8130 ± 0.00860.3240 ± 0.15380.4330 ± 0.18620.4393 ± 0.0746
GLM-5.330.5252 ± 0.07640.0909 ± 0.05620.7559 ± 0.11090.3464 ± 0.10720.3066 ± 0.10210.4050 ± 0.0506
Gemini 3.1 Pro30.6036 ± 0.01750.3180 ± 0.06440.7264 ± 0.13440.4726 ± 0.19120.1802 ± 0.07870.4602 ± 0.0657
Kimi K330.5846 ± 0.06390.0887 ± 0.05420.8180 ± 0.00380.3498 ± 0.03890.4495 ± 0.19250.4581 ± 0.0537

Q1.4 Confidence Intervals

We further calculated paired 95% confidence intervals for the handoff and Frontier comparisons, as shown in Table R5 (and reported in Appendix C.4). The intervals include zero, indicating that the observed differences are not statistically significant. Together with the substantial reduction in LLM token consumption, these results suggest that numerical optimizers can sometimes effectively continue the search from agent-generated trajectories at a much lower LLM cost.

Table R5. Exploratory two-sided 95% paired confidence intervals (CIs) for selected comparisons. Each interval is computed over seed-level paired differences using Student's t approximation, with sample SD and n − 1 degrees of freedom.

Comparison (Δ)Benchmark / taskSeeds95% CI
Handoff − AgentBBOB F028[−0.0177, +0.0243]
BBOB F158[−0.0704, +0.0918]
Breast/SVM8[−0.0178, +0.0244]
Diabetes/RF8[−0.0077, +0.0111]
Astra − FlashFrontier average3[−0.0861, +0.1325]

We have incorporated these additional results and discussions into Section 5.7 and Appendices C.3 and C.4 of the revised manuscript. Thank you again for your helpful suggestions!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers and Area Chair,

Thank you for your constructive feedback and suggestions. In response to your comments, we highlight the main text changes in red and report full experimental details in the main text and Appendix C.

Statistical analysis across the benchmark (CZtV, Ro1U). We added per-task variance across seeds and paired statistical comparisons with Holm correction. Across matched seeds, Agentic consistently outperforms Direct across all task families and across the majority of numerical tasks after within-task averaging. It also maintains the top overall average rank under final-only scoring and across all tested GP normalization settings. We now report raw BBOB regret and unnormalized objective values for the remaining domains so performance can be inspected in task-specific units (Appendices C.2, C.3, and C.5).

Agent-to-optimizer handoff (CZtV, Ro1U). We re-evaluated continued agent control against agent-to-TuRBO handoff across multiple tasks and matched seeds. Starting from the same agent-generated search history, handoff achieves comparable final performance while substantially reducing LLM token consumption (Appendix C.4).

Frontier model comparison (CZtV, Ro1U). We expanded the multi-task challenge across multiple seeds for all evaluated frontier models. While GPT-6 Astra and DeepSeek V4.1 Flash achieve the highest mean scores, their paired confidence interval spans zero, indicating no statistically significant gap between them. In the revised text, we examine task-level performance differences and report token usage alongside estimated inference costs (Appendix C.4).

Code execution, persistent workspace, and task semantics (Ro1U). We introduced two ablation controls across real-world tasks with matched seeds. First, Code Direct (which has Python execution access but starts each round with a fresh workspace) reaches comparable final scores, whereas the persistent agent requires substantially fewer tokens. Second, restoring parameter bounds and transformations while keeping variable names anonymous yields minimal change in average rank, while restoring task semantics leads to a clear rank improvement (Appendices C.6–C.7).

Diagnostic analysis of GP suggestions (PVrx). We documented the selection criteria for the diagnostic subsets and added task-level comparisons alongside full search trajectories. On two database tuning tasks, incorporating GP suggestions substantially improves the score on PostgreSQL (where the agent uses GP outputs to evaluate its own candidates) but degrades performance on Sysbench. On Bigblue1, the base agent models distances between macros, whereas an agent that relies heavily on GP suggestions across rounds achieves a lower score (Section 5.3 and Appendices C.2 and C.8).

Other revisions. We clarified Graph-GA’s hyperparameters and evaluation budget, documented GP-BO’s multistart gradient-based acquisition optimization, and included a representative search trajectory in our discussion of potential benchmark recall (Appendices C.2 and C.9).

Thank you again for your time and thoughtful reviews.

We address each reviewer's questions in detail in the individual replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.