- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes agentic Bayesian optimization, where an LLM agent runs the BO loop and a GP backend does the modeling and acquisition. Before each evaluation, the agent can probe the surrogate, ask it for proposals, or change the bounds, acquisition function, objectives or constraints. Then it decides which point to evaluate. The authors frame this as a metalevel decision process, with LLM-as-optimizer methods and standard BO as special cases. They implement the idea as an agent called Sara and a BoTorch-based backend called lenz. lenz keeps the full trial history, so the agent can reformulate the problem mid-run without losing data. The experiments cover three setting: the first has no prior knowledge, using synthetic functions, the second gives the agent a short natural-language description, using hyperparameter tuning and reaction-yield tasks, and the third is a cost-aware AutoML task with a wall-clock deadline. Sara is compared with standard BO and with LLM-based optimizers. It roughly matches standard BO when there is no context, does clearly better when context is available, and beats the LLM-based methods throughout.
Critical strengths
- The core design that keeps the GP as a tool the agent calls, instead of asking the LLM to act as the surrogate, fixes the main weakness of LLM-as-optimizer methods, and the results support this. Sara beats every LLM-only baseline by a wide margin on every suite.
- The evaluation handles LLM contamination more carefully than existing works. The authors shift optima, rename parameters, run each agent in an anonymous sandbox, and add a memorization probe on LCBench.
Critical weaknesses
- The paper never isolates what the agent adds on top of a well-configured GP. The central claim is about run-time policy control. However, every comparison mixes the LLM's warm start, lenz's own BO defaults and the agent's actual decisions. I believe two baselines could separate these effects. The first is lenz on autopilot, where a script calls suggest and observe with no agent. The second is an LLM warm start followed by plain BO, such as Ax seeded with the LLM's opening points, or πBO with an LLM-elicited prior. Without the second baseline, the gains on the prior-informed tasks could come mostly from a few good first points, which is an effect LLAMBO and Centaur already show early in their runs. Without the first, I can't tell whether Sara's no-context win over Ax on the reaction tasks comes from the agent or from lenz's settings
- The prior-informed results are probably inflated, for two reasons. First, the reaction benchmarks are easy for an LLM by construction. The authors simulated them and placed each optimum where textbook chemistry says it should be. For example, the Grignard optimum is cold and dry, and the Heck optimum is hot with little water. The prompt then gives the reaction's name. An LLM that knows basic chemistry can therefore guess the right region from the name alone, before it sees any data. The objective is also close to separable, so tuning one variable at a time works well, and the authors admit this in G.1. A real reaction dataset such as Shields et al. (2021) would be a much fairer test.
- The baselines also leave out the strongest competitors. Each baseline lacks one of Sara's two advantages. Ax has a GP but never sees the problem description, while LLAMBO and Centaur see the description but have no GP. Hybrid methods such as BORA and LLINBO have both, and Table 1 lists them as Sara's closest relatives, but none of them is in the experiments. JAHS has a similar gap. Only Sara can read the training code, which shows which operation index is a convolution. The baselines see only the numbers 0 to 4. So the paper shows that Sara beats methods that are missing either the GP or the context. It doesn't show that Sara beats methods that have both.
- The mid-run reformulation is shown as a single qualitative trajectory. A small quantitative comparison would make the claim stronger, for example against restarting multi-objective BO from scratch, or against Ax reusing the same trials. I'd also soften the claim in the abstract that this is "a capability not available in standard BO." Any BO library that keeps its data can switch from a constrained problem to a multi-objective one. What's new is that the agent makes the switch from a natural-language instruction.
- It would also help to report the cost per run next to Ax's in the main text. Figure 23 shows up to about 200k new tokens per run, which is fine for wet-lab experiments but matters for cheaper ones.
- These presentation issues need clarifying. The Metrics paragraph says yield objectives report the raw best value, but Table 2 shows regret, and the GP-sample rows are negative and unlabeled. Finally, there are a few typos: "exposed" (l. 256), "the the" (l. 1279), "versionn" (l. 1499), "significiant" (l. 1549), "accessable" (l. 2388) and "durration" (Figure 14).
Shields et al., "Bayesian reaction optimization as a tool for chemical synthesis", Nature, 2021.
Questions for authors
- Could you add a lenz-autopilot baseline and an LLM-warm-start-plus-BO baseline on the reaction and LCBench tasks?
- In the full-system runs, how often does Sara take a suggestion unchanged, edit it, or propose a point of her own? How often does she reconfigure the problem, and does any of this correlate with final regret?
- Did the full Sara also recognize the synthetic functions, especially Ackley-10? If so, how much of her lead over Ax there comes from recognition?
- The original LLAMBO ranks sampled candidates with an LLM-based surrogate, but Appendix D describes only direct proposal. Which version did you run?
- Why is there no hybrid LLM–BO baseline? Those methods seem like the most natural comparison for the prior-informed tasks.
5 comments
W2: The prior-informed results are probably inflated [...] A real reaction dataset such as Shields et al. (2021) would be a much fairer test.
Thank you for the comment, and we agree with the observation. Note, however, that this knowledge is shared by all LLM baselines. They receive the same description and use the same LLM, so if the LLM could pin down the optimum from the name alone, all LLM-based baselines would one-shot it. They do not. LLAMBO and Centaur find the right regime quickly but fail to refine it, whereas Sara can use the probabilistic backend to fine-tune the solution towards the maximum yield. Leveraging such textbook knowledge is exactly what we wanted to highlight. Embedding it in standard BO is not trivial, while an LLM in the loop can use it directly, and combining it with a BO backend gives the best final performance. We also want to note that LCBench is a real benchmark where the prior is only generic (model family and parameter scales) and there is no textbook answer to recall as also shown by the memorization probes in Figure 11.
We agree that a real reaction dataset would be an interesting test. We did not include one because it is not central to our contribution. We want to show that the agent can effectively derive and use priors, not that it is necessarily a good chemist. The simulated tasks give us a controlled setting with known optima to effectively compare performance.
W3: The baselines also leave out the strongest competitors. [...]
We agree that a comparator combining a GP with language-model context is important. We added LLINBO-Justify [R1] on Covertype, Dionis, Heck and Buchwald, using the same base LLM as Sara (Opus 4.8 on high reasoning effort). LLINBO compares an LLM proposal against a GP-UCB alternative using its acceptance rule. We evaluated two initialization conditions: LLINBO with its own five LLM-generated opening points, and LLINBO supplied with each Sara run’s exact first five evaluated configurations and outcomes. Both conditions use ten seeds per task and the same total number of max. evaluations. The table reports best observed validation accuracy (LCBench) or yield (reactions), in percent, as median [25th, 75th percentile]; higher is better.
| Task | Sara | LLINBO | LLINBO: Sara prefix |
|---|---|---|---|
| Covertype | 74.20 [74.07, 74.25] | 73.57 [73.45, 73.65] | 73.56 [73.48, 73.65] |
| Dionis | 85.22 [85.07, 85.29] | 84.87 [84.75, 85.05] | 85.07 [84.94, 85.20] |
| Heck | 87.12 [81.51, 87.49] | 83.85 [81.51, 85.71] | 85.12 [84.64, 85.81] |
| Buchwald | 90.15 [89.98, 91.75] | 89.14 [88.30, 89.53] | 89.39 [89.04, 89.76] |
Sara achieves the highest median on all four tasks, though the interquartile ranges overlap on Dionis and Heck. Giving LLINBO Sara’s first five evaluations improves its median on three of the four tasks, which shows that the starting points resulting from Sara’s agentic multi-step reasoning are also helpful in this framework. It does not close the gap, however, suggesting that Sara’s advantage is not due to initialization alone. We thank the reviewer for suggesting this baseline and will add these results to the final version!
W4: The mid-run reformulation is shown as a single qualitative trajectory. [...] I'd also soften the claim in the abstract [...]
With this example we mainly wanted to showcase how the optimization can be steered effectively by a human (or another agent), and we therefore opted for a qualitative showcase. The key aspect is that the reformulation is done automatically by the agent from a high-level natural-language instruction, and no human has to operate the BO library. This covers both new information or context (“I asked my colleague and he said that for these types of models values around 0.1 usually work best”) and changed requirements as in the example. Having the agent do this reconfiguration through the BO backend also adds a level of robustness, since all existing observations are carried over consistently instead of being re-specified by hand. One could of course let an LLM operate a standard BO library on behalf of the human, but this would essentially arrive at what we propose. We agree that the wording in the abstract does not make this distinction clear, and we will rephrase it in the final version. We see this example as a demonstration of a capability and not as a performance claim, so we would keep it as a qualitative showcase.
W5: It would also help to report the cost per run next to Ax's in the main text. [...]
Yes, that is a good idea! We were under space constraints and therefore moved the discussion of computational cost to the appendix. We will state in the main text that running an optimization with an LLM in the loop comes with additional token costs (up to roughly 200k new tokens per run) and a small latency from deliberation. For applications where a single evaluation is expensive, such as wet-lab experiments or training runs, this overhead is small relative to the cost of an evaluation. You are right that the tradeoff should be explicit, and we will report the cost next to Ax’s.
W6: These presentation issues need clarifying. [...]
Thank you very much for catching these, we will fix the typos and clean up the metrics description in the final version. For clarification, we do not know the true optima for the GP samples and therefore report the best value there. We will label this in the main text and make the metric description consistent with Table 2.
Q1: Could you add a lenz-autopilot baseline and an LLM-warm-start-plus-BO baseline on the reaction and LCBench tasks?
Yes, we ran both (see W1) and will include the results in the final version.
Q2: In the full-system runs, how often does Sara take a suggestion unchanged, edit it, or propose a point of her own? How often does she reconfigure the problem, and does any of this correlate with final regret?
Thank you for the question. We characterize the agent’s behavior at the level of tool usage. Figure 5 shows how the different lenz calls evolve over normalized trial progress, and the full per-seed sequences of tool calls are provided in the appendix in Figures 24 and 25. These show that Sara combines the different backend calls in varying ways across seeds and tasks, and that she tends to shift towards local refinement around the incumbent in the later stages of a run. As discussed in W1, this refinement can be critical for reaching the best final performance.
To quantify this, we audited the ten Covertype runs, the task from W1 on which Sara still reaches a slightly higher median than the post-hoc heuristic. 25% of the evaluated configurations are lenz suggestions taken unchanged, while 75% are edited or proposed by Sara herself. In all ten seeds, the configuration that first reached the final best value was not an unchanged suggestion. Regarding reconfiguration, the only full reconfiguration of the problem is the mid-run reformulation example. There are also smaller reconfigurations at the level of the BO backend, namely changing the acquisition function and changing the bounds from which the next proposal is generated, which are visible as set-acqf and set-bounds calls in the traces. Sara made such a change persistently in 4/10 seeds for the bounds and in 4/10 seeds for the acquisition function for the Covertype runs. In addition, she used non-persistent changes for individual suggest calls, e.g., suggest --around the incumbent in 9/10 seeds, which allow local evaluations without changing the backend configuration, as well as acquisition override for an individual suggest call in 6/10 seeds.
Note, however, that these numbers are descriptive. Since Sara’s actions depend on all outcomes observed so far, attributing final regret to individual decisions is difficult, as it already is in standard BO. These numbers are also specific to Opus 4.8 at high reasoning effort with our system prompt. As discussed in Section G.4, the way Sara interacts with lenz differs between models, e.g., GPT 5.5 follows the trial-by-trial instruction more closely, so we would expect these counts to change with the underlying LLM.
We thank the reviewer for the detailed and constructive review. Since the review raises many points, our response is split across four comments: W1 here, W2 to W4 in the second, W5 to Q2 in the third, and the remaining questions in the fourth. New results are in W1 (controls isolating the agent’s contribution) and W3 (LLINBO as a hybrid baseline).
Thank you, this is a very helpful suggestion. To clarify the setup: lenz and Ax use the same defaults for the model, ie, a Matérn GP with the same hyperpriors and a logEI-based acquisition functions as defaults. The only difference is that Ax additionally winsorizes the output. As suggested, we also ran lenz in autopilot against Ax on Ackley-10D and Hartmann-6D and observe similar performance and convergence behavior.
(Median [25th, 75th percentile] over ten seeds; lower is better.)
We would also frame the comparison of Sara (no context) and Ax on the reaction tasks as essentially the same performance when averaged over the tasks. A pattern we do see is that Sara tends to make more suggest calls restricted to a region around the incumbent towards the end of a run (see also Figure 5), which lets Sara fine-tune the solution. This likely explains the slightly better final performance on the two yield tasks, since these have a single global optimum for which fine-tuning an incumbent is a good strategy.
We nevertheless agree that the ablation can isolate the agent’s contribution more cleanly. We therefore ran the suggested controls on the reaction and LCBench tasks: lenz on autopilot, and Sara’s first five evaluations followed by either lenz or Ax (Sara prefix + lenz/Ax), which separates the effect of good first points from the agent’s decisions during the run. All three reach a lower final median than Sara on all four tasks, as shown in the table below. Autopilot is considerably worse on the yield tasks. The Sara prefix recovers most of this gap on Dionis and the yield tasks, but not on Covertype, and still falls short on the yield tasks. The first points therefore explain a large part of the benefit, but not all of it. To understand where the remaining benefit comes from, we examined Sara’s behavior further and saw the local fine-tuning around the incumbent described above. To test this, we designed an additional, deliberately simple baseline that switches to local refinement around the incumbent after 50% of the evaluations. We devised it on Covertype and kept it fixed for the other three tasks. This heuristic matches or exceeds Sara’s median on all four tasks, which confirms where the remaining gain comes from. We want to be clear that this baseline was derived and constructed from Sara’s own behavior. The controls one would specify in advance, autopilot and a warm start followed by BO, both fall short, whereas the agent arrives at this strategy on its own without being told what to do.
(Final median minus Sara's final median in percentage points over ten seeds; positive means better than Sara.)
Finally, Centaur+, which uses the LLM for a warm start, improves markedly early on but plateaus at a worse final value than Sara. The bash-only and evaluate-only ablations additionally separate the surrogate backend from reasoning in context. We will include all new results in the final version.