- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper leverages GP to construct preference supervision for fine-tuning LLM to generate high-quality candidates. It evaluates the approach on five scientific sequence optimization benchmarks.
Critical strengths
- The motivation is sound: using LLMs as generative Bayesian policies is a natural choice for sequence optimization.
- The method is reasonable: GP-derived supervision addresses the scarcity of data for LLM fine-tuning.
Critical weaknesses
Although the authors discuss and compare against ToSFiT, the two approaches appear similar in motivation and methodological design. The authors should clarify the advantages of their method over ToSFiT at the methodological level.
Questions for authors
- Could the authors provide a reproducibility statement?
- Could the authors evaluate the method’s sensitivity to prompts?
- Could the authors evaluate the method on SQL query optimization?
- Could the proposed algorithm support recursive self-improvement?
- Could the authors compare against LLM-assisted Bayesian optimization methods such as LLAMBO, LGBO, and LABO?
I would be willing to raise my score if the authors adequately address these concerns.
4 comments
Response to Reviewer 6TYZ (2/4)
Q1. Reproducibility statement
The components are already in the paper: the algorithm box, the configuration table (identical on all tasks), the benchmarks, objectives and cold-start pools, the baselines, constrained decoding and the exact prompts are all given in the appendix. What was missing was a statement pointing to each of them, which we have added, together with the seed-to-seed noise floor so readers can tell which differences between our numbers are meaningful. All base models and objectives are public, so the setup is reconstructible from the paper alone; code and analysis scripts are released with the final version.
Q2. Sensitivity to prompts
Prompt sensitivity can be posed on the two Qwen tasks, where the prompt lists the parameter space and asks for one candidate. On the other four the generator is gpt2-zinc-87M or PeptideGPT, BOS-conditioned samplers with no chat template and no parameter space to describe, so there is no prompt to vary. We measured it on Catechol: four paraphrases × 20 seeds, varying wording rather than output format, with variant 0 byte-identical to the published prompt.
The four prompts differ in exactly two lines: the opening framing and the closing output rule. The parameter block between them, which lists the solvents, SolventB%, residence time and temperature, is byte-identical across all four, so the spread below comes from framing alone and not from different information. The opening line of each (full prompts in the appendix):
| variant | opening framing line | best-so-far |
|---|---|---|
| 0 (published) | "You are an expert optimizer. Your goal is to suggest a candidate to maximize the objective." | 2.743 ± 0.073 |
| 1 (terse) | "Propose one candidate that maximizes the objective." | 2.742 ± 0.045 |
| 2 (verbose) | "You are a research scientist planning the next experiment in a costly design campaign. Propose the single candidate you believe will give the highest objective value." | 2.678 ± 0.068 |
| 3 (domain-neutral) | "Select one input for a function whose output you are trying to make as large as possible." | 2.559 ± 0.130 |
No pair differs significantly (all p ≥ 0.20; ANOVA p = 0.38): 0.18 separates best and worst. Rephrasing is essentially free: dropping the persona changes the result by 0.001 and tripling the verbosity by 0.065. The largest numerical difference arises when the domain framing is removed. Variant 3, the only one that stops describing the task as an experiment, sits 0.18 below variant 0 with the widest seed-to-seed spread, but that difference is not statistically significant at 20 seeds. We therefore read the results as suggesting that task framing may matter, while prompt sensitivity is not something we can demonstrate across these variants. Variant 3's parameter block still names the solvents, so any effect comes from framing rather than from information.
Q3. SQL query optimization
We would rather decline this one than answer it badly. SQL plan optimization is a different problem class: methods there train on one set of query shapes and are scored on held-out ones, so the object optimized is a policy that generalizes across queries, whereas every task in this paper optimizes a single objective within a single budget. A number produced under our protocol would not be comparable to that literature.
We did extend domain coverage during this rebuttal, where the comparison is exact. GB1 (Wu et al., 2016) is all 149,361 measured four-site variants of protein G domain B1, so every reward is a lookup and the optimum is known. GGOLLuM reaches 0.707 ± 0.035 within 100 iterations against 0.605 ± 0.026 for classical BO (p = 0.03), 0.503 for LLM (ICL), 0.437 for GOLLuM and 0.311 for the untrained LLM, on a general-purpose Qwen2.5-7B backbone with no protein-specific pretraining.
Q4. Recursive self-improvement
In a limited but measurable sense the loop already is: the actor's samples become its next training set, the critic refits on what the actor chose, and Section 6.4 shows the critic's ranking accuracy on the actor's own proposals rising over a run. The grounding is what stops it drifting: every update is anchored to measured points, so the critic cannot reinforce a region the oracle has not confirmed. What we do not do is modify the algorithm or objective itself, which we regard as future work.
Response to Reviewer 6TYZ (3/4)
Q5. LLAMBO, LGBO and LABO
LLAMBO, LGBO and LABO are implemented and run.
On the backbone. We ran both on Qwen2.5-7B base, matching our backbone so the comparison isolates method from model, and also on Qwen2.5-7B-Instruct. We headline the base arms because they are the stronger result for both baselines, not because they are the faithful ones.
LLAMBO (Liu et al., ICLR 2024)
We transcribed their discriminative in-context surrogate into our stack. The comparison is clean because their components map onto ours: zero-shot warmstarting is disabled, since every method starts from the identical cold start, and their history-conditioned sampler is our LLM (ICL) baseline, reused unchanged. The only thing the baseline adds is the in-context surrogate.
| task | arm | best-so-far | surrogate adds | p |
|---|---|---|---|---|
| Catechol | LLM (ICL), their sampler without the surrogate | 1.930 ± 0.106 | n/a | n/a |
| as released | 2.139 ± 0.124 | +0.209 | 0.30 | |
| format-forced | 2.510 ± 0.157 | +0.580 | 0.0097 | |
| GGOLLuM | 2.772 ± 0.055 | n/a | n/a | |
| GB1 | LLM (ICL), their sampler without the surrogate | 0.503 ± 0.034 | n/a | n/a |
| as released | 0.467 ± 0.028 | −0.037 | 0.39 | |
| rescaled | 0.525 ± 0.030 | +0.022 | 0.54 | |
| rescaled + forced | 0.537 ± 0.037 | +0.034 | 0.49 | |
| GGOLLuM | 0.707 ± 0.035 | n/a | n/a |
20 seeds per cell; "surrogate adds" is the seed-paired difference against LLAMBO's own history-conditioned sampler, which is our LLM (ICL) baseline, so that column isolates the in-context surrogate and nothing else.
The surrogate's contribution is task-dependent. On GB1 it adds nothing: −0.037 to +0.034 across the three arms, every p ≥ 0.39, including the arm whose surrogate answers every query. On Catechol it does real work with +0.580 at p = 0.0097 in the format-forced arm, and +0.209 (p = 0.30) as released. This split is consistent with LLAMBO's own positioning as a hyperparameter-optimization method: it helps on a small, well-described parameter space and not on a fully measured sequence landscape. GGOLLuM leads on both (2.772 and 0.707).
The surrogate's rank correlation with the true objective, averaged over a run, is 0.54 (as released) and 0.66 (format-forced) on Catechol, and 0.22 / 0.18 / 0.36 on GB1, with parse rates of 0.24 and 0.97 on Catechol and 0.17 / 0.09 / 1.00 on GB1. Where it ranks best (ρ = 0.66) it only matches our critic with a frozen kernel (0.66) and falls short of our learned deep kernel (0.81).
The Instruct runs. Parse rate rises to 1.000 with zero invalid candidates, yet performance collapses, because the optimizer starves before the budget is spent:
| arm | best-so-far | evaluations consumed |
|---|---|---|
| Catechol, base + forced | 2.510 ± 0.157 | 110/110, all 20 seeds |
| Catechol, Instruct | 0.647 ± 0.075 | median 95; 7/20 reached 110 |
| GB1, base, rescaled + forced | 0.537 ± 0.037 | 110/110, all 20 seeds |
| GB1, Instruct | 0.145 ± 0.033 | median 22; 0/20 reached 110 |
Those values are not like-for-like. 87.5% of its GB1 proposals duplicate earlier ones (72.3% on Catechol) against 9.3% and 6.1% for base, novel-candidate rate falls 0.70 → 0.08, and surrogate ρ drops to roughly zero (−0.02 GB1, 0.04 Catechol). The model becomes syntactically well-behaved but highly repetitive.
Two added arms, both of which can only help the baseline: their strict parser accepts 28–38% of base
draws on Catechol, hence a format-constrained arm; and on GB1 their six-decimal serialization renders
rewards of order 1e-4 as 0.000300, so the model answers 0.000000 and EI goes flat, hence a rescaled
arm. Applicability is limited to Catechol and GB1.
LGBO (Yuan et al., ICLR 2026)
LGBO is now cited in Section 2 as an LLM prior that reshapes the acquisition, and run under our protocol. Its mechanism is a region-lifted mean tilt: the LLM returns a point or region plus a confidence c, and the posterior mean shifts along Cov(X, G) by exactly c posterior standard deviations, covariance unchanged, with no free scale parameter.
Because LGBO is a prior injected into standard BO, we gave it our own BO baseline as its starting point with identical featurisation, kernel, 512-candidate pool and acquisition, so the only difference is the LLM prior, and added the same loop with the tilt off as a seed-paired control.
| task | arm | best-so-far | paired diff vs no tilt | seeds with identical final best-so-far |
|---|---|---|---|---|
| Catechol | no tilt (control) | 2.028 ± 0.061 | n/a | n/a |
| as released | 2.003 ± 0.061 | −0.025 (p = 0.14) | 80% | |
| format-forced | 2.073 ± 0.038 | +0.045 (p = 0.47) | 75% | |
| GGOLLuM | 2.772 ± 0.055 | n/a | n/a | |
| GB1 | no tilt (control) | 0.648 ± 0.029 | n/a | n/a |
| as released | 0.639 ± 0.022 | +0.003 (p = 0.94) | 21% | |
| format-forced | 0.653 ± 0.023 | +0.006 (p = 0.86) | 25% | |
| GGOLLuM | 0.707 ± 0.035 | n/a | n/a |
Response to Reviewer 6TYZ (4/4)
The LLM prior has no measurable effect on either task. Every seed-paired difference is within noise, and on Catechol most seeds finish on precisely the value they reach with no prior. The tilt changes which candidate the acquisition selects in only 5–15% of iterations on Catechol and ~4–7% on GB1, because a preference moves the decision only when the pool holds candidates the GP already considers strongly correlated with the preferred region.
Paired differences are computed over the seeds both arms completed, so on GB1 they need not equal the difference of the column means.
LGBO accordingly tracks the no-tilt control it sits on rather than closing any of the gap to GGOLLuM, and is indifferent to backbone (2.003 vs 1.969; 0.639 vs 0.630) even as its parse rate improves from 0.14 to 0.50 and 0.65 to 0.94. Letting the prior be heard far more often changes nothing. An LLM prior on the surrogate is not sufficient; what distinguishes our method is that the generator itself moves. Two implementation deviations are disclosed in the appendix.
LABO (Chen et al., ICML 2026)
LABO is a dual-fidelity method rather than a competitor at the same fidelity: it treats LLM predictions as a cheap low-fidelity source, fits a Kennedy–O'Hagan joint GP, and gates when a real experiment is worth running. We ran it at 20 seeds, budget-matched at 110 real-fidelity evaluations and paired per seed with every other method on identical cold starts (its 50-point LHS warm-up is low-fidelity only and spends no real budget):
| method | Catechol | GB1 |
|---|---|---|
| LABO | 1.967 ± 0.061 | 0.613 ± 0.032 |
| classical BO | 1.907 ± 0.087 | 0.605 ± 0.026 |
| GOLLuM | 2.308 ± 0.044 | 0.437 ± 0.035 |
| GGOLLuM | 2.772 ± 0.055 | 0.707 ± 0.035 |
LABO lands at classical-BO parity on both tasks (paired p = 0.60 and 0.84). It is not a weak baseline: on GB1 it beats GOLLuM by +0.176 (p = 0.001), the gated joint GP recovering what GOLLuM's deep kernel does not on that landscape. GGOLLuM is above it by +0.805 on Catechol (paired p = 3e-9) and +0.093 on GB1 (paired p = 0.033).
Your review asked us to separate the method from ToSFiT at the methodological level and to position it against LLM-assisted BO. Both are now answered with experiments: the ToSFiT comparison is a controlled ablation in which every other component is ours, and all three methods you named are now run: an LLM prior on the acquisition (LGBO) is a measurable no-op on both tasks, an LLM surrogate (LLAMBO) helps only on the small parameter space and not on the measured sequence landscape, and a dual-fidelity joint GP (LABO) lands at classical-BO parity. None closes the gap to GGOLLuM, which is our evidence that moving the generator is what matters. With the reproducibility statement, the prompt study and the new GB1 benchmark, we believe your concerns are addressed and hope you will consider raising your score. If any part is unconvincing, tell us during the discussion period and we will run what is needed.
Response to Reviewer 6TYZ (1/4)
We thank the reviewer for a precise review. The ToSFiT question prompted two experiments, and the request for LLM-assisted BO baselines produced the largest addition to this revision: LLAMBO, LGBO and LABO are all implemented and run under our protocol.
The four key points.
Budgets are 100 BO iterations after a shared 10-point cold start; every number is absolute best-so-far and every ± a standard error over 20 seeds.
W1. Advantages over ToSFiT at the methodological level
ToSFiT is the closest prior work to ours and addresses the same problem, which is why we compare against it directly rather than only citing it. These details were distributed across a figure caption, the ablation section and the appendix; we consolidate them here for clarity. The paper adapts ToSFiT to the low-evaluation regime before comparing: same deep-kernel critic (Section 2), tuned over learning rate, optimizer and updates per iteration, sample-matched at 16 samples per iteration (Figure 1 caption and the ToSFiT appendix), with two controls already reported: four times the samples does not close the gap, and replacing its advantage with our UCB acquisition changes nothing. It also already reports the result that most directly answers your question: with our critic, ToSFiT stays at or below GOLLuM, the same critic used only to select among the untrained model's samples, with no generator training at all (2.17 vs 2.31 on Catechol, 0.62 vs 0.80 on SMILES-DRD2).
The ToSFiT in our Table 1 is therefore not ToSFiT as published. Every component except the training signal is ours:
What remains is how the GP posterior becomes a training signal. ToSFiT reduces it to a scalar pseudo-reward and follows that gradient; we construct pairwise preferences over the generated candidates and k measured anchors, and optimize IPO. The signal is therefore relative rather than a scalar read off the acquisition, and partly tied to points whose value is known rather than predicted: removing the anchors lets the policy collapse onto its own samples (pair-pool ablation).
New in this revision: neither substitution disadvantages ToSFiT. Replacing its published critic with our deep-kernel critic increases mean performance on all five benchmarks (everything else fixed, 20 seeds):
ToSFiT as published, with full fine-tuning and a frozen critic, gives 0.51-0.53 on SMILES-DRD2 across three learning rates, against 0.513 for that same critic under LoRA, and all of these sit below the 0.62 reported for it in Table 1. The Table 1 comparison therefore isolates the training signal: the critic and adaptation setup are matched, while the policy-gradient update is replaced by IPO. We attribute that to how the Bayesian signal is converted into a policy update rather than to the quality of the critic or the backbone, and we read it as a statement about the ~100-evaluation regime rather than about ToSFiT in the setting it was designed for: one scope limit remains, now stated, in that their published protein experiment uses a 1,000-evaluation budget against our 100.